bug(ios): Host-page error - cannot find the keyboard JS file

[sentry]

Sentry Issue: KEYMAN-WEB-M9

Error: Cannot find the Ethiopic (Modern Geʾez) keyboard for Geez at file:///private/var/mobile/Containers/Shared/AppGroup/1000806A-329F-4962-AC58-637FCFFA2874/keyman/keyboards/gff_ethiopic/gff_ethiopic.js.
  at new KeyboardMissingError (@keymanapp/keyman/common/web/keyboard-processor/src/keyboards/keyboardLoaderBase.ts:34:5)
  at StubBasedErrorBuilder.missingError (@keymanapp/keyman/common/web/keyboard-processor/src/keyboards/keyboardLoaderBase.ts:67:16)
  at script.onerror (@keymanapp/keyman/common/web/keyboard-processor/src/keyboards/loaders/domKeyboardLoader.ts:50:37)

[jahorton]

I've been investigating this one the past couple of days. I won't say that I've definitively identified what's going on, but I've got lots of notes that should prove useful in reaching some sort of resolution... eventually.

The high number of events compared to user counts is due to the fact that once the error triggers, it tends to loop. The host-page fails to load the keyboard correctly, leading to a reset that tries again... for the same file it already failed to load the previous time. There are no single, isolated cases - there's always a tightly-timed loop of several events when the issue occurs.

Also of note: a separate Sentry event would have occurred if the file did not exist at the time that the Swift-side setKeyboard call were triggered, such as with this set: https://keyman.sentry.io/share/issue/534f01f1d42a49c4a35692bff26ad830/

This indicates that a protection against this exact sort of behavior, which we added in #5178, triggered:

keyman/ios/engine/KMEI/KeymanEngine/Classes/Keyboard/KeymanWebViewController.swift

Lines 304 to 309 in 60b00e8

	// Warning: without special handling, any `guard` that fails here can trigger an
	// infinite keyboard reload, as the keyboard page itself will note that the keyboard
	// failed to initialize properly.
	guard FileManager.default.fileExists(atPath: fileURL.path) else {
	let event = Sentry.Event(level: .error)
	let errorMessage = "File missing for keyboard"

(To be clear, the Sentry link above within this comment is titled "File missing for keyboard", clearly generated from the block permalinked above - as per text and stacktrace.)

So, at the time of the Swift-side setKeyboard, the file:

1. does exist

One potential avenue I considered - what if the keyboard JS files somehow got permissions-locked? As it turns out, we'd be seeing a different error for such a case. I made an experimental setup to test this, and both Firefox and Safari (and thus, likely iOS Safari / the WKWebView) would return a script error, rather than a "file not found" sort of error, if this were due to file permissions. Thus...

2. is not permissions-locked

---

When investigating this error along with a few others, I did run into the following: Apple Secure Coding Guide: Race Conditions and Secure File Operations

One of their recommendations is to not check for the existence of a file before triggering an operation - just do it and catch any errors that result. The reason: concurrency issues could cause the file to go missing between the check and actual file-load operation.

Note: an iOS WKWebView is explicitly stated to run in a separate process, let alone separate thread. If there's anything within the app or keyboard control-flows that could cause the file to be, say... deleted, between the setKeyboard call and the WebView's actual attempt to load the file, such a scenario could definitely trigger this error.

Of course... "that's not possible" is a natural first reaction. At no point do our keyboard-oriented classes do any file-write or file-delete operations. That said, there are within-app operations that can delete keyboard files - we permit this within the Settings menu. The Settings menu can also trigger updates, which do overwrite the updated file. (Also, Settings operations usually trigger a keyboard-reload.)

An alternate avenue I had in mind: this does feel parallel in a way to both #5178 and #7412. I also noted that a large portion of the events, likely near a majority, are centered around sil_euro_latin - our default keyboard. Could this be a case where a power-user has decided to remove the current default keyboard at a point capable of triggering a file-based race condition?

I wonder if the fact that we see both variants - Swift side and Web side - is indicative of this being a race condition? The exact timing of when the file disappears could lead... whatever triggers this... to trigger both errors, with it being a matter of chance which version arises per instance.

There are a few other keyboards that appear in the logs, though:

• Ethiopic (Ge'ez)
• SIL IPA
• Thok Nath

There's also a notable set of events within the logs - a single user had this event-loop trigger for the same keyboard 2 times occurring two weeks apart. (For Ethiopic / Ge'ez)

---

Finally, known limitations:

The host-page logging has no way of indicating its host app or context. We do not know if this is from the Keyman app, the SWKeyboard extension, or from FirstVoices. (Granted, we wouldn't see African-language keyboards from the FV app.)

[jahorton]

Of further note: the host page currently does not attempt to catch the error in a manner by which it could send a message about the keyboard file being missing. There is an existing mechanism we could extend in order to create a failsafe for this purpose, though.

[mcdurdin]

The host-page logging has no way of indicating its host app or context. We do not know if this is from the Keyman app, the SWKeyboard extension, or from FirstVoices. (Granted, we wouldn't see African-language keyboards from the FV app.)

Surely this should be easy to solve though -- we know from the app where we are so we could pass a context param into the host page?

[jahorton]

Surely this should be easy to solve though -- we know from the app where we are so we could pass a context param into the host page?

Was already working on this; seemed like the obvious first step. Did it as a breadcrumb at first, but then realized a better spot to put it. (See #13105.)

[jahorton]

That said, there are within-app operations that can delete keyboard files - we permit this within the Settings menu.

Pursuing this avenue: there are no notable breadcrumbs in the most recent entry for the "File missing for keyboard" error - and those should exist if the file just went missing due to removeKeyboard.

Said error also does not appear to loop - each occurrence seems to have a distinct timestamp.

[jahorton]

Process of elimination: I'm fairly sure that means ResourceFileManager.copyWithOverwrite, which doesn't breadcrumb, and is part of the package download/install process, is the remaining option for something that could have triggered this scenario.

If so... perhaps triggering an update and quickly backing out of settings has a chance to trigger the race condition if the post-Settings keyboard lines up "just right" with a downloaded update? I'd want to spend time lining up the timeline of event triggers vs keyboard updates to see if there's any merit to the idea before moving forward along these lines, though.