Blocked puts when running Khepri on ephemeral fly.io storage

[KagemniKarimu]

What does not work?

Context:

We are running a Phoenix app on Fly.io, using Khepri as a single-node Raft-backed key-value store. The app runs as the nobody user and stores Khepri data in a mounted Fly.io volume.

✅ When It Works:

Data directory: /app/priv/khepri_data (inside the container’s ephemeral FS, not on a fly.io volume)
Behavior: :khepri.start/3 and subsequent :khepri.put/3 succeed
WAL files are written and visible in that ephemeral directory, but obviously don't persist

❌ When It Fails:

Data directory: /app/data/khepri_data (a Fly volume mount at /app/data)

Behavior:

• :khepri.start/3 succeeds and logs normal Raft startup
• But the first :khepri.put/3 call hangs indefinitely
• No crash, timeout, or error is reported

WAL files are written (which suggests at least partial Raft write starts)

Permissions:

• The mounted volume and target directory are owned by nobody:nogroup
• File.mkdir_p!() succeeds
• File.stat/1 shows read-write access

Expected behavior

❌ The behavior should not be:

• Hanging or deadlocking during put/3
• Stalling indefinitely on first write
• Inconsistent WAL state (e.g., WAL file appears but no response from Raft)
• Sensitive to whether the data directory is on an ephemeral vs. mounted disk — as long as it is writable and fsync-capable

How to reproduce

To reproduce, simply setup khepri, point the start_dir at a fly.io volume path, deploy fly.io app and try to put value in khepri.

🧪 Diagnostic Findings:

• Issue does not occur in development (at ALL) or in production when using ephemeral disk
• Confirmed that environment variable KHEPRI_DATA_DIR is set properly and the correct path is used
• Hangs are specific to writing into a Fly-mounted volume
• I have heard Fly volumes use a network filesystem (virtiofs) and may have fsync semantics that block indefinitely under some conditions ? This slightly contradicts my own evidence of the following:

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vdc       ext4  974M  116K  908M   1% /app/data

🧰 Suggested Investigation Points:

• Is Khepri waiting on an unacknowledged WAL fsync?
• Is :ra sensitive to certain fsync delays on non-ext4 filesystems (e.g. virtiofs)?
• Should Khepri expose configurable fsync / sync_interval behavior for low-reliability disks?

[KagemniKarimu]

I was able to resolve this by deleting the khepri_data folder manually (rm -rf) and letting the db restore it. It turns out that it was somehow corrupted.

[michaelklishin]

@KagemniKarimu it could have been a lot of things and you have destroyed all evidence by doing that.

For example, a Raft cluster (including Khepri) that does not have an online majority will not be able to perform any writes. This is how Raft works, and a fundamental limitation of every Raft-based system out there, have an online majority or you won't have any availability.

Removing node's data and restarting a node messes with cluster members and has destroyed all evidence.

In any case, as stated below, running Khepri or any Raft-based system on ephemeral storage directly contradicts another fundamental Raft assumption: that the storage is durable, including between node restarts and such.

Sounds like you have managed to violate two fundamental assumptions of Raft out of two or three, depending on who you ask, and then "fixed" it by wiping a node entirely.

[michaelklishin]

@KagemniKarimu this should have been a discussion, you haven't provided a whole lot of details and removing a data directory is not a "fix".

Running Khepri or any other Raft-based system on ephemeral storage is not only a terrible idea, it directly violates the assumptions in the Raft paper. Use durable storage and make sure the cluster always has an online majority, or use a different storage option entirely.

[KagemniKarimu]

Thanks definitively for the insights. I don't know what in my ignorance has offended you, but I appreciate the education.

The "terrible idea" came from a last-shot effort to debug a problem quickly and it got me back running immediately. Ephemeral storage was never my original plan - moreso a step I tried to debug my Khepri configuration and see if it was working on the server at all. Once I was able to diagnose that it was functional, I filed the issue and moved things back to my durable storage volume.

My "fix" was just a quick way to get me back running and my "destorying of evidence" was a move of desperation and lack of knowledge, not malice. Much of what I was doing was not respecting Khepri's defined APIs and potentially writing corrupted paths. I closed the issue so that it would be clear that I recognize things were likely user error (and not an issue with Khepri itself).

In any event, thanks for the software.