Too Many Open Files - Causes Container (and Docker host) to Stop Functioning

[gonenutsbrb]

Output of rest-server --version

rest-server version rest-server 0.14.0 compiled with go1.24.3 on linux/amd64

Problem description / Steps to reproduce

I am getting filesystem lockout errors when running a 'check' function from my restic frontend. The errors look something like this from the docker logs:

ERROR: open /data/locks/f582ae9a31b06ac2721f860effb2e6da27eb9611424b806708d4f141a5d6799b.rest-server-temp3847862705250625670: too many open files in system ERROR: open /data/locks/f582ae9a31b06ac2721f860effb2e6da27eb9611424b806708d4f141a5d6799b.rest-server-temp8186166381684637943: too many open files in system ERROR: open /data/locks/f582ae9a31b06ac2721f860effb2e6da27eb9611424b806708d4f141a5d6799b.rest-server-temp1594954907149831599: too many open files in system ERROR: open /data/locks/f582ae9a31b06ac2721f860effb2e6da27eb9611424b806708d4f141a5d6799b.rest-server-temp491155930888957836: too many open files in system ERROR: lstat /data/locks/d32b409c3ea7bf22fd9c3c96749380d0d0a4fdbdaadfbbffcca5d74ff3aa7d89.rest-server-temp2286757414886870265: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system

I am currently running a prune on the repo now to see if I can clean out some of the unnecessary files to make the check easier, but we'll see. I will start experimenting with ulimit in a bit from inside the container, though this issue seems to affect the host as well.

Looks like the total amount of files in the repo currently exceeds 2M:

/PrimaryPool/RESTServer/data# find . -type f | wc -l
2090515

Expected behavior

Server should not reach a filesystem max limit with 40TB of data.

Actual behavior

Container ceases to function until restarted and throws the above errors.

Do you have any idea what may have caused this?

I finally finished the initial backup for a large amount of data (around 40TB), and went to run a check on the repo.

I think this may be along the lines of #215, where large datasets for these backups are going to run into some default filesystem issues. Though I'm not sure if deeper folder structure would help this particular problem.

Did rest-server help you today? Did it make you happy in any way?

Rest-server has been a godsend for acting as an endpoint on the large backup end of things. I look forward to figuring this out, since the backups already work and I can't imagine that this is going to be too complicated, though it will likely require adjustments on the host (and maybe the...super-host(?) docker is running on a VM, on a proxmox instance...)

[gonenutsbrb]

Oh boy, this is definitely going to require some changes on the host from what it looks like...

[MichaelEischer]

Can you check using lsof or in /proc/<pid of rest-server>/fd/ how many open files rest-server actually has?

For such a large repository, you want to pass --pack-size 128 to restic.

I've found a few potential file descriptor leaks in #385 but most likely those aren't relevant for your use case.