rfs rearchitecture (Part 1)

[xmonader]

Files operations

• Should be able to store a file deduped into the server to save server's storage (chunking into blocks and checking for the existence of the chunks should happen on the client to save upload bandwidth).
  • the client should be able to keep track of the file hash the relation with all of its blocks hashes
  • the server should be able to keep track of the file hash the relation with all of its blocks hashes (for faster calcuation)
• Should be able to download a file by its hash
• Should be able to verify if a hash exists on the server or not

Directory operations

• should be able to push a directory and all of its files to the server and get its flist's hash
• should be able to download a directory by its flist's hash

Flist operations

#105

• Should be able to inspect(preview) flist by its hash (on the server) or it could be a path locally (LOW PRIO)
  • rfs flist inspect flistpath
  • rfs flist inspect flisthash --server-url
• Should be able to tree an flist by its hash (on the server) or it could be a path locally. (LOW PRIO)
  • rfs flist tree flistpath
  • rfs flist tree flisthash --server-url

Containers (Docker/Nerdctl)

• Should be able to convert docker images to flists with docker convert-to-flist subcommand
• Should be able to import an flist into docker images (lowest priority)
• Should be able to convert nerdctl images to flists with nerdctl convert-to-flist subcommand (LOW PRIO)
• Should be able to import an flist into nerdctl images (lowest priority) (LOW PRIO)

Website operations

• Should be able to publish a site on the server on /websites/:flisthash by using its flist hash (static sites only)
  • the publishing is similar to uploading a directory, but gives you the access URL

Syncing operations

• Should be able to sync between to servers by hash (a block, a file, an flist)
• Should be able to sync all of the source server's blocks to the target's blocks (iterate over blocks paginated from the source maybe and then uploaded to the target)

FUSE

• should be able to mount an flist with a fuse filesystem with on demand access.

WebDAV

#102

• should be able to use a WebDAV client to navigate a directory e.g $serverurl/webdav/FLIST_HASH (LOWEST PRIO)

[rawdaGastan]

Progress

• Upload commands (can upload a file by its name or hash - also can be a block hash) file is split into blocks

• file hash has a relation to its block hashes in the server database

• Download command to be able to retrieve blocks and files

• Exist command to check the existence of a file by its path or hash - also can be a block hash)

• Upload-dir command uploads a directory to the server - can create an flist too

• Download-dir downloads a dir from the server using its flist hash

• docker command converts docker images to flists

• website-publish command to upload a directory and create and flist for it to be able to serve files using the flist hash through the server

• sync to be able to sync blocks between 2 servers can be a block or file hash - all

[rawdaGastan]

In progress

• e2e tests

[rawdaGastan]

Update

• e2e tests completed

[rawdaGastan]

In progress

• dnslink for publishing websites

[xmonader]

Did we support the flist subcommand or not yet?

Note: I merged the PR, we can continue the remaining of the ticket in a new PR.