Assemblies tracking and storage

[emiliorighi]

Right now the TSV is recreated from scratch weekly and stored as a GH release: even if just one new assembly is present in NCBI we end up recomputing everything:

• Implement a way to keep track of what has already been collected and what's new
• Look for other possible compressed formats that can be directly stored in the GH repo instead of the GH release (parquet, sqllite, etc..)
• Split assemblies into RefSeq and Genbank
• Keep adding the new assemblies (and related aliases) in the existing files instead of recreating everything from scratch
• Add commit comment when the job is finished with the number of the new assemblies found
• Implement an index of the collected assemblies for faster retrieval/lookup

[Max25R]

Still open and needed:

• Track which assemblies have been collected vs which are new.
• Append-only updates instead of full weekly rebuild.
• Commit/Release note with new-assembly count per run.

Done/covered:

• Index of collected assemblies = single index on accession is in the v1 SQLite. Detection queries do full scans (sub-second on 3M rows) so additional indexes aren't needed for now.

changed by v2:

• Compressed format stored in the repo instead of Releases: we went the other way. v2 ships only the TSV via Releases (Update TSV/SQL data #7) and builds the SQLite client-side (SQL db creation on cli startup? #6, done). So we're keeping data out of the repo entirely.
• Split assemblies into RefSeq and GenBank: Update TSV/SQL data #7 does the opposite: merges paired GenBank+RefSeq into one row. Closing this sub-item in favor of Update TSV/SQL data #7.

The incremental-update piece still important, current weekly run takes ~4.5h re-fetching all 66k assemblies; an incremental version would run in minutes.