Lazy snv access

[FlorianTrigodet]

Adds an index on variable_nucleotides(split_name) and a new anvio/snvops.py module with SNVAccessor, a small lazy class that fetches SNV slices per-split or per-region instead of the whole table.

• v42_to_v43.py migration: creates the index on existing profile DBs.
• ProfileDatabase.touch(): creates the index for newly-built profile DBs.
• SNVAccessor: picklable accessor with get_snvs_for_splits(), get_snvs_for_region(), and cached SELECT DISTINCT helpers. Returns DataFrames.

But WHY?

The variable_nucleotides table is registered as indexed=True in tables/init.py but no column index has ever existed, so WHERE split_name = ... queries are FULL table scans. On (VERY) large profile DBs (100M–800M SNV rows) this makes the current pattern unusable (particularly for anvi-report-dgrs in the DGR_Finder branch). With the index and SNVAccessor, future consumers can fetch only what they need from SQLite directly.

SNVAccessor has no master-side caller yet. But we will use it in the DGR_Finder branch.

Is it a good idea?

I have been following an idea by Claude here, to better process SNV in the DGR branch. The indexing sounds nice, but the migration is painful for large dbs: the migration will take a long time for large db and grow their disk space. @meren, let me know if that is a bad idea or if I'm missing something.

Also, is anvio/snvops.py the right place for SNVAccessor? Should this instead extend variabilityops.py, or live in ProfileDatabase?

Happy to make changes now, I'd rather get this right once than re-migrate later :)

[meren]

Hey @FlorianTrigodet,

The indexing would have been great, but it does come with a huge cost. And in this case, I don't think this is the way to go as we should not dramatically increase the size of our profile database files everwhere with a mandatory migration step.

I first thought we could maintain indexes on-demand as separate files, but one of the biggest limitations of SQLite is that it does not allow tables and their indexes to be in separate files. I imagine we can solve that issue with workarounds (i.e., companion .idx files that contain indexed tables and so on), but I can see how it would snowball from there and turn into a complete design and maintenance mess.

So what do we do. I think there is an intermediate solution:

We implement a new program, anvi-index-table that literally indexes any given table in anvi'o (with a --drop-index flag so we can reverse it (so db files can be shared in smaller size)). We still have lazy access class in snvops.py for your application, but when the class is instantiated, it complains and ultimately asks the user to "run anvi-index-table with the following parameters".

This way reversible table indexing is an option for anyone through an anvi'o program, and over time we can add more classes to work with indexed tables for performance.

My 2 cents.

[FlorianTrigodet]

I'll revert the version change and make a anvi-index-table. I'll make a set of predefined indexing options (table, column) so a user cannot create an index for ANY column and create a monstrous db. And also the possibility to overwrite the column to index with a just-do-it

[FlorianTrigodet]

Reverted migration and automatic indexing.
Now we have anvi-index-table with a set of defined indexable table/column which can be asked with --list and a --just-do-it overwrite the default list and lets anyone index what ever table/column(s).
And SNVAccessor check for the index and if the db is large (>10Gb) it stops and give the user the anvi-index-table command to run. But for small dbs, SNVAccessor just warn that it could be slow (and that anvi-index-table is available) but otherwise proceed with full db scan.
I updated the test to include anvi-profile-table. Looks like it is working 🎉