Fix Docker build (Ubuntu 24.04 / PG17 / setuptools-scm) and restructure docs by bsgiles73 · Pull Request #280 · biocommons/uta

bsgiles73 · 2026-06-27T18:35:35Z

Body:

Summary

Upgrade build environment to Ubuntu 24.04, Python 3.12, PostgreSQL 17, and fix the setuptools-scm Docker build failure
Replace deprecated pkg_resources with importlib stdlib equivalents
Fall back to curl for NCBI paths that are not valid rsync modules
Simplify ncbi-parse-gbff by removing the NM/NR pre-filter
Split the README into docs/ (development, update runbook, migrations), document the SeqRepo export/snapshot/publish steps, and move mito/manual-splign loads into a "Non-standard steps" section

Notes

Documents the new uta service snapshot bind-mount prerequisite and fixes a $UTA_ETL_OLD_UTA_IMAGE_TAG typo in the update procedure

- Bump base image from ubuntu:22.04 to ubuntu:24.04 and Python 3.10 to 3.12 - Add PostgreSQL PGDG apt repo to install postgresql-client-17 (ubuntu:24.04 ships only client-16; the UTA build target is PG 17) - Set PIP_BREAK_SYSTEM_PACKAGES=1, required by Ubuntu 24.04 externally-managed Python to allow system-wide pip installs inside a container - Copy README.md into the build context so setuptools-scm can read it during pip install -e .; also bump setuptools_scm pin from ==1.11.1 to >=7.0 to fix the Docker build failure that prompted this branch - Bind-mount the old UTA .pgd.gz snapshot into the old-uta compose service so the file is available at container startup without baking it into the image Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

pkg_resources is deprecated in favor of importlib.metadata and importlib.resources, which are available in Python 3.9+ stdlib. - uta/__init__.py: use importlib.metadata.version() instead of pkg_resources.get_distribution() to retrieve the package version - uta/cli.py: remove pkg_resources import; also replace deprecated cf.readfp() with cf.read_file() and remove dead commented-out logging config code - sbin/fasta-to-seqinfo: use importlib.resources.files() instead of pkg_resources.resource_filename() to locate the logging config file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The NCBI rsync daemon only exposes a fixed set of top-level modules. Paths outside that set (e.g. direct file URLs) were silently failing when passed to rsync. Now the script checks the path prefix against the known module list and falls back to curl over HTTPS for any path that is not a valid rsync module. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The gbff_filter function pre-filtered genbank records by LOCUS prefix (NM/NR) before passing them to Bio.SeqIO.parse. This filtering is now handled upstream in the pipeline, so the pre-filter and associated Counter/seq_md5 imports were removed. The main loop now reads directly from gbff_block_reader, and logging no longer reports prefix counts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Break the README into audience-focused docs and surface links to them near the top: - README.md keeps overview, public-instance access, and local install, plus a Documentation section linking out to docs/. - docs/development.md: developer setup (venv, docker build, tests). - docs/updating-uta.md: maintainer update runbook (setup, SeqRepo pull, extract/transform, load, and the SeqRepo export/snapshot/publish steps). Mitochondrial and manual-splign loads are moved into a "Non-standard steps" section since they are not part of the standard release. - docs/migrations.md: alembic migration workflow. Also fix several procedure errors: the old-uta snapshot bind-mount prerequisite and $UTA_ETL_OLD_UTA_IMAGE_TAG var typo, reference the correct `uta` compose service (not "old-uta"), and run the splign-manual service via its override compose file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bsgiles73 · 2026-06-27T18:37:55Z

-    echo "Downloading $DOWNLOAD_SRC to $DOWNLOAD_DST"
-    rsync --no-motd -DHPRprtv "$DOWNLOAD_SRC" "$DOWNLOAD_DST"
+
+    if echo "$RSYNC_MODULES" | grep -qw "$DOWNLOAD_MODULE"; then


@reece I had trouble rsync'ing all of the files listed in ncbi-files.txt. So as a short term fix this script falls back to curl. Open to better suggestions.

That's fine. Seems like a reasonable fallback. Thanks for the explanation!

bsgiles73 · 2026-06-27T18:38:37Z

 Users can access a public instance of UTA or build their own instance of
 the database.

+## Documentation


Reorganized some of the documentation

test_load_exonset_with_exon_structure_mismatch used `mock_logger.warning.called_with(...)`, which is not a real Mock assertion. On Python <=3.11 it silently created a truthy child mock so the assert always passed; Python 3.12 hardened unittest.mock to raise AttributeError on such typo'd names, surfacing the failure. The assertion was also checking the wrong sink: load_exonset emits the "Exon structure mismatch" message via logger.exception(e) (with the exception object), never logger.warning(). Assert against logger.exception's recorded calls instead, matching the exception message text. A sweep of tests/ for `.called_with(` and similar no-op patterns found no other occurrences. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

reece

I ran into problems with docker compose run ncbi-download (see below).
I'll investigate later this week. Just wanted to let you know that I am looking at it!

reece · 2026-07-13T03:03:47Z

-    echo "Downloading $DOWNLOAD_SRC to $DOWNLOAD_DST"
-    rsync --no-motd -DHPRprtv "$DOWNLOAD_SRC" "$DOWNLOAD_DST"
+
+    if echo "$RSYNC_MODULES" | grep -qw "$DOWNLOAD_MODULE"; then


That's fine. Seems like a reasonable fallback. Thanks for the explanation!

reece · 2026-07-13T03:07:37Z

+## Virtual Environment
+To develop UTA, follow these steps.
+
+1.  Set up a virtual environment using your preferred method.


This is consistent with the previous instructions, I think, but not with the current methodology elsewhere in biocommons to use uv sync. I'll probably update eventually. No changes needed now.

reece · 2026-07-13T03:09:33Z

    return opts


-def gbff_filter(it):


Do you remember why you removed this?

reece · 2026-07-13T03:14:36Z

+
+Make directories:
+```
+mkdir -p $(pwd)/ncbi-data


super minor: drop the $(pwd)/... it's implied conceptually and wrong programmatically (pwd isn't defined typically, so copy-paste doesn't work). I recommend the simpler mkdir -p ncbi-data output/artifacts output/logs on a single line.

BUT, do consider if you care where pwd is. Top of the uta tree?

reece · 2026-07-13T03:38:49Z

+docker build --target uta -t uta-update .
+```
+
+## 1. Download SeqRepo data


In a future version, I think we should figure out how to make an installed version of seqrepo a dependency rather than installing. I think mounting an existing repo into the container would do it.

reece · 2026-07-13T03:54:57Z

+Download files from NCBI, extract into intermediate files, and load the nuclear
+transcripts into UTA and SeqRepo:
+```
+docker compose run ncbi-download


I got stuck here:

» docker compose run ncbi-download WARN[0000] /home/reece/projects/biocommons/uta/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion WARN[0000] Found orphan containers ([uta-ncbi-download-run-bff412de337f uta-seqrepo-pull-run-0d61ee193416]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. Downloading files to /ncbi-dir Downloading ftp.ncbi.nlm.nih.gov::gene/DATA/gene2refseq.gz to /ncbi-dir/gene rsync: [Receiver] failed to connect to ftp.ncbi.nlm.nih.gov (2607:f220:41e:250::13): Connection refused (111) rsync: [Receiver] failed to connect to ftp.ncbi.nlm.nih.gov (2607:f220:41e:250::12): Connection refused (111) rsync: [Receiver] failed to connect to ftp.ncbi.nlm.nih.gov (130.14.250.13): Connection refused (111) rsync: [Receiver] failed to connect to ftp.ncbi.nlm.nih.gov (130.14.250.12): Connection refused (111) rsync error: error in socket IO (code 10) at clientserver.c(141) [Receiver=3.2.7]

I'll investigate later this week.

I assume this is related to the ncbi rsync/curl swap.

This looks like a slightly different error than I ran into. If I remember correctly it was a @ERROR: Unknown module error for one file. So I added the modules that were listed to the script.

Locally I am also getting a connect refused error, but curl is working. Should we just switch?

Shane Giles and others added 5 commits June 3, 2026 08:39

bsgiles73 requested a review from a team as a code owner June 27, 2026 18:35

bsgiles73 commented Jun 27, 2026

View reviewed changes

bsgiles73 requested a review from reece June 27, 2026 18:47

reece reviewed Jul 13, 2026

View reviewed changes

Navigation Menu

Fix Docker build (Ubuntu 24.04 / PG17 / setuptools-scm) and restructure docs - #280

Fix Docker build (Ubuntu 24.04 / PG17 / setuptools-scm) and restructure docs#280
bsgiles73 wants to merge 6 commits into
mainfrom
fix-docker-build-setuptools-scm

bsgiles73 commented Jun 27, 2026

Uh oh!

bsgiles73 Jun 27, 2026

Uh oh!

reece Jul 13, 2026

Uh oh!

bsgiles73 Jun 27, 2026

Uh oh!

reece left a comment

Uh oh!

reece Jul 13, 2026

Uh oh!

reece Jul 13, 2026

Uh oh!

reece Jul 13, 2026

Uh oh!

reece Jul 13, 2026

Uh oh!

reece Jul 13, 2026

Uh oh!

reece Jul 13, 2026

Uh oh!

reece Jul 13, 2026

Uh oh!

bsgiles Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bsgiles73 commented Jun 27, 2026

Summary

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reece left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants