refactor(htsget): use htslurp as htsget client

[cmdoret]

Context

The current implementation of the htsget client relies on pysam to return standard record objects to library users.
pysam does not have support for iterating records from a network stream, as it needs a seekable filesystem object.

As a consequence, we are forced to accumulate the stream into a temporary file when streaming htsget data from the library.

This is not an issue on the CLI, as the stream can be piped into tools such as samtools, which support it.

Goal

Allow direct streaming from the htsget-server from the library.

Changes

This PR replaces the htsget client implementation with htslurp, a rust-based htsget client that adds python bindings on top of the noodles implementation.

It supports iterating directly over the network stream and should be usable as a drop-in replacement.

Open questions

For now, htslurp can return text records, which modos then instantiate into pysam objects. This only works for {B,CR,S}AM files, as {B,V}CF records can only be instantiated from the file object. There are 3 solutions:

1. use a different library to instantiate those records (e.g. pyvcf).
2. implement dedicated modos records that can be instantiated from strings.
3. implement the structs directly in htslurp, in rust with bindings, or in python.

[cmdoret]

Additionally, htslurp is currently focused on returning records, but the modos CLI still emits the bytestream:

API stream:

sequenceDiagram
    autonumber
    participant API as Python API<br/>MODO.stream()
    participant Conn as HtsgetConnection
    participant HL as htslurp
    participant SRV as htsget + data server (S3)

    API->>Conn: to_pysam(region, reference)
    Conn->>HL: stream_records(base_url, id, format, region)
    HL->>SRV: GET ticket, then block byte ranges
    SRV-->>HL: SAM records + header
    loop per record
        HL-->>Conn: record bytes
        Conn->>Conn: AlignedSegment.fromstring(record, header)
        Conn-->>API: pysam AlignedSegment
    end

Loading

CLI stream:

sequenceDiagram
    autonumber
    participant CLI as CLI<br/>modos stream
    participant Conn as HtsgetConnection
    participant TK as HtsgetStream / _HtsgetBlockIter
    participant SRV as htsget + data server (S3)

    CLI->>Conn: open()
    Conn->>SRV: get_session().get(url)
    SRV-->>Conn: ticket JSON
    Conn->>TK: HtsgetStream(ticket urls)
    loop per block
        TK->>SRV: fetch byte range (requests.get / data URI)
        SRV-->>TK: raw bytes
        TK-->>CLI: raw byte chunk
        CLI->>CLI: write to stdout
    end

Loading

If we also expose a function to recover the raw bytestream from htslurp (e.g. htslurp.stream_bytes), we could drop most of the boilerplate in modos.genomics.htsget (HtsgetStream, the entire notion of a ticket, (de)construction of htsget urls, ...). HtsgetConnection would then become a thin wrapper over htslurp holding metadata and exporter methods:

CLI stream (potential)

sequenceDiagram
    autonumber
    participant CLI as CLI<br/>modos stream
    participant Conn as HtsgetConnection
    participant HL as htslurp
    participant SRV as htsget + data server (S3)

    CLI->>Conn: stream_bytes(region, reference)
    Conn->>HL: stream_bytes(base_url, id, format, region)
    HL->>SRV: GET ticket, then block byte ranges
    SRV-->>HL: raw byte stream
    loop per chunk
        HL-->>Conn: raw bytes
        Conn-->>CLI: raw byte chunk
        CLI->>CLI: write to stdout
    end

Loading