Repository navigation
Add a gs:// binary cache store with a credential helper - #84
Open
joshheinrichs-shopify wants to merge 7 commits into
Open
joshheinrichs-shopify wants to merge 7 commits into
joshheinrichs-shopify wants to merge 7 commits into
Conversation
The constructor template was only defined in filetransfer.cc, so only the instantiations used there existed, and nothing else could construct a FileTransferError. Other implementations of the FileTransfer interface (such as the in-memory fake GCS server used by the GCS store's unit tests) need to report NotFound and friends, so move the definition into the header.
Allow a `gs://` URL to carry an `endpoint` query parameter, either an absolute `http(s)://` URL or a `host[:port]` authority, which replaces `https://storage.googleapis.com` when building the API URLs. This is what lets the GCS store be pointed at an emulator such as fake-gcs-server. Anything else (another scheme, a URL without a host) is rejected rather than read as `host:port`.
Add a `gs://bucket[/prefix]` store type, so that a Google Cloud Storage bucket can be used directly as a substituter or as the target of `nix copy`, authenticated with OAuth2 bearer tokens from Application Default Credentials (service account key, gcloud user credentials or the GCE metadata server). Reads fall back to anonymous access when no credentials are found. Writes to Google's endpoint always require credentials; writes to another `endpoint` (an emulator) use them only when there are any. Objects are read through the XML API, path-style, at `https://storage.googleapis.com/<bucket>/<key>`, and written with the JSON API's simple upload. The object's name is the key its download URL has, so a cache under a prefix uploads under that prefix too, and a compressed narinfo's encoding goes in the `contentEncoding` query parameter: as a `Content-Encoding` header it would describe the request body, and the object would be served back still compressed. (Both learnt from the emulator test in #82.) GCS imposes no size limit on a single-request upload below the 5 TiB object maximum, so there is no multipart machinery here; the 5 GiB limit of the S3 store is S3's. The store builds these requests itself (via a new virtual `HttpBinaryCacheStore::makeRequest`) rather than relying on the `gs://` rewriting in the file transfer layer, so that it can pick the right token scope per operation and so that the whole store can be tested against a fake `FileTransfer`. The unit tests drive it against an in-memory emulation of the two APIs: authentication, prefixes, compressed uploads, anonymous reads and emulator writes, and a store path round trip.
Previously any failure to obtain a GCS access token made reads fall back to anonymous requests. On a private bucket that turns an expired or revoked `gcloud` login into 403s, which the binary cache code treats as "not found": the substituter silently looks empty and everything builds from source. Now only the absence of any configured credentials leads to anonymous reads (`GcsNoCredentials`); credentials that exist but cannot be refreshed are an error, with a hint to log in again. In the same spirit, `GOOGLE_APPLICATION_CREDENTIALS` naming a file that does not exist is now an error rather than a warning followed by the next fallback. The 403-as-404 treatment itself is an S3 workaround (S3 answers 403 for a missing object when the bucket isn't listable), and it is wrong for GCS, which answers 404 for a missing object whenever the caller may ask and 403 only for denied access. `HttpBinaryCacheStore` now asks a virtual `isMissing()` which the GCS store narrows to 404, so wrong credentials fail the build with GCS's own explanation instead of being remembered as a miss in the negative narinfo cache for an hour. Building locally regardless remains what `--fallback` is for. There is a second place where the same thing happened: a substituter that fails to *open* is dropped with a warning, so that an unreachable cache doesn't stop builds, and the GCS store's `init()` reads `nix-cache-info`. On a cold narinfo disk cache, wrong credentials therefore meant a warning and no substituter, i.e. building from source again, while on a warm cache they meant an error. The GCS store now opens regardless of a credential error (helper failure, 401, 403) and lets its first query fail instead, where that stops the build; without the narinfo disk cache, since nothing is known about the cache to record. Doing that exposed a pre-existing crash: `enqueueFileTransfer()` is `noexcept`, but the `s3://` and `gs://` request setup it performs (URL parsing, credential lookup) can throw, so e.g. `builtins.fetchurl "s3://bucket"` terminated the process. Setup failures are now delivered to the callback like any other transfer error.
Add the `gcs-credential-helper` setting: a Git credential helper that provides the OAuth2 access token for `gs://` stores. When set, it replaces the Application Default Credentials lookup. Nix calls the helper the way Go's `GOAUTH=git` and Cargo do: it runs `<command> get` with `protocol`, `host`, `path` and the `authtype` capability on its standard input, and reads attributes back. GCS authenticates with a bearer token, so the answer must be `authtype=Bearer` with `credential=<token>` (or `password=<token>` from a helper without the capability), and `password_expiry_utc` says when it expires. Git's protocol rather than one of our own, or Bazel's credential-helper spec, because it carries the same information, has far more consumers outside its origin, and is what the credential stores and token brokers on a developer's machine already speak. A store asks once, for its root URL, and uses the answer for every request under it; `builtins.fetchurl` of a `gs://` URL asks for that object. The helper's expiry is the only lifetime policy: an answer is reused until a minute before it, so a helper handing out hour-long tokens is run about once an hour per daemon worker rather than once per request. A helper that reports no expiry is run for every request, with a one-time warning: a fixed reuse window would be a second policy that is correct only while it happens to agree with the helper's own, and nothing would tell anyone when it stopped agreeing. A failed run is retried once; after that, a failure, a missing credential, an `authtype` other than `Bearer` or an already expired credential is an error rather than a fall back to anonymous access. Per-user credential stores and token brokers only answer processes of the user they serve, and the Nix daemon runs as root. The daemon forks one worker per client connection and already knows the client's uid and gid (that's how `trusted-users` is decided), so the worker now records them (`daemon::setClient`) and the helper runs *as that user*, with that user's home directory and, where `/run/user/<uid>` exists, its runtime directory. The helper then sees an ordinary same-user caller, and the daemon never holds a long-lived credential. Peer credentials are now read even when trust is forced, so forced-trust daemons know their client too; on macOS the peer's gid and pid are now filled in as well (`LOCAL_PEERCRED` carries the effective gid, `LOCAL_PEERPID` the pid). `runProgram()` cannot feed a child's standard input (that was removed on purpose along with its I/O thread), so the helper is started with `startProcess()` directly: the request is a few dozen bytes, written in full before the answer is read.
A single-machine NixOS VM test with fake-gcs-server, after #82's, for the two things the unit tests cannot cover: the real HTTP stack against an independent implementation of the API (an upload with a compressed narinfo, read back with the signature verified), and the daemon running `gcs-credential-helper` as the user it serves. For the latter the helper is a script that records which user ran it; the daemon is root, so the assertion that every run was by the unprivileged user is the test of the run-as-client path.
Why there are two credential modes and no third, why the helper speaks Git's credential protocol and runs as the daemon's client, why the helper's expiry is the only lifetime policy, and the three places where an authentication failure used to look like a cache miss and how each is closed. Plus what was deliberately left out, what was verified, and what remains open.
joshheinrichs-shopify
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Adds a
gs://binary cache store, and a way to authenticate to it on developer machines without a long-lived secret:gcs-credential-helper, a Git credential helper that the daemon runs as the connecting user. Authentication problems fail the build with the reason instead of looking like cache misses.Approach
<command> get, request on stdin,credential+password_expiry_utcout), the way Go'sGOAUTH=gitcalls one. Asked once per store, reused until a minute before expiry; no expiry means it runs per request, with a warning. Runs as the daemon's client (uid/gid, HOME, minimal PATH), so per-user token brokers work and the daemon holds no credential.--fallbackremains the opt-in to build locally anyway.GOOGLE_APPLICATION_CREDENTIALSalready does that).The store builds on #82. Design notes and open follow-ups:
plans/gcs-binary-cache/intent.md. Details are in the commit messages.Config
Verified
fetchurl gs://through a helper; 980 narinfo lookups in 4 s with one helper run; a daemon with broken, absent and working credentials gives the helper's message, GCS's message, and a substitution respectively.