Skip to content

version_checker_is_available reports 0 for images whose tag still exists but was re-pushed (cached tag is mutated) #467

Open

Description

@george-angel

Describe the bug

version_checker_is_available is 0 for a running image whose tag still exists
upstream, when that tag has been re-pushed to a new digest. Every other image
reports 1. Found on v0.12.0 (PR #447).

Example: running registry:3.1.2 with digest sha256:d4032d4c...c6c3.
Docker Hub still lists tag 3.1.2, but now with digest sha256:82f6ff4a...bc3c
(14 child images, none matching the running digest). The metric is 0 on every
reconcile, even though the tag exists.

Root cause

isLatestSemver (pkg/controller/checker/checker.go, lines 186 and 191)
appends "@" to latestImage.Tag when the tag is the same version but the
SHA differs upstream. latestImage is a pointer into the shared tag cache
(&cachedTags[i]), so the cached tag "3.1.2" becomes "3.1.2@sha256:82f6...".

Container() then calls CurrentImage, which falls back to a tag-name match
(pkg/version/version.go:134, tags[i].Tag == imageTag). The cached entry no
longer equals "3.1.2", so nothing matches, the result is nil and Available
is set to false.

When the running digest is still current, the mutating branch does not run
and the metric is 1, which is why only re-pushed tags are affected.

Reproduction

A unit test using a fake client.ClientHandler returning Docker Hub shaped
data: tags "3.1.1", "3.1.2" (SHA 82f6ff4a..., 14 children) and "latest".
Run Container() for image "registry:3.1.2" with pod imageID
docker.io/library/registry@sha256:d4032d4c...

  • default options (semver path): Available=false (bug)
  • running digest equal to the upstream digest: Available=true
  • opts.UseSHA path (does not mutate the cache): Available=true

Called directly, CurrentImage returns the tag correctly before Container()
has run, and nil afterwards, which isolates the mutation as the cause.

Other effects of the same mutation

  • A second reconcile inside the search cache TTL reuses the mutated entry.
    IsLatest flips to true and LatestVersion is "3.1.2@sha256:82f6...".
  • After the version cache refetches, the corrupted tag is skipped as metadata
    by pkg/version/filters.go:26 and LatestVersion can regress to 3.1.1.
  • image_timestamp is not set for affected images, as Timestamp stays zero.
  • is_latest_version can show both 0 and 1 series for the same image.

Suggested fix

Do not modify the cached tag. Copy before changing it, e.g. in
isLatestSemver:

result := *latestImage
result.Tag = fmt.Sprintf("%s@%s", result.Tag, result.SHA)
return &result, isLatest, nil

and likewise for the child SHA branch. A regression test that calls
Container() and then CurrentImage() on the same search instance, with a
re-pushed tag, would catch this.

Environment

version-checker v0.12.0, Kubernetes, Docker Hub official images, with
--test-all-containers=true.

Related: #323, #395, #447.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions