[1fanwang]

Tracking issue

Related to #8130

Why are the changes needed?

Overwriting cached outputs can leave datacatalog serving the old ones. When an overwrite-cache run replaces a task's cached outputs with larger ones, datacatalog writes the new data to the same location, but its in-memory storage cache keeps the old copy. Every later cache hit in that process returns the old outputs until the entry is evicted or the process restarts.

The in-memory cache only accepts entries up to 1/1024 of its configured size, about 10 KB with datacatalog's sample config. A bigger write lands in the store, the cache rejects it, and the storage layer treats the rejection as harmless, so the previous entry is never removed.

What changes were proposed in this pull request?

When the cache rejects new bytes, the storage layer now drops the cached entry for that path, so the next read goes to the store. Writes that fit still replace the entry as before. Other datacatalog replicas keep their own caches, so this fixes reads in the process that did the write.

How was this patch tested?

Added a unit test that overwrites a cached value with one too big for the cache.

Testing Done

I built datacatalog from master (2682923) and from this branch, ran each with the sample config's cache (max_size_mbs: 10), local file storage and an empty Postgres 14 database, and drove it over gRPC. The script follows the cache client: create and tag the artifact, read it by tag, overwrite it with UpdateArtifact, then read it again.

Reproducer source: cache-e2e.sh

#!/usr/bin/env bash
set -e
call() { grpcurl -plaintext -d @ localhost:18081 "datacatalog.DataCatalog/$1"; }
ds='{"project":"flytesnacks","domain":"development","name":"flyte_task-wf.t1","version":"1.0"}'
out() { printf '[{"name":"o0","value":{"scalar":{"primitive":{"stringValue":"%s"}}}}]' "$(head -c "$1" /dev/zero | tr '\0' x)"; }
size() { call GetArtifact <<<"{\"dataset\":$ds,\"tagName\":\"k1\"}" | jq '.artifact.data[0].value.scalar.primitive.stringValue | length'; }
call CreateDataset <<<"{\"dataset\":{\"id\":$ds}}" >/dev/null
call CreateArtifact <<<"{\"artifact\":{\"id\":\"a1\",\"dataset\":$ds,\"data\":$(out 100)}}" >/dev/null
call AddTag <<<"{\"tag\":{\"name\":\"k1\",\"artifactId\":\"a1\",\"dataset\":$ds}}" >/dev/null
echo "cache hit before overwrite: $(size) chars"
call UpdateArtifact <<<"{\"dataset\":$ds,\"tagName\":\"k1\",\"data\":$(out 16384)}" >/dev/null
echo "cache hit after overwrite:  $(size) chars"
echo "data.pb on disk: $(wc -c < "$(find blobs -name data.pb)") bytes"

Reproducer source: datacatalog.yaml

logger:
  level: 4
application:
  grpcPort: 18081
  httpPort: 18080
  grpcServerReflection: true
  grpcMaxRecvMsgSizeMBs: 6
datacatalog:
  storage-prefix: "metadata"
  metrics-scope: "datacatalog"
  profiler-port: 18254
  heartbeat-grace-period-multiplier: 3
  max-reservation-heartbeat: 10s
storage:
  type: local
  container: flyte
  stow:
    kind: local
    config:
      path: ./blobs
  cache:
    max_size_mbs: 10
    target_gc_percent: 100
database:
  port: 15499
  username: postgres
  host: localhost
  dbname: datacatalog
  options: "sslmode=disable"

./datacatalog --config datacatalog.yaml migrate run
./datacatalog --config datacatalog.yaml serve &
bash cache-e2e.sh

On master, the cache keeps returning the 100-character output after the overwrite, while the stored file holds the new one:

cache hit before overwrite: 100 chars
cache hit after overwrite:  100 chars
data.pb on disk:    16396 bytes

On this branch, the read after the overwrite returns the new output:

cache hit before overwrite: 100 chars
cache hit after overwrite:  16384 chars
data.pb on disk:    16396 bytes

Labels

• fixed

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

#8130

Stack

No git town stack.

Docs link

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.34%. Comparing base (2682923) to head (d68c370).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8239      +/-   ##
==========================================
+ Coverage   56.56%   57.34%   +0.78%     
==========================================
  Files         772      931     +159     
  Lines       52305    58338    +6033     
==========================================
+ Hits        29587    33455    +3868     
- Misses      19961    21832    +1871     
- Partials     2757     3051     +294     

Flag	Coverage Δ
unittests-datacatalog	53.62% <ø> (ø)
unittests-flyteadmin	53.26% <ø> (ø)
unittests-flytecopilot	48.05% <ø> (ø)
unittests-flytectl	64.11% <ø> (?)
unittests-flyteidl	76.63% <ø> (ø)
unittests-flyteplugins	60.64% <ø> (ø)
unittests-flytepropeller	53.88% <ø> (ø)
unittests-flytestdlib	64.40% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.