Skip to content

test(provisioner): real-backend integration round-trips for Deprovision/Regrade DDL (toward 100% integration)#45

Merged
mastermanas805 merged 1 commit into
masterfrom
test/provisioner-redis-integration-roundtrip
Jun 4, 2026
Merged

test(provisioner): real-backend integration round-trips for Deprovision/Regrade DDL (toward 100% integration)#45
mastermanas805 merged 1 commit into
masterfrom
test/provisioner-redis-integration-roundtrip

Conversation

@mastermanas805

Copy link
Copy Markdown
Member

What

Env-gated real-backend integration tests that drive the gRPC server layer's Provision → Regrade → Deprovision lifecycle against a real Postgres and a real Redis, through the actual RPC handlers (server.ProvisionResource / RegradeResource / DeprovisionResource) — not injected fakes.

Why (truehomie-db DROP incident class, 2026-06-03)

Every existing server_test.go / server_coverage_test.go test injects a fake backend, so the real DROP DATABASE / DROP USER / ALTER ROLE DDL had never executed through the gRPC handler path (breaker wrapping, tier→connLimit routing, mapError, response shaping, idempotent re-deprovision). 98.9% statement coverage from mocks did not prove the destroy/regrade DDL is correct end-to-end — exactly the regression class where a real DROP path hides.

Tests (all t.Skip cleanly when the backend URL is unset → CI without a backend stays green)

  • TestServer_Postgres_Provision_Regrade_Deprovision_LiveRoundTrip — asserts db_/usr_ created, role CONNECTION LIMIT actually ALTERed (cross-checked against pg_roles.rolconnlimit), db_/usr_ DROPped, and a second Deprovision is a clean idempotent no-op (the fix(bugbash 2026-05-21): Provisioner P1 wave-1 — Redis k8s sizingForTier yearly + plus tiers #9 DROP IF EXISTS fix).
  • TestServer_Postgres_Reprovision_AfterDeprovision_LiveRoundTrip — re-provisions the same token after teardown; guards a partial-DROP leak that would block reuse.
  • TestServer_Redis_Provision_Deprovision_LiveRoundTrip — ACL user created, ACL user + namespace keys reaped, idempotent second Deprovision. (Redis LocalBackend has no Regrade — only the k8s backend implements redis.Regrader.)

Coverage delta (integration-only, per §1.4 mechanism — -coverpkg=./...)

Measured from the server test package (the gRPC layer):

Metric Before After
server test pkg vs ./... 18.9% 24.8% (+5.9pp)
backend/postgres/local.go Provision (from gRPC layer) 0.0% 68.3%
backend/postgres/local.go Deprovision 0.0% 62.5%
backend/postgres/local.go Regrade 0.0% 73.3%
backend/redis/local.go Provision 0.0% 70.0%
backend/redis/local.go Deprovision 0.0% 83.3%

The backend DDL methods, as exercised FROM the gRPC handlers, go from 0% to 60–83% — the real destroy/regrade path is now integration-covered, not just unit-covered in isolation.

Env-blocked / out of scope (this increment)

  • mongo / queue (NATS) / storage backend round-trips remain GAP — no local TEST_MONGO_URL/TEST_NATS_URL. Postgres + Redis prioritized per the bounded scope.

Gate

  • go build ./... && go vet ./... && go test ./... -short — green (exit 0, no FAIL).
  • gofmt (via $(go env GOROOT)/bin/gofmt) + golangci-lint — clean (0 issues).
  • Verified locally: all three tests PASS against local Postgres (localhost:5432) + Redis (localhost:6379), and SKIP cleanly with backends unset.

🤖 Generated with Claude Code

…on/Regrade DDL (toward 100% integration)

Adds env-gated real-backend integration tests that drive the gRPC server
layer's Provision → Regrade → Deprovision lifecycle against a REAL Postgres
and a REAL Redis, through the actual RPC handlers (not injected fakes).

Motivation: the truehomie-db DROP incident class (2026-06-03). Every existing
server test injects a fake backend, so the real DROP DATABASE / DROP USER /
ALTER ROLE DDL had never executed through the gRPC handler path (breaker
wrapping, tier→connLimit routing, mapError, response shaping, idempotent
re-deprovision). High statement coverage from mocks did not prove the
destroy/regrade DDL was correct end-to-end.

Tests (all skip cleanly when the backend URL is unset):
- TestServer_Postgres_Provision_Regrade_Deprovision_LiveRoundTrip: asserts
  db_/usr_ created, role CONNECTION LIMIT actually ALTERed (pg_roles cross-
  check), db_/usr_ DROPped, and a second Deprovision is a clean idempotent
  no-op (#9 DROP IF EXISTS).
- TestServer_Postgres_Reprovision_AfterDeprovision_LiveRoundTrip: re-provisions
  the same token after teardown — guards a partial-DROP leak that would block
  reuse.
- TestServer_Redis_Provision_Deprovision_LiveRoundTrip: ACL user created, ACL
  user + namespace keys reaped, idempotent second Deprovision.

Coverage delta (integration-only, server test pkg vs ./... per the §1.4
mechanism): server package 18.9% → 24.8% (+5.9pp). The backend DDL methods,
as exercised FROM the gRPC layer, go 0% → 60-83%
(postgres Provision 0→68.3%, Deprovision 0→62.5%, Regrade 0→73.3%;
redis Provision 0→70.0%, Deprovision 0→83.3%).

Gate: go build/vet/test ./... -short green; gofmt + golangci-lint clean.
Env-blocked: mongo/queue(NATS)/storage backend round-trips remain GAP
(no local TEST_MONGO_URL/TEST_NATS_URL) — out of scope for this increment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mastermanas805 mastermanas805 enabled auto-merge (squash) June 4, 2026 15:56
@mastermanas805 mastermanas805 merged commit b2557e7 into master Jun 4, 2026
12 checks passed
mastermanas805 added a commit that referenced this pull request Jun 4, 2026
…omie DROP-incident class) (#46)

PR #45 proved single-tenant Provision/Regrade/Deprovision/idempotency round-trips
for postgres + redis through the real gRPC handlers. It did NOT prove SCOPING:
that a deprovision is confined to the target tenant. That is the gap the
2026-06-03 truehomie-db DROP incident exposed (an active Pro customer's db+role
dropped while a co-resident tenant shared the cluster).

Adds server_multitenant_scoping_test.go: provisions TWO co-resident tenants A+B
through the genuine gRPC ProvisionResource handler against a real Postgres / real
Redis, seeds data into each, deprovisions ONLY A, and asserts B fully survives.

- Postgres: after Deprovision(A), A's db_/usr_ are gone (DROP ran) AND B's
  database + role still exist, B's seeded row is intact, and B can still CONNECT
  with its own ConnectionUrl credentials.
- Redis: after Deprovision(A), A's ACL user + namespace key are reaped AND B's
  ACL user + namespace key + value survive.

Env-gated identically to server_live_roundtrip_test.go (skips clean under
-short / no backend; runs for real in coverage.yml's pg+redis services and local
dev backends). Verified PASS locally against Postgres 16 + Redis 7.

Coverage block:
  Symptom:        unscoped DROP DATABASE/DROP USER (or ACL DELUSER/SCAN+DEL) on
                  deprovision takes out a co-resident tenant (truehomie 2026-06-03)
  Enumeration:    gRPC DeprovisionResource handler path, postgres+redis LocalBackend
  Sites found:    2 (postgres deprovision, redis deprovision)
  Sites touched:  2 (both have a co-resident-survival regression test)
  Coverage test:  TestServer_{Postgres,Redis}_Deprovision_IsScopedToTargetTenant
  Live verified:  PASS vs local Postgres 16 + Redis 7 (real backends); B survives A

Co-authored-by: Manas Srivastava <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mastermanas805 added a commit that referenced this pull request Jun 4, 2026
…age (Wave 4) (#47)

* test(provisioner): real-backend gRPC round-trips for Mongo/Queue/Storage (Wave 4)

Closes the remaining cells of the provisioner gRPC × backend matrix
(INTEGRATION-COVERAGE-PLAN §2.3 / Wave 4). Postgres + Redis server-layer
round-trips already shipped (#45, #46); this adds the real-backend
Provision → assert artifact → Deprovision → assert-gone lifecycle for the
three remaining backends, all driven through the genuine gRPC handlers
(breaker wrapping, tier routing, mapError, response shaping):

- Mongo: ProvisionResource creates usr_/db_ on a real MongoDB, GetStorageBytes
  reads real dbStats (>0 after seeding), DeprovisionResource runs the real
  dropUser/dropDatabase (truehomie DROP-incident class), second Deprovision is
  a clean idempotent no-op, and Regrade(mongo) asserts the documented skip path.
- Queue (NATS): ProvisionResource passes the real NATS monitor health check and
  returns nats:// URL + subject prefix, GetStorageBytes(queue)=0 (message-metered),
  Deprovision is the shared-backend no-op, idempotent.
- Storage (MinIO/S3): GetStorageBytes object-walk — empty prefix=0, after a real
  PutObject=exact byte count, after delete=0. (Storage Provision/Deprovision are
  API-side; provisioner only meters.)

All tests env-gated (skip cleanly under `go test -short`, the deploy gate; run
for real when the backend env is present). CI: added NATS service container +
MinIO docker-run step + the TEST_NATS_HOST / TEST_MINIO_* / CUSTOMER_MONGO_AUTH_URL
env wiring to coverage.yml so they execute (mongo was already provided).

Verified locally against real mongo/nats/minio containers: all 8 server
round-trip tests PASS; integration-only coverage for internal/server = 99.2%
(provisionMongo/provisionQueue/GetStorageBytes/DeprovisionResource/RegradeResource
all 100%). No bug found in the destroy/regrade paths. Added
INTEGRATION-COVERAGE-EXCLUSIONS.md documenting the ≥80 floor method + the only
genuinely-unreachable lines (k8s dedicated-backend boot wiring, cmd/ entrypoints).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(provisioner): start NATS via docker run, not a services container

The minimal nats:2 image has no wget/curl/nc, so the service-container
--health-cmd ('wget ... :8222/healthz') could never pass — GitHub Actions
marked the container unhealthy and aborted the coverage job before any test
ran (NATS logged 'Server is ready'). Mirror the MinIO pattern: docker run
nats:2 -js -m 8222 + a runner-side curl wait on /healthz. Unblocks #47.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Manas Srivastava <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant