Make recursive delete operations more intuitive

Issue

Currently recursive delete operations like org, space, app delete, which implicitly delete service bindings and/or service instances, will fail if one of the service related deletions are handled asynchronously by the service broker. This is not optimal as users have to trigger the deletion of the parent resource again. Also currently users will get the following error message, which does not really reveal what is going on:
An operation for the service binding between app myapp and service instance myinstance is in progress.

Context

[provide more detailed introduction]

Steps to Reproduce

For example delete an app, which is bound to a service instance. The broker should answer unbinding requests with a 202.

Current result

The following tables describe the resulting behaviour for different resources and responses by the broker. In general all the recursive deletions fail if a sub-resource gets deleted asynchronously. The behaviour is the same for service instances and service bindings.

Delete service instance when service binding present

API call	resulting request to broker	broker response	current behaviour
DELETE /v3/service_instance/ <service_instance_guid>	DELETE /v2/service_instances/ <si_guid>/service_bindings/<sb_guid>	202	Starts polling service binding last operation and sets service instance delete job and service instance last operation to failed with message "delete could not be completed: An operation for the service binding between app myapp and service instance myinstance is in progress."
		500	Both the service instance and service binding delete operations fail
		200	Binding will be gone immediately, then a delete service instance request will be sent to the broker and either its gone too or polling starts

Delete app when bound to service

API call	resulting request to broker	broker response	current behaviour
DELETE /v3/apps/<app_guid>	DELETE /v2/service_instances/ <si_guid>/service_bindings/<sb_guid>	202	Starts polling service binding last operation and sets app delete job to failed with message "Job (3d1d051f-6c94-47ab-85e2-dec27e0db75a) failed: An operation for the service binding between app myapp and service instance myinstance is in progress."
		500	The app delete job fails and the service binding operations state is set to failed. Error message: "Job (17662f61-4111-4712-91da-d3ab11629ba7) failed: Service broker failed to delete service binding for instance myinstance: The service broker returned an invalid response. Status Code: 500 Internal Server Error, Body: {"state":"in progress"}"
		200	Service binding gets deleted, app gets deleted and app delete job set to COMPLETE

Delete space which contains a service binding

API call	resulting request to broker	broker response	current behaviour
DELETE /v3/spaces/ <space_guid>	DELETE /v2/service_instances/ <si_guid>/service_bindings/<sb_guid>	202	Starts polling service binding last operation and sets space delete job to failed as well as service instance last operation with message "Job (5b91523b-fe11-4cdb-bbc9-063b65fa8dee) failed: Deletion of space myspace failed because one or more resources within could not be deleted. An operation for the service binding between app myapp and service instance myinstance is in progress."
		500	The space delete job fails and the service binding and service instance last operations state is set to failed. Error message: "Job (ebce0407-9560-44fb-9b39-7b06f35edb4f) failed: Deletion of space d071102 failed because one or more resources within could not be deleted. Service broker failed to delete service binding for instance myinstance: The service broker returned an invalid response. Status Code: 500 Internal Server Error, Body: {"state":"in progress"}"
		200	Service binding, service instance, app and space gets deleted, and space delete job set to COMPLETE

Delete org, which contains a space which contains a service binding

API call	resulting request to broker	broker response	current behaviour
DELETE /v3/organizations/ <organization_guid>	DELETE /v2/service_instances/ <si_guid>/service_bindings/<sb_guid>	202	Starts polling service binding last operation and sets org delete job to failed as well as service instance last operation with message "Job (0daca787-a8b5-4433-967c-3b0c8d2e1798) failed: Deletion of organization d071102 failed because one or more resources within could not be deleted. Deletion of space d071102 failed because one or more resources within could not be deleted. An operation for the service binding between app myapp and service instance myinstance is in progress."
		500	The org delete job fails and the service binding and service instance last operations state is set to failed. Error message: "Job (5c7ffdc1-7bb3-4a95-96da-33d19c5d4e79) failed: Deletion of organization d071102 failed because one or more resources within could not be deleted. Deletion of space d071102 failed because one or more resources within could not be deleted. Service broker failed to delete service binding for instance myinstance: The service broker returned an invalid response. Status Code: 500 Internal Server Error, Body: {"state":"in progress"}"
		200	Everything gets deleted and organization delete job set to COMPLETE

Further findings

All recursive deletions will trigger the deletion of all sub-resources (except they depend on each other). For example an app delete will trigger the deletion of all service bindings of that app. If one binding fails to delete or is being deleted asynchronously, the job will continue to trigger the deletion of all other bindings. Service instances, which have bindings which are in deletion won't be deleted.

Expected result

Best case would be if the recursive delete operations can handle asynchronously deleted sub-resources. See next section for some ideas on how to achieve this.

Possible Fix

Re-enqueue recursive jobs instead of setting them to failed

The deletion jobs could be re-enqueued similarly to what we do for the polling mechanism of service related operations. The job could then check whether the resources have been deleted successfully and if so delete the parent resource.

Some thoughts on this:

Probably we would need a "locking mechanism" to prevent that in an org or space etc., which are being deleted, new resources are being created.
If asynchronous deletions fail, the job should remember that it tried to delete this resource already, otherwise this might be an endless loop.
A parameter, which allows configuring a maximum timeout for such jobs would be good
When the job fails because sub-resources could not be deleted, it would be good to show the original error message, why the deletion failed.

Delete parent resource immediately and continue asynchronous deletion of sub-resources in the background

If a service broker responds with a 202 for an unbind or deprovision request we can assume that the broker will take care of the deletion and at least "delete" it from the user perspective. The CC could then continue polling the last operation state from the broker. If the deletion fails, orphan mitigation could take over.

Some thoughts on this:

In the worst case the deletion of the service binding times out after the max_poll_intervall. How to proceed with the service instance then?
What if a user wants to create resources with the same names again after the CC stated they have been deleted, but in reality the deletion is still going on in the background?

Related issues

/v3/app delete does not wait until service binding are unbound #3333 - Describes the behaviour for app delete already
Add none recursive service instance deletion parameter #3532 - Ask for a "recursive" parameter

We've been prototyping a solution for this. The idea is to reuse the existing ReoccurringJob / PollableJobModel infrastructure and only create additional jobs when a broker actually returns 202. A new parent_guid column on the jobs table tracks the parent-child relationship.

New jobs: DeleteSpaceJob, DeleteAppJob, DeleteOrgJob, plus modifications to the existing DeleteServiceInstanceJob. These replace DeleteActionJob for the affected endpoints.

The core pattern: try to delete inline (sync). If a broker returns 202, enqueue a child ReoccurringJob to poll, and wait for it. If everything is sync, the whole operation completes in a single job cycle.

  DELETE /v3/organizations/:guid
    └── DeleteOrgJob
          └── DeleteSpaceJob (per space, maybe batched)
                │
                ├── Phase 1 — Apps (maybe batched):
                │   for each app:
                │     stop app → try AppDelete inline
                │     ├── broker 200 → app deleted (no extra jobs)
                │     └── broker 202 → enqueue DeleteBindingJob (child)
                │                      wait → retry → app deleted
                │
                ├── Phase 2 — Managed Service Instances (batched):
                │   for each SI:
                │     try ServiceInstanceDelete inline
                │     ├── bindings/keys sync   → deprovision
                │     │   ├── broker 200 → SI deleted (no extra jobs)
                │     │   └── broker 202 → enqueue DeleteServiceInstanceJob (child)
                │     │                    polls broker → SI deleted
                │     └── bindings/keys async  → enqueue DeleteBindingJob(s) (children)
                │                                wait → retry deprovision
                │
                └── Phase 3 — Cleanup (sync):
                      destroy UPSIs, unshare SIs, remove brokers, destroy space

Pollable jobs created:

All sync brokers: 1 (just the DeleteSpaceJob itself)
1 async binding: 1 + 1 DeleteBindingJob = 2
1 async deprov.: 1 + 1 DeleteServiceInstanceJob = 2
Mixed (n async): 1 + n child jobs

We'd appreciate any feedback on this approach!

Space Deletion Details

A new DeleteSpaceJob (ReoccurringJob) orchestrates deletion in phases:

Apps — delete inline, enqueue DeleteBindingJob only if broker returns 202
Service Instances — deprovision inline, enqueue DeleteServiceInstanceJob only if broker returns 202
Cleanup — UPSIs, route shares, space-scoped brokers, destroy space

The root job re-checks periodically until all sub-jobs complete, then advances to the next phase. It reschedules itself based on when its sub-jobs are next due to run - this accounts for broker-returned Retry-After values, so it never wakes up unnecessarily. Sub-jobs use the existing job priority mechanism - no changes to scheduling priority needed.

Schema changes

jobs.root_job_guid VARCHAR(255) (indexed) -> links sub-jobs to the root delete job
spaces.status VARCHAR(255) DEFAULT 'active' -> values: active, deleting. Similar to organizations.status but without suspended. Exposed in the API (read-only) but not settable by users -> managed internally by the delete job

Model changes

PollableJobModel: one_to_many :sub_jobs association via root_job_guid. find_active_delete class method for idempotent controller behavior.
Space: mark_deleting! (atomic, idempotent), deleting? predicate.

Execution flow (happy path)

┌───────────────────────────┬────────────────────────────────────┬────────────────────────────────┐
│ CLI                       │ Worker                             │ jobs table                     │
├───────────────────────────┼────────────────────────────────────┼────────────────────────────────┤
│ $ cf delete-space myspace │                                    │ space.delete   PROCESSING [R]  │
│                           │ space.delete: delete app,          │                                │
│                           │  unbind returns 202                │ bindings.del   PROCESSING [S]  │
│ Deletion in progress...   │                                    │ space.delete   POLLING    [R]  │
│                           │ bindings.del: poll → "succeeded"   │ bindings.del   COMPLETE ✓ [S]  │
│                           │ space.delete: deprovision SI,      │                                │
│                           │  broker returns 202                │ si.delete      PROCESSING [S]  │
│                           │                                    │ space.delete   POLLING    [R]  │
│                           │ si.delete: poll → "succeeded"      │ si.delete      COMPLETE ✓ [S]  │
│                           │ space.delete: cleanup, destroy     │ space.delete   COMPLETE ✓ [R]  │
│ OK                        │                                    │                                │
└───────────────────────────┴────────────────────────────────────┴────────────────────────────────┘
[R] = root job   [S] = sub-job (root_job_guid → R)

Error case (broker fails)

┌───────────────────────────┬────────────────────────────────────┬────────────────────────────────┐
│ CLI                       │ Worker                             │ jobs table                     │
├───────────────────────────┼────────────────────────────────────┼────────────────────────────────┤
│ $ cf delete-space myspace │                                    │ space.delete   PROCESSING [R]  │
│                           │ space.delete: deprovision → 202    │ si.delete      PROCESSING [S]  │
│ Deletion in progress...   │                                    │ space.delete   POLLING    [R]  │
│                           │ si.delete: poll → "in progress"    │ si.delete      POLLING    [S]  │
│                           │ si.delete: poll → "failed"         │ si.delete      FAILED ✗   [S]  │
│                           │ space.delete: detects failure      │ space.delete   FAILED ✗   [R]  │
│ FAILED: Deletion of the   │                                    │                                │
│ following resources       │                                    │                                │
│ failed: service_instance  │                                    │                                │
│ <guid>                    │                                    │                                │
└───────────────────────────┴────────────────────────────────────┴────────────────────────────────┘

@johha

Hi @johha ! I had come across issue recently also, so glad to see the proposal! Some questions / ideas for ya that maybe are here but i missed, so bare with me

Would we want / need a global timeout that if this job (DeleteSpaceJob ) keeps failing and rescheduling itself that it eventually stops?
Any thoughts on preventing apps being pushing while the space is in this new spaces.status = deleting state?
Am I right in reading that Phase 2 (deleting Service Instances) doesn't start until all Apps (Phase 1) are fully deleted and unbound? If one app takes 24 hours to asynchronously unbind from a database, potentially many other completely unbound service instances in that space will sit idle waiting to be deleted, correct? - probably fine, just might be wasteful.
Should we expose Sub-Job Progress in the V3 API?
When cf delete-space hangs for X hours, the CLI will just say "Deletion in progress...". Operators and users will want to know what it's waiting on. Perhaps we could update the GET /v3/jobs/:guid endpoint for the root space deletion job. When the job is in the POLLING state, it should include a warnings or details array listing the specific sub-jobs it is currently waiting on (e.g., "Waiting on async unbind for service instance 'my-db'"). This prevents operators from having to dig into the ccdb to see why a space deletion is taking so long.

In general though i think the proposal is great, and will really help with the current issues in the jobs structure. The concept of sub-jobs will be useful and maybe a further proposal can expand on the jobs API to expose these sub-jobs and their failures via the parent. (or at least be linked)

@neil-hickey

@neil-hickey thanks for your feedback 👍 Quick responses:

No separate limit currently. The root job reschedules itself to wake up just after its sub-jobs' next execution (plus a small buffer so sub-jobs run first). Each job (root and sub) independently respects broker_client_max_async_poll_duration_minutes (default 14 days) as its hard limit; once a job exceeds that it expires to FAILED and the root propagates the failure. Operators can tune this config to enforce a tighter limit.
All create/update endpoints (apps, packages, droplets, tasks, SIs, bindings, manifests…) call being_deleted! similar to suspended! to prevent modifications of any resource. Parallel DELETEs are intentionally still allowed so users can clean up stuck resources - deletes should be idempotent in general.
Correct, it could take a while and leave unused service instances for a while. But changing this would require bigger changes and we couldn't reuse the existing delete actions like app delete. This issue also exists today already when user try delete spaces with multiple attempts where each attempt only makes progress on one slow binding before erroring out.
This would be an API change and should be handled as a separate topic or follow-up. At the moment we would set a job warning so that users are aware that sub resources are being deleted and that this could take a while depending on the service broker. We're also thinking of adding the option for users to cancel a job (as follow-up).

Add parent_guid to jobs table and status to spaces

Add parent_guid support to job enqueue pipeline

Add DeleteSpaceJob, DeleteAppJob for async recursive delete

Block create/update operations on deleting spaces and orgs

Add tests for recursive delete and deletion guards

Refactor sub-job tracking to shared enqueuer pattern

Add parent_guid to jobs table and status to spaces

Add parent_guid support to job enqueue pipeline

Add DeleteSpaceJob, DeleteAppJob for async recursive delete

Block create/update operations on deleting spaces and orgs

Add tests for recursive delete and deletion guards

Refactor sub-job tracking to shared enqueuer pattern

Update: merged 🎉

The first slice of this has landed — #5299 and cloudfoundry/capi-release#677.

What's included: async recursive delete for apps and service instances. When a broker responds to an unbind/deprovision with 202, the delete job now waits for the async operation to finish (polling via a child job) instead of failing the whole delete. On sub-resource failures, all of the original broker errors are surfaced.

What changed vs. the earlier comments:

Scope is narrower to start. This PR covers only apps and service instances. DeleteSpaceJob / DeleteOrgJob and the phased space/org orchestration described above are still to come - the app + SI jobs build the foundation (the RecursiveDeleteRootJobMixin and sub-job tracking) that those will reuse.
Column name: the parent→child link on the jobs table is root_job_guid (not parent_guid as the first comment called it).
Root-job timeout: rather than every job independently expiring at broker_client_max_async_poll_duration_minutes, the root job no longer counts the time spent waiting on its sub-jobs against its own budget - once the sub-jobs finish, its max-duration budget is measured from that point, so the root's own broker delete always gets its full duration. Sub-jobs remain independently bounded.
Feature flags: gated per-resource via cc.temporary_enable_async_recursive_delete.apps and cc.temporary_enable_async_recursive_delete.service_instances.

Rollout note: both flags are disabled by default. Because this introduces new job types and new job serialization formats, only enable the flags after the release containing these changes has been deployed at least once - otherwise workers may fail to deserialize in-flight jobs. We plan to validate on pre-prod environments first before changing the default in a couple of releases.

Space and org support will follow as separate PRs.

We'll assign this again once we work on root jobs for org and spaces.

Navigation Menu

Repository navigation

Make recursive delete operations more intuitive #3589

Issue

Context

Steps to Reproduce

Current result

Delete service instance when service binding present

Delete app when bound to service

Delete space which contains a service binding

Delete org, which contains a space which contains a service binding

Further findings

Expected result

Possible Fix

Re-enqueue recursive jobs instead of setting them to failed

Delete parent resource immediately and continue asynchronous deletion of sub-resources in the background

Related issues

Space Deletion Details

Schema changes

Model changes

Execution flow (happy path)

Error case (broker fails)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make recursive delete operations more intuitive #3589

Description

Issue

Context

Steps to Reproduce

Current result

Delete service instance when service binding present

Delete app when bound to service

Delete space which contains a service binding

Delete org, which contains a space which contains a service binding

Further findings

Expected result

Possible Fix

Re-enqueue recursive jobs instead of setting them to failed

Delete parent resource immediately and continue asynchronous deletion of sub-resources in the background

Related issues

Activity

johha commented on Apr 28, 2026

johha commented on May 12, 2026

Space Deletion Details

Schema changes

Model changes

Execution flow (happy path)

Error case (broker fails)

neil-hickey commented on May 15, 2026

johha commented on May 20, 2026

johha commented on Aug 11, 2026

johha commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions