Repository navigation
Tools/pm 43847/delete item sends when ciphers deleted - #8899
mcamirault wants to merge 5 commits into
Conversation
🤖 Bitwarden Claude Code ReviewOverall Assessment: APPROVE This review covers the full PR and the new commit 6387d4e. That commit has Code Review DetailsNo new findings. |
| [dbo].[SendView] S | ||
| INNER JOIN | ||
| @ItemIds I ON S.[Data] LIKE '%' + CONVERT(VARCHAR(36), I.[Id]) + '%' | ||
| WHERE | ||
| S.[Type] = 2 |
There was a problem hiding this comment.
Send table every time a cipher is permanently deleted, including in synchronous API requests.
Details and fix
[dbo].[Send] only has PK_Send, IX_Send_UserId_OrganizationId, and IX_Send_DeletionDate. Nothing indexes [Type], and [Data] LIKE '%' + ... + '%' has a leading wildcard. So every call scans the clustered index across all Sends, then runs up to 500 substring comparisons against each item Send's VARCHAR(MAX) Data.
This proc is now on the request path for:
CipherService.DeleteAsync: one scan for every single-item permanent deleteCipherService.DeleteManyAsync: emptying the trash, one scan per 500 idsPurgeAsync/PurgeUserVaultAsyncand both org-delete pathsDeleteCiphersJob:ceil(expiredCount / 500)full scans in one nightly run
In CipherService.DeleteAsync the cleanup also runs before the Cipher_Deleted event and the sync push. If this query is slow or times out, the cipher is already gone but the event is never logged and clients are never told.
Possible fixes:
- Persist the item id in its own indexed column when an item Send is saved, or add a persisted computed column over
JSON_VALUE([Data], '$.Metadata.ItemId')with a filtered index on[Type] = 2. Then join on equality. - At minimum, add a filtered index on
[Type] = 2so only item Sends are scanned. Also consider moving the cleanup after the event/push inDeleteAsync, or making it best-effort there.
The EF implementation has the same shape: an OR'd Contains over the full Sends set.
code-review
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8899 +/- ##
==========================================
- Coverage 75.07% 68.85% -6.22%
==========================================
Files 2759 2757 -2
Lines 118256 118301 +45
Branches 10865 10872 +7
==========================================
- Hits 88777 81456 -7321
- Misses 26815 34289 +7474
+ Partials 2664 2556 -108 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| FROM | ||
| [dbo].[SendView] S | ||
| INNER JOIN | ||
| @ItemIds I ON S.[Data] LIKE '%' + CONVERT(VARCHAR(36), I.[Id]) + '%' |
There was a problem hiding this comment.
Send once for every id in the batch. Please read the item id out of the JSON and join on it.
LIKE '%' + id + '%' on [Data] (VARCHAR(MAX)) can't use an index, and the plan scans the table once per id. On a restored copy of production (86,867 Sends, 3,974 item Sends, MAXDOP 1):
| Ids in the call | LIKE (this PR) |
JSON_VALUE join |
|---|---|---|
| 1 | 82 ms, 10,467 reads | 123 ms, 10,538 reads |
| 50 | 3,552 ms, 513,811 reads | 121 ms, 10,979 reads |
| 500 | 32,671 ms, 5,134,160 reads | 128 ms, 13,502 reads |
About 225,000 ciphers a day leave the trash, so DeleteCiphersJob would make about 450 of these 500-id calls a night: roughly 4 hours of CPU as written, about a minute with the join below. Both forms returned the same 261 Sends for the same 500 ids.
SELECT
S.*
FROM
[dbo].[SendView] S
INNER JOIN
@ItemIds I ON I.[Id] = TRY_CONVERT(
UNIQUEIDENTIFIER,
CASE WHEN ISJSON(S.[Data]) = 1 THEN JSON_VALUE(S.[Data], '$.Metadata.ItemId') END)
WHERE
S.[Type] = 2ISJSON and TRY_CONVERT keep a malformed Send from failing the call, which matches ItemIdOf in ItemSendCleanupService. The same change is needed in util/Migrator/DbScripts/2026-11-12_00_AddSendReadByItemIds.sql.
|
|
||
| await _cipherRepository.DeleteAsync(cipherDetails); | ||
| await _attachmentStorageService.DeleteAttachmentsForCipherAsync(cipherDetails.Id); | ||
| await _itemSendCleanupService.DeleteItemSendsForCiphersAsync([cipherDetails.Id]); |
There was a problem hiding this comment.
The cipher is already deleted at this point. If the Send lookup times out or throws, Cipher_Deleted is never logged and clients are never told the cipher is gone. The same applies in DeleteManyAsync at line 505.
await _cipherRepository.DeleteAsync(cipherDetails);
await _attachmentStorageService.DeleteAttachmentsForCipherAsync(cipherDetails.Id);
await _eventService.LogCipherEventAsync(cipherDetails, EventType.Cipher_Deleted);
// push
await _cipherSyncPushService.PushSyncCipherDeleteAsync(cipherDetails, collectionIds);
await _itemSendCleanupService.DeleteItemSendsForCiphersAsync([cipherDetails.Id]);|
@rkac-bw With an eye towards the comments you had about this upcoming work I decided it made more sense to follow a similar approach here of locking Send access immediately and deleting the Send via cleanup job afterwards. That way we don't have to worry about any code that gets added in the future that deletes ciphers. |
| UPDATE | ||
| [dbo].[Send] | ||
| SET | ||
| [CipherId] = TRY_CONVERT(UNIQUEIDENTIFIER, JSON_VALUE([Data], '$.Metadata.ItemId')) | ||
| WHERE | ||
| [Type] = 2 | ||
| AND [CipherId] IS NULL | ||
| AND ISJSON([Data]) = 1 |
There was a problem hiding this comment.
The backfill is fine on a prod copy: 3,974 item Sends, 7.7 s, none left NULL. What happens to an item Send created by the current release after this script runs but before the new code is live? It gets CipherId NULL, and Send_ReadItemSendsWithDeletedCipher skips NULLs, so it's denied on access but never deleted. Could the same statement run again in a follow-up script once the release is out? It's already idempotent:
UPDATE
[dbo].[Send]
SET
[CipherId] = TRY_CONVERT(UNIQUEIDENTIFIER, JSON_VALUE([Data], '$.Metadata.ItemId'))
WHERE
[Type] = 2
AND [CipherId] IS NULL
AND ISJSON([Data]) = 1There was a problem hiding this comment.
The access check reads the item ID from the metadata via this function so this shouldn't be a concern. As long as the metadata is there (and that's been enforced the whole time this feature has been enabled, I believe) access won't be lost
| } | ||
|
|
||
| // `send` already carries the incoming data; compare against the persisted Send. | ||
| // An unchanged item keeps the persisted CipherId, which may be null for Sends whose item was already gone. |
There was a problem hiding this comment.
A Send with a NULL CipherId stays NULL when it's saved with the same item. Setting it here would let old rows heal on edit, and if the item is already gone the job then removes the Send:
if (stored != null && ItemIdOf(stored) == itemId)
{
send.CipherId = itemId;
return;
}There was a problem hiding this comment.
I hadn't included it because Sends have a limited lifespan (though that can technically be worked around) so any Sends with a null CipherId should get purged in about 30 days anyway, but the row healing will let them potentially get cleaned up faster. Added.
|
Agreed, this is the better shape. The LIKE lookup is gone and the delete paths don't need to know about Sends. I ran the migration on a prod copy: backfill covers all 3,974 item Sends in under 8 s and the job query takes 45 ms. Two gaps in what gets a |
| /// <inheritdoc /> | ||
| protected override void Up(MigrationBuilder migrationBuilder) | ||
| { | ||
| migrationBuilder.CreateIndex( |
There was a problem hiding this comment.
The SQL Server script backfills CipherId from Data, but the three EF migrations only create the index. On self-host, do existing item Sends ever get a CipherId? As written the job won't pick them up after their item is deleted. Could each provider get the equivalent backfill via migrationBuilder.Sql?
There was a problem hiding this comment.
Similar to my above comment I hadn't included a backfill for those providers because the old Sends should get purged within about 30 days, and also because adding that backfill would require hand-editing some of those generated migration scripts (which our Claude instructions expressly forbid and I wasn't sure if that extends to human developers as well). Is my understanding of that ban accurate?
🎟️ Tracking
https://bitwarden.atlassian.net/browse/PM-43847
📔 Objective
This PR adds code that cleans up item-type Sends whose items have been deleted using the existing
DeleteSendsJob. Access to the Send is locked immediately via a new condition on the access check, but the Send itself doesn't get deleted until the next time the automated job runs (max of 5 minutes, currently). With the client changes introduced here this state is distinguished by a new callout.