Skip to content

[PM-42172] feat: Separate the Identity and SSO certificates, and rotate each one without downtime - #8850

Open
enmande wants to merge 60 commits into
mainfrom
auth/pm-42172/identity-and-sso-certificate-framework
Open

enmande wants to merge 60 commits into
mainfrom
auth/pm-42172/identity-and-sso-certificate-framework

Conversation

@enmande

@enmande enmande commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🎟️ Tracking

PM-42172

📔 Objective

Give each server certificate its own settings, and let each one rotate with configuration changes and restarts only.

Role Host Settings
Identity signing /identity globalSettings:identityServer
SSO signing /sso globalSettings:ssoServer
SSO encryption /sso globalSettings:ssoServer:encryption

What changes:

  • Each role has a futureCertificate slot (enabled, active) and a list of outgoing certificates (validationCertificates or decryptionCertificates).
  • The Identity JWKS publishes the staged and validation keys. Only the active key signs.
  • /sso signs its OIDC tokens with the SSO signing certificate, not the Identity certificate.
  • SAML signing and SAML decryption use separate certificates. /sso decrypts with the active, staged, and outgoing encryption certificates.
  • A host stops at startup when a configured certificate does not load, has no RSA key, or has no private key where it needs one. The error does not show the password.
  • Each host logs the source and the thumbprints of each role at startup.
  • src/Core/Settings/IDENTITY_SSO_CERTIFICATE_ROTATION.md describes the settings, the rotation steps, and rollback.

Deployment

  • A cloud /sso host must set globalSettings:ssoServer and globalSettings:ssoServer:encryption. Without them, /sso does not start.
  • A self-hosted or Development /sso host with no SSO settings uses the Identity certificate for all three roles, as it does today.

Testing

  • Unit and integration tests cover loading, the JWKS, the SAML metadata, the fallback, and the startup failures.
  • An end-to-end test on real local hosts (development, self-hosted, and cloud with Azurite) passed 76 of 76 checks, including each rollback.
  • A continuity test signed in a password user and a SAML user with encrypted assertions at each rotation step and rollback. The only failures were expected: the first request after promotion to a token consumer that still had the old JWKS cached. The next request passed.

📸 Screenshots

Recordings of the continuity test: a real SAML sign-in in the browser at each rotation step.

Identity

Shows promotion steps with rollbacks for each.

r1-identity-rotation-cloud.mp4

SSO

Shows promotion steps with rollbacks for each.

r2-sso-split-rotation-cloud.mp4

🤖 AI-assisted review

  • Path: multi-agent review of the full branch against main.
  • Result: no blocker and no important finding. Three refactor findings are fixed in this branch. One finding was dismissed as a duplicate.
  • Not covered: the Claude configuration check did not run, because its plugin is not installed. The branch changes bitwarden_license/src/Sso/CLAUDE.md.

…handling

These tests record the current behavior of CouldHandleAsync before
PM-42283 changes it. The tests cover:
- Requests with the scheme query parameter, with and without a SAML
  message.
- RSA 1.5 and OAEP encrypted assertions, signed and unsigned.
- The message that the parser reads for each HTTP method, and the
  precedence of SAMLRequest and SAMLResponse.
…ing, behind a flag

When the pm-42283-rsa-1_5-deprecation flag is on, CouldHandleAsync reads
the SAML message with the same Sustainsys.Saml2 binding that the handler
uses. Thus, CouldHandleAsync inspects the same message that the handler
processes.

The scheme query parameter now applies only to a request without a SAML
message, such as a metadata or login-initiation request. Each request
with a SAML message gets the issuer check and the signature check.

When the flag is off, the behavior does not change.
…sertions

When the pm-42283-rsa-1_5-deprecation flag is on, CouldHandleAsync
accepts an encrypted assertion only when each key uses rsa-oaep or
rsa-oaep-mgf1p key transport. CouldHandleAsync rejects an encrypted
assertion that has no key.

This check runs before the signature check. This check also applies
when the configuration does not require signed assertions.

The RSA 1.5 deprecation notice runs before this check. Thus,
administrators get the notice when login fails.

The exception message is constant.
@enmande
enmande marked this pull request as ready for review November 11, 2026 13:45
@enmande
enmande requested a review from a team as a code owner November 11, 2026 13:45
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.39759% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.05%. Comparing base (d781428) to head (c2a3bd4).
⚠️ Report is 22 commits behind head on main.

Files with missing lines Patch % Lines
...SharedWeb/Utilities/ServiceCollectionExtensions.cs 97.87% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8850      +/-   ##
==========================================
+ Coverage   68.69%   69.05%   +0.36%     
==========================================
  Files        2759     2762       +3     
  Lines      118196   118543     +347     
  Branches    10859    10891      +32     
==========================================
+ Hits        81189    81859     +670     
+ Misses      34454    34125     -329     
- Partials     2553     2559       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JaredSnider-Bitwarden JaredSnider-Bitwarden left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great so far. I'm not done reviewing but will be out rest of day so leaving these here for ya for now.

…ettings

Declare the certificate settings members as nullable strings.
Require a blob name before the host loads a certificate from blob storage.
Add a test that a cloud future certificate with only a password stops startup.
Keep the pinned Azurite image in one constant in test/Common.
Point the Renovate regex manager at that file, so one update changes every suite.
…y-and-sso-certificate-framework

# Conflicts:
#	bitwarden_license/test/Sso.IntegrationTest/Saml2/Acs/SsoCertificateTests.cs
#	test/Identity.IntegrationTest/TokenEndpoint/DeviceLastActivity/IdentityJwksTests.cs
…in Certificates folders

Move the Identity certificate tests into Certificates/ and split them into
IdentitySigningKeyPublicationTests, IdentityTokenSigningTests, and
IdentityCertificateStartupTests. Move the SSO certificate tests into
Certificates/ and split them into SsoSigningKeyTests,
SsoMetadataCertificateTests, SsoAssertionDecryptionTests, and
SsoCertificateStartupTests.

Move the helpers that more than one class uses into CertificateTestHelpers and
SsoCertificateTestHelpers. Rename the test about a retired certificate that
failed naming "the list entry" to name "the retired entry".

No test logic changed.
…fier tests

The tests wait for a background task. On a loaded CI runner, the task can start after the 5-second limit. A 30-second limit stops these false failures. A passing test does not wait longer.
…y-and-sso-certificate-framework

# Conflicts:
#	bitwarden_license/test/Sso.IntegrationTest/Flows/Saml2SpInitiatedLogin/Saml2SpInitiatedLoginFlowTests.cs
#	bitwarden_license/test/Sso.IntegrationTest/Helpers/Saml2SpInitiatedLoginSteps.cs
#	bitwarden_license/test/Sso.IntegrationTest/Helpers/SsoCertificateTestHelpers.cs
…e source check after the slots

The Identity certificate type holds the certificates that the Identity settings configure in each slot. The new name, IdentityCertificateSlots, matches SsoCertificateSlots. Its Validation property is now Retired, which matches the slot names in the rotation document. The active source check now says that it reads only the settings.
…dentity fallback

SsoCertificateSlots now records whether the role has an active certificate settings source. It no longer records the Identity fallback. The two resolve methods decide the fallback and set it on the set that they return. The summaries of both sets now say that the set holds the certificates that the role uses after the fallback.
…t read the Identity slots

No test checked this case. The existing tests used Identity certificates that load. The new test makes each Identity slot read throw. It fails when a resolve method reads an Identity slot before it checks its own source. It also fails when the Identity slots load eagerly.
… when each host loads them

The document now defines the terms role, slot, and set. It also lists the slots that each host loads at startup. The self-hosted section now calls the Identity validation certificates the retired certificates.
… certificate sets

The signing and encryption sets now call their third list Retired, as the slots, the startup report, and the rotation document do. The staged certificates also validate or decrypt, so the old names did not tell the two lists apart. The setting names do not change.
The comment on the Azurite image now gives the reason for the pin.
…ate does not load

A host now names globalSettings:identityServer and what to check when the Identity root certificate does not load. Hosts start and stop in the same cases as before. A Development host still uses a developer signing key.
…espaces

The certificate role, slot, set, and report types are models, not utilities. They now live in Bit.SharedWeb.Certificates and Bit.Sso.Certificates, and their tests follow the same folders.
…tificate, and name rollback tests by rotation step

@JaredSnider-Bitwarden JaredSnider-Bitwarden left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work on a really hard problem. Thank you for your iterations and thoroughness in testing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-review Request a Claude code review needs-qa t:feature Change Type - Feature Development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants