feat: KMS-backed JWT signing for GitHub App authentication (private key never leaves HSM)

[vegardx]

Problem

The GitHub App private key is currently stored in SSM Parameter Store (encrypted at rest) and loaded into Lambda memory for every JWT signing operation. This means:

1. The plaintext private key is in Lambda memory during execution — any memory dump, debug log, or compromised dependency could exfiltrate it
2. No hardware security boundary — the key exists as a software secret, not in an HSM
3. Single App limitation — the module is hardwired to one App's credentials via global env vars, making multi-App deployments impossible without separate infrastructure stacks

Proposal

Move JWT signing to AWS KMS using an asymmetric RSA key (RSA_2048, SIGN_VERIFY, RSASSA_PKCS1_V1_5_SHA_256). The private key never leaves the HSM — Lambda calls kms:Sign and receives only the signature bytes.

Why this is a clean change

The codebase already has the perfect abstraction point. @octokit/auth-app accepts a custom createJwt callback (mutually exclusive with privateKey), and this repo already uses it:

// auth.ts — current implementation
const createJwt = async (appId, timeDifference?) => {
  const jwt = signJwt({ iat, exp, iss: appId, jti: randomUUID() }, privateKey);
  return { jwt, expiresAt: ... };
};
return createAppAuth({ appId, createJwt });

The change is swapping what happens inside signJwt:

// Before: node:crypto with in-memory key
const signature = createSign('RSA-SHA256').update(message).sign(privateKey, 'base64url');

// After: KMS Sign API (key never exposed)
const result = await kmsClient.send(new SignCommand({
  KeyId: kmsKeyArn,
  Message: Buffer.from(message),
  MessageType: 'RAW',
  SigningAlgorithm: 'RSASSA_PKCS1_V1_5_SHA_256',
}));
const signature = Buffer.from(result.Signature!).toString('base64url');

Design

// New: jwt-signer.ts
export interface JwtSigner {
  sign(message: string): Promise<string>; // returns base64url signature
}

export class LocalJwtSigner implements JwtSigner { /* existing node:crypto path */ }
export class KmsJwtSigner implements JwtSigner { /* KMS Sign API */ }

Guard: env var GITHUB_APP_KMS_KEY_ARN — if set, use KMS signer. If unset, fall back to SSM PEM (100% backwards compatible).

Multi-App foundation

This architecture naturally extends to multiple GitHub Apps:

1. All Apps share the same webhook secret → single webhook Lambda verifies all payloads
2. Webhook payload carries installation.id → already passed through to scale-up Lambda
3. Scale-up resolves credentials by installation_id → each App has its own KMS key
4. Token cache (PR feat(lambda): cross-Lambda installation token cache via DynamoDB #5132) is already keyed by installation_id → each App's tokens cached independently

The initial PR implements single-App KMS signing. Multi-App credential resolution (DynamoDB lookup by installation_id) is a follow-up.

Key import constraint

GitHub generates the RSA key pair when you create an App. The private key must be imported into KMS (EXTERNAL key material origin). This is a one-time setup operation.

Terraform: aws_kms_external_key resource handles the lifecycle.

KMS rate limits

Default KMS quota: 300 TPS for RSA signing (region-dependent). Combined with the token cache (#5132, ~1 JWT sign/hour/installation), this is never a bottleneck.

Terraform changes

• Optional aws_kms_external_key resource (gated by variable)
• IAM policy: kms:Sign scoped to the key
• Lambda env var: GITHUB_APP_KMS_KEY_ARN
• Documentation on importing the GitHub App PEM into KMS

Benefits

Aspect	SSM (current)	KMS (proposed)
Key at rest	Encrypted in SSM	HSM-backed, never extractable
Key in memory	Plaintext in Lambda during execution	Never in Lambda memory
Audit trail	SSM GetParameter logs	KMS Sign logs (CloudTrail)
Key rotation	Manual SSM param swap	KMS key material re-import
Multi-App	Not supported	Natural extension via per-App keys

Related

• Support for multiple GitHub Apps to overcome API rate limits at scale #5037 — Support for multiple GitHub Apps (this enables it)
• feat(lambda): cross-Lambda installation token cache via DynamoDB #5132 — Token cache (prerequisite — ensures KMS rate limits are never hit)

---

I will provide a PR for this.

[vegardx]

Note: Multi-App webhook deduplication

If multiple GitHub Apps are installed on the same org/repo, GitHub delivers a separate webhook event per App for the same workflow_job. Each event has a different installation.id but the same workflow_job.id.

Without deduplication, the webhook Lambda would dispatch the same job to SQS N times (once per App), causing N runners to be created for a single job.

Solutions to explore:

1. Deduplicate at webhook Lambda — track workflow_job.id in a short-lived cache (DynamoDB with TTL, or SQS FIFO with MessageDeduplicationId = workflow_job.id) and only dispatch the first event received
2. Single App receives webhooks, others are API-only — only one App is configured with the webhook URL; the others are used purely for token minting (installed for API access, no webhook endpoint configured)
3. Partition by org — each App is installed on a disjoint set of orgs, so no duplicate events occur

Option 2 is likely the simplest: one "webhook App" handles event delivery, and additional "API Apps" are only used to spread rate limit budget for token minting and API calls. The scale-up Lambda picks which App to use for API calls based on a round-robin or load-based strategy, independent of which App delivered the webhook.

[vegardx]

Correction to the above — the rate limit budget is per-installation (= per-App-per-org). If you have one App on an org, you get one 5,000 req/hour bucket. To get more budget for the same org, you need multiple Apps installed on that org — each gets its own bucket.

This means Option 2 ("API-only Apps") and Option 3 ("partition by org") don't solve the problem. You specifically need multiple Apps on the same org to multiply the rate limit budget, which guarantees duplicate webhook delivery.

So deduplication is the core challenge, not a corner case. The webhook Lambda must handle N events for the same workflow_job.id and only dispatch once. SQS FIFO with MessageDeduplicationId = workflow_job.id is probably the cleanest solution — exactly-once delivery within the 5-minute dedup window, no external state needed.

[vegardx]

Splitting the thread: my two comments above went into multi-App webhook deduplication, which is a different problem from what this issue is titled for. Moved to $NEW so both can be tracked properly.

This issue stays as originally written: moving JWT signing to KMS so the App private key never leaves the HSM.