Cloud push gateway accepts pushes (200, counted in Cloud console) but delivers to neither FCM nor APNs — registered workspace, official apps

[hw-marksun]

Description:

Push notifications are never delivered to ANY mobile device on our registered self-hosted workspace, even though every link up to the Rocket.Chat cloud gateway is verified working. This affects real messages (priority 10), not just push_test — see verification below. In-app (websocket) messaging works. Opening the mobile app syncs unread counts from 0 to N, confirming the device receives ZERO background pushes.

Related: #40041 has the identical symptom, but its resolution ("testing methodology — real messages work") does not apply here: we verified with real DMs sent to an offline user and delivery still fails on both platforms.

We instrumented the server and captured the gateway's actual responses through a transparent proxy, so this report contains verified data:

1. Server → gateway works. For every push, debug logs show ExternalRequest https://gateway.rocket.chat/push/gcm/send (and /push/apn/send) followed by Push: Sent message to apps { iosApps: 1, androidApps: 1 }. No 401/406/422.
2. Gateway validates and accepts our tokens. Raw upstream responses captured via proxy: real FCM token → HTTP 200 (empty body); real APNs token → HTTP 200 (empty body); deliberately bogus FCM token → HTTP 422 {"code":125,"error":"invalid push notification payload"}. The gateway distinguishes valid from invalid tokens and accepts ours.
3. Real-message test (decisive). With the recipient confirmed offline (both apps force-closed), another user sent a real DM. The offline email notification for that exact message was delivered (SMTP configured) — proving the notification pipeline fired — while the push leg delivered nothing to either device.
4. Cloud console counts our pushes. cloud.rocket.chat shows "Push Notifications (current month)" incrementing with each send. No limit/block warning; plan shows "Unlimited push notifications".
5. Devices ruled out. Official apps reinstalled, notification permission granted, battery optimization/deep-sleep disabled, devices rebooted. Both devices receive pushes from other apps normally. Tested on Wi-Fi AND cellular. iOS (APNs) and Android (FCM) fail identically — two independent push channels.
6. Tokens registered correctly in _raix_push_app_tokens (one gcm, one apn + voipToken, both enabled:true, appNames chat.rocket.android / chat.rocket.ios).
7. Settings verified: Register_Server=true, Cloud_Service_Agree_PrivacyTerms=true, Push_enable=true, Push_enable_gateway=true, Push_gateway=https://gateway.rocket.chat, Push_production=true. Restarted after changes; shouldUseGateway() satisfied (no native fallback in logs).

The only element common to both failing channels (Apple APNs + Google FCM) is the gateway's downstream delivery for this workspace (Cloud Workspace ID: 6a637449bf7ef133a3aed1ea, rocket.keybase.kr). Could someone with access to gateway logs check the downstream FCM/APNs delivery results for this workspace (e.g. credential/provisioning issue for a newly registered workspace)? It would also help operators if the gateway returned the downstream result instead of an empty 200.

Steps to reproduce:

1. Register a self-hosted 8.6.1 workspace, accept Cloud privacy terms, enable gateway push (Enable + Gateway + Production), restart.
2. Log in with the official mobile app (Android or iOS), allow notifications, force-close the app (user offline).
3. Send a real DM from another user (or use "Send a test push to my user").

Expected behavior:

Pushes accepted by the gateway (HTTP 200) for valid tokens are delivered to FCM/APNs and arrive on devices.

Actual behavior:

Server logs success, the gateway returns 200, the Cloud console counts the sends, and the offline email notification for the same message arrives — but nothing ever arrives on either device (Android FCM and iOS APNs both). Zero deliveries.

Server Setup Information:

• Version of Rocket.Chat Server: 8.6.1
• License Type: Starter (self-managed, workspace registered & Active)
• Number of Users: 2
• Operating System: Ubuntu Linux (Docker host)
• Deployment Method: docker (official image)
• Number of Running Instances: 1
• DB Replicaset Oplog: enabled
• NodeJS Version: 22.22.3 - x64
• MongoDB Version: 8.0.28 (wiredTiger)

Client Setup Information

• Desktop App or Browser Version: N/A (issue is mobile push)
• Operating System: Android (Samsung Galaxy Fold, Play Services healthy) and iOS (iPhone) — official store apps

Additional context

• Site URL https://rocket.keybase.kr (valid HTTPS, matches ROOT_URL).
• Support ticket #115256 was auto-rejected (Starter plan has no dedicated support), so filing here.

Relevant logs:

Push: Sending notification { title: '@shw', text: '...', ... }
Push: send to token { gcm: 'fQA_...' }
ExternalRequest: https://gateway.rocket.chat/push/gcm/send
Push: Sent message to apps { iosApps: 1, androidApps: 1 }
# Proxy-captured upstream responses:
# PATH=/push/gcm/send TOKEN=fQA_... -> STATUS=200 BODY=(empty)
# PATH=/push/apn/send TOKEN=241b... -> STATUS=200 BODY=(empty)
# PATH=/push/gcm/send TOKEN=BOGUS... -> STATUS=422 BODY={"code":125,"error":"invalid push notification payload"}

[coderabbitai]

🔗 Related PRs

#41341 - fix: server crash when re-enabling push notifications [merged]
#41382 - refactor: index token string via tokenValue/tokenType pair [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[Nahid-NHB]

i would love to work on this issue