GitLab Slack Notifications After Token Rotation: A Runbook

A platform engineer rotates an expired secret on Tuesday morning, updates the project integration settings, and assumes delivery continues as usual. Two hours later, an engineering lead asks in the triage channel why nobody has reviewed a critical hotfix. The review channel has gone completely silent.

Credential rotation is standard security hygiene, but notification pipelines break all the time during it. Teams mix up different credentials, update the wrong configuration field, or revoke the active token before the replacement is verified. In a 20-person engineering team, a silent break in merge request notifications halts code review for hours.

Restoring GitLab Slack notifications after token rotation means treating the notification pipeline like production infrastructure. You need to know which secret handles which boundary, follow an overlapping update sequence, and verify the pipeline with real merge request events before you revoke the old credentials.

Four credentials that get called 'the token' (and why not every setup uses all four)

When an engineer says the GitLab Slack token stopped working, they usually mean one of four distinct credentials. They aren't interchangeable, and treating them that way causes configuration drift and broken deliveries.

The first is the GitLab Webhook Secret Token. GitLab includes this string in the X-Gitlab-Token HTTP header only when a secret token is configured for that webhook, rather than sending it with every outgoing payload. Your receiving endpoint checks this header to verify that the incoming POST request came from your GitLab instance and not an unauthorized third party.

The second is the Slack Incoming Webhook URL. This URL carries an encoded secret directly in its path, formatted as hooks.slack.com/services/T.../B.../XXX. Anyone who holds this URL can post messages to the associated Slack channel without further authentication. If the URL leaks or expires, GitLab cannot deliver messages to Slack.

The third is the Slack App OAuth Bot Token, which begins with the prefix xoxb-. Notification bots and apps use it to call the Slack Web API directly, such as the chat.postMessage or chat.update methods. Unlike static incoming webhook URLs, bot tokens let applications read channel history, post threaded replies, and update existing messages.

The fourth is the GitLab API Token, which can be a Project Access Token, Group Access Token, or Personal Access Token. Slack-side applications use it to query GitLab for merge request details, check pipeline statuses, or resolve user identities. A native webhook integration doesn't use this token at all. Dedicated notification apps like MergeMe, which posts Slack notifications for GitHub pull requests and GitLab merge requests as one updating card per PR or MR, connect through API integrations rather than raw static webhooks.

Before changing any values, identify which of these four tokens you are actually rotating. Reading up on GitLab Slack webhook security will tell you whether your team is managing raw URL endpoints or authenticated bot tokens.

Build a credential inventory without recording secret values

Don't start rotating credentials by opening a secret manager and pasting values into Slack or Jira. Build a metadata-only inventory that lists the role and location of every credential in the chain.

Create a short registry document. For each credential, record four fields: the credential type, the source system, the target system, and the storage location. A typical entry looks like this: Credential Type: Slack Bot OAuth Token; Source: Slack App Admin Console; Target: Self-hosted GitLab CI/CD Variables; Storage: HashiCorp Vault path secret/data/gitlab/slack.

Never record actual secret strings, raw webhook URLs, or token prefixes in your planning notes. If a webhook URL leaks into an issue tracker or team chat, that credential is compromised immediately. Track only the expiration date, the credential identifier or key fingerprint, and the team member responsible for approving the rotation.

Find all downstream consumers before you change anything. A single Slack incoming webhook URL might be pasted across twelve different GitLab repositories within a sub-group. Rotate the URL in the Slack app dashboard without updating eleven of those repositories, and those eleven fail quietly while the one you updated keeps working. Mapping the blast radius protects the whole workspace from silent notification outages.

A controlled change sequence: create, update, verify, then revoke

The most common mistake in secret rotation is revoking the old credential before verifying the new one. Revoke first and you force an immediate outage on your review workflow, plus time pressure that leads to errors.

Use a four-step sequence for every credential change: create, update, verify, then revoke.

First, create the new credential in the authorizing system. In Slack, generate a new incoming webhook URL or app token while leaving the current one active. In GitLab, create a new Project Access Token or generate a new secret token string.

Second, update the consuming system with the new secret. In GitLab, go to Settings > Webhooks, edit the target webhook, and paste the new Secret Token or URL. If you manage configuration through infrastructure-as-code or environment variables, deploy the updated value to your staging or production runners.

Third, verify live delivery while both credentials remain valid. Send an actual test event through the pipeline. Inspect the HTTP response code on the sender side and confirm how the message renders in Slack. If delivery fails, your old credential is still active, so you have time to diagnose the misconfiguration without an emergency.

Fourth, revoke the old credential. Once the verification event passes, go back to the originating dashboard and delete the old secret. Log the revocation timestamp in your audit records so your security team has a clear trail.

Test one merge request event end to end

Don't rely on the Test button in the GitLab Webhook settings page. It sends a mock payload with hardcoded values, and that payload frequently bypasses branch filters, assignee parsing logic, and channel mapping rules.

To verify that notifications work after rotation, run an end-to-end test with a real merge request. Create a dedicated test branch in an active repository, make a trivial edit such as updating a documentation comment, and open a draft merge request assigned to yourself.

Watch the target Slack channel and confirm three behaviors. The notification should appear in the correct channel, not fall back to a default administrative channel. Author and reviewer mentions should render as active Slack mentions, not raw text strings. And modifying the merge request (pushing another commit or changing the MR title) should update the existing Slack notification or post an appropriate threaded reply.

If you see mock test messages in Slack but your real merge request produces nothing, your credentials are valid and your event filters or branch rules are broken. Our guide on why a GitLab Slack test works but merge request notifications missing covers the exact gaps between sample webhooks and live event processing.

Check GitLab Recent events and Slack-side results before trusting the fix

Nobody complaining yet doesn't mean your credentials work. Inspect the delivery logs on both the GitLab side and the Slack side.

In GitLab, go to Project > Settings > Webhooks, find your webhook configuration, and scroll down to the Recent events section. GitLab keeps a detailed delivery log for every outbound POST request. Look at the status code column. A healthy pipeline shows 200 OK. Click View details next to the latest delivery to see the raw HTTP request headers, request body, and response payload.

If delivery failed, the status code tells you where. An HTTP 401 Unauthorized means the Slack incoming webhook URL is invalid or the receiving service rejected your X-Gitlab-Token header. An HTTP 403 Forbidden often points to channel permissions or revoked app scopes. An HTTP 404 Not Found means the target Slack webhook URL was deleted or typed incorrectly.

Next, read the Slack response body directly in the GitLab log viewer. If Slack rejected the payload, the response body contains plain text errors such as invalid_token, channel_not_found, or action_prohibited. For OAuth-based applications, Slack returns a JSON object with an explicit error field, such as {"ok": false, "error": "token_revoked"}. An HTTP 200 in GitLab plus an active notification card in Slack is the only reliable proof that the rotation worked.

Replaying missed events safely: resend rules and duplicate awareness

If notifications were down for several hours during an uncoordinated token rotation, developers probably merged branches and opened review requests with no Slack alerts. GitLab records failed webhook attempts, so you can resend them manually.

In the GitLab Webhook Recent events log, find the deliveries marked with red error codes from the outage window. Click Resend Request next to a failed event to retransmit the exact JSON payload to your Slack endpoint.

Be careful with replays, though. Naive Slack integrations and basic incoming webhooks post a brand-new message into the channel for every request they receive. Resend twenty failed payloads at once and you dump twenty disconnected messages into the team channel, cluttering the room and pinging engineers about pull requests that have already been merged.

Before you trigger bulk replays, review the timestamps. If a merge request was already merged or closed during the outage, resending its open event creates confusing noise. Only replay events for open, pending merge requests that actually need team review. Tools like MergeMe avoid this spam pattern by keeping one updating card per PR or MR across GitHub.com, GitLab.com, and self-hosted GitLab, updating existing context instead of firing new channel alerts.

Rollback limits after revocation, and what to do instead

When a token rotation breaks notifications and your team can't immediately diagnose why, the instinct is to roll back to the previous credential. In modern API architectures, a true rollback after revocation is technically impossible.

Once you click Revoke on a Slack bot token (xoxb-) or delete a GitLab Project Access Token, that secret is permanently destroyed in the database. You cannot restore the old value. Pasting an expired or revoked string back into your configuration just swaps one broken state for another.

Skip the rollback and recover forward. Generate a brand-new credential in the authorizing system with standard administrative scopes. Update the receiving endpoint with the fresh token, verify that basic connectivity returns, and do the root cause analysis afterward in a non-production test repository.

To make forward recovery fast, document the exact scopes your integration needs in your runbook. For Slack bot tokens, record the required OAuth scopes, such as chat:write, chat:write.public, and channels:read. For GitLab Project Access Tokens, note whether the token needs the api or read_api permission scope. With permissions pre-documented, nobody has to guess during an outage.

Post-rotation checklist and where to go for network and mapping issues

Complete a formal post-rotation checklist before closing your maintenance ticket. Confirm that all old tokens have been revoked in the source system, the new credential is saved in your permanent secret store, the GitLab Recent events log shows 200 OK for at least three consecutive live events, and team members can trigger updates on active merge requests.

If the Recent events log shows ongoing connection timeouts or 500-series errors after a verified credential change, the token isn't your problem. It's a network or firewall boundary. Self-hosted GitLab instances often have trouble reaching external Slack API endpoints when outbound proxy rules or local egress security groups change during maintenance. Check our guide on self-hosted GitLab Slack webhook troubleshooting to sort out local proxy and network issues.

Finally, verify user ID mappings. If notifications post to the channel but fail to @mention the right developers, your Slack user mapping table may have lost sync during the rotation. Make sure the integration can match GitLab usernames to Slack member IDs so reviewers keep getting direct alerts.

Conclusion

Rotating notification credentials should be an orderly routine, not an event that halts code reviews. Separate webhook secrets from OAuth bot tokens, follow a create-update-verify-revoke sequence, and test against live merge requests. Do that and silent notification failures mostly disappear.

If your team is tired of managing brittle webhook URLs and sorting through duplicate notification spam across multiple repositories, look at MergeMe. MergeMe posts Slack notifications for GitHub pull requests and GitLab merge requests as one updating card per PR/MR, and supports GitHub.com, GitLab.com, and self-hosted GitLab in one shared workspace. The Hobby tier is free forever with 1 channel mapping and 5 user mappings, and paid Team plans are £5 per dev seat monthly. Set up MergeMe and give your engineering team clean, dependable review notifications without the maintenance overhead.