Mar 3, 05:54 UTC
Resolved - Between March 2, 21:42 UTC and March 3, 05:54 UTC project board updates, including adding new issues, PRs, and draft items to boards, were delayed from 30 minutes to over 2 hours, as a large backlog of messages accumulated in the Projects data denormalization pipeline.

The incident was caused by an anomalously large event that required longer processing time than expected. Processing this message exceeded the Kafka consumer heartbeat timeout, triggering repeated consumer group rebalances. As a result, the consumer group was unable to make forward progress, creating head-of-line blocking that delayed processing of subsequent project board updates.

We mitigated the issue by deploying a targeted fix that safely bypassed the offending message and allowed normal message consumption to resume. Consumer group stability recovered at 04:10 UTC, after which the backlog began draining. All queued messages were fully processed by 05:53 UTC, returning project board updates to normal processing latency.

We have identified several follow-up improvements to reduce the likelihood and impact of similar incidents in the future, including improved monitoring and alerting, as well as introducing limits for unusually large project events.

Mar 3, 05:53 UTC
Update - This incident has been resolved. Project board updates are now processing in near-real-time.

Mar 3, 04:36 UTC
Update - The backlog of delayed updates is expected to fully clear within approximately 1 hour, after which project board updates will return to near-real-time.

Mar 3, 04:17 UTC
Update - The fix has been deployed and processing speeds have returned to normal. There is a backlog of delayed updates that will continue to be worked through — we're estimating how long that will take and will provide an update in the next 60 minutes.

Mar 3, 03:22 UTC
Update - The fix is still building and is expected to deploy within 60 minutes. The current delay for GitHub Projects updates has increased to up to 5 hours.

Mar 3, 02:27 UTC
Update - We're deploying a fix targeting the increased delay in GitHub Projects updates. The rollout should complete within 60 minutes. If successful, the current delay of up to 4 hours should begin to decrease.

Mar 3, 01:40 UTC
Update - The delay for project board updates has increased to up to 3 hours. We've identified a potential cause and are working on remediation.

Mar 3, 00:52 UTC
Update - Project board updates — including adding issues, pull requests, and changing fields such as "Status" — are currently delayed by 1–2 hours. Normal behavior is near-real-time. We're actively investigating the root cause.

Mar 3, 00:05 UTC
Update - The impact extends beyond adding issues to project boards. Adding pull requests and updating fields such as "Status" may also be affected. We're continuing to investigate the root cause.

Mar 2, 23:46 UTC
Update - Newly added issues are taking 30–60 minutes to appear on project boards, compared to the normal near-real-time behavior. We're investigating the root cause and possible mitigations.

Mar 2, 23:12 UTC
Update - Newly added issues can take up to 30 minutes to appear on project boards. We're investigating the cause of this delay.

Mar 2, 23:11 UTC
Update - Issues is experiencing degraded performance. We are continuing to investigate.

Mar 2, 23:10 UTC
Investigating - We are investigating reports of impacted performance for some GitHub services.