Slow down in event queuing and potential dropped events
Incident Report for Automation for Jira
Resolved
All events are now being processed as we receive them.
Underlying cause was an error deserialising comments on incoming web hooks.
This code has been reverted and we sure up event queuing, test and redeploy once we are happy.

Apologies if you saw any impacts from this.
Posted 11 months ago. Dec 19, 2017 - 22:10 AEDT
Investigating
Alarms indicated that event queuing times (we queue them before we process them) were creeping up from sub 10 millisecond to ~15 seconds. This caused rules to execute on a slight delay.
We also investigating whether any events were dropped during this period.
We are reverting to handling events as soon as we receive them.
Posted 11 months ago. Dec 19, 2017 - 21:46 AEDT