Hydrogen Bot Experiencing high Websocket Pings
Incident Report for Hydrogen Studio
Postmortem

Affected Service

  • Hydrogen Bot
  • Hydrogen API
  • H.D. Backend Server Connection
  • H.D. Database Connection

Summary

  • Caused by discord websocket error in Discord’s side.
  • There is a increase after the first fix which resulted in a second issue.
  • Installed ping reducing libraries on our VPS to make sure it happens less often.
  • Learned that everytime we start the bot, there will be about 40 second of high ping(usually > 1000ms)
Posted May 14, 2020 - 11:45 PDT

Resolved
This incident has been resolved.
Posted May 14, 2020 - 10:11 PDT
Update
We had noticed that the ping is now over 600. We will try to let it sit for some time and see if it go down. As I said earlier, this might not be our issue. We will resolve the issue as of now, and will create a new one if needed later.
Posted May 14, 2020 - 10:10 PDT
Update
We are continuing to monitor as we heard that discord api is having high ping today too. This overall might be a discord side issue.
Posted May 14, 2020 - 10:06 PDT
Update
We are continuing to monitor for any further issues.
Posted May 14, 2020 - 10:01 PDT
Monitoring
A fix have been implemented and it does seems to work. We are continuing to monitor.
Posted May 14, 2020 - 09:40 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted May 14, 2020 - 09:33 PDT
Update
We are continuing to investigate the issue. During this time, please do not purchase premium nor redeem anything, as the transactions might be lost.
Posted May 14, 2020 - 09:16 PDT
Investigating
The ping is still unstable, we are investigating the issue
Posted May 14, 2020 - 09:03 PDT
Monitoring
We have finished with the upgrade and we are monitoring the results.
Posted May 14, 2020 - 08:40 PDT
Identified
The issue has been identified and fixed, we are now implementing a websocket upgrade to decrease websocket ping time.
Posted May 14, 2020 - 08:38 PDT
Investigating
We are currently investigating this issue.
Posted May 14, 2020 - 08:27 PDT
This incident affected: Hydrogen Bot, Hydrogen API and Other Components (H.D. - Backend Server Connection, H.D. - Database Connection).