Bad Gateway
laineethecat
REGISTERED
Game: freecell
Game #: 2068911363
Browser: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
When is the "bad gateway" problem going to be fixed? Very annoying.
Comments
ditto
We'll fix it when we figure out exactly what it is! ;-) I've attached a screenshot of our server monitoring stats. We can see the symptoms very clearly. Notice the white line (server process load)—it spikes up every day around 4:30pm PST and oscillates up and down once an hour. In the lower graph you can see the blue line spike up at the exact same times. That's the disk response time and it's what we believe is driving the load up. While it is high the server starts running slowly and people start getting 502 Gateway timeouts (because it can't respond to individual requests fast enough).
Since the spikes happen at precisely the same time each hour, we're pretty sure at least one of the components of the problem is a scheduled task (called a 'cron job'). But the confusing thing is that this task runs 24 times a day, yet only the ones in the late afternoon/evening make the server go crazy. That part doesn't make sense to us. Perhaps the job has different characteristics at different times of day? Or maybe it just gets slower and slower as the day goes on until it hits a breaking point that makes everything bad? Or perhaps the disk that's getting bad response times is going bad? (knock on wood it's not that one!)
To sum up…we're working on it. It's not as straightforward as it seems.
If we can definitively identify the issue and solve it, we'll probably be excited enough to write a blog post about it. :-)