four bean soup

The other day, we took tweet hopper down for some unexpected maintenance. I just wanted to take a minute to talk about what we did and why it had to happen.

Server additions

Since we changed out all our hardware when we switch from Handcrafted to Four Bean Soup, we didn’t have the need for the same shared resources as before. All our linode servers are now solely dedicated to tweet hopper, we moved from six shared servers down to three dedicated boxes.

Three servers proved to work ok, but once we got some started getting stats, it wasn’t what we were used to serving up to customers. We try to keep most page loads under 200ms and a YSlow score of 80+. We were averaging a few seconds for page loads, as you can see in this graph below.

The unexpected downtime came about because I didn’t realize how slow the service was running. I knew it wasn’t up to par with what we offered before, but I didn’t have any insight into real metrics. The morning the service went down was the morning New Relic finally released a version that supported Unicorn for stat tracking. We immediately set to work installing a dedicated mysql server and put it into service at the first real slow time in the EST time zone - usually we shoot for midnight, same this time.

The results are spot on with what we wanted out of the upgrade. Our page load times dropped to 85ms on average, with some spikes hitting 150ms. If anyone is looking for a good mysql config, we used Evan Weaver’s 512mb vps setup. We will continue to use New Relic’s rpm service to monitor and find slow spots in our app, we love it!