Bees with machine guns! Low-cost, distributed load-testing using EC2. | News Apps Blog
In June we launched Chicago Breaking Business, which is, at least in terms of traffic, our largest site so far. Both before and during that project I had been iterating toward a load-testing solution that would meet the challenge of the above quote and provide us with a reasonable certainty that sites of this size (and larger) would not collapse under sudden waves of traffic. The traditional load-testing tools we tried (ab, flood, jmeter, etc.), while often useful, suffered from a number of problems when trying to simulate this sort of event:
What we needed was a solution that allowed us to use multiple, topographically-close clients to all simultaneously assault our servers with traffic.
Bees with machines guns is a fabric script that I put together to aggressively load-test our sites. It provides methods to spin up an army of EC2 instances (the bees), assault a target server with a configurable amount of traffic, and then spin those instances back down.
(Note: Bees assumes you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set as environment variables–just as you would if you were using the Amazon command line tools.)
Using the Bees I was able to verify that our new load-balanced and Varnished servers could handle a sustained load significantly beyond our requirements. Obviously this testing strategy doesn’t attempt to address all scenarios and it should be used in conjunction with traditional testing tools such as jmeter. That being said, it also provides a pretty interesting example of using EC2 to solve problems that can really only be properly addressed with distributed computing. This particular example is very well-suited to validating caching and scaling strategies.