Deconstructing Amazon EC2 Spot Instance Pricing
But what if you look at the price, compared with the fraction of the time in which it was possible to get an instance for this price? We call this the availability of the price. Then some funny patterns show. Beautiful Straight lines. The patterns become even funnier, when the prices are normalized by the on-demand price, so they are all on the same scale. Then all the supposedly market-driven prices form graphs which fall exactly one on top of the other. That is, for all regions but one. What is so special about US-east? And, for Linux. Windows instances have price groups. Do market rules work differently when instances run Linux?
More funny features are changes in the floor price, or in the often visited highest price, which were coordinated across different continents and instance types. As far as I stretch my imagination, I cannot believe this to be the result of market forces.
If prices are not set by market forces, they are artificially set, although Amazon claim the prices are market driven. If prices are artificial, clients should not look for market effects in the prices. For example, the often visited band of prices, above which we hardly ever see prices, is not market driven. Rises in the price do not necessarily say anything about supply and demand, nor do low prices. These are all the result of an algorithm we are told little about.
If prices are the result of a secret algorithm, then pricing methods are subject to quick and unforeseen qualitative changes. In particular, it seems like Amazon update their algorithms every 6 months. Sometimes they announce the change (December 2009, July 2011), and sometimes they don't (Jan and Feb 2011, Feb 2010). So clients, watch out during January 2012 - you might be surprised then.
If the algorithm Amazon use is artificial, it should be possible to deconstruct it. In our recent study, we present a model which accurately mimics the EC2 pricing algorithm. Our model shows that EC2 traces are consistent with Amazon using a dynamic hidden reserve price. That is, Amazon uses a random method (a truncated auto-regressive process) to determine the next reserve price: a price below which no instances are granted, even if it means Amazon's computers will remain unused. If you want to read more, see our finding in Technion
CS Tech report number CS-2011-09.