PTS Results on Provisioned IOPS on AWS EBS

Initial results running the Phoronix Test Suite with Provision IOPS…

Continuing my evaluation of cloud hosted file-systems, I started looking at Provisioned IOPS as offered by AWS. This post is light on instructions and is basically just graphs of performance results.

test configurations

• small-attached-ebs: m1.small EC2 instance, standard EBS volume (ext4)
• small-attached-ebs-piops200: m1.small EC2 instance, EBS volume (ext4) provisioned for 200 IOPS
• small-attached-ebs-piops400: m1.small EC2 instance, EBS volume (ext4) provisioned for 400 IOPS

Note: there’s a 10:1 minimum ratio between the size of an AWS EBS volume and the IOPS you provision for it. As such, the smallest EBS volume that can be provisioned for 200 IOPS is 20GB, and the smallest volume with support for 400 IOPS is 40GB.

results

OpenBenchmarking.org

I ran these tests with the same test name, however PTS didn’t join the results together when it submitted them. There must be a trick to getting PTS to collect results from multiple machines.

Anyway, here are the separate result pages:

• 1308270-SO-JOKEYRHYM25
• 1309225-SO-JOKEYRHYM57
• 1309224-SO-JOKEYRHYM67

comparison

Note: each test is actually run several times, with the below scores representing the average. The “error” column is the maximum deviation from the average by one or more test runs.

AIO-STRESS

FLEXIBLE IO TESTER

SQLITE

FS-MARK

DBENCH

With 48 clients, we see one of those rare moments when performance variance with Provisioned IOPS is higher than without. Strange.

IOZONE

THREADED I/O TESTER

This benchmark highlights a specific case where provisioning 200-400 IOPS performs abysmally compared to no-PIOPS.

COMPILE BENCH

Here’s one of the few cases where provisioning only 200 IOPS still managed to perform significantly better than without.

UNPACKING THE LINUX KERNEL

POSTMARK

GZIP COMPRESSION

Here’s another use case where Provisioned IOPS (at least at the 200 and 400 level) actually has greater performance variance than without.

POSTGRESQL PGBENCH

Not sure why this benchmark didn’t run for the Provisioned IOPS cases. It may simply have been disabled in the version of PTS I used.

APACHE BENCHMARK

Here’s the last case where Provisioned IOPS (at least at the 200 and 400 level) actually has greater performance variance than without.

observations

For almost every benchmark, EBS with Provisioned IOPS had a lower variance between test runs, often drastically lower. This is the stated value proposition for this feature: you pay more, but you get a guaranteed level of performance. Without Provisioned IOPS, EBS performance varies quite a bit with spikes and valleys. If your application depends on consistent storage performance, then you really ought to turn Provisioned IOPS on.

That said, provisioning only 200 IOPS tended to worsen performance in almost every benchmark. While it’s the cheapest option, the compromise here may not be suitable for all applications: trading spikey but decent performance for consistently poor performance.

Provisioning 400 IOPS tended to performance somewhere between no-PIOPS and 200 PIOPS.

conclusions

I chose to examine the Provisioned IOPS feature for 200 IOPS and 400 IOPS. While the results are interesting, I did not find the point where paying for PIOPS brings consistently better performance than not.

According to Amazon’s documentation on EC2 Instance Types, m1.small instances only have a “low” network performance rating. This is likely to affect my test results, as EC2 instances must communicate with EBS volumes over their network interface.

I will have to follow-up these results with a wider range of Provisioned IOPS levels, and perhaps a wider range of instances sizes.

series

1. Phoronix Test Suite on AWS
2. PTS Results on Provisioned IOPS on AWS EBS
3. PTS Results on Provisioned IOPS on AWS EBS, Part 2
4. PTS on AWS with m1.medium EC2 and EBS with Provisioned IOPS

Ron - 23 Sep 2013