PTS Results on Provisioned IOPS on AWS EBS, Part 2

More results running PTS with PIOPS 1000 and 2000

Continuing my evaluation of cloud hosted file-systems, I started looking at Provisioned IOPS as offered by AWS. This post is light on instructions and is basically just graphs of performance results.

test configurations

• small-attached-ebs: m1.small EC2 instance, standard EBS volume (ext4)
• small-attached-ebs-piops200: m1.small EC2 instance, EBS volume (ext4) provisioned for 200 IOPS
• small-attached-ebs-piops400: m1.small EC2 instance, EBS volume (ext4) provisioned for 400 IOPS
• small-attached-ebs-piops1000: m1.small EC2 instance, EBS volume (ext4) provisioned for 1000 IOPS
• small-attached-ebs-piops2000: m1.small EC2 instance, EBS volume (ext4) provisioned for 2000 IOPS

Note: there’s a 10:1 minimum ratio between the size of an AWS EBS volume and the IOPS you provision for it. As such, the smallest EBS volume that can be provisioned for 200 IOPS is 20GB, and the smallest volume with support for 400 IOPS is 40GB.

results

OpenBenchmarking.org

I ran these tests with the same test name, however PTS didn’t join the results together when it submitted them. There must be a trick to getting PTS to collect results from multiple machines.

Anyway, here are the separate result pages:

• 1308270-SO-JOKEYRHYM25
• 1309225-SO-JOKEYRHYM57
• 1309224-SO-JOKEYRHYM67
• 1309230-SO-JOKEYRHYM39
• 1309233-SO-JOKEYRHYM49

comparison

Note: each test is actually run several times, with the below scores representing the average. The “error” column is the maximum deviation from the average by one or more test runs.

AIO-STRESS

FLEXIBLE IO TESTER

SQLITE

FS-MARK

DBENCH

IOZONE

THREADED I/O TESTER

This is one benchmark where PIOPS 2000 performance was actually double that of PIOPS 1000. Now that we have 4 data points for PIOPS levels, it looks like this test scales linearly with the PIOPS level.

With a bit of napkin math, we can see that PIOPS 2000 is roughly 5 time slower than no-PIOPS for random reads, and PIOPS 1000 is roughly 3 times slower than no-PIOPS for random writes. So, for this test, it would take PIOPS 10k to match no-PIOPS read performance, and PIOPS 3k to match its write performance.

COMPILE BENCH

UNPACKING THE LINUX KERNEL

POSTMARK

GZIP COMPRESSION

POSTGRESQL PGBENCH

So apparently this benchmark ran as expected with PIOPS 1000 and 2000. At some point, I’ll need to go back and re-run with 200 and 400 to see what really happened here. :S

APACHE BENCHMARK

This is a pretty dull benchmark. It doesn’t look like PIOPS 200 through to 2000 makes much of a difference in performance.

observations

PIOPS 1000 and 2000 tend to out-perform PIOPS 400, which in turn out-performs PIOPS 200.

Cases where PIOPS 1000 clearly out-performed no-PIOPS:

• AIO-Stress
• SQLite
• FS-Mark
• Dbench with 1 client
• Compile Bench: Compile
• PostgreSQL pgbench

Cases where PIOPS 1000 performed about the same as no-PIOPS (± 10%):

• Flexible IO Tester
• Dbench with 12, 48 and 128 clients
• IOzone
• Compile Bench: Initial Create and Read Compiled Tree
• Unpacking The Linux Kernel
• PostMark
• Gzip Compression
• Apache Benchmark

Cases where PIOPS 1000 and 2000 were out-classed by no-PIOPS:

• Threaded I/O Tester with 32 threads

conclusions

For most cases, it seems that PIOPS 1000 is sufficient to smooth out the spikey performance of no-PIOPS without sacrificing performance.

PIOPS 2000 was not anywhere near double the performance of PIOPS 1000, except for the Threaded I/O Tester benchmark. This may be due to the “low” performance rating of the network interface on the m1.small EC2 instance. If you’re going to be using m1.small instances, it does not seem worth going past PIOPS 1000.

If you actually care about EBS performance and can afford it, then you’ll probably be using an “EBS-optimized” size (e.g. m1.large) that offers additional network bandwidth and EBS performance. The tests I’ve been running so far cover cases where capacity and cost-effectiveness are more important than performance, but I will be testing with different sizes in future articles.

series

1. Phoronix Test Suite on AWS
2. PTS Results on Provisioned IOPS on AWS EBS
3. PTS Results on Provisioned IOPS on AWS EBS, Part 2
4. PTS on AWS with m1.medium EC2 and EBS with Provisioned IOPS

Ron - 24 Sep 2013