After the latest price drops and performance improvement announces it’s time to test Amazon’s RDS performance with Provisioned IOPS Storage using EC2 EBS-Optimized instances.
Why this benchmark?
The goal is to measure the crude performance of different Provisioned IOPS RDS instances together with different types of EC2 EBS Optimized instances. Main metrics considered for this test were Throughput and Average Latency. Benchmark was conducted using the YCSB: “Yahoo! Cloud Serving Benchmark” and its workloads.
A brief introduction to YCSB
Yahoo! Cloud Serving Benchmark project is aimed to ‘develop a framework and common set of workloads for evaluating the performance of different “key-value” and “cloud” serving stores’.
YCSB is written in Java and available thru GitHub. Current release (0.1.4) contains code to interface with Cassandra, DynamoDB, VMware vFabric GemFire, GigaSpaces XAP, HBase, Infinispan, JDBC, MapKeeper, MongoDB, Oracle NoSQL Database, Redis and Voldemort.
…and its Core Workloads
YCSB comes with a set of workloads ready for obtaining significant benchmark numbers for wide number of database systems. Even if people are encouraged his own, we used the core ones to make our results reproducibles. That’s a short description of the workloads (as in the GitHub page) to help you to understand the benchmark results:
- Workload A: Update heavy workload This workload has a mix of 50/50 reads and writes. An application example is a session store recording recent actions.
- Workload B: Read mostly workload This workload has a 95/5 reads/write mix. Application example: photo tagging; add a tag is an update, but most operations are to read tags.
- Workload C: Read only This workload is 100% read. Application example: user profile cache, where profiles are constructed elsewhere (e.g., Hadoop).
- Workload D: Read latest workload In this workload, new records are inserted, and the most recently inserted records are the most popular. Application example: user status updates; people want to read the latest.
- Workload E: Short ranges In this workload, short ranges of records are queried, instead of individual records. Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id).
- Workload F: Read-modify-write In this workload, the client will read a record, modify it, and write back the changes. Application example: user database, where user records are read and modified by the user or to record user activity.
To run this benchmark we used:
- YCSB version 0.1.4 and its core workloads;
- The latest MySQL JDBC client available: version 5.1.25;
- Default Oracle JDK version 7-21;
- Linux AMI 2013.03;
- RDS Instances with MySQL engine version 5.5, single AZ, no Backup and Maintenance window set outside tests hours;
- EC2 and RDS instances were running in eu-west-1 region and in the same availability zone, executing only YCSB workloads, nothing more;
- S3 to save batch-conducted tests for further processing.
Our goal was to measure Latency and maximum throughput obtainable using a combination of EC2 EBS Optimized instances and RDS instances using Provisioned IOPS. Instance type were chosen on the “average man needs” following the scaling-out principle as in cloud environments we usually spread workloads on many cost-saving machines to improve resiliency instead of concentrate all resources in a few costly machines. Dataset size are a compromise between number big enough to be significative for our testing needs and benchmark’s execution time.
To obtain this we followed two distinct plans:
Measuring Average Latency constancy using PIOPS RDS Instances
In this test we used a single EC2 and RDS Instances, summarized in the following table:
First Batch: different EC2 and RDS types with the same dataset size:
To better represent data we used Google Chart for this post’s graphics. Please note that where the Legend isn’t repeated is because it’s the same of the previous graph. Single line legend will show hovering the pointer over a particular point/line.
Conclusions on First Batch
Using EC2′s instances non ebs-optimized and RDS instances without Provisioned IOPS produces variable results especially in Write Latency, even if sometimes throughput is higher in our test it was always lower; Mixing EBS-Optimized instances with non PIOPS RDS instances do not change previous statement.
EC2 EBS Optimized instances and RDS instances using PIOPS can guarantee a constant throughput even if increment is not as linear as RDS Provisioned IOPS increments and it will be explained later on. Measured Read and Update (write) latencies were basically stable, tending to decrease while increasing PIOPS.
Second Batch Same instances and different dataset sizes:
Conclusions on Second Batch
As we can see in the graph, latencies and throughput are practically constant despite the datasets’ size growth and PIOPS increments; the slight decrease of latency while throughput increases is more to be attributed to the statistical calculation that to the increase of Provisioned IOPS. Comparing same instances with the same dataset leads to First Batch conclusions.
Notes on workloads d, e and f
Workload d insert continuously new records and it’s heavily affected by latency variations proper of Instances not using PIOPS.
As workload e involves short ranges scan it shows more variations while changing RDS Type than when increasing PIOPS. Changing dataset sizes clearly affect SCAN AverageLatency(us) while INSERT AverageLatency(us) is practically constant.
Workload f requires reading, modifying and writing back to the database the same records, here again RDS Instances not using PIOPS are not the best choice. Raising Provisioned IOPS does not produce performance improvement using 1 or 2 clients.
Measuring Throughput increments while scaling up PIOPS and vary instances types
As noted in the previous tests, using a single EC2 Instance and a single RDS, the Average Overall Throughput is ~1100 ops/sec, with small increases while scaling PIOPS and instances types. To effectively measure the throughput we added to the testing rig more EC2 machines running YCSB clients in cluster. We used the a workload considering this workload a good OLTP test with its mix of 50/50 reads and writes. Client number were raised until a significative deterioration in the Update Latency which indicates that we reached the maximum performance for a Provisioned IOPS RDS Instance.
In this graphs (reporting only the significative tests) we can see that the maximum throughput for the combination of EC2 m1.large and RDS db.m1.large with 1000 IOPS is reached with 3 clients obtaining 5600 Operations/second.
Raising PIOPS to 2000 we reached an Overall Throughput of 12960 Ops/sec using 8 clients before Update Latency started increasing.
These results can be a good base for taking decisions about choosing the RDS instance type and the use of Provisioned IOPS coming from a balanced workload composed at 50% of writes.
In this benchmark we learned:
- The use of Reserved IOPS in RDS is mandatory if we need constant performances, maintaining both latency and throughput fixed;
- RDS Instances without Reserved IOPS performances are lower than a 1000 IOPS one even if sometimes capables of an higher throughput during insert operations;
- EBS Optimized EC2 instances are highly recommended to take full advantage from Reserved IOPS RDS Instances;
- Raising Reserved IOPS over an instance capacity results only on a lower latency not improving the overall throughput;
- Databases requiring frequent Scan operations will take most advantages from scaling RDS Instance to a more CPU powerful one than raising IOPS.
A note on Throughput and Latency measurement:
As stated at the beginning these are crude numbers reached without database engine optimization, caching or other non standard element. Methodology, environment and tools are listed to leave the opportunity to run other tests and eventually compare the results. As for the other posts you are welcome to contribute with suggestions or questions.
Working with Provisioned IOPS Storage http://docs.aws.amazon.com/AmazonRDS/…USER_PIOPS.html
Yahoo! Cloud Serving Benchmark https://github.com/brianfrankcooper/YCSB/wiki