Benchmarking is a topic I’ve seen discussed for my entire career working with databases. We used to watch the TPC-C results published, when I was working with primarily OLTP systems, and was amazed to see software I worked with every day making the most recent amazing result. Granted, the hardware to do that far exceeded the systems I worked with.
Benchmarking History for Databases
I am not a database historian. One of my favorite historical database stories is how the DeWitt clause made it into modern software licenses. My understanding is that in the 80’s a professor(DeWitt) published a comparison of database systems, and Larry Elison of Oracle was not happy that Oracle did not perform well. This led to Oracle adding a clause to their licensing agreement that benchmarks could not be published without approval from Oracle. When I learned about this, the first thing I checked was whether the software I worked with at the time – Db2 – included this clause. I was sad to learn that other database vendors also included this clause in their licensing.
Some modern data platforms have eliminated the DeWitt clause to encourage benchmarking.
What Does a Benchmark Measure, Really?
With more than 20 years of experience working with databases, I see why benchmarks only offer so much value. This comes down to a couple of issues. The first issue is understanding what benchmarks really measure. Do they measure the maximum possible speed of the software being used? Rarely.
Any benchmark is only as good as the person who is running the benchmark = their skill both with the software being used AND with their prowess with testing a data system. Often those running benchmarks are trying to answer a question – which software is faster, A or B? The problem is that in modern software there are a million different factors that contribute to the performance. With most database systems, it requires an expert in that software to properly set up a system. In the early days of my career this would include selecting appropriate page sizes, mapping extent sizes to stripe sizes, the knobs for tuning a database system, and what indexes were added. It is pretty easy to get a single setting wrong and get a disastrously slow result – without even knowing the mistake was made.
I have seen some abysmally flawed performance tests in general in my career. Even when the system is configured by an expert, with excellent data structures and well-aligned settings, there is another area of expertise needed. Expertise in load testing itself is critical. I remember load testing for an e-commerce app that caused horrific deadlocking on DB2. Was the problem Db2? No, the problem was that the load test used fewer than 100 user accounts that were used to simulate thousands of unique users. As a result many different threads trying to check out the same cart at the same time! I have seen other problems caused by things like simulating thousands of users from only one thread on one machine – the bottleneck sure wasn’t on the database server, but on the server running the tests!
Once we get beyond the honest mistakes caused by a lack of expertise in each of these areas, we sometimes find more nefarious intentional gaming of the system. Setting a metric invites manipulation. People naturally game the system. In benchmarking, specific knowledge of a workload creates a trap. Engineers optimize for those exact queries. These “improvements” often fail to help general performance. Worse, they frequently damage critical database functions. This could include transactional logging for recovery or even true referential integrity checks. IBM Db2 has many hidden parameters. Could any of them be used to speed up benchmarks?
The hardware configurations of the systems I saw years ago in the TPC-C benchmarks were staggeringly huge for the time. They would have enough memory to store the entire database in-memory – something that few real-world implementations could afford at the time. The saying I have heard over and over again is “Cheap, Fast, Good – pick any two”. Choosing fast involves compromising at least one of the other two.
Benchmarking aims to measure platform speed. Instead, it captures team effort. The result reflects human ingenuity over a short sprint. This creates a false comparison. Personal expertise distorts the data. Teams naturally favor familiar systems, even when they have the best intentions.
Other Problems with Benchmarks
Another problem with benchmarks is that they don’t measure the real-world production workload. A different data model makes a huge difference, no matter what the data platform or RDBMS. Even with the exact same data model, the distribution of data values makes a huge difference in the execution of some queries. There are a large number of things make it difficult (if not impossible) to extrapolate performance from a small-scale test. The only benchmark that should ultimately matter is the workload of queries you consider important against your specific data model and data. Anything else might give a general idea if done exceptionally well, but is more likely to lead you astray.
This actually goes beyond benchmarks to POCs. Sometimes a targeted POC can prove a specific point you’re worried about with an unfamiliar system. But if a POC is used to compare two systems, you have the same potential pitfalls as a benchmark.
Benchmarks at Snowflake
I am not involved with benchmarking at Snowflake. My focus is on working with customers’ performance on real workloads. I have no special insight in this area, but greatly respect the approach Snowflake takes to benchmarking.
The Snowflake founders have made their stance on benchmarks clear in Industry Benchmarks and Competing with Integrity from November 2021. When I went and re-read that article, I noted the times they quoted, and today’s times for the same benchmark are far faster.
More recently, a Principal Product Manager, who is one of the top minds on performance inside Snowflake, talked about some of the benchmarking we do internally in Benchmarking Concurrent Workloads for Operational Analytics and BI Workloads. That article focused on a specific type of workload. One of my favorite things we’ve published is the Snowflake Performance Index. This is not a benchmark, but is instead based on the performance improvement in stable real customer workloads over time. It is amazing to see this constantly improving over time.
Perhaps the most fun thing is that if you do care about a benchmark, you can easily run it on Snowflake. TPC-DS data is freely available in the SNOWFLAKE_SAMPLE_DATA database. That link includes a file of queries that can run as-is against that existing data. The schemas in SNOWFLAKE_SAMPLE_DATA include both a 10TB scale and a 100TB scale. The data is already clustered in those schemas in reasonable ways. To my knowledge, there is no specific gaming of the benchmarks going on.
So if you really want to test how well YOU, personally, can run a stock benchmark in Snowflake, it doesn’t require much to do so. It can probably be done with a free trial account. I think using a 4xl for the 100TB scale TPCDS_SF100TCL schema will fit within the free trial limits. I’m not sure it’ll tell you much, but it is there as an option.
Benchmarks – Closing Thoughts
There is limited value in a benchmark. Instead of using them to compare systems, they’re often actually comparing the skill level of the implementers on the given systems. Stick to real-world data, data models, and query workloads whenever possible. Play with benchmarks for sport. Avoid trusting the results as gospel.
