In-Memory Microbenchmark (Scaling)

Symas Corp., September 2014

Test Overview

Loading the DB

Engine	Load Time			CPU	DB Size	Process Size	Total Size	Context Switches
	Wall	User	Sys	%	KB	KB	KB	Vol	Invol
LevelDB	09:24.39	04:26.36	00:52.31	56	11702148	540488	12242636	204233	625
Basho	44:28.12	07:15.42	02:15.19	21	13128020	127916	13255936	85254	12304
BDB	52:32.77	11:51.71	02:59.45	28	38590332	21499600	60089932	1196561	3162
Hyper	09:12.10	05:15.21	01:11.65	70	11644136	725100	12369236	219303	2023
LMDB	01:05.73	00:45.22	00:20.43	99	12580708	12583076	12583076	9	192
RocksDB	10:53.73	02:39.76	00:46.74	31	11745460	624916	12370376	398909	325
RocksDBpfx	10:09.24	10:27.97	00:44.74	110	12207504	13456836	25664340	195089	1134
TokuDB	24:12.76	09:35.02	04:20.47	57	14991692	18798900	33790592	1091241	1951
WiredLSM	24:27.22	17:08.86	11:53.50	118	12607732	1156020	13763752	18163720	4903
WiredBtree	07:35.84	01:24.96	00:18.78	22	11909312	314008	12223320	7268	151

We're running RocksDB in both a basic configuration and again in the tuned configuration used in the original RocksDB benchmark report. The tuned run is denoted as "RocksDBpfx" in all of these results.

The "Wall" time is the total wall-clock time taken to run the loading process. Obviously shorter times are faster/better. The actual CPU time used is shown for both User mode and System mode. User mode represents time spent in actual application code; time spent in System mode shows operating system overhead where the OS must do something on behalf of the application, but not actual application work. In a pure RAM workload where no I/O occurs, ideally the computer should be spending 100% of its time in User mode, processing the actual work of the application.

The "CPU" column is the ratio of adding the User and System time together, then dividing by the Wall time, expressed as a percentage. This shows how much work of the DB load occurred in background threads. Ideally this value should be 100, all foreground and no background work. If the value is greater than 100 then a significant portion of work was done in the background. If the value is less than 100 then a significant portion of time was spent waiting for I/O. When a DB engine relies heavily on background processing to achieve its throughput, it will bog down more noticeably when the system gets busy. I.e., if the system is already busy doing work on behalf of users, there will not be any idle system resources available for background processing. LMDB is effectively ideal.

The "Context Switches" columns show the number of Voluntary and Involuntary context switches that occurred during the load. Voluntary context switches are those which occur when a program calls a function that can block - system calls, mutexes and other synchronization primitives, etc. Involuntary context switches occur e.g. when a CPU must handle an interrupt, or when the running thread's time slice has been fully consumed. LMDB performs no blocking calls and uses no background threads, so there are practically no context switches during its run. All of the other DBs have multiple orders of magnitude more context switches, yielding much lower overall efficiency. None of the other DBs can perform the initial load anywhere near as quickly as LMDB.

The DB Size column shows the size of the DB files on disk after the load completes. The Process Size column shows the maximum size of the benchmark process during the load. The Total Size column shows the total amount of RAM consumed during the load; it is usually the sum of the DB and Process sizes since the DB occupies space in the filesystem cache. For LMDB it is simply the Process size, since the memory used by the filesystem cache is included in the LMDB process.

Throughput

RocksDB in its minimally tuned configuration appears to sustain write load the best across various threading loads. HyperLevelDB performs extremely well when there's just a single reader, but as soon as another read thread is added its performance drops sharply.

LMDB's write performance is pretty good up to 16 readers. The decline at 32 and 64 was a bit surprising and bears further investigation.

LMDB is clearly the leader here, with read performance scaling linearly with the number of cores. RocksDB in its highly tuned configuration is almost as fast. Too bad that tuning RocksDB is so complex that even its own developers don't understand it. LMDB outperforms all of the rest, with no special tuning.

Software optimization is always about optimizing both time and space. Often we have a good idea of what speeds we require for a given task, but provisioning the size can be much trickier. For most of these DB engines, the amount of memory consumed varies wildly with the changing workload. For BDB and LMDB the size is essentially constant. Also, as noted in our original design paper, we targeted LMDB at consuming about one-third as much memory as BDB. In practice we usually come in at only one-fourth as much, as shown here.

For expedience, the test duration is set to 10 minutes for each run. The DB-specific results follow.

LevelDB

In this graph, the Write Rate is on the left Y-axis and the Read Rate uses the right Y-axis. As is typical of LevelDB, the write speed is highly erratic. The read speed doesn't scale linearly with threads, and it's clear that above 32 threads, the locking overhead becomes so great that additional threads simply slow things down (negative scaling).

Other items to note - even though the test duration is only 10 minutes, the actual running time is longer. This shows that the engine spends a significant amount of time cleaning up at the end of a test. Long cleanup times are a significant vulnerability; in an emergency shutdown a DB with a long cleanup time is likely to become corrupted if it doesn't have sufficient time to finish processing.

In the ideal case, since each run has a given number of reader threads plus one writer thread, the CPU % column should read 200 / 300 / 500 / 900 / 1700 / 3300 / 6400. (In the last case, since we only have 64 cores, the writer has to fight for CPU access with the readers but in every other case all tasks should be able to fully utilize a CPU core to themselves.) The CPU utilization here is far below ideal. This shows both heavy locking overhead and significant I/O wait time.

Also in the ideal case, User CPU time should always be much greater than System time. Here we see the System time increase drastically at 16 threads, and becoming several times greater than User time at 64 threads. This also indicates heavy locking overhead.

Basho LevelDB

Basho's fork of LevelDB shows extremely long cleanup times for these tests. That along with the extremely low CPU utilization indicates severe I/O waits in these tests. This is quite unusual considering that the writes are asynchronous and the 11GB database easily fits in the server's 256GB RAM.

Aside from the strange glitch in read performance at 2 and 4 threads, it shows a general decline in read performance with increasing number of readers. The write speed is relatively constant. Overall this DB is a poor choice for multi-threaded work.

Comparing the CPU usage to the throughput shows that additional threads just cause the DB to spin, wasting cycles for each thread.

BerkeleyDB

In OpenLDAP we have occasionally had reports of long shutdown times for slapd using BDB. We see here that in some cases there is a lengthy cleanup time, although nowhere near as bad as for BashoLevelDB.

BDB runs into lock scaling issues at 8 threads and things go downhill quickly from there.

The behavior is exactly the opposite of what is desired - the System CPU time increases with number of threads, instead of the User CPU time. Instead of getting useful application work done, it is just spinning in mutex overhead.

HyperLevelDB

Aside from the sharp drop in write rate between the 1 and 2 reader case, not much stands out in these results.
HyperLevelDB runs into locking overhead problems at 32 threads, and additional threads beyond this point just slow things down.
Somewhere between 16 and 32 threads the System CPU overhead gets very high as well. Overall, increasing the number of readers causes a commensurate decrease in writer throughput.

LMDB

The CPU use here is fairly close to the ideal 200 / 300 / 500 / 900 / 1700 / 3300 / 6400.

The write rate from 4 to 16 threads is basically unchanged while the read rate increases. The drop in write rate at 64 threads is expected since the writer must compete with readers for CPU time at that point. It was not expected at 32 threads since there are still 31 idle cores on the system. More profiling will be needed to see what happened there.

The CPU usage is as expected; User CPU time increases with the number of reader threads and is proportional to the read throughput.

RocksDB

As with Basho there's a strange drop in read throughput at 2 and 4 threads, though it's not quite as drastic here. CPU utilization overall is low, indicating a lot of I/O overhead. The runtime also indicates a significant amount of cleanup time at the end of each test.

While the write rate is fairly steady, reads start to drop off at 16 threads and adding more threads just slows things down from there.

System CPU time gets out of hand past 16 threads.

RocksDBpfx

$TIME ./db_bench.rocks --num_levels=6 --key_size=16 --prefix_size=16 --keys_per_prefix=0 --block_size=4096 --cache_size=$CACHE --cache_numshardbits=6 --compression_type=none --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --sync=0 --disable_data_sync=1 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_grandparent_overlap_factor=10 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=$STATS --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --perf_level=0 --duration=$DUR --benchmarks=readwhilewriting --use_existing_db=1 --num=$NUM --threads=$THREADS --writes_per_second=$WRATE --duration=$DUR

The various performance options definitely help, although they haven't eliminated the cleanup time at the end of the test. In some cases it's actually worse than for the default settings.

The read rate scales linearly with the number of threads, which is very good. The write rate is relatively unaffected by the number of readers, which is also very good.

The User CPU time is proportional to the read rate, and the System CPU time remains small. Again, these are very good results.

TokuDB

We're told that something is misconfigured in our TokuDB driver, but unfortunately there is no documentation on what proper settings might be. As it is, the write rates here are ridiculously slow and the code is both I/O bound and lock bound.

Increasing threads beyond 8 just slows everything down.

The System CPU time gets out of hand above 8 threads. This indicates heavy locking overhead.

WiredTiger LSM

Reads scale fairly well up to 16 threads. The write rates are quite slow; it's unusual to see the write rate ramp up before ramping down again as it does here.

Comparing throughput to CPU use implies that this library uses its own locking mechanism, not just the system's mutexes.
The increase in System CPU at 64 threads indicates heavy mutex overhead.

WiredTiger Btree

This DB shows significant cleanup time at the end of each test. Also it appears to have heavy lock overhead from 4 threads on up, based on the low CPU usage.

The read rate scales well up to about 8 threads, then tapers off. Additional threads past 16 just slow things down.

The System CPU time is noticeable at all load levels, but gets out of hand past 16 threads.

NUMA Considerations

Conclusion

The definition of "what fits in memory" also varies wildly depending on whose DB you're talking about, what kind of workload you throw at it, and how you tune the DB engine. One of the reasons for using an embedded DB is not only that it has a small footprint in terms of code and resource usage, but also in terms of administration and maintenance. Some of these DBs have quite complex configurations that must be carefully tuned in order to obtain usable performance; they seem to have lost the point of simplicity in their quest for performance. With LMDB no tuning is required. If you're looking to jump on the In-Memory bandwagon, there's only one viable choice.

Files

The software versions we used:

Software revisions used:

violino:/home/software/leveldb> g++ --version
g++ (Ubuntu 4.8.2-19ubuntu1) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

violino:/home/software/leveldb> git log -1 --pretty=format:"%H %ci" master
e353fbc7ea81f12a5694991b708f8f45343594b1 2014-05-01 13:44:03 -0700

violino:/home/software/basho_leveldb> git log -1 --pretty=format:"%H %ci" develop
d1a95db0418d4e17223504849b9823bba160dfaa 2014-08-21 15:41:50 -0400

violino:/home/software/db-5.3.21> ls -l README 
-rw-r--r-- 1 hyc hyc 234 May 11  2012 README

violino:/home/software/HyperLevelDB> git log -1 --pretty=format:"%H %ci" master
02ad33ccecc762fc611cc47b26a51bf8e023b92e 2014-08-20 16:44:03 -0400

violino:~/OD/mdb> git log -1 --pretty=format:"%H %ci"
a054a194e8a0aadfac138fa441c8f67f5d7caa35 2014-08-24 21:18:03 +0100

violino:/home/software/rocksdb> git log -1 --pretty=format:"%H %ci"
7e9f28cb232248b58f22545733169137a907a97f 2014-08-29 21:21:49 -0700

violino:/home/software/ft-index> git log -1 --pretty=format:"%H %ci" master
f17aaee73d14948962cc5dea7713d95800399e65 2014-08-30 06:35:59 -0400

violino:/home/software/wiredtiger> git log -1 --pretty=format:"%H %ci"
1831ce607baf61939ddede382ee27e193fa1bbef 2014-08-14 12:31:38 +1000