In-Memory Microbenchmark (Scaling/NUMA)

Symas Corp., September 2014

The cost of accessing a remote node can be from 60% greater to over 100% greater than accessing the local node, as shown by numactl -H:

node distances:
node   0   1   2   3   4   5   6   7 
  0:  10  16  16  22  16  22  16  22 
  1:  16  10  22  16  22  16  22  16 
  2:  16  22  10  16  16  22  16  22 
  3:  22  16  16  10  22  16  22  16 
  4:  16  22  16  22  10  16  16  22 
  5:  22  16  22  16  16  10  22  16 
  6:  16  22  16  22  16  22  10  16 
  7:  22  16  22  16  22  16  16  10 

To discover how much this was affecting the benchmark performance, the entire series of tests was rerun using numactl --interleave=all which causes memory pages to be allocated across all of the nodes in round-robin fashion. Interleaving in this way should cause the single-thread case to run slower, since all of the accesses could feasibly be handled inside a single node. But for the 16 thru 64 thread cases it should speed up, since threads will be distributed across all of the nodes and at least 1/8th of their accesses will be local.

The results of running with NUMA interleaving are summarized below:

Loading the DB

Local Alloc	Load Time			CPU %	DB Size	Process Size	Total Size	Context Switches
	Wall	User	Sys		KB	KB	KB	Vol	Invol
LevelDB	09:24.39	04:26.36	00:52.31	56	11702148	540488	12242636	204233	625
Basho	44:28.12	07:15.42	02:15.19	21	13128020	127916	13255936	85254	12304
BDB	52:32.77	11:51.71	02:59.45	28	38590332	21499600	60089932	1196561	3162
Hyper	09:12.10	05:15.21	01:11.65	70	11644136	725100	12369236	219303	2023
LMDB	01:05.73	00:45.22	00:20.43	99	12580708	12583076	12583076	9	192
RocksDB	10:53.73	02:39.76	00:46.74	31	11745460	624916	12370376	398909	325
RocksDBpfx	10:09.24	10:27.97	00:44.74	110	12207504	13456836	25664340	195089	1134
TokuDB	24:12.76	09:35.02	04:20.47	57	14991692	18798900	33790592	1091241	1951
WiredLSM	24:27.22	17:08.86	11:53.50	118	12607732	1156020	13763752	18163720	4903
WiredBtree	07:35.84	01:24.96	00:18.78	22	11909312	314008	12223320	7268	151
Interleaved	Load Time			CPU %	DB Size	Process Size	Total Size	Context Switches
	Wall	User	Sys		KB	KB	KB	Vol	Invol
LevelDB	06:34.32	04:33.70	00:57.06	83	11702148	539660	12241808	195036	466
Basho	43:43.51	07:25.25	02:52.59	23	18093052	125328	18218380	73666	13061
BDB	50:19.81	13:09.63	02:45.54	31	39144188	21498344	60642532	1293051	3199
Hyper	10:28.11	05:15.28	01:21.64	63	11644136	719448	12363584	210183	1647
LMDB	01:09.76	00:45.37	00:24.29	99	12580708	12583116	12583116	10	243
RocksDB	10:54.27	02:39.77	00:51.77	32	11745460	624480	12369940	416688	366
RocksDBpfx	11:15.31	12:06.02	00:54.43	115	12207504	12344096	24551600	219820	2093
TokuDB	24:00.29	10:21.51	03:37.79	58	14351024	17673108	32024132	817791	2192
WiredLSM	25:05.43	17:52.91	10:43.36	114	13132576	1068356	14200932	19109520	3868
WiredBtree	06:19.92	01:27.93	00:20.30	28	11909312	314032	12223344	6335	199

Since the data set is the same as before the DB Size and Process Size were expected to be the same as before. The DB Size was identical to the previous run for LevelDB, HyperLevelDB, LMDB, RocksDB, and WiredTiger-Btree. But due to the vagaries of background processing, the DB Sizes were different for the other engines. The Process Sizes also changed slightly due to the change in memory allocation policy.

Since LMDB's load phase is purely single-threaded, it was expected to slow down slightly when switching to Interleaved memory. The results show this to be the case, with the load taking about 6% longer with Interleaving. For the other engines the difference is harder to quantify. Engines with multi-threaded loaders tended to improve throughput, but the differences are slight and runtime was mostly dominated by I/O waits. The following graph shows the difference in times between the original and Interleaved run.

The Wall, User, and System CPU times are shown twice in the graph. For each pair of measurements, the first column is from the original test and the second column is from the Interleaved test.

Summary

Several of the engines improved their Write throughput compared to the original test. Surprisingly, LMDB attains the fastest write speed from 1 to 16 threads. TokuDB remains the slowest at all load levels.

LMDB is still the leader here, with read performance scaling linearly with the number of cores. RocksDB in its highly tuned configuration is almost as fast. At 64 threads both appear to have hit the limits of the hardware; there aren't enough CPU cycles left to do any more work.

With the notable exception of BashoLevelDB, space usage appears to be about the same as before. The DB-specific results follow.

LevelDB

In this graph, the Write Rates are on the left Y-axis and the Read Rates use the right Y-axis. In each pair of columns, the first column is the result from the original test and the second is the result from the Interleaved test. The erraticness of the write speeds makes it difficult to show the effect of Interleaving. The read speed shows a more obvious improvement for all tests except the 2-reader test. It's still not enough of an efficiency improvement to overcome the locking overhead and negative scaling above 32 threads though.

The User CPU time increases, which is the expected result since more useful work is getting done. However, the System CPU time increases even more at 64 threads, which shows that NUMA memory management was not the primary bottleneck in this job.

Basho LevelDB

Interleaving has very little effect on this engine. There is a small improvement in write speed and a small decline in read speed. The difference is negligible.

There is a very small difference in CPU usage as well. Again, negligible. The jump in User CPU at 32 threads is unexplained; since neither write nor read rate improved it must be attributed to miscellaneous background threads.

BerkeleyDB

There are some slight improvements from 2-32 threads. The lock scaling issues still override everything else.

The CPU usage is about the same as before. Again, NUMA memory management is not the main concurrency problem in BDB.

HyperLevelDB

Writes are a mixed bag here, but reads show slight improvement for 2-64 threads. Memory Interleaving definitely has a beneficial effect overall.
At 2-16 threads the User CPU time increases, as does overall throughput. From 32 threads read throughput has still increased but write throughput has correspondingly decreased, and User CPU time decreases overall. From 32-64 threads the System CPU usage still dominates.

LMDB

The CPU use here is even closer to the ideal 200 / 300 / 500 / 900 / 1700 / 3300 / 6400.

The write and read rates both improve at all load levels. With all other overheads removed the NUMA memory layout was the only remaining bottleneck.

The User CPU time increases since more useful work is being done. The System CPU time disappears once the NUMA overhead is removed. The increase in User CPU time is just about equal to the decrease in System CPU time, which indicates that there is no other overhead in this engine. All of the CPU time is dedicated to getting useful work done.

RocksDB

Again a mixed bag for writes but mostly showing a slight decrease in throughput. Reads increase from 2-16 threads, along with a sizable increase at 64 threads.
User CPU increased and system CPU decreased from 2-16 threads, corresponding to increased throughput at those loads. System CPU increased drastically from 32-64 threads, and overall throughput is lower at those loads. NUMA Interleave definitely had an effect, just not always a positive one.

RocksDBpfx

The write rates are too erratic to judge, but the read rates generally improve.

The User CPU time increases with the increased throughput, and System CPU time disappears. NUMA Interleave clearly was a significant factor here.

TokuDB

Write speeds are largely unaffected. Read speeds improve slightly from 2-64 threads.

System CPU time increases, indicating that removing the NUMA bottleneck exposed another bottleneck.

WiredTiger LSM

Write rates improved from 1-4 threads but declined after that. Read reates improved minimally from 2-8 threads, and also a small bump at 64 threads. The overall effect is small.

System CPU use was generally small to begin with. Given the high NUMA overheads seen in other engines that suggests that WiredTiger already does some NUMA-aware memory management of its own. Still, the increases in User CPU time don't translate into higher throughput; it appears to be absorbed in their user-level lock manager instead.

WiredTiger Btree

Reads improve at all load levels. Writes improve at all loads except the 16 thread case.

The System CPU time mostly disappears. User CPU time increases by a larger amount than the removed System CPU time; overall CPU utilization is higher than in the original test but there's still significant I/O wait involved.

Conclusion

Files

The software versions we used:

Software revisions used:

violino:/home/software/leveldb> g++ --version
g++ (Ubuntu 4.8.2-19ubuntu1) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

violino:/home/software/leveldb> git log -1 --pretty=format:"%H %ci" master
e353fbc7ea81f12a5694991b708f8f45343594b1 2014-05-01 13:44:03 -0700

violino:/home/software/basho_leveldb> git log -1 --pretty=format:"%H %ci" develop
d1a95db0418d4e17223504849b9823bba160dfaa 2014-08-21 15:41:50 -0400

violino:/home/software/db-5.3.21> ls -l README 
-rw-r--r-- 1 hyc hyc 234 May 11  2012 README

violino:/home/software/HyperLevelDB> git log -1 --pretty=format:"%H %ci" master
02ad33ccecc762fc611cc47b26a51bf8e023b92e 2014-08-20 16:44:03 -0400

violino:~/OD/mdb> git log -1 --pretty=format:"%H %ci"
a054a194e8a0aadfac138fa441c8f67f5d7caa35 2014-08-24 21:18:03 +0100

violino:/home/software/rocksdb> git log -1 --pretty=format:"%H %ci"
7e9f28cb232248b58f22545733169137a907a97f 2014-08-29 21:21:49 -0700

violino:/home/software/ft-index> git log -1 --pretty=format:"%H %ci" master
f17aaee73d14948962cc5dea7713d95800399e65 2014-08-30 06:35:59 -0400

violino:/home/software/wiredtiger> git log -1 --pretty=format:"%H %ci"
1831ce607baf61939ddede382ee27e193fa1bbef 2014-08-14 12:31:38 +1000

This used revision

violino:~/OD/mdb> git log -1 --pretty=format:"%H %ci"
3646ba966c75137b01e38fc5baea6d5864189c8e 2014-09-09 19:44:23 +0100