Database Microbenchmarks

Symas Corp., September 2012

This page follows on from Google's LevelDB benchmarks published in July 2011 at LevelDB. (A snapshot of that document is available here for reference. In addition to the benchmarks tested there, we add the venerable BerkeleyDB as well as the OpenLDAP MDB database. For this test, we compare LevelDB version 1.5 (git rev dd0d562b4d4fbd07db6a44f9e221f8d368fee8e4), SQLite3 (version 3.7.7.1) and Kyoto Cabinet's (version 1.2.76) TreeDB (a B+Tree based key-value store), Berkeley DB 5.3.21, and OpenLDAP MDB (git rev 2e677bcb995b63d36461eea254f2134ebfe29da2). We would like to acknowledge the LevelDB project for the original benchmark code.

This page shows updated results for MDB write performance when using its writable mmap option. The previous benchmark report is available here. Using the memory map in read/write mode allows much faster writes and even lower memory overhead, but also opens the possibility of undetected database corruption if bugs in the application cause overwrites of arbitrary ranges of the map.

The writable mmap option was implemented after discussions at the LinuxCon in August. The results here obsolete the results in the August LinuxCon presentation.

Benchmarks were all performed on a Dell Precision M4400 laptop with a quad-core Intel(R) Core(TM)2 ExtremeCPU Q9300 @ 2.53GHz, with 6144 KB of total L3 cache and 8 GB of DDR2 RAM at 800 MHz. (Note that LevelDB uses at most two CPUs since the benchmarks are single threaded: one to run the benchmark, and one for background compactions.) The benchmarks were run on three different filesystems: tmpfs, reiserfs on an SSD, and ext2 on an HDD. (See our previous report for performance tests on btrfs, ext2, ext3, ext4, jfs, ntfs, reiserfs, xfs, and zfs.) The SSD is a Crucial M4 512GB, purchased new in August 2012. The HDD is a Seagate ST9500420AS Momentus 7200.4 500GB notebook hard drive. The system had Ubuntu 12.04 installed, with kernel 3.5.0-030500. Tests were all run in single-user mode to prevent variations due to other system activity. CPU performance scaling was disabled (scaling_governor = performance), to ensure a consistent CPU clock speed for all tests. The numbers reported below are the median of three measurements. The databases are completely deleted between each of the three measurements.

Benchmark Source Code

We wrote benchmark tools for SQLite, BerkeleyDB, MDB, and Kyoto TreeDB based on LevelDB's db_bench. The LevelDB, SQLite3, and TreeDB benchmark programs were originally provided in the LevelDB source distribution but we've made additional fixes to the versions used here. The code for each of the benchmarks resides here:

• LevelDB: db_bench.cc.
• SQLite: db_bench_sqlite3.cc.
• Kyoto TreeDB: db_bench_tree_db.cc.
• OpenLDAP MDB: db_bench_mdb.cc.
• BerkeleyDB: db_bench_bdb.cc.
• util/random.h: random.h.

Google's original benchmark code had a number of flaws, as we noted in this post to the LevelDB Discussion Group. We've fixed these flaws for these tests:

• The Random write tests now use a shuffled list of all the keys, so no duplicate keys are used. Thus the resulting databases should have the same content as from the Sequential tests.
• The synchronous tests on tmpfs are identical in length to the asynch tests, to ensure that the results are comparable. (We still shortened them on SSD and HDD due to their lengthy runtimes.)

In addition, we found that the Kyoto TreeDB results were being unfairly penalized relative to the LevelDB results, because all of its data items were undergoing an extra alloc/copy before being sent to the database. The data items for LevelDB are now similarly copied before use, to compensate for this issue. It's not clear how relevant this flaw is in real world applications but it definitely skewed write speeds in LevelDB's favor in these benchmarks.

Custom Build Specifications

• LevelDB: Assertions were disabled.
• TreeDB: We enabled the TSMALL and TLINEAR options when opening the database in order to reduce the footprint of each record.
• SQLite: We tuned SQLite's performance, by setting its locking mode to exclusive. We also enabled SQLite's write-ahead logging.
• BerkeleyDB: We configure with --with-mutex=POSIX/pthreads to avoid using the default hybrid mutex implementation.
• MDB: Assertions were disabled.

Most database vendors claim their product is fast and lightweight. Looking at the total size of each application gives some insight into the footprint of each database implementation.

size db_bench*
   text	   data	    bss	    dec	    hex	filename
 272247	   1456	    328	 274031	  42e6f	db_bench
1675911	   2288	    304	1678503	 199ca7	db_bench_bdb
  90423	   1508	    304	  92235	  1684b	db_bench_mdb
 653480	   7768	   1688	 662936	  a1d98	db_bench_sqlite3
 296572	   4808	   1096	 302476	  49d8c	db_bench_tree_db

This section gives the baseline performance of all the databases. Following sections show how performance changes as various parameters are varied. For the baseline:

• All operations are running on tmpfs. This shows the pure CPU time each database requires, independent of I/O speed.
• Each database is allowed 4 MB of cache memory. (MDB has no cache, so this is irrelevant.)
• Databases are opened in asynchronous write mode. (LevelDB's sync option, TreeDB's OAUTOSYNC option, SQLite3's synchronous options are all turned off, MDB uses the MDB_NOSYNC option, and BerkeleyDB uses the DB_TXN_WRITE_NOSYNC option). I.e., every write is pushed to the operating system, but the benchmark does not wait for the write to reach the disk.
• Keys are 16 bytes each.
• Values are 100 bytes each.
• Sequential reads/writes traverse the key space in increasing order.
• Random reads/writes traverse the key space in random order. The entire key space is shuffled so that every key is visited once; there are no duplicates or collisions in the random sequence.

A. Sequential Reads

B. Random Reads

C. Sequential Writes

D. Random Writes

MDB has the fastest read operations by a huge margin, due to its single-level-store architecture. MDB's write performance is good but not the fastest, due to the overhead of its copy-on-write design. This copy overhead is most significant on small records like those used in this test.

E. Batch Writes

A batch write is a set of writes that are applied atomically to the underlying database. A single batch of N writes may be significantly faster than N individual writes. The following benchmark writes one thousand batches where each batch contains one thousand 100-byte values. TreeDB does not support batch writes so its baseline numbers are repeated here for reference.

Sequential Writes

Random Writes

Batching allows some of MDB's copy-on-write overhead to be amortized across multiple records. MDB's Append Mode for sequential writes is most effective in batched operation.

F. Synchronous Writes

In the following benchmark, we enable the synchronous writing modes of all of the databases. Otherwise the test is identical to the Baseline.

• For LevelDB, we set WriteOptions.sync = true.
• In TreeDB, we enabled TreeDB's OAUTOSYNC option.
• For SQLite3, we set "PRAGMA synchronous = FULL".
• For MDB, we set no options since full sync is its default mode.
• For BerkeleyDB, we set no options since full sync is its default mode.

Sequential Writes

Random Writes

While most of the databases are only slowed down a little, TreeDB performs extremely poorly in synchronous mode.

G. Disk Usage

The amount of disk space each database consumed is also a factor when considering the footprint of each database. The results for each write test are presented here, in KBytes used per test. The results are obtained using du on the test directory at the completion of each test, after all data has been flushed to disk. It includes space used by log files as well as actual data files, if any.

LevelDB is the most efficient for disk usage, even without compression. We chose to omit the built in compression from these tests because it is clearly not a core feature of a database; any compression library could easily be used to handle compression for any database using simple wrappers around their put and get APIs.

MDB's sequential write packing gives the most deterministic file sizes; the DB size is the same regardless of batched or unbatched or synchronous writing. All of the other databases and modes yield different sizes due to varying overheads in each mode.

BerkeleyDB's transaction log files form the bulk of its disk usage. We configured it with DB_LOG_AUTO_REMOVE to run these tests, otherwise the consumption would be even higher. In regular deployments it may be unsafe to use this setting, since removal of the log files may prevent database recovery from working after a system crash. BerkeleyDB's transaction log management has been a continual inconvenience in real world deployments.

We increased the overall cache size for each database to 128 MB. For SQLite3, we kept the page size at 1024 bytes, but increased the number of pages to 131,072 (up from 4096). For TreeDB, we also kept the page size at 1024 bytes, but increased the cache size to 128 MB (up from 4 MB). For MDB there is no cache, so the numbers are simply a copy of the baseline. Both MDB and BerkeleyDB use the default system page size (4096 bytes).

A. Sequential Reads

B. Random Reads

C. Sequential Writes

D. Random Writes

E. Batch Writes

Sequential Writes

Random Writes

F. Synchronous Writes

Sequential Writes

Random Writes

Performance gains from the increased cache are minimal at best; some of the tests are slower with the increased caches. Overall, application-level caching exacts a great cost in terms of software and administrative complexity, and yields little (if any) benefit in return.

For this benchmark, we use 100,000 byte values. To keep the benchmark running time reasonable, we stop after writing 10,000 values. Otherwise, all of the same tests as for the Baseline are run.

A. Sequential Reads

B. Random Reads

MDB's single-level-store architecture clearly outclasses all of the other designs; the others barely even register on the results. MDB's zero-memcpy reads mean its read rate is essentially independent of the size of the data items being fetched; it is only affected by the total number of keys in the database.

C. Sequential Writes

D. Random Writes

E. Batch Writes

Sequential Writes

Random Writes

Unlike the other databases, MDB handles large value writes with ease.

In Google's original test only 1,000 entries are used, but that is too few to characterize each database. With 10,000 entries, we see that LevelDB's performance drops off a cliff in the random write tests. This is one of the most common problems with microbenchmarks - very often the data sets are too small to yield any meaningful results.

F. Synchronous Writes

Sequential Writes

Random Writes

None of the other databases even begin to approach MDB's performance when using large data values.

G. Disk Usage

The same tests as in Section 2 are performed again, this time using the Crucial M4 SSD with reiserfs. This drive has been in light use for only a few weeks, so it should still be near its original performance levels.

A. Sequential Reads

B. Random Reads

Read performance is essentially the same as for tmpfs since all of the data is present in the filesystem cache.

C. Sequential Writes

D. Random Writes

Most of the databases perform at close to their tmpfs speeds, which is expected since these are asynchronous writes. However, BerkeleyDB shows a large reduction in throughput. Also, surprisingly, MDB beats LevelDB's write performance here. Even though the writes are fully cached, the underlying storage device still has an impact on write throughput.

E. Batch Writes

Sequential Writes

Random Writes

F. Synchronous Writes

Here the difference between SSD and tmpfs is made obvious. Note that due to the overall slowness of these operations, they were only performed 1000 times each.

Sequential Writes

Random Writes

The slowness of the SSD overshadows any difference between sequential and random write performance here.

MDB's synchronous writes are slower because by default it does two separate syncs per commit - one for the data, and one for the meta page update. Generally there is a large seek penalty for the sync of the meta page, although that should not be an issue with an SSD. Using MDB's NOMETASYNC option will skip the explicit sync of the meta page update. Omitting the explicit sync of the meta page means the meta page update for one transaction may not be sync'd along with that transaction but definitely will be sync'd by the commit of the next following transaction. In this case, if the system crashes, the last committed transaction may be lost. Some apps can tolerate this reduced level of reliability. Also, on filesystems which support ordered writes, no transactions will be lost.

We increased the overall cache size for each database to 128 MB, as in Section 3. The "baseline" in these tests refers to the values from Section 5.

A. Sequential Reads

B. Random Reads

C. Sequential Writes

D. Random Writes

E. Batch Writes

Sequential Writes

Random Writes

F. Synchronous Writes

Sequential Writes

Random Writes

The impact of the increased cache is inconsistent at best.

This is the same as the test in Section 4, using the SSD.

A. Sequential Reads

B. Random Reads

The read results are about the same as for tmpfs.

C. Sequential Writes

D. Random Writes

E. Batch Writes

Sequential Writes

Random Writes

MDB dominates the asynchronous write tests.

F. Synchronous Writes

Sequential Writes

Random Writes

The benefit of MDB's NOMETASYNC option is marginal here; most of the time required for these operations is simply in writing the 100,000 byte data values.

The same tests as in Section 2 are performed again, this time using the Seagate ST9500420AS HDD with EXT2 fs. The EXT2 filesystem was chosen based on our previous survey of multiple filesystems.

A. Sequential Reads

B. Random Reads

Read performance is essentially the same as the previous tests since all of the data is present in the filesystem cache.

C. Sequential Writes

D. Random Writes

E. Batch Writes

Sequential Writes

Random Writes

F. Synchronous Writes

As before, only 1000 operations are performed, due to the slowness of these tests. The HDD's cache was not disabled for these tests.

Sequential Writes

Random Writes

The slowness of the HDD overshadows any difference between sequential and random write performance here. LevelDB seems to benefit more from the HDD's caching than any of the other databases.

MDB's NOMETASYNC option made no discernible difference here so those results are omitted. Ultimately the seek overhead cannot be avoided using a single storage device. Ideally the meta pages should be isolated in a separate file that can be stored on a separate spindle to avoid the seek overhead.

We increased the overall cache size for each database to 128 MB, as in Section 3. The "baseline" in these tests refers to the values from Section 8.

A. Sequential Reads

B. Random Reads

C. Sequential Writes

D. Random Writes

E. Batch Writes

Sequential Writes

Random Writes

F. Synchronous Writes

Sequential Writes

Random Writes

This is the same as the test in Section 4, using the HDD.

A. Sequential Reads

B. Random Reads

Again, the read results are about the same as for tmpfs.

C. Sequential Writes

D. Random Writes

E. Batch Writes

Sequential Writes

Random Writes

F. Synchronous Writes

Sequential Writes

Random Writes

The slowness of the HDD makes most of the database implementations perform about the same. As before, Kyoto Cabinet is much slower than the rest.

The raw data for all of these tests is also available. tmpfs, SSD, and HDD. The results are also tabulated in an OpenOffice spreadsheet for further analysis here. The raw data includes additional tests (e.g. reverse sequential read) which were omitted from this already lengthy report for space reasons.