From: Stephen Morris Date: Tue, 4 Sep 2012 19:19:55 +0000 (+0100) Subject: [2143] Editorial changes to layout and language X-Git-Tag: trac2351_base~47^2~13^2~2 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=7a60b5784ca9ccda3e18c740b8ff26edabbf4732;p=thirdparty%2Fkea.git [2143] Editorial changes to layout and language --- diff --git a/tests/tools/dhcp-ubench/dhcp-perf-guide.html b/tests/tools/dhcp-ubench/dhcp-perf-guide.html index 0fc4a617bf..08475d323b 100644 --- a/tests/tools/dhcp-ubench/dhcp-perf-guide.html +++ b/tests/tools/dhcp-ubench/dhcp-perf-guide.html @@ -1,69 +1,71 @@ -DHCP Performance Guide

DHCP Performance Guide

Tomasz Mrugalski

This is a companion document for BIND 10 version - 20120817.

Copyright © 2012 Internet Systems Consortium, Inc. ("ISC")

Abstract

BIND 10 is a framework that features Domain Name System - (DNS) suite and Dynamic Host Configuration Protocol (DHCP) - servers with development managed by Internet Systems Consortium (ISC). +DHCP Performance Guide

DHCP Performance Guide

Tomasz Mrugalski

This is a companion document for BIND 10 version + 20120405.

Copyright © 2012 Internet Systems Consortium, Inc. ("ISC")

Abstract

BIND 10 is a framework that features Domain Name System + (DNS) and Dynamic Host Configuration Protocol (DHCP) + software with development managed by Internet Systems Consortium (ISC). This document describes various aspects of DHCP performance, measurements and tuning. It covers BIND 10 DHCP (codename Kea), existing ISC DHCP4 software, perfdhcp (a DHCP performance - measurement tool) and other related topics.

Table of Contents

1. Acknowledgements

1. Introduction

2. ISC DHCP 4.x

3.1. Backend performance evaluation

3.2. MySQL backend

3.2.1. MySQL tweaks

3.3. SQLite-ubench

3.3.1. SQLite tweaks

3.4. memfile-ubench

3.4.1. memfile tweaks

3.5. Basic performance measurements

3.6. Optimized performance measurements

3.7. Conclusions

3.8. Possible further optimizations

4. perfdhcp

List of Tables

3.1. Synchronous results (basic)
3.2. Asynchronous results (basic)
3.3. Estimated basic performance
3.4. Synchronous results (optimized)
3.5. Asynchronous results (optimized)
3.6. Estimated optimized performance

Preface

Table of Contents

1. Acknowledgements

1. Acknowledgements

ISC would like to acknowledge generous support for + measurement tool) and other related topics.

Table of Contents

1. Acknowledgements

1. Introduction

2. ISC DHCP 4.x

3.1. Backend performance evaluation

3.2. MySQL backend

3.2.1. MySQL tweaks

3.3. SQLite-ubench

3.3.1. SQLite tweaks

3.4. memfile-ubench

3.4.1. memfile tweaks

3.5. Basic performance measurements

3.6. Optimized performance measurements

3.7. Conclusions

3.8. Possible further optimizations

4. perfdhcp

List of Tables

3.1. Synchronous results (basic)
3.2. Asynchronous results (basic)
3.3. Estimated basic performance
3.4. Synchronous results (optimized)
3.5. Asynchronous results (optimized)
3.6. Estimated optimized performance

Preface

Table of Contents

1. Acknowledgements

1. Acknowledgements

ISC would like to acknowledge generous support for BIND 10 development of DHCPv4 and DHCPv6 components provided by Comcast.

Chapter 1. Introduction

- This document is in its early stages of development. It is - expected to grow significantly in a near future. It will + This document is in the early stages of development. It is + expected to grow significantly in the near future. It will cover topics like database backend perfomance measurements, - pros an cons of various optimization techniques and - tools. + tools, and the pros an cons of various optimization techniques.

Chapter 2. ISC DHCP 4.x

TODO: Write something about ISC DHCP4 here. -

Chapter 3. Kea

Table of Contents

3.1. Backend performance evaluation

3.2. MySQL backend

3.2.1. MySQL tweaks

3.3. SQLite-ubench

3.3.1. SQLite tweaks

3.4. memfile-ubench

3.4.1. memfile tweaks

3.5. Basic performance measurements

3.6. Optimized performance measurements

3.7. Conclusions

3.8. Possible further optimizations

+

Chapter 3. Kea

Table of Contents

3.1. Backend performance evaluation

3.2. MySQL backend

3.2.1. MySQL tweaks

3.3. SQLite-ubench

3.3.1. SQLite tweaks

3.4. memfile-ubench

3.4.1. memfile tweaks

3.5. Basic performance measurements

3.6. Optimized performance measurements

3.7. Conclusions

3.8. Possible further optimizations

-

3.1. Backend performance evaluation

+

3.1. Backend performance evaluation

Kea will support several different database backends, using both popular databases (like MySQL or SQLite) and - custom-developed solutions (like in-memory database). BIND 10 - source code features set of performance microbenchmarks. - These are small tools written in C/C++ that simulate expected + custom-developed solutions (such as an in-memory database). + To aid in the choice of backend, the BIND 10 + source code features a set of performance microbenchmarks. + Written in C/C++, these are small tools that simulate expected DHCP server behaviour and evaluate the performance of - considered databases. As implemented benchmarks are not really + considered databases. As implemented, the benchmarks are not really simulating DHCP operation, but rather use set of primitives - that can be used by a real server, they are called + that can be used by a real server. For this reason, they are called micro-benchmarks.

Although there are many operations and data types that server could store in a database, the most frequently used data - type is lease information. Although lease information for IPv4 - and IPv6 differs slightly, it is expected that the performance + type is lease information. Although the information held for IPv4 + and IPv6 leases differs slightly, it is expected that the performance differences will be minimal between IPv4 and IPv6 lease operations. - Therefore each test uses lease4 table for performance measurements. + Therefore each test uses the lease4 table (in which IPv4 leases are stored) + for performance measurements.

All benchmarks are implemented as single threaded applications that take advantage of a single database connection.

Those benchmarks are stored in tests/tools/dhcp-ubench - directory. This directory contains simplified prototypes for - various DB back-ends that are planned or considered as a - backend engine for BIND10 DHCP. Athough trivial now, they are + directory of the BIND 10 source tree. This directory contains simplified prototypes for + the various database back-ends that are planned or considered as a + possibly for BIND10 DHCP. Athough trivial now, the benchmarks are expected to evolve into useful tools that will allow users to measure performance in their specific environment.

Currently the following benchmarks are implemented: -

in memory+flat file
SQLite
MySQL

+

In memory + flat file
SQLite
MySQL

- As they require additional (sometimes heavy) dependencies, they are not - built by default. Actually, their build system is completely separated. - It will be eventually merged with the main BIND10 makefile system, but + As the benchmarks require additional (sometimes heavy) dependencies, they are not + built by default. Actually, their build system is completely separate from that + of the rest of BIND 10. + It is anticipated that they will be eventually merged into the rest of BIND 10, but that is a low priority for now.

All benchmarks will follow the same pattern: -

prepare operation (connect to a database, create a file etc.)
Measure timestamp 0
Commit new lease4 (repeated X times)
Measure timestamp 1
Search for random lease4 (repeated X times)
Measure timestamp 2
Update existing lease4 (repeated X times)
Measure timestamp 3
Delete existing lease4 (repeated X times)
Measure timestamp 4
Print out statistics, based on X and measured timestamps.

+

Prepare operation (connect to a database, create a file etc.)
Measure timestamp 0
Commit new lease4 record (repeated N times)
Measure timestamp 1
Search for random lease4 record (repeated N times)
Measure timestamp 2
Update existing lease4 record (repeated N times)
Measure timestamp 3
Delete existing lease4 record (repeated N times)
Measure timestamp 4
Print out statistics, based on N and measured timestamps.

Although this approach does not attempt to simulate actual DHCP server - operation that has mix of all steps intervening, it answers the - questions about basic database strenghts and weak points. In particular - it can show what is the impact of specific DB optimizations, like + operation that has mix of all steps, it answers the + questions about basic database strengths and weak points. In particular + it can show what is the impact of specific database optimizations, such as changing engine, optimizing for writes/reads etc.

- The framework attempts to do the same amount of operations for every + The framework attempts to do the same amount of work for every backend thus allowing fair complarison between them. -

3.2. MySQL backend

MySQL backend requires MySQL client development libraries. It uses - mysql_config tool (that works similar to pkg-config) to discover required +

3.2. MySQL backend

The MySQL backend requires the MySQL client development libraries. It uses + the mysql_config tool (similar to pkg-config) to discover required compilation and linking options. To install required packages on Ubuntu, use the following command: @@ -71,27 +73,27 @@ Make sure that MySQL server is running. Make sure that you have your setup configured so there is a user that is able to modify used database.

Before running tests, you need to initialize your database. You can - use mysql.schema script for that purpose. WARNING: It will drop existing - Kea database. Do not run this on your production server. Assuming your - MySQL user is kea, you can initialize your test database by: + use mysql.schema script for that purpose.

WARNING: It will drop existing + Kea database. Do not run this on your production server.

Assuming your + MySQL user is "kea", you can initialize your test database by:

$ mysql -u kea -p < mysql.schema

-

After database is initialized, you are ready to run the test: +

After the database is initialized, you are ready to run the test:

$ ./mysql_ubench

or -

$ ./mysql_ubench > results->mysql.txt

+

$ ./mysql_ubench > results-mysql.txt

Redirecting output to a file is important, because for each operation there is a single character printed to show progress. If you have a slow - terminal, this may considerably affect test perfromance. On the other hand, - printing something after each operation is required, as poor DB setting - may slow down operations to around 20 per second. Observant user is expected - to note that initial dots are printed too slowly and abort the test.

Currently all default parameters are hardcoded. Default values can be - overwritten using command line switches. Although all benchmarks take - the same list of parameters, some of them are specific to a given backend - type. To get a list of supported parameters, run your benchmark with -h option: + terminal, this may considerably affect test performance. On the other hand, + printing something after each operation is required as poor database settings + may slow down operations to around 20 per second. (The observant user is expected + to note that the initial dots are printed too slowly and abort the test.)

Currently all default parameters are hardcoded. Default values can be + overridden using command line switches. Although all benchmarks take + the same list of parameters, some of them are specific to a given backend. + To get a list of supported parameters, run the benchmark with the "-h" option:

$ ./mysql_ubench -h
 This is a benchmark designed to measure expected performance
@@ -111,20 +113,20 @@ Possible command-line parameters:
  -c yes|no - should compiled statements be used (MySQL only)

-

3.2.1. MySQL tweaks

One parameter that has huge impact on performance is a a backend engine. +

3.2.1. MySQL tweaks

One parameter that has huge impact on performance is the choice of backend engine. You can get a list of engines of your MySQL implementation by using

> show engines;

in your mysql client. Two notable engines are MyISAM and InnoDB. mysql_ubench uses use MyISAM for synchronous mode and InnoDB for asynchronous. Please use - '-s 0|1' to choose whether you want synchronous or asynchronous operations.

Another parameter that affects performance are precompiled statements. + '-s yes|no' to choose whether you want synchronous or asynchronous operations.

Another parameter that affects performance are precompiled statements. In a basic approach, the actual SQL query is passed as a text string that is then parsed by the database engine. Alternative is a so called precompiled statement. In this approach the SQL query is compiled an specific values are being bound to it. In the next iteration the query remains the same, only bound values are changing (e.g. searching for a different address). Usage of basic or precompiled - statements is controlled with '-c 0|1'.

3.3. SQLite-ubench

SQLite backend requires both sqlite3 development and run-time package. Their + statements is controlled with '-c no|yes'.

3.3. SQLite-ubench

The SQLite backend requires both the sqlite3 development and run-time packages. Their names may vary from system to system, but on Ubuntu 12.04 they are called sqlite3 libsqlite3-dev. To install them, use the following command: @@ -140,33 +142,33 @@ Possible command-line parameters: or

> ./sqlite_ubench > results-sqlite.txt

3.3.1. SQLite tweaks

To modify default sqlite_ubench parameters, command line - switches can be used. Currently supported parameters are + switches can be used. The currently supported switches are (default values specified in brackets): -

-f filename - name of the database file ("sqlite.db")
-n num - number of iterations (100)
-s yes|no - should the operations be performend in synchronous (yes) +
1. -f filename - name of the database file ("sqlite.db")
2. -n num - number of iterations (100)
3. -s yes|no - should the operations be performed in a synchronous (yes) or asynchronous (no) manner (yes)
4. -v yes|no - verbose mode. Should the test print out progress? (yes)
5. -c yes|no - precompiled statements. Should the SQL statements be precompiled?
SQLite can run in asynchronous or synchronous mode. This - mode can be controlled by using sync parameter. It is set - using (PRAGMA synchronous = ON or OFF).
Another tweakable feature is journal mode. It can be + mode can be controlled by using "synchronous" parameter. It is set + using the SQLite command:
PRAGMA synchronous = ON|OFF
Another tweakable feature is journal mode. It can be turned to several modes of operation. Its value can be modified in SQLite_uBenchmark::connect(). See http://www.sqlite.org/pragma.html#pragma_journal_mode for detailed explanantion.
sqlite_bench supports precompiled statements. Please use - '-c 0|1' to define which should be used: basic SQL query (0) or - precompiled statement (1).

3.4. memfile-ubench

Memfile backend is custom developed prototype backend that - somewhat mimics operation of ISC DHCP4. It uses in-memory + '-c no|yes' to define which should be used: basic SQL query (no) or + precompiled statement (yes).

3.4. memfile-ubench

The memfile backend is a custom backend that + somewhat mimics operation of ISC DHCP4. It implements in-memory storage using standard C++ and boost mechanisms (std::map and boost::shared_ptr<>). All database changes are also - written to a lease file. That file is strictly write-only. This + written to a lease file, which is strictly write-only. This approach takes advantage of the fact that simple append is faster than edition with potential whole file relocation.

3.4.1. memfile tweaks

To modify default memfile_ubench parameters, command line - switches can be used. Currently supported parameters are + switches can be used. Currently supported switches are (default values specified in brackets): -

-f filename - name of the database file ("dhcpd.leases")
-n num - number of iterations (100)
-s yes|no - should the operations be performend in synchronous (yes) +
1. -f filename - name of the database file ("dhcpd.leases")
2. -n num - number of iterations (100)
3. -s yes|no - should the operations be performend in a synchronous (yes) or asynchronous (no) manner (yes)
4. -v yes|no - verbose mode. Should the test print out progress? (yes)
memfile can run in asynchronous or synchronous mode. This mode can be controlled by using sync parameter. It uses fflush() and fsync() in synchronous mode to make sure that - data is not buffered and physically stored on disk.

3.5. Basic performance measurements

This section contains sample results for backend performance measurements, + data is not buffered and physically stored on disk.

3.5. Basic performance measurements

This section contains sample results for backend performance measurements, taken using microbenchmarks. Tests were conducted on reasonably powerful machine:

 CPU: Quad-core Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 logical cores)
@@ -175,19 +177,19 @@ OS: Ubuntu 12.04, running kernel 3.2.0-26-generic SMP x86_64
 compiler: g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
 MySQL version: 5.5.24
 SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe77b41d959e9df13f8c9b5e

-

Benchmarks were run without using precompiled statements. - The code was compiled wit -O0 flag (no code optimizations). - Each run was executed once.

Benchmarks were run in two series: synchronous and +

The benchmarks were run without using precompiled statements. + The code was compiled with the -O0 flag (no code optimizations). + Each run was executed once.

Two series of measures were made, synchronous and asynchronous. As those modes offer radically different - performances, synchronous mode was conducted for 1000 (one - thousand) repetitions and asynchronous mode was conducted for - 100000 (hundred thousand) repetitions.

Table 3.1. Synchronous results (basic)

Backend	Operations	Create	Search	Update	Delete	Average
MySQL	1000	31.603978s	0.116612s	27.964191s	27.695209s	21.844998s
SQLite	1000	61.421356s	0.033283s	59.476638s	56.034150s	44.241357s
memfile	1000	38.223757s	0.000817s	38.041153s	38.017293s	28.570755s

Following parameters were measured for asynchronous mode. - MySQL and SQLite were run with 100 thousand repetitions. Memfile - was run for 1 million repetitions due to much larger performance.

Table 3.2. Asynchronous results (basic)

Backend	Operations	Create [s]	Search [s]	Update [s]	Delete [s]	Average [s]
MySQL	100000	10.584842s	10.386402s	10.062384s	8.890197s	9.980956s
SQLite	100000	3.710356s	3.159129s	2.865354s	2.439406s	3.043561s
memfile	100000	1.299642s	0.039330s	1.307112s	1.277641s	0.980931s

Presented performance results can be computed into operations per second metrics. - It should be noted that due to large differences between various operations (sometime - over 3 orders of magnitude), it is difficult to create a simple, readable chart with - that data.

Table 3.3. Estimated basic performance

Backend	Create [oper/s]	Search [oper/s]	Update [oper/s]	Delete [oper/s]	Average [oper/s]
MySQL (async)	9447.47	9627.97	9938.00	11248.34	10065.45
SQLite (async)	26951.59	31654.29	34899.70	40993.59	33624.79
memfile (async)	76944.27	2542588.35	76504.54	78269.25	693576.60
MySQL (sync)	31.64	8575.45	35.76	36.11	2169.74
SQLite (sync)	16.28	20045.37	16.81	17.85	7524.08
memfile (sync)	26.16	1223990.21	26.29	26.30	306017.24

Basic performance measurements

Graphical representation of the basic performance results - presented in table Table 3.3, “Estimated basic performance”.

3.6. Optimized performance measurements

This section contains sample results for backend performance measurements, + performances, synchronous mode was conducted for one + thousand repetitions and asynchronous mode was conducted for + one hundred thousand repetitions.

Table 3.1. Synchronous results (basic)

Backend	Operations	Create [s]	Search [s]	Update [s]	Delete [s]	Average [s]
MySQL	1,000	31.603978	0.116612	27.964191	27.695209	21.844998
SQLite	1,000	61.421356	0.033283	59.476638	56.034150	44.241357
memfile	1,000	38.223757	0.000817	38.041153	38.017293	28.570755

The following parameters were measured for asynchronous mode. + MySQL and SQLite were run with one hundred thousand repetitions. Memfile + was run for one million repetitions due to its much higher performance.

Table 3.2. Asynchronous results (basic)

Backend	Operations	Create [s]	Search [s]	Update [s]	Delete [s]	Average [s]
MySQL	100,000	10.584842	10.386402	10.062384	8.890197	9.980956
SQLite	100,000	3.710356	3.159129	2.865354	2.439406	3.043561
memfile	1,000,000	1.299642	0.039330	1.307112	1.277641	0.980931

The presented performance results can be converted into operations per second metrics. + It should be noted that due to large differences between various operations (sometimes + over three orders of magnitude), it is difficult to create a simple, readable chart with + that data.

Table 3.3. Estimated basic performance

Backend	Create [oper/s]	Search [oper/s]	Update [oper/s]	Delete [oper/s]	Average [oper/s]
MySQL (async)	9447.47	9627.97	9938.00	11248.34	10065.45
SQLite (async)	26951.59	31654.29	34899.70	40993.59	33624.79
memfile (async)	76944.27	2542588.35	76504.54	78269.25	693576.60
MySQL (sync)	31.64	8575.45	35.76	36.11	2169.74
SQLite (sync)	16.28	20045.37	16.81	17.85	7524.08
memfile (sync)	26.16	1223990.21	26.29	26.30	306017.24

Basic performance measurements

Graphical representation of the basic performance results + presented in table Table 3.3, “Estimated basic performance”.

3.6. Optimized performance measurements

This section contains sample results for backend performance measurements, taken using microbenchmarks. Tests were conducted on reasonably powerful machine:

 CPU: Quad-core Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 logical cores)
@@ -196,19 +198,19 @@ OS: Ubuntu 12.04, running kernel 3.2.0-26-generic SMP x86_64
 compiler: g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
 MySQL version: 5.5.24
 SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe77b41d959e9df13f8c9b5e

-

Benchmarks were run with precompiled statements enabled. - The code was compiled wit -Ofast flag (optimize compilation for speed). - Each run was repeated 3 times and measured values were averaged.

Benchmarks were run in two series: synchronous and +

The benchmarks were run with precompiled statements enabled. + The code was compiled with the -Ofast flag (optimize compilation for speed). + Each run was repeated three times and measured values were averaged.

Again the benchmarks were run in two series, synchronous and asynchronous. As those modes offer radically different - performances, synchronous mode was conducted for 1000 (one - thousand) repetitions and asynchronous mode was conducted for - 100000 (hundred thousand) repetitions.

Table 3.4. Synchronous results (optimized)

Backend	Operations	Create	Search	Update	Delete	Average
MySQL	1000	27.887s	0.106s	28.223s	27.696s	20.978s
SQLite	1000	61.299s	0.015s	59.648s	61.098s	45.626s
memfile	1000	39.564s	0.000724s	39.543s	39.326w	29.608s

Following parameters were measured for asynchronous mode. - MySQL and SQLite were run with 100 thousand repetitions. Memfile - was run for 1 million repetitions due to much larger performance.

Table 3.5. Asynchronous results (optimized)

Backend	Operations	Create [s]	Search [s]	Update [s]	Delete [s]	Average [s]
MySQL	100000	8.507s	9.698s	7.785s	8.326s	8.579s
SQLite	100000	1.562s	0.949s	1.513s	1.502s	1.382s
memfile	100000	1.302s	0.038s	1.306s	1.263s	0.977s

Presented performance results can be computed into operations per second metrics. + performances, synchronous mode was conducted for one + thousand repetitions and asynchronous mode was conducted for + one hundred thousand repetitions.

Table 3.4. Synchronous results (optimized)

Backend	Operations	Create [s]	Search [s]	Update [s]	Delete [s]	Average [s]
MySQL	1,000	27.887	0.106	28.223	27.696	20.978
SQLite	1,000	61.299	0.015	59.648	61.098	45.626
memfile	1,000	39.564	0.000724	39.543	39.326	29.608

The following parameters were measured for asynchronous mode. + MySQL and SQLite were run with one hundred thousand repetitions. Memfile + was run for one million repetitions due to its much higher performance.

Table 3.5. Asynchronous results (optimized)

Backend	Operations	Create [s]	Search [s]	Update [s]	Delete [s]	Average [s]
MySQL	100,000	8.507	9.698	7.785	8.326	8.579
SQLite	100,000	1.562	0.949	1.513	1.502	1.382
memfile	1,000,000	1.302	0.038	1.306	1.263	0.977

The presented performance results can be converted into operations per second metrics. It should be noted that due to large differences between various operations (sometime - over 3 orders of magnitude), it is difficult to create a simple, readable chart with - that data.

Table 3.6. Estimated optimized performance

Backend	Create [oper/s]	Search [oper/s]	Update [oper/s]	Delete [oper/s]	Average [oper/s]
MySQL (async)	11754.84	10311.34	12845.35	12010.24	11730.44
SQLite (async)	64005.90	105391.29	66075.51	66566.43	75509.78
memfile (async)	76832.16	2636018.56	76542.50	79188.81	717145.51
MySQL (sync)	35.86	9461.10	35.43	36.11	2392.12
SQLite (sync)	16.31	67036.11	16.76	16.37	16771.39
memfile (sync)	25.28	3460207.61	25.29	25.43	865070.90

Optimized performance measurements

Graphical representation of the optimized performance - results presented in table Table 3.6, “Estimated optimized performance”.

3.7. Conclusions

+ over three orders of magnitude), it is difficult to create a simple, readable chart with + the data.

Table 3.6. Estimated optimized performance

Backend	Create [oper/s]	Search [oper/s]	Update [oper/s]	Delete [oper/s]	Average [oper/s]
MySQL (async)	11754.84	10311.34	12845.35	12010.24	11730.44
SQLite (async)	64005.90	105391.29	66075.51	66566.43	75509.78
memfile (async)	76832.16	2636018.56	76542.50	79188.81	717145.51
MySQL (sync)	35.86	9461.10	35.43	36.11	2392.12
SQLite (sync)	16.31	67036.11	16.76	16.37	16771.39
memfile (sync)	25.28	3460207.61	25.29	25.43	865070.90

Optimized performance measurements

Graphical representation of the optimized performance + results presented in table Table 3.6, “Estimated optimized performance”.

3.7. Conclusions

Improvements gained by introducing support for precompiled statements in MySQL is somewhat disappointing - between 6 and 29%. On the other hand, the improvement in SQLite is @@ -222,15 +224,15 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 STL code. The memfile backend is in some operations is almost twice as fast.

- If synchronous operation is required the current performance + If synchronous operation is required, the current performance results are likely to be deemed inadequate. The limiting factor here is a disk access time. Even migrating to high - performance 15.000rpm disk is expected to only roughly double + performance 15,000 rpm disk is expected to only roughly double number of leases per second, compared to the current results. - The reason is that to write a file to disk, at lease 2 writes + The reason is that to write a file to disk, at least two writes are required: the new content and i-node modification of the file. The easiest way to boost synchronous performance is to - switch to SSD disks. Memory-backed RAM disks are also viable + switch to SSD disks. Memory-backed RAM disks are also a viable solution. However, care should be taken to properly engineer backup strategy for RAM disks.

@@ -253,10 +255,10 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 translated to expected leases per second or queries per second performance by an actual server. The DHCP server must do much more than just query the database to properly process client's - message. Provided results should be considered as only rough + message. The provided results should be considered as only rough estimates. They can also be used for relative comparisons between backends. -

3.8. Possible further optimizations

+

3.8. Possible further optimizations

For basic measurements the code was compiled with -g -O0 flags. For optimized measurements the benchmarking code was compiled with -Ofast (optimize for speed). In both cases, the @@ -269,8 +271,8 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7

Currently all operations are conducted on one by one basis. Each operation is treated as a separate - transaction. Grouping X operations together will potentially - bring almost X fold increase in synchronous operations. Such a + transaction. Grouping N operations together will potentially + bring almost N fold increase in synchronous operations. Such a feature is present in ISC DHCP4 and is called cache-threshold. Extension for this benchmark in this regard should be considered. That affects only write operations (insert, diff --git a/tests/tools/dhcp-ubench/dhcp-perf-guide.xml b/tests/tools/dhcp-ubench/dhcp-perf-guide.xml index 1ddd79b4c1..f149c44ea6 100644 --- a/tests/tools/dhcp-ubench/dhcp-perf-guide.xml +++ b/tests/tools/dhcp-ubench/dhcp-perf-guide.xml @@ -40,8 +40,8 @@ BIND 10 is a framework that features Domain Name System - (DNS) suite and Dynamic Host Configuration Protocol (DHCP) - servers with development managed by Internet Systems Consortium (ISC). + (DNS) and Dynamic Host Configuration Protocol (DHCP) + software with development managed by Internet Systems Consortium (ISC). This document describes various aspects of DHCP performance, measurements and tuning. It covers BIND 10 DHCP (codename Kea), existing ISC DHCP4 software, perfdhcp (a DHCP performance @@ -70,11 +70,10 @@ Introduction - This document is in its early stages of development. It is - expected to grow significantly in a near future. It will + This document is in the early stages of development. It is + expected to grow significantly in the near future. It will cover topics like database backend perfomance measurements, - pros an cons of various optimization techniques and - tools. + tools, and the pros an cons of various optimization techniques. @@ -97,22 +96,24 @@ Kea will support several different database backends, using both popular databases (like MySQL or SQLite) and - custom-developed solutions (like in-memory database). BIND 10 - source code features set of performance microbenchmarks. - These are small tools written in C/C++ that simulate expected + custom-developed solutions (such as an in-memory database). + To aid in the choice of backend, the BIND 10 + source code features a set of performance microbenchmarks. + Written in C/C++, these are small tools that simulate expected DHCP server behaviour and evaluate the performance of - considered databases. As implemented benchmarks are not really + considered databases. As implemented, the benchmarks are not really simulating DHCP operation, but rather use set of primitives - that can be used by a real server, they are called + that can be used by a real server. For this reason, they are called micro-benchmarks. Although there are many operations and data types that server could store in a database, the most frequently used data - type is lease information. Although lease information for IPv4 - and IPv6 differs slightly, it is expected that the performance + type is lease information. Although the information held for IPv4 + and IPv6 leases differs slightly, it is expected that the performance differences will be minimal between IPv4 and IPv6 lease operations. - Therefore each test uses lease4 table for performance measurements. + Therefore each test uses the lease4 table (in which IPv4 leases are stored) + for performance measurements. All benchmarks are implemented as single threaded applications @@ -120,9 +121,9 @@ Those benchmarks are stored in tests/tools/dhcp-ubench - directory. This directory contains simplified prototypes for - various DB back-ends that are planned or considered as a - backend engine for BIND10 DHCP. Athough trivial now, they are + directory of the BIND 10 source tree. This directory contains simplified prototypes for + the various database back-ends that are planned or considered as a + possibly for BIND10 DHCP. Athough trivial now, the benchmarks are expected to evolve into useful tools that will allow users to measure performance in their specific environment. @@ -130,52 +131,53 @@ Currently the following benchmarks are implemented: - in memory+flat file + In memory + flat file SQLite MySQL - As they require additional (sometimes heavy) dependencies, they are not - built by default. Actually, their build system is completely separated. - It will be eventually merged with the main BIND10 makefile system, but + As the benchmarks require additional (sometimes heavy) dependencies, they are not + built by default. Actually, their build system is completely separate from that + of the rest of BIND 10. + It is anticipated that they will be eventually merged into the rest of BIND 10, but that is a low priority for now. All benchmarks will follow the same pattern: - prepare operation (connect to a database, create a file etc.) + Prepare operation (connect to a database, create a file etc.) Measure timestamp 0 - Commit new lease4 (repeated X times) + Commit new lease4 record (repeated N times) Measure timestamp 1 - Search for random lease4 (repeated X times) + Search for random lease4 record (repeated N times) Measure timestamp 2 - Update existing lease4 (repeated X times) + Update existing lease4 record (repeated N times) Measure timestamp 3 - Delete existing lease4 (repeated X times) + Delete existing lease4 record (repeated N times) Measure timestamp 4 - Print out statistics, based on X and measured timestamps. + Print out statistics, based on N and measured timestamps. Although this approach does not attempt to simulate actual DHCP server - operation that has mix of all steps intervening, it answers the - questions about basic database strenghts and weak points. In particular - it can show what is the impact of specific DB optimizations, like + operation that has mix of all steps, it answers the + questions about basic database strengths and weak points. In particular + it can show what is the impact of specific database optimizations, such as changing engine, optimizing for writes/reads etc. - The framework attempts to do the same amount of operations for every + The framework attempts to do the same amount of work for every backend thus allowing fair complarison between them.

MySQL backend - MySQL backend requires MySQL client development libraries. It uses - mysql_config tool (that works similar to pkg-config) to discover required + The MySQL backend requires the MySQL client development libraries. It uses + the mysql_config tool (similar to pkg-config) to discover required compilation and linking options. To install required packages on Ubuntu, use the following command: @@ -185,31 +187,35 @@ configured so there is a user that is able to modify used database. Before running tests, you need to initialize your database. You can - use mysql.schema script for that purpose. WARNING: It will drop existing - Kea database. Do not run this on your production server. Assuming your - MySQL user is kea, you can initialize your test database by: + use mysql.schema script for that purpose. + + WARNING: It will drop existing + Kea database. Do not run this on your production server. + + Assuming your + MySQL user is "kea", you can initialize your test database by: $ mysql -u kea -p < mysql.schema - After database is initialized, you are ready to run the test: + After the database is initialized, you are ready to run the test: $ ./mysql_ubench or - $ ./mysql_ubench > results->mysql.txt + $ ./mysql_ubench > results-mysql.txt Redirecting output to a file is important, because for each operation there is a single character printed to show progress. If you have a slow - terminal, this may considerably affect test perfromance. On the other hand, - printing something after each operation is required, as poor DB setting - may slow down operations to around 20 per second. Observant user is expected - to note that initial dots are printed too slowly and abort the test. + terminal, this may considerably affect test performance. On the other hand, + printing something after each operation is required as poor database settings + may slow down operations to around 20 per second. (The observant user is expected + to note that the initial dots are printed too slowly and abort the test.) Currently all default parameters are hardcoded. Default values can be - overwritten using command line switches. Although all benchmarks take - the same list of parameters, some of them are specific to a given backend - type. To get a list of supported parameters, run your benchmark with -h option: + overridden using command line switches. Although all benchmarks take + the same list of parameters, some of them are specific to a given backend. + To get a list of supported parameters, run the benchmark with the "-h" option: $ ./mysql_ubench -h This is a benchmark designed to measure expected performance @@ -234,14 +240,14 @@ Possible command-line parameters:

MySQL tweaks - One parameter that has huge impact on performance is a a backend engine. + One parameter that has huge impact on performance is the choice of backend engine. You can get a list of engines of your MySQL implementation by using > show engines; in your mysql client. Two notable engines are MyISAM and InnoDB. mysql_ubench uses use MyISAM for synchronous mode and InnoDB for asynchronous. Please use - '-s 0|1' to choose whether you want synchronous or asynchronous operations. + '-s yes|no' to choose whether you want synchronous or asynchronous operations. Another parameter that affects performance are precompiled statements. In a basic approach, the actual SQL query is passed as a text string that is @@ -249,14 +255,14 @@ Possible command-line parameters: statement. In this approach the SQL query is compiled an specific values are being bound to it. In the next iteration the query remains the same, only bound values are changing (e.g. searching for a different address). Usage of basic or precompiled - statements is controlled with '-c 0|1'. + statements is controlled with '-c no|yes'.

SQLite-ubench - SQLite backend requires both sqlite3 development and run-time package. Their + The SQLite backend requires both the sqlite3 development and run-time packages. Their names may vary from system to system, but on Ubuntu 12.04 they are called sqlite3 libsqlite3-dev. To install them, use the following command: @@ -278,12 +284,12 @@ Possible command-line parameters:

SQLite tweaks To modify default sqlite_ubench parameters, command line - switches can be used. Currently supported parameters are + switches can be used. The currently supported switches are (default values specified in brackets): -f filename - name of the database file ("sqlite.db") -n num - number of iterations (100) - -s yes|no - should the operations be performend in synchronous (yes) + -s yes|no - should the operations be performed in a synchronous (yes) or asynchronous (no) manner (yes) -v yes|no - verbose mode. Should the test print out progress? (yes) -c yes|no - precompiled statements. Should the SQL statements be precompiled? @@ -291,8 +297,10 @@ Possible command-line parameters: SQLite can run in asynchronous or synchronous mode. This - mode can be controlled by using sync parameter. It is set - using (PRAGMA synchronous = ON or OFF). + mode can be controlled by using "synchronous" parameter. It is set + using the SQLite command: + + PRAGMA synchronous = ON|OFF Another tweakable feature is journal mode. It can be turned to several modes of operation. Its value can be @@ -301,30 +309,30 @@ Possible command-line parameters: detailed explanantion. sqlite_bench supports precompiled statements. Please use - '-c 0|1' to define which should be used: basic SQL query (0) or - precompiled statement (1). + '-c no|yes' to define which should be used: basic SQL query (no) or + precompiled statement (yes).

memfile-ubench - Memfile backend is custom developed prototype backend that - somewhat mimics operation of ISC DHCP4. It uses in-memory + The memfile backend is a custom backend that + somewhat mimics operation of ISC DHCP4. It implements in-memory storage using standard C++ and boost mechanisms (std::map and boost::shared_ptr<>). All database changes are also - written to a lease file. That file is strictly write-only. This + written to a lease file, which is strictly write-only. This approach takes advantage of the fact that simple append is faster than edition with potential whole file relocation.

memfile tweaks To modify default memfile_ubench parameters, command line - switches can be used. Currently supported parameters are + switches can be used. Currently supported switches are (default values specified in brackets): -f filename - name of the database file ("dhcpd.leases") -n num - number of iterations (100) - -s yes|no - should the operations be performend in synchronous (yes) + -s yes|no - should the operations be performend in a synchronous (yes) or asynchronous (no) manner (yes) -v yes|no - verbose mode. Should the test print out progress? (yes) @@ -350,15 +358,15 @@ MySQL version: 5.5.24 SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe77b41d959e9df13f8c9b5e - Benchmarks were run without using precompiled statements. - The code was compiled wit -O0 flag (no code optimizations). + The benchmarks were run without using precompiled statements. + The code was compiled with the -O0 flag (no code optimizations). Each run was executed once. - Benchmarks were run in two series: synchronous and + Two series of measures were made, synchronous and asynchronous. As those modes offer radically different - performances, synchronous mode was conducted for 1000 (one - thousand) repetitions and asynchronous mode was conducted for - 100000 (hundred thousand) repetitions. + performances, synchronous mode was conducted for one + thousand repetitions and asynchronous mode was conducted for + one hundred thousand repetitions. Synchronous results (basic) @@ -374,51 +382,51 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 Backend Operations - Create - Search - Update - Delete - Average + Create [s] + Search [s] + Update [s] + Delete [s] + Average [s] MySQL - 1000 - 31.603978s - 0.116612s - 27.964191s - 27.695209s - 21.844998s + 1,000 + 31.603978 + 0.116612 + 27.964191 + 27.695209 + 21.844998 SQLite - 1000 - 61.421356s - 0.033283s - 59.476638s - 56.034150s - 44.241357s + 1,000 + 61.421356 + 0.033283 + 59.476638 + 56.034150 + 44.241357 memfile - 1000 - 38.223757s - 0.000817s - 38.041153s - 38.017293s - 28.570755s + 1,000 + 38.223757 + 0.000817 + 38.041153 + 38.017293 + 28.570755

- Following parameters were measured for asynchronous mode. - MySQL and SQLite were run with 100 thousand repetitions. Memfile - was run for 1 million repetitions due to much larger performance. + The following parameters were measured for asynchronous mode. + MySQL and SQLite were run with one hundred thousand repetitions. Memfile + was run for one million repetitions due to its much higher performance. Asynchronous results (basic) @@ -444,41 +452,41 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 MySQL - 100000 - 10.584842s - 10.386402s - 10.062384s - 8.890197s - 9.980956s + 100,000 + 10.584842 + 10.386402 + 10.062384 + 8.890197 + 9.980956 SQLite - 100000 - 3.710356s - 3.159129s - 2.865354s - 2.439406s - 3.043561s + 100,000 + 3.710356 + 3.159129 + 2.865354 + 2.439406 + 3.043561 memfile - 100000 - 1.299642s - 0.039330s - 1.307112s - 1.277641s - 0.980931s + 1,000,000 + 1.299642 + 0.039330 + 1.307112 + 1.277641 + 0.980931

- Presented performance results can be computed into operations per second metrics. - It should be noted that due to large differences between various operations (sometime - over 3 orders of magnitude), it is difficult to create a simple, readable chart with + The presented performance results can be converted into operations per second metrics. + It should be noted that due to large differences between various operations (sometimes + over three orders of magnitude), it is difficult to create a simple, readable chart with that data. Estimated basic performance @@ -587,15 +595,15 @@ MySQL version: 5.5.24 SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe77b41d959e9df13f8c9b5e - Benchmarks were run with precompiled statements enabled. - The code was compiled wit -Ofast flag (optimize compilation for speed). - Each run was repeated 3 times and measured values were averaged. + The benchmarks were run with precompiled statements enabled. + The code was compiled with the -Ofast flag (optimize compilation for speed). + Each run was repeated three times and measured values were averaged. - Benchmarks were run in two series: synchronous and + Again the benchmarks were run in two series, synchronous and asynchronous. As those modes offer radically different - performances, synchronous mode was conducted for 1000 (one - thousand) repetitions and asynchronous mode was conducted for - 100000 (hundred thousand) repetitions. + performances, synchronous mode was conducted for one + thousand repetitions and asynchronous mode was conducted for + one hundred thousand repetitions.

Synchronous results (optimized) @@ -611,51 +619,51 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 Backend Operations - Create - Search - Update - Delete - Average + Create [s] + Search [s] + Update [s] + Delete [s] + Average [s] MySQL - 1000 - 27.887s - 0.106s - 28.223s - 27.696s - 20.978s + 1,000 + 27.887 + 0.106 + 28.223 + 27.696 + 20.978 SQLite - 1000 - 61.299s - 0.015s - 59.648s - 61.098s - 45.626s + 1,000 + 61.299 + 0.015 + 59.648 + 61.098 + 45.626 memfile - 1000 - 39.564s - 0.000724s - 39.543s - 39.326w - 29.608s + 1,000 + 39.564 + 0.000724 + 39.543 + 39.326 + 29.608

- Following parameters were measured for asynchronous mode. - MySQL and SQLite were run with 100 thousand repetitions. Memfile - was run for 1 million repetitions due to much larger performance. + The following parameters were measured for asynchronous mode. + MySQL and SQLite were run with one hundred thousand repetitions. Memfile + was run for one million repetitions due to its much higher performance. Asynchronous results (optimized) @@ -681,42 +689,42 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 MySQL - 100000 - 8.507s - 9.698s - 7.785s - 8.326s - 8.579s + 100,000 + 8.507 + 9.698 + 7.785 + 8.326 + 8.579 SQLite - 100000 - 1.562s - 0.949s - 1.513s - 1.502s - 1.382s + 100,000 + 1.562 + 0.949 + 1.513 + 1.502 + 1.382 memfile - 100000 - 1.302s - 0.038s - 1.306s - 1.263s - 0.977s + 1,000,000 + 1.302 + 0.038 + 1.306 + 1.263 + 0.977

- Presented performance results can be computed into operations per second metrics. + The presented performance results can be converted into operations per second metrics. It should be noted that due to large differences between various operations (sometime - over 3 orders of magnitude), it is difficult to create a simple, readable chart with - that data. + over three orders of magnitude), it is difficult to create a simple, readable chart with + the data. Estimated optimized performance @@ -832,15 +840,15 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 - If synchronous operation is required the current performance + If synchronous operation is required, the current performance results are likely to be deemed inadequate. The limiting factor here is a disk access time. Even migrating to high - performance 15.000rpm disk is expected to only roughly double + performance 15,000 rpm disk is expected to only roughly double number of leases per second, compared to the current results. - The reason is that to write a file to disk, at lease 2 writes + The reason is that to write a file to disk, at least two writes are required: the new content and i-node modification of the file. The easiest way to boost synchronous performance is to - switch to SSD disks. Memory-backed RAM disks are also viable + switch to SSD disks. Memory-backed RAM disks are also a viable solution. However, care should be taken to properly engineer backup strategy for RAM disks. @@ -867,7 +875,7 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 translated to expected leases per second or queries per second performance by an actual server. The DHCP server must do much more than just query the database to properly process client's - message. Provided results should be considered as only rough + message. The provided results should be considered as only rough estimates. They can also be used for relative comparisons between backends. @@ -891,8 +899,8 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 Currently all operations are conducted on one by one basis. Each operation is treated as a separate - transaction. Grouping X operations together will potentially - bring almost X fold increase in synchronous operations. Such a + transaction. Grouping N operations together will potentially + bring almost N fold increase in synchronous operations. Such a feature is present in ISC DHCP4 and is called cache-threshold. Extension for this benchmark in this regard should be considered. That affects only write operations (insert,