From: Tomek Mrugalski Date: Tue, 11 Sep 2012 12:16:40 +0000 (+0200) Subject: [2143] Minor corrections in DHCP Performance Guide after review. X-Git-Tag: trac2351_base~47^2~13^2 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=0ed7d2f838ec4bf431ea8a960a5c44e9c0951481;p=thirdparty%2Fkea.git [2143] Minor corrections in DHCP Performance Guide after review. --- diff --git a/tests/tools/dhcp-ubench/dhcp-perf-guide.html b/tests/tools/dhcp-ubench/dhcp-perf-guide.html index 40a07581dc..df2308341f 100644 --- a/tests/tools/dhcp-ubench/dhcp-perf-guide.html +++ b/tests/tools/dhcp-ubench/dhcp-perf-guide.html @@ -1,11 +1,11 @@ -DHCP Performance Guide

DHCP Performance Guide

Tomasz Mrugalski

This is a companion document for BIND 10 version +DHCP Performance Guide

DHCP Performance Guide

Tomasz Mrugalski

This is a companion document for BIND 10 version 20120712.

Abstract

BIND 10 is a framework that features Domain Name System (DNS) and Dynamic Host Configuration Protocol (DHCP) software with development managed by Internet Systems Consortium (ISC). This document describes various aspects of DHCP performance, measurements and tuning. It covers BIND 10 DHCP (codename Kea), existing ISC DHCP4 software, perfdhcp (a DHCP performance - measurement tool) and other related topics.


Preface

Table of Contents

1. Acknowledgements

1. Acknowledgements

ISC would like to acknowledge generous support for + measurement tool) and other related topics.


Preface

Table of Contents

1. Acknowledgements

1. Acknowledgements

ISC would like to acknowledge generous support for BIND 10 development of DHCPv4 and DHCPv6 components provided by Comcast.

Chapter 1. Introduction

This document is in the early stages of development. It is @@ -14,9 +14,9 @@ tools, and the pros an cons of various optimization techniques.

Chapter 2. ISC DHCP 4.x

TODO: Write something about ISC DHCP4 here. -

Chapter 3. Kea

-

3.1. Backend performance evaluation

+

3.1. Backend performance evaluation

Kea will support several different database backends, using both popular databases (like MySQL or SQLite) and custom-developed solutions (such as an in-memory database). @@ -37,21 +37,20 @@ for performance measurements.

All benchmarks are implemented as single threaded applications that take advantage of a single database connection.

- Those benchmarks are stored in tests/tools/dhcp-ubench - directory of the BIND 10 source tree. This directory contains simplified prototypes for + Those benchmarks are stored in tests/tools/dhcp-ubench directory of the + BIND 10 source tree. This directory contains simplified prototypes for the various database back-ends that are planned or considered as a - possibly for BIND10 DHCP. Athough trivial now, the benchmarks are - expected to evolve into useful tools that will allow users to - measure performance in their specific environment. + possibly for BIND10 DHCP. These benchmarks are expected to evolve into + useful tools that will allow users to measure performance in their + specific environment.

Currently the following benchmarks are implemented:

  • In memory + flat file

  • SQLite

  • MySQL

- As the benchmarks require additional (sometimes heavy) dependencies, they are not - built by default. Actually, their build system is completely separate from that - of the rest of BIND 10. - It is anticipated that they will be eventually merged into the rest of BIND 10, but - that is a low priority for now. + As the benchmarks require additional (sometimes heavy) dependencies, they + are not built by default. Actually, their build system is completely + separate from that of the rest of BIND 10. It will be eventually merged + with the main BIND 10 build system.

All benchmarks will follow the same pattern:

  1. Prepare operation (connect to a database, create a file etc.)

  2. Measure timestamp 0

  3. Commit new lease4 record (repeated N times)

  4. Measure timestamp 1

  5. Search for random lease4 record (repeated N times)

  6. Measure timestamp 2

  7. Update existing lease4 record (repeated N times)

  8. Measure timestamp 3

  9. Delete existing lease4 record (repeated N times)

  10. Measure timestamp 4

  11. Print out statistics, based on N and measured timestamps.

@@ -95,31 +94,19 @@ the same list of parameters, some of them are specific to a given backend. To get a list of supported parameters, run the benchmark with the "-h" option: -

$ ./mysql_ubench -h
-This is a benchmark designed to measure expected performance
-of several backends. This particular version identifies itself
-as following:
-MySQL client version is 5.5.24
-
-Possible command-line parameters:
- -h - help (you are reading this)
- -m hostname - specifies MySQL server to connect (MySQL backend only)
- -u username - specifies MySQL user name (MySQL backend only)
- -p password - specifies MySQL passwod (MySQL backend only)
- -f name - database or filename (MySQL, SQLite and memfile)
- -n integer - number of test repetitions (MySQL, SQLite and memfile)
- -s yes|no - synchronous/asynchronous operation (MySQL, SQLite and memfile)
- -v yes|no - verbose mode (MySQL, SQLite and memfile)
- -c yes|no - should compiled statements be used (MySQL only)
-

- +

$ ./mysql_ubench -h

Synchronous operation requires database backend to physically store changes to disk before proceeding. This property ensures that no data is lost in case of the server failure. Unfortunately, it slows operation considerably. Asynchronous mode allows database to write data at a later time (usually controlled by the database engine on OS - disk buffering mechanism).

3.2.1. MySQL tweaks

One parameter that has huge impact on performance is the choice of backend engine. + disk buffering mechanism).

3.2.1. MySQL tweaks

To modify the default mysql_ubench parameters, command line + switches can be used. The currently supported switches are + (default values specified in brackets): +

  1. -f name - name of the database ("kea")

  2. -m hostname - name of the database host ("localhost")

  3. -u user - MySQL username ("root")

  4. -p password - MySQL password ("secret")

  5. -n num - number of iterations (100)

  6. -s yes|no - should the operations be performed in a synchronous (yes) + or asynchronous (no) manner (yes)

  7. -v yes|no - verbose mode. Should the test print out progress? (yes)

  8. -c yes|no - precompiled statements. Should the SQL statements be precompiled? (yes)

+

One parameter that has huge impact on performance is the choice of backend engine. You can get a list of engines of your MySQL implementation by using

> show engines;

@@ -151,7 +138,7 @@ Possible command-line parameters: switches can be used. The currently supported switches are (default values specified in brackets):

  1. -f filename - name of the database file ("sqlite.db")

  2. -n num - number of iterations (100)

  3. -s yes|no - should the operations be performed in a synchronous (yes) - or asynchronous (no) manner (yes)

  4. -v yes|no - verbose mode. Should the test print out progress? (yes)

  5. -c yes|no - precompiled statements. Should the SQL statements be precompiled?

+ or asynchronous (no) manner (yes)

  • -v yes|no - verbose mode. Should the test print out progress? (yes)

  • -c yes|no - precompiled statements. Should the SQL statements be precompiled? (yes)

  • SQLite can run in asynchronous or synchronous mode. This mode can be controlled by using "synchronous" parameter. It is set using the SQLite command:

    PRAGMA synchronous = ON|OFF

    Another tweakable feature is journal mode. It can be @@ -169,7 +156,14 @@ Possible command-line parameters: that file append operation is faster than modifications introduced in the middle of the file (as it often requires moving all data after modified point, effectively requiring writing large parts of - the whole file, not just changed fragment).

    3.4.1. memfile tweaks

    To modify default memfile_ubench parameters, command line + the whole file, not just changed fragment).

    There are no preparatory steps required for memfile benchmark. + The only requirement is the ability to create and write specified lease + file (dhcpd.leases in the current directory). The tests can be run + as follows: +

    > ./memfile_ubench

    + or +

    > ./memfile_ubench > results-memfile.txt

    +

    3.4.1. memfile tweaks

    To modify default memfile_ubench parameters, command line switches can be used. Currently supported switches are (default values specified in brackets):

    1. -f filename - name of the database file ("dhcpd.leases")

    2. -n num - number of iterations (100)

    3. -s yes|no - should the operations be performend in a synchronous (yes) @@ -177,7 +171,7 @@ Possible command-line parameters:

      memfile can run in asynchronous or synchronous mode. This mode can be controlled by using sync parameter. It uses fflush() and fsync() in synchronous mode to make sure that - data is not buffered and physically stored on disk.

    3.5. Basic performance measurements

    This section contains sample results for backend performance measurements, + data is not buffered and physically stored on disk.

    3.5. Basic performance measurements

    This section contains sample results for backend performance measurements, taken using microbenchmarks. Tests were conducted on reasonably powerful machine:

     CPU: Quad-core Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 logical cores)
    @@ -192,13 +186,12 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7
           asynchronous. As those modes offer radically different
           performances, synchronous mode was conducted for one
           thousand repetitions and asynchronous mode was conducted for
    -      one hundred thousand repetitions.

    Table 3.1. Synchronous results (basic)

    BackendOperationsCreate [s]Search [s]Update [s]Delete [s]Average [s]
    MySQL1,00031.604 0.11727.96427.69521.845
    SQLite1,00061.421 0.03359.47756.03444.241
    memfile1,00038.224 0.00138.04138.01728.571

    The following parameters were measured for asynchronous mode. - MySQL and SQLite were run with one hundred thousand repetitions. Memfile - was run for one million repetitions due to its much higher performance.

    Table 3.2. Asynchronous results (basic)

    BackendOperationsCreate [s]Search [s]Update [s]Delete [s]Average [s]
    MySQL100,00010.58510.38610.062 8.890 9.981
    SQLite100,000 3.710 3.159 2.865 2.439 3.044
    memfile1,000,000 1.300 0.039 1.307 1.278 0.981

    The presented performance results can be converted into operations per second metrics. + one hundred thousand repetitions.

    Table 3.1. Synchronous results (basic)

    BackendOperationsCreate [s]Search [s]Update [s]Delete [s]Average [s]
    MySQL1,00031.604 0.11727.96427.69521.845
    SQLite1,00061.421 0.03359.47756.03444.241
    memfile1,00038.224 0.00138.04138.01728.571

    The following parameters were measured for asynchronous mode. + MySQL and SQLite were run with one hundred thousand repetitions.

    Table 3.2. Asynchronous results (basic)

    BackendOperationsCreate [s]Search [s]Update [s]Delete [s]Average [s]
    MySQL100,00010.58510.38610.062 8.890 9.981
    SQLite100,000 3.710 3.159 2.865 2.439 3.044
    memfile100,000 1.300 0.039 1.307 1.278 0.981

    The presented performance results can be converted into operations per second metrics. It should be noted that due to large differences between various operations (sometimes over three orders of magnitude), it is difficult to create a simple, readable chart with - that data.

    Table 3.3. Estimated basic performance

    BackendCreate [oper/s]Search [oper/s]Update [oper/s]Delete [oper/s]Average [oper/s]
    MySQL (async)9447.479627.979938.0011248.3410065.45
    SQLite (async)26951.5931654.2934899.7040993.5933624.79
    memfile (async)76944.272542588.3576504.5478269.25693576.60
    MySQL (sync)31.648575.4535.7636.112169.74
    SQLite (sync)16.2820045.3716.8117.857524.08
    memfile (sync)26.161223990.2126.2926.30306017.24

    Basic performance measurements

    Graphical representation of the basic performance results - presented in table Table 3.3, “Estimated basic performance”.

    3.6. Optimized performance measurements

    This section contains sample results for backend performance measurements, + that data.

    Table 3.3. Estimated basic performance

    BackendCreate [oper/s]Search [oper/s]Update [oper/s]Delete [oper/s]Average [oper/s]
    MySQL (async)9447.479627.979938.0011248.3410065.45
    SQLite (async)26951.5931654.2934899.7040993.5933624.79
    memfile (async)76944.272542588.3576504.5478269.25693576.60
    MySQL (sync)31.648575.4535.7636.112169.74
    SQLite (sync)16.2820045.3716.8117.857524.08
    memfile (sync)26.161223990.2126.2926.30306017.24

    Graphical representation of the basic performance results + presented in table Table 3.3, “Estimated basic performance”.

    3.6. Optimized performance measurements

    This section contains sample results for backend performance measurements, taken using microbenchmarks. Tests were conducted on reasonably powerful machine:

     CPU: Quad-core Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 logical cores)
    @@ -213,13 +206,12 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7
           asynchronous. As those modes offer radically different
           performances, synchronous mode was conducted for one
           thousand repetitions and asynchronous mode was conducted for
    -      one hundred thousand repetitions.

    Table 3.4. Synchronous results (optimized)

    BackendOperationsCreate [s]Search [s]Update [s]Delete [s]Average [s]
    MySQL1,00027.887 0.10628.22327.69620.978
    SQLite1,00061.299 0.01559.64861.09845.626
    memfile1,00039.564 0.00139.54339.32629.608

    The following parameters were measured for asynchronous mode. - MySQL and SQLite were run with one hundred thousand repetitions. Memfile - was run for one million repetitions due to its much higher performance.

    Table 3.5. Asynchronous results (optimized)

    BackendOperationsCreate [s]Search [s]Update [s]Delete [s]Average [s]
    MySQL100,0008.5079.6987.7858.3268.579
    SQLite100,000 1.562 0.949 1.513 1.502 1.382
    memfile1,000,0001.3020.0381.3061.2630.977

    The presented performance results can be converted into operations per second metrics. + one hundred thousand repetitions.

    Table 3.4. Synchronous results (optimized)

    BackendOperationsCreate [s]Search [s]Update [s]Delete [s]Average [s]
    MySQL1,00027.887 0.10628.22327.69620.978
    SQLite1,00061.299 0.01559.64861.09845.626
    memfile1,00039.564 0.00139.54339.32629.608

    The following parameters were measured for asynchronous mode. + MySQL and SQLite were run with one hundred thousand repetitions.

    Table 3.5. Asynchronous results (optimized)

    BackendOperationsCreate [s]Search [s]Update [s]Delete [s]Average [s]
    MySQL100,0008.5079.6987.7858.3268.579
    SQLite100,000 1.562 0.949 1.513 1.502 1.382
    memfile100,0001.3020.0381.3061.2630.977

    The presented performance results can be converted into operations per second metrics. It should be noted that due to large differences between various operations (sometime over three orders of magnitude), it is difficult to create a simple, readable chart with the data.

    Table 3.6. Estimated optimized performance

    BackendCreate [oper/s]Search [oper/s]Update [oper/s]Delete [oper/s]Average [oper/s]
    MySQL (async)11754.8410311.3412845.3512010.2411730.44
    SQLite (async)64005.90105391.2966075.5166566.4375509.78
    memfile (async)76832.162636018.5676542.5079188.81717145.51
    MySQL (sync)35.869461.1035.4336.112392.12
    SQLite (sync)16.3167036.1116.7616.3716771.39
    memfile (sync)25.283460207.6125.2925.43865070.90

    Optimized performance measurements

    Graphical representation of the optimized performance - results presented in table Table 3.6, “Estimated optimized performance”.

    3.7. Conclusions

    + results presented in table Table 3.6, “Estimated optimized performance”.

    3.7. Conclusions

    Improvements gained by introducing support for precompiled statements in MySQL is somewhat disappointing - between 6 and 29%. On the other hand, the improvement in SQLite is @@ -238,12 +230,13 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 factor here is a disk access time. Even migrating to high performance 15,000 rpm disk is expected to only roughly double number of leases per second, compared to the current results. - The reason is that to write a file to disk, at least two writes + The reason is that to write a file to disk, at least two disk + sector writes are required: the new content and i-node modification of the file. The easiest way to boost synchronous performance is to switch to SSD disks. Memory-backed RAM disks are also a viable solution. However, care should be taken to properly engineer - backup strategy for RAM disks. + backup strategy for such RAM disks.

    While the custom made backend (memfile) provides the best perfomance, it carries over all the limitations existing in @@ -267,7 +260,7 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 message. The provided results should be considered as only rough estimates. They can also be used for relative comparisons between backends. -

    3.8. Possible further optimizations

    +

    3.8. Possible further optimizations

    For basic measurements the code was compiled with -g -O0 flags. For optimized measurements the benchmarking code was compiled with -Ofast (optimize for speed). In both cases, the diff --git a/tests/tools/dhcp-ubench/dhcp-perf-guide.xml b/tests/tools/dhcp-ubench/dhcp-perf-guide.xml index 6e3846731d..2bfe9bff90 100644 --- a/tests/tools/dhcp-ubench/dhcp-perf-guide.xml +++ b/tests/tools/dhcp-ubench/dhcp-perf-guide.xml @@ -120,12 +120,12 @@ that take advantage of a single database connection. - Those benchmarks are stored in tests/tools/dhcp-ubench - directory of the BIND 10 source tree. This directory contains simplified prototypes for + Those benchmarks are stored in tests/tools/dhcp-ubench directory of the + BIND 10 source tree. This directory contains simplified prototypes for the various database back-ends that are planned or considered as a - possibly for BIND10 DHCP. Athough trivial now, the benchmarks are - expected to evolve into useful tools that will allow users to - measure performance in their specific environment. + possibly for BIND10 DHCP. These benchmarks are expected to evolve into + useful tools that will allow users to measure performance in their + specific environment. @@ -138,11 +138,10 @@ - As the benchmarks require additional (sometimes heavy) dependencies, they are not - built by default. Actually, their build system is completely separate from that - of the rest of BIND 10. - It is anticipated that they will be eventually merged into the rest of BIND 10, but - that is a low priority for now. + As the benchmarks require additional (sometimes heavy) dependencies, they + are not built by default. Actually, their build system is completely + separate from that of the rest of BIND 10. It will be eventually merged + with the main BIND 10 build system. @@ -188,10 +187,10 @@ Before running tests, you need to initialize your database. You can use mysql.schema script for that purpose. - + WARNING: It will drop existing Kea database. Do not run this on your production server. - + Assuming your MySQL user is "kea", you can initialize your test database by: @@ -217,24 +216,7 @@ the same list of parameters, some of them are specific to a given backend. To get a list of supported parameters, run the benchmark with the "-h" option: - $ ./mysql_ubench -h -This is a benchmark designed to measure expected performance -of several backends. This particular version identifies itself -as following: -MySQL client version is 5.5.24 - -Possible command-line parameters: - -h - help (you are reading this) - -m hostname - specifies MySQL server to connect (MySQL backend only) - -u username - specifies MySQL user name (MySQL backend only) - -p password - specifies MySQL passwod (MySQL backend only) - -f name - database or filename (MySQL, SQLite and memfile) - -n integer - number of test repetitions (MySQL, SQLite and memfile) - -s yes|no - synchronous/asynchronous operation (MySQL, SQLite and memfile) - -v yes|no - verbose mode (MySQL, SQLite and memfile) - -c yes|no - should compiled statements be used (MySQL only) - - + $ ./mysql_ubench -h Synchronous operation requires database backend to @@ -248,6 +230,23 @@ Possible command-line parameters:

    MySQL tweaks + To modify the default mysql_ubench parameters, command line + switches can be used. The currently supported switches are + (default values specified in brackets): + + -f name - name of the database ("kea") + -m hostname - name of the database host ("localhost") + -u user - MySQL username ("root") + -p password - MySQL password ("secret") + -n num - number of iterations (100) + -s yes|no - should the operations be performed in a synchronous (yes) + or asynchronous (no) manner (yes) + -v yes|no - verbose mode. Should the test print out progress? (yes) + -c yes|no - precompiled statements. Should the SQL statements be precompiled? (yes) + + + + One parameter that has huge impact on performance is the choice of backend engine. You can get a list of engines of your MySQL implementation by using @@ -300,14 +299,14 @@ Possible command-line parameters: -s yes|no - should the operations be performed in a synchronous (yes) or asynchronous (no) manner (yes) -v yes|no - verbose mode. Should the test print out progress? (yes) - -c yes|no - precompiled statements. Should the SQL statements be precompiled? + -c yes|no - precompiled statements. Should the SQL statements be precompiled? (yes) SQLite can run in asynchronous or synchronous mode. This mode can be controlled by using "synchronous" parameter. It is set using the SQLite command: - + PRAGMA synchronous = ON|OFF Another tweakable feature is journal mode. It can be @@ -334,6 +333,15 @@ Possible command-line parameters: after modified point, effectively requiring writing large parts of the whole file, not just changed fragment). + There are no preparatory steps required for memfile benchmark. + The only requirement is the ability to create and write specified lease + file (dhcpd.leases in the current directory). The tests can be run + as follows: + > ./memfile_ubench + or + > ./memfile_ubench > results-memfile.txt + +
    memfile tweaks To modify default memfile_ubench parameters, command line @@ -435,8 +443,7 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 The following parameters were measured for asynchronous mode. - MySQL and SQLite were run with one hundred thousand repetitions. Memfile - was run for one million repetitions due to its much higher performance. + MySQL and SQLite were run with one hundred thousand repetitions. Asynchronous results (basic) @@ -482,7 +489,7 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 memfile - 1,000,000 + 100,000 1.300 0.039 1.307 @@ -577,13 +584,21 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7
    + + + + - + - - Basic performance measurements - Graphical representation of the basic performance results presented in table . @@ -672,8 +687,7 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 The following parameters were measured for asynchronous mode. - MySQL and SQLite were run with one hundred thousand repetitions. Memfile - was run for one million repetitions due to its much higher performance. + MySQL and SQLite were run with one hundred thousand repetitions. Asynchronous results (optimized) @@ -719,7 +733,7 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 memfile - 1,000,000 + 100,000 1.302 0.038 1.306 @@ -855,12 +869,13 @@ SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe7 factor here is a disk access time. Even migrating to high performance 15,000 rpm disk is expected to only roughly double number of leases per second, compared to the current results. - The reason is that to write a file to disk, at least two writes + The reason is that to write a file to disk, at least two disk + sector writes are required: the new content and i-node modification of the file. The easiest way to boost synchronous performance is to switch to SSD disks. Memory-backed RAM disks are also a viable solution. However, care should be taken to properly engineer - backup strategy for RAM disks. + backup strategy for such RAM disks.