gcc/jit/docs/internals/index.rst

   1 .. Copyright (C) 2014-2018 Free Software Foundation, Inc.
   2    Originally contributed by David Malcolm <dmalcolm@redhat.com>
   3
   4    This is free software: you can redistribute it and/or modify it
   5    under the terms of the GNU General Public License as published by
   6    the Free Software Foundation, either version 3 of the License, or
   7    (at your option) any later version.
   8
   9    This program is distributed in the hope that it will be useful, but
  10    WITHOUT ANY WARRANTY; without even the implied warranty of
  11    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  12    General Public License for more details.
  13
  14    You should have received a copy of the GNU General Public License
  15    along with this program.  If not, see
  16    <http://www.gnu.org/licenses/>.
  17
  18 Internals
  19 =========
  20
  21 Working on the JIT library
  22 --------------------------
  23 Having checked out the source code (to "src"), you can configure and build
  24 the JIT library like this:
  25
  26 .. code-block:: bash
  27
  28   mkdir build
  29   mkdir install
  30   PREFIX=$(pwd)/install
  31   cd build
  32   ../src/configure \
  33      --enable-host-shared \
  34      --enable-languages=jit,c++ \
  35      --disable-bootstrap \
  36      --enable-checking=release \
  37      --prefix=$PREFIX
  38   nice make -j4 # altering the "4" to however many cores you have
  39
  40 This should build a libgccjit.so within jit/build/gcc:
  41
  42 .. code-block:: console
  43
  44  [build] $ file gcc/libgccjit.so*
  45  gcc/libgccjit.so:       symbolic link to `libgccjit.so.0'
  46  gcc/libgccjit.so.0:     symbolic link to `libgccjit.so.0.0.1'
  47  gcc/libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
  48
  49 Here's what those configuration options mean:
  50
  51 .. option:: --enable-host-shared
  52
  53   Configuring with this option means that the compiler is built as
  54   position-independent code, which incurs a slight performance hit,
  55   but it necessary for a shared library.
  56
  57 .. option:: --enable-languages=jit,c++
  58
  59   This specifies which frontends to build.  The JIT library looks like
  60   a frontend to the rest of the code.
  61
  62   The C++ portion of the JIT test suite requires the C++ frontend to be
  63   enabled at configure-time, or you may see errors like this when
  64   running the test suite:
  65
  66   .. code-block:: console
  67
  68     xgcc: error: /home/david/jit/src/gcc/testsuite/jit.dg/test-quadratic.cc: C++ compiler not installed on this system
  69     c++: error trying to exec 'cc1plus': execvp: No such file or directory
  70
  71 .. option:: --disable-bootstrap
  72
  73   For hacking on the "jit" subdirectory, performing a full
  74   bootstrap can be overkill, since it's unused by a bootstrap.  However,
  75   when submitting patches, you should remove this option, to ensure that
  76   the compiler can still bootstrap itself.
  77
  78 .. option:: --enable-checking=release
  79
  80   The compile can perform extensive self-checking as it runs, useful when
  81   debugging, but slowing things down.
  82
  83   For maximum speed, configure with ``--enable-checking=release`` to
  84   disable this self-checking.
  85
  86 Running the test suite
  87 ----------------------
  88
  89 .. code-block:: console
  90
  91   [build] $ cd gcc
  92   [gcc] $ make check-jit RUNTESTFLAGS="-v -v -v"
  93
  94 A summary of the tests can then be seen in:
  95
  96 .. code-block:: console
  97
  98   jit/build/gcc/testsuite/jit/jit.sum
  99
 100 and detailed logs in:
 101
 102 .. code-block:: console
 103
 104   jit/build/gcc/testsuite/jit/jit.log
 105
 106 The test executables are normally deleted after each test is run.  For
 107 debugging, they can be preserved by setting :envvar:`PRESERVE_EXECUTABLES`
 108 in the environment.  If so, they can then be seen as:
 109
 110 .. code-block:: console
 111
 112   jit/build/gcc/testsuite/jit/*.exe
 113
 114 which can be run independently.
 115
 116 You can compile and run individual tests by passing "jit.exp=TESTNAME" to RUNTESTFLAGS e.g.:
 117
 118 .. code-block:: console
 119
 120    [gcc] $ PRESERVE_EXECUTABLES= \
 121              make check-jit \
 122                RUNTESTFLAGS="-v -v -v jit.exp=test-factorial.c"
 123
 124 and once a test has been compiled, you can debug it directly:
 125
 126 .. code-block:: console
 127
 128    [gcc] $ PATH=.:$PATH \
 129            LD_LIBRARY_PATH=. \
 130            LIBRARY_PATH=. \
 131              gdb --args \
 132                testsuite/jit/test-factorial.c.exe
 133
 134 Running under valgrind
 135 **********************
 136
 137 The jit testsuite detects if :envvar:`RUN_UNDER_VALGRIND` is present in the
 138 environment (with any value).  If it is present, it runs the test client
 139 code under `valgrind <http://valgrind.org>`_,
 140 specifcally, the default
 141 `memcheck <http://valgrind.org/docs/manual/mc-manual.html>`_
 142 tool with
 143 `--leak-check=full
 144 <http://valgrind.org/docs/manual/mc-manual.html#opt.leak-check>`_.
 145
 146 It automatically parses the output from valgrind, injecting XFAIL results if
 147 any issues are found, or PASS results if the output is clean.  The output
 148 is saved to ``TESTNAME.exe.valgrind.txt``.
 149
 150 For example, the following invocation verbosely runs the testcase
 151 ``test-sum-of-squares.c`` under valgrind, showing an issue:
 152
 153 .. code-block:: console
 154
 155   $ RUN_UNDER_VALGRIND= \
 156       make check-jit \
 157         RUNTESTFLAGS="-v -v -v jit.exp=test-sum-of-squares.c"
 158
 159   (...verbose log contains detailed valgrind errors, if any...)
 160
 161                   === jit Summary ===
 162
 163   # of expected passes            28
 164   # of expected failures          2
 165
 166   $ less testsuite/jit/jit.sum
 167   (...other results...)
 168   XFAIL: jit.dg/test-sum-of-squares.c: test-sum-of-squares.c.exe.valgrind.txt: definitely lost: 8 bytes in 1 blocks
 169   XFAIL: jit.dg/test-sum-of-squares.c: test-sum-of-squares.c.exe.valgrind.txt: unsuppressed errors: 1
 170   (...other results...)
 171
 172   $ less testsuite/jit/test-sum-of-squares.c.exe.valgrind.txt
 173   (...shows full valgrind report for this test case...)
 174
 175 When running under valgrind, it's best to have configured gcc with
 176 :option:`--enable-valgrind-annotations`, which automatically suppresses
 177 various known false positives.
 178
 179 Environment variables
 180 ---------------------
 181 When running client code against a locally-built libgccjit, three
 182 environment variables need to be set up:
 183
 184 .. envvar:: LD_LIBRARY_PATH
 185
 186    `libgccjit.so` is dynamically linked into client code, so if running
 187    against a locally-built library, ``LD_LIBRARY_PATH`` needs to be set
 188    up appropriately.  The library can be found within the "gcc"
 189    subdirectory of the build tree:
 190
 191   .. code-block:: console
 192
 193     $ file libgccjit.so*
 194     libgccjit.so:       symbolic link to `libgccjit.so.0'
 195     libgccjit.so.0:     symbolic link to `libgccjit.so.0.0.1'
 196     libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped
 197
 198 .. envvar:: PATH
 199
 200   The library uses a driver executable for converting from .s assembler
 201   files to .so shared libraries.  Specifically, it looks for a name
 202   expanded from
 203   ``${target_noncanonical}-gcc-${gcc_BASEVER}${exeext}``
 204   such as ``x86_64-unknown-linux-gnu-gcc-5.0.0``.
 205
 206   Hence ``PATH`` needs to include a directory where the library can
 207   locate this executable.
 208
 209   The executable is normally installed to the installation bindir
 210   (e.g. /usr/bin), but a copy is also created within the "gcc"
 211   subdirectory of the build tree for running the testsuite, and for ease
 212   of development.
 213
 214 .. envvar:: LIBRARY_PATH
 215
 216   The driver executable invokes the linker, and the latter needs to locate
 217   support libraries needed by the generated code, or you will see errors
 218   like:
 219
 220   .. code-block:: console
 221
 222     ld: cannot find crtbeginS.o: No such file or directory
 223     ld: cannot find -lgcc
 224     ld: cannot find -lgcc_s
 225
 226   Hence if running directly from a locally-built copy (without installing),
 227   ``LIBRARY_PATH`` needs to contain the "gcc" subdirectory of the build
 228   tree.
 229
 230 For example, to run a binary that uses the library against a non-installed
 231 build of the library in LIBGCCJIT_BUILD_DIR you need an invocation of the
 232 client code like this, to preprend the dir to each of the environment
 233 variables:
 234
 235 .. code-block:: console
 236
 237   $ LD_LIBRARY_PATH=$(LIBGCCJIT_BUILD_DIR):$(LD_LIBRARY_PATH) \
 238     PATH=$(LIBGCCJIT_BUILD_DIR):$(PATH) \
 239     LIBRARY_PATH=$(LIBGCCJIT_BUILD_DIR):$(LIBRARY_PATH) \
 240       ./jit-hello-world
 241   hello world
 242
 243 Packaging notes
 244 ---------------
 245 The configure-time option :option:`--enable-host-shared` is needed when
 246 building the jit in order to get position-independent code.  This will
 247 slow down the regular compiler by a few percent.  Hence when packaging gcc
 248 with libgccjit, please configure and build twice:
 249
 250   * once without :option:`--enable-host-shared` for most languages, and
 251
 252   * once with :option:`--enable-host-shared` for the jit
 253
 254 For example:
 255
 256 .. code-block:: bash
 257
 258   # Configure and build with --enable-host-shared
 259   # for the jit:
 260   mkdir configuration-for-jit
 261   pushd configuration-for-jit
 262   $(SRCDIR)/configure \
 263     --enable-host-shared \
 264     --enable-languages=jit \
 265     --prefix=$(DESTDIR)
 266   make
 267   popd
 268
 269   # Configure and build *without* --enable-host-shared
 270   # for maximum speed:
 271   mkdir standard-configuration
 272   pushd standard-configuration
 273   $(SRCDIR)/configure \
 274     --enable-languages=all \
 275     --prefix=$(DESTDIR)
 276   make
 277   popd
 278
 279   # Both of the above are configured to install to $(DESTDIR)
 280   # Install the configuration with --enable-host-shared first
 281   # *then* the one without, so that the faster build
 282   # of "cc1" et al overwrites the slower build.
 283   pushd configuration-for-jit
 284   make install
 285   popd
 286
 287   pushd standard-configuration
 288   make install
 289   popd
 290
 291 Overview of code structure
 292 --------------------------
 293
 294 The library is implemented in C++.  The source files have the ``.c``
 295 extension for legacy reasons.
 296
 297 * ``libgccjit.c`` implements the API entrypoints.  It performs error
 298   checking, then calls into classes of the gcc::jit::recording namespace
 299   within ``jit-recording.c`` and ``jit-recording.h``.
 300
 301 * The gcc::jit::recording classes (within ``jit-recording.c`` and
 302   ``jit-recording.h``) record the API calls that are made:
 303
 304    .. literalinclude:: ../../jit-common.h
 305     :start-after: /* Recording types.  */
 306     :end-before: /* End of recording types. */
 307     :language: c++
 308
 309 * When the context is compiled, the gcc::jit::playback classes (within
 310   ``jit-playback.c`` and ``jit-playback.h``) replay the API calls
 311   within langhook:parse_file:
 312
 313    .. literalinclude:: ../../jit-common.h
 314     :start-after: /* Playback types.  */
 315     :end-before: /* End of playback types. */
 316     :language: c++
 317
 318    .. literalinclude:: ../../notes.txt
 319     :lines: 1-
 320
 321 Here is a high-level summary from ``jit-common.h``:
 322
 323 .. include:: ../../jit-common.h
 324   :start-after: This comment is included by the docs.
 325   :end-before: End of comment for inclusion in the docs.  */
 326
 327 .. _example-of-log-file:
 328
 329 Another way to understand the structure of the code is to enable logging,
 330 via :c:func:`gcc_jit_context_set_logfile`.  Here is an example of a log
 331 generated via this call:
 332
 333 .. literalinclude:: test-hello-world.exe.log.txt
 334     :lines: 1-
 335
 336 Design notes
 337 ------------
 338 It should not be possible for client code to cause an internal compiler
 339 error.  If this *does* happen, the root cause should be isolated (perhaps
 340 using :c:func:`gcc_jit_context_dump_reproducer_to_file`) and the cause
 341 should be rejected via additional checking.  The checking ideally should
 342 be within the libgccjit API entrypoints in libgccjit.c, since this is as
 343 close as possible to the error; failing that, a good place is within
 344 ``recording::context::validate ()`` in jit-recording.c.
 345
 346 Submitting patches
 347 ------------------
 348 Please read the contribution guidelines for gcc at
 349 https://gcc.gnu.org/contribute.html.
 350
 351 Patches for the jit should be sent to both the
 352 gcc-patches@gcc.gnu.org and jit@gcc.gnu.org mailing lists,
 353 with "jit" and "PATCH" in the Subject line.
 354
 355 You don't need to do a full bootstrap for code that just touches the
 356 ``jit`` and ``testsuite/jit.dg`` subdirectories.  However, please run
 357 ``make check-jit`` before submitting the patch, and mention the results
 358 in your email (along with the host triple that the tests were run on).
 359
 360 A good patch should contain the information listed in the
 361 gcc contribution guide linked to above; for a ``jit`` patch, the patch
 362 shold contain:
 363
 364   * the code itself (for example, a new API entrypoint will typically
 365     touch ``libgccjit.h`` and ``.c``, along with support code in
 366     ``jit-recording.[ch]`` and ``jit-playback.[ch]`` as appropriate)
 367
 368   * test coverage
 369
 370   * documentation for the C API
 371
 372   * documentation for the C++ API
 373
 374 A patch that adds new API entrypoints should also contain:
 375
 376   * a feature macro in ``libgccjit.h`` so that client code that doesn't
 377     use a "configure" mechanism can still easily detect the presence of
 378     the entrypoint.  See e.g. ``LIBGCCJIT_HAVE_SWITCH_STATEMENTS`` (for
 379     a category of entrypoints) and
 380     ``LIBGCCJIT_HAVE_gcc_jit_context_set_bool_allow_unreachable_blocks``
 381     (for an individual entrypoint).
 382
 383   * a new ABI tag containing the new symbols (in ``libgccjit.map``), so
 384     that we can detect client code that uses them
 385
 386   * Support for :c:func:`gcc_jit_context_dump_reproducer_to_file`.  Most
 387     jit testcases attempt to dump their contexts to a .c file; ``jit.exp``
 388     then sanity-checks the generated c by compiling them (though
 389     not running them).   A new API entrypoint
 390     needs to "know" how to write itself back out to C (by implementing
 391     ``gcc::jit::recording::memento::write_reproducer`` for the appropriate
 392     ``memento`` subclass).
 393
 394   * C++ bindings for the new entrypoints (see ``libgccjit++.h``); ideally
 395     with test coverage, though the C++ API test coverage is admittedly
 396     spotty at the moment
 397
 398   * documentation for the new C entrypoints
 399
 400   * documentation for the new C++ entrypoints
 401
 402   * documentation for the new ABI tag (see ``topics/compatibility.rst``).
 403
 404 Depending on the patch you can either extend an existing test case, or
 405 add a new test case.  If you add an entirely new testcase: ``jit.exp``
 406 expects jit testcases to begin with ``test-``, or ``test-error-`` (for a
 407 testcase that generates an error on a :c:type:`gcc_jit_context`).
 408
 409 Every new testcase that doesn't generate errors should also touch
 410 ``gcc/testsuite/jit.dg/all-non-failing-tests.h``:
 411
 412   * Testcases that don't generate errors should ideally be added to the
 413     ``testcases`` array in that file; this means that, in addition
 414     to being run standalone, they also get run within
 415     ``test-combination.c`` (which runs all successful tests inside one
 416     big :c:type:`gcc_jit_context`), and ``test-threads.c`` (which runs all
 417     successful tests in one process, each one running in a different
 418     thread on a different :c:type:`gcc_jit_context`).
 419
 420     .. note::
 421
 422        Given that exported functions within a :c:type:`gcc_jit_context`
 423        must have unique names, and most testcases are run within
 424        ``test-combination.c``, this means that every jit-compiled test
 425        function typically needs a name that's unique across the entire
 426        test suite.
 427
 428   * Testcases that aren't to be added to the ``testcases`` array should
 429     instead add a comment to the file clarifying why they're not in that
 430     array. See the file for examples.
 431
 432 Typically a patch that touches the .rst documentation will also need the
 433 texinfo to be regenerated.  You can do this with
 434 `Sphinx 1.0 <http://sphinx-doc.org/>`_ or later by
 435 running ``make texinfo`` within ``SRCDIR/gcc/jit/docs``.   Don't do this
 436 within the patch sent to the mailing list; it can often be relatively
 437 large and inconsequential (e.g. anchor renumbering), rather like generated
 438 "configure" changes from configure.ac.  You can regenerate it when
 439 committing to svn.