Configurations/README.design

   1 Design document for the unified scheme data
   2 ===========================================
   3
   4 How are things connected?
   5 -------------------------
   6
   7 The unified scheme takes all its data from the build.info files seen
   8 throughout the source tree.  These files hold the minimum information
   9 needed to build end product files from diverse sources.  See the
  10 section on build.info files below.
  11
  12 From the information in build.info files, Configure builds up an
  13 information database as a hash table called %unified_info, which is
  14 stored in configdata.pm, found at the top of the build tree (which may
  15 or may not be the same as the source tree).
  16
  17 Configurations/common.tmpl uses the data from %unified_info to
  18 generate the rules for building end product files as well as
  19 intermediary files with the help of a few functions found in the
  20 build-file templates.  See the section on build-file templates further
  21 down for more information.
  22
  23 build.info files
  24 ----------------
  25
  26 As mentioned earlier, build.info files are meant to hold the minimum
  27 information needed to build output files, and therefore only (with a
  28 few possible exceptions [1]) have information about end products (such
  29 as scripts, library files and programs) and source files (such as C
  30 files, C header files, assembler files, etc).  Intermediate files such
  31 as object files are rarely directly referred to in build.info files (and
  32 when they are, it's always with the file name extension .o), they are
  33 inferred by Configure.  By the same rule of minimalism, end product
  34 file name extensions (such as .so, .a, .exe, etc) are never mentioned
  35 in build.info.  Their file name extensions will be inferred by the
  36 build-file templates, adapted for the platform they are meant for (see
  37 sections on %unified_info and build-file templates further down).
  38
  39 The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare
  40 end products.  There are variants for them with '_NO_INST' as suffix
  41 (PROGRAM_NO_INST etc) to specify end products that shouldn't get
  42 installed.
  43
  44 The variables SOURCE, DEPEND, INCLUDE and DEFINE are indexed by a
  45 produced file, and their values are the source used to produce that
  46 particular produced file, extra dependencies, include directories
  47 needed, or C macros to be defined.
  48
  49 All their values in all the build.info throughout the source tree are
  50 collected together and form a set of programs, libraries, engines and
  51 scripts to be produced, source files, dependencies, etc etc etc.
  52
  53 Let's have a pretend example, a very limited contraption of OpenSSL,
  54 composed of the program 'apps/openssl', the libraries 'libssl' and
  55 'libcrypto', an engine 'engines/ossltest' and their sources and
  56 dependencies.
  57
  58     # build.info
  59     LIBS=libcrypto libssl
  60     INCLUDE[libcrypto]=include
  61     INCLUDE[libssl]=include
  62     DEPEND[libssl]=libcrypto
  63
  64 This is the top directory build.info file, and it tells us that two
  65 libraries are to be built, the include directory 'include/' shall be
  66 used throughout when building anything that will end up in each
  67 library, and that the library 'libssl' depend on the library
  68 'libcrypto' to function properly.
  69
  70     # apps/build.info
  71     PROGRAMS=openssl
  72     SOURCE[openssl]=openssl.c
  73     INCLUDE[openssl]=.. ../include
  74     DEPEND[openssl]=../libssl
  75
  76 This is the build.info file in 'apps/', one may notice that all file
  77 paths mentioned are relative to the directory the build.info file is
  78 located in.  This one tells us that there's a program to be built
  79 called 'apps/openssl' (the file name extension will depend on the
  80 platform and is therefore not mentioned in the build.info file).  It's
  81 built from one source file, 'apps/openssl.c', and building it requires
  82 the use of '.' and 'include' include directories (both are declared
  83 from the point of view of the 'apps/' directory), and that the program
  84 depends on the library 'libssl' to function properly.
  85
  86     # crypto/build.info
  87     LIBS=../libcrypto
  88     SOURCE[../libcrypto]=aes.c evp.c cversion.c
  89     DEPEND[cversion.o]=buildinf.h
  90
  91     GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
  92     DEPEND[buildinf.h]=../Makefile
  93     DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm
  94
  95 This is the build.info file in 'crypto', and it tells us a little more
  96 about what's needed to produce 'libcrypto'.  LIBS is used again to
  97 declare that 'libcrypto' is to be produced.  This declaration is
  98 really unnecessary as it's already mentioned in the top build.info
  99 file, but can make the info file easier to understand.  This is to
 100 show that duplicate information isn't an issue.
 101
 102 This build.info file informs us that 'libcrypto' is built from a few
 103 source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'.
 104 It also shows us that building the object file inferred from
 105 'crypto/cversion.c' depends on 'crypto/buildinf.h'.  Finally, it
 106 also shows the possibility to declare how some files are generated
 107 using some script, in this case a perl script, and how such scripts
 108 can be declared to depend on other files, in this case a perl module.
 109
 110 Two things are worth an extra note:
 111
 112 'DEPEND[cversion.o]' mentions an object file.  DEPEND indexes is the
 113 only location where it's valid to mention them
 114
 115 Lines in 'BEGINRAW'..'ENDRAW' sections must always mention files as
 116 seen from the top directory, no exception.
 117
 118     # ssl/build.info
 119     LIBS=../libssl
 120     SOURCE[../libssl]=tls.c
 121
 122 This is the build.info file in 'ssl/', and it tells us that the
 123 library 'libssl' is built from the source file 'ssl/tls.c'.
 124
 125     # engines/build.info
 126     ENGINES=dasync
 127     SOURCE[dasync]=e_dasync.c
 128     DEPEND[dasync]=../libcrypto
 129     INCLUDE[dasync]=../include
 130
 131     ENGINES_NO_INST=ossltest
 132     SOURCE[ossltest]=e_ossltest.c
 133     DEPEND[ossltest]=../libcrypto.a
 134     INCLUDE[ossltest]=../include
 135
 136 This is the build.info file in 'engines/', telling us that two engines
 137 called 'engines/dasync' and 'engines/ossltest' shall be built, that
 138 dasync's source is 'engines/e_dasync.c' and ossltest's source is
 139 'engines/e_ossltest.c' and that the include directory 'include/' may
 140 be used when building anything that will be part of these engines.
 141 Also, both engines depend on the library 'libcrypto' to function
 142 properly.  ossltest is explicitly linked with the static variant of
 143 the library 'libcrypto'.  Finally, only dasync is being installed, as
 144 ossltest is only for internal testing.
 145
 146 When Configure digests these build.info files, the accumulated
 147 information comes down to this:
 148
 149     LIBS=libcrypto libssl
 150     SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
 151     DEPEND[crypto/cversion.o]=crypto/buildinf.h
 152     INCLUDE[libcrypto]=include
 153     SOURCE[libssl]=ssl/tls.c
 154     INCLUDE[libssl]=include
 155     DEPEND[libssl]=libcrypto
 156
 157     PROGRAMS=apps/openssl
 158     SOURCE[apps/openssl]=apps/openssl.c
 159     INCLUDE[apps/openssl]=. include
 160     DEPEND[apps/openssl]=libssl
 161
 162     ENGINES=engines/dasync
 163     SOURCE[engines/dasync]=engines/e_dasync.c
 164     DEPEND[engines/dasync]=libcrypto
 165     INCLUDE[engines/dasync]=include
 166
 167     ENGINES_NO_INST=engines/ossltest
 168     SOURCE[engines/ossltest]=engines/e_ossltest.c
 169     DEPEND[engines/ossltest]=libcrypto.a
 170     INCLUDE[engines/ossltest]=include
 171
 172     GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
 173     DEPEND[crypto/buildinf.h]=Makefile
 174     DEPEND[util/mkbuildinf.pl]=util/Foo.pm
 175
 176
 177 A few notes worth mentioning:
 178
 179 LIBS may be used to declare routine libraries only.
 180
 181 PROGRAMS may be used to declare programs only.
 182
 183 ENGINES may be used to declare engines only.
 184
 185 The indexes for SOURCE must only be end product files, such as
 186 libraries, programs or engines.  The values of SOURCE variables must
 187 only be source files (possibly generated).
 188
 189 INCLUDE and DEPEND shows a relationship between different files
 190 (usually produced files) or between files and directories, such as a
 191 program depending on a library, or between an object file and some
 192 extra source file.
 193
 194 When Configure processes the build.info files, it will take it as
 195 truth without question, and will therefore perform very few checks.
 196 If the build tree is separate from the source tree, it will assume
 197 that all built files and up in the build directory and that all source
 198 files are to be found in the source tree, if they can be found there.
 199 Configure will assume that source files that can't be found in the
 200 source tree (such as 'crypto/bildinf.h' in the example above) are
 201 generated and will be found in the build tree.
 202
 203
 204 The %unified_info database
 205 --------------------------
 206
 207 The information in all the build.info get digested by Configure and
 208 collected into the %unified_info database, divided into the following
 209 indexes:
 210
 211   depends   => a hash table containing 'file' => [ 'dependency' ... ]
 212                pairs.  These are directly inferred from the DEPEND
 213                variables in build.info files.
 214
 215   engines   => a list of engines.  These are directly inferred from
 216                the ENGINES variable in build.info files.
 217
 218   generate  => a hash table containing 'file' => [ 'generator' ... ]
 219                pairs.  These are directly inferred from the GENERATE
 220                variables in build.info files.
 221
 222   includes  => a hash table containing 'file' => [ 'include' ... ]
 223                pairs.  These are directly inferred from the INCLUDE
 224                variables in build.info files.
 225
 226   install   => a hash table containing 'type' => [ 'file' ... ] pairs.
 227                The types are 'programs', 'libraries', 'engines' and
 228                'scripts', and the array of files list the files of
 229                that type that should be installed.
 230
 231   libraries => a list of libraries.  These are directly inferred from
 232                the LIBS variable in build.info files.
 233
 234   programs  => a list of programs.  These are directly inferred from
 235                the PROGRAMS variable in build.info files.
 236
 237   rawlines  => a list of build-file lines.  These are a direct copy of
 238                the BEGINRAW..ENDRAW lines in build.info files.  Note:
 239                only the BEGINRAW..ENDRAW section for the current
 240                platform are copied, the rest are ignored.
 241
 242   scripts   => a list of scripts.  There are directly inferred from
 243                the SCRIPTS variable in build.info files.
 244
 245   sources   => a hash table containing 'file' => [ 'sourcefile' ... ]
 246                pairs.  These are indirectly inferred from the SOURCE
 247                variables in build.info files.  Object files are
 248                mentioned in this hash table, with source files from
 249                SOURCE variables, and AS source files for programs and
 250                libraries.
 251
 252   shared_sources =>
 253                a hash table just like 'sources', but only as source
 254                files (object files) for building shared libraries.
 255
 256 As an example, here is how the build.info files example from the
 257 section above would be digested into a %unified_info table:
 258
 259     our %unified_info = (
 260         "depends" =>
 261             {
 262                 "apps/openssl" =>
 263                     [
 264                         "libssl",
 265                     ],
 266                 "crypto/buildinf.h" =>
 267                     [
 268                         "Makefile",
 269                     ],
 270                 "crypto/cversion.o" =>
 271                     [
 272                         "crypto/buildinf.h",
 273                     ],
 274                 "engines/dasync" =>
 275                     [
 276                         "libcrypto",
 277                     ],
 278                 "engines/ossltest" =>
 279                     [
 280                         "libcrypto.a",
 281                     ],
 282                 "libssl" =>
 283                     [
 284                         "libcrypto",
 285                     ],
 286                 "util/mkbuildinf.pl" =>
 287                     [
 288                         "util/Foo.pm",
 289                     ],
 290             },
 291         "engines" =>
 292             [
 293                 "engines/dasync",
 294                 "engines/ossltest",
 295             ],
 296         "generate" =>
 297             {
 298                 "crypto/buildinf.h" =>
 299                     [
 300                         "util/mkbuildinf.pl",
 301                         "\"\$(CC)",
 302                         "\$(CFLAGS)\"",
 303                         "\"$(PLATFORM)\"",
 304                     ],
 305             },
 306         "includes" =>
 307             {
 308                 "apps/openssl" =>
 309                     [
 310                         ".",
 311                         "include",
 312                     ],
 313                 "engines/ossltest" =>
 314                     [
 315                         "include"
 316                     ],
 317                 "libcrypto" =>
 318                     [
 319                         "include",
 320                     ],
 321                 "libssl" =>
 322                     [
 323                         "include",
 324                     ],
 325                 "util/mkbuildinf.pl" =>
 326                     [
 327                         "util",
 328                     ],
 329             }
 330         "install" =>
 331             {
 332                 "engines" =>
 333                     [
 334                         "engines/dasync",
 335                     ],
 336                 "libraries" =>
 337                     [
 338                         "libcrypto",
 339                         "libssl",
 340                     ],
 341                 "programs" =>
 342                     [
 343                         "apps/openssl",
 344                     ],
 345            },
 346         "libraries" =>
 347             [
 348                 "libcrypto",
 349                 "libssl",
 350             ],
 351         "programs" =>
 352             [
 353                 "apps/openssl",
 354             ],
 355         "rawlines" =>
 356             [
 357             ],
 358         "sources" =>
 359             {
 360                 "apps/openssl" =>
 361                     [
 362                         "apps/openssl.o",
 363                     ],
 364                 "apps/openssl.o" =>
 365                     [
 366                         "apps/openssl.c",
 367                     ],
 368                 "crypto/aes.o" =>
 369                     [
 370                         "crypto/aes.c",
 371                     ],
 372                 "crypto/cversion.o" =>
 373                     [
 374                         "crypto/cversion.c",
 375                     ],
 376                 "crypto/evp.o" =>
 377                     [
 378                         "crypto/evp.c",
 379                     ],
 380                 "engines/e_dasync.o" =>
 381                     [
 382                         "engines/e_dasync.c",
 383                     ],
 384                 "engines/dasync" =>
 385                     [
 386                         "engines/e_dasync.o",
 387                     ],
 388                 "engines/e_ossltest.o" =>
 389                     [
 390                         "engines/e_ossltest.c",
 391                     ],
 392                 "engines/ossltest" =>
 393                     [
 394                         "engines/e_ossltest.o",
 395                     ],
 396                 "libcrypto" =>
 397                     [
 398                         "crypto/aes.c",
 399                         "crypto/cversion.c",
 400                         "crypto/evp.c",
 401                     ],
 402                 "libssl" =>
 403                     [
 404                         "ssl/tls.c",
 405                     ],
 406                 "ssl/tls.o" =>
 407                     [
 408                         "ssl/tls.c",
 409                     ],
 410             },
 411     );
 412
 413 As can be seen, everything in %unified_info is fairly simple suggest
 414 of information.  Still, it tells us that to build all programs, we
 415 must build 'apps/openssl', and to build the latter, we will need to
 416 build all its sources ('apps/openssl.o' in this case) and all the
 417 other things it depends on (such as 'libssl').  All those dependencies
 418 need to be built as well, using the same logic, so to build 'libssl',
 419 we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the
 420 latter...
 421
 422
 423 Build-file templates
 424 --------------------
 425
 426 Build-file templates are essentially build-files (such as Makefile on
 427 Unix) with perl code fragments mixed in.  Those perl code fragment
 428 will generate all the configuration dependent data, including all the
 429 rules needed to build end product files and intermediary files alike.
 430 At a minimum, there must be a perl code fragment that defines a set of
 431 functions that are used to generates specific build-file rules, to
 432 build static libraries from object files, to build shared libraries
 433 from static libraries, to programs from object files and libraries,
 434 etc.
 435
 436     generatesrc - function that produces build file lines to generate
 437                   a source file from some input.
 438
 439                   It's called like this:
 440
 441                         generatesrc(src => "PATH/TO/tobegenerated",
 442                                     generator => [ "generatingfile", ... ]
 443                                     generator_incs => [ "INCL/PATH", ... ]
 444                                     generator_deps => [ "dep1", ... ]
 445                                     incs => [ "INCL/PATH", ... ],
 446                                     deps => [ "dep1", ... ],
 447                                     intent => one of "libs", "dso", "bin" );
 448
 449                   'src' has the name of the file to be generated.
 450                   'generator' is the command or part of command to
 451                   generate the file, of which the first item is
 452                   expected to be the file to generate from.
 453                   generatesrc() is expected to analyse and figure out
 454                   exactly how to apply that file and how to capture
 455                   the result.  'generator_incs' and 'generator_deps'
 456                   are include directories and files that the generator
 457                   file itself depends on.  'incs' and 'deps' are
 458                   include directories and files that are used if $(CC)
 459                   is used as an intermediary step when generating the
 460                   end product (the file indicated by 'src').  'intent'
 461                   indicates what the generated file is going to be
 462                   used for.
 463
 464     src2obj     - function that produces build file lines to build an
 465                   object file from source files and associated data.
 466
 467                   It's called like this:
 468
 469                         src2obj(obj => "PATH/TO/objectfile",
 470                                 srcs => [ "PATH/TO/sourcefile", ... ],
 471                                 deps => [ "dep1", ... ],
 472                                 incs => [ "INCL/PATH", ... ]
 473                                 intent => one of "lib", "dso", "bin" );
 474
 475                   'obj' has the intended object file with '.o'
 476                   extension, src2obj() is expected to change it to
 477                   something more suitable for the platform.
 478                   'srcs' has the list of source files to build the
 479                   object file, with the first item being the source
 480                   file that directly corresponds to the object file.
 481                   'deps' is a list of explicit dependencies.  'incs'
 482                   is a list of include file directories.  Finally,
 483                   'intent' indicates what this object file is going
 484                   to be used for.
 485
 486     obj2lib     - function that produces build file lines to build a
 487                   static library file ("libfoo.a" in Unix terms) from
 488                   object files.
 489
 490                   called like this:
 491
 492                         obj2lib(lib => "PATH/TO/libfile",
 493                                 objs => [ "PATH/TO/objectfile", ... ]);
 494
 495                   'lib' has the intended library file name *without*
 496                   extension, obj2lib is expected to add that.  'objs'
 497                   has the list of object files to build this library.
 498
 499     libobj2shlib - backward compatibility function that's used the
 500                   same way as obj2shlib (described next), and was
 501                   expected to build the shared library from the
 502                   corresponding static library when that was suitable.
 503                   NOTE: building a shared library from a static
 504                   library is now DEPRECATED, as they no longer share
 505                   object files.  Attempting to do this will fail.
 506
 507     obj2shlib   - function that produces build file lines to build a
 508                   shareable object library file ("libfoo.so" in Unix
 509                   terms) from the corresponding object files.
 510
 511                   called like this:
 512
 513                         obj2shlib(shlib => "PATH/TO/shlibfile",
 514                                   lib => "PATH/TO/libfile",
 515                                   objs => [ "PATH/TO/objectfile", ... ],
 516                                   deps => [ "PATH/TO/otherlibfile", ... ]);
 517
 518                   'lib' has the base (static) library file name
 519                   *without* extension.  This is useful in case
 520                   supporting files are needed (such as import
 521                   libraries on Windows).
 522                   'shlib' has the corresponding shared library name
 523                   *without* extension.  'deps' has the list of other
 524                   libraries (also *without* extension) this library
 525                   needs to be linked with.  'objs' has the list of
 526                   object files to build this library.
 527
 528     obj2dso     - function that produces build file lines to build a
 529                   dynamic shared object file from object files.
 530
 531                   called like this:
 532
 533                         obj2dso(lib => "PATH/TO/libfile",
 534                                 objs => [ "PATH/TO/objectfile", ... ],
 535                                 deps => [ "PATH/TO/otherlibfile",
 536                                 ... ]);
 537
 538                   This is almost the same as obj2shlib, but the
 539                   intent is to build a shareable library that can be
 540                   loaded in runtime (a "plugin"...).
 541
 542     obj2bin     - function that produces build file lines to build an
 543                   executable file from object files.
 544
 545                   called like this:
 546
 547                         obj2bin(bin => "PATH/TO/binfile",
 548                                 objs => [ "PATH/TO/objectfile", ... ],
 549                                 deps => [ "PATH/TO/libfile", ... ]);
 550
 551                   'bin' has the intended executable file name
 552                   *without* extension, obj2bin is expected to add
 553                   that.  'objs' has the list of object files to build
 554                   this library.  'deps' has the list of library files
 555                   (also *without* extension) that the programs needs
 556                   to be linked with.
 557
 558     in2script   - function that produces build file lines to build a
 559                   script file from some input.
 560
 561                   called like this:
 562
 563                         in2script(script => "PATH/TO/scriptfile",
 564                                   sources => [ "PATH/TO/infile", ... ]);
 565
 566                   'script' has the intended script file name.
 567                   'sources' has the list of source files to build the
 568                   resulting script from.
 569
 570 Along with the build-file templates is the driving engine
 571 Configurations/common.tmpl, which looks through all the information in
 572 %unified_info and generates all the rulesets to build libraries,
 573 programs and all intermediate files, using the rule generating
 574 functions defined in the build-file template.
 575
 576 As an example with the smaller build.info set we've seen as an
 577 example, producing the rules to build 'libcrypto' would result in the
 578 following calls:
 579
 580     # Note: obj2shlib will only be called if shared libraries are
 581     # to be produced.
 582     # Note 2: obj2shlib must convert the '.o' extension to whatever
 583     # is suitable on the local platform.
 584     obj2shlib(shlib => "libcrypto",
 585               objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ],
 586               deps => [  ]);
 587
 588     obj2lib(lib => "libcrypto"
 589             objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ]);
 590
 591     src2obj(obj => "crypto/aes.o"
 592             srcs => [ "crypto/aes.c" ],
 593             deps => [ ],
 594             incs => [ "include" ],
 595             intent => "lib");
 596
 597     src2obj(obj => "crypto/evp.o"
 598             srcs => [ "crypto/evp.c" ],
 599             deps => [ ],
 600             incs => [ "include" ],
 601             intent => "lib");
 602
 603     src2obj(obj => "crypto/cversion.o"
 604             srcs => [ "crypto/cversion.c" ],
 605             deps => [ "crypto/buildinf.h" ],
 606             incs => [ "include" ],
 607             intent => "lib");
 608
 609     generatesrc(src => "crypto/buildinf.h",
 610                 generator => [ "util/mkbuildinf.pl", "\"$(CC)",
 611                                "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],
 612                 generator_incs => [ "util" ],
 613                 generator_deps => [ "util/Foo.pm" ],
 614                 incs => [ ],
 615                 deps => [ ],
 616                 intent => "lib");
 617
 618 The returned strings from all those calls are then concatenated
 619 together and written to the resulting build-file.