<para>
Because the OpenEmbedded build system uses
- "<ulink url='&YOCTO_DOCS_REF_URL;#checksums'>signatures</ulink>",
+ "<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>signatures</ulink>",
which are unique to a given build, the build system
knows when to rebuild packages.
All the inputs into a given task are represented by a
<para>
For more information on shared state, see the
- "<ulink url='&YOCTO_DOCS_REF_URL;#shared-state-cache'>Shared State Cache</ulink>"
- section in the Yocto Project Reference Manual.
+ "<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>Shared State Cache</ulink>"
+ section in the Yocto Project Overview Manual.
</para>
</note>
</section>
</note>
</section>
+
+
+
+ <section id="shared-state-cache">
+ <title>Shared State Cache</title>
+
+ <para>
+ By design, the OpenEmbedded build system builds everything from
+ scratch unless BitBake can determine that parts do not need to be
+ rebuilt.
+ Fundamentally, building from scratch is attractive as it means all
+ parts are built fresh and there is no possibility of stale data
+ causing problems.
+ When developers hit problems, they typically default back to
+ building from scratch so they know the state of things from the
+ start.
+ </para>
+
+ <para>
+ Building an image from scratch is both an advantage and a
+ disadvantage to the process.
+ As mentioned in the previous paragraph, building from scratch
+ ensures that everything is current and starts from a known state.
+ However, building from scratch also takes much longer as it
+ generally means rebuilding things that do not necessarily need
+ to be rebuilt.
+ </para>
+
+ <para>
+ The Yocto Project implements shared state code that supports
+ incremental builds.
+ The implementation of the shared state code answers the following
+ questions that were fundamental roadblocks within the OpenEmbedded
+ incremental build support system:
+ <itemizedlist>
+ <listitem><para>
+ What pieces of the system have changed and what pieces have
+ not changed?
+ </para></listitem>
+ <listitem><para>
+ How are changed pieces of software removed and replaced?
+ </para></listitem>
+ <listitem><para>
+ How are pre-built components that do not need to be rebuilt
+ from scratch used when they are available?
+ </para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ For the first question, the build system detects changes in the
+ "inputs" to a given task by creating a checksum (or signature) of
+ the task's inputs.
+ If the checksum changes, the system assumes the inputs have changed
+ and the task needs to be rerun.
+ For the second question, the shared state (sstate) code tracks
+ which tasks add which output to the build process.
+ This means the output from a given task can be removed, upgraded
+ or otherwise manipulated.
+ The third question is partly addressed by the solution for the
+ second question assuming the build system can fetch the sstate
+ objects from remote locations and install them if they are deemed
+ to be valid.
+ <note>
+ The OpenEmbedded build system does not maintain
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
+ information as part of the shared state packages.
+ Consequently, considerations exist that affect maintaining
+ shared state feeds.
+ For information on how the OpenEmbedded build system
+ works with packages and can track incrementing
+ <filename>PR</filename> information, see the
+ "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
+ section in the Yocto Project Development Tasks Manual.
+ </note>
+ </para>
+
+ <para>
+ The rest of this section goes into detail about the overall
+ incremental build architecture, the checksums (signatures), shared
+ state, and some tips and tricks.
+ </para>
+
+ <section id='overall-architecture'>
+ <title>Overall Architecture</title>
+
+ <para>
+ When determining what parts of the system need to be built,
+ BitBake works on a per-task basis rather than a per-recipe
+ basis.
+ You might wonder why using a per-task basis is preferred over
+ a per-recipe basis.
+ To help explain, consider having the IPK packaging backend
+ enabled and then switching to DEB.
+ In this case, the
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
+ and
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
+ task outputs are still valid.
+ However, with a per-recipe approach, the build would not
+ include the <filename>.deb</filename> files.
+ Consequently, you would have to invalidate the whole build and
+ rerun it.
+ Rerunning everything is not the best solution.
+ Also, in this case, the core must be "taught" much about
+ specific tasks.
+ This methodology does not scale well and does not allow users
+ to easily add new tasks in layers or as external recipes
+ without touching the packaged-staging core.
+ </para>
+ </section>
+
+ <section id='overview-checksums'>
+ <title>Checksums (Signatures)</title>
+
+ <para>
+ The shared state code uses a checksum, which is a unique
+ signature of a task's inputs, to determine if a task needs to
+ be run again.
+ Because it is a change in a task's inputs that triggers a
+ rerun, the process needs to detect all the inputs to a given
+ task.
+ For shell tasks, this turns out to be fairly easy because
+ the build process generates a "run" shell script for each task
+ and it is possible to create a checksum that gives you a good
+ idea of when the task's data changes.
+ </para>
+
+ <para>
+ To complicate the problem, there are things that should not be
+ included in the checksum.
+ First, there is the actual specific build path of a given
+ task - the
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
+ It does not matter if the work directory changes because it
+ should not affect the output for target packages.
+ Also, the build process has the objective of making native
+ or cross packages relocatable.
+ <note>
+ Both native and cross packages run on the build host.
+ However, cross packages generate output for the target
+ architecture.
+ </note>
+ The checksum therefore needs to exclude
+ <filename>WORKDIR</filename>.
+ The simplistic approach for excluding the work directory is to
+ set <filename>WORKDIR</filename> to some fixed value and
+ create the checksum for the "run" script.
+ </para>
+
+ <para>
+ Another problem results from the "run" scripts containing
+ functions that might or might not get called.
+ The incremental build solution contains code that figures out
+ dependencies between shell functions.
+ This code is used to prune the "run" scripts down to the
+ minimum set, thereby alleviating this problem and making the
+ "run" scripts much more readable as a bonus.
+ </para>
+
+ <para>
+ So far we have solutions for shell scripts.
+ What about Python tasks?
+ The same approach applies even though these tasks are more
+ difficult.
+ The process needs to figure out what variables a Python
+ function accesses and what functions it calls.
+ Again, the incremental build solution contains code that first
+ figures out the variable and function dependencies, and then
+ creates a checksum for the data used as the input to the task.
+ </para>
+
+ <para>
+ Like the <filename>WORKDIR</filename> case, situations exist
+ where dependencies should be ignored.
+ For these cases, you can instruct the build process to
+ ignore a dependency by using a line like the following:
+ <literallayout class='monospaced'>
+ PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
+ </literallayout>
+ This example ensures that the
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
+ variable does not depend on the value of
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
+ even if it does reference it.
+ </para>
+
+ <para>
+ Equally, there are cases where we need to add dependencies
+ BitBake is not able to find.
+ You can accomplish this by using a line like the following:
+ <literallayout class='monospaced'>
+ PACKAGE_ARCHS[vardeps] = "MACHINE"
+ </literallayout>
+ This example explicitly adds the <filename>MACHINE</filename>
+ variable as a dependency for
+ <filename>PACKAGE_ARCHS</filename>.
+ </para>
+
+ <para>
+ Consider a case with in-line Python, for example, where
+ BitBake is not able to figure out dependencies.
+ When running in debug mode (i.e. using
+ <filename>-DDD</filename>), BitBake produces output when it
+ discovers something for which it cannot figure out dependencies.
+ The Yocto Project team has currently not managed to cover
+ those dependencies in detail and is aware of the need to fix
+ this situation.
+ </para>
+
+ <para>
+ Thus far, this section has limited discussion to the direct
+ inputs into a task.
+ Information based on direct inputs is referred to as the
+ "basehash" in the code.
+ However, there is still the question of a task's indirect
+ inputs - the things that were already built and present in the
+ <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
+ The checksum (or signature) for a particular task needs to add
+ the hashes of all the tasks on which the particular task
+ depends.
+ Choosing which dependencies to add is a policy decision.
+ However, the effect is to generate a master checksum that
+ combines the basehash and the hashes of the task's
+ dependencies.
+ </para>
+
+ <para>
+ At the code level, there are a variety of ways both the
+ basehash and the dependent task hashes can be influenced.
+ Within the BitBake configuration file, we can give BitBake
+ some extra information to help it construct the basehash.
+ The following statement effectively results in a list of
+ global variable dependency excludes - variables never
+ included in any checksum:
+ <literallayout class='monospaced'>
+ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
+ SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
+ USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
+ PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
+ CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
+ </literallayout>
+ The previous example excludes
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
+ since that variable is actually constructed as a path within
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
+ which is on the whitelist.
+ </para>
+
+ <para>
+ The rules for deciding which hashes of dependent tasks to
+ include through dependency chains are more complex and are
+ generally accomplished with a Python function.
+ The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
+ two examples of this and also illustrates how you can insert
+ your own policy into the system if so desired.
+ This file defines the two basic signature generators
+ <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
+ uses: "OEBasic" and "OEBasicHash".
+ By default, there is a dummy "noop" signature handler enabled
+ in BitBake.
+ This means that behavior is unchanged from previous versions.
+ OE-Core uses the "OEBasicHash" signature handler by default
+ through this setting in the <filename>bitbake.conf</filename>
+ file:
+ <literallayout class='monospaced'>
+ BB_SIGNATURE_HANDLER ?= "OEBasicHash"
+ </literallayout>
+ The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
+ is the same as the "OEBasic" version but adds the task hash to
+ the stamp files.
+ This results in any
+ <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
+ change that changes the task hash, automatically
+ causing the task to be run again.
+ This removes the need to bump
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
+ values, and changes to Metadata automatically ripple across
+ the build.
+ </para>
+
+ <para>
+ It is also worth noting that the end result of these
+ signature generators is to make some dependency and hash
+ information available to the build.
+ This information includes:
+ <itemizedlist>
+ <listitem><para>
+ <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
+ The base hashes for each task in the recipe.
+ </para></listitem>
+ <listitem><para>
+ <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+ The base hashes for each dependent task.
+ </para></listitem>
+ <listitem><para>
+ <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+ The task dependencies for each task.
+ </para></listitem>
+ <listitem><para>
+ <filename>BB_TASKHASH</filename>:
+ The hash of the currently running task.
+ </para></listitem>
+ </itemizedlist>
+ </para>
+ </section>
+
+ <section id='shared-state'>
+ <title>Shared State</title>
+
+ <para>
+ Checksums and dependencies, as discussed in the previous
+ section, solve half the problem of supporting a shared state.
+ The other part of the problem is being able to use checksum
+ information during the build and being able to reuse or rebuild
+ specific components.
+ </para>
+
+ <para>
+ The
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
+ class is a relatively generic implementation of how to
+ "capture" a snapshot of a given task.
+ The idea is that the build process does not care about the
+ source of a task's output.
+ Output could be freshly built or it could be downloaded and
+ unpacked from somewhere - the build process does not need to
+ worry about its origin.
+ </para>
+
+ <para>
+ There are two types of output, one is just about creating a
+ directory in
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
+ A good example is the output of either
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
+ or
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
+ The other type of output occurs when a set of data is merged
+ into a shared directory tree such as the sysroot.
+ </para>
+
+ <para>
+ The Yocto Project team has tried to keep the details of the
+ implementation hidden in <filename>sstate</filename> class.
+ From a user's perspective, adding shared state wrapping to a task
+ is as simple as this
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
+ example taken from the
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
+ class:
+ <literallayout class='monospaced'>
+ DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
+ SSTATETASKS += "do_deploy"
+ do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
+ do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+
+ python do_deploy_setscene () {
+ sstate_setscene(d)
+ }
+ addtask do_deploy_setscene
+ do_deploy[dirs] = "${DEPLOYDIR} ${B}"
+ </literallayout>
+ The following list explains the previous example:
+ <itemizedlist>
+ <listitem><para>
+ Adding "do_deploy" to <filename>SSTATETASKS</filename>
+ adds some required sstate-related processing, which is
+ implemented in the
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
+ class, to before and after the
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
+ task.
+ </para></listitem>
+ <listitem><para>
+ The
+ <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
+ declares that <filename>do_deploy</filename> places its
+ output in <filename>${DEPLOYDIR}</filename> when run
+ normally (i.e. when not using the sstate cache).
+ This output becomes the input to the shared state cache.
+ </para></listitem>
+ <listitem><para>
+ The
+ <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
+ line causes the contents of the shared state cache to be
+ copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
+ <note>
+ If <filename>do_deploy</filename> is not already in
+ the shared state cache or if its input checksum
+ (signature) has changed from when the output was
+ cached, the task will be run to populate the shared
+ state cache, after which the contents of the shared
+ state cache is copied to
+ <filename>${DEPLOY_DIR_IMAGE}</filename>.
+ If <filename>do_deploy</filename> is in the shared
+ state cache and its signature indicates that the
+ cached output is still valid (i.e. if no
+ relevant task inputs have changed), then the
+ contents of the shared state cache will be copied
+ directly to
+ <filename>${DEPLOY_DIR_IMAGE}</filename> by the
+ <filename>do_deploy_setscene</filename> task
+ instead, skipping the
+ <filename>do_deploy</filename> task.
+ </note>
+ </para></listitem>
+ <listitem><para>
+ The following task definition is glue logic needed to
+ make the previous settings effective:
+ <literallayout class='monospaced'>
+ python do_deploy_setscene () {
+ sstate_setscene(d)
+ }
+ addtask do_deploy_setscene
+ </literallayout>
+ <filename>sstate_setscene()</filename> takes the flags
+ above as input and accelerates the
+ <filename>do_deploy</filename> task through the
+ shared state cache if possible.
+ If the task was accelerated,
+ <filename>sstate_setscene()</filename> returns True.
+ Otherwise, it returns False, and the normal
+ <filename>do_deploy</filename> task runs.
+ For more information, see the
+ "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
+ section in the BitBake User Manual.
+ </para></listitem>
+ <listitem><para>
+ The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
+ line creates <filename>${DEPLOYDIR}</filename> and
+ <filename>${B}</filename> before the
+ <filename>do_deploy</filename> task runs, and also sets
+ the current working directory of
+ <filename>do_deploy</filename> to
+ <filename>${B}</filename>.
+ For more information, see the
+ "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
+ section in the BitBake User Manual.
+ <note>
+ In cases where
+ <filename>sstate-inputdirs</filename> and
+ <filename>sstate-outputdirs</filename> would be the
+ same, you can use
+ <filename>sstate-plaindirs</filename>.
+ For example, to preserve the
+ <filename>${PKGD}</filename> and
+ <filename>${PKGDEST}</filename> output from the
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
+ task, use the following:
+ <literallayout class='monospaced'>
+ do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
+ </literallayout>
+ </note>
+ </para></listitem>
+ <listitem><para>
+ <filename>sstate-inputdirs</filename> and
+ <filename>sstate-outputdirs</filename> can also be used
+ with multiple directories.
+ For example, the following declares
+ <filename>PKGDESTWORK</filename> and
+ <filename>SHLIBWORK</filename> as shared state
+ input directories, which populates the shared state
+ cache, and <filename>PKGDATA_DIR</filename> and
+ <filename>SHLIBSDIR</filename> as the corresponding
+ shared state output directories:
+ <literallayout class='monospaced'>
+ do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
+ do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
+ </literallayout>
+ </para></listitem>
+ <listitem><para>
+ These methods also include the ability to take a
+ lockfile when manipulating shared state directory
+ structures, for cases where file additions or removals
+ are sensitive:
+ <literallayout class='monospaced'>
+ do_package[sstate-lockfile] = "${PACKAGELOCK}"
+ </literallayout>
+ </para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Behind the scenes, the shared state code works by looking in
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
+ and
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
+ for shared state files.
+ Here is an example:
+ <literallayout class='monospaced'>
+ SSTATE_MIRRORS ?= "\
+ file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
+ file://.* file:///some/local/dir/sstate/PATH"
+ </literallayout>
+ <note>
+ The shared state directory
+ (<filename>SSTATE_DIR</filename>) is organized into
+ two-character subdirectories, where the subdirectory
+ names are based on the first two characters of the hash.
+ If the shared state directory structure for a mirror has the
+ same structure as <filename>SSTATE_DIR</filename>, you must
+ specify "PATH" as part of the URI to enable the build system
+ to map to the appropriate subdirectory.
+ </note>
+ </para>
+
+ <para>
+ The shared state package validity can be detected just by
+ looking at the filename since the filename contains the task
+ checksum (or signature) as described earlier in this section.
+ If a valid shared state package is found, the build process
+ downloads it and uses it to accelerate the task.
+ </para>
+
+ <para>
+ The build processes use the <filename>*_setscene</filename>
+ tasks for the task acceleration phase.
+ BitBake goes through this phase before the main execution
+ code and tries to accelerate any tasks for which it can find
+ shared state packages.
+ If a shared state package for a task is available, the
+ shared state package is used.
+ This means the task and any tasks on which it is dependent
+ are not executed.
+ </para>
+
+ <para>
+ As a real world example, the aim is when building an IPK-based
+ image, only the
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
+ tasks would have their shared state packages fetched and
+ extracted.
+ Since the sysroot is not used, it would never get extracted.
+ This is another reason why a task-based approach is preferred
+ over a recipe-based approach, which would have to install the
+ output from every task.
+ </para>
+ </section>
+
+ <section id='tips-and-tricks'>
+ <title>Tips and Tricks</title>
+
+ <para>
+ The code in the build system that supports incremental builds
+ is not simple code.
+ This section presents some tips and tricks that help you work
+ around issues related to shared state code.
+ </para>
+
+ <section id='overview-debugging'>
+ <title>Debugging</title>
+
+ <para>
+ Seeing what metadata went into creating the input signature
+ of a shared state (sstate) task can be a useful debugging
+ aid.
+ This information is available in signature information
+ (<filename>siginfo</filename>) files in
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>.
+ For information on how to view and interpret information in
+ <filename>siginfo</filename> files, see the
+ "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
+ section in the Yocto Project Reference Manual.
+ </para>
+ </section>
+
+ <section id='invalidating-shared-state'>
+ <title>Invalidating Shared State</title>
+
+ <para>
+ The OpenEmbedded build system uses checksums and shared
+ state cache to avoid unnecessarily rebuilding tasks.
+ Collectively, this scheme is known as "shared state code."
+ </para>
+
+ <para>
+ As with all schemes, this one has some drawbacks.
+ It is possible that you could make implicit changes to your
+ code that the checksum calculations do not take into
+ account.
+ These implicit changes affect a task's output but do not
+ trigger the shared state code into rebuilding a recipe.
+ Consider an example during which a tool changes its output.
+ Assume that the output of <filename>rpmdeps</filename>
+ changes.
+ The result of the change should be that all the
+ <filename>package</filename> and
+ <filename>package_write_rpm</filename> shared state cache
+ items become invalid.
+ However, because the change to the output is
+ external to the code and therefore implicit,
+ the associated shared state cache items do not become
+ invalidated.
+ In this case, the build process uses the cached items
+ rather than running the task again.
+ Obviously, these types of implicit changes can cause
+ problems.
+ </para>
+
+ <para>
+ To avoid these problems during the build, you need to
+ understand the effects of any changes you make.
+ Realize that changes you make directly to a function
+ are automatically factored into the checksum calculation.
+ Thus, these explicit changes invalidate the associated
+ area of shared state cache.
+ However, you need to be aware of any implicit changes that
+ are not obvious changes to the code and could affect
+ the output of a given task.
+ </para>
+
+ <para>
+ When you identify an implicit change, you can easily
+ take steps to invalidate the cache and force the tasks
+ to run.
+ The steps you can take are as simple as changing a
+ function's comments in the source code.
+ For example, to invalidate package shared state files,
+ change the comment statements of
+ <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
+ or the comments of one of the functions it calls.
+ Even though the change is purely cosmetic, it causes the
+ checksum to be recalculated and forces the OpenEmbedded
+ build system to run the task again.
+ <note>
+ For an example of a commit that makes a cosmetic
+ change to invalidate shared state, see this
+ <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
+ </note>
+ </para>
+ </section>
+ </section>
+ </section>
+
<section id='x32'>
<title>x32 psABI</title>
<ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
variable, and the end of the name consists of the task's name
and current
- <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>.
+ <link linkend='overview-checksums'>input checksum</link>.
<note>
This naming scheme assumes that
<ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
If doing so results in unnecessary rebuilds of tasks, you can whitelist the
variable so that the shared state code ignores the dependency when it creates
checksums.
- For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename>
- example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section.
+ For information on this process, see the
+ <filename>BB_HASHBASE_WHITELIST</filename> example in the
+ "<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>Checksums (Signatures)</ulink>"
+ section in the Yocto Project Overview Manual.
</note>
</section>
<para>
For more information on sstate, see the
- "<link linkend='shared-state-cache'>Shared State Cache</link>"
- section.
+ "<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>Shared State Cache</ulink>"
+ section in the Yocto Project Overview Manual.
</para>
</section>
<para>
Running this task does not remove the
- <link linkend='shared-state-cache'>sstate</link>) cache
- files.
+ <ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>
+ cache files.
Consequently, if no changes have been made and the recipe is
rebuilt after cleaning, output files are simply restored from the
sstate cache.
<para>
Removes all output files, shared state
- (<link linkend='shared-state-cache'>sstate</link>) cache, and
- downloaded source files for a target (i.e. the contents of
+ (<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
+ cache, and downloaded source files for a target (i.e. the contents
+ of
<link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>).
Essentially, the <filename>do_cleanall</filename> task is
identical to the
<para>
Removes all output files and shared state
- (<link linkend='shared-state-cache'>sstate</link>)
+ (<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
cache for a target.
Essentially, the <filename>do_cleansstate</filename> task is
identical to the
<link linkend='ref-tasks-clean'><filename>do_clean</filename></link>
task with the added removal of shared state
- (<link linkend='shared-state-cache'>sstate</link>) cache.
+ (<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
+ cache.
</para>
<para>
<filename>PR</filename> to know when to rebuild a
recipe.
The build system uses the task
- <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksums</ulink>
+ <ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>input checksums</ulink>
along with the
<link linkend='structure-build-tmp-stamps'>stamp</link>
and
- <link linkend='shared-state-cache'>shared state cache</link>
+ <ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>shared state cache</ulink>
mechanisms.
</note>
The <filename>PR</filename> variable primarily becomes
x32, Wayland support, and Licenses.
</para>
-<section id="shared-state-cache">
- <title>Shared State Cache</title>
-
- <para>
- By design, the OpenEmbedded build system builds everything from scratch unless
- BitBake can determine that parts do not need to be rebuilt.
- Fundamentally, building from scratch is attractive as it means all parts are
- built fresh and there is no possibility of stale data causing problems.
- When developers hit problems, they typically default back to building from scratch
- so they know the state of things from the start.
- </para>
-
- <para>
- Building an image from scratch is both an advantage and a disadvantage to the process.
- As mentioned in the previous paragraph, building from scratch ensures that
- everything is current and starts from a known state.
- However, building from scratch also takes much longer as it generally means
- rebuilding things that do not necessarily need to be rebuilt.
- </para>
-
- <para>
- The Yocto Project implements shared state code that supports incremental builds.
- The implementation of the shared state code answers the following questions that
- were fundamental roadblocks within the OpenEmbedded incremental build support system:
- <itemizedlist>
- <listitem><para>What pieces of the system have changed and what pieces have
- not changed?</para></listitem>
- <listitem><para>How are changed pieces of software removed and replaced?</para></listitem>
- <listitem><para>How are pre-built components that do not need to be rebuilt from scratch
- used when they are available?</para></listitem>
- </itemizedlist>
- </para>
-
- <para>
- For the first question, the build system detects changes in the "inputs" to a given task by
- creating a checksum (or signature) of the task's inputs.
- If the checksum changes, the system assumes the inputs have changed and the task needs to be
- rerun.
- For the second question, the shared state (sstate) code tracks which tasks add which output
- to the build process.
- This means the output from a given task can be removed, upgraded or otherwise manipulated.
- The third question is partly addressed by the solution for the second question
- assuming the build system can fetch the sstate objects from remote locations and
- install them if they are deemed to be valid.
- </para>
-
- <note>
- The OpenEmbedded build system does not maintain
- <link linkend='var-PR'><filename>PR</filename></link> information
- as part of the shared state packages.
- Consequently, considerations exist that affect maintaining shared
- state feeds.
- For information on how the OpenEmbedded build system
- works with packages and can
- track incrementing <filename>PR</filename> information, see the
- "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
- section in the Yocto Project Development Tasks Manual.
- </note>
-
- <para>
- The rest of this section goes into detail about the overall incremental build
- architecture, the checksums (signatures), shared state, and some tips and tricks.
- </para>
-
- <section id='overall-architecture'>
- <title>Overall Architecture</title>
-
- <para>
- When determining what parts of the system need to be built, BitBake
- works on a per-task basis rather than a per-recipe basis.
- You might wonder why using a per-task basis is preferred over a per-recipe basis.
- To help explain, consider having the IPK packaging backend enabled and then switching to DEB.
- In this case, the
- <link linkend='ref-tasks-install'><filename>do_install</filename></link>
- and
- <link linkend='ref-tasks-package'><filename>do_package</filename></link>
- task outputs are still valid.
- However, with a per-recipe approach, the build would not include the
- <filename>.deb</filename> files.
- Consequently, you would have to invalidate the whole build and rerun it.
- Rerunning everything is not the best solution.
- Also, in this case, the core must be "taught" much about specific tasks.
- This methodology does not scale well and does not allow users to easily add new tasks
- in layers or as external recipes without touching the packaged-staging core.
- </para>
- </section>
-
- <section id='checksums'>
- <title>Checksums (Signatures)</title>
-
- <para>
- The shared state code uses a checksum, which is a unique signature of a task's
- inputs, to determine if a task needs to be run again.
- Because it is a change in a task's inputs that triggers a rerun, the process
- needs to detect all the inputs to a given task.
- For shell tasks, this turns out to be fairly easy because
- the build process generates a "run" shell script for each task and
- it is possible to create a checksum that gives you a good idea of when
- the task's data changes.
- </para>
-
- <para>
- To complicate the problem, there are things that should not be
- included in the checksum.
- First, there is the actual specific build path of a given task -
- the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
- It does not matter if the work directory changes because it should
- not affect the output for target packages.
- Also, the build process has the objective of making native
- or cross packages relocatable.
- <note>
- Both native and cross packages run on the build host.
- However, cross packages generate output for the target
- architecture.
- </note>
- The checksum therefore needs to exclude
- <filename>WORKDIR</filename>.
- The simplistic approach for excluding the work directory is to set
- <filename>WORKDIR</filename> to some fixed value and create the
- checksum for the "run" script.
- </para>
-
- <para>
- Another problem results from the "run" scripts containing functions that
- might or might not get called.
- The incremental build solution contains code that figures out dependencies
- between shell functions.
- This code is used to prune the "run" scripts down to the minimum set,
- thereby alleviating this problem and making the "run" scripts much more
- readable as a bonus.
- </para>
-
- <para>
- So far we have solutions for shell scripts.
- What about Python tasks?
- The same approach applies even though these tasks are more difficult.
- The process needs to figure out what variables a Python function accesses
- and what functions it calls.
- Again, the incremental build solution contains code that first figures out
- the variable and function dependencies, and then creates a checksum for the data
- used as the input to the task.
- </para>
-
- <para>
- Like the <filename>WORKDIR</filename> case, situations exist where dependencies
- should be ignored.
- For these cases, you can instruct the build process to ignore a dependency
- by using a line like the following:
- <literallayout class='monospaced'>
- PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
- </literallayout>
- This example ensures that the
- <link linkend='var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></link>
- variable does not
- depend on the value of
- <link linkend='var-MACHINE'><filename>MACHINE</filename></link>,
- even if it does reference it.
- </para>
-
- <para>
- Equally, there are cases where we need to add dependencies BitBake is not able to find.
- You can accomplish this by using a line like the following:
- <literallayout class='monospaced'>
- PACKAGE_ARCHS[vardeps] = "MACHINE"
- </literallayout>
- This example explicitly adds the <filename>MACHINE</filename> variable as a
- dependency for <filename>PACKAGE_ARCHS</filename>.
- </para>
-
- <para>
- Consider a case with in-line Python, for example, where BitBake is not
- able to figure out dependencies.
- When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
- produces output when it discovers something for which it cannot figure out
- dependencies.
- The Yocto Project team has currently not managed to cover those dependencies
- in detail and is aware of the need to fix this situation.
- </para>
-
- <para>
- Thus far, this section has limited discussion to the direct inputs into a task.
- Information based on direct inputs is referred to as the "basehash" in the
- code.
- However, there is still the question of a task's indirect inputs - the
- things that were already built and present in the
- <link linkend='build-directory'>Build Directory</link>.
- The checksum (or signature) for a particular task needs to add the hashes
- of all the tasks on which the particular task depends.
- Choosing which dependencies to add is a policy decision.
- However, the effect is to generate a master checksum that combines the basehash
- and the hashes of the task's dependencies.
- </para>
-
- <para>
- At the code level, there are a variety of ways both the basehash and the
- dependent task hashes can be influenced.
- Within the BitBake configuration file, we can give BitBake some extra information
- to help it construct the basehash.
- The following statement effectively results in a list of global variable
- dependency excludes - variables never included in any checksum:
- <literallayout class='monospaced'>
- BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
- SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
- USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
- PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
- CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
- </literallayout>
- The previous example excludes
- <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
- since that variable is actually constructed as a path within
- <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on
- the whitelist.
- </para>
-
- <para>
- The rules for deciding which hashes of dependent tasks to include through
- dependency chains are more complex and are generally accomplished with a
- Python function.
- The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
- of this and also illustrates how you can insert your own policy into the system
- if so desired.
- This file defines the two basic signature generators
- <link linkend='oe-core'>OE-Core</link> uses: "OEBasic" and
- "OEBasicHash".
- By default, there is a dummy "noop" signature handler enabled in BitBake.
- This means that behavior is unchanged from previous versions.
- OE-Core uses the "OEBasicHash" signature handler by default
- through this setting in the <filename>bitbake.conf</filename> file:
- <literallayout class='monospaced'>
- BB_SIGNATURE_HANDLER ?= "OEBasicHash"
- </literallayout>
- The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
- "OEBasic" version but adds the task hash to the stamp files.
- This results in any
- <link linkend='metadata'>Metadata</link>
- change that changes the task hash, automatically
- causing the task to be run again.
- This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link>
- values, and changes to Metadata automatically ripple across the build.
- </para>
-
- <para>
- It is also worth noting that the end result of these signature generators is to
- make some dependency and hash information available to the build.
- This information includes:
- <itemizedlist>
- <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
- The base hashes for each task in the recipe.
- </para></listitem>
- <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
- The base hashes for each dependent task.
- </para></listitem>
- <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
- The task dependencies for each task.
- </para></listitem>
- <listitem><para><filename>BB_TASKHASH</filename>:
- The hash of the currently running task.
- </para></listitem>
- </itemizedlist>
- </para>
- </section>
-
- <section id='shared-state'>
- <title>Shared State</title>
-
- <para>
- Checksums and dependencies, as discussed in the previous section, solve half the
- problem of supporting a shared state.
- The other part of the problem is being able to use checksum information during the build
- and being able to reuse or rebuild specific components.
- </para>
-
- <para>
- The
- <link linkend='ref-classes-sstate'><filename>sstate</filename></link>
- class is a relatively generic implementation of how to "capture"
- a snapshot of a given task.
- The idea is that the build process does not care about the source of a task's output.
- Output could be freshly built or it could be downloaded and unpacked from
- somewhere - the build process does not need to worry about its origin.
- </para>
-
- <para>
- There are two types of output, one is just about creating a directory
- in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
- A good example is the output of either
- <link linkend='ref-tasks-install'><filename>do_install</filename></link>
- or
- <link linkend='ref-tasks-package'><filename>do_package</filename></link>.
- The other type of output occurs when a set of data is merged into a shared directory
- tree such as the sysroot.
- </para>
-
- <para>
- The Yocto Project team has tried to keep the details of the
- implementation hidden in <filename>sstate</filename> class.
- From a user's perspective, adding shared state wrapping to a task
- is as simple as this
- <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>
- example taken from the
- <link linkend='ref-classes-deploy'><filename>deploy</filename></link>
- class:
- <literallayout class='monospaced'>
- DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
- SSTATETASKS += "do_deploy"
- do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
- do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
-
- python do_deploy_setscene () {
- sstate_setscene(d)
- }
- addtask do_deploy_setscene
- do_deploy[dirs] = "${DEPLOYDIR} ${B}"
- </literallayout>
- The following list explains the previous example:
- <itemizedlist>
- <listitem><para>
- Adding "do_deploy" to <filename>SSTATETASKS</filename>
- adds some required sstate-related processing, which is
- implemented in the
- <link linkend='ref-classes-sstate'><filename>sstate</filename></link>
- class, to before and after the
- <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>
- task.
- </para></listitem>
- <listitem><para>
- The
- <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
- declares that <filename>do_deploy</filename> places its
- output in <filename>${DEPLOYDIR}</filename> when run
- normally (i.e. when not using the sstate cache).
- This output becomes the input to the shared state cache.
- </para></listitem>
- <listitem><para>
- The
- <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
- line causes the contents of the shared state cache to be
- copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
- <note>
- If <filename>do_deploy</filename> is not already in
- the shared state cache or if its input checksum
- (signature) has changed from when the output was
- cached, the task will be run to populate the shared
- state cache, after which the contents of the shared
- state cache is copied to
- <filename>${DEPLOY_DIR_IMAGE}</filename>.
- If <filename>do_deploy</filename> is in the shared
- state cache and its signature indicates that the
- cached output is still valid (i.e. if no
- relevant task inputs have changed), then the contents
- of the shared state cache will be copied directly to
- <filename>${DEPLOY_DIR_IMAGE}</filename> by the
- <filename>do_deploy_setscene</filename> task instead,
- skipping the <filename>do_deploy</filename> task.
- </note>
- </para></listitem>
- <listitem><para>
- The following task definition is glue logic needed to make
- the previous settings effective:
- <literallayout class='monospaced'>
- python do_deploy_setscene () {
- sstate_setscene(d)
- }
- addtask do_deploy_setscene
- </literallayout>
- <filename>sstate_setscene()</filename> takes the flags
- above as input and accelerates the
- <filename>do_deploy</filename> task through the
- shared state cache if possible.
- If the task was accelerated,
- <filename>sstate_setscene()</filename> returns True.
- Otherwise, it returns False, and the normal
- <filename>do_deploy</filename> task runs.
- For more information, see the
- "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
- section in the BitBake User Manual.
- </para></listitem>
- <listitem><para>
- The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
- line creates <filename>${DEPLOYDIR}</filename> and
- <filename>${B}</filename> before the
- <filename>do_deploy</filename> task runs, and also sets
- the current working directory of
- <filename>do_deploy</filename> to
- <filename>${B}</filename>.
- For more information, see the
- "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
- section in the BitBake User Manual.
- <note>
- In cases where
- <filename>sstate-inputdirs</filename> and
- <filename>sstate-outputdirs</filename> would be the
- same, you can use
- <filename>sstate-plaindirs</filename>.
- For example, to preserve the
- <filename>${PKGD}</filename> and
- <filename>${PKGDEST}</filename> output from the
- <link linkend='ref-tasks-package'><filename>do_package</filename></link>
- task, use the following:
- <literallayout class='monospaced'>
- do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
- </literallayout>
- </note>
- </para></listitem>
- <listitem><para>
- <filename>sstate-inputdirs</filename> and
- <filename>sstate-outputdirs</filename> can also be used
- with multiple directories.
- For example, the following declares
- <filename>PKGDESTWORK</filename> and
- <filename>SHLIBWORK</filename> as shared state
- input directories, which populates the shared state
- cache, and <filename>PKGDATA_DIR</filename> and
- <filename>SHLIBSDIR</filename> as the corresponding
- shared state output directories:
- <literallayout class='monospaced'>
- do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
- do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
- </literallayout>
- </para></listitem>
- <listitem><para>
- These methods also include the ability to take a lockfile
- when manipulating shared state directory structures,
- for cases where file additions or removals are sensitive:
- <literallayout class='monospaced'>
- do_package[sstate-lockfile] = "${PACKAGELOCK}"
- </literallayout>
- </para></listitem>
- </itemizedlist>
- </para>
-
-<!--
- <para>
- In this example, we add some extra flags to the task, a name field ("deploy"), an
- input directory where the task sends data, and the output
- directory where the data from the task should eventually be copied.
- We also add a <filename>_setscene</filename> variant of the task and add the task
- name to the <filename>SSTATETASKS</filename> list.
- </para>
-
- <para>
- If you have a directory whose contents you need to preserve, you can do this with
- a line like the following:
- <literallayout class='monospaced'>
- do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
- </literallayout>
- This method, as well as the following example, also works for multiple directories.
- <literallayout class='monospaced'>
- do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
- do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
- do_package[sstate-lockfile] = "${PACKAGELOCK}"
- </literallayout>
- These methods also include the ability to take a lockfile when manipulating
- shared state directory structures since some cases are sensitive to file
- additions or removals.
- </para>
--->
-
- <para>
- Behind the scenes, the shared state code works by looking in
- <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> and
- <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link>
- for shared state files.
- Here is an example:
- <literallayout class='monospaced'>
- SSTATE_MIRRORS ?= "\
- file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
- file://.* file:///some/local/dir/sstate/PATH"
- </literallayout>
- <note>
- The shared state directory (<filename>SSTATE_DIR</filename>) is
- organized into two-character subdirectories, where the subdirectory
- names are based on the first two characters of the hash.
- If the shared state directory structure for a mirror has the
- same structure as <filename>SSTATE_DIR</filename>, you must
- specify "PATH" as part of the URI to enable the build system
- to map to the appropriate subdirectory.
- </note>
- </para>
-
- <para>
- The shared state package validity can be detected just by looking at the
- filename since the filename contains the task checksum (or signature) as
- described earlier in this section.
- If a valid shared state package is found, the build process downloads it
- and uses it to accelerate the task.
- </para>
-
- <para>
- The build processes use the <filename>*_setscene</filename> tasks
- for the task acceleration phase.
- BitBake goes through this phase before the main execution code and tries
- to accelerate any tasks for which it can find shared state packages.
- If a shared state package for a task is available, the shared state
- package is used.
- This means the task and any tasks on which it is dependent are not
- executed.
- </para>
-
- <para>
- As a real world example, the aim is when building an IPK-based image,
- only the
- <link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link>
- tasks would have their
- shared state packages fetched and extracted.
- Since the sysroot is not used, it would never get extracted.
- This is another reason why a task-based approach is preferred over a
- recipe-based approach, which would have to install the output from every task.
- </para>
- </section>
-
- <section id='tips-and-tricks'>
- <title>Tips and Tricks</title>
-
- <para>
- The code in the build system that supports incremental builds is not
- simple code.
- This section presents some tips and tricks that help you work around
- issues related to shared state code.
- </para>
-
- <section id='debugging'>
- <title>Debugging</title>
-
- <para>
- Seeing what metadata went into creating the input signature
- of a shared state (sstate) task can be a useful debugging aid.
- This information is available in signature information
- (<filename>siginfo</filename>) files in
- <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>.
- For information on how to view and interpret information in
- <filename>siginfo</filename> files, see the
- "<link linkend='usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>"
- section.
- </para>
- </section>
-
- <section id='invalidating-shared-state'>
- <title>Invalidating Shared State</title>
-
- <para>
- The OpenEmbedded build system uses checksums and shared state
- cache to avoid unnecessarily rebuilding tasks.
- Collectively, this scheme is known as "shared state code."
- </para>
-
- <para>
- As with all schemes, this one has some drawbacks.
- It is possible that you could make implicit changes to your
- code that the checksum calculations do not take into
- account.
- These implicit changes affect a task's output but do not trigger
- the shared state code into rebuilding a recipe.
- Consider an example during which a tool changes its output.
- Assume that the output of <filename>rpmdeps</filename> changes.
- The result of the change should be that all the
- <filename>package</filename> and
- <filename>package_write_rpm</filename> shared state cache
- items become invalid.
- However, because the change to the output is
- external to the code and therefore implicit,
- the associated shared state cache items do not become
- invalidated.
- In this case, the build process uses the cached items rather
- than running the task again.
- Obviously, these types of implicit changes can cause problems.
- </para>
-
- <para>
- To avoid these problems during the build, you need to
- understand the effects of any changes you make.
- Realize that changes you make directly to a function
- are automatically factored into the checksum calculation.
- Thus, these explicit changes invalidate the associated area of
- shared state cache.
- However, you need to be aware of any implicit changes that
- are not obvious changes to the code and could affect the output
- of a given task.
- </para>
-
- <para>
- When you identify an implicit change, you can easily take steps
- to invalidate the cache and force the tasks to run.
- The steps you can take are as simple as changing a function's
- comments in the source code.
- For example, to invalidate package shared state files, change
- the comment statements of
- <link linkend='ref-tasks-package'><filename>do_package</filename></link>
- or the comments of one of the functions it calls.
- Even though the change is purely cosmetic, it causes the
- checksum to be recalculated and forces the OpenEmbedded build
- system to run the task again.
- </para>
-
- <note>
- For an example of a commit that makes a cosmetic change to
- invalidate shared state, see this
- <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
- </note>
- </section>
- </section>
-</section>
-
<section id='automatically-added-runtime-dependencies'>
<title>Automatically Added Runtime Dependencies</title>
${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
</literallayout>
For tasks that are accelerated through the shared state
- (<link linkend='shared-state-cache'>sstate</link>)
+ (<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
cache, an additional <filename>siginfo</filename> file is
written into
<link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
projects / thirdparty / openembedded / openembedded-core-contrib.git / commitdiff
