Richard Purdie [Fri, 20 Jan 2017 11:28:04 +0000 (11:28 +0000)]
data_smart: Fix unneeded variable manipulation
If was pointed out that if we have:
XXX = " A"
XXX_remove_inactive-override = "YY"
then XXX can become "A" and the leading space can be removed. This is because
the remove override code changes the variable value even when there is no
removals active. In the process it dirties the cache.
We don't really need to do this so tweak the code accordingly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly [Thu, 19 Jan 2017 19:35:16 +0000 (20:35 +0100)]
runqueue.py: revised completion scheduler
The idea is that tasks which complete building a recipe (like
do_package_qa) are more important than tasks which start building new
recipes (do_fetch) or those which increase disk usage
(do_compile). Therefore tasks get ordered like this (most important
first, do_rm_work before do_build because the enhanced rm_work.bbclass
was used):
1. ID /work/poky/meta/recipes-support/popt/popt_1.16.bb:do_build
2. ID /work/poky/meta/recipes-core/readline/readline_6.3.bb:do_build
3. ID /work/poky/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb:do_build
...
464. ID /work/poky/meta/recipes-sato/images/core-image-sato.bb:do_build
465. ID /work/poky/meta/recipes-graphics/xorg-proto/inputproto_2.3.2.bb:do_rm_work
466. ID /work/poky/meta/recipes-devtools/python/python3_3.5.2.bb:do_rm_work
467. ID /work/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_rm_work
...
3620. ID virtual:native:/work/poky/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_install
3621. ID /work/poky/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb:do_install
3622. ID /work/poky/meta/recipes-core/zlib/zlib_1.2.8.bb:do_compile_ptest_base
3623. ID /work/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_compile_ptest_base
...
3645. ID /work/poky/meta/recipes-support/libevent/libevent_2.0.22.bb:do_compile_ptest_base
3646. ID /work/poky/meta/recipes-core/busybox/busybox_1.24.1.bb:do_compile_ptest_base
3647. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_uboot_mkimage
3648. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_sizecheck
3649. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_strip
3650. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_compile_kernelmodules
3651. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_shared_workdir
3652. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_kernel_link_images
3653. ID /work/poky/meta/recipes-devtools/quilt/quilt-native_0.64.bb:do_compile
3654. ID /work/poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_compile
...
The order of the same task between different recipes is the same as
with the speed scheduler, i.e. more important recipes come first.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly [Thu, 19 Jan 2017 19:35:15 +0000 (20:35 +0100)]
build.py: add preceedtask() API
The API is required by the revised rm_work.bbclass implementation,
which needs to know all tasks that do_build depends so that it
can properly inject itself between do_build and those tasks.
The new API primarily hides the internal implementation of the "after"
and "before" dependency tracking. Because tasks defined as
precondition via "recrdeptask" may or may not be relevant (they are for
rm_work.bclass), the API also includes support for that.
There's no default value for including recrdeptasks, so developers
have to think about what they need.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sat, 14 Jan 2017 14:17:13 +0000 (14:17 +0000)]
fetch2: Avoid recursive errors
If PATH contains WORKDIR which contains PV which contains SRCPV we can end
up in circular recursion within the fetcher. This code change allows for the recursion
to be broken by giving PV a temporary dummy value in a data store copy.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sat, 14 Jan 2017 14:16:44 +0000 (14:16 +0000)]
cooker: Add BB_LIMITEDDEPS support
When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited.
We really need a way to know this from the metadata and this new variable
provides this in worker context. This means existing code can stop having
to guess.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 12 Jan 2017 13:25:36 +0000 (13:25 +0000)]
event/ast: Add RecipeTaskPreProcess event before task finalisation
There are various pieces of code which need to run after the tasks are
finalised but before bitbake locks in on the task dependencies. This
adds such an event so dependency changes in anonymous python can
be accounted for and acted upon by these specific event handlers.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 16 Jan 2017 14:40:24 +0000 (14:40 +0000)]
cooker: Only add universe targets where the task exists
Currently "bitbake universe -c fetch" will error if every task in the system
doesn't have a fetch task. This seems unreasonable so filter the list based
upon whether the task exists. This implementation does assume if a task exits
for a given provider name, it can run, even if the preferred provider that
ends up getting built might be the namespace without the task. This is unlikely
to be a real world issue at this point.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Juro Bystricky [Sun, 15 Jan 2017 19:13:36 +0000 (11:13 -0800)]
cooker.py: new multiconfig '*' syntax support
Currently you cannot build a target for all the configured multiconfigs without
specifying a list. The list can be quite long, requiring to type several lines
of text.
This enhancement is to support globbing so that you can do this,
e.g. instead of:
There are real world use cases where it is desirable to use multiconfig with
two different tasks. For example: SDKs with multiple toolchains but also
containing set of additional host tools, or multiconfig builds requiring one image for
the main CPU(s) and a different co-image for a companion CPU.
For this reason, two variations of the new syntax are supported.
brian avery [Sun, 15 Jan 2017 03:18:32 +0000 (19:18 -0800)]
toaster: bin/toaster whitelist TOASTER_DIR
Some of the tinfoil2 changes resulted in TOASTER_DIR being cleared by
the memory resident bitbake server toaster starts up. This prevented
toaster from being able to connect to its sqlite database. Adding
TOASTER_DIR to the BB_ENV_EXTRAWHITE list stops the cooker from clearing
out the TOASTER_DIR variable.
[YOCTO #9252]
Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ola Redell [Thu, 12 Jan 2017 10:33:27 +0000 (11:33 +0100)]
gitsm.py: Add force flag to git checkout command in update_submodules
When the gitsm fetcher is used with a repo that includes a .gitattributes
file that makes git modify files on cloning (e.g. line break characters),
the subsequent checkout performed in the update_submodules function fails.
This is fixed by adding the force flag (-f) to the checkout command.
Signed-off-by: Ola Redell <ola.redell@retotech.se> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Jianxun Zhang [Wed, 21 Dec 2016 20:27:37 +0000 (12:27 -0800)]
use multiple processes to dump signatures.
This change significantly shortens the time on reparsing stage
of '-S' option.
Each file is reparsed and then dumped within a dedicated
process. The maximum number of the running processes is not
greater than the value of BB_NUMBER_PARSE_THREADS if it is set.
The dump_sigs() in class SignatureGeneratorBasic is _replaced_
by a new dump_sigfn() interface, so calls from the outside and
subclasses are dispatched to the implementation in the base
class of SignatureGeneratorBasic.
Fixes [YOCTO #10352]
Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Aníbal Limón [Thu, 5 Jan 2017 23:44:46 +0000 (17:44 -0600)]
bb/cooker: BBCooker stops notifier at shutdown
At end of BBCooker needs to release fd's associated with pyinotify
watchers to avoid: Too many open files (EMFILE) error in different
scenarios like several instances of tinfoil.
[YOCTO #10873]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 4 Jan 2017 23:23:52 +0000 (23:23 +0000)]
prserv/serv: Tweak stdout manipulation to be stream safe
We've been seeing oe-selftest failures under puzzling circumstances. It
turns out if you run oe-selftest on a machine with xmlrunner installed
and have the recent tinfoil2 changes, the launching of PR server crashes
leading to selftest hanging if using an autoloaded PR server.
The reason is that xmlrunner uses an io.StringIO object as stdout/stderr
instead of the usual io.TextIOWrapper and StringIO lacks a fileno() method.
We have to deal with both cases and in the python way, we try and then seek
forgivness if we see an AttributeError or UnSupportedOperation exception.
Unfortunately we have to deal with both cases as we may be performing a
traditiional double fork() from the commandline, or a larger python program.
[YOCTO #10866]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 20 Dec 2016 19:02:11 +0000 (19:02 +0000)]
cookerdata: Convert multiconfig to use BB_CURRENT_MC
People are struggling with multiconfig as the point the conf file
is injected into the data store is not what people expect. We can't
really use a post config since that is too late and we can't really
use a pre config file since that is too early. In OE terms, we need
something right around the local.conf point so it behaves in a similar
way.
A way to handle this is to set the new variable BB_CURRENT_MC to be the
currently selected multiconfig, then the metadata itself can choose
when to inject the approriate configuration.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 21 Dec 2016 14:11:24 +0000 (14:11 +0000)]
runqueue: Only start fakeroot workers when needed
Fakeroot workers usually have dependencies that need to be ready before they
can be started. Starting them as a block therefore doesn't work as the
dependencies may or may not have been built. Therefore start the multiconfig
fakeworkers individually upon demand.
[YOCTO #10344]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 21 Dec 2016 13:36:25 +0000 (13:36 +0000)]
runqueue: Ensure setscene tasks with overlapping stamps don't parallel execute
In multiconfig, mutliple tasks can execute which share the same stamp file. These
must not execute in parallel, the idea is the first should execute, the subsequent
ones should see a valid stamp and get skipped.
The normal task execution code has stamps code to handle this, this adds similar
code to the setscene execute() function to handle the issue there too.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 21 Dec 2016 13:35:53 +0000 (13:35 +0000)]
runqueue: Fix setscene issues with multiconfig
setscene was being entirelu skipped for multiconfig variants as the tasks
were simply not being spotted. If the default config was also being built
it masked the problem. When this was fixed by using taskfn instead of fn
in lookups against dataCache, several other instances of this problem were
highlighted.
This goes through and corrects the setscene code to correclty use taskfn
instead of fn in the appropriate places meaning setscene tasks for multiconfig
now work correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 20 Dec 2016 23:31:11 +0000 (23:31 +0000)]
siggen: Fix clean_basepath to work with multiconfig
Tasknames can now start with "multiconfig:" which broke the virtual: comparison code and
lead to unpredictable checksums with nativesdk recipes. This adds in handling for
the new additional prefix which unbreaks nativesdk builds when using multiconfig.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 20 Dec 2016 21:15:31 +0000 (21:15 +0000)]
build/uihelper: Show better information about multiconfig tasks on UI
Currently the UI shows X is building, possibly multiple times but doesn't
say which of the multilibs that might be. This adds a prefix to the task
name so the mulitconfig being built can be identified.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 20 Dec 2016 20:51:02 +0000 (20:51 +0000)]
runqueue: Ensure pseudo executes from the correct place (use the right datastore with multiconfig)
The location of the fakeroot command and the various environmental values need
to be taken from the right multiconfig datastore, not the shared one. This
patch ensures the right one is used for cases like a split TMPDIR.
[YOCTO #10344]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mark Asselstine [Wed, 21 Dec 2016 02:50:01 +0000 (21:50 -0500)]
fetch2/wget: fixup case with no useful netrc data
Commit 873e33d0479e977520106b65d149ff1799195bf6 [fetch2/wget:
add Basic Auth from netrc to checkstatus()] causes "Fetcher failure
for URL: 'https://www.example.com/'. URL https://www.example.com/
doesn't work." on new builds when a user has a .netrc file but there
is no default and no matching host. The call to netrc.authenticators()
will return None in these cases and the attempted assignment to the
3-tuple will raise a TypeError exception. Add the TypeError to the
exceptions caught to get around this issue.
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/wget: add Basic Auth from netrc to checkstatus()
fetch2/wget uses urllib to check the status of the mirrors, wget will
use netrc to pass login and password information however checkstatus
will skip that.
This adds netrc login and password to checkstatus so both will work the
same.
Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Thu, 15 Dec 2016 18:09:13 +0000 (07:09 +1300)]
runqueue: dry-run real tasks when BB_SETSCENE_ENFORCE is set
For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used
by the installation process for the extensible SDK), we don't actually
need the whitelisted real tasks to execute - we just need to have them
in the dependency tree so that we get all of the setscene tasks they
depend on to run. Therefore we can actually dry-run those real tasks
i.e. they won't be run (and thus we won't waste a significant amount of
time doing so) and won't be stamped as having run either. We do already
have a dry-run mode in BitBake (activated by the -n or --dry-run command
line option), but it dry-runs the setscene tasks as well which we don't
want here.
Note that this has no effect on the checking we are doing with
BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are
scheduled to run - that's handled separately.
Fixes [YOCTO #10369].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 15 Dec 2016 09:36:30 +0000 (09:36 +0000)]
cooker/command: Drop expanded_data
Some of our metadata assumes that BuildStarted and BuildCompleted events
see the same data store. This is the case for buildTarget but not for
buildFile and recent changes mean this is now a problem.
The update_data() call is now an empty operation and there is no difference
between the expanded_data and data so we can simply remove the expanded_data
and its references and use data everywhere. This has been inteded for a while
but the above issue makes this more pressing to finally clean up.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Mon, 12 Dec 2016 21:53:51 +0000 (10:53 +1300)]
lib/bb/build: enable access to logger within tasks
In certain circumstances it can be useful to get access to BitBake's
logger within a task; the main example is in OpenEmbedded's image
construction code where we want to be able to check the log file for
errors and warnings, but we don't want to see any errors or warnings
that were emitted through the logger; so we need a way to exclude those.
In order to do this, pass the logger object into the task via a
BB_TASK_LOGGER variable, and add a logging handler class to bb.utils
that can be added to it in order to keep a list of warnings/errors that
have been emitted.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:14 +0000 (20:07 +1300)]
server/process: don't change UI process signal handler on terminate
On terminating the connection to the server, we were disabling SIGINT -
and this is executed on the UI side. I'm not sure whether the intention
here was to undo the SIGINT disabling we did in the server, and it was
just a mistake that it disabled rather than restored and it's run on the
wrong side, or whether we wanted to stop the user from breaking out of
the shutdown code - the commit message provides no clues either way.
Regardless, we do not want to permanently disable Ctrl+C here - it's
legitimate to terminate the connection to the server and then
re-establish it within the same process; at least currently, devtool
modify by virtue of using tinfoil in two separate parts of the code does
this, and the result of this disabling is that during the second tinfoil
usage we can potentially be parsing all recipes without the ability to
easily interrupt the process.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:13 +0000 (20:07 +1300)]
siggen: add means of ignoring basehash mismatch
If you run the setVariable command to set variables then you end up
causing the basehash to not match the previously computed values, which
triggers error messages. These mismatches are expected, so add a means
of disabling them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:12 +0000 (20:07 +1300)]
runqueue: enable setVariable command to affect task execution
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:11 +0000 (20:07 +1300)]
data_smart: support serialisation
The COW object used within VariableHistory can't be serialised itself,
but we can convert it to a dict when serializing and then back when
deserialising. This finally allows DataSmart objects to be serialized.
NOTE: "serialisation" here means pickling, not over XMLRPC or any other
transport.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:09 +0000 (20:07 +1300)]
tinfoil: pass datastore to server when expanding python references
If you're expanding a value that refers to the value of a variable in
python code, we need to ensure that the datastore that gets used to get
the value of that variable is the client-side datastore and not just the
part of it that's on the server side. For example, suppose you are in
client code doing the following:
d.setVar('HELLO', 'there')
result = d.expand('${@d.getVar("HELLO", True)}')
result should be "there" but if the client part wasn't taken into
account, it would be whatever value HELLO had in the server portion of
the datastore (if any).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:08 +0000 (20:07 +1300)]
tinfoil: implement server-side recipe parsing
It's not really practical for us to parse recipes on the client side, we
need to do it on the server because that's where we have the full python
environment (including any "pure" python functions defined in classes).
Thus, add some functions to tinfoil do this including a few shortcut
functions.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:07 +0000 (20:07 +1300)]
remotedata: enable transporting datastore from the client to the server
For the purposes of server-side parsing and expansion allowing for
client-side use of the datastore, we need a means of sending a datastore
from the client back to the server, where the datastore probably
consists of a remote (server-side) original plus some client-side
modifications. To do this we need to take care of a couple of things:
1) xmlrpc can't handle nested dicts, so if you enable memres and simply
try passing a serialised datastore then things break. Instead of
serialising the entire datastore, just take the naive option of
transferring the internal dict alone (as a list of tuples) for now.
2) Change the TinfoilDataStoreConnector object into simply the handle
(number) when transmitting; it gets substituted with the real
datastore when the server receives it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:06 +0000 (20:07 +1300)]
tinfoil: rewrite as a wrapper around the UI
Rewrite tinfoil as a wrapper around the UI, instead of the earlier
approach of starting up just enough of cooker to do what we want. This
has several advantages:
* It now works when bitbake is memory-resident instead of failing with
"ERROR: Only one copy of bitbake should be run against a build
directory".
* We can now connect an actual UI, thus you get things like the recipe
parsing / cache loading progress bar and parse error handling for free
* We can now handle events generated by the server if we wish to do so
* We can potentially extend this to do more stuff, e.g. actually running
build operations - this needs to be made more practical before we can
use it though (since you effectively have to become the UI yourself
for this at the moment.)
The downside is that tinfoil no longer has direct access to cooker, the
global datastore, or the cache. To mitigate this I have extended
data_smart to provide remote access capability for the datastore, and
created "fake" cooker and cooker.recipecache / cooker.collection adapter
objects in order to avoid breaking too many tinfoil-using scripts that
might be out there (we've never officially documented tinfoil or
BitBake's internal code, but we can still make accommodations where
practical). I've at least gone far enough to support all of the
utilities that use tinfoil in OE-Core with some changes, but I know
there are scripts such as Chris Larson's "bb" out there that do make
other calls into BitBake code that I'm not currently providing access to
through the adapters.
Part of the fix for [YOCTO #5470].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows you to maintain a local reference to a remote datastore. The
actual implementation of the remote connection is delegated to a
connector object that the caller must define and supply. There is
support for getting variable values and expanding python references
(i.e. ${@...} remotely, however setting variables remotely is not
supported - any variable setting is done locally as if the datastore
were a copy (which it kind of is).
Loosely based on an earlier prototype implementation by Qing He.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:03 +0000 (20:07 +1300)]
server/xmlrpc: send back 503 response with correct encoding
If you send back a string here you get "TypeError: 'str' does not
support the buffer interface" errors in bitbake-cookerdaemon.log and
"IncompleteRead(0 bytes read, 22 more expected)" errors on the client
side.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:02 +0000 (20:07 +1300)]
knotty: fix --observe-only option
If we're in observe-only mode then we cannot run commands that would
affect the server's state, including getSetVariable, so prevent that
from being called in observe-only mode.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:01 +0000 (20:07 +1300)]
knotty: make quiet option a level option
Allow you to specify -q / --quiet more than once to reduce the messages
even further. It will now operate as follows:
Level Option Result
----- ------ ----------------------------------------
0 Print usual output
1 -q Only show progress and warnings or above
2 -qq Only show warnings or above
3+ -qqq Only show errors
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 13 Dec 2016 07:07:00 +0000 (20:07 +1300)]
data_smart: fix resetting of reference on variablehistory
There is no "datasmart" member, only dataroot. This dates back to the
original implementation of variable history support - it's surprising we
haven't noticed the issue until now, but I guess it's rare to change a
copy of a datastore in a manner which using the old reference would
cause an issue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Fri, 9 Dec 2016 16:52:50 +0000 (16:52 +0000)]
toaster: importlayer Add git revision typeahead to that input field
Add the front end mechanism to load the typeahead for the git revision
field on importing a layer.
Also fix one indentation issue and update the js test.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Fri, 9 Dec 2016 16:52:49 +0000 (16:52 +0000)]
toaster: typeaheads Add a git revisions suggestions
When we're importing a layer it's useful to suggest available git revisions of the
layers. This uses git ls-remote to fetch the revisions and then filter on this.
Caching is added for this typeahead to avoid having to fetch this
information multiple times in a single session.
[YOCTO #8429]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 14 Nov 2016 09:51:38 +0000 (09:51 +0000)]
cooker: Fix world taskgraph generation issue
The processing of the "do_" prefix to tasks is currently inconsistent
and has resulted in "bitbake world -g" being broken as task prefixes
don't get handled correctly.
Make the "do_" task prefix handling consistent through various codepaths.
[YOCTO #10651]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 7 Dec 2016 12:04:45 +0000 (12:04 +0000)]
runqueue: Send BB_TASKDEPDATA for setscene tasks
We now have code in OE that needs BB_TASKDEPDATA for setscene tasks. Therefore
generate and send this data. In this case its a "pre collapsed" tree
but that is fine for the use cases in question.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 7 Dec 2016 12:04:43 +0000 (12:04 +0000)]
bitbake-worker: Further IO performance tweaks
Looking further at the CPU loads on systems running large numbers of tasks,
the following things helps performance:
* Loop on waitpid until there are no processes still waiting
* Using select to wait for the cooker pipe to be writable before writing
avoiding pointless 100% cpu usage
* Only reading from worker pipes that select highlights are readable
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Daniel Lublin [Mon, 5 Dec 2016 17:42:05 +0000 (18:42 +0100)]
lib/bs4: Fix imports from html5lib >= 0.9999999/1.0b8
As of html5lib 0.9999999/1.0b8 (released on July 14, 2016), some modules
have moved from _base to base. Handle this, while staying compatible
with earlier versions.
Signed-off-by: Daniel Lublin <daniel@lublin.se> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sujith H [Thu, 1 Dec 2016 05:43:09 +0000 (11:13 +0530)]
cooker: convert type which needs to be marshalled
We assume that the value taken by variable v can be string,
integer or any type which can be marshalled by xmlrpc. This
change would help us to convert the non marshallable types
to string. So that we don't get exception from xmlrpc.
[YOCTO #10740]
Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
brian avery [Thu, 1 Dec 2016 03:35:27 +0000 (19:35 -0800)]
toaster: browser tests - add Selenium Docker container as driver
Adds the ability to specify a Selenium Docker container server as
a driver. This allows for repeatable tests independent of host.
Currently we assume you are using the Firefox container. Instructions
are located in the README in tests/browser.
Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly [Tue, 29 Nov 2016 16:47:45 +0000 (17:47 +0100)]
monitordisk: add event
The current disk usage is interesting and may be worth logging over
time as part of the build statistics. Instead of re-implementing the
code and the configuration option (BB_DISKMON_DIRS), the information
gathered by monitordisk.py is made available to buildstats.bbclass via
a new event.
This has pros and cons:
- there is already a useful default configuration for "interesting" directories
- no code duplication
- on the other hand, users cannot configure recording separately from
monitoring (probably not that important)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly [Tue, 29 Nov 2016 16:47:43 +0000 (17:47 +0100)]
runqueue.py: monitor disk space at regular time intervals
Hooking the disk monitor into the regular heatbeat event instead
of the runqueue solves two problems:
- When there is just one long running task which fills up the disk,
the previous approach did not notice that until after the completion
of the task because _execute_runqueue() only gets called on task
state changes. As a result, aborting a build did not work in this
case.
- When there are many short-lived tasks, disk space was getting
checked very frequently. When the storage that is getting checked
is on an NFS server, that can lead to noticable traffic to the
server.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly [Tue, 29 Nov 2016 16:47:42 +0000 (17:47 +0100)]
cooker process: fire heartbeat event at regular time intervals
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.
All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.
The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.
If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly [Fri, 18 Nov 2016 15:23:22 +0000 (16:23 +0100)]
codeparser.py: support deeply nested tokens
For shell constructs like
echo hello & wait $!
the process_tokens() method ended up with a situation where "token"
in the "name, value = token" assignment was a list of tuples
and not the expected tuple, causing the assignment to fail.
There were already two for loops (one in _parse_shell(), one in
process_tokens()) which iterated over token lists. Apparently the
actual nesting can also be deeper.
Now there is just one such loop in process_token_list() which calls
itself recursively when it detects that a list entry is another list.
As a side effect (improvement?!) of the loop removal in
_parse_shell(), the local function definitions in process_tokens() get
executed less often.
Fixes: [YOCTO #10668] Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 24 Nov 2016 11:20:04 +0000 (11:20 +0000)]
toaster: buildinfohelper Simplify layer event to toaster layer function
Simplify the layer event information to layer version object in toaster
function. Previously this attempted many different methods of trying to
obtain the correct layer from toaster by manipulating the data from the
event or the data from the known layers to try and match them together.
We speed up and simplify this process by making better use of django's
orm methods and by working down the most likely matching methods in order
of accuracy.
[YOCTO #10220]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 24 Nov 2016 11:20:03 +0000 (11:20 +0000)]
toaster: bldcontrol Move CustomImageRecipe file creation into own function
Move the custom image file creation (i.e. create the layer file
structure, conf and recipe file) into it's own function and remove the
creation of the BRLayer as this is done at schedule_build just like all
the other layers.
Fix a bug where the toaster-custom-images layer was always being appened
to the layer list if the directory exists.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 24 Nov 2016 11:20:02 +0000 (11:20 +0000)]
toaster: orm models Handle CustomImageRecipe BRLayer here
The schedule_build function on the project object is where the BRLayers
are created for the build. Instead of creating the BRLayer for the
CustomImageRecipe in the localhostbbcontroller create it here so that
all that mechanism is in one place.
Also fix a number of pyflake errors.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes the unidentified layers issue by making the
toaster-custom-images layer a local layer. By doing this we also fix the
git assumptions made for the local layers which stop recipes and other
meta data being associated with them. This also removed some of the
special casing previously needed when we didn't have the concept of a
local (non git) layer.
Also rename created flag var to a have a different var for each returned
value so that the same value isn't used multiple times.
[YOCTO #10220]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sujith H [Thu, 24 Nov 2016 11:19:58 +0000 (11:19 +0000)]
toaster: localhostbecontroller accept custom init script for build
When passed variable CUSTOM_BUILD_INIT_SCRIPT to toaster
setting, it would be nice to use it. Else toaster
can use oe-init script. This gives an oppurtunity to
use customized build init scritps.
Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reyna, David [Thu, 24 Nov 2016 11:19:56 +0000 (11:19 +0000)]
toaster: orm gen_layerdeps Protect against circular Layer dependencies
Limit the recursion (to say 20 levels) when processing layer dependencies
so that circular dependecies do not cause infinite decent and an
out-of-memory failure. The duplicate found layers are already immediately
filtered in the code.
[YOCTO #10630]
Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 24 Nov 2016 11:19:55 +0000 (11:19 +0000)]
toaster: customrecipejs Consume click event on 'a' link if disabled
Consume the click event on the download recipe link if it's disabled. To
prevent the link from sending user to an error page.
See http://getbootstrap.com/css/#forms-disabled-fieldsets and a link
caveat.
[YOCTO #10151]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 24 Nov 2016 11:19:52 +0000 (11:19 +0000)]
toaster: runbuilds Write the pidfile in python rather than shell script
Write the pid file out in the start up of this management command. This
ensures this has happened instead of relying on the shell command having
been run which may or may not be the case. This also makes it simpler for
testing.
Couple of clean ups of runbuilds as identified by pyflake
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I noiced builds where tasks seemed to be taking a surprisingly long time.
When I looked at the output of top/pstree, these tasks were no longer
running despite being listed in knotty. Some were in D/Z state waiting for
their exit code to be collected, others were simply not present at all.
strace showed communication problems between the worker and cooker, each
was trying to write to the other and nearly deadlocking. Eventually, timeouts
would allow them to echange 64kb of data but this was only happening every
few seconds.
Whilst this particularly affected builds on machines with large numbers
of cores (and hence highly parallal task execution) and in cases where
I had a lot of debug enabled, this situation is clearly bad in general.
This patch introduces a thread to the worker which is used to write data
back to cooker. This means that the deadlock can't occur and data flows
much more freely and effectively.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Kai Kang [Sun, 9 Oct 2016 08:34:32 +0000 (16:34 +0800)]
COW.py: fix sample codes
The call of methods iteritems() and itervalues() in sample codes were
replaced by items() and values() to convert to Python 3 by Bitbake rev d0f904d407f57998419bd9c305ce53e5eaa36b24. But the methods iteritems()
and itervalues() belong to class COWDictMeta not class dict or set. The
modifications should not be made in purpose that it fails to run sample
codes, so revert them.
Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 10 Nov 2016 03:52:34 +0000 (19:52 -0800)]
toaster: layerindex updater Take into account layers being predefined
As we can now provide layer definitions through fixtures we need to be
more clever how we update the metadata in the database to avoid
duplicate metadata being created. To do this we make more effort to
match existing data in the database and update only the fields which
will be better provided by the layer index.
This removes the need for us to special case layers which are provided
as part of poky such as openembedded-core or meta-poky which exist on
the layerindex but with different git urls.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 10 Nov 2016 03:52:33 +0000 (19:52 -0800)]
toaster: orm/fixtures Add the master release and correct morty release
Add the master release option to base your project on and correct the
morty release so that for poky based setups we use the poky provided
version of the layer rather than checking out the layer from its own git
repository.
[YOCTO #10497]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
