Markus Lehtonen [Fri, 19 Feb 2016 12:22:58 +0000 (14:22 +0200)]
SignatureGenerator: add checksum_cache argument to get_taskhash()
Extend the SignatureGenerator API by adding a new argument to
get_taskhash() for defining the file checksum cache to use. If the
checksum cache argument is not provided, we use the cache in bb.fetch2,
as before. This is a step towards removing checksum cache from
bb.fetch2.
Unfortunately, extending the API is a bit more involved than just adding
a new optional argument because it has a sort of cyclic dependecy: the
API is defined in bitbake, the implementation may be modified elsewhere
(e.g. oe.sstatesig in oe-core layer) by overriding SignatureGenerator's
method(s), which are eventually consumed back at bitbake. In bitbake, we
must be prepared to use the old interface because oe-core may override
our new shiny API with an old one. Thus, we need to inspect the API in
RunQueueData and adjust call arguments accordingly. If the signature
generator instance implements the new API, we use a checksum cache
instance specific to runque. Print a warning if an implementation of the
old API is detected.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
bb.{cooker, data}: only emit a var as python if 'func' is set
This avoids a common issue where PACKAGECONFIG is emitted as a function in
bitbake -e when the 'python' flag exists. It isn't a python function unless
both 'func' and 'python' are set. This aligns with the behavior of
emit_func_python.
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It seems the frozenset constructor in pypy runs len(), so we can't pass the
DataSmart instance directly to it, instead pass the iterator. Fixes pypy
support.
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: fetch2: Safer check for BB_ORIGENV datastore
BB_ORIGENV value on the datastore can be NoneType thus raising an AttributeError
exception when calling the getVar method. To avoid this, a check is done before
accesing it.
[YOCTO #9567]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 21 Apr 2016 13:22:06 +0000 (14:22 +0100)]
toaster: bin/toaster Fix noweb command
For the noweb command we don't need to check if a socket is in use or
not as we're not starting the django development server. We're just
setting up the environment and running the runbuilds scheduler.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 21 Apr 2016 13:22:05 +0000 (14:22 +0100)]
toaster: bin/toaster Add ability to specify port and address to bind to
Allow binding to a specified address and port to start the django
development server on.
Remove the assumption that you want to bind to 0.0.0.0 and set the
default to localhost.
Add some additional help text after Toaster webserver has started
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 21 Apr 2016 13:22:04 +0000 (14:22 +0100)]
toaster: bin/toaster Add help text for unrecognised command
Add help text for unrecognised or missing command for toaster script
Remove assumption that no command is 'start' as the default.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 20 Apr 2016 07:31:29 +0000 (10:31 +0300)]
bitbake: main: set defaults from env variables
Environment variables BBSERVER, BBTOKEN and BBEVENTLOG silently
overwrite bitbake command line arguments. This is confusing and
can cause issues that are difficult to debug. It's better to use
them as default values instead.
Used environment variables BBSERVER, BBTOKEN and BBEVENTLOG to set
default values for command line arguments.
Changed setting default value of --ui command line argument from
BITBAKE_UI to look similar way.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 20 Apr 2016 07:31:28 +0000 (10:31 +0300)]
bitbake: main: add 2 environment variables
Bitbake uses set of environment variables to set command line
options, e.g. seeting BBTOKEN variable has the same effect
as using --token command line option.
Added new environment variables BBPRECONF and BBPOSTCONF that
are equivalents of --read and --postread command line options.
They can be used by high level scripts to append or prepend
configuration files to conf/local.conf
[YOCTO #9235]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 26 Apr 2016 16:18:07 +0000 (17:18 +0100)]
toaster: customrecipe Only show download icon or button if it's possible
If the based on recipe has not yet been checked out/cloned we cannot
generate the custom image recipe file that uses it. So disable/remove
the option to download it.
[YOCTO #9425]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 26 Apr 2016 16:18:06 +0000 (17:18 +0100)]
toaster: orm Add get_base_recipe_file to CustomImageRecipe
This function returns the base recipe file path only if it currently
exists. This allows us to know whether we can proceed at this point with
generating a custom image recipe. It also enables us to call this
function from the templates to enable visual indication of this state.
Some whitespace fixes also added in generate_recipe_file_contents
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Fri, 22 Apr 2016 15:56:50 +0000 (16:56 +0100)]
fetch2: export DBUS_SESSION_BUS_ADDRESS to support authentication agents
Some users may want to use authenticated SSH connections with credentials stored
in a keyring, such as gnome-keyring. These typically need a DBus session bus
connection, so pass DBUS_SESSION_BUS_ADDRESS into the fetcher environment.
To avoid the user needing to set it in their local.conf (which wouldn't be
usable) or adding it to the environment-cleansing whitelist (which would
potentially impact builds) allow the variables being passed to the fetchers to
come from the data store (first) or the original environment (second).
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:47 +0000 (17:28 +0100)]
toaster-tests: tests for build dashboard
Convert existing tests to Selenium.
Add basic tests to check that the modal contains radio buttons to select
a custom image to edit when a build built multiple custom images, and
to create a new custom image from one of the images built during
the build.
[YOCTO #9123]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:46 +0000 (17:28 +0100)]
toaster: add modal to select custom image for editing
Add functionality to the placeholder button on the build dashboard
to open a modal dialog displaying editable custom images, in cases
where multiple custom images were built by the build. Where there
is only one editable custom image, go direct to its edit page.
The images shown in the modal are custom recipes for the project
which were built during the build shown in the dashboard.
This also affects the new custom image dialog, as that also has
to show custom image recipes as well as image recipes built during
the build. Modify the API on the Build object to support both.
Also modify and rename the queryset_to_list template filter so that
it can deal with lists as well as querysets, as the new custom image
modal has to show a list of image recipes which is an amalgam of two
querysets.
[YOCTO #9123]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:45 +0000 (17:28 +0100)]
toaster: add build dashboard buttons to edit/create custom images
When a build is viewed in the dashboard, enable users to edit
a custom image which was built during that build, and/or create
a new custom image based on one of the image recipes built during
the build.
Add methods to the Build model to enable querying for the
set of image recipes built during a build.
Add buttons to the dashboard, with the "Edit custom image"
button opening a basic modal for now. The "New custom image"
button opens the existing new custom image modal, but is modified
to show a list of images available as a base for a new custom image.
Add a new function to the new custom image modal's script which
enables multiple potential custom images to be shown as radio
buttons in the dialog (if there is more than 1). Modify existing
code to use this new function.
Add a template filter which allows the queryset of recipes for
a build to be available to client-side scripts, and from there
be used to populate the new custom image modal.
[YOCTO #9123]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:44 +0000 (17:28 +0100)]
toaster-tests: make helper click on input before entering text
The Selenium helper's enter_text() method doesn't cause
keyup events to trigger unless the element where text is
being entered has been clicked.
Prefix all text entry with a click() on the element to ensure
that keyup events fire.
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:43 +0000 (17:28 +0100)]
toaster-tests: add tests for new custom image page
Test adding a new custom image when:
1. No custom images are in the project yet.
2. User tries to add custom image which duplicates the name of
an existing custom image.
3. User tries to add custom image which duplicates the name
of a non-image recipe.
[YOCTO #9209]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:42 +0000 (17:28 +0100)]
toaster: prevent exception when Project.release is null
Project.release can be null. This causes an exception when calling
get_all_compatible_layer_versions(), as the query to fetch
the layer versions references release.branch_name.
Add a guard to the function so that an empty queryset is returned
if the release isn't set for a project.
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:41 +0000 (17:28 +0100)]
toaster: only prevent duplicate custom image names within a project
We currently prevent the same name being used for multiple custom
images, but make the check across all projects. This means that
custom image names have to be unique across all projects in
the Toaster installation.
Modify how we validate the name of a custom image so that we
only prevent duplication of custom image names within a project,
while ensuring that the name of a custom image doesn't duplicate
the name of a recipe which is not a custom image recipe.
[YOCTO #9209]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:40 +0000 (17:28 +0100)]
toaster: disable/enable "Add layer" button according to input's content
In the import layer page, the "Add layer" button in the layer dependencies
section doesn't accurately reflect whether the layer name in the
corresponding input can be added. A partial or empty layer name can
leave the button active, such that when it is clicked, a
previously-selected layer can be accidentally added.
Fix by keeping track of the items currently available in the typeahead,
only activating the "Add layer" button when the input matches the name
of one of those items.
[YOCTO #8511]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 19 Apr 2016 16:28:39 +0000 (17:28 +0100)]
toaster: fix sorting after hiding a column in build tables
When hiding a column in the build tasks or build packages included
table, if the column is set as the current order by for the table, the
order by is not reset to the default. The result is that the table
stays sorted by the hidden column.
Set the default_orderby for these two tables correctly to ensure the
corresponding table is re-sorted when a column is hidden, if that column
was being used as the order by.
[YOCTO #9011]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 19 Apr 2016 16:28:38 +0000 (17:28 +0100)]
toaster: ensure ToasterTable headings are reset when order by changes
If a ToasterTable is ordered by an optional column and that
column is subsequently hidden, the table ordering switches back
to the default ordering for the table. However, the table headings
don't update to reflect the new ordering. This is because the
code which sets the heading weight and hides/shows the caret symbols
only runs when the table is first loaded.
Store the default order by and re-apply it when the data is updated.
[YOCTO #9011]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 14 Apr 2016 13:30:52 +0000 (14:30 +0100)]
providers: Add PREFERRED_RPROVIDER support
Sometimes you can end up in a situation where you need to specify that
a specific runtime entity should be provided by a specific entry.
An example of this is bluez where you could end up in a situation where
for example:
NOTE: multiple providers are available for runtime libasound-module-bluez (bluez4, bluez5)
NOTE: consider defining a PREFERRED_PROVIDER entry to match libasound-module-bluez
NOTE: multiple providers are available for runtime bluez-hcidump (bluez-hcidump, bluez5)
NOTE: consider defining a PREFERRED_PROVIDER entry to match bluez-hcidump
The only option here is to set something like PREFERRED_PROVIDER_bluez4 = "bluez4"
which is clearly not very informative.
I've actually held off adding RPROVIDER support for a long while as this
does have sigificant potential for misuse. It doesn't for example allow
multiple runtime providers of the same name to coexist, that simply isn't
supported. It therefore doesn't replace some of the name mappings such
as busybox verses coreutils that OE-Core faces as that is a different
problem with different constraints. This mechanism is simply to provide
bitbake with a hint to decide what the dependency tree should look like.
Also, this allows us to stop printing a confusing message telling the user
to set PREFERRED_PROVIDER when the setting needed would be rather ambiguous.
[YOCTO #5044]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 14 Apr 2016 13:30:04 +0000 (14:30 +0100)]
providers: We don't depend on previous build results
Back in history the code did depend on previous build results. This was
bad for determinism and we no longer do that. Update comments to match
the current behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 14 Apr 2016 13:29:27 +0000 (14:29 +0100)]
cooker/knotty: Prefix parse logs with filename being parsed
We now prefix log messages coming from worker task context with the
PF and task info, however parsing messages all have to be manually
prefixed which is ugly and error prone. This change modifies the log
handler filter so this happens automatically, meaning we don't have
to change every message to include that information. This makes error
messages longer but more usable.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 29 Feb 2016 14:23:33 +0000 (14:23 +0000)]
cooker: pass exception to finishAsyncCommand
An invalid task causes bitbake to exit incorrectly, firing a
CommandCompleted event rather than a CommandFailed one. This
means that clients listening for CommandFailed events are
unable to detect the build failure even though one occurred.
Passing an exception string to finishAsyncCommand when a task
fails causes the CommandFailed event to be fired correctly.
[YOCTO #9087]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sun, 10 Apr 2016 10:09:21 +0000 (11:09 +0100)]
runqueue: Improve 'mulitiple .bb files are due to be built' message
When multiple recipes which both provide something are being built, bitbake
informs us that most likely one of them provides something the other doesn't,
which is usually correct, but unfortunately it's rather painful to figure out
exactly what that is.
This patch dumps two sets of information, one is the provides information for
each recipe, filtered so only common components are removed. The other is a list
of dependees on the recipe, since sometimes this can easily identify why something
is being built.
Its not straightforward for bitbake to obtain the information but since the
warning/error code path isn't the normal one, we can afford to go through some
less than optimal processing to aid debugging.
Also provide the same information even if we're showing a warning since its still
useful.
[YOCTO #8032]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dave Lerner [Fri, 8 Apr 2016 08:39:28 +0000 (09:39 +0100)]
toaster: fixes for customimage package not found
For a custom image, if a search for a package results in no packages
found, then additional information should be presented to the user.
This is different than a 'no results' found for a search in other
contexts, for example, a search for a package in a non-customised build.
For a custom image, a package search failure can happen because the
package was not added to the custom image. This commit presents more
information to the user, suggesting why the package was not found in the
custom image.
The generic table view handling js changes to handle a new div
element no-results-special-... such that, if present, that template
section is shown rather than the default no-results-... section.
[YOCTO #9154]
Signed-off-by: Dave Lerner <dave.lerner@windriver.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Randy Witt [Thu, 7 Apr 2016 06:55:40 +0000 (23:55 -0700)]
tests/utils.py: test origvalue in a callback matches what is expected
There were no tests that verified the value of origvalue in the callback
routines used by edit_metadata(). This patch adds one for a simple
multiline variable.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Randy Witt [Thu, 7 Apr 2016 06:55:39 +0000 (23:55 -0700)]
lib/bb/utils.py: Fix a bug in edit_metadata() that could corrupt vars
edit_metadata() would corrupt a variable that was multiline, but
had the ending quotes on the same line as the last value. For example:
TEST_VAR = " foo \
bar"
would become " foo ba" because the code would always delete the last
character on the line and then do it again if the line ended in the
quote. This however doesn't show up if you have:
TEST_VAR = " foo \
bar \
"
which is how all the test cases were written.
This patch fixes that bug and adds and fixes a test that matched the bugs
behavior rather than the expected behavior.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Mon, 4 Apr 2016 10:52:49 +0000 (11:52 +0100)]
tests: browser Add test to run the js unit tests
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Mon, 4 Apr 2016 10:47:31 +0000 (11:47 +0100)]
toaster: views jsunittest Add MACHINE and an extra layer to test project
Add set a MACHINE if needed and add a layer. When we're running in the
context of the django unit tests we don't have these defaults setup for
the project so add them.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Fri, 1 Apr 2016 10:47:32 +0000 (11:47 +0100)]
toaster: tests Set MACHINE for the test projects
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Fri, 1 Apr 2016 10:32:35 +0000 (11:32 +0100)]
toaster: Add quint to project so that it can be used offline
Currently we're using a hosted version of quint however this means that the
testing has to be online or have a cached version of quint. Add the
files to Toaster to be able to use offline.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dave Lerner [Wed, 6 Apr 2016 13:44:41 +0000 (14:44 +0100)]
toaster: add rev dep column to image detail pages
Add a column to the custom image pages that shows the reverse
dependencies in a format matching the dependencies column:
- either blank or a button showing the count of reverse dependencies,
- when the button is clicked, a popover appears showing the list
of reverse dependencies, with each package's size, and the total
size of all of the reverse dependencies.
The implementation adds a packages table method to retreive the reverse
dependency total size, and adds a separate 'popover' html template. Both
of these changes follow the pattern for the dependencies column.
[YOCTO #9163]
Signed-off-by: Dave Lerner <dave.lerner@windriver.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have been seeing UnicodeDecodeErrors when handling the
ImagePkgList MetadataEvent in ORMWrapper's
save_target_file_information() if the event includes filenames
that include non-ASCII characters.
In the short term work around this by converting paths to the
unicode type when passing them to Django's ORM. This is a bit of
a hack but it's too late in the cycle to do anything more invasive.
[YOCTO #9142]
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:48 +0000 (17:46 +0100)]
toasterui: update build in internal state
buildinfohelper stores current Build object in its internal
state. Any changes to Build object will be lost if internal
state is not updated as current buildinfohelper code
saves Build object from internal state when build is
completed.
This bug causes incorrect build state when build is cancelled.
Updating internal state should fix it.
Note, that this commit updates internal state after status of
the build is changed to Build.CANCELLED. There are several other
places in the code where Build object is updated without updating
internal state. They should be carefully analyzed and fixed.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:47 +0000 (17:46 +0100)]
buildinfohelper: fix KeyError
When bitbake doesn't need to build anything it still sends
ImagePkgList event with empty 'pkgdata', 'imgdata' and 'filedata'
fields. This causes crash in buildinfohelper code as it's assumed
that above mentioned fields always have data keyed by build target:
ERROR: u'core-image-minimal'
Traceback (most recent call last):
File "toasterui.py", line 423, in main
buildinfohelper.store_target_package_data(event)
File "buildinfohelper.py", line 1218, in store_target_package_data
imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target]
KeyError: u'core-image-minimal'
Fixed this by using dict.get method with empty dictionary as default
return value instead of trying to get value without checking if target
key is in the data.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:46 +0000 (17:46 +0100)]
toaster: get bitbake location from BBBASEDIR
It was incorrectly assumed in the current code that bitbake is in
../bitbake/bin/ directory. It's not always the case.
Using bitbake from $BBBASEDIR should be .
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:45 +0000 (17:46 +0100)]
toaster: export BBBASEDIR variable
This variable will be used in localhost controller code
to determine correct bitbake location, so it has to be
exported.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sujith H [Wed, 6 Apr 2016 16:46:44 +0000 (17:46 +0100)]
toaster: update projectconf.html for DL_DIR and SSTATE_DIR
Modified the projectconf.html to include DL_DIR and
SSTATE_DIR. Updated the script section in the html
to handle the changes made by the user on DL_DIR
and SSTATE_DIR. Included validation check for the
folder names.
[YOCTO #8422]
Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sujith H [Wed, 6 Apr 2016 16:46:43 +0000 (17:46 +0100)]
toaster: update view to support DL_DIR and SSTATE_DIR
Update toaster's views.py to support DL_DIR and SSTATE_DIR for
page projectconf.html. Removed DL_DIR and SSTATE_DIR from
blacklist. Initial value of DL_DIR and SSTATE_DIR comes from
BuildEnvironment.
[YOCTO #8422]
Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:42 +0000 (17:46 +0100)]
toaster: use empty token
If client xmlrpc token is not provided in the command line, bitbake
generates random token. Server token in --server-only mode is always
empty. This doesn't allow clients with non-empty tokens to connect
to the server. Specifying empty token should stop generation of
random token and make it possible for clients to communicate with
the server.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Wed, 6 Apr 2016 16:46:41 +0000 (17:46 +0100)]
toaster: runbuilds Clean up runbuilds
- Organise the imports into logical groups
- Fix 80 col wrapping
- Remove catch all exceptions
- Log to the toaster log
- Use QuerySet functions such as .first() and Q()
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the cancel build button to the mrb section template and add the event
handlers to cancelABuild.
Also clean up the calls to startABuild to use the updated libtoaster methods
and to make the code consistent with it's cancelABuild counterpart.
Co-Author: Sujith H <sujith.h@gmail.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exclude cancelled builds from showing in the builds table
[YOCTO #6787]
Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Wed, 6 Apr 2016 16:46:35 +0000 (17:46 +0100)]
toaster: bldcontrol models Add a cancelling state the BuildRequest
To accurately reflect the state of a build request we also need a
cancelling state. This is set when we've started a build and then for
whatever reason cancel it, cancelling is not instantaneous so we have
this state to indicate that a cancel is in progress.
Also add a state transition guard. As the state of a BuildRequest can
currently be modified by three processes; Toastergui,
Runbuilds/bldcontrol and the buildinofhelper we cannot say for sure
which process will be running at the time of cancellation so in order to
avoid one of these processes making an incorrect transition only allow
transitions of state to increase.
e.g. CREATED -> QUEUED -> INPROGRESS
And to ignore such requested changes such as
INPROGRESS -> CREATED
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sujith H [Wed, 6 Apr 2016 16:46:34 +0000 (17:46 +0100)]
toaster: models Add cancelled state to build outcome
A new state CANCELLED is introduced to, distinguish
the state of build.
[YOCTO #6787]
Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sujith H [Wed, 6 Apr 2016 16:46:33 +0000 (17:46 +0100)]
toaster: update BuildEnvironmentController and BitbakeController
Remove getBBController function from BuildEnvironmentController.
The constructor of BitbakeController is updated appropriately so that
call can be made to connect to running server.
The call to startBBServer is removed from bbcontroller and handledin
localhostbecontroller.
[YOCTO #6787]
Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Wed, 6 Apr 2016 16:46:32 +0000 (17:46 +0100)]
toaster: libtoaster Update implementation of startABuild and cancelABuild
Update the implementation of startABuild and cancelAbuild to reflect
changes to the backend api. We now have a dedicated endpoint to make
calls into so add this url to libtoaster.ctx and allow passing null in
as a url value to indicate that we want to use the current project
Also:
- Fix some documentation comments
- Add the convenience of passing in an array of targets to startABuild
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Wed, 6 Apr 2016 16:46:31 +0000 (17:46 +0100)]
toaster: xhr Update the implementation of the build cancellation request
Update the implementation of the backend api for cancelling builds with
the new cancelling BuildRequest state and cancelled Build state.
Also added some docstring about general usage.
Co-Author: Sujith H <sujith.h@gmail.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Wed, 6 Apr 2016 16:46:30 +0000 (17:46 +0100)]
toaster: Move xhr calls for starting and stopping builds
Move the backend xhr implementation of the build request changes
into it's own file and out of the ToasterTable definition.
It used to live in the views.py but in a hope of starting to collate logical
groups of views move this to a new file called api.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sujith H [Wed, 6 Apr 2016 16:46:29 +0000 (17:46 +0100)]
toaster: bldcontrol Add forceShutDown function to BitbakeController
Add forceShutDown function to BitbakeController class. This function
provides a mechanism to cancel the build from toaster. An API which
can be used safely to cancel build and hence shutdown running bitbake
server.
[YOCTO #6787]
Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:28 +0000 (17:46 +0100)]
toasterui: shutdown on BuildCompleted event
toasterui exits event loop on one of the following events:
CommandCompleted, CommandFailed or CommandExit.
Unfortunately none of them come from bitbake when build fails.
This is normai if toasterui runs in observer mode. However, if it's
in build mode this causes toasterui to stuck in the infinite loop
waiting for new events.
The only event we can rely on is BuildCompleted as it always
comes from bitbake unlike 3 above mentioned events.
Modified the code to always shutdown toasterui in build mode
on BuildCompleted event.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:27 +0000 (17:46 +0100)]
toaster: use bash explicitly
Toaster can't run builds on Ubuntu as default /bin/sh points
to dash there. The reason is that oe-init-build-env can't
be sourced under dash for various reasons. It can be fixed
or work arounded, but it wouldn't fix builds for older releases.
Explicitly using bash to start builds should fix the issue.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:26 +0000 (17:46 +0100)]
toaster: fix jethro build
The keys 'started', 'ended', 'cpu_time_user', 'disk_io_read' and
'disk_io_write' were added to the event recently, so they don't
exist in the events generated by bitbake server from older releases.
Checking if task_to_update structure has these keys before using
them should fix build of older releases.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:25 +0000 (17:46 +0100)]
toaster: update conf/local.conf
Added 'INHERIT+="toaster buildhistory"' line to the conf/local conf
when Toaster starts. It should make commandline builds to provide
all required information to Toaster backend.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:24 +0000 (17:46 +0100)]
toaster: stop bitbake server after the build
Bitbake server is used only during the build. There is no need
in keeping server running after the build.
Running bitbake -m in the subshell after the build should stop
the server.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:23 +0000 (17:46 +0100)]
toaster: add new parameter to _shellcmd
Added 'nowait' parameter to _shellcmd method to support
running chain of commands in a subshell.
This is going to be used to stop bitbake server after
the build.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:22 +0000 (17:46 +0100)]
toaster: reimplement triggerBuild
Reimplemented triggerBuild method to support one build directory
per project:
- start bitbake server from the cloned repository
- don't run observer
- run bitbake build directly instead of triggering it
through xmlrpc
[YOCTO #7880]
[YOCTO #9058]
[YOCTO #8958]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:21 +0000 (17:46 +0100)]
toaster: modified setLayers API
Removed updating of bblayers.conf. It will be done in runBuild method.
Changed return value: return list of layers.
Removed _updateBBLayers method.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:20 +0000 (17:46 +0100)]
toaster: add brbe parameter to triggerBuild
Called triggerBuild with brbe parameter instead of adding TOASTER_BRBE
variable to the database and fetching it in triggerBuild.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:19 +0000 (17:46 +0100)]
toaster: remove release API
This API is quite dangerous as it removes build directory.
It's not used anywhere and most probably will not be used in future
as toaster is going to have one build directory per project.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:18 +0000 (17:46 +0100)]
toaster: remove startBBServer API
We still will have to run bitbake server, but it will be done
different way and the code will be in triggerBuild function.
Removed startBBServer API from BuildEnvironmentController and
LocalhostBEController classes.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:17 +0000 (17:46 +0100)]
toasterui: fix brbe reporting
buildinfohelper.brbe is lost when buildinfohelper is closed.
This causes incorrect report of brbe when build is done.
Saved brbe attribute before closing buildinfohelper and used
it to report correct brbe.
Got rid of useless and confusing 'ToasterUI build done 1'
log message.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:16 +0000 (17:46 +0100)]
buildinfohelper: improve handling of providermap
DepTreeGenerated event doesn't contain 'providermap' data in jethro.
Modified buildinfohelper to handle events without this data. This
should make it possible to handle jethro events coming from jethro
bitbake server by the latest buildinfohelper.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:15 +0000 (17:46 +0100)]
uievent: improve BBUIEventQueue code
Return value of self.BBServer.registerEventHandler differs between
jethro and master. To be able to build jethro toaster should be
able to communicate with jethro bitbake server i.e. it must work
with both old and new registerEventHandler call.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:14 +0000 (17:46 +0100)]
toasterui: add brbe parameter to buildinfohelper
In current toaster code BRBE(build request:build environment) value
is passed from toaster to buildinfohelper through the 'SetBRBE' event.
Passing it through environment variable is easier as it doesn't
involve rpc communication between toaster and bitbake server.
It also eliminates the need in running bitbake observer process.
Added parameter 'brbe' to BuildInfoHelper.__init__
Used environment variable TOASTER_BRBE to set brbe for
buildinfohelper object.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:13 +0000 (17:46 +0100)]
toaster: set BITBAKE_UI environment variable
Set BITBAKE_UI variable to 'toastergui' for command line builds
to use toasterui as a default ui module for bitbake.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:12 +0000 (17:46 +0100)]
toaster: get rid of noui option
noui command line option doesn't makes sense anymore as toaster doesn't
run bitbake. It should be safe to to remove it.
The purpose of this option was to skip running bitbake observer process.
This was never used before as it's not possible to run toaster build
without running observer.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 6 Apr 2016 16:46:11 +0000 (17:46 +0100)]
toaster: don't start bitbake server
>From now on toaster script will not run bitbake server.
It will be started by runbuilds and stopped after the build.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Klauer, Daniel [Tue, 5 Apr 2016 08:49:15 +0000 (08:49 +0000)]
persist_data: Return str instead of unicode for sqlite3 text queries
Python 2's sqlite3 module defaults to returning Unicode strings for SQL
text queries, which could trickle down to other parts of bitbake code and
cause unexpected Unicode conversions. Using byte strings avoids this issue.
For example, the git fetcher's AUTOREV support caches HEAD SHA1's using
bb.persist_data, so sometimes the git command strings passed to fetch2's
runfetchcmd() were unicode, potentially causing UnicodeDecodeErrors when
it appended the values of environment variables containing non-ASCII chars.
[YOCTO #9382]
Signed-off-by: Daniel Klauer <daniel.klauer@gin.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sat, 2 Apr 2016 22:15:06 +0000 (23:15 +0100)]
siggen: Fix nostamp taint handling
The taint values need to be passed from the server to the workers to
ensure they see the same stamp values. Also ensure that the "nostamp:"
prefix isn't included in the checksum value to match the server
calculation. This ensures the checksums are all consistent.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sat, 2 Apr 2016 16:11:26 +0000 (17:11 +0100)]
siggen: Add checksum recalculation/checking code
In theory all the information to recalcuate the task signatures was written
into the siginfo/sigdata files. In reality, some of the information was
written into the filename.
Firstly this patch duplicates that info into the file itself just for easy
of use since its small.
Secondly, we abstract out the existing "calculate the checksum" code for
the taskhash, and add a function to calculate the bashhash based on the
informaiton within the file.
Finally, we call these functions when we're writing out the data to check
that the data we're writing is consistent. I've found a couple of places
it wasn't and its good to know about these in advance, rather than having
a siginfo/sigdata file which a given hash in its filename but a contents
which give a different result.
This should all combine to avoid a certain class of checksum bugs making
it into world, and identifying problems in advance.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
