Ed Bartosh [Tue, 29 Sep 2015 04:45:20 +0000 (21:45 -0700)]
toaster: implement decorator for REST responses
Implemented xhr_response decorator to decorate responses
from REST methods into Django HttpResponse objects.
This decorator should reduce amount of repeated code in
REST methods and make them more readable.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 29 Sep 2015 04:45:19 +0000 (21:45 -0700)]
toaster: add toggle for enabling image customisation feeature
This feature is currently under heavy development and should be used
with caution.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 29 Sep 2015 04:45:18 +0000 (21:45 -0700)]
toaster: Add CustomImageRecipe model
This model lists custom image recipes for the project.
It is populated when new custom image is created.
It holds reference to the base recipe and list of packages
included into custom image.
For CustomImageRecipes the packages will be copied in and
therefore not associated with a build so Remove the requirement
for the package to have a Build.
co-author: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 29 Sep 2015 04:45:17 +0000 (21:45 -0700)]
toaster: ToasterTable remove unused class definition
Remove ToasterTemplateView as this isn't used by anything.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 29 Sep 2015 04:45:16 +0000 (21:45 -0700)]
toaster: add nocache option to the ToasterTable widget
Useful for skipping the cache mechanism when debugging.
Simply append nocache=true to the end of the url with the table in
it.
Also adds a debug message if the table is using cached data.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 29 Sep 2015 04:45:15 +0000 (21:45 -0700)]
toaster: widgets ToasterTable Add more info to search field exception
If we get a search fields exception then also print out the model name
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 29 Sep 2015 04:45:14 +0000 (21:45 -0700)]
toaster: widgets ToasterTable add logger to notify when cache hit
Useful for debugging to remind you if you're hitting a cache or not
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 29 Sep 2015 04:45:13 +0000 (21:45 -0700)]
toaster: create custom layer and recipes for Image customisation
When building customised recipes toaster creates custom layer directory
and puts layer.conf and custom recipes to it.
[YOCTO #8075]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 29 Sep 2015 04:45:12 +0000 (21:45 -0700)]
toaster: tables Move the title and name into the widget
For historical reasons this was being set in the urls definition. We
can set this in the actual definition of the table and defaults in the
widget.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 29 Sep 2015 04:45:11 +0000 (21:45 -0700)]
toaster: make a workaround for old style index
For a new style indexes 'inherits' field is used to determine if
recipe is an image recipe.
As old style indexes don't have 'inherits' field this can be
guessed from recipe name. Let's consider recipe an image recipe
if recipe name contains '-image-'.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/serv.py: Better messaging when starting/stopping the server with port=0
When starting the server using port=0, the server actually starts with a
different port, so print a message with this new value. When stopping the
server with port=0, advise the user which ports the server is listening to,
so next time it tries to close it, user can pick up the correct one.
[YOCTO #8560]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/serv: Close the DB connection out of class destructor
When launching the PR server daemon, the PRData __del__ function was being
called (no reason found yet) where the DB connection closed, thus following
PR updates were not getting into the DB. This patch closes the connection
explicitly, not relaying on the __del__ function execution.
Closing the connection in turn causes all WAL file transactions to be moved
into the database (checkpoint), thus effectively updating the database.
[YOCTO #8215]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/serv: Start/Stop daemon using ip instead of host
In cases where hostname is given instead of an IP (i.e. localhost
instead of 127.0.0.1) when stopping the server with bitbake-prserv --stop,
the server shows a misleading message indicating that the daemon was not
found, where it is actually stopped. This patch converts host to IP values
before starting/stopping the daemon, so it will always work on IP, not on
hostnames, avoiding problems like the latter.
[YOCTO #8258]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Markus Lehtonen [Mon, 21 Sep 2015 13:07:57 +0000 (16:07 +0300)]
tinfoil: remove logging handler at shutdown
Otherwise the logger gets multiple handers (and the user get duplicate
logging output) if another tinfoil instance is initialized after one is
shut down().
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the 'latest builds' section of the all builds page,
completed builds show the time they completed at. Builds
in progress should not display such time, since they
haven't completed yet.
This patch removes the time information that was showing
for builds in progress, whatever that time actually was.
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Tue, 22 Sep 2015 02:42:35 +0000 (19:42 -0700)]
toaster: Add fake entry to Target_File for filesystem root
The files-in-image.txt file is produced by bitbake after an
image is created, listing all the files in the image.
However, this list doesn't include the root directory ('/').
buildinfohelper.py then tries to construct the filesystem
tree from this file, assuming that every directory apart from
the root directory (which is special-cased) can be assigned
a parent. But because the root directory isn't listed in
files-in-image.txt, an object for the root directory is never
created.
The direct subdirectories of the root ('./bin', './usr' etc.)
then can't be assigned a parent directory, as the object
representing the root directory doesn't exist. This
results in a Target_File lookup error and causes the
directory listing page to fail.
Fix this by creating a fake entry for the root directory
in the Target_File table, so that the direct subdirectories
of / can be assigned a parent. Note that it doesn't matter
that the root is faked, as its properties are never shown
in the directory structure tree.
[YOCTO #8280]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Tue, 22 Sep 2015 02:45:24 +0000 (19:45 -0700)]
toaster: layerdetails Fix back button tab behaviour
This completes the behaviour fix of the back button in the layerdetails
page as we not only have parameters in our history we also have the hash
to indicate which tab is active. As we pop our history we need to show
the corresponding tab.
[YOCTO #8252]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ke Zou [Wed, 23 Sep 2015 14:18:11 +0000 (15:18 +0100)]
toaster: UI test improvements
* Add decorator in logging mechanism
* Add more debug information
Signed-off-by: Ke Zou <ke.zou@windriver.com> Signed-off-by: Stanciu Mihail <stanciux.mihail@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
brian avery [Wed, 23 Sep 2015 14:05:40 +0000 (15:05 +0100)]
toaster: support selenium testing from mac OS X
* added a mac section to the cfg file
* added mac specific screenshot code
Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 22 Sep 2015 09:34:55 +0000 (10:34 +0100)]
toaster: add 2 UI tests
Tested that UI shows task names for the builds in
both all-builds and projectbuilds views.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 22 Sep 2015 09:34:54 +0000 (10:34 +0100)]
toaster: change UI to show tasks
Changed toaster UI to show tasks if they're specified for the
builds and use them when restarting builds.
[YOCTO #7442]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 22 Sep 2015 09:34:53 +0000 (10:34 +0100)]
toaster: don't re-create Target objects
Due to re-creating Target objects from bitbake events task information
stored in original objects is lost.
There is no valid reason to remove existing objects. It's safer to query
them instead of re-creating as original object contain more information
than events coming from bitbake.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 22 Sep 2015 09:34:52 +0000 (10:34 +0100)]
toaster: store task name in Target objects
Information about a task is not stored in Target objects.
This makes it impossible to correctly operate with the builds
where task is specified.
Storing taks name is an enabler for other fixes in UI and backend
related to restarting builds.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently this module is dereferencing errno.ENOENT but the python module "errno"
is not imported, which causes a crash when fetching from a git repository.
Signed-off-by: Romain Perier <romain.perier@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 24 Jul 2015 15:05:20 +0000 (16:05 +0100)]
toaster: hide irrelevant builds in the project builds view
This patch fixes the project builds view so it doesn't show
"in progress" builds or builds for other projects.
Note that this also modifies the "all builds" view to use
the same queryset filtering as the project builds. This is
to avoid excluding "in progress" builds more than once, which
is what was happening before.
The patch also has a minor change to ensure that when
displaying the project builds page, only builds for that
project are in the results.
The queryset filtering is now split over several lines so
you can see what's going on.
[YOCTO #8236]
[YOCTO #8187]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 18 Sep 2015 07:55:47 +0000 (07:55 +0000)]
data_smart: Ensure OVERRIDES dependencies account for contains()
The dependencies of OVERRIDES were not including DEFAULTTUNE in OE-Core.
This is pulled in by a bb.utils.contains() reference which the override
dependency tracking code wasn't accounting for.
This patch ensures we do track contains references too.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 16 Sep 2015 20:59:44 +0000 (21:59 +0100)]
data_smart: When considering OVERRIDE dependencies, do so recursively
Sadly its not enough to consider the dependencies of OVERRIDES, we
need to resolve their dependencies and so on recursively. If we don't
do this, some variable can be changed and the resulting data store is
incorrect.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 16 Sep 2015 20:57:55 +0000 (21:57 +0100)]
data_smart: Expand overrides cache recursively
If the values that make up OVERRIDES are themselves overridden,
we end up into some horrible circular logic. Unfortunately some
metadata does depend on this functionality.
e.g:
DEFAULTTUNE_virtclass-multilib-xxx = Y
which changes TUNE_ARCH
which changes TARGET_ARCH
which changes OVERRIDES
As a solution, we iterate override expansion until the values don't
change. If we iterate more than 5 times we abort and tell the user to
report the issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.fetch2.{git, hg}: remove tarball if it needs updating
We were maintaining state in the form of ud.repochanged to determine whether
we need to write the tarball in the case where the tarball already exists, but
this state is maintained only in memory. If we need to update the git repo,
set ud.repochanged to True, and then are interrupted or killed, the tarball
will then be out of date.
Rather than maintaining this state, simply remove the out of date tarball when
we update the git repo, and it will be re-created with updated content in
build_mirror_data.
[YOCTO #6366]
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Fri, 4 Sep 2015 09:37:47 +0000 (10:37 +0100)]
toaster: Simplify redirects when build page parameters are missing
A RedirectException is used to redirect the client to the
correct page when their original request is missing the
required parameters (page, orderby etc.). However, the code
is difficult to follow.
Rather than catching RedirectExceptions and rethrowing them with
different view URLs, ensure that the RedirectException has the
correct URL in it when thrown by passing the original page
name to the _build_list_helper() method.
Modified from an original patch by
David Reyna <david.reyna@windriver.com>.
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Fri, 4 Sep 2015 09:37:46 +0000 (10:37 +0100)]
toaster: Fix date range pickers on the project builds page
These were referring to the old HTML elements with ids
"date_from_created" and "date_from_updated", but their
ids have changed to "date_from_started_on" and
"date_from_completed_on". This meant that they weren't
functional.
This fixes the references.
Modified from an original patch by David Reyna
<david.reyna@windriver.com>.
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Fri, 4 Sep 2015 09:37:45 +0000 (10:37 +0100)]
toaster: Show correct builds count on project pages
The counter for completed builds on the project pages
includes builds in progress.
Instead use the completedbuilds queryset to count the
number of completed builds for display on project pages.
Modify how the completedbuilds queryset is constructed so
it only excludes builds with status "in progress".
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toaster: hide irrelevant builds in the project builds view
This patch fixes the project builds view so it doesn't show
"in progress" builds or builds for other projects.
Note that this also modifies the "all builds" view to use
the same queryset filtering as the project builds. This is
to avoid excluding "in progress" builds more than once, which
is what was happening before.
The patch also has a minor change to ensure that when
displaying the project builds page, only builds for that
project are in the results.
The queryset filtering is now split over several lines so
you can see what's going on.
[YOCTO #8236]
[YOCTO #8187]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mark Hatle [Thu, 10 Sep 2015 23:15:27 +0000 (18:15 -0500)]
tests/fetch.py: Fix recursion failure in url mapping
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list. This permits us to re-evaluate the list
while excluding all matches that have previousily occured.
Without this fix, adding this test results in a failure:
RuntimeError: maximum recursion depth exceeded in cmp
Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Fri, 11 Sep 2015 20:57:30 +0000 (13:57 -0700)]
toaster: Don't def a function for each call to build_artifact()
Cache the mimetype object and only define the function for
getting a mimetype once.
Also ensure that filemagic is listed as a requirement of
toaster. Doing this also means we can remove the code
which tries multiple different "magic" libraries, as we know
we have the right version available.
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Fri, 11 Sep 2015 20:57:29 +0000 (13:57 -0700)]
toaster: Avoid unnecessary local to local copy of cooker log
The cooker log is copied from its original (bitbake) location
to an artifact directory chosen by the user (chosen when
checksettings.py runs as part of the managed mode;
it's one of the annoying questions you're asked at startup).
The copy happens as part of the runbuilds script run, which
is started in a loop from the toaster startup script
in managed mode.
When a user requests the log for a build via toaster, they
are getting the log which has been copied to the artifact
directory, not the original bitbake log.
This works for the managed case, where the runbuilds command is
running in a loop and copying log files for completed builds to
the artifact directory. However, in analysis mode, there are two
problems:
1. checksettings isn't run, so the artifacts directory isn't
set. toaster is then unable to figure out where the log files
should have been copied to.
2. The log files aren't copied to the artifacts directory
anyway, as runbuilds isn't running in analysis mode.
To fix this, just point the user to the local bitbake log file
in its original location. This avoids the copy step, and means
we can remove a whole question from the toaster startup sequence,
as we no longer need an artifact directory.
The only downside to this is that it's not appropriate for
remote bitbake servers. We will need to revisit this and
possibly reinstate the copy step once we have to reconcile
local and remote builds and make their logs available in
the same way.
[YOCTO #8209]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Fri, 11 Sep 2015 20:57:28 +0000 (13:57 -0700)]
toaster: Read correct cooker log path from toasterui
The BB_CONSOLELOG variable changes by the time we read it in
BuildInfoHelper. This means that the log file location we
are using is incorrect, so the links to the cooker logs don't
work.
Instead, read it at the point when the BuildStarted event occurs
in toasterui. The BB_CONSOLELOG variable has the correct value
here, so pass that to BuildInfoHelper.
[YOCTO #8209]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
brian avery [Mon, 14 Sep 2015 15:45:40 +0000 (16:45 +0100)]
toaster: delete multiple builds cleanup
This patch cleans up the multiple delete. It:
1) skips build id's that don't exist rather than giving a traceback.
2) let you pass in the ids as a space separated list
3) fixes the usage to match the space separated list format
[YOCTO #7726]
Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 16 Sep 2015 20:44:20 +0000 (21:44 +0100)]
cache: Handle spaces and colons in directory names for file-checksums
If there is a space in a directory name containing a file in file-checksums
(e.g. from a file:// url), you currently get tracebacks from bitbake. This
improves the code to handle colons and spaces in the file-checksums names
since it possible to figure out the correct names.
[YOCTO #8267]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 9 Sep 2015 21:44:37 +0000 (22:44 +0100)]
bitbake-worker: Ensure pipe closure doesn't crash before killpg()
If the pipe is closed, we want to ensure that we kill any child processes
by triggering the sigterm handler before we exit. This code does that,
hopefully avoiding the remaining process left behind issues on the autobuilder.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Wed, 9 Sep 2015 02:39:44 +0000 (19:39 -0700)]
toaster: increase waiting time
Increased waiting time for toasterUI from 10 to 25 seconds. Bitbake
takes longer time to start, so toaster should wait longer.
[YOCTO #8240]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Changes the breadcrumb to provide access to either
the project builds or the project configuration, as
appropriate
* Changes the left navigation in the project configuration
to reflect the hierarchical relationship between the
basic configuration and all other configuration pages
* Changes the left navigation in the build history to bring
it in line with the changes in the project configuration
This way the breadcrumb explicitly exposes the hierarchy
of the application, which is its correct behaviour, making it
easier to move around within Toaster.
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 8 Sep 2015 22:37:34 +0000 (23:37 +0100)]
bitbake-worker: Handle SIGKILL of parent gracefully
If we SIGKILL cooker (the parent process), ensure the worker notices
and shuts down gracefully. To do this:
* trigger the sigterm handler if the parent exits
* ensure broken pipe writes don't trigger backtraces which interfer with
other exit work
* notice if our command pipe is broken due to EOF and sigterm if so
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 8 Sep 2015 22:32:07 +0000 (23:32 +0100)]
prserv: SIGTERM and deamonization fixes
Worryingly, if you SIGKILL the bitbake cooker, an autostarted PR server
will remain behind. It turns out there are a few things we should do:
* The PR service doesn't need to daemonize when started from cooker,
it just complicated the process lifecycle. Add a fork() method
to handle this and use the non-daemon mode for the singleton.
* Reset the sigterm and sigint handlers. Bitbake cooker installs its
own which we inherit meaning PR server was ignoring SIGTERM. Installing
our own handlers which include a sync makes most sense here. Since
we're in the code, make it sync the database on SIGINT.
* Use the new bb.utils.signal_on_parent_exit() call so that we get a
SIGTERM when the parent (usually cooker) exits and we can shutdown
too. Alternatives would be having an open pipe or polling
os.getppid() for changes but this seems more effective.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 8 Sep 2015 22:31:13 +0000 (23:31 +0100)]
utils: Add signal_on_parent_exit() function
Add a new bb.utils.signal_on_parent_exit() function so that a process
can register to recieve a signal when the parent dies. There is no
POSIX standard for this and the implementation is Linux specific.
Alternatives would be having an open pipe or polling os.getppid()
for changes but this seems more effective and less invasive to most
of bitbake's code structure.
We need to be able to determine when parents die to ensure child
processes stop running in a variety of circumstances to avoid
locks being held and ensure clean shutdown.
Roughly based on https://gist.github.com/evansd/2346614
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 7 Sep 2015 15:48:16 +0000 (15:48 +0000)]
server/process: Handle SIGTERM more gracefully
Currently if you send a SIGTERM to the bitbake UI process, the system basically
hangs if tasks are executing. This is because the server process doesn't
actually try any kind of shutdown before exiting.
This patch trys executing a stateForceShutdown command first, which is
enough to stop any active tasks before the system exits.
I also noticed that terminate can execute multiple times, once at SIGTERM
from the handler and once from the real exit. Double execution leads to
stack traces and potential hangs (writes to dead pipes), so ensure
the code only can run once.
With these fixes, bitbake much more correctly deals with SIGTERM to the
UI process.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ETA we show for builds in progress is woefully
inaccurate. In the 1.8 release we replaced it with
the % of tasks completed. Somehow, we regressed to
the ETA, so bringing the task % back.
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.fetch: don't remove the clone when an update fails
When our clone exists, but is out of date, and the attempt to update it fails,
we don't necessarily want to remove the entire clone, particularly if it's
a large repository.
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.fetch: handle checksums consistently for mirrors
If the main fetch method doesn't support checksums, the user will not be
defining them in the recipe, so we don't want to check them for
premirrors/mirrors either. This ensures that we never error due to missing
checksums on a git mirror tarball.
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using an absolute file URI, there's no host, and the path starts with
'/', the dir under ${DL_DIR}/git2/ ends up starting with '.', so is hidden.
Remove any leading '.' to fix this.
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alex Franco [Wed, 2 Sep 2015 20:11:30 +0000 (15:11 -0500)]
cooker/runqueue: Allow bitbake commands starting with do_
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).
[YOCTO #7818]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 3 Sep 2015 14:20:05 +0000 (15:20 +0100)]
event/server: Add _uiready flag to handle missing error messages
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.
The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.
This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.
[YOCTO #8239]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake utils' movefile is now prone to malform the destination
file with duplicated file name strings. Fixing it to force a file
name append iff the dest argument is a dir not a file name
Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Thu, 3 Sep 2015 00:25:12 +0000 (17:25 -0700)]
toaster: Add tests for pages which show the default project
Test that the correct landing page redirect is applied, depending
on the state of the default project.
Test that the default project is only shown on the /projects page
if it has at least one build.
[YOCTO #7932]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Thu, 3 Sep 2015 00:25:11 +0000 (17:25 -0700)]
toaster: Fix test for projects page JSON
The tests expect data to be returned in a particular format,
which doesn't include is_default or the num_builds fields
added to support default project improvements.
Add the extra fields so the tests pass again.
[YOCTO #7932]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Thu, 3 Sep 2015 00:25:10 +0000 (17:25 -0700)]
toaster: Exclude default project unless it has builds
Don't include the default "command line builds" project in
the projects view unless it has builds associated with it.
[YOCTO #7932]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Thu, 3 Sep 2015 00:25:09 +0000 (17:25 -0700)]
toaster: Only redirect to projects page if user has added projects
The landing page currently redirects the user if there are any
projects in the db. Because we now always have at least one
(the default one added by a migration), we always get the redirect.
Change this so that when the user hits the landing page,
we only redirect them to the projects page if there is at least
one user-added project and there are no builds.
[YOCTO #7932]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Thu, 3 Sep 2015 00:25:08 +0000 (17:25 -0700)]
toaster: Improve how default project is identified and fetched
Command line builds are associated with a "default project"
(as we currently require a build to have a project). This
acts as a container for builds initiated outside Toaster.
Currently, this project is marked as the default by its ID
being 0. However, this doesn't work with MySQL, as MySQL
won't allow 0 in a foreign key which references an
autoincrement field.
Instead, use an is_default field to track the default Project
for builds initiated outside Toaster.
Add a method to fetch this default project, rather than fetching
a project with a magic ID.
Add this default project in a migration, rather than as a side
effect of a get_or_create() style method.
Also ensure that builds always have a project explicitly assigned
to avoid any magic with a build's project foreign key defaulting to
0 (as it no longer does).
[YOCTO #7932]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Elliot Smith [Thu, 3 Sep 2015 00:25:07 +0000 (17:25 -0700)]
toaster: Make 0021 migration compatible with MySQL
Two issues prevent this migration from working correctly
with a MySQL back-end:
1. MySQL won't allow a default value to be set for an
AutoField, which is what the migration tries to do
for project_id ("ValueError: The database backend does not accept
0 as a value for AutoField.")
2. When migrations are applied to a MySQL back-end, Django
(via South) attempts a dry run of the migration first: it
applies the forward migration then rolls it back. However, this
migration raises an exception on roll back, which causes the
whole series of migrations to fail.
This commit fixes both issues.
[YOCTO #7932]
Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want addtask to be able to bring back a deleted task, but we don't want its
previous dependencies to come back with it, so rather than marking a task as
deleted and then skipping tasks marked as such, actually delete the task and
its dependency information in deltask.
While we're in that part of the code, also fix a couple 'not foo in bar'
instances to 'foo not in bar', which is preferred in python.
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Mon, 17 Aug 2015 16:47:47 +0000 (17:47 +0100)]
toastergui: Add frontend javascript unit tests
Use Jquery's Qunit tests to create some unit tests for javascript
components used in toaster.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toaster: fix show / hide columns in all builds table
The classes errors_no and warnings_no were removed from the
td tags in the builds.html template. That broke the show / hide
columns functionality, which would hide only the table heading,
but not the data cells.
This patch brings back those classes so that you can hide
and show the errors and warnings information.
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bian Naimeng [Fri, 28 Aug 2015 15:37:38 +0000 (16:37 +0100)]
toaster: Support environments which have proxies set
In an environment with a proxy which requires authentication, e.g. with
$http_proxy = 'http://user:password@ip:port', the following error
occurs when running Toaster:
EE: Using proxy http://user:password@ip:port
EE: could not connect to 'url', skipping update: 'error message'
This prevents Toaster from fetching layer, recipe and machine information
from remote repositories.
This patch allows Toaster to use the proxy settings from the
environment for HTTP/HTTPS requests.
Signed-off-by: Bian Naimeng <biannm@cn.fujitsu.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
David Reyna [Tue, 25 Aug 2015 08:32:15 +0000 (09:32 +0100)]
toaster: update directory tree expansion
The ajax query that fetches the directory data on an expand request
is now returning the response data already in an object form. so the
parseJSON() call is no longer needed (and is currently returning a
parse error).
[YOCTO #7810]
Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Wed, 26 Aug 2015 12:50:47 +0000 (13:50 +0100)]
lib/bb/cooker: add interrupted flag to BuildCompleted event
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the Toaster screenshot we use in the landing page
to make sure it cannot be confused with the real interface.
Also, remove the MANAGE conditional we were using to set the
screenshot. We no longer have different applications for the
two modes, so it should no longer be needed.
[YOCTO #7743]
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Mon, 24 Aug 2015 14:00:22 +0000 (15:00 +0100)]
lib/bb/main: avoid importing all server/UI modules on every execution
We're importing the server and UI modules in order to check they are
valid, but it turns out that that has some nasty side-effects. We don't
actually need to do this except when --help is passed or the module
doesn't exist, so rearrange the code so that we only do the module
listing in those two cases.
Additionally, let's just go ahead and catch all errors on import; we
really don't care to have them cause a failure now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Randy Witt [Thu, 20 Aug 2015 21:00:36 +0000 (14:00 -0700)]
main: Handle RuntimeError exception in list_extension_modules
This exception was triggered in toaster from recent changes and is completely
breaking the whole of bitbake. Add the exception to the list so at least only
toaster is affected.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:29:00 +0000 (17:29 +0100)]
toastergui: fix projectbuilds page
This patch fixes the redirection projectbuilds page and
the template layout in the projectbuilds page.
* The _build_list_helper now returns an empty RedirectException
that is properly customized by the caller and re-raised
to achieve redirection to the original page (poor man's overloading)
* The template for ProjectBuilds is updated as to properly display
Build objects instead of BuildRequest objects.
[YOCTO #7995]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:59 +0000 (17:28 +0100)]
toastergui: do not show in-progress builds in the table
This patch fixes the all-builds table as not to show
in-progress builds.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:58 +0000 (17:28 +0100)]
toaster logger: fix pylint issues
This patch fixes pylint issues in the toaster build data logger,
toasterui. The following types of warnings are touched here:
* fixing imports
* unused variables are set to _
* logger calls now use lazy evaluation instead of formatting
the string
* correct whitespace identation
* removes unneeded "pass" statements, and extra parantheses
* disable specific pylint warnings when decideing to override
them
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru Damian [Tue, 18 Aug 2015 16:28:57 +0000 (17:28 +0100)]
toaster: fix pylint errors
Prompted by issues discovered during running pylint on
the toaster sources, this patch fixes:
* missing import in toaster ui
* improper call of function in toaster ui (logger.debug)
* improper function definitions in bbcontroller
* invalid references to objects in bldcontrol.models
* proper initialization of object fields in ToasterTables
Also inhibiting specific pylint errors in files where
the problems are mis-identified.
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:56 +0000 (17:28 +0100)]
toaster: fix updates on failed build requests
The patch to fix the original mistake is wrong, in the sense
that a constant from the BuildRequest class is used on the build
object.
Fixing the patch to bring in the correct constant in.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:55 +0000 (17:28 +0100)]
toaster: replace raising Exceptions in loadconf
This patch replaces throwing Exceptions in toaster loadconf
command with proper explicit error messages. This allows
the user to properly understand and debug what went wrong when
loading a config file.
Additionally we change a bit the logic around auto-detecting relative
giturl handling so the user gets proper error messages when
trying to load an invalid toasterconf.json file.
[YOCTO #7945]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:54 +0000 (17:28 +0100)]
toaster: do not stop data import on bad data
The layer data update in Layer source needlessly stops
processing by throwing an Exception when bad layer data cannot
be inserted in the database (e.g. invalid dependencies).
This patch changes the behaviour to ignore bad data, and
continue with data import after detailed logging through the
logging system.
[YOCTO #7955]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:53 +0000 (17:28 +0100)]
toastergui: rename variables in views.py
Renaming various variables in the views.py file to make
the code more readable, and prevent overshadowing of global
variables with local variables that have a different name.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:52 +0000 (17:28 +0100)]
toaster: use loggers instead of prints
Switching debugging from using print statement to using
loggers, as it uses the logging infrastructure to process
the messages.
The messages are logged with the "toaster" logger which
is defined in the toastermain/settings.py.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:51 +0000 (17:28 +0100)]
toaster: models field initialization
As prompted by pylint, the object fields need to be
initialized properly, so this patch fixes that.
Also adds some casts to int, because, sometimes, the IntegerField
is not returning an int, but a string.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 18 Aug 2015 16:28:50 +0000 (17:28 +0100)]
toaster: improve exception handling
This patch improves exception handling, fixing issues
as prompted by pylint, including exception renaming.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improving the TEMPLATECONF detection by verifying the return
code and dumping extra debug information in case of exception.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
