toaster: base Only show change project icon when > one project
In the new build button, it only makes sense to change
the selected project when there is more than one project
in the Toaster instance. If the number of projects is 1,
we hide the change project icon.
This patch adds a simple UI-less POST endpoint, where
bitbake_eventlog.json files generated by a bitbake run can be
uploaded to the running toaster instance for insertion into
the database.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toastergui: implement UI changes to allow file download
This patchset adds download links in the build analisys pages
if toaster runs in managed mode. This allows the user to access
data directly from the web interface.
[YOCTO #6837]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
This patch adds a "-w/--write-log" option to bitbake
that writes an event log file for the current build.
The name of the file is passed as a parameter to the "-w"
argument. If the parameter is the empty string '', the file
name is generated in the form bitbake_eventlog_DATE.json,
where DATE is the current date and time, with second precision.
The "-w" option can also be supplied as the BBEVENTLOG
environment variable.
We add a script, toater-eventreplay, that reads an event
log file and loads the data into a Toaster database, creating
a build entry.
We modify the toasterui to fix minor issues with reading
events from an event log file.
Performance impact is undetectable under no-task executed builds.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Mon, 20 Jan 2014 09:39:34 +0000 (09:39 +0000)]
toasterui: add extra debug and development infos
We update and add logs throughout the code in order to help
with development. The extra logging is turned off by default,
but it can be enabled by using environment variables.
All logging happens through the Python logging facilities.
The toaster UI will save a log of all incoming events if the
TOASTER_EVENTLOG variable is set.
If TOASTER_SQLDEBUG is set all DB queries will be logged.
If TOASTER_DEVEL is set and the django-fresh module is available,
the module is enabled to allow auto-reload of pages when the
source is changed.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Scott Rifenbark [Fri, 15 Aug 2014 06:01:49 +0000 (09:01 +0300)]
bitbake-user-manual-metadata.xml: Added [eventmask] flag information.
Reported-by: Laszlo Papp <lpapp@kde.org> Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 8 Dec 2014 16:38:14 +0000 (16:38 +0000)]
runqueue: Fix 100% cpu use after keyboard interrupt
After Ctrl+C is pressed to interrupt bitbake, it loops continually, running
at 100% cpu. This patch selects on the correct file descriptors resolving
the excess cpu usage.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 8 Dec 2014 16:37:26 +0000 (16:37 +0000)]
data: Handle BASH_FUNC shellshock implication
The shellshock patches changed the way bash functions are exported.
Unfortunately different distros used slightly different formats,
Fedora went with BASH_FUNC_XXX()=() { echo foo; } and Ubuntu went with
BASH_FUNC_foo%%=() { echo foo; }.
The former causes errors in dealing with out output from emit_env,
the functions are not exported in either case any more.
This patch handles things so the functions work as expected in either
case.
[YOCTO #6880]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Mon, 8 Dec 2014 10:50:25 +0000 (10:50 +0000)]
event: fix resetting class handlers object
If you don't explicitly specify to use a global variable when doing an
assignment, you will be setting a local variable instead, which means
this function wasn't working at all. It explains some odd behaviour we
have seen in the layer index where event handlers were sometimes
bleeding into other contexts where they should not have been.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Mon, 8 Dec 2014 10:50:24 +0000 (10:50 +0000)]
event: add a means of filtering events internally
When using external tinfoil-based utilities, it is useful to be able to
turn off most of the event handlers; for example sstate_eventhandler
doesn't like being sent events for any recipe which has been skipped.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Mon, 8 Dec 2014 10:50:23 +0000 (10:50 +0000)]
fetch2: add means of disabling SRC_URI checksums
If we're fetching outside of the context of a recipe, it's handy to be
able to disable checksum functionality so you don't get a meaningless
warning about the signatures being missing.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Mon, 1 Dec 2014 20:50:14 +0000 (20:50 +0000)]
data: rename defaultval to _defaultval
The defaultval field is intended to be internal and the only use of that field
outside of data.py is to skip over it when iterating over a value's flags.
For clarity and convenience, rename the field to _defaultval so that it is
considered internal and not exposed through the data API.
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 27 Nov 2014 14:59:21 +0000 (14:59 +0000)]
cooker: Allow featureset in error state
Currently, if an invalid PR service is selected the server will error
with a traceback. This is because its set into the error state and the
setFeature code will then fail since its not in the initial state.
Modifying the featureset in the error state is acceptable, we just need
to ensure we don't trigger a reset, that would happen from whichever
code handles the error.
[YOCTO #6934]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Aníbal Limón [Fri, 28 Nov 2014 01:12:05 +0000 (19:12 -0600)]
fetch/wget: latest_versionstring improvments in searching
Validate if package contain version string if not return the current
version cases for spectrum-fw and corpus recipes.
_check_latest_version return the latest version available don't
take into account the current version previous this only return
the upstream version if it greater than the current version.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package_custom_regex_comp is built with the current package name and
then used to search upstream version this reduces custom regex'es in
sites that have different packages in the same directory.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toaster: remove the word 'project' from layers and machine
Remove the word 'project' from the 'Layers' and 'Machine'
sections of the project page, following feedback from the
QA team, who suggested that the word 'project'
raised questions regarding the scope of the configuration,
Small changes to the project, new project and all layers
pages to ensure consitency in release naming across the
interface.
The changes are:
* In the new project page, change the label 'release version' to
'release'
* In the new project page, sort the releases in the dropdown menu
in ascending alphabetical order
* In the new project page, remove the release name that was
showing between brackets after the release description in the
dropdown menu
* In the project page, make sure the release information
shows the release description field instead of the release name,
to keep consistency with the new project page
* In the all layers page, provide some help text for the branch 'HEAD'
In managed mode, we show the name of the project in the build
notifications of the all builds page. The way we show the
project requires modifying the default padding of the .alert
class. This patch makes sure the padding modification is
only applied in managed mode, i.e., when the project name
needs to be shown.
Alexandru DAMIAN [Tue, 25 Nov 2014 10:12:46 +0000 (10:12 +0000)]
toaster: display Toaster exceptions and other fixes
Changing ToasterUI to log toaster exceptions on a different level than
build errors.
Updating the build dashboard to show Toaster exceptions.
We add extra logging to console for exceptions.
Fixed a problem where packages database entries were created instead of
being looked up in the database, conficting with entries created to
satisfy dependency information.
Toaster now checks for invalid states at startup and performs needed
cleanups.
Removed loading reference to jquery-ui.min.css as we do not have this
file.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Tue, 25 Nov 2014 11:21:53 +0000 (11:21 +0000)]
toastergui: do not show project info in interactive mode
The most recent builds section showed the project name and the Run Again
buttons regardless whenever the Toaster was in interactive or managed
modes. These widgets have no meaning in interactive mode, and this
patch disables the widgets if toaster is not running in Managed mode.
[YOCTO #6776]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Fri, 21 Nov 2014 13:58:51 +0000 (13:58 +0000)]
toaster: use http proxies to fetch data
Under some network configurations http proxies are used
for Internet access. This patch makes Toaster obey
the http_proxy environment variable when fetching
information from layer indexes.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
When you search for a layer and your search returns no results,
we should show you an alert and a link to show back all layers,
as we do in all other tables.
Michael Wood [Fri, 14 Nov 2014 15:44:55 +0000 (15:44 +0000)]
toaster: project page remove additional tooltip for Change pencil
We have a global pencil icon tooltip so avoid two tooltips appearing by
removing them from the project page. This also means it's one less place to
update it.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Michael Wood [Fri, 14 Nov 2014 18:12:20 +0000 (18:12 +0000)]
toaster: libtoaster: Add getProjectInfo utility function
Add a utility function to return a specified project's info/config
This re-uses the existing server code path for the project edit
except that it allows any project id to be used as a parameter
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Michael Wood [Tue, 11 Nov 2014 16:23:58 +0000 (16:23 +0000)]
toaster: Create libtoaster.js with some utility functions
Replace main.js with libtoaster. This can be a place for common
functionality for toaster. Calling it lib... makes it more obvious as
well as helps with code readability due to the namespacing.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Fri, 14 Nov 2014 17:07:06 +0000 (17:07 +0000)]
toastergui: layer name correlation
This patch modifies how layers are identified and matched.
Layers were primarely organized by the source of layer information,
and Releases were separated by both layer git branches and originating
source of layer information. This setup prevented mixing layers from
different sources for a certain release, which didn't match the way
people use Yocto Project / bitbake.
This patch brings name-based indentification, where layers with the
same name are assumed to be equivalent, in the sense of being able
to substitute one another. To facilitate this identification to
humans, layers are differentiated by GIT URI instead of layer sources,
which was a rather arbitrary abstraction.
Additional changes include modification to models in order accomodate
for the new data structure, and to config file loading to match
the new toasterconf.json layout.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Michael Wood [Mon, 27 Oct 2014 19:03:06 +0000 (19:03 +0000)]
toaster: skip virtualenv when searching for django apps
If we are using a virtualenv in the current search path we end up trying
to add modules to the django apps list which do not have the correct
load path or are internal to other modules. Fix this by skipping the
virtualenv path.
[YOCTO #6896]
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Robert Yang [Mon, 17 Nov 2014 10:27:36 +0000 (02:27 -0800)]
bitbake-worker: exit normally when SIGHUP
Fixed:
1) Run "bitbake recipe" in the terminal
2) Close the terminal while building
3) $ ps aux | grep bitbake-worker
There will be many processes, and they will keep the resources (e.g.,
memory), and won't exit unless kill or kill -9.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otavio Salvador [Wed, 12 Nov 2014 10:33:23 +0000 (08:33 -0200)]
bb.codeparser.py: Remove reference for oe.utils.contains
The oe.utils.contains function has been removed from OE-Core metadata
as the references for it has been replaced to use the
bb.utils.contains.
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Brendan Le Foll [Wed, 12 Nov 2014 11:42:39 +0000 (11:42 +0000)]
cooker.py: fix loginfo op being set to an invalid value
loginfo "op" was being set to invalid values in saveConfigurationVar it was
working because set is a python global but append is not. This replaces both
the "op" with strings and removes the crash when calling appendConfigurationVar
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 11 Nov 2014 17:01:09 +0000 (17:01 +0000)]
toastergui: fix XSS injection points in projects page
We close XSS injection points in Projects page.
* modify the json filter to properly escape HTML tags in strings
* enable $sanitize to automatically sanitize dangerous HTML in
user-supplied input
* clean dangerous characters in targets field, as that field contents
will be directly passed to a shell command
Based on the vulnerability discovered and the patch provided by Michael Wood.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toastergui: changes for the Project page, round 3 of reviews
This patch implements the round 3 of reviews for the Project
page, including fixing the time display, fixing the build list
display, with fade-in and fade-out animations, and various
small layout fixes.
[YOCTO #6587]
[YOCTO #6731]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toaster script: webport option and other improvements
We add the "webport=" command line option as to allow starting
the web server on a custom port.
The bitbake server port is now auto-allocated. This is needed
to be able to run multiple toaster environments on a single machine.
We tackle bug 6023 (toaster refusing to start when lock file is present)
by using more specific checks, and automatically recover from
bitbake server down / webserver up error mode.
Command line parameters are now read on both interactive and managed
modes.
The localhost and ssh controllers are updated to work with the modified
toaster launcher script.
[YOCTO #6023]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Martin Hundebøll [Thu, 30 Oct 2014 10:43:24 +0000 (11:43 +0100)]
progressbar: use '/usr/bin/env' in shebangs with python
To support yocto on systems with python3 as default version, scripts
should use /usr/bin/env python in the shebang, as this allows the use of
a fake env to mimic python2 as default version.
This patch simply replaces occurrences of #!/usr/bin/python with
#!/usr/bin/env python and was done with this oneliner:
Signed-off-by: Martin Hundebøll <martin@hundeboll.net> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Aníbal Limón [Wed, 5 Nov 2014 18:10:29 +0000 (12:10 -0600)]
fetch/wget: Add latest_versionstring method
Being able to query whether updated versions of a url are available
is useful, not least for the package reporting system. Since such code
is closely linked to the url type and the url itself, the fetcher
makes a locical place to contain this code.
For wget based urls this means taking upstream directory listings
and searching those for later versions, returning those that are
found.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
Aníbal Limón [Wed, 5 Nov 2014 18:10:28 +0000 (12:10 -0600)]
fetch/git: Add latest_versionstring method
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
Richard Purdie [Fri, 7 Nov 2014 17:47:37 +0000 (17:47 +0000)]
siggen: Fix shared work checksum mismatch/rebuild issues
Similar to the last shared work task signature bug, we've found another
one. Looking at the improved output of diffsigs in this case:
runtaskdeps changed from [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native',
'libgcc-initial_4.9.bb.do_patch:virtual:nativesdk'
] to [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gcc-crosssdk-initial_4.9.bb.do_patch',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native'
]
so we can get a different task hash since libgcc sorts before gnu-config
and gcc sorts after it. We could do with a way of fixing this, the best
I can come up with is to include a single parent directory. Since
recipes are never at the top of any metadata trees I've seen, this
should suffice for now.
I'm planning to burn the concept of shared work within bitbake
and do something at the metadata level in the 1.8 timeframe as its just
too fragile as things stand and hard to fix well.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 7 Nov 2014 17:45:07 +0000 (17:45 +0000)]
siggen: Ensure we output if the ordering of runtaskdeps changes
Order of runtaskdeps is important. If the hashes differ we should print output.
This is complicated by shared work where the filenames themselves can differ,
but the checksum should not.
This fixes a case where two different checksums could show no output with
bitbake-diffsigs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 31 Oct 2014 19:17:02 +0000 (19:17 +0000)]
toastergui: fix invalid build url usage
We need to pass the URL for the build action as javascript
parameter, since the project ID is different for each build,
instead of using the project id of the most-recently-used project.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 31 Oct 2014 19:17:01 +0000 (19:17 +0000)]
toasterui: improvements in data reading
We add improvements in data reading, following
issues discovered in testing.
- elapsed_time is now read from buildstats
- we add safeguards to not fail logging if the build was triggered
with a toaster_brbe configuration, but it's running in
1.6 mode
- added log markups for build finish to let other programs
known when the work is done.
[YOCTO #6833]
[YOCTO #6685]
[YOCTO #6887]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 5 Nov 2014 16:58:13 +0000 (16:58 +0000)]
prserv: Use WAL mode
Ideally, we want the PR service to have minimal influence from
queued disk IO. sqlite tends to be paranoid about data loss and
locks/fsync calls. There is a "WAL mode" which changes the journalling
mechanism and would appear much better suited to our use case.
This patch therefore switches the database to use WAL mode. With this
change, write overhead appears significantly reduced.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 4 Nov 2014 14:02:32 +0000 (14:02 +0000)]
prserv/serv: Ensure sync happens in the correct thread
The sync/commit calls are happening in the submission thread which can
race against the handler. The handler may start new transactions which
then causes the submission thread to error with "cannot start a
transaction within a transaction".
The fix is to move the calls to the correct thread.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 14 Oct 2014 12:57:33 +0000 (13:57 +0100)]
toastergui: provide download file capability
We add, for the localhost environments, the capability to
download build artifacts. This is a pontentially dangerous API,
because it gives unrestricted read access to the build environment
file system - do not expose the functionality directly to the
web layer, but use filtering/translation code, such as
exemplified in the build_artifact view.
The capability for remote build environments is dependent
on bug 6835, as to use the collect storage as intermediary
storage for serving files.
[YOCTO #6834]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Mon, 13 Oct 2014 16:10:39 +0000 (17:10 +0100)]
toaster: change startup parameter passing to avoid race
We avoid a race between the setting the TOASTER_BRBE variable
and reading the variable in toaster ui by supplying the variable
at server startup time through the toaster.conf post-read file.
Additional small changes are included, including marking the
build request with the environment id of where the build took place.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
The default django URL validator marks git URLs as not valid,
so we change the default validators to a URLValidator-derived
that adds git and ssh protocol schemas and inhibits the
frontend URL validator.
We update the branch names based on the white-listed
branch names in the Releases name.
Adds proper class string to ease usage in admin section.
Other small improvements.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Conflicts:
bitbake/lib/toaster/orm/admin.py
toaster: use modified validators for git url fields
The default django URL validator marks git URLs as not valid,
so we change the default validators to a URLValidator-derived
that adds git and ssh protocol schemas and inhibits the
frontend URL validator.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Tue, 30 Sep 2014 12:45:51 +0000 (13:45 +0100)]
toastergui: protect variable value reads
We make sure we don't throw an exception when reading
non-existent variables from the database, and we return
empty data that can be displayed without ill effects.
This fixes the Configuration page on empty builds.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Tue, 30 Sep 2014 12:31:57 +0000 (13:31 +0100)]
toastergui: refactor objects get in views.py
We refactor the objects.filter()[0] pattern to the proper
Django QuerySet usage of objects.get() to make sure we
get one and only one object for a query.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toaster: fix Project page in order to trigger builds
This patch rewrites the Project page and the additional
infrastructure in order to fix a bug that makes triggering
builds through UI impossible, and to introduce data feeds
for suggestions for the user.
Alexandru DAMIAN [Tue, 30 Sep 2014 12:57:14 +0000 (13:57 +0100)]
toastergui: update breadcrumb in analysis pages
If we are in managed mode, and we inspect a build that comes
from a project, we modify the breadcrumb as to display
the project from where the build originated.
[YOCTO #6776]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Mon, 29 Sep 2014 14:35:59 +0000 (15:35 +0100)]
toasterui: save build id in build request on first chance
We change the saving of the build id to the build request as
soon is the build is created, as to allow for a consistent
display of build data while build is in progress.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
