toastergui: remove time link for failed build requests
Failed build requests do not have a build time page where
you can look at the time spent on each task. Therefore,
their build time information cannot be a link to such
a page.
Alexandru DAMIAN [Wed, 28 Jan 2015 13:18:09 +0000 (13:18 +0000)]
toaster: new layer checkout logic
This patch implements a new layer checkout logic that brings more
flexibility in getting layers under different commits work with
Toaster.
The new hibrid logic will checkout separately each layer and commit id;
the task execution will be delegated to the checkedout bitbake, but the
data logger will be executed from the current toaster version as to
bring in enough data to sustain the updates in UI models.
[YOCTO #7171]
[YOCTO #6892]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Richard Purdie [Thu, 29 Jan 2015 14:35:37 +0000 (14:35 +0000)]
data_smart: Don't use mutable objects as default args
These only have one instance created which means your subsequent datastores
can contain echos of previous ones. Obviously this is not the behaviour
we want/expect. It doesn't affect bitbake too badly as we only have one
datastore, it does massively potentially break our selftests though.
Thanks to Tim Amsell for pointing out the now obvious problem!
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 23 Jan 2015 17:36:14 +0000 (17:36 +0000)]
toastergui: update project build listing
We update the build listings in the project mode to enable
proper display and selection of build requests that do not have
an actual build object because the bitbake process did not start.
We add a page to display error details for build requests that
did not start a build.
Fixing errors and misspelling in build sections.
Sorting by "timespent" is disabled for build-listing pages.
toastergui: add 'show all' link to no results page
When you search for targets or machines in the layer details
page and your search returns no results, you should
see a 'show all' link that you can click to clear the
search and display all table entries.
toastergui: Generic message for the layer details empty states
In the layer details page, change the message you see when
the number of targets or machines provided by a layer is
0, either because that's what's reported by a layer index
instance, or because Toaster does not have any information
about the layer.
The new message is more generic, in order to fit layers from
all layer sources.
toaster: apply common styles to right hand column of detail pages
Layer, recipe and package details pages have a right column
with additional information. This commit standardises the
formatting of the content in that right column across
all these pages by creating a new css class .item-info
in default.css
The layer details page should have the same layout as all
other detail pages (recipes, packages). For this to happen,
the basebuilddetailpage.html and baseprojectpage.html
templates should use the same layout, and so should the
layerdetails.html and recipe.html templates.
This patch also capitalises the string 'All layers' in the
breadcrumb of layerdetails.html as all other breadcrumb elements.
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Richard Purdie [Wed, 21 Jan 2015 16:07:19 +0000 (16:07 +0000)]
cooker: Rename event_data -> expanded_data
event_data would be better named expanded_data, then we can widen its scope
to other places in cooker where we need to access an expanded data store.
We certainly don't want multiple expanded data stores.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 21 Jan 2015 17:43:51 +0000 (17:43 +0000)]
server/process: Fix select call
There was a report that bitbake -e | less would use 100% cpu when it shouldn't
really. The issue appears to be a bogus file descriptor in the select call. We
shouldn't be blocking if there is event data pending to a *reader* from server
context.
[YOCTO #7138]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Wed, 21 Jan 2015 12:59:51 +0000 (12:59 +0000)]
toastergui: enable re-ordering when columns are hidden
This patch enables table re-ordering when a sorted-by column is hidden.
This is default functionality in toaster table pages, but was not
enabled for all columns in this situation.
[YOCTO #7125]
[YOCTO #7022]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toastergui: fix the second 'view all targets' link
The project page has 2 links to the all targets page.
The one at the top of the page is working, but not
the one further down. This patch sorts that out.
Richard Purdie [Wed, 21 Jan 2015 13:53:59 +0000 (13:53 +0000)]
prserv/serv: Improve exit handling
Currently, I'm not sure how the prserver managed to shut down cleanly. These
issues may explain some of the hangs people have reported.
This change:
* Ensures the connection acceptance thread monitors self.quit
* We wait for the thread to exit before exitting
* We sync the database when the thread exits
* We do what the comment mentions, timeout after 30s and sync the database
if needed. Previously, there was no timeout (the 0.5 applies to sockets,
not the Queue object)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
David Reyna [Fri, 16 Jan 2015 16:42:41 +0000 (16:42 +0000)]
toaster: project configuration variables page
Implement the project configuration variables page.
AlexD made whitespace changes and a minor fix.
[YOCTO #6588]
Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 21 Jan 2015 11:16:20 +0000 (11:16 +0000)]
cooker: Further optimise pyinotify
We currently add crazy numbers of watches on files. The per user limit is 8192
by default and on a system handling multiple builds, this can be an issue.
We don't need to watch all files individually, we can watch the directory containing
the file instead. This gives better resource utilisation and better performance
further reverting some of the performance regression seen with the introduction
of pyinotify.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 21 Jan 2015 10:49:01 +0000 (10:49 +0000)]
command/cooker/knotty: Fix memres handling of command environment changes
If the environment changes, we need memory resident bitbake to adapt to those
changes. This adds in functionality to handle this alongside the configuration
option handling code. This means that the common usage:
MACHINE=X bitbake Y
now works with the memory resident server.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Wed, 14 Jan 2015 12:46:55 +0000 (12:46 +0000)]
toasterui: fix variable data error
A bug slipped in the toaster ui that prevented saving of
build configuration despite the data being retrieved
from the server. This patch fixes the shaming mistake.
[YOCTO #7117]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Wed, 14 Jan 2015 12:46:54 +0000 (12:46 +0000)]
toastergui: all builds page lists failed build requests
This patch modifies the all builds page by splitting the page
into two variants - the "interactive" (default) and "managed" mode
versions.
In the "managed" mode version, we display build requests instead of
builds, including the failed build requests that have no build
associated with them.
[YOCTO #6671]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Wed, 14 Jan 2015 12:46:52 +0000 (12:46 +0000)]
toaster: Add layer details page feature
This commit adds the layer details page which shows the metadata for the
layer such as layer description, machines associated with the layer as well
as the targets provided.
If the layer is an imported layer this page also allows you to update
the layer's configuration.
>From this page you can add/remove the layer from the current project
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 15 Jan 2015 09:42:23 +0000 (09:42 +0000)]
ConfHandler: Clean up bogus imports
The import statements here are plain bizarre. Remove them, tweaking
some of the function calls to match current practices. I can't find any
reason these old imports are as they are.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 15 Jan 2015 09:41:23 +0000 (09:41 +0000)]
ConfHandler: Rename oldfn to parentfn to be clearer
Looking at this function I had no idea what oldfn was, I doubt anyone
else would either without looking up what the caller does. "parentfn"
would seem a more appropriate name so rename it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 15 Jan 2015 09:39:34 +0000 (09:39 +0000)]
cooker: Improve pyinotify performance
Benchmarks show that the introduction of pyinotify regressed
performance. This patch ensures we only call the add_watch() function
for new entries, not ones we've already processed which does improve
performance as measured by "time bitbake -p".
This doesn't completely remove the overhead but it does substantially
reduce it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 14 Jan 2015 12:18:27 +0000 (12:18 +0000)]
cooker: Fix pyinotify handling of ENOENT issues
We try and add watches for files that don't exist but if they did, would influence
the parser. The parent directory of these files may not exist, in which case we need
to watch any parent that does exist for changes. This change implements that fallback
handling.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 13 Jan 2015 14:13:53 +0000 (14:13 +0000)]
cooker/cache/parse: Implement pyinofity based reconfigure
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).
This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.
Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.
Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.
We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.
For education and QA purposes I can document a workflow that illustrates this:
$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]
As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.
The new pyinotify dependency is covered in the previous commit.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 13 Jan 2015 14:11:36 +0000 (14:11 +0000)]
bitbake: Add pyinotify to lib/
We need inotify support within bitbake and pyinotify provides the best
mechanism to add this. We have a few options:
a) Depend on pyinotify from the system
b) Add in our own copy
c) Only use pyinotify in cases like the memory resident server
For a), it would mean adding in dependencies, updating documentation and
generally creating churn for users as well as having implications for things
like the build-appliance recipe.
It turns out that glibc has the C functionality we need from version 2.4
onwards (2006) and that we just need a single python file for b), there
is no binary module needed. We therefore add in a copy of pyinotify 0.9.5
into the tree meaning we can depend on it simply and unconditionally.
c) is unattractive as we need fewer possible code paths, not more.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Aníbal Limón [Tue, 13 Jan 2015 00:36:13 +0000 (18:36 -0600)]
bb/fetch2/ssh.py: Fix urldata.localpath use os.path.normpath
When urls ends with trailing slash os.path.basename return "" [1]
and built urldata.localpath only with DL_DIR, it causes that
donestamp is built as DL_DIR + '.done' and seems that ssh resource
was already download.
Richard Purdie [Thu, 8 Jan 2015 20:57:45 +0000 (20:57 +0000)]
fetch/git: Improve ls-remote handling for latest_revision
Currently the code ignores lightweight tags which has caused some user
complaints. We can't put the right search list in place easily since
the results don't come back in a good order, head happens to sort
before tags.
In the end I refactored the function so we get the complete list of
remotes and then we can filter it ourselves in the order we chose,
including checking for light weight tags, preferring the proper ones.
Hopefully this resolves the issues people have been seeing.
[YOCTO #6881]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toaster: do not launch web browser on noweb option
This patch makes Toaster managed mode obey the "noweb" parameter
by not starting the web server and launching the web browser
command if the "noweb" parameter is specified.
The web browser will be pointed at 127.0.0.1 instead of 0.0.0.0
[YOCTO #7039]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toaster: write pre-read conf file for build variables
We change the setting of variables from directly injection
into the set-up cooker to writing a conf file that is pre-read
on bitbake server startup. This is needed because the injection
can only happen after the variable set is parsed, and the variables
already inferred, so setting up variables happens too late.
[YOCTO #7045]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Thu, 8 Jan 2015 13:15:10 +0000 (13:15 +0000)]
toaster: Improve client side error handling
Make use of the toastermain.settings.DEBUG flag to toggle the client
side error logging. Make the error logging consistent by using
console.warn/error across the project, this adds traceability to the
warnings. Also handles the case where console is not available by
stubbing it in libtoaster.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 8 Jan 2015 10:35:16 +0000 (10:35 +0000)]
ast: Add error when trying to use dash in sh function names
A dash character is illegal in function names in sh (but not bash). Since
our shell tasks run under sh and the shell parser is sh based, EXPORT_FUNCTIONS
won't work with class names containing a dash.
We can't change sh, we can ensure the user is warned about the problem
straight away though.
[YOCTO #7006]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 8 Dec 2014 21:25:23 +0000 (21:25 +0000)]
cache/fetch2/siggen: Ensure we track include history for file checksums
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 8 Dec 2014 21:22:26 +0000 (21:22 +0000)]
wget: Add localpaths method which gives localpath with history
In some cases for cache purpoes we not only need to know which file
is going to be used but also which paths were considered. Add a
localpaths method which includes the history.
The core which() funciton already supports this, this just extends
the function to preserve the extra data we need. localpath becomes
just a special case of the case with history.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 8 Dec 2014 21:31:56 +0000 (21:31 +0000)]
bitbake-worker: Use setsid() rather than setpgid()
The bug has a long discussion of this. Basically, in some environments,
the exact details of which aren't understood, a Ctrl+C signal to the
UI is being transmitted to all the process children. Looking at the output
of "ps ax -O tpgid", its clear the main process is still the terminal
owner of these processes.
stty -a on a problematic system shows: "-ignbrk brkint"
and on a working system shows: "-ignbrk -brkint"
The description of brkint would suggest this is the problem, setting up
that terminal environment wasn't able to reproduce the problem though.
It was confirmed that using setsid() caused the problem to be resolved
and is probably the right thing to be doing anyway, so lets do it.
[YOCTO #6949]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 8 Jan 2015 09:41:03 +0000 (09:41 +0000)]
cooker: Shut down the parser in error state
If the cooker is in an error state, we shouldn't continue to try parsing.
This fixes an issue where an invalid PR server is detected when bitbake
is started and ensures bitbake exits cleanly rather than hanging.
[YOCTO #6934]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 23 Dec 2014 12:32:36 +0000 (12:32 +0000)]
fetch2/wget: Fix horrible temp file handling
Hardcoding a temp directory is bad practice and leads to races between
the tests. There is no longer any good reason for doing this, drop it
and ensure the files get cleaned up correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Aníbal Limón [Thu, 18 Dec 2014 23:58:57 +0000 (17:58 -0600)]
bb/fetch2/wget.py: latest_versionstring fix multithread and remove deprecated code
In order to fix multithread usage of latest_versionstring moves package_custom_regex_comp
from class to method level because need to be defined by package.
Remove code for build url's with /download suffix because it's deprecated since you can
specify the download directory using package_regex.inc file.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Fri, 19 Dec 2014 10:20:32 +0000 (10:20 +0000)]
cooker: add ability to ignore unmatched regexes in BBFILE_PATTERN
Add a BBFILE_PATTERN_IGNORE_EMPTY variable to allow ignoring the fact
that a regex specified in BBFILE_PATTERN for a particular collection
doesn't match any recipes. This will be used in OpenEmbedded in the
workspace layers created by "devtool" which may not always contain any
recipes (which is not cause for warning the user).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Fri, 19 Dec 2014 10:20:30 +0000 (10:20 +0000)]
utils: add basic metadata manipulation functions
* Add a generic edit_metadata_file() function to modify variable
assignments in any metadata file (conf, bb, bbappend) using a callback
for flexibility
* Add a specific edit_bblayers_conf() function to modify
conf/bblayers.conf and add and/or remove layers from the BBLAYERS
value within it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Fri, 19 Dec 2014 10:20:29 +0000 (10:20 +0000)]
siggen: ensure nostamp tasks force dependent tasks to re-execute
If a nostamp task is depended on by a non-nostamp task, then we want the
signature of that task to change such that it re-executes afterwards.
This is an unusual situation, but we want this to work in OE in
externalsrc.bbclass so that compilation happens every time it is
requested.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running bitbake inside make results in the exported environment variable
MAKEOVERRIDES="${-*-command-variables-*-}", which the shell chokes on
when trying to expand it. But of course, it probably shouldn't have been
trying to expand it in the first place -- so just escape the dollar
sign.
Signed-off-by: Richard Tollerton <rich.tollerton@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Michael Wood [Fri, 5 Dec 2014 17:25:51 +0000 (17:25 +0000)]
toaster: Importlayer add notify exactly which layers changed
This changes when the dependencies are added to the project so that we
can know which ones were successfully added by waiting for the server to
respond with a list. This is more reliable because we may have specified
dependencies which are already in the project.
To pass this information to the project page a temporary cookie is used
with the values for the notification.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Michael Wood [Fri, 28 Nov 2014 20:12:18 +0000 (20:12 +0000)]
toaster: Add import layer feature.
This feature allows users to import layers from git into their current
project and associate it with the release of the current project and the
dependencies for the newly imported layer with existing layers.
It will also resolve the child dependencies of the dependencies added.
[YOCTO #6595]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
This is a basic refactoring of the code computing
the layer equivalence classes for a project, in order to
bring common bits of logic in a single place.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
All 'change' icons should have a tooltip that appears
when you hover over them. The tootlip says (fittingly):
"Change". Initialise those icons in the libtoaster.js
file so that they work outside the project page.
toaster: base Only show change project icon when > one project
In the new build button, it only makes sense to change
the selected project when there is more than one project
in the Toaster instance. If the number of projects is 1,
we hide the change project icon.
This patch adds a simple UI-less POST endpoint, where
bitbake_eventlog.json files generated by a bitbake run can be
uploaded to the running toaster instance for insertion into
the database.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
