Alexandru DAMIAN [Fri, 21 Nov 2014 13:58:51 +0000 (13:58 +0000)]
toaster: use http proxies to fetch data
Under some network configurations http proxies are used
for Internet access. This patch makes Toaster obey
the http_proxy environment variable when fetching
information from layer indexes.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
When you search for a layer and your search returns no results,
we should show you an alert and a link to show back all layers,
as we do in all other tables.
Michael Wood [Fri, 14 Nov 2014 15:44:55 +0000 (15:44 +0000)]
toaster: project page remove additional tooltip for Change pencil
We have a global pencil icon tooltip so avoid two tooltips appearing by
removing them from the project page. This also means it's one less place to
update it.
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Michael Wood [Fri, 14 Nov 2014 18:12:20 +0000 (18:12 +0000)]
toaster: libtoaster: Add getProjectInfo utility function
Add a utility function to return a specified project's info/config
This re-uses the existing server code path for the project edit
except that it allows any project id to be used as a parameter
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Michael Wood [Tue, 11 Nov 2014 16:23:58 +0000 (16:23 +0000)]
toaster: Create libtoaster.js with some utility functions
Replace main.js with libtoaster. This can be a place for common
functionality for toaster. Calling it lib... makes it more obvious as
well as helps with code readability due to the namespacing.
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Fri, 14 Nov 2014 17:07:06 +0000 (17:07 +0000)]
toastergui: layer name correlation
This patch modifies how layers are identified and matched.
Layers were primarely organized by the source of layer information,
and Releases were separated by both layer git branches and originating
source of layer information. This setup prevented mixing layers from
different sources for a certain release, which didn't match the way
people use Yocto Project / bitbake.
This patch brings name-based indentification, where layers with the
same name are assumed to be equivalent, in the sense of being able
to substitute one another. To facilitate this identification to
humans, layers are differentiated by GIT URI instead of layer sources,
which was a rather arbitrary abstraction.
Additional changes include modification to models in order accomodate
for the new data structure, and to config file loading to match
the new toasterconf.json layout.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Michael Wood [Mon, 27 Oct 2014 19:03:06 +0000 (19:03 +0000)]
toaster: skip virtualenv when searching for django apps
If we are using a virtualenv in the current search path we end up trying
to add modules to the django apps list which do not have the correct
load path or are internal to other modules. Fix this by skipping the
virtualenv path.
[YOCTO #6896]
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Robert Yang [Mon, 17 Nov 2014 10:27:36 +0000 (02:27 -0800)]
bitbake-worker: exit normally when SIGHUP
Fixed:
1) Run "bitbake recipe" in the terminal
2) Close the terminal while building
3) $ ps aux | grep bitbake-worker
There will be many processes, and they will keep the resources (e.g.,
memory), and won't exit unless kill or kill -9.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otavio Salvador [Wed, 12 Nov 2014 10:33:23 +0000 (08:33 -0200)]
bb.codeparser.py: Remove reference for oe.utils.contains
The oe.utils.contains function has been removed from OE-Core metadata
as the references for it has been replaced to use the
bb.utils.contains.
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Brendan Le Foll [Wed, 12 Nov 2014 11:42:39 +0000 (11:42 +0000)]
cooker.py: fix loginfo op being set to an invalid value
loginfo "op" was being set to invalid values in saveConfigurationVar it was
working because set is a python global but append is not. This replaces both
the "op" with strings and removes the crash when calling appendConfigurationVar
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 11 Nov 2014 17:01:09 +0000 (17:01 +0000)]
toastergui: fix XSS injection points in projects page
We close XSS injection points in Projects page.
* modify the json filter to properly escape HTML tags in strings
* enable $sanitize to automatically sanitize dangerous HTML in
user-supplied input
* clean dangerous characters in targets field, as that field contents
will be directly passed to a shell command
Based on the vulnerability discovered and the patch provided by Michael Wood.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toastergui: changes for the Project page, round 3 of reviews
This patch implements the round 3 of reviews for the Project
page, including fixing the time display, fixing the build list
display, with fade-in and fade-out animations, and various
small layout fixes.
[YOCTO #6587]
[YOCTO #6731]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toaster script: webport option and other improvements
We add the "webport=" command line option as to allow starting
the web server on a custom port.
The bitbake server port is now auto-allocated. This is needed
to be able to run multiple toaster environments on a single machine.
We tackle bug 6023 (toaster refusing to start when lock file is present)
by using more specific checks, and automatically recover from
bitbake server down / webserver up error mode.
Command line parameters are now read on both interactive and managed
modes.
The localhost and ssh controllers are updated to work with the modified
toaster launcher script.
[YOCTO #6023]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Martin Hundebøll [Thu, 30 Oct 2014 10:43:24 +0000 (11:43 +0100)]
progressbar: use '/usr/bin/env' in shebangs with python
To support yocto on systems with python3 as default version, scripts
should use /usr/bin/env python in the shebang, as this allows the use of
a fake env to mimic python2 as default version.
This patch simply replaces occurrences of #!/usr/bin/python with
#!/usr/bin/env python and was done with this oneliner:
Signed-off-by: Martin Hundebøll <martin@hundeboll.net> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Aníbal Limón [Wed, 5 Nov 2014 18:10:29 +0000 (12:10 -0600)]
fetch/wget: Add latest_versionstring method
Being able to query whether updated versions of a url are available
is useful, not least for the package reporting system. Since such code
is closely linked to the url type and the url itself, the fetcher
makes a locical place to contain this code.
For wget based urls this means taking upstream directory listings
and searching those for later versions, returning those that are
found.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
Aníbal Limón [Wed, 5 Nov 2014 18:10:28 +0000 (12:10 -0600)]
fetch/git: Add latest_versionstring method
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
Richard Purdie [Fri, 7 Nov 2014 17:47:37 +0000 (17:47 +0000)]
siggen: Fix shared work checksum mismatch/rebuild issues
Similar to the last shared work task signature bug, we've found another
one. Looking at the improved output of diffsigs in this case:
runtaskdeps changed from [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native',
'libgcc-initial_4.9.bb.do_patch:virtual:nativesdk'
] to [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gcc-crosssdk-initial_4.9.bb.do_patch',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native'
]
so we can get a different task hash since libgcc sorts before gnu-config
and gcc sorts after it. We could do with a way of fixing this, the best
I can come up with is to include a single parent directory. Since
recipes are never at the top of any metadata trees I've seen, this
should suffice for now.
I'm planning to burn the concept of shared work within bitbake
and do something at the metadata level in the 1.8 timeframe as its just
too fragile as things stand and hard to fix well.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 7 Nov 2014 17:45:07 +0000 (17:45 +0000)]
siggen: Ensure we output if the ordering of runtaskdeps changes
Order of runtaskdeps is important. If the hashes differ we should print output.
This is complicated by shared work where the filenames themselves can differ,
but the checksum should not.
This fixes a case where two different checksums could show no output with
bitbake-diffsigs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 31 Oct 2014 19:17:02 +0000 (19:17 +0000)]
toastergui: fix invalid build url usage
We need to pass the URL for the build action as javascript
parameter, since the project ID is different for each build,
instead of using the project id of the most-recently-used project.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 31 Oct 2014 19:17:01 +0000 (19:17 +0000)]
toasterui: improvements in data reading
We add improvements in data reading, following
issues discovered in testing.
- elapsed_time is now read from buildstats
- we add safeguards to not fail logging if the build was triggered
with a toaster_brbe configuration, but it's running in
1.6 mode
- added log markups for build finish to let other programs
known when the work is done.
[YOCTO #6833]
[YOCTO #6685]
[YOCTO #6887]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 5 Nov 2014 16:58:13 +0000 (16:58 +0000)]
prserv: Use WAL mode
Ideally, we want the PR service to have minimal influence from
queued disk IO. sqlite tends to be paranoid about data loss and
locks/fsync calls. There is a "WAL mode" which changes the journalling
mechanism and would appear much better suited to our use case.
This patch therefore switches the database to use WAL mode. With this
change, write overhead appears significantly reduced.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 4 Nov 2014 14:02:32 +0000 (14:02 +0000)]
prserv/serv: Ensure sync happens in the correct thread
The sync/commit calls are happening in the submission thread which can
race against the handler. The handler may start new transactions which
then causes the submission thread to error with "cannot start a
transaction within a transaction".
The fix is to move the calls to the correct thread.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 14 Oct 2014 12:57:33 +0000 (13:57 +0100)]
toastergui: provide download file capability
We add, for the localhost environments, the capability to
download build artifacts. This is a pontentially dangerous API,
because it gives unrestricted read access to the build environment
file system - do not expose the functionality directly to the
web layer, but use filtering/translation code, such as
exemplified in the build_artifact view.
The capability for remote build environments is dependent
on bug 6835, as to use the collect storage as intermediary
storage for serving files.
[YOCTO #6834]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Mon, 13 Oct 2014 16:10:39 +0000 (17:10 +0100)]
toaster: change startup parameter passing to avoid race
We avoid a race between the setting the TOASTER_BRBE variable
and reading the variable in toaster ui by supplying the variable
at server startup time through the toaster.conf post-read file.
Additional small changes are included, including marking the
build request with the environment id of where the build took place.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
The default django URL validator marks git URLs as not valid,
so we change the default validators to a URLValidator-derived
that adds git and ssh protocol schemas and inhibits the
frontend URL validator.
We update the branch names based on the white-listed
branch names in the Releases name.
Adds proper class string to ease usage in admin section.
Other small improvements.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Conflicts:
bitbake/lib/toaster/orm/admin.py
toaster: use modified validators for git url fields
The default django URL validator marks git URLs as not valid,
so we change the default validators to a URLValidator-derived
that adds git and ssh protocol schemas and inhibits the
frontend URL validator.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Tue, 30 Sep 2014 12:45:51 +0000 (13:45 +0100)]
toastergui: protect variable value reads
We make sure we don't throw an exception when reading
non-existent variables from the database, and we return
empty data that can be displayed without ill effects.
This fixes the Configuration page on empty builds.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Tue, 30 Sep 2014 12:31:57 +0000 (13:31 +0100)]
toastergui: refactor objects get in views.py
We refactor the objects.filter()[0] pattern to the proper
Django QuerySet usage of objects.get() to make sure we
get one and only one object for a query.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toaster: fix Project page in order to trigger builds
This patch rewrites the Project page and the additional
infrastructure in order to fix a bug that makes triggering
builds through UI impossible, and to introduce data feeds
for suggestions for the user.
Alexandru DAMIAN [Tue, 30 Sep 2014 12:57:14 +0000 (13:57 +0100)]
toastergui: update breadcrumb in analysis pages
If we are in managed mode, and we inspect a build that comes
from a project, we modify the breadcrumb as to display
the project from where the build originated.
[YOCTO #6776]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Alexandru DAMIAN [Mon, 29 Sep 2014 14:35:59 +0000 (15:35 +0100)]
toasterui: save build id in build request on first chance
We change the saving of the build id to the build request as
soon is the build is created, as to allow for a consistent
display of build data while build is in progress.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toastergui: Remove extra <div> from build dashboard
For builds that generate no rootfs files, an extra <div>
was being added between the build status and the build
summary section. This commit removes the extra <div> and the
white space it created.
toastergui: Remove sorting from "Size over total" column
The sorting in this column is tied to the sorting by
"Size". If we make both columns sortable, when you sort
by one of them it also sorts the other, and the table looks
as if you have sorted by 2 columns at the same time.
This was fixed in YOCTO #6061, but turned up again.
Change the background colour of table rows with the .error
class from white to transparent, to make sure the blue
highlight animation works when you click on the order
number in the task details page.
toastergui: Fix reverse dependencies tab for packages included
Make sure the reverse dependencies tab for included packages
shows a notification when the number of reverse dependencies
is 0, instead of an empty table.
The "Prebuilt task could be based on" table we provide in the task
details page for prebuilt tasks is missing the task descriptions.
This patch adds them.
[YOCTO #6328]
Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Belen Barros [Fri, 29 Aug 2014 12:46:41 +0000 (13:46 +0100)]
toastergui: Amend show rows options
In the templates basetable_bottom.html and basetable_top.html
remove the '2' and '5' options from the 'show rows' menu, and
add the '150' option as per design specs.
Alex edited conflicts.
Signed-off-by: Belen Barros <belen.barros.pena@intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Michael Wood [Tue, 28 Oct 2014 17:47:12 +0000 (17:47 +0000)]
buildinfohelper: Make sure we use the orm defined value for loglevel
We need to consistently use LogMessage.INFO/WARNING/ERROR to make sure toaster knows
how to categories these rather than passing in the "raw" loglevel value
which in best case comes from python logging but worst case any value.
[YOCTO 6885]
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ben Shelton [Mon, 27 Oct 2014 17:27:23 +0000 (12:27 -0500)]
prserv: don't wait until exit to sync
In the commit 'prserv: Ensure data is committed', the PR server moved to
only committing transactions to the database when the PR server is
stopped. This improves performance, but it means that if the machine
running the PR server loses power unexpectedly or if the PR server
process gets SIGKILL, the uncommitted package revision data is lost.
To fix this issue, sync the database periodically, once per 30 seconds
by default, if it has been marked as dirty. To be safe, continue to
sync the database at exit regardless of its status.
Signed-off-by: Ben Shelton <ben.shelton@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Thu, 25 Sep 2014 15:10:33 +0000 (16:10 +0100)]
toastergui fix size rendering in dirinfo page
We fix the rendering of the size field in dirinfo-related pages
by directly calling filtered_filesizeformat and not rendering
it through the template engine.
Additionally, we enable error dumping into logs if an
Exception happens.
[YOCTO #6669]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 24 Sep 2014 16:51:47 +0000 (16:51 +0000)]
hob: Fix sstate mirrors mangling
hob was adding the redundant characters "\1" in SSTATE_MIRRORS variable. If
needed it is expected the user will add this instead so remove the code
that was doing this.
[YOCTO #6600]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch: SRC_URI parameter "subdir" does not work for local files
Check if the 'subdir' parameter exists and assign it to 'destdir' so that
files are copied in ${WORKDIR}/destdir. This fixes urls that are of the form
file://a;subdir=b.
Signed-off-by: Roxana Ciobanu <roxana.ciobanu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Thu, 25 Sep 2014 14:28:52 +0000 (15:28 +0100)]
monitordisk: don't log when not monitoring a filesystem for inodes
Writing a log that the filesystem isn't being monitored for inode usage just
confuses users who are not aware about the nature of inodes in their filesystem,
so don't say anything, just silently disable the monitor. In general this only
happens on filesystems which don't have a limit on inodes.
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Chad Nelson [Fri, 12 Sep 2014 20:42:21 +0000 (15:42 -0500)]
fetch2/perforce: Fix localfile name if labels are used
I could apply the label "release_1.0" to a super project that contains
many sub projects. If my recipes have SRC_URI's that use that label but
grab different sub-folders, than there's a bug where the cached localfile
(tar.gz) will not be unique and reused at the wrong times.
Richard Purdie [Tue, 23 Sep 2014 11:58:22 +0000 (12:58 +0100)]
knotty: Ensure commandline parameters are updated in memres server
When using options like -k, -f, -v and so on with the memory resident
server, they'd currently only be set on the initial values passed to
the original command. This ensures they now match those specified
on the commandline for the options where this makes sense.
To make this work, a command to update the options on the server side
is required so this is added.
[YOCTO #5292]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sat, 13 Sep 2014 07:50:26 +0000 (08:50 +0100)]
data_smart: Fix remove operator and its interaction with data expansion
If you have:
FOO = "${bindir}/X Y"
FOO_remove = "${bindir}/X"
the expected result is "Y". Currently this doesn't work since the removed
expressions are not expanded first. This patch adjusts things so the
expressions are expanded before being processed for removal.
Also add a test to ensure this case continues to work.
[YOCTO #6624]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 5 Sep 2014 09:34:41 +0000 (10:34 +0100)]
siggen/runqueue/bitbake-worker: Improve siggen data transfer interface
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Wed, 3 Sep 2014 16:06:42 +0000 (17:06 +0100)]
tinfoil: add a means of enabling variable history tracking
Unfortunately it seems like the external use of the cooker
enableDataTracking() function broke at some point since the code that
reads it now runs within BBCooker's constructor. Since this now has to
be done early, add a parameter to Tinfoil's constructor to allow
enabling variable history tracking.
Fixes [YOCTO #6676].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Marius Avram [Fri, 29 Aug 2014 12:51:01 +0000 (15:51 +0300)]
toaster: use cookies for count and sorting in templates tables
Until now cookies were used to save which columns were shown and which
were hidden in toaster tables. The tables from the templates also have
functionalities like sorting the entries on a certain column and
limiting the number of entries displayed on a page. The later however
were not saved using cookies. This patch brings this new feature.
The cookies are not saved only in the front-end. They are saved both
in the frontend in case the user uses the inputs/buttons to change
a parameter and also in the backend in case the user specifies manually
using GET variables the value of the parameters.
When no GET parameters are given the views will redirect the url to one
containg the parameters saved as cookies. When no cookies exist, default
values will be used.
[YOCTO #6126]
Signed-off-by: Marius Avram <marius.avram@intel.com>
In order to avoid problems when using JSON serializer for
saving sessions, we move from storing the objects themselves
in the session to storing the object id and reloading the
object when retrieved.
This allows, for example, to use cookie-storage sessions if the
infrastructure owner so desires.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
We rename the projecttags in bldviewer.templatetags to
simple_projecttags in order to avoid conflict with the
similarly named tagset in toastergui.
The conflict leads to an intermittent bug where proper
tags are not read correctly since Django uses only the
module name as global tag library identificator.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Dan McGregor [Fri, 29 Aug 2014 19:46:18 +0000 (13:46 -0600)]
knotty: ignore interrupted system calls
With the improved exception handling added in an earlier commit bitbake
now stops when recieving a SIGWINCH. This happens frequently when
disconnecting and reconnecting tmux sessions and bitbake didn't survive.
Restore old behaviour of ignoring interrupted system calls but keep
proper exception handling for other errors.
Signed-off-by: Dan McGregor <dan.mcgregor@usask.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
