Robert Yang [Mon, 9 Jul 2018 03:28:38 +0000 (11:28 +0800)]
update.py: check whether branch existed when nocheckout
Fixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master
INFO: Skipping update of layer hello - branch master doesn't exist
This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout
[snip]
INFO: Sorting layers for branch mater:
WARNING: Cannot find required collections on branch master:
WARNING: hello: LAYERDEPENDS: <snip>
This is incorrect, this patch fixed the problem, now it skips it since the
branch doesn't exists when --nocheckout.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Robert Yang [Fri, 6 Jul 2018 07:20:47 +0000 (15:20 +0800)]
update.py: add layers when RECOMMENDS isn't satisfied
When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if
layer_b is not found, still add layer_a.
And print summary error mesage:
$ update.py -b master
ERROR: Issues found on branch master:
openembedded-core: Added without LAYERRECOMMENDS
meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Robert Yang [Fri, 6 Jul 2018 04:31:06 +0000 (12:31 +0800)]
utils.py: fix checkout_repo when no HEAD
Fixed:
$ git clone <url>
warning: remote HEAD refers to nonexistent ref, unable to checkout.
$ git rev-parse HEAD
HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Catch the error and avoid that.
And use "git reset --hard" to replace of "git reset --hard HEAD", HEAD is
default for git reset, so they are the same, but the later one reports error
when remote HEAD doesn't exist:
$ git reset --hard HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
[snip]
$ git reset --hard
No errors.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Robert Yang [Fri, 6 Jul 2018 03:09:37 +0000 (11:09 +0800)]
update_layer.py: avoid calling setup_core_layer_sys_path() when --initial
Fixed:
$ update.py -b <new_branch>
[snip]
NOTE: Starting bitbake server...
Traceback (most recent call last):
File "update_layer.py", line 471, in main
utils.setup_core_layer_sys_path(settings, branch.name)
File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path
core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
AttributeError: 'NoneType' object has no attribute 'vcs_subdir'
[snip]
This is because core_layerbranch is not in database yet for completely new
branch, so it is None and we will get the error. Avoid calling
setup_core_layer_sys_path() when "update_layer.py --initial" will fix the
problem.
And also only add core layer's sys path when it is present, since core layer
may not be added yet for completely new branch.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Paul Eggleton [Mon, 7 May 2018 00:04:38 +0000 (12:04 +1200)]
Add CSV export for layer recipes
Add the ability to export the recipe listing for a layer to a CSV file
for importing into external tools. At the moment we include name,
version and license, but there is a parameter that lets you specify the
fields to include in the URL if desired.
Implements [YOCTO #12722].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Wed, 18 Apr 2018 04:43:05 +0000 (16:43 +1200)]
rrs: fix unique constraint on RecipeMaintainerHistory sha1 field
Although it's unlikely to be an issue, technically we shouldn't be
insisting the sha1 field be unique globally, just within each
layerbranch, so adjust the constraints.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Wed, 18 Apr 2018 04:11:51 +0000 (16:11 +1200)]
rrs: add flag to MaintenancePlan to specify layer-wide maintainers
Most layers do not track maintenance on a per-recipe basis, and for
those layers we will hide some of the per-recipe maintainer features
and on the recipe detail show the layer maintainer(s) as the
maintainer(s) of the recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 17 Apr 2018 23:33:10 +0000 (11:33 +1200)]
rrs_upgrade_history: skip commits that don't touch the layer
If we're in a repository containing multiple layers, we don't care about
commits that don't affect the layer we are processing, so skip those
commits rather than passing them to upgrade_history_internal.py which
will ignore them (which is significantly slower).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Fri, 13 Apr 2018 04:03:42 +0000 (16:03 +1200)]
templates/rrs: replace use of = with ==
I can't quite tell which Django version broke this, but in any case
based on what's in pagination.html it seems we ought to have been
using == already.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 10 Apr 2018 23:01:33 +0000 (11:01 +1200)]
rrs: link maintenance/upstream history to layerbranch
RecipeUpstreamHistory was not linked to the layer it was produced from,
which meant that it wasn't easy to query for a different maintenance
plan (i.e. a different layer) and thus the maintenance plan selection
on the recipe list didn't really work. Add a link field, populate it in
a migration and then make it required.
We had added a link earlier from RecipeMaintainerHistory to LayerBranch
but it was optional; for the same reasons we now populate it and make it
required.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 10 Apr 2018 22:35:56 +0000 (10:35 +1200)]
rrs: drop a couple of unused functions from Raw class
These two functions aren't being used anywhere. In the interest of
having as little code directly reading the database using SQL as
possible, drop them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Mon, 9 Apr 2018 05:37:34 +0000 (17:37 +1200)]
rrs: improve admin for Release/Milestone objects
* Ensure the Release and Milestone names are separated by a space when
listing Milesones
* Include the maintenance plan name in the name shown for each
Release/Milestone
* Allow filtering Releases/Milestones by maintenance plan
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Mon, 9 Apr 2018 05:37:56 +0000 (17:37 +1200)]
rrs: duplicate releases from first plan when adding a new plan
It's a pain to have to add all the releases when adding a new
maintenance plan. Since these are likely to be the same (or similar) for
every plan, then duplicate them across from the first plan when you save
a new one.
Also add "default" milestones on the assumption that other layers
probably won't want to use the 4-milestone split per release, but there
do have to be some milestone records, so just create one milestone for
each release.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Mon, 9 Apr 2018 05:33:49 +0000 (17:33 +1200)]
rrs: default python2/3 environments for new maintenance plan layer branches
It's a bit of a pain to have to set the two python environment fields on
every record in order to have things set correctly, and it can easily
get forgotten, so try to set them automatically by default (assuming
reasonable naming).
Note that this does introduce an annoying behaviour whereby if you click
"Add another Maintenance plan layer branch" and then decide you don't
want it, the admin form will insist you fill in the fields unless you
clear out the python2/3 environment fields. I'm not sure how to fix
that, so I'm leaving it as-is for now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Sun, 8 Apr 2018 22:53:45 +0000 (10:53 +1200)]
rrs: restore full-width pages
Upon consideration, for the width of the information we want to present
we do actually want full-width pages for the RRS. When this was changed
earlier in the rrs branch it was changed in the base template, but we
want to keep the same style elsewhere, so put a block in that will let
use the "container-fluid" style (full width) in the RRS pages and
"container" by default everywhere else.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Fri, 6 Apr 2018 05:19:22 +0000 (17:19 +1200)]
rrs_upgrade_history: improve checkout logic
* Consolidate the code for checking out a repository, using the newly
added utils.checkout_repo() function
* Check out a layer's dependencies, not just the layer itself
* Only check out if the desired revision isn't already checked out
(mostly useful for bitbake which we would otherwise be checking out
much more frequently than necessary since it may not have changed
even if we've moved to a new commit in the layer).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Fri, 6 Apr 2018 11:27:00 +0000 (23:27 +1200)]
utils: add common function to check out a specific git revision
Checking out a revision in the bitbake/layer repositories is something
we are doing in a few places, so add a checkout_repo() function that
does this, ensuring that we don't get tripped up by any junk files,
and avoids checking out if the repository is already at the desired
revision (thus avoiding the clean operation and e.g. preserving any
.pyc files that aren't stale and would speed things up a little). Note
that we do the clean before checking out in case there are untracked
files that are tracked in the commit we are checking out.
In addition to adding this function, change the existing code that we
use in the update script to check out a layer use the new function.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 3 Apr 2018 11:19:46 +0000 (23:19 +1200)]
Implement layer web repo commit URL
The Recipe Reporting System needs to be able to provide links to commits
in the web interface for the repository, but we can only do this if we
have a custom template URL just like we do for file/tree links, since
it's different for different git web interfaces. Add support in all the
various places for such a URL and make use of it in the RRS.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 3 Apr 2018 05:22:31 +0000 (17:22 +1200)]
rrs: support new source structure in recipe detail
In the rrs branch we used to just store SRC_URI in a field, however we
now have a proper model to store sources, so use that in the RRS recipe
detail page.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Mon, 8 Jan 2018 20:46:40 +0000 (09:46 +1300)]
Save recipe source URLs
Save each remote SRC_URI so we can use these for the recipe reporting
system. This replaces an earlier implementation in the rrs branch where
we simply stored SRC_URI verbatim.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 17 Apr 2018 20:57:12 +0000 (08:57 +1200)]
admin: use more dynamic method of setting recipe read-only fields
Every time we add something that links to Recipe we had to add it to the
exclusions list in the readonly_fields line for RecipeAdmin (and
ClassicRecipeAdmin), which is tedious and easily forgotten. We can avoid
this by looking at each field and excluding it by its attributes rather
than having a hardcoded list of names.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 3 Apr 2018 04:36:17 +0000 (16:36 +1200)]
Add a link from the Tools drop-down to the RRS if enabled
If the RRS is enabled, then add a link to it in the tools menu. I don't
expect this to be used a lot, but otherwise the only way you'd get there
would be either externally or via the link from the layer detail.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 3 Apr 2018 04:26:45 +0000 (16:26 +1200)]
rrs: Add link to layer detail to breadcrumb on recipe detail
Add a convenience link to the layer detail to the breadcrumb (also as an
indicator, since it's possible to have more than one layer in the
maintenance plan).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 3 Apr 2018 03:30:00 +0000 (15:30 +1200)]
rrs: handle linking maintainership
Provide a mechanism set the maintainer for things like gcc-cross-<arch>
to the same as gcc. (We do have entries in the .inc file for these,
however they aren't useful as they don't match the recipe name when we
parse it, and due to the fact that RecipeMaintainer objects link
directly to Recipe objects, we can't handle entries that don't map to a
real recipe).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't actually need to check out the repository until we actually
analyse a commit, so avoid doing so. Additionally, there's not much
point in checking out master at the end, let the next script invocation
do that if needed (if it needs to, it should since otherwise there's no
guarantee what state the repository is in).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Wed, 28 Mar 2018 11:45:23 +0000 (00:45 +1300)]
rrs_upgrade_history: handle only .inc file changing
If a recipe upgrade only consists of a .inc file changing, we were not
picking it up, since we were only looking specifically for recipes
(.bb). If a .inc file changes, assume all .bb files in the same
directory (if any) should be parsed to look for an upgrade.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 27 Mar 2018 03:16:29 +0000 (16:16 +1300)]
rrs_upgrade_history: ignore files outside of the layer
We were parsing recipes that were in the repository but not inside the
actual layer we're dealing with (e.g. we have meta-selftest within the
OE-Core repository, containing a number of recipes that are only
intended for testing purposes and should not be looked at by this
script).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 27 Mar 2018 02:59:22 +0000 (15:59 +1300)]
rrs_upgrade_history: handle broken version numbers
In OE-Core revision e0531174119bff21e9014b95ed1bbd0e1c01af26 we
accidentally committed a new e2fsprogs recipe with ..bb at the end of
its name instead of .bb. This was fixed immediately afterwards, but when
the RRS hits this commit, it doesn't fail immediately, but the bogus
version "1.43." gets into the database and all subsequent commits
touching the e2fsprogs recipe cause bb.utils.vercmp_part() to blow up
because one of the version parts in the "previous" version in the database
is apparently empty. To work around this and any similar issues, just
reject any change that results in such a broken version string (on the
assumption that it'll be corrected in a subsequent commit and thus we
will get to re-parse the recipe then and therefore not miss the
upgrade.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Mon, 26 Mar 2018 04:40:33 +0000 (17:40 +1300)]
rrs/tools: use layer index lock
We check out different revisions while we do this processing, and so
does the layer index update script, so we shouldn't be allowing both to
run at once or nasty stuff will happen.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Mon, 26 Mar 2018 01:53:29 +0000 (14:53 +1300)]
rrs_upgrade_history: only look at commits that actually change recipes
Since we're now executing a separate script per commit, we should try
not to do that unless the commit actually touches recipe files in order
to avoid wasting time.
(Whilst it's possible that a change to a bbclass might alter what's in
the recipe, we can ignore that since we are only concerned with actual
upgrades which would always require some sort of change to the recipe or
an include file, so we can safely skip commits that don't do that.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Sun, 25 Mar 2018 21:40:01 +0000 (10:40 +1300)]
rrs_upgrade_history: use date/commit of last recipe upgrade import
Instead of arbitrarily importing the last 8 days of upgrades, record the
date and commit when we do an import, and then use that information the
next time the script is run.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Fri, 23 Mar 2018 12:06:36 +0000 (01:06 +1300)]
rrs/tools/common.py: ensure we pass a string to loadDataFull()
We're calling translate() on the string deep in the bowels of the
parsing code and that doesn't work well if the string is unicode, so
convert it to a plain string first. That won't work well if the filename
is unicode but the chances of that with a recipe is pretty small I would
think.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Mon, 26 Mar 2018 03:59:32 +0000 (16:59 +1300)]
rrs_upgrade_history.py: ensure we shut down tinfoil safely
Use try...finally to ensure we shut down tinfoil, but since we are
potentially dealing with older bitbake releases when importing older
upgrades, only call shutdown() if it's actually there (and although it's
unlikely, guard against the broken shutdown() in fido as we do in the
main layer index update script).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Thu, 22 Mar 2018 12:30:56 +0000 (01:30 +1300)]
rrs_upgrade_history.py: enable to work across python2/3 versions
If you want to go back and get history for the earlier releases (krogoth
and previous) then we need to be able to support both python 2 and 3,
which practically means we need the same split for this script as we
have for the main layer index update script.
The catch here is that since we are going back and following the history
of changes forward, we basically need to use the same version of bitbake
that was current at that time. This works except for around the
transition between python 2 to 3 where the metadata lagged behind a bit,
so we need to take that into account. In order to keep things generic we
have a date field on the maintenance plan layer branch that specifies
the date in the metadata where we should switch over to python 3, and
then link to PythonEnvironment records that should be used for python 2
and 3 respectively.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Wed, 21 Mar 2018 03:52:49 +0000 (16:52 +1300)]
rrs: handle dependency field differences
The old RRS branch had its own addition of dependency support, but in the
mean time we added that to the layer index in the master branch using a
different structure. Adapt the RRS recipe detail page to that structure.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 20 Mar 2018 03:35:43 +0000 (16:35 +1300)]
rrs/views: fix case in table names
Table names should be lowercase. I'm unsure if the collation settings
have an effect on this, but with the mariadb database I set up here I
needed to make this change or else I got errors about missing tables.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 17 Apr 2018 22:16:33 +0000 (10:16 +1200)]
rrs_upstream_email: Adapt to template rendering API change
With Django 1.10+, if you use get_template() to retrieve a template,
then you can't pass a context when calling .render() on it, you need to
pass a dict instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Thu, 1 Mar 2018 20:09:35 +0000 (09:09 +1300)]
rrs_upstream_email: rework
* Use maintenance plans to get layerbranches
* Use from/to/subject and admin contact from maintenance plan
* Use an actual template to render the email (and drop tabulate
dependency)
* Improve grammar in the email text
* Use a single line to represent the most recent commit
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Thu, 1 Mar 2018 19:31:01 +0000 (08:31 +1300)]
rrs/models: add fields for more flexible email handling
* Add a flag to say whether emails should be sent
* Add fields for from/to/subject (to replace what's currently in
settings)
* Add user link for administrator to replace the hardcoded values in the
email template
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Thu, 1 Mar 2018 04:30:42 +0000 (17:30 +1300)]
rrs/tools: drop update_repo()
This function is no longer needed - we now rely upon the layer index's
code to do this (update through the update script, and checkout through
setup_layer()).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the Recipe object doesn't exist here then an exception will be raised
rather than None being returned, and this will also trigger if multiple
recipes match. This may have never triggered in the past because this
would have been run right after updating all the recipes in the layer and
clearing out duplicates (which we were doing earlier), and thus what is
in the database would match the recipe files in the repository, assuming
no errors occurred during parsing). We can't remove duplicates though so
we need to switch over to using filter() and taking the first recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Thu, 1 Mar 2018 03:33:58 +0000 (16:33 +1300)]
rrs_upstream_history.py: drop threading
This is never going to work, the only way we can practically do parallel
execution is to run these things in a task which would basically amount
to distrodata's do_checkpkg; we may move to that in future but for now
just drop the threading code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Wed, 28 Feb 2018 20:10:20 +0000 (09:10 +1300)]
rrs_maintainer_history.py: support maintenance plans and remove poky hardcoding
Instead of hardcoded references to the poky repository, look for any
maintainers.inc file in layers associated with the layerbranches for all
enabled maintenance plans. At present few layers have this file, but at
least it will now work generically in any layer index instance.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Wed, 28 Feb 2018 19:26:57 +0000 (08:26 +1300)]
utils: decode command output in runcmd() as UTF-8
Sometimes we get back UTF-8 characters (particularly when picking up
names from git commits), and the ascii codec will error out if that
happens, so switch over to utf-8.
We can as a result remove the decode() calls from the bulk change view.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 27 Feb 2018 04:17:16 +0000 (17:17 +1300)]
rrs_upgrade_history.py: add maintenance plan handling
Instead of processing all layerbranches, only process those associated
with an enabled maintenance plan. This is one step towards being able to
use the RRS together with a more general layer index database.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 27 Feb 2018 04:16:12 +0000 (17:16 +1300)]
rrs/tools: avoid unnecessary checkouts
We don't need to create branches here, and we don't need to actually
check anything out unless we're going to parse, so we can save a bit of
time by not doing so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 27 Feb 2018 21:03:27 +0000 (10:03 +1300)]
rrs: add Release link to MaintenancePlan
If any Releases exist then we create a default MaintenancePlan and then
link all Releases to it - this needs to be done in multiple upgrade
steps since the field needs to be non-null.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 27 Feb 2018 04:15:32 +0000 (17:15 +1300)]
rrs: add maintenance plans
The MaintenancePlan will provide some context for the RRS records so we
can effectively enable RRS functionality only on certain layers where we
want it. You can also consider the maintenance of multiple layers
together under a single plan if desired.
Here we add just the MaintenancePlan and the corresponding migration;
a non-null link from the Release requires a separate migration and thus
that will be done in a separate commit.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Tue, 27 Feb 2018 20:49:46 +0000 (09:49 +1300)]
rrs: add initial migration
This is of course a new Django migration rather than the previous South
one. This will have to be applied using --fake-initial on a database
that already has the RRS tables.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Paul Eggleton [Mon, 26 Feb 2018 20:40:19 +0000 (09:40 +1300)]
rrs/tools/rrs_unique_recipes: drop
We can't just delete arbitrary recipes that do exist in the layer if
we're in a general layer index database. We'll need to handle this in a
different way, or just live with the fact that there will be duplicate
entries.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
