Joshua Lock [Fri, 27 May 2011 20:04:44 +0000 (13:04 -0700)]
uievent: fix queueing of events for xmlrpc before UI has loaded
The change to Queue up events before the UI is spawned broke the xmlrpc
server because the uievent implementation of BBUIEventQueue expects pickled
strings for its queue_event() method.
This is because the RPC exposed event.send() method must accept pickled
strings, but for xmlrpc event.send() is just mapped to queue_event(). Work
around this by adding a send_event method which unpickles strings and hands
them off to queue_event() which can then be used for the remapping.
Richard Purdie [Fri, 27 May 2011 15:14:48 +0000 (16:14 +0100)]
bitbake/codeparser: Improve cache handling
The current codeparser cache handling hurts performance badly even
over a couple of cores and certainly on many core systems, it can
spent huge amounts of time in the codeparser cache save functions.
This patch reworks the cache handling so that each parsing thread
saves out its own "differences" file compared to any existing core
cache and then the main bitbake thread picks these up and merges
things back together.
This was tested on systems with small and large numbers of cores
and was found to perform orders of magnitude better in all cases
despite the more complex code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 27 May 2011 14:03:51 +0000 (15:03 +0100)]
bitbake/runqueue.py: Ensure existing setscene stamp files are taken into account
JaMa reported issues where bitbake would rebuild things instead of using the
existing built tasks. This was tracked to a case where:
a) rm_work is uses
b) A depends on B
c) B has a version change (e.g. PR bump)
and A *and* B would then rebuild.
It turns out that rm_work was correctly turning stamp files into the correct
_setscene varients but bitbake was then ignoring them during setscene processing.
If the correct sstate checksumed files didn't exist, everything would seemingly
rebuild.
The fix is to check for existing *_setscene stamps and if present, honour them.
If "basichash" is enabled, the hash is included with the stamps so everything
should then function as intended.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 27 May 2011 14:11:49 +0000 (15:11 +0100)]
bitbake/cooker.py: Fix -b option regexp handling
bitbake -b core-image was showing no matches when it should list all targets
containing the fragment "core-image". This patch only calls os.path.abspath()
on things that look like paths and passed the path around more consistently to
various functions to get this use case working again.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Yu Ke [Thu, 26 May 2011 07:37:56 +0000 (15:37 +0800)]
fetch2/git: enhance the unpack by using "git checkout"
current git fetcher unpack method only checkout index and working tree,
but did not did not update the git branch in ref/heads, so user may not
get right info in ${S} by using git.
this patch enhance the unpack by using git checkout to fix this issue.
Fix bug [YOCTO #1089]
Signed-off-by: Yu Ke <ke.yu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 25 May 2011 22:45:31 +0000 (23:45 +0100)]
cooker.py: Don't show spurious warnings for collections of .bbappend files
Seeing warnings like:
WARNING: No bb files matched BBFILE_PATTERN_yocto '^/xxx/meta-yocto/'
are not encouraging to users and we shouldn't show these if we found
.bbappend files (but no .bb files). This change stops these warnings
from appearing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 25 May 2011 22:19:41 +0000 (23:19 +0100)]
build/siggen: Ensure a task signature file is generated for each task that is executed
Tracing down signature changes and dependencies is hard unless the complete task
chain can be evaultated. Since we have the data available at task exeuction
time, writing it to disk makes sense.
This change is suboptimal internal knowledge stamp_internal() has leaks into
make_stamp() and the concepts used by this code need revisiting but the change
brings enough usability benefit to warrant its inclusion despite that.
Addresses issue [YOCTO #1074]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Yu Ke [Sun, 15 May 2011 14:33:30 +0000 (22:33 +0800)]
git fetcher: add support for rebaseable git repo
Some upstream git repo may rebase in the future, which means current
revision may disappear from the upstream repo after the rebase.
current git fetcher can not handle this case, because the git mirror
tar ball is per repo, and may also change in the rebase and lost the
current revision info.
To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this
case, even upstream rebase, the git mirror still has the current
revision info.
- for rebaseable repo, generate mirror tar ball by default, since the
repo may change in the future.
Richard Purdie [Tue, 8 Mar 2011 19:23:34 +0000 (11:23 -0800)]
bitbake/fetch2: Allow local file:// urls to be found on mirrors
With the current implementation, file:// urls as used by sstate don't access the
mirror code, breaking sstate mirror support. This change enables the usual
mirror handling. To do this, we remove the localfile special case, using the basename
paramemter instead. We also ensure the downloads directory is checked for files.
The drawback of this change is that file urls containing "*" globing require special
casing in the core.
Richard Purdie [Wed, 20 Apr 2011 01:13:23 +0000 (02:13 +0100)]
bitbake/fetch2: Fix the problems introduced by the git fetcher AUTOREV fix
The ordering constrains on the urldata_init functions are not straight
forward. To avoid further problems, create a helper function to setup
the source revisions which the init functions can all at the appropriate
point.
Richard Purdie [Tue, 19 Apr 2011 22:31:09 +0000 (23:31 +0100)]
bitbake/fetch2/git: Fix a bug where AUTOREV and the git fetcher interact badly
Fix a bug where ud.branches were being referenced before it was set by
the git fetcher when using AUTOREV. To do this some ordering needed
to be changed. This fixes errors like:
ERROR: Error parsing /recipes-kernel/linux/rt-tests_git.bb: Failure expanding variable
SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
AttributeError: 'FetchData' object has no attribute 'branches'
Darren Hart [Wed, 9 Mar 2011 20:55:20 +0000 (12:55 -0800)]
bitbake docs: use dblatex to build the pdf bitbake manual
Fix [BUGID #593]
The current manual build fails for printing formats which use latex as an
intermediate format. This bug has been reported in multiple locations and I
haven't found a solution posted to any of them.
Using --with-dblatex uses dblatex to make the conversion and successfully
generates the pdf. It adds a dependency on dblatex and its dependencies.
Yu Ke [Mon, 18 Apr 2011 05:53:53 +0000 (13:53 +0800)]
git fetcher: make tag back to work, fix Yocto bug 972
In current git fetcher, tag does not work due to commit http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=5920e85c561624e657c126df58f5c378a8950bbc. Tag is not in sha256 form, so it will be treated invalid, and silently replaced by latest revision.
To fix it, this patch treat tag name as branches name, thus it will be handled correctly later. Thanks Richard for reviewing and proposing the better approach.
Richard Purdie [Sun, 10 Apr 2011 17:52:29 +0000 (10:52 -0700)]
bitbake/cooker/codeparser: Ensure the code parser cache is saved for each parsing process
Before this change, the codeparser cache was only being saved for the main
server process. This is suboptimal as it leaves code being re-evaluated at
task execution time and increases parse time.
We use the multiprocess Finalize() functionality to ensure each process
saves out its cache. We need to update the cache save function to be multiprocess
friendly with locking.
Chris Larson [Fri, 8 Apr 2011 14:10:40 +0000 (07:10 -0700)]
build: run fakeroot if FAKEROOT is set
FAKEROOT contains the fakeroot command to be run. This should not be set for
those using pseudo + FAKEROOTENV, but should be set for people like upstream
OE, and its value should be 'fakeroot' for them, for compatibility with the
old mechanisms.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Thu, 10 Feb 2011 00:57:03 +0000 (17:57 -0700)]
persist_data: make SQLTable a context manager
This can be used for more control over the underlying transactions. Unlike
the context manager of, say, a file object, we can still use the object even
after the end of a given with block, as the context manager exit only ensures
we've committed to the database, not that we have closed the database.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Mon, 4 Apr 2011 18:31:39 +0000 (11:31 -0700)]
event: improve output when eventhandler exec fails
- Name the event handler by its actual name, so the traceback shows it rather
than 'tmpHandler'.
- Rather than immediately aborting when encountering an event handler error,
display an error message and try to continue.
- Show a traceback for ordinary exceptions, skipping the first entry in the
traceback, so it only shows the useful information.
- Show an error, but no traceback, for SystemExit with a code other than 0.
- For for SystemExit with a code of 0, simply continue silently.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
In Poky fakeroot processing is handled exclusively by FAKEROOTENV. The
fakeroot binary itself is never invoked directly. (This is due to Poky
using pseudo, a fakeroot replacement that is more efficiently run by
setting environment variables.)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
Mark Hatle [Thu, 17 Mar 2011 23:10:18 +0000 (19:10 -0400)]
runqueue.py: Enable PSEUDO (fakeroot) before the fork
PSEUDO when used as the fakeroot program is usually preloaded into memory,
but disabled by default. The trigger for enabling it is a set of environment
variables and a fork() or exec*() operation.
We need to setup the environment, specifically PSEUDO_DISABLED=0, prior to
the fork() so that python tasks can be run under PSEUDO control.
This patch is based on the work in Poky primarily from Richard Purdie.
See the following Poky commits for detailed history on this change:
bitbake Revert bitbake exec() and go back to fork() for performace wins (first draft)
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
commit 52aada2eaf08d57d5f715f155f2d878831dbaab0
Author: Richard Purdie <rpurdie@linux.intel.com>
Date: Tue Dec 21 00:51:24 2010 +0000
bitbake/runqueue.py: Somehow the python environment mapping is failing so do it manually
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
commit 1b08a7eb8b708f4d0fc119cf89deb450fa62fea1
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Mon Feb 28 15:31:20 2011 +0000
bitbake/cache/runqueue.py: Move workload for recipe parsing to the child process
Parsing the recipe in the parent before forking off the child worker
can mean the parent doesn't hit the idle loop and becomes a bottleneck
when lauching many short lived processes.
The reason we need this in the parent is to figure out the fakeroot
environmental options. To address this, add the fakeroot variables
to the cache and move recipe loadData into the child task.
For a poky-image-sato build this results in about a 2 minute speedup
(1.8%).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Integrated and merged into bitbake upstream
Signed-off-by: Mark Hatle <Mark.Hatle@windriver.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Wed, 16 Mar 2011 15:07:29 +0000 (08:07 -0700)]
Initial work on getting bitbake working under pypy
- use os.chmod, not os.fchmod, as the latter is missing under pypy
- rearrange our imports a bit
- don't die if sqlite3 is missing shared cache support
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Tue, 8 Mar 2011 16:33:40 +0000 (09:33 -0700)]
lockfile: ask for forgiveness, not permission
Create the lockfile directory if it doesn't exist, rather than erroring out if
it doesn't exist (was also racy).
Also improve the wording of the error message shown when the lockfile's
directory is not writable.
Note for the future, this function should be improved, particularly with
regard to its exception handling. It should be catching the *exact*
exception(s) it will encounter when the file is locked, and continuing in that
case only. If it did that, there'd be no need for the proactive directory
writability check, as bb.utils.lockfile() would raise an appropriate IOError
for that case.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Joshua Lock [Wed, 2 Feb 2011 12:48:10 +0000 (12:48 +0000)]
bitbake/cooker: don't drop possible_world ref count
We need this if we want to run the buildWorldTargetList function more than
once, for example in a UI where we can change the MACHINE and DISTRO as much
as we like before triggering a build.
Joshua Lock [Tue, 1 Feb 2011 15:23:47 +0000 (15:23 +0000)]
bitbake/progress: add method to pulse the progress bar
When we're running a long operation with indeterminate duration it's useful
to use the gtk.ProgressBar's pulse method to show that something is happening
but we don't know how long it will take.
Joshua Lock [Wed, 12 Jan 2011 12:24:04 +0000 (12:24 +0000)]
introduce crumbs.TaskListModel a gtk.ListStore subclass
Provide a gtk.ListStore subclass which includes a function,
populate(), which takes as input the data emitted by
bb.event.TargetsTreeGenerated and fills the ListStore model
appropriately.
Furthermore convenience functions are provided by which the caller can
get gtk.TreeModel subclasses which provide filtered views of the data.
Joshua Lock [Fri, 7 Jan 2011 12:57:18 +0000 (12:57 +0000)]
bitbake/[cooker|cache]: cache summary, license and group. Add to targets tree
Add summary, license and group metadata to RecipeInfo and the cache.
Unfortunately this impacts parse speed but gives us a much richer set of
metadata to expose through UI's which can be accessed via the
generateTargetsTree command.
Joshua Lock [Fri, 7 Jan 2011 12:00:09 +0000 (12:00 +0000)]
bitbake/cooker: reduce code duplication
Move runqueua and taskdata initialisation into a new function,
prepareTreeData(), so that generateDepTreeData() and
generateTargetsTreeData() are not duplicating the same logic.
Joshua Lock [Tue, 4 Jan 2011 20:28:20 +0000 (20:28 +0000)]
implement command to find configuration files for a config variable
Some configuration variables (MACHINE, MACHINE-SDK and DISTRO) set which
confguration files bitbake should use.
The added command , findConfigFiles, enables a UI to query which files are
suitable values for a specified parameter.
Joshua Lock [Tue, 4 Jan 2011 20:08:51 +0000 (20:08 +0000)]
implement command to get all possible targets and their dependencies
Add a new command generateTargetsTree() which returns a dependency tree of
possible targets (tasks and recipes) as well as their dependency information.
Optional parameter 'klass' also ensures any recipes which inherit the
specified class path (i.e. 'classes/image.bbclass') are included in the model
Chris Larson [Sun, 27 Feb 2011 22:11:18 +0000 (15:11 -0700)]
server: use local fixed _bootstrap when appropriate
When running on python versions 2.6.0 through 2.6.2, we use a local copy
of the python 2.6.6 _bootstrap method of Process, to ensure that we have
the fix for http://bugs.python.org/issue5313. This avoids the "hang" of
the bitbake process at 0% progress during the parsing on older distros
like Fedora 12.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Thu, 24 Feb 2011 20:16:43 +0000 (13:16 -0700)]
cooker: use BBHandler.inherit for INHERIT
Ideally we'd avoid direct BBHandler usage, but honestly, to say the whole
bb.parse abstraction is incredibly leaky is an understatement. If we try to
make handle() not reparse classes, things get ugly fairly quickly, as
inherit() calls handle() itself after adding the class to the inherit cache.
This change fixes it so we no longer risk reparsing a class if:
- it's listed in INHERIT multiple times
- it's listed in INHERIT and is 'inherit'ed from a class in INHERIT
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Richard Purdie [Thu, 24 Feb 2011 22:02:19 +0000 (22:02 +0000)]
build.py: Operate logfile handling at the task level
Previously, not all task output could end up in the logfile and
sub functions run using exec_func() calls would end up logging to
/dev/null. This is extrememly confusing for users who want to find
the output from a given task as there are several possible log files,
or it might not be there at all.
This patch changes exec_task() to redirect standard output/error
to the logfile along with any python logging output. This change is
applied for all python/shell tasks so it doesn't matter whether output
comes from os.system() calls or wherever, the logs are always correct.
This also removed the ton of log files previous code could generate
in ${T}, again making this simpler for users.
Based upon patches from Poky.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If expanding a variable triggers an exception the caller currently has no
way to supress the error message or otherwise handle the siutation. An
example of where this is a problem is "bitbake -e" showing tracebacks and
errors for variables like SRCPV in OE/Poky.
Secondly in a chained expansion fails, log mesages are recorded for
every step of the expansion, not just the innermost error which is
where the real failure occured.
To fix this we introduce a new exception ExpansionError which callers
can handle as appropriate.
Richard Purdie [Wed, 23 Feb 2011 11:48:01 +0000 (11:48 +0000)]
bitbake/data_smart: Improve the way lazyassignment works
Currently, if a variable has been set with ??= and the code looks it up
before the data finalisation phase, no value is found. This is causes
serious problems for anonymous python functions which manipulate data, or
for the fetcher revision handling code where revisions can be set with
??=.
There is also a significant performance implication for processing lazy
assignment in finalise.
Moving the check for a default value into getVarFlag addresses both
the timing issue and the performace. This change gives a 7% real time
performance improvement to parsing the Poky metadata. The cost of the
check at this point is minimal since we have all the data flags available.
Richard Purdie [Wed, 23 Feb 2011 11:09:07 +0000 (11:09 +0000)]
bitbake/cooker: Fix parsing failure zombie problem
When parsing if a SystemExit event is triggered, it causes the parsing thread to
exit and the main process hangs waiting for it to finish indefintely. Add code to
catch BaseExceptions and raise these with the main process gracefully instead
of just hanging indefinitely with zombie processes.
Mark Hatle [Fri, 11 Feb 2011 17:43:54 +0000 (11:43 -0600)]
fetch2: unpack revision
Revise the unpack function to have a way to disable the unpack. This is
based on the work from "Andreas Oberritter <obi@opendreambox.org>", see
http://cgit.openembedded.net/cgit.cgi/openembedded/commit/?id=2bdfe8519eda8067845019a699acdf19a21ba380
In addition, the to_boolean function comes from the work of
"Chris Larson <chris_larson@mentor.com>", see
http://cgit.openembedded.net/cgit.cgi/openembedded/commit/?id=900cc29b603691eb3a077cb660545ead3715ed54
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
