Yu Ke [Mon, 24 Jan 2011 06:58:26 +0000 (14:58 +0800)]
bb.fetch2: add "BB_NO_NETWORK" option
Sometime user want a purely local fetching, i.e. using local mirror without
any remote netowrk access. BB_NO_NETWORK option is introduced for this purpose
check_network_access() is the guard for BB_NO_NETWOKR option. it should be
put in any place that fetcher use network access
Yu Ke [Tue, 18 Jan 2011 15:35:30 +0000 (23:35 +0800)]
git.py: split download to download() and build_mirror_data()
the download is to fetch the source from URL, the build_mirror_data is
to create the mirror tar ball. the original go() method mix them together,
it is more clean to split them.
Yu Ke [Tue, 18 Jan 2011 14:41:23 +0000 (22:41 +0800)]
bb.fetch: add fetch version to distinguish bb.fetch and bb.fetch2
there is case that we need to distingush bb.fetch and bb.fetch2,
and use different API for bb.fetch and bb.fetch2. so it is necessary
to add version info for distinguish purpose
Yu Ke [Tue, 18 Jan 2011 14:08:09 +0000 (22:08 +0800)]
bb.fetch2: revise the Fetch.unpack API
change the unpack to use the urldata and rootdir parameter
- urldata is the FetchData instance
- rootdir is the dir to put the extracted source. the original unpack
use current dir (os.getcwd) as destination dir, which is not flexible
and error-prone (error will occur if caller not chdir to dest dir)
Yu Ke [Mon, 27 Dec 2010 08:20:34 +0000 (16:20 +0800)]
Fetcher: only set __BB_DONT_CACHE when SRCREV = "${AUTOREV}"
we should cache SRCREV whenever possible, the only exception is
when SREREV is auto rev. so change the logic to only set __BB_DONT_CACHE
at SRCREV = "${AUTOREV}" case
Yu Ke [Mon, 27 Dec 2010 01:31:38 +0000 (09:31 +0800)]
Fetcher: break the "SRCREVINACTION" deadlock
Current fetcher has annoying "SRCREVINACTION" deadlock,
which occurs when SRCREV=${AUTOREV}=@bb.fetch.get_srcrev():
get_srcrev()->setup_localpath()->srcrev_internal_helper()
->evaluate SRCREV->get_srcrev()
current fetcher resolve the deadlock by introducing a
"SRCREVINACTION" condition check. Althoguh it works, it is
indeed not clean.
This patch use antoehr idea to break the deadlock: break
the dependency among SRCREV and get_srcrev(), i.e. assign
a specific keyword "AUTOINC" to AUTOREV. when Fetcher meet
this keyword, it will check and set the latest revision to
urldata.revision. get_srcrev later can use the urldata.revision
for value evaluation(SRCPV etc). In this case, SRCREV no longer
depends on get_srcrev, and there is not deadlock anymore.
Yu Ke [Sat, 8 Jan 2011 08:40:18 +0000 (16:40 +0800)]
bb.fetch2: add urldata_init call for Fetch class
FetchData has some fetch method specific data, and only fetch method knows how
to initialize it. originally it is mostly initialized in Fetch.localpath().
But now there is requirement to call Fetch.latest_revision() before
Fetch.localpath(), thus require another earlier place for initialization. so
urldata_init is introduced for this purpose. it will be called in FetchData:__init__
and make all the Fetch functions useable after that.
Chris Larson [Tue, 8 Feb 2011 20:41:36 +0000 (13:41 -0700)]
Shift traceback pre-formatting into LogHandler
It's cleaner to leave it behaving as usual, passing the exception data in the
exc_info attribute of the LogRecord where it normally lies, and then let
LogHandler make it pickleable so it can be sent to the UI.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Tue, 8 Feb 2011 16:53:47 +0000 (09:53 -0700)]
build: improve set -x handling
- set -x always, not just when debugging
- set -x immediately before executing the function, rather than at the top of
the script, so the output is no longer cluttered with all the variable and
function definitions
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Mon, 7 Feb 2011 17:50:27 +0000 (10:50 -0700)]
Add initial bitbake-layers script
This script has subcommands which operate against your bitbake layers, either
displaying useful information, or acting against them. Currently, it only
provides a show_appends command, which shows you what bbappends are in effect,
and warns you if you have appends which are not being utilized.
Currently, a bug exists when using this due to the DataContext stuff, but I'm
not certain as to the root cause, it appears to be the bb package relying
implicitly on the way the bitbake script does things. A fix for that issue
will be forthcoming, as will further subcommands.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Richard Purdie [Fri, 28 Jan 2011 10:21:41 +0000 (10:21 +0000)]
bitbake/runqueue.py: Avoid starvation of events to the server
The server UI was reading 1024 bytes, then sleeping for 0.25 seconds. Since
most new LogRecord events are larger than this it leads to a build up of data
which is only processed slowly, leading to a bottleneck and a slow down of
all bitbake processes.
Thanks to Dongxiao Xu <dongxiao.xu@intel.com> for the great work in debugging
this. A large value has been left in for the read() command just to ensure some
fairness amongst process handling if a task tries to log truly huge amounts of
data to the server, or goes crazy and ensures the main loop doesn't stall.
Chris Larson [Mon, 17 Jan 2011 15:50:40 +0000 (08:50 -0700)]
cooker: also respond to SystemExit for execute_runqueue
Without this, bitbake appears to hang when encountering an invalid task. In
reality, the SystemExit flows up uncaught, the server may go away, and the UI
doesn't know about it, so requires us to ^C to exit.
This is a temporary measure. Poky's bitbake handles invalid tasks
differently, and avoids this *particular* issue, but we do need to ensure that
the server responds to SystemExit from the idle function in a sane way.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Mon, 10 Jan 2011 16:20:50 +0000 (09:20 -0700)]
Inject taskpid into log records via our log handler
It turns out that while log filters added with addFilter are only associated
with that logger, and not its children, handlers are inherited, and handlers
can be filters. So, let's add filtering to our existing LogHandler class
which dispatches our log records as bitbake events.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
The current parameters are not useful to the stampfile generator function
as they can't uniquely define a task. This updated things so the
parameters can identify unique tasks.
Yu Ke [Wed, 29 Dec 2010 07:28:48 +0000 (15:28 +0800)]
bb.utils: check if lock file is writable, to fix Yocto bug 606
Bug 606 report that if $DL_DIR is read-only, do_fetch will
simply hang without any error message.
The root cause is that: bb.fetch.go()->bb.utils.lockfile()
will try to lock file ${DL_DIR}/xxxxx.lock. Since ${DL_DIR}
is read-only, it will cause IOError exception. Although
lockfile() can catch the exception, currently code simply
ignore all the exception and continue the loop. it make
sense if the exception is caused by locking contention,
but in the read-only $DL_DIR case, it cause endless waiting
unfortunately.
So this patch add read-only check for lockfile to avoid the
silent hang.
Chris Larson [Fri, 7 Jan 2011 15:51:43 +0000 (08:51 -0700)]
Merge branch 'poky-sync'
* poky-sync:
build: kill unneeded environment setup bits
parse: save python functions into the metadata
Include exported variables in task process env
runqueue: start implementing quieterrors
Overhaul environment handling
runqueue: resurrect use of file objects for pipein/pipeout
runqueue: clean up message log levels
runqueue: use correct task ID when checking validity of setscene tasks
Add support for 'noexec' tasks
runqueue: implement cache checking from metadata
runqueue: pass task hash information to tasks
runqueue: avoid unnecessary delays
Implement task signatures
data: Add emit_func and generate_dependencies
codeparser: Implement persistent cache
data_smart: track variable references
codeparser: add module for parsing shell/python
runqueue: implement scenequeue
fetch: add SRC_URI checksum
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 7 Jan 2011 15:38:41 +0000 (08:38 -0700)]
cache: don't expand variables for skipped recipes
Errors can result from these expansions, but for skipped recipes, we
shouldn't care about those failures. This fixes the same issue which
Richard Purdie fixed in poky, commit 847b717.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Tue, 4 Jan 2011 20:34:08 +0000 (13:34 -0700)]
parse: pass filename, lineno into the ast
We will be needing this information to improve the tracebacks of python code
from the metadata, as well as to give the user information about where
variables were defined, so they know how it ended up the way it is.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 10 Sep 2010 18:14:54 +0000 (11:14 -0700)]
Fix exit code display for task failure
Per the python documentation, os.waitpid returns the exitcode shifted up by 8
bits, and we weren't compensating, resulting in a display of 'failed with 256'
when a worker process exits with a code of 1.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Only mark fn as clean if it is clean.
This saves us from removing (prematurely added) fn from our clean set
and saves me a few percent of runtime (and misleading debugging output
from remove()).
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Previously we emitted two newlines for export and unexport.
One newline for export and unexport is enough (and makes the scripts
look better and a tad smaller).
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Richard Purdie [Sun, 28 Nov 2010 17:39:09 +0000 (17:39 +0000)]
Overhaul environment handling
Currently, anything whitelisted in the environment makes it into the worker
processes. This is undesireable and the worker environment should be as
clean as possible. This patch adapts bitbake sosme variables are loaded into
bitbake's datastore but not exported by default. Any variable can be exported
by setting its export flag.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
Kevin Tian [Tue, 14 Dec 2010 11:21:24 +0000 (19:21 +0800)]
runqueue: use correct task ID when checking validity of setscene tasks
sstate hash validation is done at initialization of RunQueueExecuteScenequeue.
However the index of 'valid' list returned from the validation doesn't
correspond to setscene task ID. It's just an intermediate namespace between
runqueue and sstate hash func. Use it as setscene task ID fully mess the flow.
Previously this doesn't cause trouble because all setscene tasks are passed. Commit 58396a5d24c62710fd0a9f3780d84ac8a95d8e7c add 'noexec' concept to setscene
tasks which grabs some tasks out of the list and thus trigger this problem
Without this fix there're ~50 recipes (gzip-native, glib, ...) rebuilt weirdly
with a minimal build, even though existing sstate packages could accelerate them.
there's another typo using wrong task ID in a debug message which further hide
this issue
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
