Chris Larson [Fri, 11 Feb 2011 16:42:38 +0000 (09:42 -0700)]
persist_data: drop SQLData indirection
The common case (if not the only case) is to only use a single domain. The
only reason SQLData exists is to make it easier to delete a domain. Yet,
there's no need for us to delete a domain if SQLTable knows how to clear
itself out. So, add clear() to the table and pass the domain to persist().
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Mark Hatle [Tue, 8 Feb 2011 00:18:18 +0000 (18:18 -0600)]
fetch2: Add SRPM knowledge
Enable the fetcher to be able to unpack and SRPM. By default the system will
unpack the contents of the SRPM into the WORKDIR.
A new syntax "unpack=file" was developed for the SRC_URI, to allow for a
recipe to extract a specific file within an SRPM. An unpack operation will
then be executed on the extracted file.
In order to apply extracted patches (or unpack files not specified with
unpack), you must specify the path using WORKDIR, i.e.:
Richard Purdie [Mon, 7 Feb 2011 09:13:44 +0000 (09:13 +0000)]
bitbake/fetch2/git: Fix localpath to point at the clone repo since we no longer always generate a mirror tarball and it isn't a good guide to fetcher success
Richard Purdie [Fri, 4 Feb 2011 10:30:54 +0000 (10:30 +0000)]
bitbake/fetch2: Drop old md5 handling code
Drop some old md5 functions since we have improved functionality now which includes
sha256 checksum support. This stops each download being md5 checksumed twice.
Also change ".md5" stamp extentions to ".done" to better describe its use as a
download complete marker file and no longer write the md5 sum to the files.
Richard Purdie [Sun, 6 Feb 2011 10:20:18 +0000 (10:20 +0000)]
bitbake/fetch2/git: Switch to completely use bare clones
We no longer need index/workdir support in the mirror tree and it causes all
kind of reference naming problems.Simplifying the code to remove this and use
just bare clones addresses this problem.
We increase the "version" number on the mirror tarballs to reflect the change
and ensure older mirror tarballs are not used as they would break.
Richard Purdie [Thu, 3 Feb 2011 20:16:05 +0000 (20:16 +0000)]
bitbake/fetch2/git: Ensure deleted branches upstream are removed, all branch tags are forced to the upstream revisions and all revisions and tags are always fetched
Richard Purdie [Thu, 3 Feb 2011 18:24:25 +0000 (18:24 +0000)]
bitbake/fetch2: Remove old local file acceleration code
Since local mirror fetches are always symlinked from the download directory
directly, there is no need for this premirrors hack which doesn't cover
mirrors and also abuses the localpath variable with inconsistent results.
Richard Purdie [Thu, 3 Feb 2011 18:22:06 +0000 (18:22 +0000)]
bitbake/fetch2: Ensure that mirror fetches are symlinked from the download directory
When files are fetched from a mirror source that happens to be local,
ensure links are created for the file since subsequent fetch calls
can then follow the links to find files.
Any other approach such as the existing manipulations of localpath
internally to the fetcher are prone to errors, races and other issues.
Richard Purdie [Thu, 3 Feb 2011 18:19:23 +0000 (18:19 +0000)]
bitbake/fetch2: When encoding a file:// url, drop user and host information
When processing a cvs SRC_URI to a file:// mirror, the user and host information
will break the mirror processing. This patch addresses it by only constructing
valid urls.
Yu Ke [Wed, 26 Jan 2011 12:14:06 +0000 (20:14 +0800)]
fetch2: Allow multiple src rev to be used in one src_uri
* SRC_URI format:
the SRC_URI are extended to allow multiple src rev:
name=<name1>,<name2>,...<name-n>
branch=<branch1>,<branch2>,...,<branch-n>
also SRCREV can be defined with
SRCREV_<name1> = xxxxx
SRCREV_<name2> = xxxxx
* FetchData extention
to support multiple src rev, several FetchData data are added:
- FetchData.names: list of name in SRC_URI, one name per srcrev. name is the index of revision and branch
- FetchData.revisions: dictionary of name->revision.
- FetchData.branches: dictionary of name->branch.
For example, linux-yocto recipes becomes:
SRC_URI = "git://git.pokylinux.org/linux-yocto-2.6.37;protocol=git;branch=${KBRANCH},meta;name=machine,meta"
* generic revision handling extension
the related revision handling code in fetch2.__init__.py are changed accordingly. the major change is add name parameter to indicate which src rev to handling. originally there is one src rev per FetchData, so FetchData parameter is enough. now since one FetchData has multiple src rev, it is necessary to use FetchData + name to specifiy src rev.
* git extension
git fetcher are also revised to take advantage of the multiple src rev in FetchData. especially the download() method are enhanced to fetch multiple src rev.
* other fetcher (svn, hg, ...) does not support multiple src rev. they just sync the API to add name, and then simply ignore the name. no actually functional change
Yu Ke [Tue, 18 Jan 2011 16:58:42 +0000 (00:58 +0800)]
git.py: Remove the source tree tarball archives
Since we're now always providing the git source control files it becomes
pointless to handle the tarballs of specific git revisions so drop this
part of the fetcher.
Yu Ke [Tue, 18 Jan 2011 16:22:13 +0000 (00:22 +0800)]
bb.fetch2: Add git unpack
The git download method clones the git repository to the local machine. The unpack process
can be optimised to be a local to local machine clone or a direct readtree operation to the
destination using git.will clone git repo to local, so git unpack can be simplified
to only checkouting the code to the work dir. For fullclone case, we also
need to manually copy all the ref info, which is needed by the later do_kernel_checkout().
Rather than use hardlinks, we reference the repository using alternatives since the
download directory may be on a different filesystem.
Yu Ke [Mon, 24 Jan 2011 06:58:26 +0000 (14:58 +0800)]
bb.fetch2: add "BB_NO_NETWORK" option
Sometime user want a purely local fetching, i.e. using local mirror without
any remote netowrk access. BB_NO_NETWORK option is introduced for this purpose
check_network_access() is the guard for BB_NO_NETWOKR option. it should be
put in any place that fetcher use network access
Yu Ke [Tue, 18 Jan 2011 15:35:30 +0000 (23:35 +0800)]
git.py: split download to download() and build_mirror_data()
the download is to fetch the source from URL, the build_mirror_data is
to create the mirror tar ball. the original go() method mix them together,
it is more clean to split them.
Yu Ke [Tue, 18 Jan 2011 14:41:23 +0000 (22:41 +0800)]
bb.fetch: add fetch version to distinguish bb.fetch and bb.fetch2
there is case that we need to distingush bb.fetch and bb.fetch2,
and use different API for bb.fetch and bb.fetch2. so it is necessary
to add version info for distinguish purpose
Yu Ke [Tue, 18 Jan 2011 14:08:09 +0000 (22:08 +0800)]
bb.fetch2: revise the Fetch.unpack API
change the unpack to use the urldata and rootdir parameter
- urldata is the FetchData instance
- rootdir is the dir to put the extracted source. the original unpack
use current dir (os.getcwd) as destination dir, which is not flexible
and error-prone (error will occur if caller not chdir to dest dir)
Yu Ke [Mon, 27 Dec 2010 08:20:34 +0000 (16:20 +0800)]
Fetcher: only set __BB_DONT_CACHE when SRCREV = "${AUTOREV}"
we should cache SRCREV whenever possible, the only exception is
when SREREV is auto rev. so change the logic to only set __BB_DONT_CACHE
at SRCREV = "${AUTOREV}" case
Yu Ke [Mon, 27 Dec 2010 01:31:38 +0000 (09:31 +0800)]
Fetcher: break the "SRCREVINACTION" deadlock
Current fetcher has annoying "SRCREVINACTION" deadlock,
which occurs when SRCREV=${AUTOREV}=@bb.fetch.get_srcrev():
get_srcrev()->setup_localpath()->srcrev_internal_helper()
->evaluate SRCREV->get_srcrev()
current fetcher resolve the deadlock by introducing a
"SRCREVINACTION" condition check. Althoguh it works, it is
indeed not clean.
This patch use antoehr idea to break the deadlock: break
the dependency among SRCREV and get_srcrev(), i.e. assign
a specific keyword "AUTOINC" to AUTOREV. when Fetcher meet
this keyword, it will check and set the latest revision to
urldata.revision. get_srcrev later can use the urldata.revision
for value evaluation(SRCPV etc). In this case, SRCREV no longer
depends on get_srcrev, and there is not deadlock anymore.
Yu Ke [Sat, 8 Jan 2011 08:40:18 +0000 (16:40 +0800)]
bb.fetch2: add urldata_init call for Fetch class
FetchData has some fetch method specific data, and only fetch method knows how
to initialize it. originally it is mostly initialized in Fetch.localpath().
But now there is requirement to call Fetch.latest_revision() before
Fetch.localpath(), thus require another earlier place for initialization. so
urldata_init is introduced for this purpose. it will be called in FetchData:__init__
and make all the Fetch functions useable after that.
Chris Larson [Tue, 8 Feb 2011 20:41:36 +0000 (13:41 -0700)]
Shift traceback pre-formatting into LogHandler
It's cleaner to leave it behaving as usual, passing the exception data in the
exc_info attribute of the LogRecord where it normally lies, and then let
LogHandler make it pickleable so it can be sent to the UI.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Tue, 8 Feb 2011 16:53:47 +0000 (09:53 -0700)]
build: improve set -x handling
- set -x always, not just when debugging
- set -x immediately before executing the function, rather than at the top of
the script, so the output is no longer cluttered with all the variable and
function definitions
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Mon, 7 Feb 2011 17:50:27 +0000 (10:50 -0700)]
Add initial bitbake-layers script
This script has subcommands which operate against your bitbake layers, either
displaying useful information, or acting against them. Currently, it only
provides a show_appends command, which shows you what bbappends are in effect,
and warns you if you have appends which are not being utilized.
Currently, a bug exists when using this due to the DataContext stuff, but I'm
not certain as to the root cause, it appears to be the bb package relying
implicitly on the way the bitbake script does things. A fix for that issue
will be forthcoming, as will further subcommands.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Richard Purdie [Fri, 28 Jan 2011 10:21:41 +0000 (10:21 +0000)]
bitbake/runqueue.py: Avoid starvation of events to the server
The server UI was reading 1024 bytes, then sleeping for 0.25 seconds. Since
most new LogRecord events are larger than this it leads to a build up of data
which is only processed slowly, leading to a bottleneck and a slow down of
all bitbake processes.
Thanks to Dongxiao Xu <dongxiao.xu@intel.com> for the great work in debugging
this. A large value has been left in for the read() command just to ensure some
fairness amongst process handling if a task tries to log truly huge amounts of
data to the server, or goes crazy and ensures the main loop doesn't stall.
Chris Larson [Mon, 17 Jan 2011 15:50:40 +0000 (08:50 -0700)]
cooker: also respond to SystemExit for execute_runqueue
Without this, bitbake appears to hang when encountering an invalid task. In
reality, the SystemExit flows up uncaught, the server may go away, and the UI
doesn't know about it, so requires us to ^C to exit.
This is a temporary measure. Poky's bitbake handles invalid tasks
differently, and avoids this *particular* issue, but we do need to ensure that
the server responds to SystemExit from the idle function in a sane way.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
