Chris Larson [Tue, 14 Dec 2010 15:10:36 +0000 (08:10 -0700)]
Fix PWD issue with new exec_func_shell
The previous attempt was incorrect. The issue isn't that subprocess fails to
set PWD, it's that PWD is in the metadata, inherited from the environment, and
is re-exported, overwriting the actual accurate one in the shell environment
with the old one from the metadata. So, ensure that PWD in the metadata is
not exported.
We can ditch this when the environment handling is reworked (e.g. poky's
commit to do so).
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 10 Dec 2010 23:53:19 +0000 (16:53 -0700)]
Rename command events, adjust compareRevisions
- Moved the logic for comparing revisions from cooker into command
- Removed 'Cooker' from the event names
- Renamed the 'ExitCode' event into CommandExit, and changed CommandFailed to
be a subclass of CommandExit
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Wed, 22 Sep 2010 01:11:54 +0000 (18:11 -0700)]
Rework the persist_data API
Rather than having to run .addDomain() and then .getValue(domain, key),
.setValue(domain, key), etc, now it just works as mappings.
As an example:
setValue(domain, key) -> persist[domain][key] = value
It also arranges things so we can construct objects of this type using any
arbitrary filename/path for the sqlite3 database, rather than being so
tightly bound to the metadata.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 10 Dec 2010 15:51:47 +0000 (10:51 -0500)]
Kill the uncaught exception handler
We now wrap the main() in a try/except, ensuring that both the main portion of
bin/bitbake and the UI raising an exception will be shown to the user. For
the server and workers, we can ensure in the server itself that exceptions are
handled correctly.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Tue, 23 Nov 2010 22:30:45 +0000 (15:30 -0700)]
cooker: use re match, not search in re_match_strings
We want to match the requested pattern at the beginning of the string,
otherwise things behave in an unintuitive manner wrt ASSUME_PROVIDED (e.g.
ASSUME_PROVIDED += "gtk+" will also assume foo-gtk+ is provided), and the user
can always use '.*gtk+' to get the old behavior.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Where a variable name consisted of an append/prepend combined with an override
and there was also an append/prepend to the variable, the override could be lost
if the override was not in OVERRIDES.
For example:
FOO = "A"
FOO_append = "B"
FOO_append_virtclass-native = "C"
could result in "AB" even though virtclass-native was in OVERRIDES.
With this patch applied, the result is "ABC" as would be expected.
The problem was the deletion of the _append/_prepend flag was happening
if *any* append/prepend was procesed, the result should really be that
it should contain any unprocessed append/prepend.
Kevin Tian deserves credit for looking into this and working out the
problem here.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Joshua Lock [Thu, 25 Nov 2010 15:15:10 +0000 (15:15 +0000)]
bitbake/depexp: Factor ProgressBar into a separate class in crumbs/
ProgressBar will be useful in other UI elements so make it it's own class.
Make ProgressBar a subclass of gtk.Dialog, rather than gtk.Window, so that we
can suggest the window manager parent the ProgressBar to the widget passed at
as parent.
Joshua Lock [Wed, 1 Dec 2010 14:40:21 +0000 (14:40 +0000)]
bitbake/xmlrpc: Modify xmlrpc server to work with Python 2.7
Python 2.7's library changes some of xmlrpclib's internal implementation such
that interacting with a proxy to BitBakes SimpleXMLRPCServer would cause
BitBake to crash.
The issue was traced to changes in the xmlrpclib.Transport implementation and
Python bug #8194 (http://bugs.python.org/issue8194).
This patch introduces a workaround by create a subclass of
xmlrpclib.Transport, which overrides the offending methods with the Python
2.6.6 implementation copy and pasted from the Python 2.6.6 xmlrpclib, and
using this BBTransport implementation for both xmlrpclib.Server objects we
create.
Joshua Lock [Mon, 6 Dec 2010 15:34:50 +0000 (15:34 +0000)]
bitbake/goggle: automatically show most recently added message
It seems likely that the user would want to view the most recently emitted
messages so this patch sets the message dislay treeview to scroll to any
newly added rows.
Joshua Lock [Thu, 11 Nov 2010 10:18:11 +0000 (10:18 +0000)]
bitbake/crumbs: fix the event name determination
Due to some recent change *somewhere* we need to explicitly look at the
name attribute on the instances class, rather than the name attribute of
the instance.
Joshua Lock [Wed, 1 Dec 2010 16:59:04 +0000 (16:59 +0000)]
bitbake/cooker: fix idle command processing in servers
idle command processing in each of the servers does not handle an explicit
None return value, which means the goggle UI ends up repeatedly adding
"Tasks Summary:" rows to the list.
This patch modifies BBCooker.buildTargets.buildTargetsIdle to return False
when BuildCompleted is fired, as is done in BBCooker.buildFile.buildFileIdle.
It may be that the correct way to fix this is to change the idle command
processing in the servers.
Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Mon, 6 Dec 2010 15:19:37 +0000 (08:19 -0700)]
Merged the following poky fetcher fixes:
- bitbake/fetch: Allow checking of a single url at a time (massive sstate
speed improvement)
- bitbake/fetch: Fix handling of mirrors when checking for url validity
- bitbake/fetch: When fetchers return errors, ensure any partial download is
cleared
- bitbake/fetch: Make URL checking slightly less verbose (distracting with the
sstate code)
- bitbake/fetch/git: Ensure fullclone repositories are fully fetched
- bitbake/fetcher: Deal with a ton of different bugs
- bitbake/fetch/git.py: Fix git fetcher to correctly use mirror tarballs
- bitbake/fetch/__init__.py: Abstract mirror variable handling and enhance to
accept \n delimitation in variables
- bitbake/fetch: fix logic to prevent fetches when the file already exists
- bitbake/fetch: ensure the go() method completes when not using premirrors
- Export ALL_PROXY for bitbake commands
- bitbake: Add proxy variables to standard export list
- bitbake/fetch/git: fix try_premirror() definition .Method takes 4
parameters.
- bitbake/fetch: fix by moving try_premirror() to the right place
- bitbake/fetch: add try_premirror method and special case git fetcher
- fetch: fix setting of localpath in SRC_URI parameters
- bitbake/fetch: ensure the mirrored repository is updated as required
- bitbake/git: define a forcefetch method
- bitbake/fetch: Respect forcefetch even when pulling from a mirror
- bitbake/git.py: Make sure different branches can have different revisions
without triggering build count increases
- bitbake/git.py: Allow tracking of branches in SRC_URI without cloning for
use with fullclone
- bitbake/git.py: Make sure a full clone checkout always updates
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Mark Hatle [Thu, 30 Sep 2010 20:04:18 +0000 (15:04 -0500)]
build.py: Add a symlink for the logging
[BUGID #375]
When configuring for the log file output, we generate a symlink to the
target filename. This link uses the same file naming, but without the
active pid, making it easier to see which log file is the last one
generated.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Chris Larson [Tue, 30 Nov 2010 15:25:13 +0000 (08:25 -0700)]
cooker: no cached in progressbar and add ETA
Rather than updating the progress bar based on the recipe being processed
(whether cached or parsed), consider only parsed recipes. This reduces the
instability in progress rate introduced by the cached entries, and allows the
ETA to be resurrected and be a bit more useful.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Mon, 29 Nov 2010 19:36:22 +0000 (12:36 -0700)]
Merge branch 'feature/parallel-parsing'
* feature/parallel-parsing:
cache: change to more incremental format
cooker: pass back child exceptions to the server
cache: bump cachever per master
cache: ensure 'pn' is included in the pkgvars
knotty: drop the ETA from the progressbar for now
cooker: ensure that the cache sync completes
cooker: get number of threads in constructor
cooker: don't add info for skipped recipes
cooker: show progress bar before initializing the cache
cooker: save progress chunk value (total/100)
cooker: stop loading the cache for -b
cooker: don't fire unnecessary parse progress events
cache: don't add info to cache if cache is disabled
cache: sync the cache file to disk in the background
Simplify cache syncing
Split up the ParseProgress event
Experimental usage of the 'progressbar' module
Implement parallel parsing support
cache: don't write out the cache unnecessarily
cache: create and use a RecipeInfo class
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Tue, 23 Nov 2010 19:02:40 +0000 (14:02 -0500)]
knotty: drop the ETA from the progressbar for now
Currently, the progress bar is an indication of the processing of our recipes,
which includes loading the cache file, then for each recipe, either adding the
existing cached information to the CacheData or parsing the recipe from disk.
These tasks clearly take different amounts of time, so the ETA is unreliable
today. We'll resurrect this functionality after we revamp the progress
handling, fully incorporating the load of the cache file.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Tue, 23 Nov 2010 17:46:49 +0000 (11:46 -0600)]
cooker: ensure that the cache sync completes
Without explicitly joining the thread, it's possible for the process to end
(e.g. after a bitbake -p) and kill off the thread without waiting for it to
exit cleanly. So, register the thread join with atexit.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Sun, 21 Nov 2010 18:59:05 +0000 (11:59 -0700)]
cooker: show progress bar before initializing the cache
This ensures that the time spent loading the cache from disk occurs with the
progress bar up. Though the progress bar stays at 0% during this period, I
think this is an improvement over the multi-second stall which occurred
previously before the progress bar came up. Ideally, we'd integrate cache
loading from disk into the progress display, but this is a first step.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 19 Nov 2010 18:46:42 +0000 (11:46 -0700)]
cooker: stop loading the cache for -b
Previously, the cache was actually being loaded from disk twice whenever using
-b or -e -b. This also moves the bb_cache instance into the CookerParser, as
it's not needed by the cooker itself at all.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 19 Nov 2010 05:47:36 +0000 (22:47 -0700)]
cache: sync the cache file to disk in the background
This version uses a thread rather than a process, to avoid problems with
waitpid handling. This gives slightly less overall build time reduction than
the separate process for it did (this reduces a -c compile coreutils-native by
about 3 seconds, while the process reduced it by 7 seconds), however this time
is quite insignificant relative to a typical build.
The biggest issue with non-backgrounded syncing is the perceived delay before
work begins, and this resolves that without breaking anything, or so it seems.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 19 Nov 2010 05:28:09 +0000 (22:28 -0700)]
Simplify cache syncing
Rather than adding nocache items to the cache, then copying the cache and
removing them to sync it, don't add them in the first place. Also use 'with'
for the cachefile.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 19 Nov 2010 04:52:15 +0000 (23:52 -0500)]
Split up the ParseProgress event
We now have ParseStarted, ParseProgress, and ParseCompleted. I think this
clarifies matters, and avoids sending all our statistics with every parse
progress event.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Fri, 19 Nov 2010 03:21:54 +0000 (20:21 -0700)]
Implement parallel parsing support
This utilizes python's multiprocessing module. The default number of threads
to be used is the same as the number of available processor cores, however,
you can manually set this with the BB_NUMBER_PARSE_THREADS variable.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Wed, 17 Nov 2010 04:55:19 +0000 (21:55 -0700)]
cache: don't write out the cache unnecessarily
If the only recipes's we reparsed this run were those flagged as not to be
cached, there's no point in re-saving the cache, as those items won't be
included anyway.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Chris Larson [Tue, 16 Nov 2010 19:58:52 +0000 (12:58 -0700)]
cache: create and use a RecipeInfo class
This class holds the particular pieces of information about a recipe which are
needed for runqueue to do its job.
By using it, I think we improve code clarity, reduce method sizes, reduce
overuse of primitive types, and prepare for parallel parsing. In addition,
this ditches the leaky abstraction whereby bb.cache attempted to hide the
difference between cached data and a full recipe parse. This was a remnant
from the way things used to be done, and the code using it had to know the
difference anyway. If we choose to reimplement caching of the full recipes,
we can do it in bb.parse, in a completely transparent way.
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Bob Foerster [Fri, 19 Nov 2010 20:39:22 +0000 (04:39 +0800)]
Prefer xrange over range for small performance gain.
range() allocates an actual list when called. xrange() is just an iterator
and creates the next range item on demand. This provides a slight
performance increase.
In python 3, range will do what xrange does currently, but the upgrade will
be handled by the 2to3 tool.
Chris Larson [Fri, 19 Nov 2010 02:51:51 +0000 (19:51 -0700)]
Revert "persist_data: cache connection and use cursor"
Caching the database connection can cause serious issues if it results in
multiple processes (e.g. multiple tasks) simultaneously using the same
connection.
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
