Richard Purdie [Fri, 21 Jun 2013 11:31:32 +0000 (11:31 +0000)]
data_smart: Fix bug with overrides and weak default values
(aka pay the cookie monster for weak defaults)
If you have code like:
MYVAR = "a"
MYVAR_override ??= "b"
then MYVAR will get the value "a" even when override is in OVERRIDES. The
reason is that the value of ??= is set as a flag not a value and the cookie
monster isn't paid.
The fix is to ensure appropriate payment is made for a defaultval varflag
matching the usual setVar case.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 20 Jun 2013 20:53:14 +0000 (21:53 +0100)]
data_smart: Ensure variable flags are accounted for in config data hash
Currently if the flags set against a variable in the base data store
change, it doesn't automatically trigger a reparse when it really
should. For example with the blacklist class setting:
will not trigger a reparse if only one entry is changed and a
blacklisted recipe can still be built.
I did consider using BB_SIGNATURE_EXCLUDE_FLAGS in here however it
doesn't make sense, we want to trigger a reparse when any of the
flags change too (which is different to the sstate signatures which
we wouldn't want to change in those cases).
[YOCTO #4627]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cristiana Voicu [Thu, 20 Jun 2013 14:21:17 +0000 (17:21 +0300)]
hob: save variables in conf/ directory
Until now the variables were saved in bitbake configuration,
now they are saved in configuration files, in order to be read by
bitbake-worker. This helps to assure the consistency for the rest
of the variables.
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cristiana Voicu [Thu, 20 Jun 2013 14:21:16 +0000 (17:21 +0300)]
hob: replace the init cooker with reset in order to keep the history
When the cooker initialization is called, the variables history is lost.
It need a reset, in order to load the configuration files.
Also, concatenated the value for INHERIT variable.
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cristiana Voicu [Thu, 20 Jun 2013 14:21:15 +0000 (17:21 +0300)]
bitbake: call the initConfiguration when reparsing
As it is implemented now, when parsing the configuration files is
needed, it also needs an initialization.
Also, implemented a function to create an extra configuration files
which can be used as postfile/prefiles when parsing.
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cristiana Voicu [Wed, 19 Jun 2013 14:23:28 +0000 (17:23 +0300)]
bitbake: remove the function for reparse because it does nothing
The command for reparse does nothing, so I have managed to remove it,
because it can be misleading. When a reparse is needed, it can be used
the parse function.
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cristiana Voicu [Thu, 20 Jun 2013 10:19:42 +0000 (13:19 +0300)]
bitbake & hob: modify some commands and call them from hob
This patch adds a new command to create an extra configuration
file, in order to create it from Hob.
Also, it calls the enable/disable data tracking when needed.
Modified the prefiles and postfiles variables because the cooker
expects some lists.
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 19 Jun 2013 13:03:39 +0000 (14:03 +0100)]
runqueue/build: Add recideptask flag
Currently, tasks like fetchall are slightly broken since if a recipe
has specific [depends] which occur after do_fetch and add items not listed
in DEPENDS and RDEPENDS, they are not caught by recrdeptask. We've gone
around in circles on this issue (e.g
http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/runqueue.py?id=5fa6036d49ed7befe6ad50ec95c61a50aec48195
) and in many cases the behaviour of recrdepends is correct but tasks like
fetchall need the other behaviour.
To address this we add a recideptask flag which can be used in conjuction
with the recrdeptask flag to specify which task to to the inspection upon.
This means entries like do_rootfs[depends] which have do_fetch tasks are
caught and run.
I'm not 100% happy with needing another flag but I don't see any rational
way to get the correct behaviour in all cases without it.
[YOCTO #4597]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Mon, 17 Jun 2013 11:11:53 +0000 (12:11 +0100)]
bitbake: read BBSERVER environement for remote server
This change allows bitbake to connect to a remote server
by reading the BBSERVER environment variable and effecting
the config. Basically a shortcut to the command line parameters.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Mon, 17 Jun 2013 11:11:51 +0000 (12:11 +0100)]
knotty, xmlrpc: add observer-only mode
I add an observer only mode for the knotty UI and
the XMLRPC server that will allow the UI to register
a callback with a server in order to receive events.
The observer-UI is able to send read-only commands to the
server, and also is able to register as an event handler.
Read-only commands are the commands that do not change
the state of the server and have been marked as such in
the command module.
The observer can switch to a full client if it calls addClient
at any time, and the server has no other client running.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the double wrapper script environment, we'd not see issues like
this as it would be masked. The problem theoretically affects code like
useradd too as well as anything using a sstate postinstall.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bogdan Marinescu [Fri, 31 May 2013 14:19:14 +0000 (14:19 +0000)]
bitbake: Add event mask flag support
Add a flag to event handlers which lists the events a given handler wishes to
process. By default event handlers recieve all events but this means
we can stop running code in many cases if we know it doesn't want the event.
This is part of the fix for YOCTO #3812, but implements filtering only
for class event handlers; the other part (events filter for UIs) will be
the subject of a different patch.
Signed-off-by: Bogdan Marinescu <bogdan.a.marinescu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 7 Jun 2013 17:13:04 +0000 (18:13 +0100)]
prserv: Adapt autostart to bitbake-worker
With the change to bitbake-worker we need to ensure the workers know
how to contact the PR service, the magic 0 port and singleton is
no longer enough.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 7 Jun 2013 17:11:49 +0000 (18:11 +0100)]
runqueue: Move the bitbake-worker execution to a higher level
The worker was being executed by each execution queue so would get
constructed twice for each build. This is wasteful so move execution
to the main runqueue so we only have to start the worker once.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 7 Jun 2013 17:11:09 +0000 (18:11 +0100)]
runqueue: Split runqueue to use bitbake-worker
This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.
This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.
Known issues:
* Hob is broken with this patch since it writes to the configuration
and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
see if we can streamline it.
These are issues which will be followed up in subsequent patches.
This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 7 Jun 2013 17:10:36 +0000 (18:10 +0100)]
fetch2: Fix AUTOINC handling
AUTOINC was meant to appear once at the start of the version string.
The list of names may not be sorted meaning it could get inserted in
the middle. This patch simplifies the code and ensures it appears at
the start.
Include cache version bump to ensure the cache picks up these changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton [Tue, 4 Jun 2013 08:59:37 +0000 (09:59 +0100)]
lib/bb/cache.py: invalidate cache when file checksum entry no longer exists
Go through the cached list of file checksums and check if any of the
files no longer exist; if any are missing then invalidate the cached
recipe, which will force it to be reparsed and thus force the list of
files to be collected again. This prevents a warning when moving a file
to a different location that is still picked up by the recipe, e.g.
moving a file from a "files" subdirectory to one named with the recipe
name (${BPN}).
Fixes [YOCTO #4474].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 31 May 2013 11:06:48 +0000 (12:06 +0100)]
bitbake: Improve remote server handling
Two fixes in bitbake related to running remote servers -
* can now specify correctly the bind port
* the information in print conforms to common server infrastructure
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 31 May 2013 11:06:47 +0000 (12:06 +0100)]
xmlrpc: Allow server to remain memory resident
This patch enables the XMLRPC server to remain
resident in memory after a task is run, and to accept
a new controlling client. To check the server after
task completion, do
lsof bitbake.lock
in the build directory. Kill the server with kill.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Fri, 31 May 2013 11:06:46 +0000 (12:06 +0100)]
bitbake server: create common server infrastructure
In an attempt to minimize code duplication, create
clear interfaces, and maximize code reuse through OOP,
bb.server adds base classes for the BitBakeServer,
BitBakeServerConnection and actual server implementations
instructed in particular server types.
These classes document the minimum interfaces that the
derived classes must implement, and provide boilerplate code.
Changes to None, Process and XMLRPC servers as to use
the common server infrastructure.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 28 May 2013 16:52:02 +0000 (16:52 +0000)]
bitbake: xmlrpc remote server
Added code in XMLRPC server that creates a stub local server
for a client-only connection and is able to connect to
a remote server, and receive events from the remote server.
Added the option to start a client with a remote server in
bitbake.
Original code by Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 28 May 2013 13:55:09 +0000 (13:55 +0000)]
bitbake: xmlrpc transport has identification token
In order to be able to identify different clients over a
stateless XMLRPC connection, we add a custom header named
Bitbake-token, which identifies each client.
We refactor the rest of the code to use the new transport.
Based on a patch by Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN [Tue, 28 May 2013 12:49:55 +0000 (12:49 +0000)]
bitbake: move extra cache collection out of cooker
The collection of the extra caching data should not
be performed by the cooker, but supplied to it.
This patch will also streamline the code for launching servers
without a UI attached.
Based on a patch by Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 28 May 2013 13:28:30 +0000 (14:28 +0100)]
server: Remove none server
The process server backend has been serving well as the default for a long
time now and the UI model is much better thought out that it used to be. With
the move to make bitbake a memory resident process, the none server is now
looking rather pointless and complicates the code needlessly. Lets therefore
now remove it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 24 May 2013 10:28:38 +0000 (11:28 +0100)]
build: Avoid empty promises
The "see xxx for further information" is misleading since it is just
the same information. Clarify just to mention the that this is the location
of the logfile without any empty promise.
[YOCTO #4343]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Originally it seemed like a good idea to keep the parameters around. Having
seen this in real life use, its incorrect, we should pull all the data we need
into the cooker's configuguration and then use this to build the datastore.
Being able to just build the datastore from the parameters seemed like a good
idea but having a dummy cooker configuration object is now looking like
the better option.
This also fixes failures in hob since the parseFiles command can call
into cooker directly now and reset the configuration prefiles and postfiles
at will, rather than the indirect calls before which were breaking the datastore
(e.g. BBPATH wasn't set).
The cleanup this allows in tinfoil illustrates how this change makes more sense.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current execution context management for bitbake is ugly and the
use of a global variable is nasty. Fixing that is hard, however we
can improve things to start to establish an API for accessing
and changing that context.
This patch also adds in an explicit reset of the context when we reparse
the configuration data which starts to improve the lifecycle of the data
in setups like hob.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 23 May 2013 09:48:26 +0000 (10:48 +0100)]
cooker.py: Remove global scope class event handlers
Its confusing when an event handler in for example a bbclass receives
events for a recipe which doesn't use the class. This is due to the
class event handlers having confusing scope. Worse, the issue is not
deterministic and the events a handler will see depends on the parse
order.
To avoid these issues, remove the class event handler global scope
and only have class handlers recieve events for recipes using the
class.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 23 May 2013 09:47:10 +0000 (10:47 +0100)]
methodpool: Retire it, remove global method scope
Having a global method scope confuses users and with the introduction
of parallel parsing, its not even possible to correctly detect conflicting
functions. Rather than try and fix that, its simpler to retire the global
method scope and restrict functions to those locations they're defined
within. This is more what users actually expect too.
If we remove the global function scope, the need for methodpool is reduced
to the point we may as well retire it. There is some small loss of caching
of parsed functions but timing measurements so the impact to be neglibile
in the overall parsing time.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cristiana Voicu [Wed, 22 May 2013 09:52:34 +0000 (12:52 +0300)]
hob: apply a fade out animation when deselecting from tables
-applied a fade out mechanism for deselctiong on all the tabs of
recipes and packages list page
-set cursor on the package/recipe when it is included(important
when the its place is changing; like when the tree si order by
included tab)
[YOCTO #4342] Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 18 Feb 2013 13:18:31 +0000 (13:18 +0000)]
runqueue.py: Ensure export flag is set for fakeroot environment variables
This means the variables show up in the shell execution "run" files since
its useful to know what the fakeroot environment is and how to set it up
manually.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As the code stands today its hard to know which configuration variables
are used by which parts of the system. Some are used by the UIs, some
by bin/bitbake itself, some by cooker.
This patch changes the configuration to just contain the variables cooker
uses, and changes bin/bitbake to access the variables it needs directly
which hopefully lets us start to untangle this mess.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 20 May 2013 22:00:31 +0000 (23:00 +0100)]
cooker: Split configuration parsing code into cookerdata
In order to have a memory resident bitbake and to allow task execution, we need
to be able to rebuild the base configuration without a cooker. This moves the
code into its own class so it can be built independently.
The interface is less than ideal here but I didn't want to add parsing methods
a subclassed DataSmart, at least until we've experimented further with this code
and are certain that makes sense. At the very least, the methods are ugly and need
cleaning up. Spliting the code out seems to be the right thing to do though and
should unblock various activities on BitBake so I believe this code is a step in
the right direction.
Based on a patch from Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 20 May 2013 21:54:41 +0000 (22:54 +0100)]
cooker: Move commandline parsing back into the UI/cookerdata
Building up a set of actions for the server is tricky since we depend upon the
commandline but fall back to values from the datastore. We should be able to build
a datastore without a commandline and vice versa. Ultimately the UI should send
the commands to the server.
This patch amounts to code rearranging, moving the heavy lifting to the UI, though
a helper in the configuration option. This will need further cleanup/tweaking but
this should be the only update needed to the UIs. The code now queries the server
for any missing data should it need to.
This code allows various knowledge of configuration variables to move to the UI side
only, partcularly pkgs_to_build but also all the command specifiers. It should also
be possible to move cmd eventually, I'm just unsure if any callers call the commands
expecting this to default to something sane right now.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 20 May 2013 21:54:30 +0000 (22:54 +0100)]
bitbake: Create cookerdata splitting config from cooker and bin/bitbake
Currently the UI and server configuration is one big incestuous mess. To
start to untangle this we creater cookerdata, a new module which contains
various confiuration modules and the code for building the base datastore.
To start with we add a ConfigParameters() class which contains information
about both the commandline configuration and the original environment.
The CookerConfiguration class is created to contain the cooker.configuration
options. This means we can transfer new paramters to the server over something
like XMLRPC and then build a new configuration from these on the server.
Based on a patch from Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Mon, 20 May 2013 22:00:10 +0000 (23:00 +0100)]
bin/bitbake: Improve exception handling
Due to the internal event processing, this excepting handler usually raises an
Empty error, masking the underlying failure. Ensure the original exception is
raised.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sun, 19 May 2013 10:21:55 +0000 (13:21 +0300)]
fetch2: Shorten long srcrevs
The long srcrevs are mainly used or the workdir construction as well as
the package version. The long entries are hashes generated by the git fetcher
and other scms using a similar revision mechanism.
We need these to change when the package changes however collisions are
unlikely to happen within the domains we care about. The long revisions
have generated negative user feedback due to the use in path and file
names.
This patch therefore truncates the revisions to 10 characters maximum.
This should be safe in the contexts where these revisions are used as
the chances of spatially close collisions is very low (distant
collisions are not a major issue in the way we use these).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sun, 19 May 2013 10:17:58 +0000 (13:17 +0300)]
fetch2/git: Clean up sortable_revision
Now we no longer try and provide increasing values from the fetcher,
we can simplify the function structure for the sortable_revision
pieces and move the AUTOINC handling directly into the function
which needs it, simplifying the code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Sun, 19 May 2013 10:16:54 +0000 (13:16 +0300)]
data: Ensure dependencies of subfunctions are accounted for
Currently we account of the top level function's vardeps but not
those of any subfunction. This would imply we'd have to manually
write the dependencies of all parent functions which would be crazy.
This patch adds the dependencies to fix the issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 10 May 2013 15:09:36 +0000 (15:09 +0000)]
cooker.py: Drop confusing updateCache calls
updateCache() gets called by command.py when needed and needs to be iterated
over. The calls in cooker.py are therefore just plain wrong/confusing now.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 10 May 2013 14:35:39 +0000 (14:35 +0000)]
cooker: Separate out collections handling code into its own class
The Cooker class is too large and needs to be split up into different
functional units. Splitting out the collections code into its own class
seems like a good place to start to try and disentangle things.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
