runqueue.py: Enable PSEUDO (fakeroot) before the fork

PSEUDO when used as the fakeroot program is usually preloaded into memory,
but disabled by default.  The trigger for enabling it is a set of environment
variables and a fork() or exec*() operation.

We need to setup the environment, specifically PSEUDO_DISABLED=0, prior to
the fork() so that python tasks can be run under PSEUDO control.

This patch is based on the work in Poky primarily from Richard Purdie.

See the following Poky commits for detailed history on this change:

commit 1f3e313fd5cc5ae8ea838bf8fcdedace3cb72584
Author: Richard Purdie <rpurdie@linux.intel.com>
Date:   Wed Dec 8 00:08:04 2010 +0000

    bitbake Revert bitbake exec() and go back to fork() for performace wins (first draft)

    Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
commit 52aada2eaf08d57d5f715f155f2d878831dbaab0
Author: Richard Purdie <rpurdie@linux.intel.com>
Date:   Tue Dec 21 00:51:24 2010 +0000

    bitbake/runqueue.py: Somehow the python environment mapping is failing so do it manually

    Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
commit 1b08a7eb8b708f4d0fc119cf89deb450fa62fea1
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Mon Feb 28 15:31:20 2011 +0000

    bitbake/cache/runqueue.py: Move workload for recipe parsing to the child process

    Parsing the recipe in the parent before forking off the child worker
    can mean the parent doesn't hit the idle loop and becomes a bottleneck
    when lauching many short lived processes.

    The reason we need this in the parent is to figure out the fakeroot
    environmental options. To address this, add the fakeroot variables
    to the cache and move recipe loadData into the child task.

    For a poky-image-sato build this results in about a 2 minute speedup
    (1.8%).

    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Integrated and merged into bitbake upstream

Signed-off-by: Mark Hatle <Mark.Hatle@windriver.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>

diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index 421bd791833fa0c4f77560f1878dd5e3b3f6d302..2d04ee77c0fafa5ccc62a297290723caba50d210 100644 (file)
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -43,7 +43,7 @@ except ImportError:
     logger.info("Importing cPickle failed. "
                 "Falling back to a very slow implementation.")
 
-__cache_version__ = "137"
+__cache_version__ = "138"
 
 recipe_fields = (
     'pn',
@@ -78,6 +78,8 @@ recipe_fields = (
     'summary',
     'license',
     'section',
+    'fakerootenv',
+    'fakerootdirs',
 )
 
 
@@ -172,6 +174,8 @@ class RecipeInfo(namedtuple('RecipeInfo', recipe_fields)):
             summary          = cls.getvar('SUMMARY', metadata),
             license          = cls.getvar('LICENSE', metadata),
             section          = cls.getvar('SECTION', metadata),
+            fakerootenv      = cls.getvar('FAKEROOTENV', metadata),
+            fakerootdirs     = cls.getvar('FAKEROOTDIRS', metadata),
         )
 
 
@@ -584,6 +588,8 @@ class CacheData(object):
         self.summary = {}
         self.license = {}
         self.section = {}
+        self.fakerootenv = {}
+        self.fakerootdirs = {}
 
         # Indirect Cache variables (set elsewhere)
         self.ignored_dependencies = []
@@ -647,3 +653,5 @@ class CacheData(object):
         self.summary[fn] = info.summary
         self.license[fn] = info.license
         self.section[fn] = info.section
+        self.fakerootenv[fn] = info.fakerootenv
+        self.fakerootdirs[fn] = info.fakerootdirs

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 85a944199d785a9a1b058643ef89f815eccee98b..57df4c0ec9376b91c42bff907c3b92d371f109dd 100644 (file)
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1058,6 +1058,28 @@ class RunQueueExecute:
         return
 
     def fork_off_task(self, fn, task, taskname, quieterrors=False):
+        # We need to setup the environment BEFORE the fork, since
+        # a fork() or exec*() activates PSEUDO...
+
+        # Capture a copy of the environment as a backup if we overwrite anything...
+        envbackup = os.environ.copy()
+        env = {}
+
+        taskdep = self.rqdata.dataCache.task_deps[fn]
+        if 'fakeroot' in taskdep and taskname in taskdep['fakeroot']:
+            envvars = (self.rqdata.dataCache.fakerootenv[fn] or "").split()
+            for var in envvars:
+                comps = var.split("=")
+                env[comps[0]] = comps[1]
+
+            fakedirs = (self.rqdata.dataCache.fakerootdirs[fn] or "").split()
+            for p in fakedirs:
+                bb.utils.mkdirhier(p)
+            logger.debug(2, "Running %s:%s under fakeroot, state dir is %s" % (fn, taskname, fakedirs))
+            # Setup fakeroot/pseudo environment
+            for e in env:
+                os.putenv(e, env[e])
+
         sys.stdout.flush()
         sys.stderr.flush()
         try:
@@ -1099,6 +1121,13 @@ class RunQueueExecute:
                 os._exit(ret)
             except:
                 os._exit(1)
+        else:
+            # Now restore the environment back to the way we found it...
+            for e in env:
+                os.unsetenv(e)
+            for e in envbackup:
+                if e in env:
+                    os.putenv(e, envbackup[e])
 
         return pid, pipein, pipeout