Jens Rehsack [Wed, 28 Nov 2018 08:15:30 +0000 (09:15 +0100)]
avahi: avoid depending on skipped package
When built without D-Bus, libavahi-client is not build:
Building libavahi-client: no (You need avahi-daemon and D-Bus!)
which causes avahi-dev RDEPENDS failing when creating an image
containing development-tools:
* - nothing provides libavahi-client = 0.7-r0 needed by avahi-dev-0.7-r0.cortexa8hf-neon
And there would be no signature info when rebuild from sstate:
$ bitbake u-boot linux-yocto -cclean
$ bitbake u-boot linux-yocto
$ grep signature tmp/deploy/images/beaglebone-yocto/*.dtb
No result
This s because kernel directly edit ${DEPLOY_DIR_IMAGE}/u-boot.dtb, (Note, it
is global ${DEPLOY_DIR_IMAGE}, not recipe's DEPLOYDIR), so that the modified
info is not in sstate, and would be lost when rebuild from sstate.
There are other problems in previouse code:
- The u-boot.dtb is provided by u-boot, but edited by kernel during signing, so
it should be deployed by kernel rather than u-boot.
- The u-boot.do_concat_dtb directly install files to global ${DEPLOY_DIR_IMAGE},
this is incorrect, the ${DEPLOY_DIR_IMAGE} should be installed by do_deploy.
- It seems that it assumes do_deploy depends on do_install according the comments,
but they have no relationships:
# do_concat_dtb is scheduled _before_ do_install as it overwrite the
# u-boot.bin in both DEPLOYDIR and DEPLOY_IMAGE_DIR.
- The do_concat_dtb should be run after do_compile, but it doesn't have this
dependency.
Make u-boot install u-boot.dtb to ${datadir}, kernel copies u-boot.dtb from
${STAGING_DATADIR} to ${B} and deploy it can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hongxu Jia [Thu, 6 Dec 2018 08:56:15 +0000 (16:56 +0800)]
glibc: improve reproducibility with multilib
Multilib builds specify several loaders which will end up embedded in
some binaries or script files. To support reproducible builds, we must
ensure the loaders are always in deterministic order.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Improve the fix and rebase it to 2.28
Here is the log of lib32-glibc
[log.do_compile]
|Adjust ldd script
|ldd "/lib64/ld-linux-x86-64.so.2 /lib/ld-linux.so.2 /lib/ld-linux.so.2
/lib64/ld-linux-x86-64.so.2" -> "/lib/ld-linux.so.2 /lib64/ld-linux-x86-64.so.2"
[log.do_compile]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Federico Sauter [Wed, 5 Dec 2018 14:22:32 +0000 (14:22 +0000)]
kernel: don't assign the build user/host
The KBUILD_BUILD_USER and KBUILD_BUILD_HOST variables were
assigned at the kernel class level, which made it impossible
to override them in the local configuration.
By setting only the default values of those variables in the
kernel class, it is now possible to override them as expected.
Richard Purdie [Thu, 29 Nov 2018 10:28:34 +0000 (10:28 +0000)]
oeqa/selftest/runcmd: Increase timeout delta
Expecting 1s accuracy on a 2s timeout on a heavily loaded system has proven to be
unreliable. Update this to a 5s timeout with a 3s delta which should be achievable.
INFO - ======================================================================
INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
clss.setUpClassMethod()
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass
cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"])
File "/usr/lib/python3.5/tempfile.py", line 929, in __init__
self.name = mkdtemp(suffix, prefix, dir)
File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
_os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc'
Joshua Watt [Tue, 4 Dec 2018 03:42:36 +0000 (21:42 -0600)]
bitbake: bitbake-worker: Pass taskhash as runtask parameter
Pass the task hash as a parameter to the 'runtask' message instead of
passing the entire dictionary of hashes when the worker is setup. This
is possible less efficient, but prevents the worker taskhashes from
being out of sync with the runqueue in the event that the taskhashes in
the runqueue change.
Turns off the shared cache. It isn't a significant factor in performance
(now that WAL is enabled), and is a really bad idea to have enabled in
processes that fork() (as bitbake it prone to do).
Joshua Watt [Tue, 4 Dec 2018 03:42:32 +0000 (21:42 -0600)]
bitbake: persist_data: Enable Write Ahead Log
Enabling the write ahead log improves database reliability, speeds up
writes (since they mostly happen sequentially), and speeds up readers
(since they are no longer blocked by most write operations). The
persistent database is very read heavy, so the auto-checkpoint size is
reduced from the default (usually 1000) to 100 so that reads remain
fast.
The original implementation of persistent data executed all SQL
statements via sqlite3.Connection.execute(). Behind the scenes, this
function created a sqlite3 Cursor object, executed the statement, then
returned the cursor. However, the implementation did not account for
this and failed to close the cursor object when it was done. The cursor
would eventually be closed when the garbage collector got around to
destroying it. However, sqlite has a limit on the number of cursors that
can exist at any given time, and once this limit is reached it will
block a query to wait for a cursor to be destroyed. Under heavy database
queries, this can result in Python deadlocking with itself, since the
SQL query will block waiting for a free cursor, but Python can no longer
run garbage collection (as it is blocked) to free one.
This restructures the SQLTable class to use two decorators to aid in
performing actions correctly. The first decorator (@retry) wraps a
member function in the retry logic that automatically restarts the
function in the event that the database is locked.
The second decorator (@transaction) wraps the function so that it occurs
in a database transaction, which will automatically COMMIT the changes
on success and ROLLBACK on failure. This function additionally creates
an explicit cursor, passes it to the wrapped function, and cleans it up
when the function is finished.
Note that it is still possible to leak cursors when iterating. This is
much less frequent, but can still be mitigated by wrapping the iteration
in a `with` statement:
with db.iteritems() as it:
for (k, v) in it:
...
As a side effect, since most statements are wrapped in a transaction,
setting the isolation_level when the connection is created is no longer
necessary.
Richard Purdie [Thu, 6 Dec 2018 23:25:49 +0000 (23:25 +0000)]
oeqa/selftest/esdk: Fix typo causing test failure
2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last):
File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
clss.setUpClassMethod()
File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass
cls.tmpdir_eSDKQA = cls.tempdirobj.name
AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj'
Kai Kang [Wed, 5 Dec 2018 13:00:56 +0000 (08:00 -0500)]
packages: respect PACKAGE_NO_GCONV
PACKAGE_NO_GCONV is set in libc-package.bbclass if not all of
'libc-charsets libc-locale-code libc-locales' included in
DISTRO_FEATURES. And then no packages glibc-gconv-* glibc-charmap-* and
glibc-localedata-* is created. Update recipes and conf file which depend
on these packages to check required distro features.
Kai Kang [Wed, 5 Dec 2018 13:00:55 +0000 (08:00 -0500)]
image.bbclass: respect PACKAGE_NO_GCONV
It installs locale-base-* packages according to var IMAGE_LINGUAS.
Packages locale-base-* are split in libc-package.bbclass if variable
PACKAGE_NO_GCONV is not set. When none of ditro features libc-charsets
libc-locales and libc-locale-code is set, PACKAGE_NO_GCONV is set. Then
no locale-base-* is created and fails to create image.
Kai Kang [Wed, 5 Dec 2018 13:00:54 +0000 (08:00 -0500)]
glibc-locale: fix installed-vs-shipped qa issue
Variable PACKAGE_NO_GCONV is set in libc-package.bbclass when none of
libc-charsets, libc-locales and libc-locale-code set in DISTRO_FEATURES.
Then it causes installed-vs-shipped qa issue of glibc-locale:
| ERROR: glibc-locale-2.28-r0 do_package: QA Issue: glibc-locale:
| Files/directories were installed but not shipped in any package:
| /usr/share/i18n
| /usr/share/i18n/charmaps
| /usr/share/i18n/locales
| /usr/share/i18n/charmaps/CP737.gz
| ...
| /usr/share/i18n/locales/ru_RU
| ...
| /usr/lib64/gconv/gconv-modules
| Please set FILES such that these items are packaged. Alternatively if they
| are unneeded, avoid installing them or delete them within do_install.
| glibc-locale: 843 installed and not shipped files. [installed-vs-shipped]
So check PACKAGE_NO_GCONV during do_install and not copy those files if
PACKAGE_NO_GCONV has been set.
Simplify call of bb.utils.contains() in libc-package.bbclass as well.
Hongxu Jia [Wed, 5 Dec 2018 02:11:48 +0000 (10:11 +0800)]
autoconf-doc: improve reproducibility
autoconf-doc package contains autoconf.info.
This file contains date when this file was created, i.e:
"This manual (31 January 2018) .."
Therefore, two builds done on two different days will show different dates for
otherwise identical files, hence breaking reproducibility.
The date is obtained from mtime of "autoconf.texi", unfortunately we patch this
file and change the mtime as a consequence.
We restore reproducibility by removing the patch modifying "autoconf.texi".
As a consequence certain supplemental information will be not added to the documentation.
In particular: more details on usage, and on workarounds for non-updated projects.
Joshua Watt [Sat, 1 Dec 2018 03:01:42 +0000 (21:01 -0600)]
classes/waf: Fix builds when B != S
Waf requires that the current working directory be ${S} (the location of
the wscript) when building. Most of the time, this was true only because
B defaults to S. However, anything that changed that behavior (notably,
using externalsrc) would break the recipe. Remedy this by explicitly
changing cwd to ${S} when running waf commands. As a happy side effect,
B can be set up for "out of tree" builds to keep the source directory
clean.
Richard Purdie [Sat, 1 Dec 2018 17:40:49 +0000 (17:40 +0000)]
bitbake: process: Flush key output to logs
Small tweak to ensure these items are printed into the log since there
is other logging code which looks for the header and this makes it clearer
the server did start but is slow somewhere in startup.
Robert Yang [Fri, 23 Nov 2018 11:13:44 +0000 (19:13 +0800)]
bitbake: bitbake: BBHandler: Check tab indentation for python code
The previous check was in data.py which only can check code like "python
funcname()" in the dependency chain, but there are 3 kinds of python functions:
- python()
- def py_funcname()
- python funcname()
Add the checking to BBHandler to check and warn for all of them.
The warning looks like:
WARNING: /path/to/recipes-core/busybox/busybox_1.29.2.bb: python should use 4 spaces indentation, but found tabs in busybox.inc, line 75
Ross Burton [Mon, 3 Dec 2018 20:35:14 +0000 (20:35 +0000)]
oeqa: don't litter /tmp with temporary directories
If we need to create a temporary directory in targetbuild or buildproject use
tempfile.TemporaryDirectory so that when the test case is finished, the
directory is deleted.
Also synchronise the logic and don't possibly store the temporary directory in
self.tmpdir as nothing uses that.
Richard Purdie [Sun, 2 Dec 2018 11:23:17 +0000 (11:23 +0000)]
oeqa/utils/qemurunner: Avoid tracebacks on closed files
Reorder the shutdown/teardown to avoid:
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 224, in launch
op = self.getOutput(output)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 90, in getOutput
fl = fcntl.fcntl(o, fcntl.F_GETFL)
ValueError: I/O operation on closed file
assertTrue doesn't give good debug information when things fail. Update
several to use assertIn which gives information upon failure, for the
others print the log information upon failure.
"ERROR: Fatal errors occurred in subprocesses, tracebacks printed above"
message from oe-selftest with no other traceback information. Improve the
traceback logging to try and give a better indication of any errors that is
ocurring.
Ross Burton [Tue, 4 Dec 2018 15:08:48 +0000 (15:08 +0000)]
python3: drop redundant patch
This patch altered the clean target's behaviour to skip the ipkg-install
directory. However this directory isn't created by opkg, opkg-utils, or the
package_ipk class; and we don't invoke the clean target as we perform
out-of-tree builds.
Ross Burton [Tue, 4 Dec 2018 15:08:47 +0000 (15:08 +0000)]
python3: don't cripple target distutils
We stop distutils for *native* Python from rewriting hashbangs when installing
(so installed scripts don't have a hashbang that refers to sysroot paths), but
this isn't needed nor desirable for the *target* Python.
Khem Raj [Tue, 4 Dec 2018 04:51:35 +0000 (20:51 -0800)]
valgrind: Skip vgpreload_memcheck shared object from stripping
This is a special library for memcheck tool, where it needs to have the
symbols intact for the stack traces to work on target, current option is
to install valgrind-dbg ( 151 MB uncompressed ) is quite big for some
systems which may not have space to install it all. Leaving it
unstripped adds about 200KB to image which is much better, this alone
gets memcheck working, as an aside we might need same solution for other
tools e.g. helgrind etc. when needed, they also have leading libraries
installed
Ross Burton [Mon, 3 Dec 2018 20:35:16 +0000 (20:35 +0000)]
insane: Clarify GNU_HASH warning
We have a fatal error if ELF objects don't have GNU_HASH segments but it
doesn't explain what the problem is. At least give a hint to users by
suggesting that LDFLAGS wasn't passed to the compiler.
linux-firmware: better packaging for TI wl12xx & wl18xx firmwares
Currently, the linux-firmware-wl12xx contains all wl12xx and
wl18xx firmwares, except for the wl18xx compatibility symlinks
for old kernels and the linux-firmware-wl18xx contains just
those compatibility symlinks and nothing else. This doesn't
make sense...
Be more specific about what to package into each package, in
particular because the existing wl12xx package is specific
about symlinks already.
At the same time, we split the common bits into a -wlcommon
package, so that the wl18xx package doesn't need to depend
on all the wl12xx firmwares, saving several MiB in the file
system.
Ross Burton [Mon, 3 Dec 2018 11:47:58 +0000 (11:47 +0000)]
oeqa/oelib/path: don't leak temporary directories
setUp() is used to populate a directory of temporary files, and deleted in
__del__. However setUp() is called once *per test* so __del__ would only be
able to remove the last directory created.
Fix the code by using the natural counterpart to setUp, tearDown(), to clean up.
Khem Raj [Sat, 1 Dec 2018 18:12:53 +0000 (10:12 -0800)]
nfs-utils: Upgrade to 2.3.3
enhance the musl-only patch to degrade certain Werros into Warnings
which helps in compiling on musl, since its checking for __GLIBC__
defines which are undefined on musl.
Ross Burton [Thu, 29 Nov 2018 17:39:33 +0000 (17:39 +0000)]
oeqa/sdk/python: add Python 2 and fix detection
Add a Python 2 form to exercise that if present, and fix the setUp() so it
actually looks for a package that exists (nativesdk-python3 is a virtual
package, the interpretter is in nativesdk-python3-core).
Joshua Watt [Thu, 29 Nov 2018 14:57:20 +0000 (08:57 -0600)]
classes/update-alternatives: Skip alternatives when disabled
Skips the update alternative steps for recipes that shouldn't have them
enabled.
Fixes errors like:
nativesdk-bzip2-1.0.6-r5 do_package: bzip2: alternative target
(/opt/poky/2.5+snapshot/sysroots/i686-pokysdk-mingw32/usr/bin/bunzip2
or
/opt/poky/2.5+snapshot/sysroots/i686-pokysdk-mingw32/usr/bin/bunzip2.bzip2)
does not exist, skipping...
Douglas Royds [Wed, 28 Nov 2018 21:13:18 +0000 (10:13 +1300)]
acl: Strip build host tool details from installed ptest file
Adopt the PTEST_BUILD_HOST_FILES mechanism to strip build host paths from the
installed builddefs file. Also strip the record of the build host rpm version.
Douglas Royds [Wed, 28 Nov 2018 04:53:10 +0000 (17:53 +1300)]
ptest: Optionally strip build host tool info from installed files
Several packages (eg. acl, attr, openssl) install Makefiles or other build host
configuration files for ptest so that they can run `make check`, for instance.
These build host files commonly include paths and versions of build host tools,
(eg. rpm or perl) whose presence and version depends on the build host distro.
Optionally strip lines from installed PTEST_BUILD_HOST_FILES using an
extended regex pattern defined in PTEST_BUILD_HOST_PATTERN. This has no
effect on the on-target ptest.
Douglas Royds [Wed, 28 Nov 2018 04:53:09 +0000 (17:53 +1300)]
ptest: Reproducibility: Strip build host paths from any installed Makefile
A common pattern for ptest is to install a Makefile that provides a make check
target. These generated Makefiles are normally full of build host paths, to
hosttools, and to files and scripts within the source or build tree. They also
commonly include the CFLAGS and CPPFLAGS. None of these build host paths
can possibly work on-target, so stripping them has no effect on the success of
the resulting ptests.
Stripping the HOSTTOOLS_DIR has this effect, for instance:
Paul Eggleton [Thu, 29 Nov 2018 23:38:28 +0000 (12:38 +1300)]
oe-selftest: add some tests for recipeutils module
Add some tests for functions in meta/lib/oe/recipeutils.py, in
particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.
The code here wasn't correctly getting the variable history for
varflags, so for example if you did a devtool upgrade on a recipe where
the SRC_URI checksums were in the .inc file (typical for python recipes
in order to support both python 2 and 3) then after the upgrade the
new values would be set in the recipe and the old values were left in
the .inc, which is not right. Teach the code here how to get the history
for varflags so it works properly.
Paul Eggleton [Wed, 28 Nov 2018 04:16:14 +0000 (17:16 +1300)]
lib/oe/recipeutils: patch_recipe(): fix handling of values across includes/classes
If we were setting a variable and part of the variable's value was being
set in a class or a .inc file, we were still just setting the value
outright on the assumption that it was too hard to do otherwise. With
some careful use of the variable history we can do better for certain
situations i.e. when the recipe does not currently set the value
outright.
Additionally, correctly remove _appends for variables we are changing if
we're trying to remove the value added in the _append.
Fixes [YOCTO #12623] and partially fixes [YOCTO #9360].
The value to SRC_URI_append_class-native was not prefixed with a space.
This was not noticed as the SRC_URI before applying the _append contains
trailing spaces. However, if one, e.g., has a .bbappend and adds to the
SRC_URI using SRC_URI += "file://foo.patch", then there no longer is any
trailing space and the _append concatenates the two URIs together,
leading to a build failue.
Khem Raj [Sat, 1 Dec 2018 01:35:04 +0000 (17:35 -0800)]
local.conf.sample: Update the sample config as per new migration manual
Syncs with oe-core and the documentation
https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#var-TESTIMAGE_AUTO
https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#migration-2.6-automatic-testing-changes
Anuj Mittal [Fri, 30 Nov 2018 07:16:09 +0000 (15:16 +0800)]
gstreamer1.0-vaapi: downgrade vaapisink to marginal rank
Using vaapisink (which doesn't supports DRI3 [1] and uses DRI2) with
default poky configuration currently results in an unresponsive display
because DRI2 rendering doesn't work (as of xserver 1.20.3) in non-composited
environments [2].
Downgrade vaapisink to marginal for now so playbin (and in turn gst-play
and gtk-play examples) uses next best sink element and works out of box.
While there is a bit of documentation regarding building a new
manifest file for python, it seems that users usually only read
the manifest file.
The manifest file is in JSON format which doesn't allow comments,
hence why instructions were initially put elsewhere.
This patch hacks the call to open the JSON manifest file by using a
marker to trick it into reading only part of the file as the manifest
itself, and keep the other part as comments, which contain instructions
for the user to run the create_manifest task after an upgrade or
when adding a new package.
While there is a bit of documentation regarding building a new
manifest file for python, it seems that users usually only read
the manifest file.
The manifest file is in JSON format which doesn't allow comments,
hence why instructions were initially put elsewhere.
This patch hacks the call to open the JSON manifest file by using a
marker to trick it into reading only part of the file as the manifest
itself, and keep the other part as comments, which contain instructions
for the user to run the create_manifest task after an upgrade or when
adding a new package.
Ross Burton [Thu, 29 Nov 2018 11:42:15 +0000 (11:42 +0000)]
image_types: use cpio-native to build cpio images
As per the previous commit, upstream cpio has a bug which means it crashes on
append. If the image being built has already had testimage ran then cpio-native
will be in the sysroot. It's also possible that some distributions are shipping
this broken CVE patch too.
Now that our cpio-native is fixed, until we can be sure that the host cpio isn't
broken depend on cpio-native if building a cpio image.
Ross Burton [Thu, 29 Nov 2018 11:42:14 +0000 (11:42 +0000)]
cpio: fix crash when appending to archives
The upstream fix for CVE-2016-2037 introduced a read from uninitialized memory
bug when appending to an existing archive, which is an operation we perform when
building an image.
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
