Richard Purdie [Fri, 14 Dec 2018 17:49:16 +0000 (17:49 +0000)]
bitbake: server/process: Ensure socket has a timeout set
We're seeing hangs in oe-selftest where server startup and shutdown are
racing. The assumption was a connect would timeout however no timeout is
set which can leave processes hanging. Set a short timeout for
the connection to avoid this.
Ross Burton [Fri, 14 Dec 2018 17:08:50 +0000 (17:08 +0000)]
libpsl: add
A Public Suffix List is a collection of Top Level Domains (TLDs) suffixes. TLDs
include Global Top Level Domains (gTLDs) like .com and .net; Country Top Level
Domains (ccTLDs) like .de and .cn; and Brand Top Level Domains like .apple and
.google. Brand TLDs allows users to register their own top level domain that
exist at the same level as ICANN's gTLDs. Brand TLDs are sometimes referred to
as Vanity Domains.
Richard Purdie [Fri, 14 Dec 2018 18:02:17 +0000 (18:02 +0000)]
oeqa/selftest/distrodata: Port to use the new recipeutils.get_recipe_upgrade_status() function
Rather than use the obsolete do_checkpkg function, use the new recipeutils
function which uses tinfoil to get the data rather than needing csv file
manipulation.
Richard Purdie [Fri, 14 Dec 2018 17:45:27 +0000 (17:45 +0000)]
lib/oe/recipeutils: Add a new function to mimic do_checkpkg
The code in distrodata.bbclass related to the do_checkpkg task is rather
dated, has holes in it (ignoring some recipes) and has horrible locking
and csv related issues.
We should use modern APIs such as tinfoil to make the calls we need directly
against bitbake, cutting out the middleman and clarifing the code.
This change imports the bits of distrodata.bbclass that are needed by the
automated upgrade helper (AUH) into a standalone function which uses the
tinfoil API. This can then be used by AUH and by the tests in
oeqa/selftest/distrodata as well as by any other standalone script that needs
this functionality. Its likely it can be further improved from here but this is a
good start and appears to function as before, with slightly wider recipe
coverage as some things skipped by distrodata are not skipped here (images,
pieces of gcc, nativesdk only recipes).
Richard Purdie [Fri, 14 Dec 2018 11:02:59 +0000 (11:02 +0000)]
bitbake: data_smart: Allow numeric characters in overrides
We're seeing problems due to the way x86-64 is handled (or not handled)
as an override. Relax the containts on overrides from being lowercase
to being lowercase or numeric. This fixes problem where MACHINE=qemux86
would work but MACHINE=qemux86-64 would fail the same tests.
Richard Purdie [Thu, 13 Dec 2018 16:21:54 +0000 (16:21 +0000)]
maintainers: Clarify/add several entries
Images were previously missing but are added, this also corrects the names
used for gcc/go/bintuils/gdb recipes and adds a few other misc missing ones
to ensure we have complete coverage of the recipes in OE-Core.
Richard Purdie [Thu, 13 Dec 2018 14:46:30 +0000 (14:46 +0000)]
go-crosssdk: PN should use SDK_SYS, not TARGET_ARCH
The crosssdk dependencies are handled using the virtual/ namespace so
this name doesn't matter in the general sense. We want to be able to provide
recipe maintainer information through overrides though, so this standardises it
with the behaviour from gcc-crosssdk and ensures the maintainer overrides work.
Kai Kang [Thu, 13 Dec 2018 14:14:10 +0000 (09:14 -0500)]
systemd: fix compile error for x32
Backport patch to fix systemd compile error for x32:
| ../git/src/timesync/timesyncd-manager.c:607:19: error: format '%lli'
| expects argument of type 'long long int', but argument 11 has type
| 'long int' [-Werror=format=]
Paul Eggleton [Thu, 13 Dec 2018 03:46:56 +0000 (16:46 +1300)]
oe-selftest: distrodata: change test_maintainers() to use tinfoil
Use tinfoil to enumerate recipes and get the value of RECIPE_MAINTAINER
to make it a bit more reliable in the face of do_checkpkg issues we are
currently seeing on the Yocto Project autobuilder. This also makes it a
little less painful to re-execute test_maintainers() since you don't
have to wait for bitbake -c checkpkg to complete every time.
Note that the new test has been written in such a way that it will still
function if RECIPE_MAINTAINER values are ever moved to the recipes.
Also, the test still currently fails as there are recipes that don't
have an assigned maintainer.
Michael Ho [Thu, 29 Nov 2018 12:21:37 +0000 (14:21 +0200)]
insane.bbclass: add package specific skips to sstate hash
The bbclass currently adds INSANE_SKIP to the sstate hash dependencies
however the package specific skips such as INSANE_SKIP_${PN} are
not added automatically because of how the class references them.
This causes the problem that modifying INSANE_SKIP_${PN} does not
invalidate the sstate cache and can mask build breaking warnings.
Add an anonymous python snippet to explicitly include these additional
relevant skips to the sstate hash.
Michael Ho [Thu, 29 Nov 2018 12:21:35 +0000 (14:21 +0200)]
cmake.bbclass: append includedir to implicit include dirs
This resolves issues with paths being marked as system includes that
differ from /usr/include but are considered implicit by the toolchain.
This enables developers to add directories to system includes
to supress compiler compiler warnings from them.
Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Cc: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Tue, 11 Dec 2018 23:26:36 +0000 (23:26 +0000)]
oeqa/sdk: rewrite cpio test
Don't use the helper class as it gets in the way more than it helps, exercise
the out-of-tree paths, and verify the installed files match the expected
architecture.
Ross Burton [Tue, 11 Dec 2018 23:26:35 +0000 (23:26 +0000)]
oeqa/sdk: rewrite lzip test
Don't use the helper class as it gets in the way more than it helps, exercise
the out-of-tree paths, and verify the installed files match the expected
architecture.
Ross Burton [Tue, 11 Dec 2018 23:26:33 +0000 (23:26 +0000)]
oeqa/sdk: show output if run() fails
Use oeqa.utils.subprocesstweak to monkey-patch the subprocess exception so that
any output is shown, and remove any explicit try/catch handling that would have
hidden this.
He Zhe [Wed, 21 Nov 2018 14:06:05 +0000 (22:06 +0800)]
linux-libc-headers: Fix build failure by using fixed temporary file instead of pipe
This is a workaround for the following possible build failure.
*** Compiler lacks asm-goto support.. Stop.
When building linux-libc-headers we need to use binutils on build machine.
binutils v2.31 introduces a bug that could cause scripts/gcc-goto.sh to fail
when running in an environment where /tmp is rarely used, e.g. in docker.
Khem Raj [Mon, 10 Dec 2018 23:52:06 +0000 (15:52 -0800)]
insane.bbclass: BPF objects may mismatch in endianness and bitness too
This ensures that bitness and endianness is ignored for BPF objects
Fixes QA issues like
Bit size did not match (32 to 64) kernel-selftest on
/work/qemumips-yoe-linux/kernel-selftest/1.0-r0/packages-split/kernel-selftest/usr/kernel-selftest/bpf/test_btf_nokv.o
Tim Orling [Sun, 9 Dec 2018 21:43:22 +0000 (13:43 -0800)]
liberror-perl: upgrade 0.17026 -> 0.17027
* add RDEPENDS
* add RDEPENDS for ptest
- tested on qemux86 with glibc and musl
* drop do_compile, not obvious why LIBC needed to be set
* Added HOMEPAGE:
https://bitbucket.org/shlomif/perl-error.pm
- Repository used to be github.com/shlomif (deadlink)
Upstream release notes:
"""
0.17027 2018-10-28
- Documentation and examples enhancements.
- https://bitbucket.org/shlomif/perl-error.pm/pull-requests/1/october-prc-some-minor-mostly-doc-changes/diff
- Thanks to https://metacpan.org/author/JMERELO .
"""
create_manifest2: Dont match filenames which contain the directory name for new manifest
When creating a new python2 manifest, there is a corner case on which
the filepath for a certain dependency that was found, could contain
the path of an existing folder, e.g. ${libdir}/python2.7/xmlrpclib.py
module path contains ${libdir}/python2.7/xml, this causes an issue where
the dependency doesnt get eventually added on FILES for that module.
This patch checks if the dependency that was found is a directory, if it
is, it checks if it matches one of the existing directories on the
manifest, if it is not, then it checks if the dependency's path (without
the filename) matches one of the directories.
create_manifest3: Dont match filenames which contain the directory name for new manifest
When creating a new python3 manifest, there is a corner case on which
the filepath for a certain dependency that was found, could contain
the path of an existing folder, e.g. ${libdir}/python3/xmlrpclib.py
module path contains ${libdir}/python3/xml, this causes an issue where
the dependency doesnt get eventually added on FILES for that module.
This patch checks if the dependency that was found is a directory, if it
is, it checks if it matches one of the existing directories on the
manifest, if it is not, then it checks if the dependency's path (without
the filename) matches one of the directories.
Robert Yang [Tue, 4 Dec 2018 02:06:03 +0000 (10:06 +0800)]
bitbake: bitbake: pysh: Improve error handling for shell code
The p_error() is used for printing errors when parse shell code, but it can't
the EOF error correctly
- Add the following lines to quilt.inc
do_configure_prepend () {
find ${s} -name "*.in" -exec sed -i -e "1s,^#\!.*@perl@ -w$,#\! @perl@\nuse warnings;," {} \;
if [ hello ]; then
}
- Before the patch:
$ rm -fr cache/ tmp/cache/; bitbake -p
[snip]
WARNING: /path/to/quilt/quilt-native_0.65.bb: Error during finalise of /path/to/quilt/quilt-native_0.65.bb
[snip]
bb.pysh.sherrors.ShellSyntaxError: None
followed by:
We can see that this isn't easy to debug, let p_error() check wheter it is EOF
and print appropriate errors can improve the error message. And don't let
codeparser.py except pyshlex.NeedMore (in fact, it never worked since p_error()
only raise ShellSyntaxError), but make it print the last 5 lines which might be
useful for debuging.
- After the patch
$ rm -fr cache/ tmp/cache/; bitbake -p
[snip]
ERROR: /path/to/quilt/quilt_0.65.bb: Error during parse shell code, the last 5 lines are:
find /path/to/quilt/0.65-r0/quilt-0.65 -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
if [ hello ]; then
autotools_do_configure
sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i /path/to/quilt/0.65-r0/quilt-0.65/Makefile
[snip]
File "/path/to/bb/pysh/pyshyacc.py", line 649, in p_error(p=None):
w('Unexpected EOF')
> raise sherrors.ShellSyntaxError(''.join(msg))
The configuration of the sqlite database can timeout due to locking
under heavy load and should be subject to the same retry logic as the
other statements.
Richard Purdie [Fri, 7 Dec 2018 15:39:15 +0000 (15:39 +0000)]
bitbake: utils: Use utils.remove() within prunedir()
We've already tweaked remove() for speed and not to error if it
races for deletion. Therefore use this for prunedir() which was
starting to show the same bug reports.
Richard Purdie [Sun, 2 Dec 2018 19:18:14 +0000 (19:18 +0000)]
bitbake: server/process: Increase server startup time delay
On loaded production systems we've seen bitbake server take over
40s to start up. Increase the timeout to 90s which tries to avoid
failures in selftests.
The delays come from setting up the inotify watches (31s) so can't
really be avoided.
After 5s delay we now warn the user we're waiting for 90s so the
interactive exeperience shouldn't be much changed and its very
unlikely the user would see that anyway.
Jens Rehsack [Wed, 28 Nov 2018 08:15:30 +0000 (09:15 +0100)]
avahi: avoid depending on skipped package
When built without D-Bus, libavahi-client is not build:
Building libavahi-client: no (You need avahi-daemon and D-Bus!)
which causes avahi-dev RDEPENDS failing when creating an image
containing development-tools:
* - nothing provides libavahi-client = 0.7-r0 needed by avahi-dev-0.7-r0.cortexa8hf-neon
And there would be no signature info when rebuild from sstate:
$ bitbake u-boot linux-yocto -cclean
$ bitbake u-boot linux-yocto
$ grep signature tmp/deploy/images/beaglebone-yocto/*.dtb
No result
This s because kernel directly edit ${DEPLOY_DIR_IMAGE}/u-boot.dtb, (Note, it
is global ${DEPLOY_DIR_IMAGE}, not recipe's DEPLOYDIR), so that the modified
info is not in sstate, and would be lost when rebuild from sstate.
There are other problems in previouse code:
- The u-boot.dtb is provided by u-boot, but edited by kernel during signing, so
it should be deployed by kernel rather than u-boot.
- The u-boot.do_concat_dtb directly install files to global ${DEPLOY_DIR_IMAGE},
this is incorrect, the ${DEPLOY_DIR_IMAGE} should be installed by do_deploy.
- It seems that it assumes do_deploy depends on do_install according the comments,
but they have no relationships:
# do_concat_dtb is scheduled _before_ do_install as it overwrite the
# u-boot.bin in both DEPLOYDIR and DEPLOY_IMAGE_DIR.
- The do_concat_dtb should be run after do_compile, but it doesn't have this
dependency.
Make u-boot install u-boot.dtb to ${datadir}, kernel copies u-boot.dtb from
${STAGING_DATADIR} to ${B} and deploy it can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hongxu Jia [Thu, 6 Dec 2018 08:56:15 +0000 (16:56 +0800)]
glibc: improve reproducibility with multilib
Multilib builds specify several loaders which will end up embedded in
some binaries or script files. To support reproducible builds, we must
ensure the loaders are always in deterministic order.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Improve the fix and rebase it to 2.28
Here is the log of lib32-glibc
[log.do_compile]
|Adjust ldd script
|ldd "/lib64/ld-linux-x86-64.so.2 /lib/ld-linux.so.2 /lib/ld-linux.so.2
/lib64/ld-linux-x86-64.so.2" -> "/lib/ld-linux.so.2 /lib64/ld-linux-x86-64.so.2"
[log.do_compile]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Federico Sauter [Wed, 5 Dec 2018 14:22:32 +0000 (14:22 +0000)]
kernel: don't assign the build user/host
The KBUILD_BUILD_USER and KBUILD_BUILD_HOST variables were
assigned at the kernel class level, which made it impossible
to override them in the local configuration.
By setting only the default values of those variables in the
kernel class, it is now possible to override them as expected.
Richard Purdie [Thu, 29 Nov 2018 10:28:34 +0000 (10:28 +0000)]
oeqa/selftest/runcmd: Increase timeout delta
Expecting 1s accuracy on a 2s timeout on a heavily loaded system has proven to be
unreliable. Update this to a 5s timeout with a 3s delta which should be achievable.
INFO - ======================================================================
INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
clss.setUpClassMethod()
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass
cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"])
File "/usr/lib/python3.5/tempfile.py", line 929, in __init__
self.name = mkdtemp(suffix, prefix, dir)
File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
_os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc'
Joshua Watt [Tue, 4 Dec 2018 03:42:36 +0000 (21:42 -0600)]
bitbake: bitbake-worker: Pass taskhash as runtask parameter
Pass the task hash as a parameter to the 'runtask' message instead of
passing the entire dictionary of hashes when the worker is setup. This
is possible less efficient, but prevents the worker taskhashes from
being out of sync with the runqueue in the event that the taskhashes in
the runqueue change.
Turns off the shared cache. It isn't a significant factor in performance
(now that WAL is enabled), and is a really bad idea to have enabled in
processes that fork() (as bitbake it prone to do).
Joshua Watt [Tue, 4 Dec 2018 03:42:32 +0000 (21:42 -0600)]
bitbake: persist_data: Enable Write Ahead Log
Enabling the write ahead log improves database reliability, speeds up
writes (since they mostly happen sequentially), and speeds up readers
(since they are no longer blocked by most write operations). The
persistent database is very read heavy, so the auto-checkpoint size is
reduced from the default (usually 1000) to 100 so that reads remain
fast.
The original implementation of persistent data executed all SQL
statements via sqlite3.Connection.execute(). Behind the scenes, this
function created a sqlite3 Cursor object, executed the statement, then
returned the cursor. However, the implementation did not account for
this and failed to close the cursor object when it was done. The cursor
would eventually be closed when the garbage collector got around to
destroying it. However, sqlite has a limit on the number of cursors that
can exist at any given time, and once this limit is reached it will
block a query to wait for a cursor to be destroyed. Under heavy database
queries, this can result in Python deadlocking with itself, since the
SQL query will block waiting for a free cursor, but Python can no longer
run garbage collection (as it is blocked) to free one.
This restructures the SQLTable class to use two decorators to aid in
performing actions correctly. The first decorator (@retry) wraps a
member function in the retry logic that automatically restarts the
function in the event that the database is locked.
The second decorator (@transaction) wraps the function so that it occurs
in a database transaction, which will automatically COMMIT the changes
on success and ROLLBACK on failure. This function additionally creates
an explicit cursor, passes it to the wrapped function, and cleans it up
when the function is finished.
Note that it is still possible to leak cursors when iterating. This is
much less frequent, but can still be mitigated by wrapping the iteration
in a `with` statement:
with db.iteritems() as it:
for (k, v) in it:
...
As a side effect, since most statements are wrapped in a transaction,
setting the isolation_level when the connection is created is no longer
necessary.
Richard Purdie [Thu, 6 Dec 2018 23:25:49 +0000 (23:25 +0000)]
oeqa/selftest/esdk: Fix typo causing test failure
2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last):
File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
clss.setUpClassMethod()
File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass
cls.tmpdir_eSDKQA = cls.tempdirobj.name
AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj'
Kai Kang [Wed, 5 Dec 2018 13:00:56 +0000 (08:00 -0500)]
packages: respect PACKAGE_NO_GCONV
PACKAGE_NO_GCONV is set in libc-package.bbclass if not all of
'libc-charsets libc-locale-code libc-locales' included in
DISTRO_FEATURES. And then no packages glibc-gconv-* glibc-charmap-* and
glibc-localedata-* is created. Update recipes and conf file which depend
on these packages to check required distro features.
Kai Kang [Wed, 5 Dec 2018 13:00:55 +0000 (08:00 -0500)]
image.bbclass: respect PACKAGE_NO_GCONV
It installs locale-base-* packages according to var IMAGE_LINGUAS.
Packages locale-base-* are split in libc-package.bbclass if variable
PACKAGE_NO_GCONV is not set. When none of ditro features libc-charsets
libc-locales and libc-locale-code is set, PACKAGE_NO_GCONV is set. Then
no locale-base-* is created and fails to create image.
Kai Kang [Wed, 5 Dec 2018 13:00:54 +0000 (08:00 -0500)]
glibc-locale: fix installed-vs-shipped qa issue
Variable PACKAGE_NO_GCONV is set in libc-package.bbclass when none of
libc-charsets, libc-locales and libc-locale-code set in DISTRO_FEATURES.
Then it causes installed-vs-shipped qa issue of glibc-locale:
| ERROR: glibc-locale-2.28-r0 do_package: QA Issue: glibc-locale:
| Files/directories were installed but not shipped in any package:
| /usr/share/i18n
| /usr/share/i18n/charmaps
| /usr/share/i18n/locales
| /usr/share/i18n/charmaps/CP737.gz
| ...
| /usr/share/i18n/locales/ru_RU
| ...
| /usr/lib64/gconv/gconv-modules
| Please set FILES such that these items are packaged. Alternatively if they
| are unneeded, avoid installing them or delete them within do_install.
| glibc-locale: 843 installed and not shipped files. [installed-vs-shipped]
So check PACKAGE_NO_GCONV during do_install and not copy those files if
PACKAGE_NO_GCONV has been set.
Simplify call of bb.utils.contains() in libc-package.bbclass as well.
Hongxu Jia [Wed, 5 Dec 2018 02:11:48 +0000 (10:11 +0800)]
autoconf-doc: improve reproducibility
autoconf-doc package contains autoconf.info.
This file contains date when this file was created, i.e:
"This manual (31 January 2018) .."
Therefore, two builds done on two different days will show different dates for
otherwise identical files, hence breaking reproducibility.
The date is obtained from mtime of "autoconf.texi", unfortunately we patch this
file and change the mtime as a consequence.
We restore reproducibility by removing the patch modifying "autoconf.texi".
As a consequence certain supplemental information will be not added to the documentation.
In particular: more details on usage, and on workarounds for non-updated projects.
Joshua Watt [Sat, 1 Dec 2018 03:01:42 +0000 (21:01 -0600)]
classes/waf: Fix builds when B != S
Waf requires that the current working directory be ${S} (the location of
the wscript) when building. Most of the time, this was true only because
B defaults to S. However, anything that changed that behavior (notably,
using externalsrc) would break the recipe. Remedy this by explicitly
changing cwd to ${S} when running waf commands. As a happy side effect,
B can be set up for "out of tree" builds to keep the source directory
clean.
Richard Purdie [Sat, 1 Dec 2018 17:40:49 +0000 (17:40 +0000)]
bitbake: process: Flush key output to logs
Small tweak to ensure these items are printed into the log since there
is other logging code which looks for the header and this makes it clearer
the server did start but is slow somewhere in startup.
Robert Yang [Fri, 23 Nov 2018 11:13:44 +0000 (19:13 +0800)]
bitbake: bitbake: BBHandler: Check tab indentation for python code
The previous check was in data.py which only can check code like "python
funcname()" in the dependency chain, but there are 3 kinds of python functions:
- python()
- def py_funcname()
- python funcname()
Add the checking to BBHandler to check and warn for all of them.
The warning looks like:
WARNING: /path/to/recipes-core/busybox/busybox_1.29.2.bb: python should use 4 spaces indentation, but found tabs in busybox.inc, line 75
Ross Burton [Mon, 3 Dec 2018 20:35:14 +0000 (20:35 +0000)]
oeqa: don't litter /tmp with temporary directories
If we need to create a temporary directory in targetbuild or buildproject use
tempfile.TemporaryDirectory so that when the test case is finished, the
directory is deleted.
Also synchronise the logic and don't possibly store the temporary directory in
self.tmpdir as nothing uses that.
projects / thirdparty / openembedded / openembedded-core-contrib.git / log
