scripts: python3: decode subprocess output

stdeout and stderr content returned by subprocess API has different
types in Python 3(bytes) and Python 2(string). Decoding it to
'utf-8' makes it unicode on both pythons.

(From OE-Core rev: ae4d36375683b6cfd48af25bfca70b2e77f7ade5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

combo-layer: python3: use tempfile.TemporaryFile

Used tempfile.TemporaryFile() API instead of deprecated
os.tmpfile().

(From OE-Core rev: bf1b411eb1cd2cc960325d5fdb0cb4f4f7b1e40e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

dirsize: python3: fix TypeError: unorderable types

Python 3 ignores the __cmp__() method and doesn't have cmp() builtin
function. This caused sorted() call to raise
    TypeError: unorderable types: Record() < Record()

Removing __cmp__ method and implementing __lt__ should solve the
problem as __lt__ is the only method needed for sort[ed] to work.

(From OE-Core rev: 391cd33720e7d7e8e261193199272739293ad881)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

combo-layer: python3: import reduce

Reduce is not a builtin function in python3.
It has to be imported from functools.

(From OE-Core rev: f9bda8b36decb5a837adcfeb1173092401988801)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: python3: use new metaclass syntax

Used metaclass=<metaclass> syntax instead old
__metaclass__ = <metaclass> as only new one is supported
in python 3.

(From OE-Core rev: 067a2156f36a12c731d49f88cf9d0b0bab7a0dcb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: python3: get rid of __future__ imports

Removed print_function and with_statement imports from __future__
as they're supported by python 3 by default.

(From OE-Core rev: 77ae2daad5d775d710b953cf0c623ce74cb2c274)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: python3: rename raw_input to input

Renamed raw_input to input as raw_input does not
exist in python 3.

(From OE-Core rev: 32765150b860ecdea74b4494b9531f5bc40252bb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

ksize.py: python3: get rid of strings.join

Used join method instead of strings.join as stings.join
doesn't exist in python 3.

(From OE-Core rev: 10529d8fbc7254523f9749f4b35b07ebcccb6205)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: python3: fix urllib imports

Some functions and classes have been moved from urllib[2]
to urllib.request and urllib.error in python 3.

Used new imports to make the code working in python 3.

(From OE-Core rev: ec3f1759e8b491a44a1fc1ecb6f89919dd30da97)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: python3: use new style except statement

Changed old syle except statements 'except <exception>, var'
to new style 'except <exception> as var' as old style is not
supported in python3.

(From OE-Core rev: 438eabc248f272e3d272aecaa4c9cec177b172d5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: python3: convert iterables to lists

Converted return value of items() keys() and values() to
lists when dictionary is modified in the loop and when
the result is added to the list.

(From OE-Core rev: 874a269eb1d70060c2f3b3f8b70800e2aea789f4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

automake: drop the Python scriptlets to determine Python's directories

python.m4 has python scriptlets that determine where the system's library
directory is. For Python 2.x distutils.sysconfig is used (which works because
we patch it), for Python 3.x plain sysconfig is used (which doesn't work
for example on x32). There is also a hardcoded fallback if the scriptlets fail.

This commit removes the scriptlets altogether and just uses the hardcoded path always,
due to above mentioned issues with x32.

(From OE-Core rev: 0c856653ce65403791e1e5c78229820ff794bae3)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python3-numpy: add a recipe

This is needed for updating piglit to use Python 3.
Python 2 based recipe can be later moved to meta-oe.

(From OE-Core rev: d08f523b71d3463f0ed604be81ae78a7a79390f3)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

hwlatdetect: move to Python 3

(From OE-Core rev: 4aa0d34155834e2e1ef7972c5dd6490a6f0c49e8)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

libcap-ng: move to Python 3

(From OE-Core rev: 671a8e58aba0a01ad8571995dadbf569706df954)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

systemtap: move to Python 3

(From OE-Core rev: fa707f700c0d2317cc5c0289e3e1fc1eb859ef40)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

lttng-ust: move to Python 3

(From OE-Core rev: f49beaf99171e081a0e5df50d38a1122045ca029)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

lttng-tools: move to Python 3

(From OE-Core rev: e0cd0bd7da24b7e68e57feb448926ea9b0007590)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

gnome-doc-utils: remove recipe

Nothing in oe-core actually requires it.

(From OE-Core rev: 8ba5fcb5232bd79d67657218b9d1b3a10bf56b08)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

libnewt-python: move to Python 3

(From OE-Core rev: 70e39d11bffd3fa2bb143ea640e470c9da6748ba)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

libuser: move to Python 3

(From OE-Core rev: b2037a6d1735755ff44913d89c674303191954e0)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python-docutils: move to Python 3

(From OE-Core rev: 798db38f993cf9bad6feb2da6e0eea0a582ba9f3)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python-imaging: remove unused recipe

(From OE-Core rev: b03ebfaee278eb98b4bd0e11a682ac0cda192086)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python-pyrex: remove unused recipe

(From OE-Core rev: 976097f72af02ae4009ab81158927fca54ea78ff)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

swig: move to Python 3

(From OE-Core rev: aa77072c9587ca89ffd64162b43ca0f61f0c2d52)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

gdb: move to Python 3

(From OE-Core rev: 675d11c73dc1f420c471af01a520f6a20d8a7337)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bootchart2: move to Python 3

(From OE-Core rev: c4829e96272376c61d8645df75c4b946f37b461c)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python-pycairo: move to Python 3

The Python 3 version comes in a separate tarball, and is relicensed
under LGPLv3:
http://cairographics.org/pycairo/

(From OE-Core rev: bfd62071befd0f20597cf9c62c281fb6e815bd2e)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

util-linux: move to Python 3

(From OE-Core rev: 0dacdb780bce78e2c12b1ac9914a1a0194d6ff39)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

systemd: drop python dependency for ptests

Python tests are not actually used, because systemd is configured
using --without-python

(From OE-Core rev: acea8caa0ce4f8fd1a0d33c01c12d4b5a81508d8)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

nfs-utils: switch to Python 3

(From OE-Core rev: 91c3d8859676a8b0615563e1f8855566359f90c7)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

packagegroup-core-full-cmdline: drop python-dbus from the list of services

Any Python scripts should include this dependency explicitly.

(From OE-Core rev: f0caf5765f7a969917639912ffccb1c32e2012d1)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

ofono: drop the custom-made revert to Python 2 from Python 3

(From OE-Core rev: 5645bd904b41fb765a03c4246ce6bcc3cda28f9d)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

connman: do not install Python test scripts

They are not compatible with python 3, and require python-dbus
and python-gobject (which are provided only for Python 3).

(From OE-Core rev: 1e248522d1141e0e48562add75373c9930c2c0be)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bluez5: switch to Python 3

(From OE-Core rev: 51250c3dd2bbbdfcc1d9dfba8ed2111335802fe9)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

neard: do not package python test scripts

They require python-dbus and python-gobject
(which are only provided for Python 3), and have not
been ported to Python 3.

(From OE-Core rev: 6334de28bfd5252343deef326e4edca785254b85)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python-pygobject: port to Python 3

This reverts commit 7f7c9ab29eba0e58916629ca13dc4a494535ce19.

(From OE-Core rev: f55e040f901371427db0f7c68a7139e55d5953d2)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

dbus-test: remove unneeded pygobject dependency

(From OE-Core rev: f0ade81f696c5c4f69db48632e69f2f23aaa3c8f)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

glib: move to Python 3

Drop python-pygobject and python-dbus dependencies, because nothing in
ptests depends on them, and it creates a circular dependency chain.

(From OE-Core rev: 95840a416afd50dc6140367570f045c5128c94f4)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python3: drop 110-enable-zlib.patch

With the patch, the zlib extension wasn't built at all.
Without the patch, it gets build just fine.

(From OE-Core rev: 59870a852cf88ecef094db0a2adec168a8997626)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python-dbus: update to 1.2.4, port to python 3

(From OE-Core rev: 691032c4f0e424c3a097a015a14279ea0a6dd71d)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bind: switch Python dependency to Python 3.x

(From OE-Core rev: a10fd8722fb7c5f2c5a206203d0c7f4237a86466)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

systemd: support systemd-boot as a stand-alone EFI bootloader

The "systemd-boot" is gummiboot now included into systemd project.
The old gummiboot project supported in OE is dead.

Our intention is to get a gummiboot-like EFI bootloader without
much dependency on systemd and its features.

This work is largely derived from the existing bbclass and recipes
of gummiboot and systemd.
(commit tip: ee25d0e3987d7732a2e46e1640693b4cf419a9fc)

Please refer to the history up to the tip for authorship and
credit information for the original works.

To enable the systemd-boot in build, add this line
EFI_PROVIDER = "systemd-boot" in your machine conf file.

(From OE-Core rev: e9add1cd01e498d2aa52528ec52342cae48a387a)

Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

pango_1.40.1.bb: Fix compilation error

On a build host not having libglib-2.0 installed compiling pango
fails with the error message

./gen-all-unicode: error while loading shared libraries: libglib-2.0.so.0: cannot open shared object file: No such file or directory

The executable doesn't have RPATH set to the library installed in
the native sysroot.

The fix sets RPATH.

(From OE-Core rev: 25168b82f82f9b205973014664fe2044ec4d4d6d)

Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

libpam: update dependency list

Depend only on bison-native as configure script is checking
only for bison binary and libpam itself doesn't need target bison.

Add libxml2-native for deterministic build (it detects xmlcatalog
and xmllint)

(From OE-Core rev: d904d24a9cb1f15688b9924981d9c3d4dac7065f)

Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

iw: update to version 4.7

4.3 -> 4.7

Refreshed the patch for new version:
    1) 0001-iw-version.sh-don-t-use-git-describe-for-versioning.patch

(From OE-Core rev: 0c6fc9a01ba58adc221f04cc79ece7efc16c0ee7)

Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

libsdl2: Fix build with wayland 1.10

(From OE-Core rev: 77187701e35e858835ae51b3ac089a385e97657a)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

syslinux.bbclass: Added configurable SYSLINUX_ALLOWOPTIONS variable

The new variable allows for images to be created without an
editable boot line in syslinux.  Default behavior remains unchanged.

(From OE-Core rev: 935578c139a260c18e437419be82d7fd7e8be81a)

Signed-off-by: Michael Davis <michael.davis@essvote.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: bitbake-layers: show-layers: disable parsing

We don't need to parse all recipes just to show the list of layers,
since that comes straight from the configuration, so save a bit of time
by not doing so. (A minor regression that came in with the the
bitbake-layers refactoring).

(Bitbake rev: a609ad7d560260a8d50dfa197cd960f496c5da73)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

poky: Add Ubuntu 16.04 to tested/supported list since I run it myself

(From meta-yocto rev: 462aaa3a4354b88fb234ebb8953080f6cef29334)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

distro_check.py: Don't mix tabs and spaces

(From OE-Core rev: 6c8d1876f5a2d53fa89531e4a5f414d5d5da77be)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: use python3 explicitly

Explicitly used python3 as default python for oe builds
will continue to be python2.

[YOCTO #9584]

(Bitbake rev: fde5c962cb69a11b072d1f238c2371a5137d030d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: fix urllib imports

Some functions have been moved from urllib to urllib.parse
in python 3. Modifying the code to import unquote, urlencode and
unquote_plus from urllib.parse if import from urllib fails should
make it working on both python 2 and python 3.

(Bitbake rev: b91aa29fa20befd9841678a727bb91100363518f)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: don't use mru.count in the template

mru is a list. It has a count method, but it differs from the
one for the queryset. Using mru.count causes 'Latest project builds'
section to silently disappear when toaster runs on python 3.

(Bitbake rev: c3ff90c9f028bff5733ebac7b5e72a4688addd3e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: use // operator instead of /

Division operator works differently in Python 3. It results in
float unlike in Python 2, where it results in int.

Explicitly used "floor division" operator instead of 'division'
operator. This should make the code to result in integer under
both pythons.

(Bitbake rev: 0c38441ed99b49dae8ef9613e320f0760853d6aa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: Revert "buildinfohelper: work around unicode exceptions"

This commit causes buildinfohelper to crash when run on python 3
as python 3 doesn't have unicode builtin function.

This reverts commit 7a309d964a86f3474eaab7df6d438ed797f935c0.

(Bitbake rev: 750ca5c8d5a25fc519b75c56352dec7823c7e240)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: xmlrpc: add parameter use_builtin_types

Added use_builtin_types parameter to XMLRPCProxyServer.__init__
to fix this error:
    ERROR: Could not connect to server 0.0.0.0:37132
    : __init__() got an unexpected keyword argument 'use_builtin_types'

(Bitbake rev: ceb6e5bd33a25c45c2afe1559b9394c466db8a92)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: use knotty when working with bitbake server

Using empty BITBAKE_UI environment variable causes bitbake server
to fail with the error:
    FATAL: Unable to import extension module "" from bb.ui
           Valid extension modules: knotty or toasterui\n'

Used BITBAKE_UI="knotty" when starting and stoping bitbake
server to solve above issue.

(Bitbake rev: b9a9ddfb6f53259be214032fb93812149d0cce19)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: fix migrations

Created two new migrations due to the models changes made
in python3 patchset.

(Bitbake rev: 9a6b211b2e15734379186408995851b50a2e0805)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: moved import bb.server.xmlrpc

Moved import xmlrpc module to the place where it's used
to avoid toaster crashes when importing bitbake code.

NOTE: This patch is made to be able to partly test toaster
with bitbake from master. It can be removed as soon as bb.server.xmlrpc
is ported to python 3.

(Bitbake rev: 0c5d691746fb02807568a8a470969a0b68e21915)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: read timezone files in binary mode

Used 'rb' mode to open files to avoid unicode error when code
runs on python 3:
   UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80
                       in position 44: invalid start byte

[YOCTO #9584]

(Bitbake rev: 1414866b84fe1fd674ea79500cd62eda3aa30b33)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: use re.sub() instead of translate()

translate has different set of parameters in python 3 and
can't be used the way it's used in toaster api module.

Replacing it with re.sub() should make the code work in
both python 2  and python 3.

[YOCTO #9584]

(Bitbake rev: 7cc3a41139abc5b8d1eef4376fb19aba3910ce0d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: replace map with list comprehensions

map returns map object in python 3. Replaced calls of
map to equivalent list comprehensions in the code which
requires lists.

[YOCTO #9584]

(Bitbake rev: d41eb6d6c061cb35ec1ecd899f856876f9077167)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: use items and range instead of old APIs

Used items() and range() APIs instead of iteritems() and
xrange() as latter don't exist in python 3

[YOCTO #9584]

(Bitbake rev: 372dd3abcb201bd9ac2c3189c5505d3578ce0dd0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: use decode('utf-8') for binary data

Decoded binary data to utf-8 to fix the following
error when runnign on python 3:
TypeError: Type str doesn't support the buffer API

[YOCTO #9584]

(Bitbake rev: 752ea00919ef054622a51ce097923309934eff2b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: modified list of dependencies

Removed dependency to argparse and wsgiref as they're in
standard python library.

wsgiref crashes during installation with python 3 as it's not
compatible with python 3.

Added dependency to pytz as it's used in the toaster code.

(Bitbake rev: 46f185afaaef74c566eec6d8fc1edc68a3b96b32)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: fix local imports

Replaced local imports with absolute imports.
Used .module for local imports.

This should make the code to work on python 2 and python 3.

[YOCTO #9584]

(Bitbake rev: 3f1e68c783308dcb51242d0fdeef758e581ccc8c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: fix imports to work for python 3

Some APIs have been moved to other modules in python 3:
 getstatusoutput: moved from commands to subproces
 urlopen: moved from urllib2 to urllib.request
 urlparse: moved from urlparse to urllib.parse

Made the imports work for both python versions by
catching ImportError and importing APIs from different
modules.

[YOCTO #9584]

(Bitbake rev: 1abaa1c6a950b327e6468192dd910549643768bb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: get rid of using reduce

Replaced compicated calls of reduce with more clear code.
As reduce was removed from python 3 this change is mandatory
for the code to work on both pythons.

Here is an example change for illustration purposes:

original code:

  querydict = dict(zip(or_keys, or_values))
  query = reduce(operator.or_, map(lambda x: __get_q_for_val(x, querydict[x]), [k for k in querydict])))

replaced with:

  query = None
  for key, val in zip(or_keys, or_values):
      x = __get_q_for_val(k, val)
      query = query | x if query else x

[YOCTO #9584]

(Bitbake rev: 249d0bc6094ec9f369a02b78d8ed634a239e5ee4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: toaster: use force_text instead of force_bytes

Usage of force_bytes in BuildRequest.__str__ method caused
python 3 to throw "__str__ returned non-string (type bytes)"
error.

Replaced force_bytes with force_text to make the code working
on both python 2 and python 3.

[YOCTO #9584]

(Bitbake rev: 9dd9c1393a84d1110c647e84253af8e0bb6acc45)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: goggle/image-writer: Drop since bitrotting and no longer used

The upgrade to python3 is the final nail in the coffin for image-writer
and the goggle UI. Neither seem used or recieve patches and are based
on old versions of GTK+ so drop them, and the remaining crumbs support
pieces.

(Bitbake rev: ee7df1ca00c76f755057c157c093294efb9078d8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: image-writer/goggle: Disable pygtkcompat problems

Disable the problematic gtk usage for use with pygtkcompat. The following
commit removes these tools/UIs entirely but we may as well leave this
piece in the history in case anyone does want a starting point for reusing
them.

(Bitbake rev: c53c7418d392452450352ca2175667dbdbd92401)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: lib/bb: Set required python 3 version to 3.4.0

get_context was added to mutliprocessing as part of 3.4.0

(Bitbake rev: 710351610e3ca4a1b61abc67564f84907e9b2f1c)

Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: data_smart: Simplify ExpansionError exception

This seemingly convoluted syntax doesn't work in python3. Instead
use the chained exception handling syntax which appears to make more
sense here.

(Bitbake rev: b19a4c5166303b1fa680582adf63e6a5564bfb4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: bitbake: Convert to python 3

Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: bitbake: Drop futures usage since we're python 3

(Bitbake rev: bf25f05ce4db11466e62f134f9a6916f886a93d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: bitbake/pyinotify.py: Upgrade to py3 version

(Bitbake rev: 5ee80d77bc278758e411048ed09551ab65b9e72d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake: bitbake/bs4: Upgrade 4.3.2 -> 4.4.1 (python 3 version)

Upgrade to 4.4.1 which has been run through 2to3 as per the maintainers
recommendation for v3 use.

(Bitbake rev: 2f4b98af93c971a8c466ffaf3c09cca0edb6e3ad)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

oeqa/selftest/recipetool.py: Tweak matching of warning line

We are observing cases (see below) where the 'WARNING:' prefix is not at the
beginning of a line, so instead of expecting it in the beginning, match it
within the string.

    ======================================================================
    FAIL: test_recipetool_appendfile_patch (oeqa.selftest.recipetool.RecipetoolTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
        return func(*args, **kwargs)
      File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/recipetool.py", line 285, in test_recipetool_appendfile_patch
        self.fail('Patch warning not found in output:\n%s' % output)
    AssertionError: Patch warning not found in output:
    Parsing recipes..WARNING: File /etc/selftest-replaceme-patched is added by the patch /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta-selftest/recipes-test/recipetool/files/add-file.patch - you may need to remove or replace this patch in order to replace the file.
    NOTE: Writing append file /tmp/recipetoolqaug0kdb4x/recipes-test/recipetool/selftest-recipetool-appendfile.bbappend
    NOTE: Copying /tmp/recipetoolqagci9tita/testfile to /tmp/recipetoolqaug0kdb4x/recipes-test/recipetool/selftest-recipetool-appendfile/testfile
    done.

(From OE-Core rev: 2289138bdaa17b764821f41a3b3fd0a01cda7440)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

qemurunner: Use surrogateescape decoding

Since the stream can contain invalid binary characters (e.g. from
ppc's bootloader) use surrogateescape decoding to ensure we do process
the character stream, else it can hang/timeout.

(From OE-Core rev: 28a0030430d4cfcaf5dfc3e71bda07cdbfbbf4a7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

devtool.py: Fix parsing of bitbake-layers' output

Current parsing was picking wrong targets, leading to the following problem:

AssertionError: Command 'bitbake  Parsing recipes..done. -e' returned non-zero exit status 1:

(From OE-Core rev: eaf83a58825d91c7445835b27d843da7532c208b)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

buildhistory_analysis: Convert stream data (bytes) to strings

The bytes type variables are threat different as strings variables in python3,
so convert bytes to strings. This was found when using the
script buildhistory-diff, where the interpreter was yielding this error

    TypeError: Type str doesn't support the buffer API

(From OE-Core rev: 3064d36186b47954eb94095217f7bb37e3fce651)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

selftest/devtool: Compare sets instead of arrays on AssertEqual

Sets are safer when comparing internal elements and positions are not
important. This commit avoid errors observed on python3 builds as reported
on the below bugzilla entry.

[YOCTO #9661]

(From OE-Core rev: f6df164d09a4d4cf58977bf6cc0bc4f4bc71183a)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

classes/lib: Complete transition to python3

This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.

(From OE-Core rev: fcd6b38bab8517d83e1ed48eef1bca9a9a190f57)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: use python3 in shebang

As most of oe-test cases work for devtool and recipetool
it makes sense to switch both tools to python 3 by
explicitly referring to python3 in their shebangs.

(From OE-Core rev: dad9617809c60ec5f11d4780b0afa1cffa1efed5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: Fix urlparse imports for python3

Used urllib.parse instead of urlparse to make code
working in python 3.

(From OE-Core rev: 0a064f2216895db0181ee033a785328e704ddc0b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: Rename ConfigParser -> configparser for python3

The ConfigParser API was renamed to configparser in python 3.
Renamed ConfigParser -> configparser in scripts/ to make the
code working in python 3.

(From OE-Core rev: de6e98f272e623ce72e724e66920eecf10cb2d41)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: Fix encoding errors for python3

Moved call of decode('utf-8') as close as possible to
call of subprocess API to avoid calling it in a lot of
other places.

Decoded binary data to utf-8 where appropriate to fix devtool
and recipetool tests in python 3 environment.

(From OE-Core rev: 30d02e2aa2d42fdf76271234b2dc9f37bc46b250)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: Replace basestring -> str for python3

Python 3 doesn't have basestring type as all string
are unicode strings.

(From OE-Core rev: e8cfab060f4ff3c4c16387871354d407910e87aa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts: Fix deprecated dict methods for python3

Replaced iteritems -> items, itervalues -> values,
iterkeys -> keys or 'in'

(From OE-Core rev: 25d4d8274bac696a484f83d7f3ada778cf95f4d0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

qemurunner: convert data when working with socket

Converted str to bytes before sending to socket.
Converted bytes to str after receiving from socket.

This should fix TypeError: 'str' does not support the buffer interface
for qemurunner.run_serial method.

(From OE-Core rev: 210e290c9251839dc74e3aabdcea3655dd707a50)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

wic: encode help text

Encoded help text before sending it to pager.communicate as
it expects binary.

[YOCTO #9412]

(From OE-Core rev: 23c27d9d936efaa17da00525f1d2e2f98c53abc7)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

wic: use python3 in shebang

Switched scripts/wic to use python3 as a default
python interpreter.

(From OE-Core rev: ea6245d2383e2ba905ef9f1ba210e5dadc779ad8)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

sanity: Switch urlparse to urllib.parse

urlparse is replaced with urllib.parse functionality in python3

(From OE-Core rev: ecfcc5dad20943b762a741546732a6c447265251)

Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

classes/lib: Update xrange -> range for python3

xrange() no longer exists in python 3, use range()

(From OE-Core rev: d022b4335100612d6596cc4c4956cb98ed5873cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

classes/lib: Update to explictly create lists where needed

Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.

(From OE-Core rev: caebd862bac7eed725e0f0321bf50793671b5312)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

classes/lib: Update to match python3 iter requirements

python3 standardises its use of iteration operations. Update
the code to match the for python3 requires.

(From OE-Core rev: 2476bdcbef591e951d11d57d53f1315848758571)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

classes/lib: Update to use python3 command pipeline decoding

In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.

(From OE-Core rev: bb4685af1bffe17b3aa92a6d21398f38a44ea874)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

classes/lib: Convert to use python3 octal syntax

The syntax for octal values changed in python3, adapt to it.

(From OE-Core rev: 737a095fcde773a36e0fee1f27b74aaa88062386)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

oe-buildenv-internal: Update to python3

Check that 'python' is a python v2 since that is what we assume everywhere
and upstream python devs recommend. We can need both python2 and python3
available since we don't control the software we might download and run.

Also check that python 3 is >= 3.4.0, our minimum version for bitbake.

(From OE-Core rev: 3dd26cd6b3d731f7698f6fbcd1947969f360cdc4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>