daemon: Block main loop more intelligently

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Fork new worker process when we receive a build job

The previous system was that the worker processes would have been
pre-forked and waiting for a job which was randomly allocated.

This model is now changing since we no longer poll, but push build jobs
from the hub to the builders.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Store configuration parameters

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Avoid any crashes when we received no message

The on_message callback can be called when the connection is closed
which made the JSON decoder crash.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Depend on "cpuinfo" Python module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logger: Add newline to the end if none already

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logger: Clear any previously configured handlers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Close queue connection immediately when shutting down

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Add flags to be more verbose

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logger: Use new logger setup in builder, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logger: Create a simple logger and use it in daemon

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Join the job queue and call a function when a job is assigned

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Add websocket capability

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Drop distro.py

This has been replaced with a much simpler implemenation.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Add a simple function to read the distro name

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Drop system.py

This has been replaced by cpuinfo and psutil.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Remove max_running/max_waiting configuration parameters

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Use Python socket module to get hostname

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Drop keepalive process

The old companion process which sent keepalive messages to the hub has
been dropped and the main daemon process is now conducting this task.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Add support for POST requests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Make it async

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Install distro.py

This module has been around all the time but was somehow not installed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Cosmetic changes when connecting to the hub

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Show progress bar on file uploads

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Drop legacy HTTP client

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

client: Make all functions async

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

client: Refactor hub communication based on tornado HTTP client

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Add a simple wrapper for clone3

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Make CLONE_INTO_CGROUP non-optional

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: List all mountpoints after mount

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Don't list everything after each mount operation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Don't mess with the helpers

For some reason, this umounts practically everything on my Debian box.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Fix typo in comment

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Use libmount to iterate over any mountpoints

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

libpakfire: Move mount operations into a new file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Link against libmount

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Use pivot_root instead of chroot

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Link against libcap

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Filter syscalls in container

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Depend on libseccomp

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Drop capabilities

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Launch any containers in a new PID namespace

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Fix compression selection

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

shell: Allow passing extra packages to install

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Don't return void

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Simply empty string check

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

request: Drop support for groups

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Only activate the relevant decompressor

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Try being more robust against non-existant files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Rename files inside the archive (again)

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Do not dump JSON in pretty format

This is not pretty enough to be easily read on the CLI, so we can then
dump it as compact as possible.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

parser: Use special "evr" variable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Automatically erase empty epoch when setting evr

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builder: Drop legacy Python build stuff

This has now all moved into the C library

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Ignore if epoll_wait() is being interrupted

This can happen if the process is receiving a signal and we should just
continue as usual.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Make installing the source package an essential job

This should help that the solver does not suggest to not install the
source package. That does not seem to be the best advice here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Parse filelist

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: makefile: Dump all package information

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: makefile: Check filelist length

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: makefile: Add test for dist

This test tries to create a source package for a dummy package. We then
evaluate if we can read correct information back again.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "build: Allow build environments to be interactive"

This reverts commit 91b5b0472158835cf53b489134d4d2e53e717598.

This is quite problematic because this flag is being used in multiple
ways. One version is that we take user input, the other one is that we
show progress bars on the terminal.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Add distribution information

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Add vendor to package metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Add source information

This patch adds support for storing the source package name, evr and
architecture in the metadata.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Explicitely pass struct pakfire on create

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Compile with correct CFLAGS

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "macros: Build for more modern processors only"

This reverts commit 547777ac363bc14975cd7ab77fed3fb160b904f4.

The version of GCC that we currently have does not seem to support this
feature, yet.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Drop installing self-provides

I have no idea what those do, and it seems to work without it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Import any data into the pool

We cannot keep it separate because solving won't work properly any more.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Automatically set build host/time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Read package information from JSON file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Give better name to metadata file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Write package metadata in JSON format

This is starting a new package format which should be easier to write
and parse than our own format. It should also be quite extensible.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Fix storing strings in repository

We tried to store all strings directly in the pool (I think) which is
not what libsolv liked. Any SOLV files written could not be re-read
which is fixed by this change which gives libsolv the chance to store
its strings wherever it wants.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Drop all repo information from memory before read

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dependencies: Fix parsing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Allow build environments to be interactive

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

progressbar: Fix output

This was broken in 23b9395887ec464084ca57c23fbe8c012f467700.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Fix build after callback changes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Drop redundant interactivity flag

There is already a flag for this which can be used.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logging: Do not crash when log callback isn't set

This is now possible because we have a new function to modify the
callback.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

callbacks: Rework them to be easier to pass around

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Introduce confirm callback

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Go back on callbacks

Instead of passing a giant struct will all sorts of callbacks, it is
probably more flexible if we would pass them individually.

However, the log callback should be initialized at the very beginning to
allow Pakfire to properly log from the very start.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Add function to reset all callbacks

When we call pakfire from Python, we will pass some callbacks which we
will hold a reference to. If those are being freed but Pakfire isn't we
might call the freed callback.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

downloader: Replace snprintf by our own string function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Check for compatible architecture

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

macros: Build for more modern processors only

This patch enables pakfire to build for processors that have been
released after 2011 only.

It allows us to use modern processor features like SSE4.1, SSE4.2,
POPCNT, SSE3, and SSSE3.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Wrap erasing packages into a transaction, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Ignore if the file was already deleted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Store a timestamp when the database was modified last

This is useful to detect whether things have changed since the database
was opened and whether things have to be re-read from it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Implement a simple locking mechanism to avoid concurrent transactions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Fix appending flags to CFLAGS/LDFLAGS

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

testsuite: Fix shadowed variables

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

testsuite: Fix prototype declarations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packagelist: Make helper function static

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

transaction: Make pakfire_transaction_downloadsize static

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logging: Include header file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

keystore: Include header file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Do not attempt to call callbacks when not set

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>