tests: execute: Add test to catch exceptions in logger

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: command: Print PID

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: command: Build a fork bomb

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Check if logger returned the correct output

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: command: All option to exhaust all memory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Generate some random output for tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Replace former dummy commands

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Add test to return environment variables

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Add a simple echo command that prints lines

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Replace exit code tests with the stub command

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Run execute tests in the stub environment

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Build scaffolding for a simple statically linked command

This can then be used to test the execution container.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

snapshot: Show throughput and ETA in progress bar

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Drop any custom timespec functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

progressbar: Make this a lot smoother

Previously, the progress bar was redrawn very often which did not look
great on the terminal. We are now redrawing the bar about 20 times a
second which gives us a smooth experience.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

_pakfire: Adjust flag to enable/disable ccache

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

_pakfire: Adjust flag to enable/disable snapshot

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Remove unneeded stuff from snapshot.c

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

snapshot: Include pakfire.h

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Check if snapshot has something in it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Don't return error in foreach when there are no mountpoints

The loop returned 1 by default when there were no mountpoints to process
which is not what we need here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

snapshots: Protect against invalid inputs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add a test that creates and restores a snapshot

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "mount: Adjust mount flags for unprivileged users"

This reverts commit c92f710524a370d8e910b74d7ba062373d02d7a6.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Adjust mount flags for unprivileged users

Bind-mounts require us to set MS_REC and remounting any mountpoint
requires us to now downgrade on noexec/nodev/nosuid.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Use mount(2) to perform any mount operations

libmount did too much voodoo here which prevented us from running
smoothly for unprivileged users.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Don't use path in error message

path seems to have been freed after the extraction has been started and
therefore we cannot use it any more.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Don't mount /tmp in container

If /tmp is a ramdisk, any temporary files written during the build
process will be lost between stages. That is rather unintuitive and we
might use excess memory.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Copy scripts into /

Formerly they were created in /tmp which could be overlayed by a tmpfs.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Make the static analyzer happy

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Move cache directory into user's home

This is only happening when running as an unprivileged user.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Fix permissions of temporary directories

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move all temporary files directly into /var/tmp

When running Pakfire as an unprivileged user, we cannot create temporary
files in a subdirectory which has been created earlier by a different
user.

Hence we now put everything directly into /var/tmp where everyone should
have write permissions all of the time.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Store errno when running pakfire_rmtree()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Store UID of running user

This patch also moves the root permission check into the safety check
function.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Use sane directory/file permissions throughout

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Only try umounting after we actually mounted something

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Use existing function to check if we are running in /

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Remove unused variable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Move pakfire_bind() into mount.c

There are no functional changes, but this function rather belongs here,
and as a bonus, we get to make pakfire_mount() static and declutter
pakfire.c slightly.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mount: Mount the interpreter every time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Mount all file systems only in namespace

If Pakfire is running as an unprivileged user, we cannot call mount() in
the original namespace. However, it is difficult to spawn a new process
in a new namespace first and then perform loads of actions in there.
Embedded Pakfire would become more difficult.

At the cost of losing the option to create an environment in a
dynamically created ramdisk, we can only mount everything when we enter
the container.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Switch back to chroot()

pivot_root() seems to be very complicated to use and will require us to
have the container run on a different file system. That is however not
possible when Pakfire is running as an un-privileged user.

Since pivot_root() does not seem to offer any advantages over chroot(),
we switch back to chroot() which is easier to use.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Remove mount_tmpfs flag

This is no longer in use

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Pass loglevel on creation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Correctly set return code on initialization errors

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

digests: Renumber them

There is no need to use them as bitfields.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Use correct value for digest types

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

execute: Disable mount propagation before calling pivot_root()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Allow passing None as empty list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Copy file payload into mtree for hashing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Drop unused function to copy archive entry

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

files: Strip leading "./" from filenames

mtree adds ./ to every file that is being written to the archive, but
that is not very useful for us.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Add SHA256 checksums to filelists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

constants: Export PAKFIRE_DIGEST_* to Python

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

files: Add Python functions to access digest/hexdigest

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Store file digests in a separate table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Refactor digests

Files can now hold more than one digest type and the interfaces have
been slightly improved.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Return filelist as list of File objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Create Python wrapper for file objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Export various functions

These seem to have never been used outside of the library

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Expose filelist as property

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Export path as Python property

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

transactions: Add dry-run mode

This is useful when we want to check whether a package can be
installed/erased/etc.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Update downloads URL

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

client: Implement creating builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Add logging during the build job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Add a new class for jobs

This is a proxy which is being used during the build to communicate with
the hub.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Add some basic steps to build a package

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

constants: Drop some old stuff

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Drop rm function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Drop old build function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Drop old worker shutdown handler

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Make workers async

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Drop unused function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Set job ID in process title

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Block main loop more intelligently

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Fork new worker process when we receive a build job

The previous system was that the worker processes would have been
pre-forked and waiting for a job which was randomly allocated.

This model is now changing since we no longer poll, but push build jobs
from the hub to the builders.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Store configuration parameters

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Avoid any crashes when we received no message

The on_message callback can be called when the connection is closed
which made the JSON decoder crash.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Depend on "cpuinfo" Python module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logger: Add newline to the end if none already

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logger: Clear any previously configured handlers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Close queue connection immediately when shutting down

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Add flags to be more verbose

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logger: Use new logger setup in builder, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logger: Create a simple logger and use it in daemon

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Join the job queue and call a function when a job is assigned

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Add websocket capability

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Drop distro.py

This has been replaced with a much simpler implemenation.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Add a simple function to read the distro name

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Drop system.py

This has been replaced by cpuinfo and psutil.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Remove max_running/max_waiting configuration parameters

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Use Python socket module to get hostname

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Drop keepalive process

The old companion process which sent keepalive messages to the hub has
been dropped and the main daemon process is now conducting this task.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Add support for POST requests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Make it async

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Install distro.py

This module has been around all the time but was somehow not installed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Cosmetic changes when connecting to the hub

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hub: Show progress bar on file uploads

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>