archive writer: Add convenience function to write JSON files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Move the layer inside an uncompressed outer tarball

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add function to add a larger file from disk

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Split off creating a new file entry

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Split off copying payload

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Drop old compression code

This became a little bit messy with so many options and therefore I
replaced this with the new archive writer.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Use the archive writer to create the layer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Rebase based on the new archive writer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add function to add a file from memory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Store the timestamp when we wrote the archive

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Implement writing files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Implement setting progress title

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add a progress indicator

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Setup the link resolver

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Configure libarchive depending on the format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add scaffolding

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfopen: Move transparent compression/decompression functions into a separate file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Add a simple way how to create an OCI image

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Add a simple function to install packages

This will come handy when generating images.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Don't silently fail if archives should not be hashed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Add the option to make gzipped archives

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Don't rewind on just opened files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Return properly on error

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log buffer: Fix error code when enqueuing a line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Ensure we always send the entire buffer

cURL returns OK even if only parts of the buffer have been sent, so we
need to walk through the loop again to make sure all data is sent.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Go back to the event-driven model to send logs

When the builder is not located in the same data center it can happen
that the send buffer fills up which results in a broken TCP connection
if we continue to send any data.

We will now first of all buffer any messages that have only been
partially sent and listen for when the socket is ready to receive the
rest. If so, we will flush our buffer.

Afterwards we will call the send callback which will go and look for any
more data to send.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Rename the receive buffer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Don't crash when no packages have been uploaded

I remove a check earlier which this code relied on.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Be less verbose when sending socket messages

cURL is already logging this for us.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Be patient when the websocket buffer is full

When the log stream is starting a little while into the build, there
might be a big burst of data sent through the socket which might fill it
up. Then we need to be more patient and send the message again
remembering what has been sent before.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Always require WebSocket support in cURL

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Drop building for Ubuntu 24.04

This version does not have a recent enough version of cURL and
therefore, the build fails. I don't think we need to support anything
old here that does not support WebSockets.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Send timestamp when streaming the log

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log buffer: Store a timestamp with each line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Add function to format timeval to ISO 8601 format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: package: Drop storing a reference to the context

This was an ugly hack to create a new package list which we don't need
any more. This caused the application to occasionally crash because ctx
wasn't set at all times.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Let reverse requires create its own package list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix more error codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Refactor creating a Python list from package list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add a simple test to create a daemon

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Listen to SIGTERM and terminate the jail

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add option to read prerequires from packages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Always turn tmpfs usage on for local builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Require a flag to turn on using tmpfs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Permit to finish jobs without uploads

This is even called when the build has failed in which case we don't
have any packages that we can use here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Untangle reading the filelist and extraction

The idea here was to re-use existing code, but I am not sure this is
currently very helpful to anyone.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Re-enable parsing checksums again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Remove header for dropped pakfire_extract() function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Open an archive in stub mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Add some tests for the main Pakfire class

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Minor code cleanup

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Rename File.digest() to File.checksum()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Return the error code when fetching the filelist

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Update translations

This is mainly to update po/POTFILES.in

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Add some simple tests for pakfire.version_compare()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Remove unused Python exceptions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Test pakfire.supported_arches()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Tidy up writing repository configuration

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Don't return an empty mirrorlist URL, return NULL instead

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Fix exporting keys

Before, the key was randomly written into the file descriptor.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

keys: Make a simple function to export a key as a string

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix setting/reading the context's cache path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Update renamed checksum field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Initialize the directory file descriptor

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log buffer: Implemnent ring buffer behaviour

I don't know why I didn't implement this straight away as this was
always intended to be a ring buffer.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Correctly set pidfd on initialization

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Allow turning on/off log streaming by the job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Handle message to abort a build job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Handle messages sent to a specific job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Improve readability of the code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add switch to turn on stub mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Add stub mode

This should be used in the build service or other situations where we
only need dependency solving.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Move path setup

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Split initialization into smaller functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Change process title to show status

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

strings: Add function to truncate strings

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Ignore any unsupported metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Remove support for repository metadata version zero

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Ignore any malformed repository metadata from disk

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Remove resetting a pointer

I don't think we need to do this here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Remove the control connection from the HTTP client

This is necessary after the connection was unexpectedly closed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Reset metadata if parsing failed

This is needed in case there were some required fields missing. We then
consider the metadata invalid and won't use it any more.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Store/search for repomd.json in the repository's root

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Force refresh if we have no metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Mark repository metadata as version 1

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cli: Send more stuff to the console instead of syslog

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hasher: Make the static analyzer happy

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hasher: Actually return any errors

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Use Zstandard level 20 for compression

This is the maximum that seems to make sense. There is pretty much no
gain from any higher levels, but time will increase by about 3x.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Move log streaming into the main process

Now, we can get entirely rid of individual control connections for jobs.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Add a common function to send messages into the control socket

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop the control connect and explicitely launch them

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Enable long mode (if available)

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hashes: Replace the old digest mechanism

This is splitting the hasher into a separate object so that we have more
flexibility here. It also renames the digests struct into "hashes" which
is a little bit shorter and I thought that it makes more sense.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

digests: Constify source when importing digests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Remove debugging line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Fix composing import paths

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Move sync decision

There are actually not a lot of changes, I just needed to move a lot of
static functions.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Refactor code to retry

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Perform a full scan for local repositories without metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>