tests: python: Update renamed checksum field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Initialize the directory file descriptor

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log buffer: Implemnent ring buffer behaviour

I don't know why I didn't implement this straight away as this was
always intended to be a ring buffer.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Correctly set pidfd on initialization

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Allow turning on/off log streaming by the job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Handle message to abort a build job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Handle messages sent to a specific job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Improve readability of the code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add switch to turn on stub mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Add stub mode

This should be used in the build service or other situations where we
only need dependency solving.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Move path setup

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Split initialization into smaller functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Change process title to show status

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

strings: Add function to truncate strings

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Ignore any unsupported metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Remove support for repository metadata version zero

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Ignore any malformed repository metadata from disk

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Remove resetting a pointer

I don't think we need to do this here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Remove the control connection from the HTTP client

This is necessary after the connection was unexpectedly closed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Reset metadata if parsing failed

This is needed in case there were some required fields missing. We then
consider the metadata invalid and won't use it any more.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Store/search for repomd.json in the repository's root

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Force refresh if we have no metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Mark repository metadata as version 1

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cli: Send more stuff to the console instead of syslog

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hasher: Make the static analyzer happy

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hasher: Actually return any errors

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Use Zstandard level 20 for compression

This is the maximum that seems to make sense. There is pretty much no
gain from any higher levels, but time will increase by about 3x.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Move log streaming into the main process

Now, we can get entirely rid of individual control connections for jobs.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Add a common function to send messages into the control socket

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop the control connect and explicitely launch them

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Enable long mode (if available)

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hashes: Replace the old digest mechanism

This is splitting the hasher into a separate object so that we have more
flexibility here. It also renames the digests struct into "hashes" which
is a little bit shorter and I thought that it makes more sense.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

digests: Constify source when importing digests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Remove debugging line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Fix composing import paths

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Move sync decision

There are actually not a lot of changes, I just needed to move a lot of
static functions.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Refactor code to retry

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Perform a full scan for local repositories without metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Download the package database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Store the downloaded metadata if it is more recent

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Parse the entire repository metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Add more helper functions to read JSON objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Read any existing repository metadata before download

This way we can verify that the downloaded version is more recent.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Add a new object to parse the metadata into

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Add helper function to read integers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Load metadata into memory and verify it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

key: Parse signatures that don't have a comment

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Add a simple buffer structure

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

key: Print correct error code when verification fails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Oops, forgot to add the files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Move repomd.json into the root

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Don't make the JSON pretty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Cleanup old metadata before writing new stuff

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Add file type to metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Add filesize to metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Some minor code cleanup when writing metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Only use a few stronger digests for metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Add checksums for files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Add an array for files

This should allow us to add more items whenever we would need to.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Add the database path to the repository metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Move functions into a separate file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Use our own JSON functions to build repository metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Sign the repository metadata instead of the database

I don't think that there is any chance that we will be able to get rid
of the JSON object.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

keys: Give signing functions better names

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: key: Remove unused variables

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Refactor importing repositories

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Make the static analyzer happy when importing repositories

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Import the correct headers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfers: Use correct error code when a transfer fails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Return positive error codes on verify

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Prevent that the event loop terminates too early

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Keep WebSocket connections open

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Implicitly handle relaunch

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Launch transfers as soon as they are enqueued when not in standalone mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Shorten some function names

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Remove circular dependency to HTTP client

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Refactor the queue management

It does not seem like a very good idea to give cURL all the handles at
the same time and let it sort it all out. Therefore, this is yet another
attempt to implement some half-decent queueing to implement parallel
downloads.

Fixes: #13812 - httpclient: Bring back a queue for transfers
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Don't verify digest when we did not get a positive response

When servers were sending a 404 error page, the transfer failed because
of a digest mismatch which is not what we want.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Ensure we always cleanup on failure

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Remove unused clone function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

snapshot: Clone pakfire with custom flags

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

snapshot: Create a new snapshot if the cache dir does not exist

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Configure some more sensible defaults

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Anchor the key to the appdata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Tidy up mirror handling

No functional changes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: For queued xfers, make sure we don't use an older mirror

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrorlist: Don't free the JSON object when checking it's type

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Disable instead of "broken"

This patch also implements this all with flags.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Evaluate the status code

With some errors, there is no point at all to keep trying.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Refactor mirror handling

xfers will now immediately get a mirror assigned when they have a
mirrorlist set so that we don't have to play any games later.

When an xfer has failed, we will ask for another mirror and run again.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirror: Ensure paths always have a trailing slash

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

contrib: Enable mirror list for stable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Achor the mirrorlist to the appdata

This is now possible because we won't circle-reference Pakfire any more
since we have the ctx object.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Instead of passing around flags, automatically free the repo

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Add reference counting to appdata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Include flags when cloning an instance

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Clean up after failing to create a temporary directory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Free the cookie if level parsing fails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Include local repo name in path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Use function to check repo name

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>