repo: Simply scan again on forced refresh of local repos

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Make the write metadata function public

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Remove the refresh function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Don't refresh the local repository sooner than the others

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Drop Pakfire.refresh()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Refresh repos once all have been created

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Move creating the local repository into Pakfire

Pakfire is kind of the thing that should do the setup. That way, we have
it all in one place instead of two places of setup.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cli: The shell environment is obviously local, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

contrib: Remove the local repo from the configuration

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Always create a local build repository in the home directory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

path: Add function to expand paths

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Create a flag to mark locally run builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Increase the Zstandard compression level

The data is actually really small and compresses well, so we can spend
some extra CPU cycles to save disk space and bandwidth.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Implement passing the compression level to XZ & Zstd

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Fix flushing Zstandard output data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Move compression detection

It is easier to run this when we read the database instead of in the
lower level SOLV reading function because inheriting the FILE handle
becomes difficult.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Add function to destroy a package

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: archive: Make the test repository virtual

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Make all internal repositories virtual

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Since we now support virtual repositories, the don't have to be local

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Fix return codes when reading SOLV data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Remove unused function to only add package metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cli: Use import function to import archives

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Optionally return the package object when importing archives

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Introduce the concept of virtual repositories

These repos can import packages, but they won't copy them but rather
leave them where they were before.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Revert changing pakfire_path_relpath

Use pakfire_path_relative instead which writes the path to the stack.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Remove fetching unused path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Don't lose build status when closing the log file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Make sure relative paths are always relative

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Unify code to import packages

This now supports the flat layout that we use for the @local build
repository as well as the UUID layout that we use to publish packages.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Automatically detect gzip-compressed data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Refactor finding the installed repository

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Refactor identity function

We could in theory have two repositories with the same name.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Drop an unused function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Refactor opening existing repositories

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Refactor creating a new repo

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Fix weird argument handling

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Remove the braindead UUID parsing/unparsing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Pass all log messages on to the upstream logger

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log file: Change compression to gzip

Although it would be nice to only use one compression algorithm, the
challenges to use Zstandard are a little bit too large. The compression
is actually not better (if even) and takes up a lot more resources. At
the same time, there is very little support for Zstandard in browsers
and Python itself, which results us writing a lot of extra code to
ensure that the files are often decoded again to be delivered in
plaintext and being DEFLATEd again by the reverse proxies.

So it all makes a lot more sense to use gzip from the start and just be
done with it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Add transparent gzip compression

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Depend on zlib

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Don't use chunked encoding for uploads

This is deprecated in HTTP/2 and on the server side we now live without
it. cURL even sends a Content-Length header which is quite helpful.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

string: Don't return the number of elements in an array after append

We don't use this anywhere and it requires to reset r whenever we store
the value in it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Drastically increase the Zstandard compression level

All files that we generate are being transferred or stored for a long
time. So it should be wise to always spend the extra CPU cycles.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log file: Add a human readable filename argument

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Create a log file and write the entire job output to it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log file: Reset the file handle immediately

Since we are calling some logging functions from here we might want to
write to the closed file handle.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buildservice: Always set title on upload

This will always be set, so there is no point in adding this check.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buildservice: Send digest without choosing some algorithm

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buildservice: Reformat how to play with arguments

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Make this more versatile

This is needed because we have differnt types of input and therefore
need to be able to handle them all.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Log something when the result callback returned non-zero

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Properly implement the log stream now

This is now a post event which will always be called when there have
been any other events processed. We will therefore try to send as many
log lines as possible before we terminate.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: compress: Tidy up the code to get better assertion messages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buildservice: Make setting the upload filename optional

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Correctly terminate on EOF

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Refactor transparent Zstandard compression

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log file: Add new object to write a (compressed) log file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buildservice: Use the URL to upload the payload

This is just a change in the protocol which hopefully will give us some
more flexibility in the future.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Remove old target implementation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add some basic tests for the log stream

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Add a function to print custom content into the stream

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Don't loop forever if we did not find a line break

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Only call the read function when there is new data to read

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Add function to close the stream

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Send an API call once the job has finished

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Upload packages after the build is done

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buildservice: Simplify how it is being initialized

This make this all more re-usable in other places.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Add a result callback and use it in the CLI builder

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Add a function that imports an archive

I know that we have been doing this in so many places, but hopefully
this is now the new (soon to be unified) way to do this.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Drop old code that copies the packages after build

This is broken and needs to be replaced by something more custom.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Make a function available to all that generates the filename

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Pass the package as well as the archive

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Add a convenient way to iterate over all local archives

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Remove remains of some logging callback

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Fix group parsing for legacy package formats

In older package formats, we stored the groups as a string. The current
test package is in this format and the added code ensures that we can
parse the group.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Fix running Python tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Use the branch prediction macros in potentially hot code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

strings: Use the branch prediction macros in potentially hot code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Introduce macros to tell the compiler how likely branches are

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

arch: Use our own string comparison function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

string: Make setting a string faster

This avoids calling a lot of libc functions that we actually don't need.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Ensure the ccache directory exists or create it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Always append a / to the base URL

When cURL merges URLs it will cut off the last part if the URL is not
terminated with /.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

string: Add a function to append something to a string

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Don't crash if xfer finishes too early

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

deps: Remove unused variable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Return an error if we could not subscribe to signals

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cgroup: Initialize return value in case there are no children

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

ELF: Initilize r just in case the binary has no program headers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Initialize return value on scan just in case we find no files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Refactor fetching dependencies and re-use existing code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Minor code cleanups in Solution class

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Identify as "pakfire" and not "_pakfire"

The module was renamed some time ago.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Use the string array function for arches

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Drop file handle wrapper

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Use strings functions to free strings in Package

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Minor code cleanups in the Package class

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Fix config.h

PACKAGE_VERSION was accidentially redefined which rendered all compiler
checks broken which then wiped out all custom CFLAGS that we have been
using.

This patch adds a new variable with the full version which fixes the
problem.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>