daemon: Don't remove the stats timer

We used to free the stats timer whenever the daemon lost the connection
to the build service. Although that should be supported, it could make
the event loop abort because of some unexpected conditions.

This patch simply allocates the timer once and we just arm and disarm it
as needed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Fix adding Python ABI dependencies

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cli: Remove unused variable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Move filtering escape characters to the PTY

This is probably the place where this should live. We are now processing
cursor positioning events inside the same line and we are filtering out
any escape sequences that we don't know.

It is a bit sad to lose the colors, but I don't have to have two places
where we process this. At least this is going to be more robust and we
can be sure to not forward anything malicious to the controlling
terminal.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log file: Try to process any ANSI escape sequences

This is a rather basic implementation and it has the problem that it
erases the entire timestamp on carriage return, too.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Add CLI switch to write the build log to file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfopen: Don't set errno when we don't fail

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Ensure that we will always send the entire message before starting the next one

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfopen: Add more tests for invalid inputs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

string: Fix error handling for strftime functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

strings: Remove an unused variable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Remove superfluous semicolon

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

strings: Refactor formatting time

The previous function was generating wrong outputs, and strtime() should
be used for timestamps instead of deltas.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Don't warn if an address always evaluates

We have a couple of statically allocate strings that will trigger some
checks in the macros which is annoying.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: string: Fail with a clear error code on invalid input

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests/string.c: add test function for time formatting

Signed-off-by: Rico Hoppe <rico.hoppe@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Add string formatting attribute

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Add some missing metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Add function to add formatted strings to an array

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

OCI: Add an entrypoint to the image

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Add a convenience function to append strings to arrays

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Add basic OCI metadata

This patch also revers back to SHA-2 256 since the tools that I have
been using to test this image with usually don't support SHA-2 512.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add string formatting attribute

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

json: Add a convenience function to format strings

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hasher: Add a convenience function to has some data in memory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Never ever pad the last block

Even for gzip, this seems to confuse some implementations.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Make sure we flush all buffers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Use gzip compression for OCI images

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Update the filename for the blob

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Remove creating Docker metadata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Write the OCI layout file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Add some JSON data for the layer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add convenience function to write JSON files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Move the layer inside an uncompressed outer tarball

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add function to add a larger file from disk

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Split off creating a new file entry

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Split off copying payload

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Drop old compression code

This became a little bit messy with so many options and therefore I
replaced this with the new archive writer.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Use the archive writer to create the layer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Rebase based on the new archive writer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add function to add a file from memory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Store the timestamp when we wrote the archive

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Implement writing files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Implement setting progress title

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add a progress indicator

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Setup the link resolver

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Configure libarchive depending on the format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive writer: Add scaffolding

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfopen: Move transparent compression/decompression functions into a separate file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

oci: Add a simple way how to create an OCI image

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Add a simple function to install packages

This will come handy when generating images.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Don't silently fail if archives should not be hashed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Add the option to make gzipped archives

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Don't rewind on just opened files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Return properly on error

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log buffer: Fix error code when enqueuing a line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Ensure we always send the entire buffer

cURL returns OK even if only parts of the buffer have been sent, so we
need to walk through the loop again to make sure all data is sent.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Go back to the event-driven model to send logs

When the builder is not located in the same data center it can happen
that the send buffer fills up which results in a broken TCP connection
if we continue to send any data.

We will now first of all buffer any messages that have only been
partially sent and listen for when the socket is ready to receive the
rest. If so, we will flush our buffer.

Afterwards we will call the send callback which will go and look for any
more data to send.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Rename the receive buffer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Don't crash when no packages have been uploaded

I remove a check earlier which this code relied on.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Be less verbose when sending socket messages

cURL is already logging this for us.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Be patient when the websocket buffer is full

When the log stream is starting a little while into the build, there
might be a big burst of data sent through the socket which might fill it
up. Then we need to be more patient and send the message again
remembering what has been sent before.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Always require WebSocket support in cURL

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Drop building for Ubuntu 24.04

This version does not have a recent enough version of cURL and
therefore, the build fails. I don't think we need to support anything
old here that does not support WebSockets.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Send timestamp when streaming the log

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log buffer: Store a timestamp with each line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Add function to format timeval to ISO 8601 format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: package: Drop storing a reference to the context

This was an ugly hack to create a new package list which we don't need
any more. This caused the application to occasionally crash because ctx
wasn't set at all times.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

package: Let reverse requires create its own package list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix more error codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Refactor creating a Python list from package list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add a simple test to create a daemon

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Listen to SIGTERM and terminate the jail

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add option to read prerequires from packages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Always turn tmpfs usage on for local builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Require a flag to turn on using tmpfs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Permit to finish jobs without uploads

This is even called when the build has failed in which case we don't
have any packages that we can use here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Untangle reading the filelist and extraction

The idea here was to re-use existing code, but I am not sure this is
currently very helpful to anyone.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Re-enable parsing checksums again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

compress: Remove header for dropped pakfire_extract() function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Open an archive in stub mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Add some tests for the main Pakfire class

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Minor code cleanup

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Rename File.digest() to File.checksum()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Return the error code when fetching the filelist

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Update translations

This is mainly to update po/POTFILES.in

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Add some simple tests for pakfire.version_compare()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Remove unused Python exceptions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Test pakfire.supported_arches()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Tidy up writing repository configuration

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Don't return an empty mirrorlist URL, return NULL instead

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Fix exporting keys

Before, the key was randomly written into the file descriptor.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

keys: Make a simple function to export a key as a string

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix setting/reading the context's cache path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: python: Update renamed checksum field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Initialize the directory file descriptor

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log buffer: Implemnent ring buffer behaviour

I don't know why I didn't implement this straight away as this was
always intended to be a ring buffer.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Correctly set pidfd on initialization

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Allow turning on/off log streaming by the job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Handle message to abort a build job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>