xfer: Fix logic error in reading WebSocket messages

After the message was read we never cleared the buffer which resulted
that on every message, the callback was called with the entire buffer.
Usually that meant that only the first JSON message was parsed and
handled. Again and again and again...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Remove recursive function call

I am not sure why I am suddenly scared of this, but there have been some
security-relevant issues where this could exhaust the stack.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Use our own string comparison macro

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Send the IDs of all running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Stream logs as base64-encoded data

There should not be anything else but ASCII or Unicode characters here,
but we can never be too sure. Therefore we will encode all data as
base64 to keep the protocol somewhat extensible.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

base64: Replace the encoder/decoder with low-level functions

The BIO interface is large and very well suited to encode larger
messages. However, we only have very small messages to encode and
therefore will spend a long time on setting up and destroying the BIO
interface all of the time.

These low-level functions are a lot faster and easier to set up.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Rename fields again

I even confused myself here because length is actually not the length of
the buffer that I wanted. It is what has been allocated. That is however
nothing that we need to access at any time. Therefore the fields have
been renamed to be a little bit more intuitive again.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Replace the send buffer with our own buffer implementation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Use our own buffer implementation for the receive buffer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Fix return codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Log a better message when a build has been interrupted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

xfer: Clear the overflow buffer after the segment could be sent

This is a stub and I will replace the send and receive buffers with our
own buffer implementation.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Store Kerberos credentials in memory

I think I am starting to become confused. At least in trixie and
bookworm-backports, we seem to be linking against MIT. This seems to
work now on my machine. Let's hope this is a general thing.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Completely destroy the Kerberos credentials cache on update

This is required because on Debian, cURL is linked against Heimdal which
does not seem to be able to skip any deleted tickets in the file.
Instead it always uses the first ticket which would eventually have
expired. Therefore we destroy the entire cache and write it again so
Heimdal cannot get confused.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Ensure we have gawk available on Fedora

Fedora 42 does not seem to install this by default any more, but our
configure script wants to use it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Always install all updates before trying to install any packages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Ensure the keyring is up to date on Archlinux

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dist: Remove the old solution to pass the filename

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive: Add function to fetch the desired filename

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add tests for dist()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Make dist() work again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

file: Don't try to resolve paths relative to Pakfire in stub mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pwd: Don't permit any UID/GID lookups in stub mode

This is simply not supported because we don't have a file system.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packager: Don't use an invalid pointer to a freed JSON object

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

archive write: Fetch length of serialized JSON string from libjson-c

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

job: Always stream logs

This was too fragile before and I think we should be able to afford the
stupid traffic...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Use function to check if a flag is set

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Retry authentication if we could not reach the KDC

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Remove any previous credentials

Otherwise krb5 will append all credentials, but Heimdal only seems to
read the first (eventually) expired ticket.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

daemon: Make Kerberos refresh time easier to understand

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Return converted timestamps as long

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Move the main callback

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Use our own string comparison function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Fix return codes of bind function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: execute: Implement bind-mounting mountpoints

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: execute: Capture stdout/stderr separately

That way, we can raise the error output and return the rest

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Actually verify the set nice level

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Bring back the open file descriptor test

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Actually check the output

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add some simple tests for the buffer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Only negative return values are considered unsuccessful

Sometimes we want to return the length of consumed input, so this should
all be considered successful.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Update translations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Don't fail if no callback has been registered

This should usually never happen.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Don't drop into a shell when build has been interrupted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Cleanly terminate on SIGINT

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Move syscall wrapper to where it should be

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: execute: Check that we are not running for forever

There was a bug once where the loop never terminated if the process
actually could never be executed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Drain the pipes after the process has terminated

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Free the input buffer after we are done

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Free string array after use

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Move the filelist streaming helper from PTY

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Move the send buffer function from the PTY

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Refactor the output callback

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

build: Remove any traces of the deprecated PTY stdin callback

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Refactor streaming data into the jail

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: jail: Consume any output of the pipe

This is just because of the log streamer complaining

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Add a helper function that fills a buffer with a callback

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Return the number of bytes written

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Only make the parent ends of the pipes non-blocking

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Don't allow to resize the buffer smaller than what it holds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Don't allow the buffer to grow larger than its defined length

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pakfire: Don't leak our locking fd into the jail

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Flush any output after we have updated the buffers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Use our own buffer implementation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: log stream: Replace PTY test with file handle test

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Remove code that maps CRNL to NL

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Remove all sorts of code that is not needed any more

This is all of the non-interactive code.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Bring back pipes for non-interactive sessions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Use the PTY only for interactive sessions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Always allocate at least one chunk

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Automatically shrink the buffer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Correctly compute the new desired length when growing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Add function to configure the maximum length

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Give the functions slightly better names

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Use the new buffer implementation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

buffer: Create a new generic buffer implementation

This is used to hold a dynamically growing buffer.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Expect the callback to consume the entire line

This is otherwise getting out of hand and we should not have a place
where this would become an issue.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Use log stream to perform line-buffered output

This helps us to split the complexity between the PTY and some extra
buffering code.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: log stream: Stream another line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: execute: Use the send buffer callback we already have

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Move data as second argument of the callback

That way, we are compatible with the PTY stdout callback.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Change first argument of the callback to ctx

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

PTY: Remove the rest of the line buffering code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log stream: Add an experimental implementation to stream from PTY

This is the first step to split all the complicated line buffering from
the PTY code.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log streamer: Move line sanitization from PTY

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Major refactor

This adds loads of comments to hopefully make this thing more robust. So
far we don't really seem to be able to implement all the features we
need without destroying others.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Pass the program exit code as return code of the loop

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: command: Always return zero after the pipe is finished

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Always disable character echoing

This usually happens later in case of the interactive version, but since
we don't need to configure much in the pipe case, we can just set this
as default.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Add a function to set up interactive mode

This way we can split off the initialization code a little bit better
from the main initialization of the PTY object.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Refactor how we capture output

This will now avoid copying all the data around between the kernel and
userland and will also simply use our own callbacks which helps us to
keep the code a little bit less of an if/else hell.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Remove use of removed PTY flags

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Re-organise flags

The CONNECT* flags have been used for two purposes which got mixed up on
the way which is why the standard input was not connected for callbacks.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Add more logging when reading/writing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

pty: Automatically drain when the event loop finishes

There was the problem that some content could have remained stuck in the
buffers on occasion.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Fail with error code if fgetc() returns an error

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

transaction: Tidy up cleanup code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jail: Tidy up creating a new jail

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: pakfire: Keep a reference to the Python context

Since we are registering all sorts of callbacks to here, we cannot
deallocate the Ctx object, or we won't be able to use them any more.

This is being fixed by holding a reference to the Python object instead
of the context itself. That will at least help us with lots of these
problems.

There is still, however, the problem that both the Python Ctx and
Pakfire object can be dereferenced when the backend objects will remain
existing.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Pakfire: Remove old callbacks from headers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>