registry: Don't fail if we could not find a file inside an image

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

registry: Add a simple in-memory cache for indexes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

registry: Respond with the correct code when manifest was not found

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

registry: Add a tags handler

This lists all available tags

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

registry: Send proper API error if something wasn't found

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

registry: Split the combined handlers

I think this might all be slightly better to read instead of all the ifs
and elses.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

registry: Stream blobs from a separate thread

We need this to not block the main thread on really large blobs.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

decorators: Create a decorator to run something in a thread

This is just for convenience and to write less code since we are using
this pattern a lot.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

registry: Create a function to find a blob

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

registry: Build a basic read-only OCI registry

This allows us to easily run any OCI images in Docker/podman/etc. for CI
testing without engaging any external storage, etc.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

images: Build out class and add the table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix relationship to groups

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix creating test groups

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix dedicated log streaming page

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Wrap avatars in <figure>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Put most recently finished jobs to the top

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix the watchers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Let Apache deliver the log files

This will be even more efficient than sending them in the main process.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Call the right function to create a group

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix fetching reverse requires

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix log streaming

This will now dynamically turn this on and off whenever it is needed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop the old log stream module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Fix sending emails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Move log stream frontend code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix sending emails to watchers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Send a push notification when a build is ready

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Hash builders by hostname for their stats connections

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix memory usage calculation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Add a combined operation to flush and refresh

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix fetching watchers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix sending stats

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix fetching the value for enabled

It has to be a boolean.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Move stats JS into the main file

We currently cannot easily add more files in the right place, so since
there is only so little code, we just ship it all together.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Associate imported packages with the job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Flush and refresh to read the finished timestamp

This feels like a stupid hack but this at least prevents utter mayhem
happening after this.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Turn JobPackage into a model

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Log less

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Remove some dead code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Let's try running autoscaling in the background

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Use the renamed checksum functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Simplify the queue estimation

It is not a very exact science anyways to try to guess how long a job
will take, so we are now just counting them which should save a couple
of complex database queries when running this.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

aws: Be a little bit less verbose

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Commit to the database as soon as we have dispatched it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix finishing jobs

Obviously the database field is nullable and NULL by default.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Actually call the add_watcher() coroutine

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Ensure that we have packages when a job finishes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Allow passing keyword arguments to background tasks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Don't cache preceeding jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Break some dependency cycle somewhere

SQLAlchemy failed to load alljobs in Builds and I could not quite figure
out why. It seems that when both properties that form the relationship
should be fetched in a single join, nothing happens and SQLAlchemy does
not even seem to complain about it. Oh well.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Require builders to be created by someone

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Drop the settings table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Use as many inner joins as possible

We also are very unlikely to fetch any deleted objects, so let's rather
pull in the users that have deleted something by "SELECT IN" so that the
queries will be smaller and we don't add any joins which almost never
will find anything.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Fetch messages in badges

We would have otherwise looped for forever since we cannot check in
advance whether the iterator actually has any elements, but we don't
want to connect to the mail relay if there is literally nothing to send.
So as a compromise we fetch messages in batches.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix fetching running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Consider a builder running when it is online

This is a lot cheaper to test and should be sufficiently accurate.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix formatting timeout time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Ensure we commit to the database after each message

We have this weird problem that after any background tasks, SQLAlchemy
won't automatically commit. Therefore all those changes were lost.

But since we want to avoid sending messages again if something else
later in a batch fails we will commit any changes immediately after each
message has been sent.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

settings: Remove database settings

We read stuff from the configuration file now.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

release monitoring: Read the API key from configuration file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Refactor the whole thing

This will now asynchronously send messages from the main thread.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Load SSL context from certificate files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Don't log arch which can be empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Fix rename installsize field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Fix relaunching pending jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix importing DependencyError

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Set cache path in context

This has now been fixed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Don't abort when already aborted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Initialize Pakfire in stub mode

This way, Pakfire does not need a path, won't initialize the database
and so on. It simply can be used to perform dependency solving which is
all we need here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop unused control connection stuff

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Update message to abort

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Rename the finished handler

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Make fetching multiple uploads async

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop the control connection

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Consider spaces when splitting Accept-Enconding headers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Import log files as they are when they are already compressed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Commit the database session before sending response

The client could reconnect very quickly, before the database has session
has been commited which will result on the upload failing. Therefore we
make sure that the implicitely opened transaction has been commited
before we send the response.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Use existing URLs to compose mirror URLs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Don't include the build service in the mirrorlist

pakfire will already fall back to it, if baseurl is set.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Refactor this once again

This is not 100% async but good enough. If the system cannot write to
its own file system we probably have bigger issues.

This will now spool the uploaded data for a little while but soon flush
it out once it has become too large to be kept in memory for forever.

The file will then be copied (yes I know) and the new path stored.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Drop an unused function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Calculate the digest when receiving the file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix sending configuration

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Return the received filename in the API response

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Add a debug handler to download any uploaded payload

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Send an URL to the client where to send the payload

Hopefully this will give us some more flexibility in the future.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Manually commit the session

This is required because the session might not be fast enough before the
client comes back with the payload. This ensures that we won't have a
race.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Pass the configuration to Pakfire as string

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Check for all required Python module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

constants: Remove unused values

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Rename to PBS

This is easier than a hyphenated Python package name. This is now also
moved out of the pakfire module which has become a pure C module.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Store control connections by their UUID

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Add crash handler

This will be called by the builders when the build process has
unexexpectedly crashed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix retrying jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix aborting jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Don't fetch index data in a transaction

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix fetching running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Refactor dispatching jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Tidy up code

No functional changes.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Make it easier to generate a text configuration

We cannot use the old method any more since we need to call async
methods.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: (Re-)raise any exceptions after the task is done

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>