Migrate to an even more async design

This monster commit replaces our database engine by SQLAlchemy so that
we can utilise eager loading and don't have to implement this on our
own. Although Jinja is a lot more flexible and allows us to
asynchronically fetch any data, we cannot really have async properties.
Therefore it is a good idea to have some eager loading happening when we
fetch certain objects that will always depend on some other objects.

This has been a major rewrite of the backend which still isn't done, but
I needed to commit this somehow. There was little point it doing it
gradually so here is this mountain of a commit. Enjoy.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Load krb5 and sync settings from config

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

AWS: Read configuration from file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Implement translation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Load some useful Jinja extensions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Convert login page to Jinja

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Set globals to avoid a namespace that is too large

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

templates: Convert the error template to Jinja

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

templates: Convert the base template to Jinja

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Make the request handler as async as possible

This code is mainly copied from upstream and made to work with async
functions for the error handlers.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Make the new template engine async

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Replace the Tornado template engine with Jinja

We need this to support any async code in the templates.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Try to make this entire application async

We will need this so that we won't run out of database file descriptors
as psycopg3 currently can only use select() which does not support a
file descriptor to have a number over 1024.

We will also need this to scale this application better. However,
Tornado has some limitations in its template engine which is why we will
need to replace the template engine with Jinja.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

httpclient: Read proxy configuration from config file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Read base URL from configuration

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Launch Pakfire with its own context

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: jobs: Include the default backend

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Drop the POST-based stats handler

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Require authentication when uploading the payload

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Regularly ping WebSocket clients to keep connections alive

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Disable autocomplete for passwords

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cookies: Set secure attribute for all cookies

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Pakfire: Remove deprecated arguments

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update setting the cache path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Update API

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Update API authentication

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix API authentication

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Make statistics percentage value range from 0 - 1

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Add a new stats handler

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Update API error response codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Show if payload has been received and block sending it again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Require SPNEGO

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Use errno codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Log any incorrect digests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Align response codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Fail quickly if no payload was received

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Fix deleting uploads if no payload was received

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Do not try to send (multiline) error messages in status

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Make APIError inherit from HTTPError

This way, tornado will catch the exception and log something useful

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Send better errors for failed upload requests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Rewrite the whole thing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Add decorator to require clients to do SPNEGO

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Return 0 when no quota has been used

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Create a temporary directory

As a non-privileged user, we cannot mount a ramdisk which is why we
cannot use this and need to pre-create an empty directory.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Only set priority if configured

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix quota calculation for release builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Fix serialising UUIDs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Implement sending debuginfo

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

events: Show when builds are moving repository

This keeps the event log shorter and makes it clearer what has happened.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Show used daily build quota

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Obviously we need to account for deleted builds/jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Introduce daily build quotas

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Make scratch builds index more efficient

This should not include any test builds.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Fix how we are calculating disk usage

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Set default storage quota for users

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Remove empty SQL statement

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix users delete field update

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Rename quota to storage quota

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Fix remaining index names

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Drop unused repo_builds table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Update users indexes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Rename deleted to deleted_at

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Show when jobs time out

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Automatcially abort any jobs after three hours

If builds hang or have some other problem, we will automatically abort
them after three hours so that we don't keep the builders up idle.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Optimise efficiency of the running jobs index

This does not need to contain any deleted jobs and therefore can be
slightly smaller.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Allow searching for files with wildcards

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix sorting of the queue

Scratch builds are now being moved to the top of the queue and packages
will be built by when their dependency check has succeeded.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix Kerberos authentication

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Add controls to approve builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Move repos above test builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Rewrite all relevant repositories when deleting builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Remove unnecessary task group when deleting builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Allow up to 1024 concurrent database connections

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logstream: Fix sending the last messages

Instead, the first messages in the buffer were sent.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Delete builds one step at a time

Creating a lot of sub-tasks caused some SQL statements to be executed
outside its transaction and the connection pool to starve.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix UUID comparison that differs in type

We use UUID for PostgreSQL and now receive the UUID in Python's UUID
type which surprisingly does not compare to strings...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Increase the maximum number of connections again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Import packages one by one

This could have caused the appliance to run out of available database
connections.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

misc: Fix format_size from running out of sizes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Move cleanup job into the main thread

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cron: Remove unneeded cronjobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Close connections after 5 seconds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Sync mirrors from the main thread

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Define background tasks here

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Limit the maximum number of database connections to 64

Our servers are configured to only allow up to 100 connections total
which had funny effects here...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Show when jobs have been aborted in queue

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

monitorings: Perform checks one at a time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Send "noarch" as build architecture to builders

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

monitoring: Catch errors when we cannot create a build

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Add an extra barrier if we are missing files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Revert back to authentication using a web form

Using Kerberos authentication for this part seems to be more complicated
than necessary and does not provide any benefits.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Force builders to refresh repository data all the time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sync: Write timestamp so that we can track when mirrors were synced last

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Fix joining paths

The original path of the mirror was unintentionally dropped.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Store groups as string array

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Store build ID as UUID

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Store build arches and apply them when creating jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Only check out those files that we need

Since the entire build service is very heavy on I/O, we should avoid
writing files that we don't need.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Create an index over the superseeded_by field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Capture & store Pakfire log on dist()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>