db: Fix remaining index names

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Drop unused repo_builds table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Update users indexes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Rename deleted to deleted_at

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Show when jobs time out

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Automatcially abort any jobs after three hours

If builds hang or have some other problem, we will automatically abort
them after three hours so that we don't keep the builders up idle.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Optimise efficiency of the running jobs index

This does not need to contain any deleted jobs and therefore can be
slightly smaller.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Allow searching for files with wildcards

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix sorting of the queue

Scratch builds are now being moved to the top of the queue and packages
will be built by when their dependency check has succeeded.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix Kerberos authentication

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Add controls to approve builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Move repos above test builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Rewrite all relevant repositories when deleting builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Remove unnecessary task group when deleting builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Allow up to 1024 concurrent database connections

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logstream: Fix sending the last messages

Instead, the first messages in the buffer were sent.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Delete builds one step at a time

Creating a lot of sub-tasks caused some SQL statements to be executed
outside its transaction and the connection pool to starve.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix UUID comparison that differs in type

We use UUID for PostgreSQL and now receive the UUID in Python's UUID
type which surprisingly does not compare to strings...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Increase the maximum number of connections again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Import packages one by one

This could have caused the appliance to run out of available database
connections.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

misc: Fix format_size from running out of sizes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Move cleanup job into the main thread

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cron: Remove unneeded cronjobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Close connections after 5 seconds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Sync mirrors from the main thread

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Define background tasks here

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Limit the maximum number of database connections to 64

Our servers are configured to only allow up to 100 connections total
which had funny effects here...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Show when jobs have been aborted in queue

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

monitorings: Perform checks one at a time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Send "noarch" as build architecture to builders

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

monitoring: Catch errors when we cannot create a build

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Add an extra barrier if we are missing files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Revert back to authentication using a web form

Using Kerberos authentication for this part seems to be more complicated
than necessary and does not provide any benefits.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Force builders to refresh repository data all the time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sync: Write timestamp so that we can track when mirrors were synced last

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Fix joining paths

The original path of the mirror was unintentionally dropped.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Store groups as string array

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Store build ID as UUID

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Store build arches and apply them when creating jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Only check out those files that we need

Since the entire build service is very heavy on I/O, we should avoid
writing files that we don't need.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Create an index over the superseeded_by field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Capture & store Pakfire log on dist()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Add PakfireLogger class

This class can be used to capture the log from Pakfire in order to debug
issues easier.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix styling of edit page

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Call dispatch after launching any jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Fix call to launch builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repo: Add simple API operations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Don't remove builds that have been removed before

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Reset jobs as successful if following jobs succeed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Show logs of previous attempts

Fixes: #13249 - jobs: Add option to download logs from previous attempts
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Immediately launch builds in the transaction

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Read archive metadata in executor thread

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Fail more gracefully if a package has been deleted from disk

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Re-add accidentially removed import of the queue module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Don't keep so many connections open and idle

Since we usually peak briefly, we will fork a lot of PostgreSQL
processes that we will never need again. In order to avoid wasting too
much memory, we close those connections quicker, but at least 8 open so
that new tasks will quickly be able to acquire a database connection.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds+jobs: Use new methods to access database objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Show message if no jobs are queued

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Log deleting packages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Try to delete packages if creating build fails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Don't try to overwrite existing packages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Remove some unnecessary imports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Extract filelists in a separate thread

Since we need to read the entire archive to extract the entire filelist
with all its meta information, we need to move this into a separate
thread.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Tail logs more efficiently

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Automatically launch builds after dist

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cache: Tidy up code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Create a simple rate limiter

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

release monitoring: Do not try to update a bug that has never been created

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cache: Implement an async redis cache driver

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Launch installcheck in transactions and wait for it

It can happen that a transaction is being rolled back and an
installcheck has been queued which will then run into errors.

Since the installcheck is now fast enough, we can simply wait for it
with the exception of relaunching any pending jobs in repositories which
will be run as one single task in the background.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

monitoring: Put the rate limiter in the correct place

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

monitoring: Complete configuration before running the first check

Because we are communicating with Bugzilla (potentially) we might undo a
lot of stuff if there is any error.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Pass configuration as file object

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Make Pakfire an async context manager

This should help us to block less when initializing a Pakfire instance.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Show controls to create repos when none exist

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

release monitoring: Use new version compare function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Use aiofiles for asyncio handling of temporary files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

index: Show recently finished jobs instead of pending ones

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Move job queue

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Add option to only show failed builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: queue: Show build time for finished jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: queue: Show status for failed/finished jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Group by date

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Add pagination to new page

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Create new page that lists all finished jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Create UI module for queued jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix indefinite loop in dispatch function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix SQL query to fetch all release/scratch builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logstreams: Increase the maximum number of log messages stored

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logstream: Lock buffer when it is being modified

This might be cheaper than copying it when it becomes large.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "logstreams: Don't wait until all messages have been delivered to all consumers"

This reverts commit 1fa487c5455f3e4a63ae0755c466c2a7ee62b4e5.

This might potentially shuffle the messages :(

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Run fewer installchecks

Only run this for jobs that have previously failed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Perform installcheck before a job is being dispatched

I would like to perform fewer of the installchecks because they unlikely
to fail and rather expensive to run.

This is a safety net so that we will never dispatch any jobs that cannot
be built.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logstreams: Copy buffer before sending it off

collections.deque() does not allow us to append new messages while we
are reading from it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

logstreams: Don't wait until all messages have been delivered to all consumers

This will just queue tasks for all new messages and allows us to get
back to processing the next message quicker.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Return zero for total build time if there are no builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Include disabled builders in autoscale

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repositories: Master repositories as async as possible

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

robots.txt: Disallow download files from packages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: make_url: Filter out any values that are None

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Commit forgotten bug templates

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>