messages: Fix sending emails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Move log stream frontend code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix sending emails to watchers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Send a push notification when a build is ready

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Hash builders by hostname for their stats connections

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix memory usage calculation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Add a combined operation to flush and refresh

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix fetching watchers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix sending stats

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix fetching the value for enabled

It has to be a boolean.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Move stats JS into the main file

We currently cannot easily add more files in the right place, so since
there is only so little code, we just ship it all together.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Associate imported packages with the job

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Flush and refresh to read the finished timestamp

This feels like a stupid hack but this at least prevents utter mayhem
happening after this.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Turn JobPackage into a model

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Log less

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Remove some dead code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Let's try running autoscaling in the background

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Use the renamed checksum functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Simplify the queue estimation

It is not a very exact science anyways to try to guess how long a job
will take, so we are now just counting them which should save a couple
of complex database queries when running this.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

aws: Be a little bit less verbose

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Commit to the database as soon as we have dispatched it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix finishing jobs

Obviously the database field is nullable and NULL by default.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Actually call the add_watcher() coroutine

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Ensure that we have packages when a job finishes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Allow passing keyword arguments to background tasks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Don't cache preceeding jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Break some dependency cycle somewhere

SQLAlchemy failed to load alljobs in Builds and I could not quite figure
out why. It seems that when both properties that form the relationship
should be fetched in a single join, nothing happens and SQLAlchemy does
not even seem to complain about it. Oh well.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Require builders to be created by someone

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Drop the settings table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Use as many inner joins as possible

We also are very unlikely to fetch any deleted objects, so let's rather
pull in the users that have deleted something by "SELECT IN" so that the
queries will be smaller and we don't add any joins which almost never
will find anything.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Fetch messages in badges

We would have otherwise looped for forever since we cannot check in
advance whether the iterator actually has any elements, but we don't
want to connect to the mail relay if there is literally nothing to send.
So as a compromise we fetch messages in batches.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix fetching running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Consider a builder running when it is online

This is a lot cheaper to test and should be sufficiently accurate.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix formatting timeout time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Ensure we commit to the database after each message

We have this weird problem that after any background tasks, SQLAlchemy
won't automatically commit. Therefore all those changes were lost.

But since we want to avoid sending messages again if something else
later in a batch fails we will commit any changes immediately after each
message has been sent.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

settings: Remove database settings

We read stuff from the configuration file now.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

release monitoring: Read the API key from configuration file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Refactor the whole thing

This will now asynchronously send messages from the main thread.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Load SSL context from certificate files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Don't log arch which can be empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Fix rename installsize field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Fix relaunching pending jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix importing DependencyError

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Set cache path in context

This has now been fixed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Don't abort when already aborted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Initialize Pakfire in stub mode

This way, Pakfire does not need a path, won't initialize the database
and so on. It simply can be used to perform dependency solving which is
all we need here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop unused control connection stuff

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Update message to abort

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Rename the finished handler

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Make fetching multiple uploads async

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop the control connection

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Consider spaces when splitting Accept-Enconding headers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Import log files as they are when they are already compressed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Commit the database session before sending response

The client could reconnect very quickly, before the database has session
has been commited which will result on the upload failing. Therefore we
make sure that the implicitely opened transaction has been commited
before we send the response.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Use existing URLs to compose mirror URLs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Don't include the build service in the mirrorlist

pakfire will already fall back to it, if baseurl is set.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Refactor this once again

This is not 100% async but good enough. If the system cannot write to
its own file system we probably have bigger issues.

This will now spool the uploaded data for a little while but soon flush
it out once it has become too large to be kept in memory for forever.

The file will then be copied (yes I know) and the new path stored.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Drop an unused function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Calculate the digest when receiving the file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix sending configuration

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Return the received filename in the API response

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Add a debug handler to download any uploaded payload

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Send an URL to the client where to send the payload

Hopefully this will give us some more flexibility in the future.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Manually commit the session

This is required because the session might not be fast enough before the
client comes back with the payload. This ensures that we won't have a
race.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Pass the configuration to Pakfire as string

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Check for all required Python module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

constants: Remove unused values

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Rename to PBS

This is easier than a hyphenated Python package name. This is now also
moved out of the pakfire module which has become a pure C module.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Store control connections by their UUID

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Add crash handler

This will be called by the builders when the build process has
unexexpectedly crashed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix retrying jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix aborting jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Don't fetch index data in a transaction

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix fetching running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Refactor dispatching jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Tidy up code

No functional changes.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Make it easier to generate a text configuration

We cannot use the old method any more since we need to call async
methods.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: (Re-)raise any exceptions after the task is done

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix least busy iterator

This must be an outer join so that we will always get all builds, even
those that don't have a job running.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix accidental join

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Use shorter code return queue length

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Fix loads of incorrect joins

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cli: Fix running installchecks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Send log files with decompressing them if client supports it

If the browser supports gzip, we simply send the raw file as there is no
need to decompress it first and then have the reverse proxy compress it
again.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

events: Rely more on "selectin"

This seems to be slightly faster in case of larger result sets.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Initialize the locale

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Log in the user before checking the XSRF cookie

In case the cookie validation fails, we need to be able to send an error
message which accesses current_user.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Implement custom logger that only logs debugging

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Bring back the welcome push message

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Don't include the TTL in the payload of the push message

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Simplify passing around push messages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Correctly set session expiry time in cookies

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sessions: Correctly set expiry time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Send push messages from the same task

This is necessary because SQLAlchemy cannot delete the push
notification from a different task.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Fix sending push notifications from command line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Fix deleting push subscriptions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Move JS for push notifications into the main file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Fix subscribing to push notifications

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix showing number of running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Store when a builder has connected last and from where

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>