jobs: Fix finishing jobs

Obviously the database field is nullable and NULL by default.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Actually call the add_watcher() coroutine

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Ensure that we have packages when a job finishes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Allow passing keyword arguments to background tasks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Don't cache preceeding jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Break some dependency cycle somewhere

SQLAlchemy failed to load alljobs in Builds and I could not quite figure
out why. It seems that when both properties that form the relationship
should be fetched in a single join, nothing happens and SQLAlchemy does
not even seem to complain about it. Oh well.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Require builders to be created by someone

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Drop the settings table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Use as many inner joins as possible

We also are very unlikely to fetch any deleted objects, so let's rather
pull in the users that have deleted something by "SELECT IN" so that the
queries will be smaller and we don't add any joins which almost never
will find anything.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Fetch messages in badges

We would have otherwise looped for forever since we cannot check in
advance whether the iterator actually has any elements, but we don't
want to connect to the mail relay if there is literally nothing to send.
So as a compromise we fetch messages in batches.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix fetching running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Consider a builder running when it is online

This is a lot cheaper to test and should be sufficiently accurate.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix formatting timeout time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Ensure we commit to the database after each message

We have this weird problem that after any background tasks, SQLAlchemy
won't automatically commit. Therefore all those changes were lost.

But since we want to avoid sending messages again if something else
later in a batch fails we will commit any changes immediately after each
message has been sent.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

settings: Remove database settings

We read stuff from the configuration file now.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

release monitoring: Read the API key from configuration file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

messages: Refactor the whole thing

This will now asynchronously send messages from the main thread.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Load SSL context from certificate files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Don't log arch which can be empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

packages: Fix rename installsize field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Fix relaunching pending jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix importing DependencyError

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Set cache path in context

This has now been fixed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Don't abort when already aborted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Initialize Pakfire in stub mode

This way, Pakfire does not need a path, won't initialize the database
and so on. It simply can be used to perform dependency solving which is
all we need here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop unused control connection stuff

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Update message to abort

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Rename the finished handler

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Make fetching multiple uploads async

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Drop the control connection

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Consider spaces when splitting Accept-Enconding headers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Import log files as they are when they are already compressed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Commit the database session before sending response

The client could reconnect very quickly, before the database has session
has been commited which will result on the upload failing. Therefore we
make sure that the implicitely opened transaction has been commited
before we send the response.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

mirrors: Use existing URLs to compose mirror URLs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

repos: Don't include the build service in the mirrorlist

pakfire will already fall back to it, if baseurl is set.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Refactor this once again

This is not 100% async but good enough. If the system cannot write to
its own file system we probably have bigger issues.

This will now spool the uploaded data for a little while but soon flush
it out once it has become too large to be kept in memory for forever.

The file will then be copied (yes I know) and the new path stored.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Drop an unused function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Calculate the digest when receiving the file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix sending configuration

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Return the received filename in the API response

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Add a debug handler to download any uploaded payload

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Send an URL to the client where to send the payload

Hopefully this will give us some more flexibility in the future.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Manually commit the session

This is required because the session might not be fast enough before the
client comes back with the payload. This ensures that we won't have a
race.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Pass the configuration to Pakfire as string

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Check for all required Python module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

constants: Remove unused values

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Rename to PBS

This is easier than a hyphenated Python package name. This is now also
moved out of the pakfire module which has become a pure C module.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Store control connections by their UUID

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Add crash handler

This will be called by the builders when the build process has
unexexpectedly crashed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix retrying jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix aborting jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Don't fetch index data in a transaction

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Fix fetching running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Refactor dispatching jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Tidy up code

No functional changes.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

config: Make it easier to generate a text configuration

We cannot use the old method any more since we need to call async
methods.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: (Re-)raise any exceptions after the task is done

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix least busy iterator

This must be an outer join so that we will always get all builds, even
those that don't have a job running.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix accidental join

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Use shorter code return queue length

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Fix loads of incorrect joins

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cli: Fix running installchecks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Send log files with decompressing them if client supports it

If the browser supports gzip, we simply send the raw file as there is no
need to decompress it first and then have the reverse proxy compress it
again.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

events: Rely more on "selectin"

This seems to be slightly faster in case of larger result sets.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Initialize the locale

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

web: Log in the user before checking the XSRF cookie

In case the cookie validation fails, we need to be able to send an error
message which accesses current_user.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Implement custom logger that only logs debugging

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Bring back the welcome push message

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Don't include the TTL in the payload of the push message

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Simplify passing around push messages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Correctly set session expiry time in cookies

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sessions: Correctly set expiry time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Send push messages from the same task

This is necessary because SQLAlchemy cannot delete the push
notification from a different task.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Fix sending push notifications from command line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Fix deleting push subscriptions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Move JS for push notifications into the main file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Fix subscribing to push notifications

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Fix showing number of running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Store when a builder has connected last and from where

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Hash connections by hostname

Since we now have one builder object per session they don't match their
hashes any more. Therefore we identify them by their name which is also
unique.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Log when builders connect or disconnect

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

uploads: Actually send an empty response

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

watchers: Fetch users as selectin

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Finish relationship to points

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

CSS: Fix broken stuff by importing CSS in the correct order

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

CSS: Drop unused style.css

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Show system information

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Rename a couple of fields and add system

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Add a helper function to trigger a commit

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Limit the amount of concurrent log tail operations

This way we will ensure that we access one file at a time and put the
others in a queue to not starve IO too hard.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Stream the log file faster

When we are downloading the entire file, we only have to decompress it
and we don't have to search for line breaks before sending each chunk.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Ratelimit access to logs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix deleting builds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

events: Correctly show the bug ID

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builders: Refactor auto-scaling

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Propagate back the group

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Make repos read-only

SQLAlchemy complained and wanted this flag set.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jobs: Add a CTE to fetch all currently running jobs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix rendering bugs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

builds: Fix incorrect attribute access

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>