]>
git.ipfire.org Git - pbs.git/log
Michael Tremer [Wed, 13 Sep 2023 17:43:55 +0000 (17:43 +0000)]
builds: Delete builds one step at a time
Creating a lot of sub-tasks caused some SQL statements to be executed
outside its transaction and the connection pool to starve.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 13 Sep 2023 15:52:19 +0000 (15:52 +0000)]
jobs: Fix UUID comparison that differs in type
We use UUID for PostgreSQL and now receive the UUID in Python's UUID
type which surprisingly does not compare to strings...
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 13 Sep 2023 15:41:10 +0000 (15:41 +0000)]
database: Increase the maximum number of connections again
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 13 Sep 2023 15:40:45 +0000 (15:40 +0000)]
jobs: Import packages one by one
This could have caused the appliance to run out of available database
connections.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 13 Sep 2023 15:40:18 +0000 (15:40 +0000)]
misc: Fix format_size from running out of sizes
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 9 Sep 2023 13:50:18 +0000 (13:50 +0000)]
backend: Move cleanup job into the main thread
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 9 Sep 2023 13:49:15 +0000 (13:49 +0000)]
cron: Remove unneeded cronjobs
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 9 Sep 2023 13:49:02 +0000 (13:49 +0000)]
database: Close connections after 5 seconds
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 9 Sep 2023 13:47:13 +0000 (13:47 +0000)]
backend: Sync mirrors from the main thread
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 9 Sep 2023 13:45:56 +0000 (13:45 +0000)]
backend: Define background tasks here
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 8 Sep 2023 16:10:10 +0000 (16:10 +0000)]
database: Limit the maximum number of database connections to 64
Our servers are configured to only allow up to 100 connections total
which had funny effects here...
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 8 Sep 2023 14:53:15 +0000 (14:53 +0000)]
jobs: Show when jobs have been aborted in queue
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 8 Sep 2023 14:49:48 +0000 (14:49 +0000)]
monitorings: Perform checks one at a time
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 8 Sep 2023 14:49:01 +0000 (14:49 +0000)]
config: Send "noarch" as build architecture to builders
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 8 Sep 2023 14:48:35 +0000 (14:48 +0000)]
monitoring: Catch errors when we cannot create a build
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 2 Sep 2023 13:09:45 +0000 (13:09 +0000)]
sources: Add an extra barrier if we are missing files
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 2 Sep 2023 12:32:28 +0000 (12:32 +0000)]
auth: Revert back to authentication using a web form
Using Kerberos authentication for this part seems to be more complicated
than necessary and does not provide any benefits.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 2 Sep 2023 09:30:23 +0000 (09:30 +0000)]
jobs: Force builders to refresh repository data all the time
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 1 Sep 2023 16:16:33 +0000 (16:16 +0000)]
sync: Write timestamp so that we can track when mirrors were synced last
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 1 Sep 2023 16:08:54 +0000 (16:08 +0000)]
mirrors: Fix joining paths
The original path of the mirror was unintentionally dropped.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 31 Aug 2023 03:28:27 +0000 (03:28 +0000)]
packages: Store groups as string array
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 30 Aug 2023 18:07:43 +0000 (18:07 +0000)]
packages: Store build ID as UUID
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 30 Aug 2023 17:49:26 +0000 (17:49 +0000)]
packages: Store build arches and apply them when creating jobs
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 26 Aug 2023 11:18:22 +0000 (11:18 +0000)]
sources: Only check out those files that we need
Since the entire build service is very heavy on I/O, we should avoid
writing files that we don't need.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 26 Aug 2023 11:14:33 +0000 (11:14 +0000)]
jobs: Create an index over the superseeded_by field
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 26 Aug 2023 11:14:03 +0000 (11:14 +0000)]
sources: Capture & store Pakfire log on dist()
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 26 Aug 2023 11:12:10 +0000 (11:12 +0000)]
config: Add PakfireLogger class
This class can be used to capture the log from Pakfire in order to debug
issues easier.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 26 Aug 2023 10:11:31 +0000 (10:11 +0000)]
builders: Fix styling of edit page
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 16:01:11 +0000 (16:01 +0000)]
jobs: Call dispatch after launching any jobs
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 16:00:52 +0000 (16:00 +0000)]
sources: Fix call to launch builds
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 15:27:31 +0000 (15:27 +0000)]
repo: Add simple API operations
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 15:27:16 +0000 (15:27 +0000)]
repos: Don't remove builds that have been removed before
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 14:11:39 +0000 (14:11 +0000)]
jobs: Reset jobs as successful if following jobs succeed
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 14:08:57 +0000 (14:08 +0000)]
jobs: Show logs of previous attempts
Fixes: #13249 - jobs: Add option to download logs from previous attempts
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 10:05:02 +0000 (10:05 +0000)]
builds: Immediately launch builds in the transaction
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 09:53:52 +0000 (09:53 +0000)]
packages: Read archive metadata in executor thread
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 09:53:30 +0000 (09:53 +0000)]
packages: Fail more gracefully if a package has been deleted from disk
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 09:52:28 +0000 (09:52 +0000)]
jobs: Re-add accidentially removed import of the queue module
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 09:51:10 +0000 (09:51 +0000)]
database: Don't keep so many connections open and idle
Since we usually peak briefly, we will fork a lot of PostgreSQL
processes that we will never need again. In order to avoid wasting too
much memory, we close those connections quicker, but at least 8 open so
that new tasks will quickly be able to acquire a database connection.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 09:50:51 +0000 (09:50 +0000)]
builds+jobs: Use new methods to access database objects
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 18 Aug 2023 09:50:07 +0000 (09:50 +0000)]
jobs: Show message if no jobs are queued
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 17 Aug 2023 11:06:43 +0000 (11:06 +0000)]
packages: Log deleting packages
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 17 Aug 2023 11:05:18 +0000 (11:05 +0000)]
builds: Try to delete packages if creating build fails
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 17 Aug 2023 11:05:03 +0000 (11:05 +0000)]
packages: Don't try to overwrite existing packages
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 17 Aug 2023 11:02:09 +0000 (11:02 +0000)]
jobs: Remove some unnecessary imports
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 17 Aug 2023 10:17:34 +0000 (10:17 +0000)]
packages: Extract filelists in a separate thread
Since we need to read the entire archive to extract the entire filelist
with all its meta information, we need to move this into a separate
thread.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 17 Aug 2023 10:16:54 +0000 (10:16 +0000)]
jobs: Tail logs more efficiently
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 17 Aug 2023 09:48:50 +0000 (09:48 +0000)]
sources: Automatically launch builds after dist
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 11 Aug 2023 14:49:58 +0000 (14:49 +0000)]
cache: Tidy up code
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 11 Aug 2023 14:45:56 +0000 (14:45 +0000)]
web: Create a simple rate limiter
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 5 Aug 2023 10:11:43 +0000 (10:11 +0000)]
release monitoring: Do not try to update a bug that has never been created
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 5 Aug 2023 10:11:09 +0000 (10:11 +0000)]
cache: Implement an async redis cache driver
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 2 Aug 2023 17:17:36 +0000 (17:17 +0000)]
jobs: Launch installcheck in transactions and wait for it
It can happen that a transaction is being rolled back and an
installcheck has been queued which will then run into errors.
Since the installcheck is now fast enough, we can simply wait for it
with the exception of relaunching any pending jobs in repositories which
will be run as one single task in the background.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 2 Aug 2023 17:02:19 +0000 (17:02 +0000)]
monitoring: Put the rate limiter in the correct place
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 2 Aug 2023 17:01:06 +0000 (17:01 +0000)]
monitoring: Complete configuration before running the first check
Because we are communicating with Bugzilla (potentially) we might undo a
lot of stuff if there is any error.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 2 Aug 2023 15:36:26 +0000 (15:36 +0000)]
config: Pass configuration as file object
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 17:18:19 +0000 (17:18 +0000)]
config: Make Pakfire an async context manager
This should help us to block less when initializing a Pakfire instance.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 17:17:58 +0000 (17:17 +0000)]
users: Show controls to create repos when none exist
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 16:58:59 +0000 (16:58 +0000)]
release monitoring: Use new version compare function
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 16:47:29 +0000 (16:47 +0000)]
backend: Use aiofiles for asyncio handling of temporary files
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 16:31:07 +0000 (16:31 +0000)]
index: Show recently finished jobs instead of pending ones
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 16:23:09 +0000 (16:23 +0000)]
builds: Move job queue
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 16:16:37 +0000 (16:16 +0000)]
jobs: Add option to only show failed builds
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 16:03:13 +0000 (16:03 +0000)]
jobs: queue: Show build time for finished jobs
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 15:52:39 +0000 (15:52 +0000)]
jobs: queue: Show status for failed/finished jobs
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 15:50:06 +0000 (15:50 +0000)]
jobs: Group by date
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 15:47:38 +0000 (15:47 +0000)]
jobs: Add pagination to new page
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Aug 2023 15:45:03 +0000 (15:45 +0000)]
jobs: Create new page that lists all finished jobs
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 31 Jul 2023 17:19:48 +0000 (17:19 +0000)]
web: Create UI module for queued jobs
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 28 Jul 2023 09:24:46 +0000 (09:24 +0000)]
jobs: Fix indefinite loop in dispatch function
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 25 Jul 2023 15:05:46 +0000 (15:05 +0000)]
builds: Fix SQL query to fetch all release/scratch builds
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 25 Jul 2023 14:15:50 +0000 (14:15 +0000)]
logstreams: Increase the maximum number of log messages stored
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 25 Jul 2023 14:15:08 +0000 (14:15 +0000)]
logstream: Lock buffer when it is being modified
This might be cheaper than copying it when it becomes large.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 25 Jul 2023 14:12:18 +0000 (14:12 +0000)]
Revert "logstreams: Don't wait until all messages have been delivered to all consumers"
This reverts commit
1fa487c5455f3e4a63ae0755c466c2a7ee62b4e5 .
This might potentially shuffle the messages :(
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 25 Jul 2023 14:10:02 +0000 (14:10 +0000)]
repos: Run fewer installchecks
Only run this for jobs that have previously failed.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 25 Jul 2023 14:08:57 +0000 (14:08 +0000)]
jobs: Perform installcheck before a job is being dispatched
I would like to perform fewer of the installchecks because they unlikely
to fail and rather expensive to run.
This is a safety net so that we will never dispatch any jobs that cannot
be built.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 25 Jul 2023 14:02:52 +0000 (14:02 +0000)]
logstreams: Copy buffer before sending it off
collections.deque() does not allow us to append new messages while we
are reading from it.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 25 Jul 2023 14:01:28 +0000 (14:01 +0000)]
logstreams: Don't wait until all messages have been delivered to all consumers
This will just queue tasks for all new messages and allows us to get
back to processing the next message quicker.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 18:17:05 +0000 (18:17 +0000)]
builders: Return zero for total build time if there are no builds
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 18:16:53 +0000 (18:16 +0000)]
builders: Include disabled builders in autoscale
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 17:42:12 +0000 (17:42 +0000)]
repositories: Master repositories as async as possible
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 17:21:25 +0000 (17:21 +0000)]
robots.txt: Disallow download files from packages
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 15:17:11 +0000 (15:17 +0000)]
web: make_url: Filter out any values that are None
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 15:14:39 +0000 (15:14 +0000)]
builds: Commit forgotten bug templates
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 15:12:54 +0000 (15:12 +0000)]
packages: Use the more efficient version comparison function
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 15:12:43 +0000 (15:12 +0000)]
database: Close connections sooner
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 15:12:16 +0000 (15:12 +0000)]
repos: Move database operations into the main thread
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 15:11:52 +0000 (15:11 +0000)]
database: Replace more occurences of the UUID type
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 13:31:31 +0000 (13:31 +0000)]
database: Move all sorts of database operations into the main thread
We no longer support any database operations from any other thread than
the main thread (because otherwise we cannot figure out which Task we
are running in).
For that reason, we have to split a couple of tasks off at different
times and send blocking operations to an executor thread later.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 13:30:12 +0000 (13:30 +0000)]
database: psycopg3 is returning UUIDs as UUID type instead of string
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 10:22:44 +0000 (10:22 +0000)]
release monitoring: Fix cleaning up the repository if none was created
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 10:16:51 +0000 (10:16 +0000)]
database: Replace the old database model with a connection pool
This patch introduces a connection pool to the database. When a function
is executing a query, a connection will be handed to the (asyncio) Task
and remains with that task until its end.
That way, multiple concurrent tasks (even when they are running in the
same thread) will have an exclusive connection to the database which
will avoid that we overwrite and commit transactions of other tasks.
Additionally, transactions can now be nested as psycopg's transactions
are more advanced than the previous Transaction class.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 09:54:07 +0000 (09:54 +0000)]
backend: Move temporary tables into CTEs
It was a nice idea, but if we want to pool connections, we will only
create the temporary tables on one connection with the others losing
out.
An alternative could have been to create all tables on all connections
first, but I suppose that is going to create a lot of unnecessary
overhead for short-lived connections.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 21 Jul 2023 09:52:35 +0000 (09:52 +0000)]
web: jobs: Add HTTP status code to exception
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 20 Jul 2023 13:31:32 +0000 (13:31 +0000)]
database: Remove aliases for common exceptions
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 20 Jul 2023 10:44:34 +0000 (10:44 +0000)]
database: Migrate to psycopg3
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 20 Jul 2023 10:16:55 +0000 (10:16 +0000)]
builds: Show better tags
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 20 Jul 2023 10:16:12 +0000 (10:16 +0000)]
CSS: Don't use our custom red as it clashes with the primary colour
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 20 Jul 2023 10:15:34 +0000 (10:15 +0000)]
sources: Fix rendering page when source has never been fetched
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 20 Jul 2023 10:15:05 +0000 (10:15 +0000)]
sources: Create dist jobs for all packages on the initial commit import
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>