exporters: Don't fail if exporting for the first time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dbl: Allow to for optimization

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Add link to the allocation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Configure allocated SIDs

  https://github.com/sidallocation/sidallocation.org/issues/37
  https://github.com/sidallocation/sidallocation.org/commit/0e1c905f5e8eb8cc0d68aa91cc03783077c44efa

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Allow trusted users to accept their own reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Treat all emails as auto-generated

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Don't encourage people to reply to refusal emails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Automatically close any reports from trusted reporters

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Create a group for trusted reporters

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Minor code cleanup

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Improve wording in emails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Include the domain name in subject on report emails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Improve wording on opening email

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Don't show own username on duplicate reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Allow to disable the email notification on reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Include the URL to the report in the API response

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Import util

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Fix the domains query endpoint

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

ratelimiter: Don't send headers if the ratelimiter has been disabled

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

ratelimiter: Use the correct variable to fetch the API key

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Add ratelimiting to the searches

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

ratelimiter: Change limit for the entire class after auth

We are accessing the limit variables outside of this function, so we
will have to reset limit. This is slightly annoying, but we cannot
perform any authentication checks in __init__().

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

ratelimiter: Purge any outdated data from the ratelimiter table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

ratelimiter: Allow to configure different limits for authenticated users

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Allow to mark some API keys to never be ratelimited

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

ratelimiter: Use the API key as bucket for authenticated users

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Implement a basic rate limiter

This code is borrowed from Pakfire.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Don't store the amount of pending reports any more

This is not very reliable and we don't show the number anyways

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Validate the name when creating reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Add a middleware that ads timing information about the request

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Generate a RSS feed with the latest reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Fix fetching closed reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Don't try to search for a list if none exists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Fix incorrect excpetion class name

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Make User objects comparable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Remove the deprecated calls to fetch reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Don't accept any repeat reports by the same user

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Create the reports search endpoint

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Actually check the API keys

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Return reports as an iterator

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Allow searching for reports by a certain reporter

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Send 409 if someone is trying to close a closed report

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Allow to post a comment when closing the report

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Implement some basic permissions check

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Add a handler to post comments to reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Add a simple check function if a report has been closed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Fix name of database field

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Log when API keys have been used last

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Require authentication to close reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Use type annotations for users

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Rename the authentication function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Deprecate the old reports endpoints in the lists handler

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Create a new endpoint to submit reports

This makes the API slightly cleaner and moves all report things
together.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: auth: The impersonation is tied to the key and not the user

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Use the authenticated user to create reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Check if the API key user actually exists, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Whenever a list is updated, we refresh the timestamp

Further down in the code, we will compare timestamps which have not been
manifested in the database, yet, and are therefore unknown.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Fail if we are trying to import some HTML

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Implement API authentication for users

This patch also implements that we can impersonate users so that the
webapp does not require an API key for each single user.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

auth: Add UID and permission to impersonate to API keys

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Create a separate table for comments

That way, we may have multiple comments on a report.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Move /search into its own section

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Don't list the index redirection as an endpoint

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Manifest timestamps before optimizing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Don't pull dead domains when optimizing

Since we have millions of dead domains, we must be more efficient here
and not pull them since we already know that they will completely
delisted.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Don't optimize the list if not necessary

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Optimize the list before exporting it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Store the timestamp when we optimized the list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Remove dead domains from the exports again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Check if an export actually needs an update

This allows us to export only if we actually have any changes which will
take a lot of load away from the server.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Implement showing false-positives

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Make them sortable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Only export blocked domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Perform whitelist check only if anything is actually whitelisted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Make it possible to manually check a domain again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Mark domains as listed for faster search

The query that is determining which domains are whitelisted has always
been very slow and there is no feasible way to accellerate it using
indexes, etc.

Therefore we will download all whitelisted domains and all potentially
blockable domains and perform the check in the Python application. That
way, we can later mark any delisted domains and fetch the entire list of
domains reasonably fast.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

domains: Add a field to mark a domain as listed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Ignore if the server suddenly closes the connection

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Redirect to the API documentation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Convert modification timestamps to UTC

The asyncpg driver does not seem to like offset-aware and offset-unaware
timestamps.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Update domains in batches of 100

The asyncpg driver does not support an infinite amount of arguments in a
single query.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Actually update all sources

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Fix running in async mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Fix updating sources

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Make the entire application async

This was needed because we seem to have some issues with the API
starting the next request or trying to access the database after the
session has been closed. Instead we will now have a single database
session per task which can be managed easier and will only be closed
after the entire task has completed.

As another benefit, we are now able to run many requests simultaneously.
So far this has not been a big bottleneck, but some operations (like
closing a report) can take a moment and would therefore have been
blocking other requests.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move the configuration file to /etc/dbl/dbl.conf

That way, we can have some companion files with credentials, etc.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Deliver emails over SMTP instead of piping them into sendmail

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Move our Suricata SID range

This is not perfectly unique, but it should be hopefully okay.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Abort if we are importing our own canary

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exports: Add a canary domain to all exports

This is so that we can identify when we are reading back our own data.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Don't remember what got notified

It is more useful to always receive an email with all open reports.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Obfuscate names when sending them over email

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Gracefully skip if a source has given us an empty file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Fix wrong database field when fetching multiple reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Implement closing reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Fix fetching open/closed reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Fix syntax error in Suricata rules

Reported-by: Matthias Fischer <matthias.fischer@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Allow to search for reports for a specific name

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Don't perform search if the query is not a valid hostname

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: domains: Revert the domain dump to something simple again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>