Rename the whole project to "IPFire DBL"

DNS is not strictly true since we are using the domain lists in other
places now, too. Therefore I prefer the more generic term.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

domains: Prevent SQLModel from throwing away my rows

The ID is not necessarily unique, so we have to add more attributes to
the primary key of the VIEW.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Show any removals of domains

This was because of an invalid SQL comparison.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

systemd: Add timers to send notifications

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Add a middleware to correctly close database sessions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Send an email to the reporter when a report is being opened

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Send an email after a report has been reviewed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Send am email to moderators if there are pending reports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

users: Add a simple system to access users and groups

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Store stats about subsumed lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Optimize the lists

This is a quick solution to find any listed subdomains. Those don't have
to be exported if we already have the parent domain listed. This will
decrease the size of the lists.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Remove the replaced Suricata exporters

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Move check results into the main domains table

This will save us some time when composing the final lists as we are
getting rid of a join. As we are storing one row for each domain in each
table it makes sense to store this all together.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Be more efficient when fetching domains

We used to fetch domains that need to be checked in batches, but that
causes a lot of database load and does not keep the resolver busy.

This patch changes this so that we once fetch a large iterator from the
database which we will iteratively feed into the pool.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Remove some debugging code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Don't follow CNAMEs any more

If we get a CNAME response, the domain exists. Maybe the target does
not, but that could change at any time.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Accept empty responses

When we are querying for something.example.org, we will get the SOA of
example.org. dnspython still considers this a NoAnswer, but actually
this is good enough to tell us that the domain is alive.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Log the result of any queries

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Allow passing some domains for manual checking

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

domains: Realise the history as a view

This avoids creating yet another complicated query and we can have a
model so that accessing attributes becomes easier.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Create an index to make domain history searches faster

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

API: Add an endpoint to check the history of a domain on a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

API: Add a simple endpoint to check where a domain is being listed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Add a systemd unit file for the API service

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Add a priority which is exported to Suricata

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Generate unique SIDs for all Suricata rules

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Compose Suricata ruleset from a dict

This makes it slightly easier to swap out some fields where needed than
manipulating a really large string.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Refactor Suricata rules

Exporting everything as a single rule is completely blowing up Suricata.
It was always going to be a tight game, but a lot of the tooling even
breaks down due to the large size of the rule files.

This is my less preferred option because it isn't easily possible for
users to enable/disable any individual domains, but at least this
performs well.

There are now only single rules for each list that enable/disable
filtering for the different protocols and there is one large list of
domains for each list. This is being parsed as a dataset which should be
the most efficient approach in terms of performance and memory usage.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Update stats immediately after closing a report

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Initialize all stats with zero

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Store false-positives

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Use the tarball exporter to write Suricata rules

That way, we will always compress them as they are rather large to be
downloaded as plaintext.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Create a nested directory exporter

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Fix the history output on the console

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporter: Attempt to remove code duplication in the exporters

This will also export a unified tarball of all Suricata rules.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Compute Suricata SIDs by a hash

This function is not perfect, but substantially faster when writing out
the list. We will have some somewhat-stable hash here which will always
return the same result per domain unless there has already been a small
collision.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "lists: Export domains with a global, unique ID for each domain"

This reverts commit 42a14f468c41a64c4ebdabaf49021c9f92e6626d.

This query seems to be very broken as it isn't returning the right data
any more and also is taking a very long time.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Implement exporting Suricata rules

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Export domains with a global, unique ID for each domain

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Convert all domains to lowercase

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Delete sources by URL as that is unique

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Create a convenience function to gracefully terminate the program

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Gracefully terminate if we could not find a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Store a stats history

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Store a stats history

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Update list stats after the checker has finished

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Don't fail if a query timed out

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Initialize the number of total domains when creating a new list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Strip any excess whitespace after the comments have been removed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Always return an integer for the length of the source

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Move any TXT records out of the apex

Some RPZ clients consider anything existing in the apex a wildcard match
for anything.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Perform stable sorting for sources

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Accept the third-party option for ABP

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Accept "[Adblock Plus 2.0]" as a valid header

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Refactor the ABP parser

Some sources don't implement the format very strictly. To not miss any
data, we will have to split off any options and then check whether we
want to add the domain to our lists.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Rework the hosts file parser

Some sources use tabs to split and since we don't want to have a large
list of IP addresses, we simply check if we now have a format where we
list an IP address and something else.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Consider ABP files as such without the header

Some files don't have the [Adblock Plus] header, but they use ! as a
comment, so we can identify them that way.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Parse any byte-order marks

https://en.wikipedia.org/wiki/Byte_order_mark

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Always strip any comments

So far it looks like all file formats are using # to separate any
comments. Therefore we can strip anything away very early on.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Force an update of empty sources

We assume that we could not parse any data and therefore we will just
try again.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Support customizing the DNS zones

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporter: Add some metadata to the apex

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporter: Support exporting for AdBlock Plus

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Split whitelisted domains on history

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Implement closing reports

This will trigger changes to the lists.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Don't import anything that isn't globally resolvable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Allow to go backwards in history

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Export the history

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Add command to show the history

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Add a custom SELECT method

This is necessary because SQLModel only spits out the first column of
any custom queries.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

domains: Store a report for adding and removing a domain

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

domains: Add a unique index for manually added domains

We don't want to have any duplicates, but we will have to accept any
duplicates that we are receiving from other sources.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Support whitelisting domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

domains: Reference the list directly

This simplifies the queries substantially and we will only have to
select from one single, although larger table.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

domains: Create a unified table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Account for how many open reports there

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

search: Return any parent domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

API: Implement a simple search

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Implement fetching reports over the API

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Export the list slug in the API

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Use a UUID as primary key

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Export any reports over the API

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

reports: Add a system to create a new report over the API

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a simple authentication mechanism

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: List the sources of a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Store the total number of domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Add endpoint to fetch a single list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Fix reading the configuration file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Don't expose some fields to the API

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Add a simple endpoint to fetch all lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

api: Create some basic (empty) API service

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

backend: Support passing the configuration as a path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Require the Python dns module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Fix listing all sources

I have no idea how I confused all these field names... Oh well...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add flag to force updating all sources of all lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Show if a domain is dead

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Store number of total and dead domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Normalize any international domain names

We don't want these to hit the database in Unicode, but only in ASCII
format.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Remove any trailing dots from domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Create a CTE to fetch all domains

This is reusable so that we don't have to copy any queries.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>