sources: Fix listing all sources

I have no idea how I confused all these field names... Oh well...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add flag to force updating all sources of all lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Show if a domain is dead

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Store number of total and dead domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Normalize any international domain names

We don't want these to hit the database in Unicode, but only in ASCII
format.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Remove any trailing dots from domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Create a CTE to fetch all domains

This is reusable so that we don't have to copy any queries.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Remove any domains that could not be resolved

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

checker: Check if domains exist by querying their SOA

This way, we should be able to massively reduce the blacklists because
so many domains actually no longer exist.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Don't hard-fail when we cannot read the server's response

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Add some ignored domains

These will never be added to any list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: analyze: Don't divide by zero when a source is empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Support other prefixes of the host format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Don't try to strip empty domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exports: List the total number of entries on the list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Strip any whitespace from domains

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Silently skip any IP addresses or URLs on import

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Parse plain lists that include comments

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Don't consider domains with a hyphen or underscore as invalid

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Implement forcing a list update

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Use Babel to format the number of domains properly

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Show update time rather than creation time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Add the correct directory to the squidGuard tarball

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Block all sub-domains, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Use an appropriate prefix for the zones

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Drop exporting zones using dnspython

dnspython is taking hours as well as several gigabytes of memory to
generate a zone file. This is not sustainable at all since we want to be
able to push out updates fast.

I could not find any alternatives, but we are able to generate ZONEMD
records externally.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Don't accept any IP addresses being listed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Fix creating lists by initializing updated_at

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Create expressions and urls files for squidguard

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Refactor is_fqdn

No functional changes.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Be more lax with domain names

There are a lot of domain names out there that start/end with a
non-alphanumerical character which is why we need to remove this check.

We also need to accept underscores in domain names.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Remove leading *.

Some lists store domains with a leading *. which we can simply remove.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Insert domains in batches

Since too many database roundtrips are making the parsing of other large
lists extremely slow, we will collect them all in a set (so that we will
never have any duplicates) and insert them into the database in batches
of 1000 domains at a time.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Support transparent decompression of compressed files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Fix sending the Etag

Obviously this must be sent in an If-None-Match header.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporter: Create the RRset only once

Since all records have the same rrset, we simply create it only once
instead of doing this every time we iterate over our loop.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Create a stopwatch to monitor how long some expensive functions take

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add an analyze command to show duplicates in lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Don't rely on the Adblock Plus header

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Log an error if we could not parse anything

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Support parsing hosts files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Show the number of domains on lists/sources

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: export-all: Create a tarball with all lists for squidGuard

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Initialize updated_at with the current timestamp

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Support exporting lists that have never been updated

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add description to lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Add an exporter for squidGuard

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporters: Split logic for text and binary exporters

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Download the entire payload before processing it

Some web servers did not keep the connection open for long enough so
that we cannot keep processing all domains on the fly.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

exporter: Build an exporter for a regular DNS BL zone

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Drop the PowerDNS views

Although this worked really well, we cannot create ZONEMD records and
catalog zones and anything else that needs to store data is becoming
very difficult to manage.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Allow to specify an output file for single exports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add ZONEMD records to the exported zones

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Move the exporters into a separate file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Use "dns" to write the zone file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Don't export deleted lists/sources to DNS

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Support output in hosts file format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Store and show when sources have been updated last

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Show the last update time in the header

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Create a sub-directory for each exported list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Allow blocking an entire TLD

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a search command

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Create an index to remove old domains quicker

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Add format detecting and parse the Adblock Plus format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Don't delete domains that have already been deleted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add command to remove a source

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a header to all exports

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lists: Fetch sources ordered by their name

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Make the list only as updated when we actually have new data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Set the mtime of the file to when the list was last updated

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Allow handling binary file descriptors on export

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add command to export all lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Support exporting RPZ zone files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Allow to pass the output format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add --quiet/-q flag to be less verbose

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a command to update all lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a command to delete a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Make "notified_serial" writable

That way, we can run PowerDNS in master mode and it will notify any
slaves if there are any changes.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Set the correct master namesever

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Add NS records to the zones

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Create some experimental VIEWs for PowerDNS

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Only download when upstream has actually changed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

sources: Don't prune domains from other sources

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Remove the "list-" prefix from all commands

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Blacklist -> Blocklist

Let's make this politically correct.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a simply way to download plaintext lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Require httpx

To download stuff.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Add a helper function to check if something is a valid FQDN

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add domains to sources and add export command

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: sources: Use the new db methods to fetch sources

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Create a command to update a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Make the backend available to all objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a command to show all lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a command to show information about a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add our own database abstraction

Otherwise we would have to carry session objects around which makes the
code incredibly messy.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Align the naming of the "list-create" command

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a command to add a source to a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

dnsbl: Add a command to create a new list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

db: Create a simple database schema for lists and sources

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Add check for Python modules

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>