Add some more country codes.

* Bonaire, Sint Eustatius and Saba (BQ)
* Curacao (CW)
* South Sudan (SS)
* Sint Maarten (SX)

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

overrides: Rename newcountry to country

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

overrides: Remove all networks with an invalid subnet

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Import custom overrides

Fixes: #11755
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Set flags when parsing A1-A3

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add countries to database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Import

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update all RIRs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix downloading databases in the default format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

add networks with faked KP location

These networks claim to be located in North Korea, but are mostly used
for VPN or other anonymisation purposes. Therefore they are added to
A1 or "others" override files.

See https://blog.trendmicro.com/trendlabs-security-intelligence/a-closer-look-at-north-koreas-internet/
for details. 57.73.224.0/19 was reassigned since then and contains
valid data now, causing it to be omitted.

The second version of this patch superseds the first
one which was broken due to bugs in the MUAs GPG implementation.

Fixes #11757.

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Import for import of LACNIC

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add parser for LACNIC

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Migrate to a binary-encoded database format

This approach is very similar to the one that we used before, but instead of storing
IP addresses as two integers, they are encoded as a big-endian 128 bit long binary
"string".

This is then stored as the BLOB data type of sqlite which has fewer limitations in
length, can be indexed and uses memcmp() which should be at least as fast as comparing
integers.

This allows a slightly easier query and seems to me significantly faster than the
PostgreSQL approach.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update data for AFRINIC

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix merging the database with SQLite

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "Replace sqlite with PostgreSQL"

This reverts commit 44c781ac664bf0f6670651e0a721fe2ddc446e66.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update ARIN

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update AfriNIC

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Ignore some blocks in the source from AfriNIC

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update RIPE

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

A1 overrides: add missing PureVPN networks and clean up

GZ Systems Ltd. has some other networks which were not added to A1 overrides
by mistake. To clean up the list a little bit and make file processing faster,
some of the networks were allocated in bigger ones (>= /23).

Sorry for this large patch, but I guess it does not make much sense to add
a bunch of networks fist and then split them apart afterwards. :-|

Signed-off-by: Peter Müller
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

A1 overrides: add IAPS Security Services, L.L.C.

IAPS is a VPN provider claiming its networks to be located in VA.

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update APNIC

These networks now contain the correct subnet

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tools: Actually print the correct subnet to file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tools: Commit to database only once when importing data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Import APNIC

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Replace sqlite with PostgreSQL

sqlite cannot match any subnets which makes it useless for us.

PostgreSQL has a data type for IP addresses which can be used
to match subnets against each other.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Set database vendor

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

add networks to overrides-other which must not be added

These networks are covered for special use by some RFCs
and must not be added to the database.

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

add A2 overrides

Add override entries for A2 networks (satellite internet providers).

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

add missing A1 overrides

Add missing A1 overrides (mostly GZ Systems Ltd./PureVPN).

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

add missing A1 overrides

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Use the organisation's name as AS name

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Import data from AFRINIC, APNIC, ARIN & RIPE

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add script to export data into a location database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: Remove any inline comments

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Skip any mntner: blocks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Ensure that country codes are uppercase and fix some bugs in the ARIN database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Skip any blocks where IP addresses could not be parsed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

util: In the RIPE database, % is a comment, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move some commonly used functions to util submodule

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tools: Move filename generation into RIR class

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add parser to parse database formats that are in RIPE format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

downloader: Only return blocks with content in them

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

downloader: Skip commented lines

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

downloader: Only strip right hand side of lines

This should make stripping a bit faster

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

downloader: Seperate downloaded files into blocks and iterate over those

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add downloader which decodes files downloaded from the RIRs on-the-fly

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

add override country code to A1 overrides

Adding a "newcountry" flag for each AS (no IP ranges currently
listed here) in the A1 override file. This makes parsing of
several override files much easier, since we do not need to
care about file names, for example.

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

A1 overrides: Sort them by ASN

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

A1 override: add AS396507 (Emerald Onion)

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

A1 override: add AS51852 (Private Layer INC)

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

A1 override: add AS10914 (Cypherpunk Privacy Network)

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

A1 override: add AS37560 (Cyberdyne S.A.)

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country override files: inital checkin

Initially create manual override files for country data.

Signed-off-by: Peter Müller <peter.mueller@link38.eu>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>