export: Remove old flattening feature

The database enumerator now only returns networks that will
never overlap.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add property to return IP addresses as bytes

This avoids calling inet_pton to parse IP addresses from string

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix download of database

This was all messed up in 0c74f6b1a3bdce5ebdc2ee452b9baf3e421dd3d1
when the change of type for the timestamp wasn't changed everywhere
else.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: End lookup after an invalid IP address was passed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Remove unnecessary db object from writers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Remove filtering for flags

The filter is an AND filter and if we set the flags from
the special country codes, we won't get back much.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Add debug output to filtering

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Free filter lists in enumerator

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Do not clean up python list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Actually clear all lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Change back to use Network objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

as: Make lists grow dynamically

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Make list grow dynamically

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

countries: Make list grow dynamically

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Simplify AS matching code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Simplify network matching code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

as: Add list for easier processing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Filter flags in C

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Only return country codes we want

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Implement filtering for multiple countries in the enumerator

This will allow us to speed up the export of the database
if only a few countries should be returned.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move network lists into an own file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Move tree flattening into C

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Unexport all tree functions

These should not be exported

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add function to pop first element from stack

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Sort result of excluded lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Filter results coming from stack

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Move network filtering into a separate function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Copy all attributes when splitting networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

test: Update API

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Add option to return networks flattened

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add function to exclude multiple networks at once

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add new subnet function

The old one is too difficult to use in terms of order
of input parameters and return value.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add more debugging output to stacks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Export networks exclude function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Reduce debugging output

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Pass flag to enumerator to flatten output

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Make lists unique

Networks that are in the list won't be added again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add function to merge two lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add function to check if network is part of a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add function to check if two networks overlap

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add functions to break network into subnets

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add function to dump lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add list to manage groups of networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

test: Add tests for database enumerator

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: always convert organisation handles into upper cases

Fixes: #12523
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location.in: do not confuse UTC with local time zones

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>

location update: Remove double conversion of timestamps

This caused that the timestamp in DNS was misinterpreted
as local time and often, databases could not be downloaded.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>

location: Fix Python syntax error in verify()

The database is now being opened before the requested
method is called and handle_verify() wasn't updated.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Flatten the tree before exporting it

This patch removes the possibility that any IP address ranges
might show up in multiple exported files.

If this was content from the database:

  * 10.0.0.0/16 - DE
  * 10.0.1.0/24 - FR

Then the IP address 10.0.1.1 would match for both countries.

The algorithm will now break the larger /16 subnet apart into
smaller subnets so that 10.0.1.0/24 is no longer overlapped.

There was some time spent on this to make this as efficient
as possible.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: avoid log spam for too small networks

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: omit historic/orphaned RIR data

Some RIRs include detailled information regarding networks not managed
by or allocated to themselves, particually APNIC. We need to filter
those networks (they usually have a characteristic network name) in
order to prevent operational quirks or returning wrong country codes.

Fixes: #12501
Partially fixes: #12499

Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer.py: fetch LACNIC data via HTTPS

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: filter bogus IP networks for both Whois and extended sources

Sanity checks for parsed networks have been put into a separate function
to avoid boilerplate code for extended sources. This makes the location
database less vulnerable to garbage written into RIR databases on
purpose or by chance.

Fixes: #12500
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export.py: fix exporting IP networks for crappy xt_geoip module

In contrast to the location database itself, the xt_geoip module
consumes a list of IP networks for each country, and returns after the
first match.

We therefore need to...

(a) sort IP networks by their size, allow as precise matches as possible
(b) export _any_ IP networks - including inverted subnets - to prevent
    undefined overlaps
(c) do the entire thing as fast as possible, consuming as less disk
    space as possible, which is why we can't just iterate over all /24
    chunks

Partially fixes: #12499

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "Revert "location-importer.in: only import relevant data from AFRINIC, APNIC and RIPE""

This reverts commit 13f67f285856e8eabfeff2daf1be3aeaa36a82cc.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "Revert "Revert "Revert "importer: Import raw sources for inetnum's again""""

This reverts commit 44341478233115b26bb27fdb24da5b0a1eedb173.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Allow adding single IP addresses and automatically add the prefix

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Restructure SQL query to be executed in parallel

There are no functional changes, this just runs quicker now.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add search index to network_overrides table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add search index to announcements table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Remove accidentially committed debug line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Test if we can add localhost (IPv6)

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Improve parsing IP addresses

loc_network_new_from_string() seem to have had some unexpected
behaviour for invalid inputs.

The function has been tidied up slightly and returns as soon as
some invalid input was detected.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Try adding an invalid network

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Free countries when the writer is being destroyed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Free array with pointer to ASes, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

as: Do not attempt to match name when it wasn't set

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

as: Fix dereferencing NULL pointer when setting AS name

Reported-by: Gisle Vanem <gisle.vanem@gmail.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "Revert "Revert "importer: Import raw sources for inetnum's again"""

This reverts commit f532841e9197ce2f40aad8c086d786b2cb783a54.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "location-importer.in: only import relevant data from AFRINIC, APNIC and RIPE"

This reverts commit a36bc686865fc87ea386fd90b389338bdcb80cbc.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Include all overridden networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: only import relevant data from AFRINIC, APNIC and RIPE

In contrast to ARIN and LACNIC, we are able to process more detailled
feeds from those RIRs, avoiding storage of obviously unnecessary data.

Thanks to various SQL optimisations, doing so now takes less time than
the first version of this did.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "Revert "importer: Import raw sources for inetnum's again""

This reverts commit 92f6abf4e272672bb0a71cfe991261b95ebe2fef.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "importer: Import raw sources for inetnum's again"

This reverts commit 64e95fa903edec8b4e4e59830b395e2e4a411853.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "importer: Purge any redundant entries"

This reverts commit c2cc55d5a6875c3838f060032eaed89dcfb92ef6.

The query stalls the database and therefore the automatic
scripts are no longer able to generate a new version of the
database.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Bump version to 0.9.4

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Purge any redundant entries

When importing inetnums, we might import various small networks
which are not relevant for us as long as they do not have a
different country code than their parent network.

Therefore we delete all these entries to keep the database
smaller without losing any information. The second version of this
patch introduces a SQL statement parallelised across all CPUs
available, while the DELETE-statement of the first version literally
took ages to complete.

However, cleaning up those data still takes about 26 hours (!) on
our location02 testing machine, making daily updates of the location
database impossible to the current knowledge.

real    1521m30.620s
user    38m45.521s
sys     9m6.027s

Special thanks goes to Michael for spending numerous hours
on this, setting up a testing environment, doing PostgreSQL magic
and providing helpful advice while debugging.

Partially fixes: #12458

Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Import raw sources for inetnum's again

The extended feeds do not have enough detailed information
for us, so that we need to import inetnums from RIRs where
possible. Filtering private networks is necessary as RIR data
may contain 0.0.0.0/0 or similar entries for administrative
purposes or due to misfilings.

Special thanks goes to Michael for spending numerous hours
on this, setting up a testing environment and providing helpful
advice while debugging.

Partially fixes: #12458

Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: avoid violating NOT NULL constraints during JOIN

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Determine family when the network is initialised

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Export family, first_address and last_address in Python

We can use this to avoid formatting the network in a string
representation which is then parsed again later to determine
the first and last addresses.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Store the first and last address of each network

This avoids recomputing the last address multiple times.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Rename start_address to first_address

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Skip writing any subnets

It is unnecessary to write any subnets in the output, because
they might already be included in a bigger parent network.

Since the tree is ordered, we only need to check if the network
to be written is a subnet of the previous one. If so, we skip it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Fix message sting if the database has been recently updated

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add test for get_continent_code() function.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add test for get_country_name() function.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python example script: Add country name and continent code.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add get_country_name() function.

This function can be used to get the stored name for a given country
code.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add test for the lookup_network_has_flag() function

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add example network flag to the python create-database example script.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add lookup_network_has_flag() function.

This function can be used to check if a given address or network has one
of the following network flags.

* LOC_NETWORK_FLAG_ANONYMOUS_PROXY
* LOC_NETWORK_FLAG_SATELLITE_PROVIDER
* LOC_NETWORK_FLAG_ANYCAST

It will return true if the given flag is set.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add get_as_name()

This function can be use to give the numer of an Autonomous System and
get back the stored name from the used location database.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add test for the database_countries() function.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add an example country to the python create-database example script.

The generated database is used by the testsuite of the perl binding.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add database_countries() function.

This function is used to the stored countries of a database, which
easily can be assigned to an array.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update translations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Provide a return code if the database does not need to be updated.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Read all routes and not only those imported through iBGP

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Implement fetch routes from Bird

If the route server is now a socket, the importer will connect
to it and fetch the routing table. This is a lot faster than telnet.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>