Bump version to 0.9.4

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Purge any redundant entries

When importing inetnums, we might import various small networks
which are not relevant for us as long as they do not have a
different country code than their parent network.

Therefore we delete all these entries to keep the database
smaller without losing any information. The second version of this
patch introduces a SQL statement parallelised across all CPUs
available, while the DELETE-statement of the first version literally
took ages to complete.

However, cleaning up those data still takes about 26 hours (!) on
our location02 testing machine, making daily updates of the location
database impossible to the current knowledge.

real    1521m30.620s
user    38m45.521s
sys     9m6.027s

Special thanks goes to Michael for spending numerous hours
on this, setting up a testing environment, doing PostgreSQL magic
and providing helpful advice while debugging.

Partially fixes: #12458

Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Import raw sources for inetnum's again

The extended feeds do not have enough detailed information
for us, so that we need to import inetnums from RIRs where
possible. Filtering private networks is necessary as RIR data
may contain 0.0.0.0/0 or similar entries for administrative
purposes or due to misfilings.

Special thanks goes to Michael for spending numerous hours
on this, setting up a testing environment and providing helpful
advice while debugging.

Partially fixes: #12458

Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: avoid violating NOT NULL constraints during JOIN

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Determine family when the network is initialised

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Export family, first_address and last_address in Python

We can use this to avoid formatting the network in a string
representation which is then parsed again later to determine
the first and last addresses.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Store the first and last address of each network

This avoids recomputing the last address multiple times.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Rename start_address to first_address

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Skip writing any subnets

It is unnecessary to write any subnets in the output, because
they might already be included in a bigger parent network.

Since the tree is ordered, we only need to check if the network
to be written is a subnet of the previous one. If so, we skip it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Fix message sting if the database has been recently updated

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add test for get_continent_code() function.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add test for get_country_name() function.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python example script: Add country name and continent code.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add get_country_name() function.

This function can be used to get the stored name for a given country
code.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add test for the lookup_network_has_flag() function

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add example network flag to the python create-database example script.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add lookup_network_has_flag() function.

This function can be used to check if a given address or network has one
of the following network flags.

* LOC_NETWORK_FLAG_ANONYMOUS_PROXY
* LOC_NETWORK_FLAG_SATELLITE_PROVIDER
* LOC_NETWORK_FLAG_ANYCAST

It will return true if the given flag is set.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add get_as_name()

This function can be use to give the numer of an Autonomous System and
get back the stored name from the used location database.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add test for the database_countries() function.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add an example country to the python create-database example script.

The generated database is used by the testsuite of the perl binding.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Add database_countries() function.

This function is used to the stored countries of a database, which
easily can be assigned to an array.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Update translations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Provide a return code if the database does not need to be updated.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Read all routes and not only those imported through iBGP

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Implement fetch routes from Bird

If the route server is now a socket, the importer will connect
to it and fetch the routing table. This is a lot faster than telnet.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country: Terminate buffer when reading from database

Compilers on ARM do not seem to initialise the buffer.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.3

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location(8): Remove any temporary files after unsuccessful downloads

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>

location(8): Fix pasing timestamp for systems not running on UTC

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>

location(8): Allow downloading a database without one on disk

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location(8): Allow limiting updates to once a day, week, month

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Correctly sort the results

The results where incorrectly ordered so that the DISTINCT clause
removed the wrong rows matching an incorrect parent subnet.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Also add networks that are not announced to the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Write NULL into database when bool is not set

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Walk through all options for flags

It could happen that a network was marked, but the AS was not.

Since the AS is checked first, we won't check the network and
therefore write the wrong information into the database.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Always require SSL

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location.in: strip trailing whitespaces in database description while dumping

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.2

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Enable debug logging for all tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Update home and contact address

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

man: Update documentation on renamed --output-format parameter

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

man: Document export command

Fixes: #12437
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

man: Order commands alphabetically on location(8)

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Fix handling of families argument

This was ignored, because it was already converted before
and the export function was expecting a string.

Fixes: #12436
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix timezone-dependant interpretation of database timestamp

The timestamp that is retrieved from DNS is in UTC and we must
not interpret as local time.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Correctly set and use __version__

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Also build for sid

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Print ASes in format like: AS204867 - Lightning Wire Labs GmbH

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Disable perl package.

The perl package cannot be cross-compiled correctly for other
architectures different from the hosts one.

So currently disable the perl package until we get this fixed at a later
time.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Return the last block, too

Fixes: #12427
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Create separate package for the importer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: build.sh: Stop printing every command that is executed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Do not destroy the build result

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: unstable is called bullseye

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Fix broken search for networks with flags

The search was ended after the first network. No matter if
it matched, or not.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export flagged networks with their faked country names, too

This will lead to some networks showing up twice. Once with
their real country and once with their faked one.

It is likely that the first one will match.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location(8): Export all countries by default

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add option to iterate over all countries and print them to the console

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Remove accidentially committed hacks for Debian

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Mark all packages as "Multi-Arch: same"

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Correctly define runtime dependencies for all packages

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Rename main package to "location"

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Update rootfiles

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Import translation strings

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Merge location-exporter(8) into location(8)

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Rename location-query(8) to location(8)

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

downloader: Change user-agent to location

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

systemd: Rename location-downloader to location-update

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-downloader: Merge man page into location-query

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move location-downloader functionality into location-query

The commands are very long and confusion. Hence we merge this
all into one command.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-exporter: Warn, but do not fail on invalid input

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-exporter: Do not mistake country AS for an AS number

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-query: Require at least one flag

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Remove Python path overrides for Debian

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-downloader: Do not change content of open database files

The database might be opened by another process. When modified,
it will return random results.

Fixes: #12420
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

test: Generate random temporary files

Debian runs the testsuite in parallel which fails when mutliple
process write to the same file at the same time.

This patch will now create and later delete a temporary file.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Bump version to 0.9.1

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "debian: Drop perl package"

This reverts commit 3d26288681601b273f034de1f2241dc4342791b5.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Enable location-download.timer after installation

This timer periodly executes the location-downloader and keeps
the location database up to date.

Fixes #12412.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Ship database signing key.

This key is needed to download and verify the database.

Fixes #12410.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

examples/create-database.py: Add hack to allow loading the module in build environment.

This script is needed to generate the database which is used by the
testsuite of the perl binding of libloc.

This allows us to revert the following commits:

* 482eb94f165507ffcb6503e162535064d6dcecac
* 3d26288681601b273f034de1f2241dc4342791b5

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Do not add auto-generated files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

man: Document quiet mode switch

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Print country name if possible

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Implement importing/exporting countries

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-query: Fix typo in "aut-num"

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

stringpool: Always add an empty string

This will create an empty string at the first byte of the pool
and is helpful for our database metadata (vendor, license, de-
scription) which can be zero and would therefore return the
first string in the pool.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

format: Reduce the length of the signature header

Since a single signature can only be up to 2048 bytes long,
we do not need 32 bits to tell us how many of those bytes
are being used.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

test: Sign database twice with the same key

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Support passing two signing keys

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Add support for two signatures

This allows us to sign the database with two different keys
in case the first key gets weakened or compromised in any other
way.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Build a source package, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Remove code that would have built a native package

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Build each build in their own directory

Otherwise we will have all packages in one large directory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Build for unstable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Do not build for riscv64

The build requirements could not be satisfied

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Setup build environment only once per release

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Drop perl package

This has runtime requirements and won't build when there is
no functional Python module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Add script to build package for various arches

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Fix rootfiles

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>