network-tree: Run deduplication once for each family

This probably never becomes a real problem, but we did not clear the
stack properly once we switched family when iterating through the
entire tree in one go.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tools: Import the copy script

This is useful to debug the library by copying one library 1:1 and then
diffing the result.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-tree: Split into a separate file

There are no functional changes here, but I find this code easier to
handle if it is split across smaller files.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Fix Spamhaus ASN-DROP parsing

The format of this list has changed, from a plain text file with a
customer schema to JSON. Adjust our routines accordingly to make use of
this list again.

The second version of this patch incorporates Michael's feedback on the
first version, and adds AS names to the autnums table in case they are
not there already, which closes some gaps on rogue ASNs in the LACNIC
area.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Tested-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Add missing area code for AWS

It is unclear what exactly "eusc" refers to ("EU Secure Cloud", maybe?),
but according to RIPE database records for IPv6 networks involved, this
area code physically traces back to Germany, likely Berlin instead of
Frankfurt.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Fix deduplication not looking far enough

When deduplicating the tree, we kept comparing subnets to the largest
supernet only - which is incorrect.

Instead, we need to compare subnets to the next largest supernet which
is now implemented in this patch.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Fix bit length check when merging networks

The check was incorrect as it only expected perfect matches, but we can
of course also merge subnets that don't necessarily require the entire
prefix length for the start address.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Make AS objects hashable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.17

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

data: Update database to 2023-07-31

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Decreate log level when deleting networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

libloc.so: Bump library revision

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix string escaping in location tool

Python 3.12 started to warn on an invalid string escape sequence:

$ location list-networks-by-as --format ipset 1
/usr/bin/location:583: SyntaxWarning: invalid escape sequence '\d'
  m = re.match("^AS(\d+)$", object)
IpsetOutputWriter requires family being set

This patch fixes the warning as well as the regular expression by
using a raw string.

<https://docs.python.org/3.12/whatsnew/changelog.html#id77>

Fixes: #13188 - Invalid escape sequence with Python 3.12
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: More elegantly prevent deleting the root node

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Delete networks from the tree on merge

Previously, the tree did not support deleting nodes, but this has now
been added to do this safely.

This saves us running another deduplication check and halves the size of
the database.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Merge networks before writing the database

This should help us keeping the database more compact where we summarize
two smaller networks to one larger one if they share the same
properties.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Actually delete any deleted nodes

Usually we only mark nodes as deleted, but finally, we should actually
delete what we can as we keep the pointers for back referencing.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Cleanup networks before writing

This patch will add a function that will search the tree for duplicate
entries which are not needed and remove them.

Afterwards, any unused nodes will be deleted.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Add flag to delete nodes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Include bash-completion when packaging

dh_missing for a standard `debuild` run on latest `master` commit
complains about the bash-completion file created for location package.
This file should be listed as for inclusion in location package
accordingly, so that auto-built Debian package picks it up properly.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Decrease the log level if Spamhaus' files are empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Don't write AS without names into the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: AWS: Add country code of NZ for ap-southeast-5

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Housekeep location.install and changelog

In order to run `debuild` for an easy Debian package locally, the
debian/*.install files must list all build results. For version 0.9.16
there is a new pre-built database.db file that belongs with `location`
package, and the `man8` pages have been moved to `man1`.

Additionally, the changelog is added for version 0.9.16 as to maintain
the centralized reference (inside this upstream repository). The
author/timestamp is based on the 0.9.16 tag metadata tracked in Git.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

geofeed: Parse and normalize any URLs

It would be nice if we had an easy way to check if the URL is valid, but
Python does not seem to have a library function for this.

Therefore we might store invalid URLs in the database, but when making a
request to them, urllib with throw an InvalidURL error.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Improve performance of AS name export query

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Log query execution time in debug mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

geofeeds: Catch any invalid URLs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

geofeeds: Delete any data that did not update within two weeks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

geofeeds: Delete any data on 404

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Log any errors to the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Catch TimeoutError when loading Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Silence invalid country code warning

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Skip lines we cannot decode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Convert country codes to uppercase from Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Silently ignore any empty country codes in Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop method to import routing information from route servers

telnetlib in Python is deprecated and will be removed in Python 3.13.
Since we don't use this feature any more, this patch removes it
entirely.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Merge remote-tracking branch 'origin/geofeed'

export: Raise an error when trying to export ipset for both families

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Fix correct set name when family is selected

This does however not fix the problem if no --family= parameter is being
given because we will then export for multiple families in one file.

This is for example not supported for ipset.

Fixes: #12897
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Drop an unused function to count all networks

Reported-by: Gisle Vanem <gvanem@online.no>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Make sure to ship bash completion files even when disabled

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Install bash-completion files.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add bash-completion file for the location command.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Add Georgian translation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Update translations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix parsing gzipped content on invalid Content-Type header

RIPE seems to have misconfigured their webserver which now sends
application/octet-stream for all gzipped files instead of
application/x-gzip or similar which is what the importer would expect.

Instead of only checking the content type, we will now test whether we
see the gzip magic and try to decompress the file then.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.16

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Drop superfluous bracket

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Install Perl files to Perl vendor directory

On Fedora, the original code placed files into /usr/lib64/perl5/5.36/.
That location is not recognized by Fedora perl.

It also supresses installation and uninstallaion of packlist and perlocal
inventory files. They cannot be reliably uninstalled without
corrupting them.

Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Fix check if a signature is set

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Assign correct file descriptor for private keys

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Fix typo in signature variable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Correct error code on verification

If the first signature check returned an error, the error code was
overwritten by the second signature check, returning an overall
incorrect result.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Create the database with read-only permissions for everyone

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move location manual from section 8 to section 1 in location-update.service

It's a tool a non-superuser can use. It's installed into /usr/bin.

Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Remove shebangs from Python modules

Shebangs only belong to standalone executables.

Fixes: #12956
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move location manual from section 8 to section 1

It's a tool a non-superuser can use. It's installed into /usr/bin.

Fixes: #12957
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Drop the test database and use the distributed one

Fixes: #12958
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Ship a recent default database

The database can easily be updated using "make update-database" which
will have to be executed before every release.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Declare make dependencies for Perl binding

Running make in parallel (make -j5) randomly failed with:

/bin/sh ./libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.  -include ./config.h -DSYSCONFDIR=\""/etc"\" -I./src  -Wall -Wchar-subscripts -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wpointer-arith -Wshadow -Wsign-compare -Wstrict-prototypes -Wtype-limits  -fno-semantic-interposition -ffunction-sections -fdata-sections -DLIBLOC_PRIVATE -fvisibility=hidden -g -O2 -c -o src/libloc_internal_la-database.lo `test -f 'src/database.c' || echo './'`src/database.c
/usr/bin/ld: cannot find -lloc: No such file or directory
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:482: blib/arch/auto/Location/Location.so] Error 1
make[2]: *** [Makefile:2982: build-perl] Error 2

That's because build-perl did not depend on libloc library.

A similar issue was with running tests and installing files. Moreover,
thess two targets rebuilt Perl binding from scratch.

This patch make the dependencies explicit and turns build-perl into
non-PHONY.

Fixes: #12961
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "perl: Remove RPATH"

This reverts commit fb574759cbd3a60e64ed2674f514037b9085f078.

Suggested-by: Petr Pisar <ppisar@redhat.com>
Fixes: #12955
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Fix generating file names for ipset output

The tag that is being used in the filename was set after creating the
filename. The order has now been fixed and the export works fine for me.

Fixes: #12944
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Reset Python path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Validate country codes from Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Ensure that we only use HTTPS URLs for Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix reading Geofeeds from remarks

Only RIPE has a dedicated geofeed field. For others, this data needs to
be read from the remarks section.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add a search index match geofeed networks quicker

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Use a GIST index for networks from geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Use geofeeds for country assignment

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Sync geofeeds

Geofeeds are kept in a separate table to only fetch them once per URL.
This needs to be kept in sync which is done before we update any feeds.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Just fetch any exception from the executor

Any exceptions will only be raised in the main thread when they are
fetched from the executor. We do not need to print any other return
values here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add command to import geofeeds into the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Store geofeed URLs from RIR data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Add an empty string to the stringpool

This is useful when we have some offset that isn't initialised. It will
point to the start of the string pool which is some random string that
happened to be added first.

An empty string at least signifies better that something is unset.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Export __version__ in location module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Fix incorrect database path

Reported-by: Jochen Sprickerhof <jochen@sprickerhof.de>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix potential SQL command injection

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: Add new Amazon region codes

Note that the precise location of "ap-southeast-6" is currently
undocumented. However, all IPv4 networks allocated are not routed at the
time of writing, hence our allocation of Asia/Pacific should be
acceptable for the time being.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: Conduct sanity checks per DROP list

Previously, the lack of distinction between different DROP lists caused
only the last one to be persisted. The second version of this patch
incorporates suggestions from Michael on the first version.

Tested-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.15

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add new open() interface to easily open a database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Make the default database path configurable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Flush everything to disk after writing finishes

This might solve a race in the testsuite.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "database: Increase page size to 64k"

This reverts commit 8a330e5575d9f39d2b7716b238e894a650e9605a.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: country: Don't crash when a country could not be found

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country: Drop unused CC_LEN

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix compilation on MacOS X

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Check for madvise

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add signing key to verify signatures

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Initialize r on create

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Drop debug line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Drop unused offset variable in objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Encourage the compiler to inline some functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Call madvise() to tell the kernel that we will randomly access the data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Read all data from the large mmap()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hexdump: Don't try to dump any empty memory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Read header from mapped data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Map the entire database into memory as a whole

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Check if this version of libloc supports the database format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>