lua: database: Add __close method

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: database: Rename the __gc method for consistency

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Actually return a network after lookup()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add a Network object

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add a Database object with a dummy lookup function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Initialize location context

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add version() function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Actually load the C module as such

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

.gitignore: Ignore some more temporary files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Create scaffolding for a module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Require Lua

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Tidy up SQL queries

No functional changes. Just better readability.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop previously imported AS names, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Refactor ARIN importer

This patch is taking advantage of Python's CSV parser.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix Python syntax error

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Replace ARIN AS names source with one that offers human-readable names

This patch replaces our previous source for AS names in ARIN's realms
with another file provided by ARIN that contains human-readable names
for organizations ASNs have been allocated to.

Please note that a

TRUNCATE autnums;

is necessary on machines previously running the old version of
location-importer, in order to make use of this changed data source.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Make the new tests executable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add more dedup/merge tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Show even large diffs in dedup tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Build out more dedup tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Implement rich comparison for Network objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix rich comparison for AS objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country: Return NULL/None for unset attributes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add some tests for the Country object

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix rich comparison function for Country

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Make Country hashable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add a simple test for deduplication

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Tidy up some code

No functional changes.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-tree: Use the raw prefix to place networks onto the tree

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-tree: Run deduplication once for each family

This probably never becomes a real problem, but we did not clear the
stack properly once we switched family when iterating through the
entire tree in one go.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tools: Import the copy script

This is useful to debug the library by copying one library 1:1 and then
diffing the result.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-tree: Split into a separate file

There are no functional changes here, but I find this code easier to
handle if it is split across smaller files.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Fix Spamhaus ASN-DROP parsing

The format of this list has changed, from a plain text file with a
customer schema to JSON. Adjust our routines accordingly to make use of
this list again.

The second version of this patch incorporates Michael's feedback on the
first version, and adds AS names to the autnums table in case they are
not there already, which closes some gaps on rogue ASNs in the LACNIC
area.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Tested-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Add missing area code for AWS

It is unclear what exactly "eusc" refers to ("EU Secure Cloud", maybe?),
but according to RIPE database records for IPv6 networks involved, this
area code physically traces back to Germany, likely Berlin instead of
Frankfurt.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Fix deduplication not looking far enough

When deduplicating the tree, we kept comparing subnets to the largest
supernet only - which is incorrect.

Instead, we need to compare subnets to the next largest supernet which
is now implemented in this patch.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Fix bit length check when merging networks

The check was incorrect as it only expected perfect matches, but we can
of course also merge subnets that don't necessarily require the entire
prefix length for the start address.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Make AS objects hashable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.17

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

data: Update database to 2023-07-31

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Decreate log level when deleting networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

libloc.so: Bump library revision

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix string escaping in location tool

Python 3.12 started to warn on an invalid string escape sequence:

$ location list-networks-by-as --format ipset 1
/usr/bin/location:583: SyntaxWarning: invalid escape sequence '\d'
  m = re.match("^AS(\d+)$", object)
IpsetOutputWriter requires family being set

This patch fixes the warning as well as the regular expression by
using a raw string.

<https://docs.python.org/3.12/whatsnew/changelog.html#id77>

Fixes: #13188 - Invalid escape sequence with Python 3.12
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: More elegantly prevent deleting the root node

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Delete networks from the tree on merge

Previously, the tree did not support deleting nodes, but this has now
been added to do this safely.

This saves us running another deduplication check and halves the size of
the database.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Merge networks before writing the database

This should help us keeping the database more compact where we summarize
two smaller networks to one larger one if they share the same
properties.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Actually delete any deleted nodes

Usually we only mark nodes as deleted, but finally, we should actually
delete what we can as we keep the pointers for back referencing.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Cleanup networks before writing

This patch will add a function that will search the tree for duplicate
entries which are not needed and remove them.

Afterwards, any unused nodes will be deleted.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Add flag to delete nodes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Include bash-completion when packaging

dh_missing for a standard `debuild` run on latest `master` commit
complains about the bash-completion file created for location package.
This file should be listed as for inclusion in location package
accordingly, so that auto-built Debian package picks it up properly.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Decrease the log level if Spamhaus' files are empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Don't write AS without names into the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: AWS: Add country code of NZ for ap-southeast-5

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Housekeep location.install and changelog

In order to run `debuild` for an easy Debian package locally, the
debian/*.install files must list all build results. For version 0.9.16
there is a new pre-built database.db file that belongs with `location`
package, and the `man8` pages have been moved to `man1`.

Additionally, the changelog is added for version 0.9.16 as to maintain
the centralized reference (inside this upstream repository). The
author/timestamp is based on the 0.9.16 tag metadata tracked in Git.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

geofeed: Parse and normalize any URLs

It would be nice if we had an easy way to check if the URL is valid, but
Python does not seem to have a library function for this.

Therefore we might store invalid URLs in the database, but when making a
request to them, urllib with throw an InvalidURL error.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Improve performance of AS name export query

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Log query execution time in debug mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

geofeeds: Catch any invalid URLs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

geofeeds: Delete any data that did not update within two weeks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

geofeeds: Delete any data on 404

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Log any errors to the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Catch TimeoutError when loading Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Silence invalid country code warning

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Skip lines we cannot decode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Convert country codes to uppercase from Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Silently ignore any empty country codes in Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop method to import routing information from route servers

telnetlib in Python is deprecated and will be removed in Python 3.13.
Since we don't use this feature any more, this patch removes it
entirely.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Merge remote-tracking branch 'origin/geofeed'

export: Raise an error when trying to export ipset for both families

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Fix correct set name when family is selected

This does however not fix the problem if no --family= parameter is being
given because we will then export for multiple families in one file.

This is for example not supported for ipset.

Fixes: #12897
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Drop an unused function to count all networks

Reported-by: Gisle Vanem <gvanem@online.no>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Make sure to ship bash completion files even when disabled

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Install bash-completion files.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add bash-completion file for the location command.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Add Georgian translation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Update translations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix parsing gzipped content on invalid Content-Type header

RIPE seems to have misconfigured their webserver which now sends
application/octet-stream for all gzipped files instead of
application/x-gzip or similar which is what the importer would expect.

Instead of only checking the content type, we will now test whether we
see the gzip magic and try to decompress the file then.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.16

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Drop superfluous bracket

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Install Perl files to Perl vendor directory

On Fedora, the original code placed files into /usr/lib64/perl5/5.36/.
That location is not recognized by Fedora perl.

It also supresses installation and uninstallaion of packlist and perlocal
inventory files. They cannot be reliably uninstalled without
corrupting them.

Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Fix check if a signature is set

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Assign correct file descriptor for private keys

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Fix typo in signature variable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Correct error code on verification

If the first signature check returned an error, the error code was
overwritten by the second signature check, returning an overall
incorrect result.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Create the database with read-only permissions for everyone

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move location manual from section 8 to section 1 in location-update.service

It's a tool a non-superuser can use. It's installed into /usr/bin.

Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Remove shebangs from Python modules

Shebangs only belong to standalone executables.

Fixes: #12956
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move location manual from section 8 to section 1

It's a tool a non-superuser can use. It's installed into /usr/bin.

Fixes: #12957
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Drop the test database and use the distributed one

Fixes: #12958
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Ship a recent default database

The database can easily be updated using "make update-database" which
will have to be executed before every release.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Declare make dependencies for Perl binding

Running make in parallel (make -j5) randomly failed with:

/bin/sh ./libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.  -include ./config.h -DSYSCONFDIR=\""/etc"\" -I./src  -Wall -Wchar-subscripts -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wpointer-arith -Wshadow -Wsign-compare -Wstrict-prototypes -Wtype-limits  -fno-semantic-interposition -ffunction-sections -fdata-sections -DLIBLOC_PRIVATE -fvisibility=hidden -g -O2 -c -o src/libloc_internal_la-database.lo `test -f 'src/database.c' || echo './'`src/database.c
/usr/bin/ld: cannot find -lloc: No such file or directory
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:482: blib/arch/auto/Location/Location.so] Error 1
make[2]: *** [Makefile:2982: build-perl] Error 2

That's because build-perl did not depend on libloc library.

A similar issue was with running tests and installing files. Moreover,
thess two targets rebuilt Perl binding from scratch.

This patch make the dependencies explicit and turns build-perl into
non-PHONY.

Fixes: #12961
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "perl: Remove RPATH"

This reverts commit fb574759cbd3a60e64ed2674f514037b9085f078.

Suggested-by: Petr Pisar <ppisar@redhat.com>
Fixes: #12955
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Fix generating file names for ipset output

The tag that is being used in the filename was set after creating the
filename. The order has now been fixed and the export works fine for me.

Fixes: #12944
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Reset Python path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Validate country codes from Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Ensure that we only use HTTPS URLs for Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix reading Geofeeds from remarks

Only RIPE has a dedicated geofeed field. For others, this data needs to
be read from the remarks section.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add a search index match geofeed networks quicker

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Use a GIST index for networks from geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Use geofeeds for country assignment

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Sync geofeeds

Geofeeds are kept in a separate table to only fetch them once per URL.
This needs to be kept in sync which is done before we update any feeds.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>