importer: Drop the geofeed sources when updating RIR data

This is a cleaner way to drop any previously imported content and should
also be faster because we can drop a lot of DELETE staments for objects
without Geofeeds.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Create a better structure to import RIRs

All information about all RIRs will now be imported in one large
database transaction per RIR which should bring us better integrity and
help us to phase out any stale data.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add option to only import specific RIRs

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Also import networks that are smaller than /48 or /24

These cannot appear in the global routing table, but that does not mean
that there isn't any value in importing them into our database.

In case they are just noise, we will filter them out later on.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Improve checks for unspecified networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Improve check for network objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Change country code logic

If we have imported something without valid country codes, we still need
to import that network object into the database. Further information
could come from a Geofeed. Otherwise we still want to have this network
here for later processing.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Skip any countries that we don't know

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: When storing country codes, make the code more straight-forward

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Return known countries as a set()

It should be possible to search through sets quicker.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Make translating country codes more extensible

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Ignore certain country codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fail if no countries have been imported, yet

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Remove superfluous function call

Normally a function call doesn't have that much overhead in Python, but
since we are calling all these that many times I am sure this is all
adding up.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Use the downloader to import Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop any data from feeds we no longer support

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop source field from overrides table

Since we now import all feeds into a separate table including their own
source, we can drop this from the *_overrides tables as we only import
our own data here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Refactor parsing Spamhaus ASNDROP

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Refactor parsing Spamhaus DROP feeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add the option to only update one feed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Completely rewrite the AWS parser

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Reformat AWS dictionary

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Simplify fetching countries

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Refactor feed parsing

This adds a bit of common code across all feeds, but is only implemented
for AWS at the moment.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Create a new subcommand to import feeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Reformat the large SQL query

There are no functional changes here. This is just to be able to extend
the script more easily.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix typo in SQL query

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Import AWS IP feed into the new feeds table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Import Spamhaus DROP feeds into the new feeds table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Don't import any AS names from Spamhaus

Those are not very human readable and we should usually have a better
source.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Create a feeds table for networks, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Import the Spamhaus ASN feed into the new feeds table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Create an extra table for feeds

We are currently using the overrides table to import any third-party
feeds. That creates the problem that those tables can only hold one row
for each AS which we cannot easily merge without losing the source of
the information.

Therefore this patch creates a new table which allow us to store this
information per feed and we will use that information if there is no
overrides information.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Pass downloader to the ARIN import

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Don't check for Lua when --disable-lua is passed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: network: Implement checking flags

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Export network flags

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Export default database path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Check for luaunit

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: database: Implement verify()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: database: Export description/license/vendor

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: database: Implement fetching countries

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: database: Implementing fetching AS objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add AS object

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: country: Export remaining properties

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Don't try to free memory that was allocated by Lua

The userdata that we have allocated cannot be freed as we don't have
control over it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Force testing garbage collection for countries

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Create Country objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Force garbage collection to test those methods

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Check if we can read the network's properties

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Fix calling methods that belong to an object

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: network: Add a __close method

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: database: Add __close method

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: database: Rename the __gc method for consistency

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Actually return a network after lookup()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add a Network object

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add a Database object with a dummy lookup function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Initialize location context

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add version() function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Actually load the C module as such

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

.gitignore: Ignore some more temporary files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Create scaffolding for a module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Require Lua

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Tidy up SQL queries

No functional changes. Just better readability.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop previously imported AS names, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Refactor ARIN importer

This patch is taking advantage of Python's CSV parser.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix Python syntax error

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Replace ARIN AS names source with one that offers human-readable names

This patch replaces our previous source for AS names in ARIN's realms
with another file provided by ARIN that contains human-readable names
for organizations ASNs have been allocated to.

Please note that a

TRUNCATE autnums;

is necessary on machines previously running the old version of
location-importer, in order to make use of this changed data source.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Make the new tests executable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add more dedup/merge tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Show even large diffs in dedup tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Build out more dedup tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Implement rich comparison for Network objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix rich comparison for AS objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country: Return NULL/None for unset attributes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add some tests for the Country object

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix rich comparison function for Country

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Make Country hashable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add a simple test for deduplication

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Tidy up some code

No functional changes.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-tree: Use the raw prefix to place networks onto the tree

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-tree: Run deduplication once for each family

This probably never becomes a real problem, but we did not clear the
stack properly once we switched family when iterating through the
entire tree in one go.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tools: Import the copy script

This is useful to debug the library by copying one library 1:1 and then
diffing the result.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-tree: Split into a separate file

There are no functional changes here, but I find this code easier to
handle if it is split across smaller files.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Fix Spamhaus ASN-DROP parsing

The format of this list has changed, from a plain text file with a
customer schema to JSON. Adjust our routines accordingly to make use of
this list again.

The second version of this patch incorporates Michael's feedback on the
first version, and adds AS names to the autnums table in case they are
not there already, which closes some gaps on rogue ASNs in the LACNIC
area.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Tested-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Add missing area code for AWS

It is unclear what exactly "eusc" refers to ("EU Secure Cloud", maybe?),
but according to RIPE database records for IPv6 networks involved, this
area code physically traces back to Germany, likely Berlin instead of
Frankfurt.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Fix deduplication not looking far enough

When deduplicating the tree, we kept comparing subnets to the largest
supernet only - which is incorrect.

Instead, we need to compare subnets to the next largest supernet which
is now implemented in this patch.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Fix bit length check when merging networks

The check was incorrect as it only expected perfect matches, but we can
of course also merge subnets that don't necessarily require the entire
prefix length for the start address.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Make AS objects hashable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.17

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

data: Update database to 2023-07-31

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Decreate log level when deleting networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

libloc.so: Bump library revision

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix string escaping in location tool

Python 3.12 started to warn on an invalid string escape sequence:

$ location list-networks-by-as --format ipset 1
/usr/bin/location:583: SyntaxWarning: invalid escape sequence '\d'
  m = re.match("^AS(\d+)$", object)
IpsetOutputWriter requires family being set

This patch fixes the warning as well as the regular expression by
using a raw string.

<https://docs.python.org/3.12/whatsnew/changelog.html#id77>

Fixes: #13188 - Invalid escape sequence with Python 3.12
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: More elegantly prevent deleting the root node

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Delete networks from the tree on merge

Previously, the tree did not support deleting nodes, but this has now
been added to do this safely.

This saves us running another deduplication check and halves the size of
the database.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Merge networks before writing the database

This should help us keeping the database more compact where we summarize
two smaller networks to one larger one if they share the same
properties.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Actually delete any deleted nodes

Usually we only mark nodes as deleted, but finally, we should actually
delete what we can as we keep the pointers for back referencing.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Cleanup networks before writing

This patch will add a function that will search the tree for duplicate
entries which are not needed and remove them.

Afterwards, any unused nodes will be deleted.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Add flag to delete nodes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>