debian: Sync version

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Restart flatten algorithm from the top when a network was dropped

We used to simply take the first element from the stack after we have
split a network. That is wrong because it is not passing through any
filters and no further subnet checks. It could have therefore been
that the tree was not entirely flat.

Reported-by: Arne Fitzenreiter <arne_f@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Exit find operation early when the list is empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Do not check whether an integer is larger than 32 bit on 32 bit arches

This cannot happen and generated a compiler warning

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: PyList_Size returns ssize_t which is signed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Silence compiler formatting warnings for size_t

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Adjust man-pages switch with documentation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

as-list/country-list: Set elements list to zero after freeing it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Add greater elements after the current one

The list was otherwise not sorted

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Add more networks to test

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Copy flags when split into subnets

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.5

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Use clear function to tidy up

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Include network header

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Remove deprecated sort call

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Massively improve performance on exclude

When we check the result for any overlaps, we can cut this short
by walking through both lists from start to end and remember the
last network that we checked.

The next one will by definition be strictly greater and therefore
we do not need to check anything before this any more.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Call subnet function with the correct order of arguments

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add more excluded networks straight to the to_check list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Do not execute with an error when the excluded result will be empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Remove comparing family

Everything is encoded in IPv6 anyways...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add tests for overlaps function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Remove useless comment

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Show index when listing networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Do not half list when popping the first element

The list was unfortunately halved in size every time an element
was taken from it, which was great for performance, but shortened
the result substantially.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Set elements pointer to NULL so that we know it is empty

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Check last element before doing binary search

This is helpful because very often we walk through a list in
order and are most interested in the last element.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Use binary search to find if a network is a subnet

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Adjust return codes of loc_network_match_address and add test

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Drop loc_network_gt which is now unused

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Drop loc_network_eq in favour of loc_network_cmp

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

test: Add more networks to list to see the algorithm work

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add excluded networks to to_check list

This is now being done immediately instead of creating a new
list which is being merged later.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Read the first element from the list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Avoid merging the same data twice

When finish splitting networks into many parts, we have
a list of subnets with the excluded subnets and merge them
together first and put them on the stack again.

This is slower than pushing it all onto the stack first
and then popping the first element.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Drop sorting functions

Since the list is always sorted, there is no point in
sorting it again...

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Make this a sorted list

The list is now organised so that it is always ordered.

This allows us to find if a network is on the list a lot
quicker using binary search as well as inserting a new
network at the right place.

This will benefit loc_network_exclude with very large networks.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Remove deprecated loc_network_is_subnet_of function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Speed up subnet check

There is no point in checking different address families
with each other and we do not need to compare addresses
when the prefix of the subnet does not fit into the
network to check.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add function to return the prefix

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Optimise _subnet function

This function used to create a network list which always
had exactly two elements. Since splitting a network in half
always returns two parts, we can simply return them as a
pointer.

This improves returning the network tree by about 17%.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Implement merging in reverse in one step

This will save us some time because we do not need to
change the list in place first and then merge it.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Flatten out code due to compiler errors

It looks like GCC could not follow the code as it was
written. Therefore I clean up more often now instead
of having a GOTO block where I do that in.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Remove debugging output

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Fix loc_network_is_subnet()

This function always returned false.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

test: Add tests to network-lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Disable some useless code when not running in debug mode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Speed-up export in xt_geoip format

Removing the loop avoids creating a tuple and then iterating over it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Remove old flattening feature

The database enumerator now only returns networks that will
never overlap.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add property to return IP addresses as bytes

This avoids calling inet_pton to parse IP addresses from string

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix download of database

This was all messed up in 0c74f6b1a3bdce5ebdc2ee452b9baf3e421dd3d1
when the change of type for the timestamp wasn't changed everywhere
else.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: End lookup after an invalid IP address was passed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Remove unnecessary db object from writers

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Remove filtering for flags

The filter is an AND filter and if we set the flags from
the special country codes, we won't get back much.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Add debug output to filtering

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Free filter lists in enumerator

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Do not clean up python list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Actually clear all lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Change back to use Network objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

as: Make lists grow dynamically

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Make list grow dynamically

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

countries: Make list grow dynamically

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Simplify AS matching code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Simplify network matching code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

as: Add list for easier processing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Filter flags in C

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Only return country codes we want

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Implement filtering for multiple countries in the enumerator

This will allow us to speed up the export of the database
if only a few countries should be returned.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move network lists into an own file

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Move tree flattening into C

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Unexport all tree functions

These should not be exported

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add function to pop first element from stack

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Sort result of excluded lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Filter results coming from stack

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Move network filtering into a separate function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Copy all attributes when splitting networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

test: Update API

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Add option to return networks flattened

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add function to exclude multiple networks at once

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add new subnet function

The old one is too difficult to use in terms of order
of input parameters and return value.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add more debugging output to stacks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Export networks exclude function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Reduce debugging output

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Pass flag to enumerator to flatten output

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Make lists unique

Networks that are in the list won't be added again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add function to merge two lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add function to check if network is part of a list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add function to check if two networks overlap

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add functions to break network into subnets

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add function to dump lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

networks: Add list to manage groups of networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

test: Add tests for database enumerator

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: always convert organisation handles into upper cases

Fixes: #12523
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location.in: do not confuse UTC with local time zones

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>

location update: Remove double conversion of timestamps

This caused that the timestamp in DNS was misinterpreted
as local time and often, databases could not be downloaded.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>

location: Fix Python syntax error in verify()

The database is now being opened before the requested
method is called and handle_verify() wasn't updated.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Flatten the tree before exporting it

This patch removes the possibility that any IP address ranges
might show up in multiple exported files.

If this was content from the database:

  * 10.0.0.0/16 - DE
  * 10.0.1.0/24 - FR

Then the IP address 10.0.1.1 would match for both countries.

The algorithm will now break the larger /16 subnet apart into
smaller subnets so that 10.0.1.0/24 is no longer overlapped.

There was some time spent on this to make this as efficient
as possible.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: avoid log spam for too small networks

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: omit historic/orphaned RIR data

Some RIRs include detailled information regarding networks not managed
by or allocated to themselves, particually APNIC. We need to filter
those networks (they usually have a characteristic network name) in
order to prevent operational quirks or returning wrong country codes.

Fixes: #12501
Partially fixes: #12499

Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer.py: fetch LACNIC data via HTTPS

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: filter bogus IP networks for both Whois and extended sources

Sanity checks for parsed networks have been put into a separate function
to avoid boilerplate code for extended sources. This makes the location
database less vulnerable to garbage written into RIR databases on
purpose or by chance.

Fixes: #12500
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>