configure: Bump version to 0.9.15

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add new open() interface to easily open a database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Make the default database path configurable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Flush everything to disk after writing finishes

This might solve a race in the testsuite.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "database: Increase page size to 64k"

This reverts commit 8a330e5575d9f39d2b7716b238e894a650e9605a.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: country: Don't crash when a country could not be found

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country: Drop unused CC_LEN

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix compilation on MacOS X

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Check for madvise

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add signing key to verify signatures

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Initialize r on create

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Drop debug line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Drop unused offset variable in objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Encourage the compiler to inline some functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Call madvise() to tell the kernel that we will randomly access the data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Read all data from the large mmap()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hexdump: Don't try to dump any empty memory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Read header from mapped data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Map the entire database into memory as a whole

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Check if this version of libloc supports the database format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Refactor checking magic

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Break opening procedure into smaller parts

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Refactor error handling on create

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: database: Expand test coverage

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Export DatabaseEnumerator type

This is useful for tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: country: Return better error codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

stringpool: Drop function to find next offset

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add some simple database tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Fall back when mmap() isn't available

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

stringpool: Implement mmap as optional

This patch will allow mmap() to fail and will make the reader
fall back to read from the given file descriptor.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix errors for Database.lookup()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Improve error reporting when the magic cannot be read

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Correctly raise any errors when opening the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Increase page size to 64k

Since libloc is currently only using mmap() to map any databases into
memory, all mapped address range needs to be page aligned.

In the past, every database was aligned to a 4k page boundary. Some
architectures use 64k pages by default which could therefore not open
the database.

This patch changes the alignment to 64k which should hopefully be a
good common denominator. However, older database cannot be opened.

If we would ever have to support an architecture which's page size
is not evenly fitting into 64k, we will have to implement an
alternative to mmap.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Log any errors when mmap() fails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Do not try to unmap failed mappings

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Use MAP_PRIVATE with mmap()

We do not have any changes that need to be propagated down to the file.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Check if stringpool has been initialized before free

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Replace obsolete AC_PROG_CC_STDC macro

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Check return value of fread() when reading header

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cron: Add a cronjob if systemd is not available

This will allow us to build this package when systemd is not available
(e.g. on Debian kfreebsd/hurd).

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

*_unref: Always expect a valid pointer

Those functions have been very inconsistent in that way that some
happily accepted a NULL pointer and others didn't.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country: Refactor storing country code and continent code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Don't abuse errno as return code

errno is not either set properly, or left set properly by any function
that we have called.

Return codes should either be zero or non-zero for functions unless they
are some value.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Replace strerror(errno) with %m in format string throughout

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: Fix dangling variable

This fixes:

Traceback (most recent call last):
  File "/usr/bin/location-importer", line 1607, in <module>
    main()
  File "/usr/bin/location-importer", line 1605, in main
    c.run()
  File "/usr/bin/location-importer", line 140, in run
    ret = args.func(args)
  File "/usr/bin/location-importer", line 1234, in handle_update_overrides
    self._update_overrides_for_spamhaus_drop()
  File "/usr/bin/location-importer", line 1504, in _update_overrides_for_spamhaus_drop
    for sline in t.readlines():
NameError: name 't' is not defined

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump release to 0.9.14

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Pack all debian files into the tarball

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Import all sources in alphabetical order

To generate deterministic results

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Tolerate that data might exist from other RIRs

Since we are breaking the import into smaller chunks now, it might be
that some data already exists in the database. This is now being
ignored and data won't be replaced.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix missing bracket

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Import each source individually

When importing source data fails for one source, we won't import
anything at all.

This patch changes that behaviour so that we will import all data one
after the other but continue if there was a problem with one source.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Try to make parsing blocks faster

This patch changes the order of some operations and drops a few
redundant ones (potentiall).

That should allow us to parse blocked data faster.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: database: Return None if no description/vendor/license set

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Move importing extended sources/ARIN into transaction

All imports should have been conducted in one large transaction so that
we can remove any previous data.

This was not the case because of an indentation issue and could have
caused that the transaction was commited without all data being
successfully re-imported.

Fixes: #12852
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Change download behaviour

The downloader used to open a connection to the web server hosting our
content which would have been decompressed (if necessary) on the fly and
also been parsed on the fly so that it could have been fed into the
database easily.

Some webservers do not seem to be patient enough to keep the connection
open if things take a little bit longer than usual. That caused the
import to fail.

This patch changes the behaviour that we would download all content
first, store it locally, and then start processing it.

Fixes: #12852
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Cc: Peter Müller <peter.mueller@ipfire.org>

importer: Actually perform the Spamhaus sanity check

The check block was indented incorrectly and would have never been
executed.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Use buster_backport for new debhelper

Improvements from Debian packaging

Merge location-importer into location package

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Rename Python package to python3-location

To comply with the Debian Python policy. Also add a transitional package
with the old name.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: AS names starting with "DNIC" actually are valid

These are all assigned to the US Department of Defense.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer: Only delete override data if we are sure to have a valid replacement

The current way of truncating all override data straight away leaves us
with no data at all, should a source turn out to be unreachable or
returning bogus files (yes, Cloudflare, I _am_ looking at you).

It is therefore better to only delete data we know to have a valid
replacement for, rather than just dropping the source altogether.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Don't fail when output stream isn't seekable

For the ipset format, we will rewrite the header after we know the total
number of entries that have been written in order to optimise the hash
table.

When the output stream isn't seekable, we cannot write the header again
which is being fixed in this patch.

Fixes: #12885
Reported-by: Jon Murphy <jon.murphy@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

verify: Show message on success

Fixes: #12887
Reported-by: Jon Murphy <jon.murphy@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Drop using functools

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

downloader: Fetch __version__ from C module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Make sources around that we can run tests without location installed

In order to run the test suite, we need to make the Python module
loadable from the build directory so that we first of all test the right
code and that it just works without running "make install" first.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "configure: Require Python >= 3.9"

This reverts commit 643b8d062ac11efef6c74cf8af7281079e2533e9.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.13

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Import test database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

.gitignore: Ignore *.db files only in main directory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Enable flattening for everything

When performing checks, it is useful to be able to rely on a flat
network plan so that any larger parent networks in some countries/ASes
won't match any subnets.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Require Python >= 3.9

We are using a couple of newer Python features which are only available
from 3.9 onwards.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: List Python test cases as scripts

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Break after exporting 1000 networks

Otherwise this test runs for forever

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: loc_network_subnets: Use correct prefix

The prefix is being stored as a total number of bits now since it is
easier for any bitwise maths later. This is however causing an incorrect
prefix being computed when splitting a network into two subnets for
IPv4.

To get the correct prefix, loc_network_prefix must be called.

Introduced in 1fd09d0b09756649968fec42c737c88b92e4f11f

Reported-by: Valter Jansons <valter.jansons@gmail.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add more debugging output when running exclude

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Log a more useful message on invalid prefix

If loc_network_subnets is being called with some invalid input, we just
fail without logging a useful message or setting errno accordingly.

This is being changed with this patch.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Allocate subnets list only once

This is a performance improvement when exporting networks flattened. For
the subnet search, we allocate an empty list many times which is often
not required.

This patch changes this behaviour that (if needed) the lists will be
allocated and will stay around and cleared if necessary.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add a simple test that lists all networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Bump version to 0.9.12

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Set default prefix if none is given

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Refactor parsing IP addresses

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: I forgot to initalize the country list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Improve performance of network export query

This patch moves the subqueries out of the large query, so that the
database will materialize them for faster lookup.

We also drop the "UNION ALL" and replace it with just "UNION" because we
do not want any duplicate networks. That will save us many iterations
later on.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Use country list internally

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country-list: Grow like AS list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Grow just like the AS list

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Use AS list internally

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

as-list: Grow faster to avoid too many re-allocations

When we add a large number of ASes to the list, we will constantly
re-allocate the whole list because we are only growing it by 64 entries
at a time.

This patch changes that we will double the list in size every time we
run out of space and we will start with 1024 elements.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

bogons: Skip any subnets of former networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

bogons: Add gaps that are only one address wide

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Add some simple tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Fix buffer overwrite

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Fix decrementing IP addresses

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Simplify functions

This patch improves looping through the octets of an IP address and
avoids that we will read anything that isn't for us.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Remove debugging line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: summarize: Break when we exhausted the network range

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network-list: Simplify debugging output on summarize

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>