importer: Catch TimeoutError when loading Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Silence invalid country code warning

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Skip lines we cannot decode

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Convert country codes to uppercase from Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Silently ignore any empty country codes in Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop method to import routing information from route servers

telnetlib in Python is deprecated and will be removed in Python 3.13.
Since we don't use this feature any more, this patch removes it
entirely.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Merge remote-tracking branch 'origin/geofeed'

export: Raise an error when trying to export ipset for both families

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location: Fix correct set name when family is selected

This does however not fix the problem if no --family= parameter is being
given because we will then export for multiple families in one file.

This is for example not supported for ipset.

Fixes: #12897
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Drop an unused function to count all networks

Reported-by: Gisle Vanem <gvanem@online.no>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Make sure to ship bash completion files even when disabled

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Install bash-completion files.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Add bash-completion file for the location command.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Add Georgian translation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Update translations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix parsing gzipped content on invalid Content-Type header

RIPE seems to have misconfigured their webserver which now sends
application/octet-stream for all gzipped files instead of
application/x-gzip or similar which is what the importer would expect.

Instead of only checking the content type, we will now test whether we
see the gzip magic and try to decompress the file then.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.16

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Drop superfluous bracket

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Install Perl files to Perl vendor directory

On Fedora, the original code placed files into /usr/lib64/perl5/5.36/.
That location is not recognized by Fedora perl.

It also supresses installation and uninstallaion of packlist and perlocal
inventory files. They cannot be reliably uninstalled without
corrupting them.

Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Fix check if a signature is set

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Assign correct file descriptor for private keys

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Fix typo in signature variable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Correct error code on verification

If the first signature check returned an error, the error code was
overwritten by the second signature check, returning an overall
incorrect result.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Create the database with read-only permissions for everyone

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move location manual from section 8 to section 1 in location-update.service

It's a tool a non-superuser can use. It's installed into /usr/bin.

Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Remove shebangs from Python modules

Shebangs only belong to standalone executables.

Fixes: #12956
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Move location manual from section 8 to section 1

It's a tool a non-superuser can use. It's installed into /usr/bin.

Fixes: #12957
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Drop the test database and use the distributed one

Fixes: #12958
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Ship a recent default database

The database can easily be updated using "make update-database" which
will have to be executed before every release.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Declare make dependencies for Perl binding

Running make in parallel (make -j5) randomly failed with:

/bin/sh ./libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.  -include ./config.h -DSYSCONFDIR=\""/etc"\" -I./src  -Wall -Wchar-subscripts -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wpointer-arith -Wshadow -Wsign-compare -Wstrict-prototypes -Wtype-limits  -fno-semantic-interposition -ffunction-sections -fdata-sections -DLIBLOC_PRIVATE -fvisibility=hidden -g -O2 -c -o src/libloc_internal_la-database.lo `test -f 'src/database.c' || echo './'`src/database.c
/usr/bin/ld: cannot find -lloc: No such file or directory
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:482: blib/arch/auto/Location/Location.so] Error 1
make[2]: *** [Makefile:2982: build-perl] Error 2

That's because build-perl did not depend on libloc library.

A similar issue was with running tests and installing files. Moreover,
thess two targets rebuilt Perl binding from scratch.

This patch make the dependencies explicit and turns build-perl into
non-PHONY.

Fixes: #12961
Signed-off-by: Petr Písař <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "perl: Remove RPATH"

This reverts commit fb574759cbd3a60e64ed2674f514037b9085f078.

Suggested-by: Petr Pisar <ppisar@redhat.com>
Fixes: #12955
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

export: Fix generating file names for ipset output

The tag that is being used in the filename was set after creating the
filename. The order has now been fixed and the export works fine for me.

Fixes: #12944
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Reset Python path

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Validate country codes from Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Ensure that we only use HTTPS URLs for Geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix reading Geofeeds from remarks

Only RIPE has a dedicated geofeed field. For others, this data needs to
be read from the remarks section.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add a search index match geofeed networks quicker

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Use a GIST index for networks from geofeeds

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Use geofeeds for country assignment

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Sync geofeeds

Geofeeds are kept in a separate table to only fetch them once per URL.
This needs to be kept in sync which is done before we update any feeds.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Just fetch any exception from the executor

Any exceptions will only be raised in the main thread when they are
fetched from the executor. We do not need to print any other return
values here.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Add command to import geofeeds into the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Store geofeed URLs from RIR data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Add an empty string to the stringpool

This is useful when we have some offset that isn't initialised. It will
point to the start of the string pool which is some random string that
happened to be added first.

An empty string at least signifies better that something is unset.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Export __version__ in location module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Fix incorrect database path

Reported-by: Jochen Sprickerhof <jochen@sprickerhof.de>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Fix potential SQL command injection

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: Add new Amazon region codes

Note that the precise location of "ap-southeast-6" is currently
undocumented. However, all IPv4 networks allocated are not routed at the
time of writing, hence our allocation of Asia/Pacific should be
acceptable for the time being.

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: Conduct sanity checks per DROP list

Previously, the lack of distinction between different DROP lists caused
only the last one to be persisted. The second version of this patch
incorporates suggestions from Michael on the first version.

Tested-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.15

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Add new open() interface to easily open a database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Make the default database path configurable

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Flush everything to disk after writing finishes

This might solve a race in the testsuite.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "database: Increase page size to 64k"

This reverts commit 8a330e5575d9f39d2b7716b238e894a650e9605a.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: country: Don't crash when a country could not be found

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country: Drop unused CC_LEN

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix compilation on MacOS X

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Check for madvise

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add signing key to verify signatures

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Initialize r on create

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Drop debug line

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Drop unused offset variable in objects

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Encourage the compiler to inline some functions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Call madvise() to tell the kernel that we will randomly access the data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Read all data from the large mmap()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

hexdump: Don't try to dump any empty memory

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Read header from mapped data

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Map the entire database into memory as a whole

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Check if this version of libloc supports the database format

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Refactor checking magic

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Break opening procedure into smaller parts

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Refactor error handling on create

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: database: Expand test coverage

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Export DatabaseEnumerator type

This is useful for tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: country: Return better error codes

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

stringpool: Drop function to find next offset

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add some simple database tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Fall back when mmap() isn't available

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

stringpool: Implement mmap as optional

This patch will allow mmap() to fail and will make the reader
fall back to read from the given file descriptor.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix errors for Database.lookup()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Improve error reporting when the magic cannot be read

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Correctly raise any errors when opening the database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Increase page size to 64k

Since libloc is currently only using mmap() to map any databases into
memory, all mapped address range needs to be page aligned.

In the past, every database was aligned to a 4k page boundary. Some
architectures use 64k pages by default which could therefore not open
the database.

This patch changes the alignment to 64k which should hopefully be a
good common denominator. However, older database cannot be opened.

If we would ever have to support an architecture which's page size
is not evenly fitting into 64k, we will have to implement an
alternative to mmap.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Log any errors when mmap() fails

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Do not try to unmap failed mappings

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Use MAP_PRIVATE with mmap()

We do not have any changes that need to be propagated down to the file.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Check if stringpool has been initialized before free

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Replace obsolete AC_PROG_CC_STDC macro

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Check return value of fread() when reading header

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

cron: Add a cronjob if systemd is not available

This will allow us to build this package when systemd is not available
(e.g. on Debian kfreebsd/hurd).

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

*_unref: Always expect a valid pointer

Those functions have been very inconsistent in that way that some
happily accepted a NULL pointer and others didn't.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

country: Refactor storing country code and continent code

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Don't abuse errno as return code

errno is not either set properly, or left set properly by any function
that we have called.

Return codes should either be zero or non-zero for functions unless they
are some value.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Replace strerror(errno) with %m in format string throughout

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

location-importer.in: Fix dangling variable

This fixes:

Traceback (most recent call last):
  File "/usr/bin/location-importer", line 1607, in <module>
    main()
  File "/usr/bin/location-importer", line 1605, in main
    c.run()
  File "/usr/bin/location-importer", line 140, in run
    ret = args.func(args)
  File "/usr/bin/location-importer", line 1234, in handle_update_overrides
    self._update_overrides_for_spamhaus_drop()
  File "/usr/bin/location-importer", line 1504, in _update_overrides_for_spamhaus_drop
    for sline in t.readlines():
NameError: name 't' is not defined

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump release to 0.9.14

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Pack all debian files into the tarball

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Import all sources in alphabetical order

To generate deterministic results

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Tolerate that data might exist from other RIRs

Since we are breaking the import into smaller chunks now, it might be
that some data already exists in the database. This is now being
ignored and data won't be replaced.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix missing bracket

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>