Michael Tremer [Fri, 2 Dec 2022 11:06:01 +0000 (11:06 +0000)]
importer: Fix parsing gzipped content on invalid Content-Type header
RIPE seems to have misconfigured their webserver which now sends
application/octet-stream for all gzipped files instead of
application/x-gzip or similar which is what the importer would expect.
Instead of only checking the content type, we will now test whether we
see the gzip magic and try to decompress the file then.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Petr Písař [Tue, 4 Oct 2022 11:51:21 +0000 (13:51 +0200)]
Install Perl files to Perl vendor directory
On Fedora, the original code placed files into /usr/lib64/perl5/5.36/.
That location is not recognized by Fedora perl.
It also supresses installation and uninstallaion of packlist and perlocal
inventory files. They cannot be reliably uninstalled without
corrupting them.
Signed-off-by: Petr Písař <ppisar@redhat.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 27 Sep 2022 16:28:44 +0000 (16:28 +0000)]
writer: Add an empty string to the stringpool
This is useful when we have some offset that isn't initialised. It will
point to the start of the string pool which is some random string that
happened to be added first.
An empty string at least signifies better that something is unset.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 26 Sep 2022 18:33:29 +0000 (18:33 +0000)]
location-importer.in: Add new Amazon region codes
Note that the precise location of "ap-southeast-6" is currently
undocumented. However, all IPv4 networks allocated are not routed at the
time of writing, hence our allocation of Asia/Pacific should be
acceptable for the time being.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 26 Sep 2022 18:26:44 +0000 (18:26 +0000)]
location-importer.in: Conduct sanity checks per DROP list
Previously, the lack of distinction between different DROP lists caused
only the last one to be persisted. The second version of this patch
incorporates suggestions from Michael on the first version.
Tested-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 22 Aug 2022 10:37:29 +0000 (10:37 +0000)]
database: Increase page size to 64k
Since libloc is currently only using mmap() to map any databases into
memory, all mapped address range needs to be page aligned.
In the past, every database was aligned to a 4k page boundary. Some
architectures use 64k pages by default which could therefore not open
the database.
This patch changes the alignment to 64k which should hopefully be a
good common denominator. However, older database cannot be opened.
If we would ever have to support an architecture which's page size
is not evenly fitting into 64k, we will have to implement an
alternative to mmap.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 14 Aug 2022 16:02:55 +0000 (16:02 +0000)]
location-importer.in: Fix dangling variable
This fixes:
Traceback (most recent call last):
File "/usr/bin/location-importer", line 1607, in <module>
main()
File "/usr/bin/location-importer", line 1605, in main
c.run()
File "/usr/bin/location-importer", line 140, in run
ret = args.func(args)
File "/usr/bin/location-importer", line 1234, in handle_update_overrides
self._update_overrides_for_spamhaus_drop()
File "/usr/bin/location-importer", line 1504, in _update_overrides_for_spamhaus_drop
for sline in t.readlines():
NameError: name 't' is not defined
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 12 Aug 2022 15:47:29 +0000 (15:47 +0000)]
importer: Tolerate that data might exist from other RIRs
Since we are breaking the import into smaller chunks now, it might be
that some data already exists in the database. This is now being
ignored and data won't be replaced.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 12 Aug 2022 13:56:17 +0000 (13:56 +0000)]
importer: Move importing extended sources/ARIN into transaction
All imports should have been conducted in one large transaction so that
we can remove any previous data.
This was not the case because of an indentation issue and could have
caused that the transaction was commited without all data being
successfully re-imported.
Fixes: #12852 Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 12 Aug 2022 13:51:20 +0000 (13:51 +0000)]
importer: Change download behaviour
The downloader used to open a connection to the web server hosting our
content which would have been decompressed (if necessary) on the fly and
also been parsed on the fly so that it could have been fed into the
database easily.
Some webservers do not seem to be patient enough to keep the connection
open if things take a little bit longer than usual. That caused the
import to fail.
This patch changes the behaviour that we would download all content
first, store it locally, and then start processing it.
Fixes: #12852 Signed-off-by: Michael Tremer <michael.tremer@ipfire.org> Cc: Peter Müller <peter.mueller@ipfire.org>
Peter Müller [Sun, 5 Jun 2022 10:04:50 +0000 (10:04 +0000)]
location-importer: Only delete override data if we are sure to have a valid replacement
The current way of truncating all override data straight away leaves us
with no data at all, should a source turn out to be unreachable or
returning bogus files (yes, Cloudflare, I _am_ looking at you).
It is therefore better to only delete data we know to have a valid
replacement for, rather than just dropping the source altogether.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 14 Apr 2022 18:31:56 +0000 (18:31 +0000)]
Make sources around that we can run tests without location installed
In order to run the test suite, we need to make the Python module
loadable from the build directory so that we first of all test the right
code and that it just works without running "make install" first.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 11 Apr 2022 17:57:22 +0000 (17:57 +0000)]
export: Enable flattening for everything
When performing checks, it is useful to be able to rely on a flat
network plan so that any larger parent networks in some countries/ASes
won't match any subnets.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
projects / location / libloc.git / log
