Michael Tremer [Wed, 6 Mar 2024 22:45:28 +0000 (22:45 +0000)]
tree: Be smarter when removing networks from the stack
Since we are working through a sorted tree, we will either only see
subnets of a network or we won't. Once we see a network that isn't a
subnet, we remove the supernet until nothing is left on the stack.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 4 Mar 2024 12:06:33 +0000 (12:06 +0000)]
importer: No longer import Geofeeds concurrently
This is kind of a problem since many servers have gone away, respond
very slowly, etc. But since we are moving towards psycopg3, we cannot do
this the old way any more.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 4 Mar 2024 11:47:10 +0000 (11:47 +0000)]
importer: Drop the geofeed sources when updating RIR data
This is a cleaner way to drop any previously imported content and should
also be faster because we can drop a lot of DELETE staments for objects
without Geofeeds.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 4 Mar 2024 11:42:04 +0000 (11:42 +0000)]
importer: Create a better structure to import RIRs
All information about all RIRs will now be imported in one large
database transaction per RIR which should bring us better integrity and
help us to phase out any stale data.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 4 Mar 2024 10:12:42 +0000 (10:12 +0000)]
importer: Change country code logic
If we have imported something without valid country codes, we still need
to import that network object into the database. Further information
could come from a Geofeed. Otherwise we still want to have this network
here for later processing.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 4 Mar 2024 09:34:14 +0000 (09:34 +0000)]
importer: Remove superfluous function call
Normally a function call doesn't have that much overhead in Python, but
since we are calling all these that many times I am sure this is all
adding up.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 2 Mar 2024 11:26:43 +0000 (11:26 +0000)]
importer: Drop source field from overrides table
Since we now import all feeds into a separate table including their own
source, we can drop this from the *_overrides tables as we only import
our own data here.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 2 Mar 2024 09:23:30 +0000 (09:23 +0000)]
importer: Create an extra table for feeds
We are currently using the overrides table to import any third-party
feeds. That creates the problem that those tables can only hold one row
for each AS which we cannot easily merge without losing the source of
the information.
Therefore this patch creates a new table which allow us to store this
information per feed and we will use that information if there is no
overrides information.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 10 Dec 2023 19:37:00 +0000 (19:37 +0000)]
location-importer: Replace ARIN AS names source with one that offers human-readable names
This patch replaces our previous source for AS names in ARIN's realms
with another file provided by ARIN that contains human-readable names
for organizations ASNs have been allocated to.
Please note that a
TRUNCATE autnums;
is necessary on machines previously running the old version of
location-importer, in order to make use of this changed data source.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Feb 2024 14:45:43 +0000 (14:45 +0000)]
network-tree: Run deduplication once for each family
This probably never becomes a real problem, but we did not clear the
stack properly once we switched family when iterating through the
entire tree in one go.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 17 Feb 2024 22:31:00 +0000 (22:31 +0000)]
location-importer: Fix Spamhaus ASN-DROP parsing
The format of this list has changed, from a plain text file with a
customer schema to JSON. Adjust our routines accordingly to make use of
this list again.
The second version of this patch incorporates Michael's feedback on the
first version, and adds AS names to the autnums table in case they are
not there already, which closes some gaps on rogue ASNs in the LACNIC
area.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Tested-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 17 Feb 2024 22:39:00 +0000 (22:39 +0000)]
location-importer: Add missing area code for AWS
It is unclear what exactly "eusc" refers to ("EU Secure Cloud", maybe?),
but according to RIPE database records for IPv6 networks involved, this
area code physically traces back to Germany, likely Berlin instead of
Frankfurt.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 17 Feb 2024 21:12:32 +0000 (21:12 +0000)]
network: Fix bit length check when merging networks
The check was incorrect as it only expected perfect matches, but we can
of course also merge subnets that don't necessarily require the entire
prefix length for the start address.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
projects / location / libloc.git / log
