]>
git.ipfire.org Git - location/libloc.git/log
Michael Tremer [Tue, 24 Nov 2020 16:57:28 +0000 (16:57 +0000)]
network-list: Use binary search to find if a network is a subnet
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 16:50:17 +0000 (16:50 +0000)]
network: Adjust return codes of loc_network_match_address and add test
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 15:50:39 +0000 (15:50 +0000)]
Drop loc_network_gt which is now unused
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 15:48:55 +0000 (15:48 +0000)]
Drop loc_network_eq in favour of loc_network_cmp
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 15:42:54 +0000 (15:42 +0000)]
test: Add more networks to list to see the algorithm work
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 15:41:53 +0000 (15:41 +0000)]
network: Add excluded networks to to_check list
This is now being done immediately instead of creating a new
list which is being merged later.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 15:22:02 +0000 (15:22 +0000)]
database: Read the first element from the list
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 15:15:59 +0000 (15:15 +0000)]
database: Avoid merging the same data twice
When finish splitting networks into many parts, we have
a list of subnets with the excluded subnets and merge them
together first and put them on the stack again.
This is slower than pushing it all onto the stack first
and then popping the first element.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 15:04:03 +0000 (15:04 +0000)]
network-list: Drop sorting functions
Since the list is always sorted, there is no point in
sorting it again...
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 14:55:08 +0000 (14:55 +0000)]
network-list: Make this a sorted list
The list is now organised so that it is always ordered.
This allows us to find if a network is on the list a lot
quicker using binary search as well as inserting a new
network at the right place.
This will benefit loc_network_exclude with very large networks.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 18:44:42 +0000 (18:44 +0000)]
network: Remove deprecated loc_network_is_subnet_of function
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 18:39:13 +0000 (18:39 +0000)]
network: Speed up subnet check
There is no point in checking different address families
with each other and we do not need to compare addresses
when the prefix of the subnet does not fit into the
network to check.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 18:38:47 +0000 (18:38 +0000)]
network: Add function to return the prefix
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 18:36:48 +0000 (18:36 +0000)]
network: Optimise _subnet function
This function used to create a network list which always
had exactly two elements. Since splitting a network in half
always returns two parts, we can simply return them as a
pointer.
This improves returning the network tree by about 17%.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 16:42:55 +0000 (16:42 +0000)]
network-list: Implement merging in reverse in one step
This will save us some time because we do not need to
change the list in place first and then merge it.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 16:25:56 +0000 (16:25 +0000)]
database: Flatten out code due to compiler errors
It looks like GCC could not follow the code as it was
written. Therefore I clean up more often now instead
of having a GOTO block where I do that in.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 16:25:27 +0000 (16:25 +0000)]
network: Remove debugging output
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 16:24:40 +0000 (16:24 +0000)]
network: Fix loc_network_is_subnet()
This function always returned false.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 14:43:31 +0000 (14:43 +0000)]
test: Add tests to network-lists
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 19 Nov 2020 12:48:46 +0000 (12:48 +0000)]
database: Disable some useless code when not running in debug mode
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 19 Nov 2020 12:41:19 +0000 (12:41 +0000)]
export: Speed-up export in xt_geoip format
Removing the loop avoids creating a tuple and then iterating over it
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 19 Nov 2020 12:40:01 +0000 (12:40 +0000)]
export: Remove old flattening feature
The database enumerator now only returns networks that will
never overlap.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 19 Nov 2020 12:34:11 +0000 (12:34 +0000)]
python: Add property to return IP addresses as bytes
This avoids calling inet_pton to parse IP addresses from string
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 19 Nov 2020 12:03:33 +0000 (12:03 +0000)]
python: Fix download of database
This was all messed up in
0c74f6b1a3bdce5ebdc2ee452b9baf3e421dd3d1
when the change of type for the timestamp wasn't changed everywhere
else.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 18 Nov 2020 13:34:50 +0000 (13:34 +0000)]
location: End lookup after an invalid IP address was passed
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 18 Nov 2020 13:33:45 +0000 (13:33 +0000)]
python: Remove unnecessary db object from writers
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 18 Nov 2020 13:19:04 +0000 (13:19 +0000)]
export: Remove filtering for flags
The filter is an AND filter and if we set the flags from
the special country codes, we won't get back much.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 18 Nov 2020 13:18:52 +0000 (13:18 +0000)]
database: Add debug output to filtering
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 19:01:04 +0000 (19:01 +0000)]
database: Free filter lists in enumerator
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 18:34:51 +0000 (18:34 +0000)]
database: Do not clean up python list
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 18:14:15 +0000 (18:14 +0000)]
Actually clear all lists
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 18:13:49 +0000 (18:13 +0000)]
export: Change back to use Network objects
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 17:57:55 +0000 (17:57 +0000)]
as: Make lists grow dynamically
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 17:55:51 +0000 (17:55 +0000)]
networks: Make list grow dynamically
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 17:50:17 +0000 (17:50 +0000)]
countries: Make list grow dynamically
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 16:58:55 +0000 (16:58 +0000)]
database: Simplify AS matching code
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 16:56:43 +0000 (16:56 +0000)]
database: Simplify network matching code
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 17 Nov 2020 16:46:48 +0000 (16:46 +0000)]
as: Add list for easier processing
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 16 Nov 2020 15:25:15 +0000 (15:25 +0000)]
database: Filter flags in C
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 16 Nov 2020 15:20:50 +0000 (15:20 +0000)]
python: Only return country codes we want
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 16 Nov 2020 15:13:28 +0000 (15:13 +0000)]
Implement filtering for multiple countries in the enumerator
This will allow us to speed up the export of the database
if only a few countries should be returned.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sun, 15 Nov 2020 15:02:28 +0000 (15:02 +0000)]
Move network lists into an own file
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 13 Nov 2020 12:09:03 +0000 (12:09 +0000)]
python: Move tree flattening into C
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 13 Nov 2020 11:43:53 +0000 (11:43 +0000)]
network: Unexport all tree functions
These should not be exported
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 13 Nov 2020 11:38:15 +0000 (11:38 +0000)]
network: Add function to pop first element from stack
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 13 Nov 2020 11:29:15 +0000 (11:29 +0000)]
network: Sort result of excluded lists
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 13 Nov 2020 11:29:02 +0000 (11:29 +0000)]
database: Filter results coming from stack
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 13 Nov 2020 11:23:33 +0000 (11:23 +0000)]
database: Move network filtering into a separate function
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 20:09:37 +0000 (20:09 +0000)]
networks: Copy all attributes when splitting networks
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 20:09:20 +0000 (20:09 +0000)]
test: Update API
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 20:02:03 +0000 (20:02 +0000)]
database: Add option to return networks flattened
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 20:01:17 +0000 (20:01 +0000)]
network: Add function to exclude multiple networks at once
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 20:00:09 +0000 (20:00 +0000)]
network: Add new subnet function
The old one is too difficult to use in terms of order
of input parameters and return value.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 19:59:22 +0000 (19:59 +0000)]
network: Add more debugging output to stacks
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 19:36:38 +0000 (19:36 +0000)]
python: Export networks exclude function
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 19:21:58 +0000 (19:21 +0000)]
network: Reduce debugging output
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 19:21:13 +0000 (19:21 +0000)]
database: Pass flag to enumerator to flatten output
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 14:35:43 +0000 (14:35 +0000)]
network: Make lists unique
Networks that are in the list won't be added again
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 14:33:22 +0000 (14:33 +0000)]
networks: Add function to merge two lists
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 14:28:15 +0000 (14:28 +0000)]
networks: Add function to check if network is part of a list
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 14:24:58 +0000 (14:24 +0000)]
networks: Add function to check if two networks overlap
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 14:18:40 +0000 (14:18 +0000)]
network: Add functions to break network into subnets
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 12 Nov 2020 13:57:35 +0000 (13:57 +0000)]
networks: Add function to dump lists
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 11 Nov 2020 23:01:19 +0000 (23:01 +0000)]
networks: Add list to manage groups of networks
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 11 Nov 2020 21:16:45 +0000 (21:16 +0000)]
test: Add tests for database enumerator
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Tue, 3 Nov 2020 15:31:08 +0000 (15:31 +0000)]
location-importer.in: always convert organisation handles into upper cases
Fixes: #12523
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Thu, 29 Oct 2020 14:36:46 +0000 (07:36 -0700)]
location.in: do not confuse UTC with local time zones
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Peter Müller [Thu, 29 Oct 2020 14:25:53 +0000 (07:25 -0700)]
location update: Remove double conversion of timestamps
This caused that the timestamp in DNS was misinterpreted
as local time and often, databases could not be downloaded.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Michael Tremer [Wed, 28 Oct 2020 09:52:36 +0000 (09:52 +0000)]
location: Fix Python syntax error in verify()
The database is now being opened before the requested
method is called and handle_verify() wasn't updated.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 27 Oct 2020 17:14:30 +0000 (17:14 +0000)]
export: Flatten the tree before exporting it
This patch removes the possibility that any IP address ranges
might show up in multiple exported files.
If this was content from the database:
* 10.0.0.0/16 - DE
* 10.0.1.0/24 - FR
Then the IP address 10.0.1.1 would match for both countries.
The algorithm will now break the larger /16 subnet apart into
smaller subnets so that 10.0.1.0/24 is no longer overlapped.
There was some time spent on this to make this as efficient
as possible.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Wed, 21 Oct 2020 14:47:43 +0000 (14:47 +0000)]
location-importer.in: avoid log spam for too small networks
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Wed, 21 Oct 2020 14:47:41 +0000 (14:47 +0000)]
location-importer.in: omit historic/orphaned RIR data
Some RIRs include detailled information regarding networks not managed
by or allocated to themselves, particually APNIC. We need to filter
those networks (they usually have a characteristic network name) in
order to prevent operational quirks or returning wrong country codes.
Fixes: #12501
Partially fixes: #12499
Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Wed, 21 Oct 2020 14:47:40 +0000 (14:47 +0000)]
importer.py: fetch LACNIC data via HTTPS
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Wed, 21 Oct 2020 14:47:39 +0000 (14:47 +0000)]
location-importer.in: filter bogus IP networks for both Whois and extended sources
Sanity checks for parsed networks have been put into a separate function
to avoid boilerplate code for extended sources. This makes the location
database less vulnerable to garbage written into RIR databases on
purpose or by chance.
Fixes: #12500
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Wed, 21 Oct 2020 14:47:38 +0000 (14:47 +0000)]
export.py: fix exporting IP networks for crappy xt_geoip module
In contrast to the location database itself, the xt_geoip module
consumes a list of IP networks for each country, and returns after the
first match.
We therefore need to...
(a) sort IP networks by their size, allow as precise matches as possible
(b) export _any_ IP networks - including inverted subnets - to prevent
undefined overlaps
(c) do the entire thing as fast as possible, consuming as less disk
space as possible, which is why we can't just iterate over all /24
chunks
Partially fixes: #12499
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Wed, 21 Oct 2020 14:47:37 +0000 (14:47 +0000)]
Revert "Revert "location-importer.in: only import relevant data from AFRINIC, APNIC and RIPE""
This reverts commit
13f67f285856e8eabfeff2daf1be3aeaa36a82cc .
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Wed, 21 Oct 2020 14:47:36 +0000 (14:47 +0000)]
Revert "Revert "Revert "Revert "importer: Import raw sources for inetnum's again""""
This reverts commit
44341478233115b26bb27fdb24da5b0a1eedb173 .
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 22 Oct 2020 12:24:34 +0000 (12:24 +0000)]
network: Allow adding single IP addresses and automatically add the prefix
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 16:01:57 +0000 (16:01 +0000)]
importer: Restructure SQL query to be executed in parallel
There are no functional changes, this just runs quicker now.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 13:54:45 +0000 (13:54 +0000)]
importer: Add search index to network_overrides table
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 13:53:36 +0000 (13:53 +0000)]
importer: Add search index to announcements table
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 13:44:50 +0000 (13:44 +0000)]
networks: Remove accidentially committed debug line
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 13:43:21 +0000 (13:43 +0000)]
networks: Test if we can add localhost (IPv6)
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 13:36:35 +0000 (13:36 +0000)]
networks: Improve parsing IP addresses
loc_network_new_from_string() seem to have had some unexpected
behaviour for invalid inputs.
The function has been tidied up slightly and returns as soon as
some invalid input was detected.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 13:19:44 +0000 (13:19 +0000)]
tests: Try adding an invalid network
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 09:31:29 +0000 (09:31 +0000)]
writer: Free countries when the writer is being destroyed
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 09:28:39 +0000 (09:28 +0000)]
writer: Free array with pointer to ASes, too
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 21 Oct 2020 09:18:08 +0000 (09:18 +0000)]
as: Do not attempt to match name when it wasn't set
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 20 Oct 2020 20:44:43 +0000 (20:44 +0000)]
as: Fix dereferencing NULL pointer when setting AS name
Reported-by: Gisle Vanem <gisle.vanem@gmail.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 16 Oct 2020 12:26:43 +0000 (12:26 +0000)]
Revert "Revert "Revert "importer: Import raw sources for inetnum's again"""
This reverts commit
f532841e9197ce2f40aad8c086d786b2cb783a54 .
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 16 Oct 2020 12:26:38 +0000 (12:26 +0000)]
Revert "location-importer.in: only import relevant data from AFRINIC, APNIC and RIPE"
This reverts commit
a36bc686865fc87ea386fd90b389338bdcb80cbc .
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 16 Oct 2020 12:24:58 +0000 (12:24 +0000)]
location-importer: Include all overridden networks
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 12 Oct 2020 20:53:32 +0000 (20:53 +0000)]
location-importer.in: only import relevant data from AFRINIC, APNIC and RIPE
In contrast to ARIN and LACNIC, we are able to process more detailled
feeds from those RIRs, avoiding storage of obviously unnecessary data.
Thanks to various SQL optimisations, doing so now takes less time than
the first version of this did.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 12 Oct 2020 20:53:31 +0000 (20:53 +0000)]
Revert "Revert "importer: Import raw sources for inetnum's again""
This reverts commit
92f6abf4e272672bb0a71cfe991261b95ebe2fef .
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 24 Sep 2020 10:18:58 +0000 (10:18 +0000)]
Revert "importer: Import raw sources for inetnum's again"
This reverts commit
64e95fa903edec8b4e4e59830b395e2e4a411853 .
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 24 Sep 2020 10:17:58 +0000 (10:17 +0000)]
Revert "importer: Purge any redundant entries"
This reverts commit
c2cc55d5a6875c3838f060032eaed89dcfb92ef6 .
The query stalls the database and therefore the automatic
scripts are no longer able to generate a new version of the
database.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 21 Sep 2020 16:21:21 +0000 (16:21 +0000)]
Bump version to 0.9.4
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 20 Sep 2020 19:21:03 +0000 (19:21 +0000)]
importer: Purge any redundant entries
When importing inetnums, we might import various small networks
which are not relevant for us as long as they do not have a
different country code than their parent network.
Therefore we delete all these entries to keep the database
smaller without losing any information. The second version of this
patch introduces a SQL statement parallelised across all CPUs
available, while the DELETE-statement of the first version literally
took ages to complete.
However, cleaning up those data still takes about 26 hours (!) on
our location02 testing machine, making daily updates of the location
database impossible to the current knowledge.
real 1521m30.620s
user 38m45.521s
sys 9m6.027s
Special thanks goes to Michael for spending numerous hours
on this, setting up a testing environment, doing PostgreSQL magic
and providing helpful advice while debugging.
Partially fixes: #12458
Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 20 Sep 2020 19:20:18 +0000 (19:20 +0000)]
importer: Import raw sources for inetnum's again
The extended feeds do not have enough detailed information
for us, so that we need to import inetnums from RIRs where
possible. Filtering private networks is necessary as RIR data
may contain 0.0.0.0/0 or similar entries for administrative
purposes or due to misfilings.
Special thanks goes to Michael for spending numerous hours
on this, setting up a testing environment and providing helpful
advice while debugging.
Partially fixes: #12458
Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 20 Sep 2020 19:19:36 +0000 (19:19 +0000)]
location-importer.in: avoid violating NOT NULL constraints during JOIN
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>