Michael Tremer [Wed, 9 Mar 2022 10:29:11 +0000 (10:29 +0000)]
importer: Improve performance of network export query
This patch moves the subqueries out of the large query, so that the
database will materialize them for faster lookup.
We also drop the "UNION ALL" and replace it with just "UNION" because we
do not want any duplicate networks. That will save us many iterations
later on.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Mon, 7 Mar 2022 11:12:17 +0000 (11:12 +0000)]
bogons: Refactor algorithms
This changes that we won't compare one network with the previous one,
but instead we will look for gaps starting from the first possible to
the last possible IP address.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Sat, 5 Mar 2022 11:56:40 +0000 (11:56 +0000)]
importer: Parse aggregated networks
This patch adds code to parse any aggregated networks.
Bird does not automatically show the last ASN of the path, but we can
collect all networks that we can see without any ASN and perform
"show route <network> all" on them to gather this information.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 3 Mar 2022 08:48:14 +0000 (08:48 +0000)]
export: Fix filtering logic
It is possible to filter for what kind of network should be exported.
This worked well when the filter list only contained country codes, or
when it only contained ASNs. If there was a mix, only networks that
match both (i.e. virtually nothing) matched.
This patch fixes that we will use for either of them.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Reported-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 2 Mar 2022 10:26:41 +0000 (10:26 +0000)]
export: Conditionally enable flattening
By default, we enabled flattening of the network tree when we export it.
However, this is only required for xt_geoip since the other formats can
deal with overlapping networks and would even benefit from a shorter
list.
Therefore this is now only enabled when needed which results in shorter
export times (9 seconds instead of 2.5 minutes) and the full ipset is
about 20% smaller when loaded into memory than before.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 2 Mar 2022 10:18:16 +0000 (10:18 +0000)]
ipset: Set maxelem to a fixed size
When we try to load a changed set which might have more entries, a
previous maxelem could have been smaller preventing us from adding new
entries.
We also cannot run the "create" command with a changed maxelem
parameter which is why this patch set the value to something that should
be large enough for everything.
The downside of this is also, that we cannot modify the hashsize when we
reload a set, which is probably okay, since sets should not change too
much in size and therefore will only run *slightly* less efficient - if
at all.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Mar 2022 12:44:21 +0000 (12:44 +0000)]
ipset: Optimise hash table size
ipset uses a hash table internally which can be dynamically sized to
chose whether more space efficiency or performance is required.
Previously to this patch, we always set the size of the hash table to
1024 buckets. Having large sets with almost half a million entries, this
is not performing well since we will spend a lot of time in searching
the linked list.
This will probably perform even slower on systems with smaller cache
sizes like the IPFire Mini Appliance.
Having more buckets that are sparesely filled, will result in less
memory fetches at the cost of more wastage. Throughout the whole IPv4
set, this ranges from about 50 MB for a factor of 4, to about 100 MB for
a factor of 0.75.
Since memory of this quantity is cheap and since we want to increase
throughput, I have chosen to set the fill factor to 0.75.
Logistically, it is a little bit complicated to know this in advance
when we have to write the header, so we will write the entire file
first, and then come back to write the header again. This is required to
keep memory consumption down during the export.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 18 Dec 2021 12:57:45 +0000 (13:57 +0100)]
location-importer.in: Do not make things more complicated than they are
Suggested-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Reviewed-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Fri, 11 Feb 2022 09:57:47 +0000 (09:57 +0000)]
location-importer.in: Add country code for AWS's "il-central-1" zone
Reported-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 11 Dec 2021 21:59:22 +0000 (22:59 +0100)]
Process LACNIC geofeed as well
This improves country code accurarcy for suballocations within IP space
managed by LACNIC, as the delegated-extended-latest file only provides
country code information at the top level of an allocated network.
Sadly, lacnic.db.gz does not contain descriptions or names of Autonomous
Systems within the space maintained by LACNIC.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 11 Dec 2021 18:01:54 +0000 (19:01 +0100)]
location-importer: Set "is_drop" to "True" even in case of conflicts
Previously, any present override for a given network or ASN would have
caused the SQL statement not to conduct anything at all. Since "is_drop"
is the only flag being actually set here, it makes sense to do so in
case of already present overrides as well.
The effect of this is limited: Our own override files are always
considered at last, so in case of conflicts they will be the ultima
ratio. This is an intended behaviour, but slipped my mind when I filed
bug #12728, so this patch can only be seen as a partial solution - the
rest is not a bug, but a feature. :-)
Partially fixes: #12728
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 1 Nov 2021 18:24:37 +0000 (19:24 +0100)]
location-importer.in: Add Spamhaus DROP lists
A while ago, it was discussed whether or not libloc should become an
"opinionated database", i. e. including any information on a network's
reputation.
In general, this idea was dismissed as libloc is neither intended nor
suitable for such tasks, and we do not want to make (political?)
decisions like these for various reasons. All we do is to provide a
useful location database in a neutral way, and leave it up to our users
on how to react on certain results.
However, there is a problematic area. Take AS55303 as an example: We
_know_ this is to be a dirty network, tampering with RIR data and
hijacking IP space, and strongly recommend against processing any
connection originating from or directed to it.
Since it appears to be loaded with proxies used by miscreants for
abusive purposes, all we can do at the time of writing is to flag it
as "anonymous proxy", but we lack possibility of telling our users
something like "this is not a safe area". The very same goes for known
bulletproof ISPs, IP hijackers, and so forth.
This patch therefore suggests to populate the "is_drop" flag introduced
in libloc 0.9.8 (albeit currently unused in production) with the
contents of Spamhaus' DROP lists (https://www.spamhaus.org/drop/), to
have at least the baddest of the bad covered. The very same lists are,
in fact, included in popular IPS rulesets as well - a decent amount of
IPFire users is therefore likely to have them already enabled, but in a
very costly way.
It is not planned to go further, partly because there is no other feed
publicly available, which would come with the same intention,
volatility, and FP rate.
The third version of this patch makes use of an auxiliary function to
sanitise ASNs, hence avoiding boilerplate code, and treats any line
starting with a semicolon as a comment, which should be sufficient.
Further, extracting ASNs from the ASN-DROP feed is done in a more clear
way, avoiding code snippets hard to read.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Thu, 23 Sep 2021 10:23:50 +0000 (13:23 +0300)]
debian: Ensure changelog distribution is tagged
UNRELEASED should not be left as-is when actually releasing.
The latest changelog entry now point at unstable instead.
The simple d/genchangelog.sh now does `dch -r ''` automatically
to ensure this distribution update doesn't get lost along the way
on future invocations.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 8 Aug 2021 21:31:58 +0000 (23:31 +0200)]
location-importer.in: Braindead me accidentally forgot a "break" statement
This one apparently went down the drain between these two patches:
- https://patchwork.ipfire.org/project/location/patch/20210522125758.28770-1-peter.mueller@ipfire.org/
- https://patchwork.ipfire.org/project/location/patch/aefd1904-4b38-f5cf-ab1d-9d69636cf914@ipfire.org/
Due to other safeguards, the current damage in production is limited to:
Peter Müller [Mon, 19 Jul 2021 21:34:40 +0000 (21:34 +0000)]
location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
A decent amount of autnum objects - especially, but not exclusively in
the APNIC sector - does not contain a link to an organisation handle.
In such cases, this patch is going to use the first description line of
the atunum object in question (if available) as a string for its name.
The overwhelming majority of affected ASNs contains a valuable
information there, so this is almost as good as having an organisation
handle linked to it.
Fixes: #12660 Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>