Michael Tremer [Sat, 5 Mar 2022 11:56:40 +0000 (11:56 +0000)]
importer: Parse aggregated networks
This patch adds code to parse any aggregated networks.
Bird does not automatically show the last ASN of the path, but we can
collect all networks that we can see without any ASN and perform
"show route <network> all" on them to gather this information.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 3 Mar 2022 08:48:14 +0000 (08:48 +0000)]
export: Fix filtering logic
It is possible to filter for what kind of network should be exported.
This worked well when the filter list only contained country codes, or
when it only contained ASNs. If there was a mix, only networks that
match both (i.e. virtually nothing) matched.
This patch fixes that we will use for either of them.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Reported-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 2 Mar 2022 10:26:41 +0000 (10:26 +0000)]
export: Conditionally enable flattening
By default, we enabled flattening of the network tree when we export it.
However, this is only required for xt_geoip since the other formats can
deal with overlapping networks and would even benefit from a shorter
list.
Therefore this is now only enabled when needed which results in shorter
export times (9 seconds instead of 2.5 minutes) and the full ipset is
about 20% smaller when loaded into memory than before.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 2 Mar 2022 10:18:16 +0000 (10:18 +0000)]
ipset: Set maxelem to a fixed size
When we try to load a changed set which might have more entries, a
previous maxelem could have been smaller preventing us from adding new
entries.
We also cannot run the "create" command with a changed maxelem
parameter which is why this patch set the value to something that should
be large enough for everything.
The downside of this is also, that we cannot modify the hashsize when we
reload a set, which is probably okay, since sets should not change too
much in size and therefore will only run *slightly* less efficient - if
at all.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Mar 2022 12:44:21 +0000 (12:44 +0000)]
ipset: Optimise hash table size
ipset uses a hash table internally which can be dynamically sized to
chose whether more space efficiency or performance is required.
Previously to this patch, we always set the size of the hash table to
1024 buckets. Having large sets with almost half a million entries, this
is not performing well since we will spend a lot of time in searching
the linked list.
This will probably perform even slower on systems with smaller cache
sizes like the IPFire Mini Appliance.
Having more buckets that are sparesely filled, will result in less
memory fetches at the cost of more wastage. Throughout the whole IPv4
set, this ranges from about 50 MB for a factor of 4, to about 100 MB for
a factor of 0.75.
Since memory of this quantity is cheap and since we want to increase
throughput, I have chosen to set the fill factor to 0.75.
Logistically, it is a little bit complicated to know this in advance
when we have to write the header, so we will write the entire file
first, and then come back to write the header again. This is required to
keep memory consumption down during the export.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 18 Dec 2021 12:57:45 +0000 (13:57 +0100)]
location-importer.in: Do not make things more complicated than they are
Suggested-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Reviewed-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Fri, 11 Feb 2022 09:57:47 +0000 (09:57 +0000)]
location-importer.in: Add country code for AWS's "il-central-1" zone
Reported-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 11 Dec 2021 21:59:22 +0000 (22:59 +0100)]
Process LACNIC geofeed as well
This improves country code accurarcy for suballocations within IP space
managed by LACNIC, as the delegated-extended-latest file only provides
country code information at the top level of an allocated network.
Sadly, lacnic.db.gz does not contain descriptions or names of Autonomous
Systems within the space maintained by LACNIC.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 11 Dec 2021 18:01:54 +0000 (19:01 +0100)]
location-importer: Set "is_drop" to "True" even in case of conflicts
Previously, any present override for a given network or ASN would have
caused the SQL statement not to conduct anything at all. Since "is_drop"
is the only flag being actually set here, it makes sense to do so in
case of already present overrides as well.
The effect of this is limited: Our own override files are always
considered at last, so in case of conflicts they will be the ultima
ratio. This is an intended behaviour, but slipped my mind when I filed
bug #12728, so this patch can only be seen as a partial solution - the
rest is not a bug, but a feature. :-)
Partially fixes: #12728
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 1 Nov 2021 18:24:37 +0000 (19:24 +0100)]
location-importer.in: Add Spamhaus DROP lists
A while ago, it was discussed whether or not libloc should become an
"opinionated database", i. e. including any information on a network's
reputation.
In general, this idea was dismissed as libloc is neither intended nor
suitable for such tasks, and we do not want to make (political?)
decisions like these for various reasons. All we do is to provide a
useful location database in a neutral way, and leave it up to our users
on how to react on certain results.
However, there is a problematic area. Take AS55303 as an example: We
_know_ this is to be a dirty network, tampering with RIR data and
hijacking IP space, and strongly recommend against processing any
connection originating from or directed to it.
Since it appears to be loaded with proxies used by miscreants for
abusive purposes, all we can do at the time of writing is to flag it
as "anonymous proxy", but we lack possibility of telling our users
something like "this is not a safe area". The very same goes for known
bulletproof ISPs, IP hijackers, and so forth.
This patch therefore suggests to populate the "is_drop" flag introduced
in libloc 0.9.8 (albeit currently unused in production) with the
contents of Spamhaus' DROP lists (https://www.spamhaus.org/drop/), to
have at least the baddest of the bad covered. The very same lists are,
in fact, included in popular IPS rulesets as well - a decent amount of
IPFire users is therefore likely to have them already enabled, but in a
very costly way.
It is not planned to go further, partly because there is no other feed
publicly available, which would come with the same intention,
volatility, and FP rate.
The third version of this patch makes use of an auxiliary function to
sanitise ASNs, hence avoiding boilerplate code, and treats any line
starting with a semicolon as a comment, which should be sufficient.
Further, extracting ASNs from the ASN-DROP feed is done in a more clear
way, avoiding code snippets hard to read.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Thu, 23 Sep 2021 10:23:50 +0000 (13:23 +0300)]
debian: Ensure changelog distribution is tagged
UNRELEASED should not be left as-is when actually releasing.
The latest changelog entry now point at unstable instead.
The simple d/genchangelog.sh now does `dch -r ''` automatically
to ensure this distribution update doesn't get lost along the way
on future invocations.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 8 Aug 2021 21:31:58 +0000 (23:31 +0200)]
location-importer.in: Braindead me accidentally forgot a "break" statement
This one apparently went down the drain between these two patches:
- https://patchwork.ipfire.org/project/location/patch/20210522125758.28770-1-peter.mueller@ipfire.org/
- https://patchwork.ipfire.org/project/location/patch/aefd1904-4b38-f5cf-ab1d-9d69636cf914@ipfire.org/
Due to other safeguards, the current damage in production is limited to:
Peter Müller [Mon, 19 Jul 2021 21:34:40 +0000 (21:34 +0000)]
location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
A decent amount of autnum objects - especially, but not exclusively in
the APNIC sector - does not contain a link to an organisation handle.
In such cases, this patch is going to use the first description line of
the atunum object in question (if available) as a string for its name.
The overwhelming majority of affected ASNs contains a valuable
information there, so this is almost as good as having an organisation
handle linked to it.
Fixes: #12660 Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Sun, 11 Jul 2021 16:50:24 +0000 (19:50 +0300)]
debian: Clean up 0.9.7 changelog
- Update for maintainer name and email address, as to reflect who
actually prepared this release of the package. It was not me,
but I was selected due to having the first commit on the package.
To mitigate against this, when running the `debchange --release`
(`dch -r`) command, environment variables DEBFULLNAME and DEBEMAIL
should be configured properly for the current user.
- Removal of NMU comment on my name, as I am not really doing a
non-maintainer upload. I would say the 'NMU' message is fairly
useless on this repository, as it is self-maintained here.
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Sun, 13 Jun 2021 16:16:25 +0000 (19:16 +0300)]
debian: Attribute all maintainers in changlog
This commit further builds on historical changelog modifications,
to properly attribute all authors of the commits.
An additional d/genchangelog.sh script has been added. This allows
generation of changelog entries, internally using `debchange` (`dch`).
The script accepts an argument, which is the commit range to generate
entries for. Each commit's subject line (first line of body) is used,
along with author name and email. This information is added to the
changelog. Automatic detection (via `debchange` built-in functionality)
is used to determine whether these entries should be added to an
existing version number. If there is no UNRELEASED version, then a new
version is automatically tagged.
The new version tag will usually need to be modified, for example,
replacing an automatically generated 0.9.6-2 with 0.9.7-1.
The final release change (s/UNRELEASED/unstable/) needs to be done
manually as well, when the Git tag is actually being tagged.
`dch -r` can be useful for this particular purpose.
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Reported-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 11 Jun 2021 07:51:07 +0000 (10:51 +0300)]
debian: Add dpkg's symbols file
There are muiltiple standards of listing symbols throughout the Linux
ecosystem. For `dpkg`, a d/package.symbols file tracks symbols, and in
which version they were added in. This is then used to allow dependency
checks/resolution.
See man:dpkg-gensymbols(1) for details about the generation,
and man:dpkg-shlibdeps(1) for how the symbols file ends up being used.
This commit adds a d/libloc1.symbols file, containing the current state
of the symbols. There is now also a d/gensymbols.sh script, which
generates this symbols file. The script tries to determine what Git
tags need to be checked for changes in symbols, by looking at current
maximum version referenced in symbols file.
After checking tags, the current revision is also processed, to allow
building symbols file for a yet unreleased version (prior to tagging it).
This is to allow symbols changes to be included in a tag.
Do keep in mind, that for the workflow above, when running the script,
the d/changelog file should contain information about what version the
current revision will be released at (potentially tagged as UNRELEASED
in the d/changelog file). Otherwise, if there is no version tagged,
the `dpkg-gensymbols` tool will use the old version information,
in turn incorrectly attributing new symbols to an old version.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 10 Jun 2021 09:37:22 +0000 (09:37 +0000)]
countries: Fix matching invalid country codes
When an invalid country code is entered, loc_country_new returns an
error which is interpreted as a match to the list since we check for a
non-zero return code.
Any invalid country codes are now silently ignored and not considered a
match.
Peter Müller [Tue, 8 Jun 2021 09:55:41 +0000 (09:55 +0000)]
location-importer.in: import additional IP information for Amazon AWS IP networks
Amazon publishes information regarding some of their IP networks
primarily used for AWS cloud services in a machine-readable format. To
improve libloc lookup results for these, we have little choice other
than importing and parsing them.
Unfortunately, there seems to be no machine-readable list of the
locations of their data centers or availability zones available. If
there _is_ any, please let the author know.
The second version of this patch adds a meaningful description for the
"source" column in the overrides tables, to make introduced changes
less intransparent.
Fixes: #12594 Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Tue, 8 Jun 2021 09:55:40 +0000 (09:55 +0000)]
location-importer.in: add source column for overrides as well
This allows us to track changes introduced by IP feeds from 3rd parties,
such as Amazon AWS, on the SQL server side.
In order not to break existing tables (which would required TRUNCATE),
there currently is no constraint set for the new column, but "NOT NULL"
is planned in the future.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Tue, 8 Jun 2021 17:03:07 +0000 (17:03 +0000)]
location-importer.in: Import (technical) AS names from ARIN
ARIN and LACNIC, unfortunately, do not seem to publish data containing
human readable AS names. For the former, we at least have a list of
tecnical names, which this patch fetches and inserts into the autnums
table.
While some of them do not seem to be suitable for human consumption (i.
e. being very cryptic), providing these data might be helpful
neverthelesss.
The second version of this patch contains some additional remarks on
efficient Python coding style from Michael, doing things more "pythonic".
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Fri, 4 Jun 2021 15:57:30 +0000 (17:57 +0200)]
Implement an additional flag for hostile networks safe to drop
This patch implements an additional flag intended for networks and
Autonomous Systems being considered hostile. While libloc does not and
should not be an opinionated database, reality shows it is being used
this way.
Hereby, we assign "XD" (drop) as a custom country code for networks
being flagged this way. According to ISO, "XA" to "XZ" are reserved for
"user-assgined codes" (https://www.iso.org/glossary-for-iso-3166.html),
so this is a safe thing to do.
This patch does not interfere with "A1" to "A3", which we currently
assign outside standardised country code ranges for historical reasons.
Neither does it specify any policy or source for tagging networks with a
"drop" flag. Doing so is beyond the scope of this - technical -
approach.
To avoid confusions with the SQL "DROP" command, "is_drop" will be used
as a column name for database operations.
Thanks to Michael for his remarks and ideas during the run-up.
Cc: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Reported-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 30 May 2021 08:50:04 +0000 (10:50 +0200)]
location-importer.in: track original countries as well
This helps us to determine how many network objects have more than one
country set, and what their original country code set looked like.
The third version of this patch uses ALTER TABLE to add the column for
original countries, preventing existing SQL setups from breaking, and is
correctly based against the current "master" branch.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 22 May 2021 20:33:51 +0000 (20:33 +0000)]
location-importer.in: keep track of sources for networks, ASNs, and organisations
This allows us to trace back concrete changes or anomalies to their RIR
source, without having to parse everything again. Further, it enables
adding 3rd party sources such as IP feeds from Amazon, without loosing
track of the changes introduced by them.
The second version of this patchset uses ALTER TABLE to add the source
columns, avoiding breaking existing SQL setups.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 3 May 2021 17:14:29 +0000 (19:14 +0200)]
location-importer.in: emit warnings due to unknown country code for valid networks only
This reduces log spam in case of processing RIR database, checking for
networks with unknown country codes assigned. If we would not have
written into the database, there is no need to warn about them.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:10 +0000 (16:06 +0300)]
debian: Drop unintended files from location-python
_location.la gets built and installed to site-packages/, however
an .la file is not expected to reside in the Python root. Additionally,
the dependency library listed does not have its respective .la file
installed. Further complicating the situation, dh-python moves the
site-packages/ files to dist-packages/ silently which then results in
a broken libdir left behind in the .la file.
The only reason the file is there is that it gets built inside the
source directory, which gets copied entirely to location-python package
as-is. Considering the situation, this commit ensures the .la files is
not packaged by deleting it from the package files subdirectory.
location-importer package pulls in two Python (.py) files from the
source directory. These files should not be included in the
location-python package as a result.
Valters Jansons [Fri, 16 Apr 2021 13:06:05 +0000 (16:06 +0300)]
debian: Add all temporary files to Gitignore
New packages have been added since the inception of the .gitignore and
as a result during build we see directories such as location-importer/
and files such as location-importer.debhelper.log.
This commit ensures all temporary subdirectories, and additional
generic build artifact files, are ignored by Git.
The subdirectory exceptions to this rule are:
- d/patches/ which may be used by Quilt
considering the source format is '3.0 (quilt)',
- d/source/ for the format file,
- d/tests/ which may be used by autopkgtest
to specify what test suites exist for the source.
See: https://salsa.debian.org/ci-team/autopkgtest/-/raw/debian/5.16/doc/README.package-tests.rst
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:12 +0000 (16:06 +0300)]
systemd: Add Documentation= to location-update
Systemd units are expected to provide some documentation information
such as manpages, or direct links, which provide more details about
that unit. This commit simply links location-update.service to the
manual for location(8) followed by a fallback to the online manual.
Valters Jansons [Fri, 16 Apr 2021 13:06:11 +0000 (16:06 +0300)]
debian: Add watch configuration for uscan
Packages defined as '3.0 (quilt)' are expected to provide information
about how the latest upstream information can be obtained,
as a special d/watch file. This can then get used by uscan(1).
To see how the metadata is utilized, and how the network requests
are made behind the scenes, you can locally run:
$ uscan --no-download --verbose --debug
Resolves: lintian: debian-watch-file-is-missing
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:07 +0000 (16:06 +0300)]
debian: Set 'Multi-Arch: foreign' hint for Python
Due to the invocation of py3compile (via dh-python) in location-importer
and location-python packages, those packages have different bytecode for
varying architectures, and as a result are not 'Multi-Arch: same'.
Valters Jansons [Thu, 15 Apr 2021 11:42:13 +0000 (14:42 +0300)]
po: Update translations
POTFILES.in should not contain src/python/__init__.py file as it
is not present in the committed tree. It has its respective .in file
which is present instead.
This commit further ensures po/POTFILES.in generator avoids such
files that Git ignores (using git-check-ignore during find).
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
projects / location / libloc.git / log
