Peter Müller [Mon, 1 Nov 2021 18:24:37 +0000 (19:24 +0100)]
location-importer.in: Add Spamhaus DROP lists
A while ago, it was discussed whether or not libloc should become an
"opinionated database", i. e. including any information on a network's
reputation.
In general, this idea was dismissed as libloc is neither intended nor
suitable for such tasks, and we do not want to make (political?)
decisions like these for various reasons. All we do is to provide a
useful location database in a neutral way, and leave it up to our users
on how to react on certain results.
However, there is a problematic area. Take AS55303 as an example: We
_know_ this is to be a dirty network, tampering with RIR data and
hijacking IP space, and strongly recommend against processing any
connection originating from or directed to it.
Since it appears to be loaded with proxies used by miscreants for
abusive purposes, all we can do at the time of writing is to flag it
as "anonymous proxy", but we lack possibility of telling our users
something like "this is not a safe area". The very same goes for known
bulletproof ISPs, IP hijackers, and so forth.
This patch therefore suggests to populate the "is_drop" flag introduced
in libloc 0.9.8 (albeit currently unused in production) with the
contents of Spamhaus' DROP lists (https://www.spamhaus.org/drop/), to
have at least the baddest of the bad covered. The very same lists are,
in fact, included in popular IPS rulesets as well - a decent amount of
IPFire users is therefore likely to have them already enabled, but in a
very costly way.
It is not planned to go further, partly because there is no other feed
publicly available, which would come with the same intention,
volatility, and FP rate.
The third version of this patch makes use of an auxiliary function to
sanitise ASNs, hence avoiding boilerplate code, and treats any line
starting with a semicolon as a comment, which should be sufficient.
Further, extracting ASNs from the ASN-DROP feed is done in a more clear
way, avoiding code snippets hard to read.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Thu, 23 Sep 2021 10:23:50 +0000 (13:23 +0300)]
debian: Ensure changelog distribution is tagged
UNRELEASED should not be left as-is when actually releasing.
The latest changelog entry now point at unstable instead.
The simple d/genchangelog.sh now does `dch -r ''` automatically
to ensure this distribution update doesn't get lost along the way
on future invocations.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 8 Aug 2021 21:31:58 +0000 (23:31 +0200)]
location-importer.in: Braindead me accidentally forgot a "break" statement
This one apparently went down the drain between these two patches:
- https://patchwork.ipfire.org/project/location/patch/20210522125758.28770-1-peter.mueller@ipfire.org/
- https://patchwork.ipfire.org/project/location/patch/aefd1904-4b38-f5cf-ab1d-9d69636cf914@ipfire.org/
Due to other safeguards, the current damage in production is limited to:
Peter Müller [Mon, 19 Jul 2021 21:34:40 +0000 (21:34 +0000)]
location-importer.in: Attempt to provide meaningful AS names if organisation handles are missing
A decent amount of autnum objects - especially, but not exclusively in
the APNIC sector - does not contain a link to an organisation handle.
In such cases, this patch is going to use the first description line of
the atunum object in question (if available) as a string for its name.
The overwhelming majority of affected ASNs contains a valuable
information there, so this is almost as good as having an organisation
handle linked to it.
Fixes: #12660 Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Sun, 11 Jul 2021 16:50:24 +0000 (19:50 +0300)]
debian: Clean up 0.9.7 changelog
- Update for maintainer name and email address, as to reflect who
actually prepared this release of the package. It was not me,
but I was selected due to having the first commit on the package.
To mitigate against this, when running the `debchange --release`
(`dch -r`) command, environment variables DEBFULLNAME and DEBEMAIL
should be configured properly for the current user.
- Removal of NMU comment on my name, as I am not really doing a
non-maintainer upload. I would say the 'NMU' message is fairly
useless on this repository, as it is self-maintained here.
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Sun, 13 Jun 2021 16:16:25 +0000 (19:16 +0300)]
debian: Attribute all maintainers in changlog
This commit further builds on historical changelog modifications,
to properly attribute all authors of the commits.
An additional d/genchangelog.sh script has been added. This allows
generation of changelog entries, internally using `debchange` (`dch`).
The script accepts an argument, which is the commit range to generate
entries for. Each commit's subject line (first line of body) is used,
along with author name and email. This information is added to the
changelog. Automatic detection (via `debchange` built-in functionality)
is used to determine whether these entries should be added to an
existing version number. If there is no UNRELEASED version, then a new
version is automatically tagged.
The new version tag will usually need to be modified, for example,
replacing an automatically generated 0.9.6-2 with 0.9.7-1.
The final release change (s/UNRELEASED/unstable/) needs to be done
manually as well, when the Git tag is actually being tagged.
`dch -r` can be useful for this particular purpose.
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Reported-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 11 Jun 2021 07:51:07 +0000 (10:51 +0300)]
debian: Add dpkg's symbols file
There are muiltiple standards of listing symbols throughout the Linux
ecosystem. For `dpkg`, a d/package.symbols file tracks symbols, and in
which version they were added in. This is then used to allow dependency
checks/resolution.
See man:dpkg-gensymbols(1) for details about the generation,
and man:dpkg-shlibdeps(1) for how the symbols file ends up being used.
This commit adds a d/libloc1.symbols file, containing the current state
of the symbols. There is now also a d/gensymbols.sh script, which
generates this symbols file. The script tries to determine what Git
tags need to be checked for changes in symbols, by looking at current
maximum version referenced in symbols file.
After checking tags, the current revision is also processed, to allow
building symbols file for a yet unreleased version (prior to tagging it).
This is to allow symbols changes to be included in a tag.
Do keep in mind, that for the workflow above, when running the script,
the d/changelog file should contain information about what version the
current revision will be released at (potentially tagged as UNRELEASED
in the d/changelog file). Otherwise, if there is no version tagged,
the `dpkg-gensymbols` tool will use the old version information,
in turn incorrectly attributing new symbols to an old version.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Thu, 10 Jun 2021 09:37:22 +0000 (09:37 +0000)]
countries: Fix matching invalid country codes
When an invalid country code is entered, loc_country_new returns an
error which is interpreted as a match to the list since we check for a
non-zero return code.
Any invalid country codes are now silently ignored and not considered a
match.
Peter Müller [Tue, 8 Jun 2021 09:55:41 +0000 (09:55 +0000)]
location-importer.in: import additional IP information for Amazon AWS IP networks
Amazon publishes information regarding some of their IP networks
primarily used for AWS cloud services in a machine-readable format. To
improve libloc lookup results for these, we have little choice other
than importing and parsing them.
Unfortunately, there seems to be no machine-readable list of the
locations of their data centers or availability zones available. If
there _is_ any, please let the author know.
The second version of this patch adds a meaningful description for the
"source" column in the overrides tables, to make introduced changes
less intransparent.
Fixes: #12594 Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Tue, 8 Jun 2021 09:55:40 +0000 (09:55 +0000)]
location-importer.in: add source column for overrides as well
This allows us to track changes introduced by IP feeds from 3rd parties,
such as Amazon AWS, on the SQL server side.
In order not to break existing tables (which would required TRUNCATE),
there currently is no constraint set for the new column, but "NOT NULL"
is planned in the future.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Tue, 8 Jun 2021 17:03:07 +0000 (17:03 +0000)]
location-importer.in: Import (technical) AS names from ARIN
ARIN and LACNIC, unfortunately, do not seem to publish data containing
human readable AS names. For the former, we at least have a list of
tecnical names, which this patch fetches and inserts into the autnums
table.
While some of them do not seem to be suitable for human consumption (i.
e. being very cryptic), providing these data might be helpful
neverthelesss.
The second version of this patch contains some additional remarks on
efficient Python coding style from Michael, doing things more "pythonic".
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Fri, 4 Jun 2021 15:57:30 +0000 (17:57 +0200)]
Implement an additional flag for hostile networks safe to drop
This patch implements an additional flag intended for networks and
Autonomous Systems being considered hostile. While libloc does not and
should not be an opinionated database, reality shows it is being used
this way.
Hereby, we assign "XD" (drop) as a custom country code for networks
being flagged this way. According to ISO, "XA" to "XZ" are reserved for
"user-assgined codes" (https://www.iso.org/glossary-for-iso-3166.html),
so this is a safe thing to do.
This patch does not interfere with "A1" to "A3", which we currently
assign outside standardised country code ranges for historical reasons.
Neither does it specify any policy or source for tagging networks with a
"drop" flag. Doing so is beyond the scope of this - technical -
approach.
To avoid confusions with the SQL "DROP" command, "is_drop" will be used
as a column name for database operations.
Thanks to Michael for his remarks and ideas during the run-up.
Cc: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Reported-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sun, 30 May 2021 08:50:04 +0000 (10:50 +0200)]
location-importer.in: track original countries as well
This helps us to determine how many network objects have more than one
country set, and what their original country code set looked like.
The third version of this patch uses ALTER TABLE to add the column for
original countries, preventing existing SQL setups from breaking, and is
correctly based against the current "master" branch.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 22 May 2021 20:33:51 +0000 (20:33 +0000)]
location-importer.in: keep track of sources for networks, ASNs, and organisations
This allows us to trace back concrete changes or anomalies to their RIR
source, without having to parse everything again. Further, it enables
adding 3rd party sources such as IP feeds from Amazon, without loosing
track of the changes introduced by them.
The second version of this patchset uses ALTER TABLE to add the source
columns, avoiding breaking existing SQL setups.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 3 May 2021 17:14:29 +0000 (19:14 +0200)]
location-importer.in: emit warnings due to unknown country code for valid networks only
This reduces log spam in case of processing RIR database, checking for
networks with unknown country codes assigned. If we would not have
written into the database, there is no need to warn about them.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:10 +0000 (16:06 +0300)]
debian: Drop unintended files from location-python
_location.la gets built and installed to site-packages/, however
an .la file is not expected to reside in the Python root. Additionally,
the dependency library listed does not have its respective .la file
installed. Further complicating the situation, dh-python moves the
site-packages/ files to dist-packages/ silently which then results in
a broken libdir left behind in the .la file.
The only reason the file is there is that it gets built inside the
source directory, which gets copied entirely to location-python package
as-is. Considering the situation, this commit ensures the .la files is
not packaged by deleting it from the package files subdirectory.
location-importer package pulls in two Python (.py) files from the
source directory. These files should not be included in the
location-python package as a result.
Valters Jansons [Fri, 16 Apr 2021 13:06:05 +0000 (16:06 +0300)]
debian: Add all temporary files to Gitignore
New packages have been added since the inception of the .gitignore and
as a result during build we see directories such as location-importer/
and files such as location-importer.debhelper.log.
This commit ensures all temporary subdirectories, and additional
generic build artifact files, are ignored by Git.
The subdirectory exceptions to this rule are:
- d/patches/ which may be used by Quilt
considering the source format is '3.0 (quilt)',
- d/source/ for the format file,
- d/tests/ which may be used by autopkgtest
to specify what test suites exist for the source.
See: https://salsa.debian.org/ci-team/autopkgtest/-/raw/debian/5.16/doc/README.package-tests.rst
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:12 +0000 (16:06 +0300)]
systemd: Add Documentation= to location-update
Systemd units are expected to provide some documentation information
such as manpages, or direct links, which provide more details about
that unit. This commit simply links location-update.service to the
manual for location(8) followed by a fallback to the online manual.
Valters Jansons [Fri, 16 Apr 2021 13:06:11 +0000 (16:06 +0300)]
debian: Add watch configuration for uscan
Packages defined as '3.0 (quilt)' are expected to provide information
about how the latest upstream information can be obtained,
as a special d/watch file. This can then get used by uscan(1).
To see how the metadata is utilized, and how the network requests
are made behind the scenes, you can locally run:
$ uscan --no-download --verbose --debug
Resolves: lintian: debian-watch-file-is-missing
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:07 +0000 (16:06 +0300)]
debian: Set 'Multi-Arch: foreign' hint for Python
Due to the invocation of py3compile (via dh-python) in location-importer
and location-python packages, those packages have different bytecode for
varying architectures, and as a result are not 'Multi-Arch: same'.
Valters Jansons [Thu, 15 Apr 2021 11:42:13 +0000 (14:42 +0300)]
po: Update translations
POTFILES.in should not contain src/python/__init__.py file as it
is not present in the committed tree. It has its respective .in file
which is present instead.
This commit further ensures po/POTFILES.in generator avoids such
files that Git ignores (using git-check-ignore during find).
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Mon, 12 Apr 2021 13:01:45 +0000 (16:01 +0300)]
debian: Rework historical changelog
Rewriting history is generally considered a "not-so-good" thing,
however here the historical data does not align with best practises
and therefore it is beneficial to provide a better example going
forward.
There is only one initial release. Everything following that should
list some kind of release notes or changelog, or at the very least
just say something along the lines of "New version" rather than
"Initial release".
In this commit, the Git history is used for this task,
filtering out "Makefile" changes as to retain only changes
that are visible to users, excluding building tooling.
For Debian packages, upon release, the target distribution should be
updated to "unstable" (or "experimental" if preferred for any reason)
when a release is finalized. During development, an invalid
distribution name is expected to be there for tracking unreleased
changes. That is why "UNRELEASED" is the standard way of specifying
ongoing development, being an invalid distribution name itself.
The "(Closes: #XXXXXX)" tag is intended for linking to Debian bug
tracker, such as linking to the initial Intent to Package ticket,
or later update/bugfix tickets. There does not appear to be a bug
tracker in use for this task here, and the XXXXXX bug ticket number
does not take you anywhere. It's therefore better to just remove it.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Mon, 12 Apr 2021 12:57:24 +0000 (15:57 +0300)]
debian: Add missing '<' in copyright
The email address information should be inside brackets. This
commit ensures the missing bracket character issue is remedied. Reviewed-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Tue, 6 Apr 2021 11:13:31 +0000 (14:13 +0300)]
debian: Add intltoolize to dh_auto_configure
Debian has automated building tools that handle source trees directly.
It is expected that you can pick up a source tarball, and with the
appropriate debian/ subtree, a successful build can be produced using
the `debuild` tool. This depends on all the build steps having been
included as part of the debian/rules file (see: `man debuild`).
This commit ensures there is no need to manually run autogen.sh
on a locally extracted source tarball prior to building for Debian.
This is accomplished by adding the `intltoolize` command to the
override_dh_auto_configure step in d/rules.
There is no need to add the `autoreconf` command due to dh-autoreconf
always handling that prior to the dh_auto_configure step.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Tue, 30 Mar 2021 15:47:10 +0000 (17:47 +0200)]
location-importer.in: skip networks with unknown country codes
There is no sense in parsing and storting networks whose country codes
cannot be found in the ISO-3166-x country code table. This avoids side
effects in applications using the location database, and introduces
another sanity check to compensate bogus RIR data.
On location02, this affects some networks from APNIC (country code: ZZ)
as well as a bunch of smaller allocations within the RIPE region still
tagged to CS or YU (Yugoslavia). To my surprise, no network tagged as SU
(Soviet Union) was found - while the NIC for .su TLD is still
operational. :-)
Applying this patch causes the countries to be processed before
update_whois() is called. In case no countries are present in the SQL
table, this check is silently omitted.
Fixes: #12510 Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 29 Mar 2021 20:24:36 +0000 (22:24 +0200)]
location-importer.in: process unaligned IP ranges in RIR data files correctly
The IP range given in an inetnum object apparently not necessarily
matches distinct subnet boundaries. As a result, the current attempt to
calculate its CIDR mask resulted in faulty subnets not covering the
entire IP range.
This patch leaves the task of enumerating subnets to the ipaddress
module itself, which handles things much more robust. Since the output
may contain of several subnets, a list for the inetnum key is necessary
as well as a loop over them when conducting the SQL statements.
Fixes: #12595 Cc: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 16 Jan 2021 18:05:33 +0000 (19:05 +0100)]
location-importer.in: delete 6to4 IPv6 space as well
2002::/16 is an anycast prefix for 6to4 scenarios, as specified in RFC
3068. We currently process an announcement from Hurricane Electric for
it, and since it is an anycast network, multiple entities across the
world announce it as well.
Thereof, it does not make sense to include it in the database - as of
today, we do not have a country for it, either.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Dec 2020 16:58:29 +0000 (16:58 +0000)]
database: Restart flatten algorithm from the top when a network was dropped
We used to simply take the first element from the stack after we have
split a network. That is wrong because it is not passing through any
filters and no further subnet checks. It could have therefore been
that the tree was not entirely flat.
Reported-by: Arne Fitzenreiter <arne_f@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 25 Nov 2020 15:16:06 +0000 (15:16 +0000)]
network: Massively improve performance on exclude
When we check the result for any overlaps, we can cut this short
by walking through both lists from start to end and remember the
last network that we checked.
The next one will by definition be strictly greater and therefore
we do not need to check anything before this any more.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 25 Nov 2020 14:42:26 +0000 (14:42 +0000)]
network-list: Do not half list when popping the first element
The list was unfortunately halved in size every time an element
was taken from it, which was great for performance, but shortened
the result substantially.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
projects / location / libloc.git / log
