Peter Müller [Sun, 30 May 2021 08:50:04 +0000 (10:50 +0200)]
location-importer.in: track original countries as well
This helps us to determine how many network objects have more than one
country set, and what their original country code set looked like.
The third version of this patch uses ALTER TABLE to add the column for
original countries, preventing existing SQL setups from breaking, and is
correctly based against the current "master" branch.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 22 May 2021 20:33:51 +0000 (20:33 +0000)]
location-importer.in: keep track of sources for networks, ASNs, and organisations
This allows us to trace back concrete changes or anomalies to their RIR
source, without having to parse everything again. Further, it enables
adding 3rd party sources such as IP feeds from Amazon, without loosing
track of the changes introduced by them.
The second version of this patchset uses ALTER TABLE to add the source
columns, avoiding breaking existing SQL setups.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 3 May 2021 17:14:29 +0000 (19:14 +0200)]
location-importer.in: emit warnings due to unknown country code for valid networks only
This reduces log spam in case of processing RIR database, checking for
networks with unknown country codes assigned. If we would not have
written into the database, there is no need to warn about them.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:10 +0000 (16:06 +0300)]
debian: Drop unintended files from location-python
_location.la gets built and installed to site-packages/, however
an .la file is not expected to reside in the Python root. Additionally,
the dependency library listed does not have its respective .la file
installed. Further complicating the situation, dh-python moves the
site-packages/ files to dist-packages/ silently which then results in
a broken libdir left behind in the .la file.
The only reason the file is there is that it gets built inside the
source directory, which gets copied entirely to location-python package
as-is. Considering the situation, this commit ensures the .la files is
not packaged by deleting it from the package files subdirectory.
location-importer package pulls in two Python (.py) files from the
source directory. These files should not be included in the
location-python package as a result.
Valters Jansons [Fri, 16 Apr 2021 13:06:05 +0000 (16:06 +0300)]
debian: Add all temporary files to Gitignore
New packages have been added since the inception of the .gitignore and
as a result during build we see directories such as location-importer/
and files such as location-importer.debhelper.log.
This commit ensures all temporary subdirectories, and additional
generic build artifact files, are ignored by Git.
The subdirectory exceptions to this rule are:
- d/patches/ which may be used by Quilt
considering the source format is '3.0 (quilt)',
- d/source/ for the format file,
- d/tests/ which may be used by autopkgtest
to specify what test suites exist for the source.
See: https://salsa.debian.org/ci-team/autopkgtest/-/raw/debian/5.16/doc/README.package-tests.rst
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:12 +0000 (16:06 +0300)]
systemd: Add Documentation= to location-update
Systemd units are expected to provide some documentation information
such as manpages, or direct links, which provide more details about
that unit. This commit simply links location-update.service to the
manual for location(8) followed by a fallback to the online manual.
Valters Jansons [Fri, 16 Apr 2021 13:06:11 +0000 (16:06 +0300)]
debian: Add watch configuration for uscan
Packages defined as '3.0 (quilt)' are expected to provide information
about how the latest upstream information can be obtained,
as a special d/watch file. This can then get used by uscan(1).
To see how the metadata is utilized, and how the network requests
are made behind the scenes, you can locally run:
$ uscan --no-download --verbose --debug
Resolves: lintian: debian-watch-file-is-missing
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Fri, 16 Apr 2021 13:06:07 +0000 (16:06 +0300)]
debian: Set 'Multi-Arch: foreign' hint for Python
Due to the invocation of py3compile (via dh-python) in location-importer
and location-python packages, those packages have different bytecode for
varying architectures, and as a result are not 'Multi-Arch: same'.
Valters Jansons [Thu, 15 Apr 2021 11:42:13 +0000 (14:42 +0300)]
po: Update translations
POTFILES.in should not contain src/python/__init__.py file as it
is not present in the committed tree. It has its respective .in file
which is present instead.
This commit further ensures po/POTFILES.in generator avoids such
files that Git ignores (using git-check-ignore during find).
Signed-off-by: Valters Jansons <valter.jansons@gmail.com> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Mon, 12 Apr 2021 13:01:45 +0000 (16:01 +0300)]
debian: Rework historical changelog
Rewriting history is generally considered a "not-so-good" thing,
however here the historical data does not align with best practises
and therefore it is beneficial to provide a better example going
forward.
There is only one initial release. Everything following that should
list some kind of release notes or changelog, or at the very least
just say something along the lines of "New version" rather than
"Initial release".
In this commit, the Git history is used for this task,
filtering out "Makefile" changes as to retain only changes
that are visible to users, excluding building tooling.
For Debian packages, upon release, the target distribution should be
updated to "unstable" (or "experimental" if preferred for any reason)
when a release is finalized. During development, an invalid
distribution name is expected to be there for tracking unreleased
changes. That is why "UNRELEASED" is the standard way of specifying
ongoing development, being an invalid distribution name itself.
The "(Closes: #XXXXXX)" tag is intended for linking to Debian bug
tracker, such as linking to the initial Intent to Package ticket,
or later update/bugfix tickets. There does not appear to be a bug
tracker in use for this task here, and the XXXXXX bug ticket number
does not take you anywhere. It's therefore better to just remove it.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Mon, 12 Apr 2021 12:57:24 +0000 (15:57 +0300)]
debian: Add missing '<' in copyright
The email address information should be inside brackets. This
commit ensures the missing bracket character issue is remedied. Reviewed-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Valters Jansons [Tue, 6 Apr 2021 11:13:31 +0000 (14:13 +0300)]
debian: Add intltoolize to dh_auto_configure
Debian has automated building tools that handle source trees directly.
It is expected that you can pick up a source tarball, and with the
appropriate debian/ subtree, a successful build can be produced using
the `debuild` tool. This depends on all the build steps having been
included as part of the debian/rules file (see: `man debuild`).
This commit ensures there is no need to manually run autogen.sh
on a locally extracted source tarball prior to building for Debian.
This is accomplished by adding the `intltoolize` command to the
override_dh_auto_configure step in d/rules.
There is no need to add the `autoreconf` command due to dh-autoreconf
always handling that prior to the dh_auto_configure step.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Tue, 30 Mar 2021 15:47:10 +0000 (17:47 +0200)]
location-importer.in: skip networks with unknown country codes
There is no sense in parsing and storting networks whose country codes
cannot be found in the ISO-3166-x country code table. This avoids side
effects in applications using the location database, and introduces
another sanity check to compensate bogus RIR data.
On location02, this affects some networks from APNIC (country code: ZZ)
as well as a bunch of smaller allocations within the RIPE region still
tagged to CS or YU (Yugoslavia). To my surprise, no network tagged as SU
(Soviet Union) was found - while the NIC for .su TLD is still
operational. :-)
Applying this patch causes the countries to be processed before
update_whois() is called. In case no countries are present in the SQL
table, this check is silently omitted.
Fixes: #12510 Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Mon, 29 Mar 2021 20:24:36 +0000 (22:24 +0200)]
location-importer.in: process unaligned IP ranges in RIR data files correctly
The IP range given in an inetnum object apparently not necessarily
matches distinct subnet boundaries. As a result, the current attempt to
calculate its CIDR mask resulted in faulty subnets not covering the
entire IP range.
This patch leaves the task of enumerating subnets to the ipaddress
module itself, which handles things much more robust. Since the output
may contain of several subnets, a list for the inetnum key is necessary
as well as a loop over them when conducting the SQL statements.
Fixes: #12595 Cc: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Peter Müller [Sat, 16 Jan 2021 18:05:33 +0000 (19:05 +0100)]
location-importer.in: delete 6to4 IPv6 space as well
2002::/16 is an anycast prefix for 6to4 scenarios, as specified in RFC
3068. We currently process an announcement from Hurricane Electric for
it, and since it is an anycast network, multiple entities across the
world announce it as well.
Thereof, it does not make sense to include it in the database - as of
today, we do not have a country for it, either.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 1 Dec 2020 16:58:29 +0000 (16:58 +0000)]
database: Restart flatten algorithm from the top when a network was dropped
We used to simply take the first element from the stack after we have
split a network. That is wrong because it is not passing through any
filters and no further subnet checks. It could have therefore been
that the tree was not entirely flat.
Reported-by: Arne Fitzenreiter <arne_f@ipfire.org> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 25 Nov 2020 15:16:06 +0000 (15:16 +0000)]
network: Massively improve performance on exclude
When we check the result for any overlaps, we can cut this short
by walking through both lists from start to end and remember the
last network that we checked.
The next one will by definition be strictly greater and therefore
we do not need to check anything before this any more.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Wed, 25 Nov 2020 14:42:26 +0000 (14:42 +0000)]
network-list: Do not half list when popping the first element
The list was unfortunately halved in size every time an element
was taken from it, which was great for performance, but shortened
the result substantially.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Tue, 24 Nov 2020 15:15:59 +0000 (15:15 +0000)]
database: Avoid merging the same data twice
When finish splitting networks into many parts, we have
a list of subnets with the excluded subnets and merge them
together first and put them on the stack again.
This is slower than pushing it all onto the stack first
and then popping the first element.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 18:39:13 +0000 (18:39 +0000)]
network: Speed up subnet check
There is no point in checking different address families
with each other and we do not need to compare addresses
when the prefix of the subnet does not fit into the
network to check.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Michael Tremer [Fri, 20 Nov 2020 18:36:48 +0000 (18:36 +0000)]
network: Optimise _subnet function
This function used to create a network list which always
had exactly two elements. Since splitting a network in half
always returns two parts, we can simply return them as a
pointer.
This improves returning the network tree by about 17%.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>