The export became a huge problem here as it was very slow. One large
query produced one very large output which was difficult to test and
validate as the total export could easily take 17-20 hrs.
This patch fixes this so that the export now takes around ~5min. Sadly
we are still not using more than one processor core, because PostgreSQL
thinks its faster with just the one, but we trade a lot of CPU time for
a little bit of extra memory instead.
We create a new temporary table that holds all networks that we are
interested in. We then run a couple of broken down queries that perform
one thing instead of multiple at the same time.
First, we add all AS numbers, then all countries which are the longest
queries taking now around 20-30s. Then we apply any data from the
external feeds and our own overrides.
The final export requires about 1.7 GiB of memory because we don't
perform any deduplication in the database any more. That has been a lot
better implemented in our own code and I don't think that < 2 GiB is a
reason for concern.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>