]> git.ipfire.org Git - location/libloc.git/commit
importer: Purge any redundant entries
authorPeter Müller <peter.mueller@ipfire.org>
Sun, 20 Sep 2020 19:21:03 +0000 (19:21 +0000)
committerMichael Tremer <michael.tremer@ipfire.org>
Mon, 21 Sep 2020 16:16:43 +0000 (16:16 +0000)
commitc2cc55d5a6875c3838f060032eaed89dcfb92ef6
tree3aac65cd733b06ae721d523bacb65531c4d1d0c0
parent64e95fa903edec8b4e4e59830b395e2e4a411853
importer: Purge any redundant entries

When importing inetnums, we might import various small networks
which are not relevant for us as long as they do not have a
different country code than their parent network.

Therefore we delete all these entries to keep the database
smaller without losing any information. The second version of this
patch introduces a SQL statement parallelised across all CPUs
available, while the DELETE-statement of the first version literally
took ages to complete.

However, cleaning up those data still takes about 26 hours (!) on
our location02 testing machine, making daily updates of the location
database impossible to the current knowledge.

real    1521m30.620s
user    38m45.521s
sys     9m6.027s

Special thanks goes to Michael for spending numerous hours
on this, setting up a testing environment, doing PostgreSQL magic
and providing helpful advice while debugging.

Partially fixes: #12458

Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
src/python/location-importer.in