From: Michael Tremer <michael.tremer@ipfire.org>
Date: Fri, 12 Aug 2022 15:38:08 +0000 (+0000)
Subject: importer: Try to make parsing blocks faster
X-Git-Tag: 0.9.14~6
X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=c5092d41671f285276068316854f1464a2101041;p=location%2Flibloc.git

importer: Try to make parsing blocks faster

This patch changes the order of some operations and drops a few
redundant ones (potentiall).

That should allow us to parse blocked data faster.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
---

diff --git a/src/python/location/importer.py b/src/python/location/importer.py
index ea1333c..a5776f7 100644
--- a/src/python/location/importer.py
+++ b/src/python/location/importer.py
@@ -200,6 +200,10 @@ def iterate_over_blocks(f, charsets=("utf-8", "latin1")):
 	block = []
 
 	for line in f:
+		# Skip commented lines
+		if line.startswith(b"#") or line.startswith(b"%"):
+			continue
+
 		# Convert to string
 		for charset in charsets:
 			try:
@@ -209,24 +213,17 @@ def iterate_over_blocks(f, charsets=("utf-8", "latin1")):
 			else:
 				break
 
-		# Skip commented lines
-		if line.startswith("#") or line.startswith("%"):
-			continue
-
-		# Strip line-endings
-		line = line.rstrip()
-
 		# Remove any comments at the end of line
 		line, hash, comment = line.partition("#")
 
-		if comment:
-			# Strip any whitespace before the comment
-			line = line.rstrip()
+		# Strip any whitespace at the end of the line
+		line = line.rstrip()
 
-			# If the line is now empty, we move on
-			if not line:
-				continue
+		# If we cut off some comment and the line is empty, we can skip it
+		if comment and not line:
+			continue
 
+		# If the line has some content, keep collecting it
 		if line:
 			block.append(line)
 			continue