Handle bad UTF-8 in xmltv (#5909)

We had a string where we had a rogue byte (0x8a) which was not part of
a UTF-8 string.  This then caused some downstream parsers to abort
processing the document; other parsers ignored the bad character.

As an interim fix, we now parse the individual characters and filter
out invalid characters.  We replace such characters with a space
character (instead of a U+FFFD replacement character) since this
is typically user presentable data on a "10ft interface".  An
alternative would be to completely discard the character, but the
examples we had would then have words combined where the invalid
character used to be.

diff --git a/src/htsbuf.c b/src/htsbuf.c
index 50f22a5006ba64d425af455b14c2bd1752986b46..c7c9b10f8bbce2cca3e12593d6bd0f0a973ccc1a 100644 (file)
--- a/src/htsbuf.c
+++ b/src/htsbuf.c
@@ -353,15 +353,50 @@ htsbuf_append_and_escape_xml(htsbuf_queue_t *hq, const char *s)
 {
   const char *c = s;
   const char *e = s + strlen(s);
-  const char *esc;
+  const char *esc = 0;
   int h;
 
   if(e == s)
     return;
 
-  while(1) {
+  while(c<e) {
     h = *c++;
 
+    if (h & 0x80) {
+      // Start of UTF-8.  But we sometimes have bad UTF-8 (#5909).
+      // So validate and handle bad characters.
+
+      // Number of top bits set indicates the total number of  bytes.
+      const int num_bytes =
+        ((h & 240) == 240) ? 4 :
+        ((h & 224) == 224) ? 3 :
+        ((h & 192) == 192) ? 2 : 0;
+
+      if (!num_bytes) {
+        // Completely invalid sequence, so we replace it with a space.
+        htsbuf_append(hq, s, c - s - 1);
+        htsbuf_append(hq, " ", 1);
+        s=c;
+        continue;
+      } else {
+        // Start of valid UTF-8.
+        if (e - c < num_bytes - 1) {
+          // Invalid sequence - too few characters left in buffer for the sequence.
+          // Append what we already have accumulated and ignore remaining characters.
+          htsbuf_append(hq, s, c - s - 1);
+          break;
+        } else {
+          // We should probably check each character in the range is also valid.
+          htsbuf_append(hq, s, c - s - 1);
+          htsbuf_append(hq, c-1, num_bytes);
+          s=c-1;
+          s+=num_bytes;
+          c=s;
+          continue;
+        }
+      }
+    }
+
     switch(h) {
     case '<':  esc = "&lt;";   break;
     case '>':  esc = "&gt;";   break;