From: E.Smith <31170571+azlm8t@users.noreply.github.com>
Date: Tue, 25 Sep 2018 11:50:57 +0000 (+0100)
Subject: python: Handle incorrect utf-8 decodes.
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=c3f3ab77febd72b3bd03fd37a76aff60ba2884e2;p=thirdparty%2Ftvheadend.git

python: Handle incorrect utf-8 decodes.

Some broadcasts can have different charsets (such as iso-8859-1) but
we assume utf-8 unless user has set it correctly. So when decode fails
we get an exception.  So we now attempt to decode with error
replacement so user sees incorrect character.

This gives "u'Denise Th\ufffd\ufffd':" as the string returned instead
when the received name contains an é that is in iso-8859-1 instead of
utf-8.
---

diff --git a/lib/py/tvh/htsmsg.py b/lib/py/tvh/htsmsg.py
index a5ea8ca36..8fe85d650 100644
--- a/lib/py/tvh/htsmsg.py
+++ b/lib/py/tvh/htsmsg.py
@@ -179,7 +179,7 @@ def deserialize0(data, typ=HMF_MAP):
             try:
                 item = data[:dlen].decode('utf-8')
             except:
-                item = data[:dlen]
+                item = data[:dlen].decode(errors='replace')
         elif typ == HMF_BIN:
             item = HMFBin(data[:dlen])
         elif typ == HMF_S64:
@@ -201,7 +201,11 @@ def deserialize0(data, typ=HMF_MAP):
         if islist:
             msg.append(item)
         else:
-            msg[name.decode()] = item
+            try:
+                n = name.decode(errors="replace")
+            except:
+                n = name
+            msg[n] = item
         data = data[dlen:]
     return msg