From: Markus Armbruster <armbru@redhat.com>
Date: Fri, 21 Feb 2014 15:42:52 +0000 (+0100)
Subject: vnc: Fix tight_detect_smooth_image() for lossless case
X-Git-Tag: v1.7.2~5
X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=02835d57444ce7308931b71cabbe5fb1d7d8b9eb;p=thirdparty%2Fqemu.git

vnc: Fix tight_detect_smooth_image() for lossless case

VncTight member uint8_t quality is either (uint8_t)-1 for lossless or
less than 10 for lossy.

tight_detect_smooth_image() first promotes it to int, then compares
with -1.  Always unequal, so we always execute the lossy code.  Reads
beyond tight_conf[] and returns crap when quality is actually
lossless.

Compare to (uint8_t)-1 instead, like we do elsewhere.

Spotted by Coverity.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
(cherry picked from commit 2e7bcdb99adbd8fc10ad9ddcf93bd2bf3c0f1f2d)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---

diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c
index e6966aebc35..59b59c0c79c 100644
--- a/ui/vnc-enc-tight.c
+++ b/ui/vnc-enc-tight.c
@@ -330,7 +330,7 @@ tight_detect_smooth_image(VncState *vs, int w, int h)
     } else {
         errors = tight_detect_smooth_image16(vs, w, h);
     }
-    if (quality != -1) {
+    if (quality != (uint8_t)-1) {
         return (errors < tight_conf[quality].jpeg_threshold);
     }
     return (errors < tight_conf[compression].gradient_threshold);