parser tests: test parsing of interesting mail

We have 4 new tests:
 - One with a header with a character that is neither 7-bit ASCII
   nor UTF-8.
 - One with UTF-8 in the body.
 - One with un-encoded UTF-8 in the headers.
 - One with a header that claims to be UTF-8 but isn't.

This also requires us to change how we open files on Python 3. Use
the binary file and message_from_binary_file setup under Python 3.

Signed-off-by: Daniel Axtens <dja@axtens.net>
Reviewed-by: Stephen Finucane <stephen@that.guru>

diff --git a/patchwork/tests/mail/0012-invalid-header-char.mbox b/patchwork/tests/mail/0012-invalid-header-char.mbox
new file mode 100644 (file)
index 0000000..3412ca6
--- /dev/null
+++ b/patchwork/tests/mail/0012-invalid-header-char.mbox
@@ -0,0 +1,72 @@
+From ferruh.yigit@intel.com  Thu Sep 15 14:06:56 2016
+Return-Path: <ferruh.yigit@intel.com>
+Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
+ by dpdk.org (Postfix) with ESMTP id E9946530F
+ for <dev@dpdk.org>; Thu, 15 Sep 2016 14:06:55 +0200 (CEST)
+Received: from fmsmga001.fm.intel.com ([10.253.24.23])
+ by fmsmga101.fm.intel.com with ESMTP; 15 Sep 2016 05:06:54 -0700
+X-ExtLoop1: 1
+X-IronPort-AV: E=Sophos;i="5.30,339,1470726000"; d="scan'208";a="1040330672"
+Received: from irvmail001.ir.intel.com ([163.33.26.43])
+ by fmsmga001.fm.intel.com with ESMTP; 15 Sep 2016 05:06:54 -0700
+Received: from sivswdev02.ir.intel.com (sivswdev02.ir.intel.com
+ [10.237.217.46])
+ by irvmail001.ir.intel.com (8.14.3/8.13.6/MailSET/Hub) with ESMTP id
+ u8FC6rSL030312; Thu, 15 Sep 2016 13:06:53 +0100
+Received: from sivswdev02.ir.intel.com (localhost [127.0.0.1])
+ by sivswdev02.ir.intel.com with ESMTP id u8FC6rwP015039;
+ Thu, 15 Sep 2016 13:06:53 +0100
+Received: (from fyigit@localhost)
+       by sivswdev02.ir.intel.com with \9c id u8FC6rZs015035;
+       Thu, 15 Sep 2016 13:06:53 +0100
+X-Authentication-Warning: sivswdev02.ir.intel.com: fyigit set sender to
+ ferruh.yigit@intel.com using -f
+From: Ferruh Yigit <ferruh.yigit@intel.com>
+To: dev@dpdk.org
+Cc: Ferruh Yigit <ferruh.yigit@itnel.com>
+Date: Thu, 15 Sep 2016 13:06:44 +0100
+Message-Id: <1473941204-14587-1-git-send-email-ferruh.yigit@intel.com>
+X-Mailer: git-send-email 1.7.0.7
+Subject: [dpdk-dev] [PATCH] kni: fix compilation error when debug enabled
+X-BeenThere: dev@dpdk.org
+X-Mailman-Version: 2.1.15
+Precedence: list
+List-Id: patches and discussions about DPDK <dev.dpdk.org>
+List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
+ <mailto:dev-request@dpdk.org?subject=unsubscribe>
+List-Archive: <http://dpdk.org/ml/archives/dev/>
+List-Post: <mailto:dev@dpdk.org>
+List-Help: <mailto:dev-request@dpdk.org?subject=help>
+List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
+ <mailto:dev-request@dpdk.org?subject=subscribe>
+X-List-Received-Date: Thu, 15 Sep 2016 12:06:56 -0000
+
+Fix build error with Linux kernel >= v4.7
+
+Fix compile error because of Linux API change, 'trans_start' field
+removed from 'struct net_device'.
+
+Linux: 9b36627acecd ("net: remove dev->trans_start")
+
+Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
+---
+ lib/librte_eal/linuxapp/kni/kni_net.c | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/lib/librte_eal/linuxapp/kni/kni_net.c b/lib/librte_eal/linuxapp/kni/kni_net.c
+index fc82193..371bbfa 100644
+--- a/lib/librte_eal/linuxapp/kni/kni_net.c
++++ b/lib/librte_eal/linuxapp/kni/kni_net.c
+@@ -508,7 +508,7 @@ kni_net_tx_timeout (struct net_device *dev)
+       struct kni_dev *kni = netdev_priv(dev);
+ 
+       KNI_DBG("Transmit timeout at %ld, latency %ld\n", jiffies,
+-                      jiffies - dev->trans_start);
++                      jiffies - dev_trans_start(dev));
+ 
+       kni->stats.tx_errors++;
+       netif_wake_queue(dev);
+-- 
+2.7.4
+
+

diff --git a/patchwork/tests/mail/0013-with-utf8-body.mbox b/patchwork/tests/mail/0013-with-utf8-body.mbox
new file mode 100644 (file)
index 0000000..c7319f2
--- /dev/null
+++ b/patchwork/tests/mail/0013-with-utf8-body.mbox
@@ -0,0 +1,46 @@
+From patchwork Wed Jun  1 20:00:54 2016
+Content-Type: text/plain; charset="utf-8"
+MIME-Version: 1.0
+Content-Transfer-Encoding: 8bit
+Subject: MAINTAINERS: Update entry for BCM5301X ARM
+From: =?utf-8?b?UmFmYcWCIE1pxYJlY2tp?= <zajec5@gmail.com>
+X-Patchwork-Id: 9148093
+Message-Id: <1464811283-10381-1-git-send-email-zajec5@gmail.com>
+To: Florian Fainelli <f.fainelli@gmail.com>
+Cc: bcm-kernel-feedback-list@broadcom.com,
+ =?UTF-8?q?Rafa=C5=82=20Mi=C5=82ecki?= <zajec5@gmail.com>,
+ Andrew Morton <akpm@linux-foundation.org>,
+ "David S. Miller" <davem@davemloft.net>,
+ Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
+ Mauro Carvalho Chehab <mchehab@osg.samsung.com>,
+ Guenter Roeck <linux@roeck-us.net>, Jiri Slaby <jslaby@suse.cz>,
+ linux-kernel@vger.kernel.org (open list)
+Date: Wed,  1 Jun 2016 22:00:54 +0200
+
+Add myself as I contribute to it. Include Broadcom's feedback ML as
+suggested by Florian. Finally modify file rule to match
+bcm5301x-nand-cs0-bch8.dtsi.
+
+Signed-off-by: Rafał Miłecki <zajec5@gmail.com>
+---
+ MAINTAINERS | 4 +++-
+ 1 file changed, 3 insertions(+), 1 deletion(-)
+
+diff --git a/MAINTAINERS b/MAINTAINERS
+index 7304d2e..a5352d8 100644
+--- a/MAINTAINERS
++++ b/MAINTAINERS
+@@ -2511,10 +2511,12 @@ F:     arch/mips/include/asm/mach-bcm47xx/*
+ 
+ BROADCOM BCM5301X ARM ARCHITECTURE
+ M:    Hauke Mehrtens <hauke@hauke-m.de>
++M:    Rafał Miłecki <zajec5@gmail.com>
+ L:    linux-arm-kernel@lists.infradead.org
++L:    bcm-kernel-feedback-list@broadcom.com
+ S:    Maintained
+ F:    arch/arm/mach-bcm/bcm_5301x.c
+-F:    arch/arm/boot/dts/bcm5301x.dtsi
++F:    arch/arm/boot/dts/bcm5301x*.dtsi
+ F:    arch/arm/boot/dts/bcm470*
+ 
+ BROADCOM BCM63XX ARM ARCHITECTURE

diff --git a/patchwork/tests/mail/0014-with-unencoded-utf8-headers.mbox b/patchwork/tests/mail/0014-with-unencoded-utf8-headers.mbox
new file mode 100644 (file)
index 0000000..43c8c45
--- /dev/null
+++ b/patchwork/tests/mail/0014-with-unencoded-utf8-headers.mbox
@@ -0,0 +1,46 @@
+From patchwork Wed Jun  1 20:00:54 2016
+Content-Type: text/plain; charset="utf-8"
+MIME-Version: 1.0
+Content-Transfer-Encoding: 8bit
+Subject: MAINTAINERS: Update entry for BCM5301X ARM to include Rafał Miłecki
+From: Rafał Miłecki <zajec5@gmail.com>
+X-Patchwork-Id: 9148093
+Message-Id: <1464811283-10381-1-git-send-email-zajec5@gmail.com>
+To: Florian Fainelli <f.fainelli@gmail.com>
+Cc: bcm-kernel-feedback-list@broadcom.com,
+ Rafał Miłecki <zajec5@gmail.com>,
+ Andrew Morton <akpm@linux-foundation.org>,
+ "David S. Miller" <davem@davemloft.net>,
+ Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
+ Mauro Carvalho Chehab <mchehab@osg.samsung.com>,
+ Guenter Roeck <linux@roeck-us.net>, Jiri Slaby <jslaby@suse.cz>,
+ linux-kernel@vger.kernel.org (open list)
+Date: Wed,  1 Jun 2016 22:00:54 +0200
+
+Add myself as I contribute to it. Include Broadcom's feedback ML as
+suggested by Florian. Finally modify file rule to match
+bcm5301x-nand-cs0-bch8.dtsi.
+
+Signed-off-by: Rafał Miłecki <zajec5@gmail.com>
+---
+ MAINTAINERS | 4 +++-
+ 1 file changed, 3 insertions(+), 1 deletion(-)
+
+diff --git a/MAINTAINERS b/MAINTAINERS
+index 7304d2e..a5352d8 100644
+--- a/MAINTAINERS
++++ b/MAINTAINERS
+@@ -2511,10 +2511,12 @@ F:     arch/mips/include/asm/mach-bcm47xx/*
+ 
+ BROADCOM BCM5301X ARM ARCHITECTURE
+ M:    Hauke Mehrtens <hauke@hauke-m.de>
++M:    Rafał Miłecki <zajec5@gmail.com>
+ L:    linux-arm-kernel@lists.infradead.org
++L:    bcm-kernel-feedback-list@broadcom.com
+ S:    Maintained
+ F:    arch/arm/mach-bcm/bcm_5301x.c
+-F:    arch/arm/boot/dts/bcm5301x.dtsi
++F:    arch/arm/boot/dts/bcm5301x*.dtsi
+ F:    arch/arm/boot/dts/bcm470*
+ 
+ BROADCOM BCM63XX ARM ARCHITECTURE

diff --git a/patchwork/tests/mail/0015-with-invalid-utf8-headers.mbox b/patchwork/tests/mail/0015-with-invalid-utf8-headers.mbox
new file mode 100644 (file)
index 0000000..57a1b40
--- /dev/null
+++ b/patchwork/tests/mail/0015-with-invalid-utf8-headers.mbox
@@ -0,0 +1,46 @@
+From patchwork Wed Jun  1 20:00:54 2016
+Content-Type: text/plain; charset="utf-8"
+MIME-Version: 1.0
+Content-Transfer-Encoding: 8bit
+Subject: MAINTAINERS: Update entry for BCM5301X ARM
+From: =?utf-8?b?UmFmYcWCIE1pxYJlY2tp?= <zajec5@gmail.com>
+X-Patchwork-Id: 9148093
+Message-Id: <1464811283-10381-1-git-send-email-zajec5@gmail.com>
+To: Florian Fainelli <f.fainelli@gmail.com>
+Cc: bcm-kernel-feedback-list@broadcom.com,
+ =?UTF-8?q?Rafa=FF=FF=20Mi=FF=FFecki?= <zajec5@gmail.com>,
+ Andrew Morton <akpm@linux-foundation.org>,
+ "David S. Miller" <davem@davemloft.net>,
+ Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
+ Mauro Carvalho Chehab <mchehab@osg.samsung.com>,
+ Guenter Roeck <linux@roeck-us.net>, Jiri Slaby <jslaby@suse.cz>,
+ linux-kernel@vger.kernel.org (open list)
+Date: Wed,  1 Jun 2016 22:00:54 +0200
+
+Add myself as I contribute to it. Include Broadcom's feedback ML as
+suggested by Florian. Finally modify file rule to match
+bcm5301x-nand-cs0-bch8.dtsi.
+
+Signed-off-by: Rafał Miłecki <zajec5@gmail.com>
+---
+ MAINTAINERS | 4 +++-
+ 1 file changed, 3 insertions(+), 1 deletion(-)
+
+diff --git a/MAINTAINERS b/MAINTAINERS
+index 7304d2e..a5352d8 100644
+--- a/MAINTAINERS
++++ b/MAINTAINERS
+@@ -2511,10 +2511,12 @@ F:     arch/mips/include/asm/mach-bcm47xx/*
+ 
+ BROADCOM BCM5301X ARM ARCHITECTURE
+ M:    Hauke Mehrtens <hauke@hauke-m.de>
++M:    Rafał Miłecki <zajec5@gmail.com>
+ L:    linux-arm-kernel@lists.infradead.org
++L:    bcm-kernel-feedback-list@broadcom.com
+ S:    Maintained
+ F:    arch/arm/mach-bcm/bcm_5301x.c
+-F:    arch/arm/boot/dts/bcm5301x.dtsi
++F:    arch/arm/boot/dts/bcm5301x*.dtsi
+ F:    arch/arm/boot/dts/bcm470*
+ 
+ BROADCOM BCM63XX ARM ARCHITECTURE

diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index 496818dc4fd42506eff08419615c47fc8982cf90..83ce20429fa472501f61713b559b606f4fb41e58 100644 (file)
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -17,7 +17,7 @@
 # along with Patchwork; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
-from email import message_from_file
+import email
 from email import message_from_string
 from email.mime.multipart import MIMEMultipart
 from email.mime.text import MIMEText
@@ -25,6 +25,7 @@ from email.utils import make_msgid
 import os
 
 from django.test import TestCase
+from django.utils import six
 
 from patchwork.models import Comment
 from patchwork.models import Patch
@@ -52,7 +53,12 @@ TEST_MAIL_DIR = os.path.join(os.path.dirname(__file__), 'mail')
 def read_mail(filename, project=None):
     """Read a mail from a file."""
     file_path = os.path.join(TEST_MAIL_DIR, filename)
-    mail = message_from_file(open(file_path))
+    if six.PY3:
+        with open(file_path, 'rb') as f:
+            mail = email.message_from_binary_file(f)
+    else:
+        with open(file_path) as f:
+            mail = email.message_from_file(f)
     if 'Message-Id' not in mail:
         mail['Message-Id'] = make_msgid()
     if project is not None:
@@ -537,6 +543,37 @@ class PatchParseTest(PatchTest):
         self.assertEqual(2, diff.count('\ No newline at end of file'))
 
 
+class MailParsingTest(TestCase):
+
+    def setUp(self):
+        self.project = create_project()
+
+    def test_invalid_header_char(self):
+        """Validate behaviour when an invalid character is in a header."""
+        mail = read_mail('0012-invalid-header-char.mbox', self.project)
+        parse_mail(mail, list_id=self.project.listid)
+        self.assertEqual(Patch.objects.all().count(), 1)
+
+    def test_utf8_mail(self):
+        """Validate behaviour when a UTF-8 char is in a message."""
+        mail = read_mail('0013-with-utf8-body.mbox')
+        parse_mail(mail, list_id=self.project.listid)
+        self.assertEqual(Patch.objects.all().count(), 1)
+
+    def test_utf8_unencoded_headers(self):
+        """Validate behaviour when unencoded UTF-8 is in headers,
+        including subject and from."""
+        mail = read_mail('0014-with-unencoded-utf8-headers.mbox')
+        parse_mail(mail, list_id=self.project.listid)
+        self.assertEqual(Patch.objects.all().count(), 1)
+
+    def test_invalid_utf8_headers(self):
+        """Validate behaviour when invalid encoded UTF-8 is in headers."""
+        mail = read_mail('0015-with-invalid-utf8-headers.mbox')
+        parse_mail(mail, list_id=self.project.listid)
+        self.assertEqual(Patch.objects.all().count(), 1)
+
+
 class DelegateRequestTest(TestCase):
 
     patch_filename = '0001-add-line.patch'