Daniel Axtens [Thu, 17 Nov 2016 05:11:59 +0000 (16:11 +1100)]
Fix parsing of interesing series reply structures
There are some things you probably shouldn't do on public
mailing lists, but which people do anyway.
The first, and most understandable, is this:
- [PATCH 1/2] test: Add some lorem ipsum
- [PATCH 2/2] test: Convert to Markdown
- [PATCH v2 1/2] test: Add some lorem ipsum
- [PATCH v2 2/2] test: Convert to Markdown
We should correctly parse these by:
- creating a new series if the version number changes
- when deciding whether to create a SeriesReference, search by
message-id alone, not the message-id/series pair. (Otherwise,
we try to create a series ref for v1 2/2 in the series for v2,
which breaks a uniqueness constraint.
The second, and less excusable, is this:
- [PATCH 1/2] test: Add some lorem ipsum
- [PATCH 2/2] test: Convert to Markdown
- [PATCH 1/2] test: Add some lorem ipsum
- [PATCH 2/2] test: Convert to Markdown
With this patch:
- if we get a x/n for a series that already has an x/n, create a
new series for it.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Wed, 16 Nov 2016 05:58:44 +0000 (16:58 +1100)]
Fix parsing of un-numbered messages in series
Say we are sent the following:
- [PATCH 0/2] A sample series
- [PATCH 1/2] test: Add some lorem ipsum
- Random message with diff
We expect that:
1) we parse normally without errors
2) we get a series with a cover letter and a patch
3) the random message is orphaned
What happens is that we get an integrity error, boiling down to:
(1048, "Column 'number' cannot be null")
This is caused because we believe that the random message belongs
to the series because of the headers, but because there are no
numbers in the Subject, we pass "None" into the number field of
SeriesPatch. That turns into a null, and rightly hits an integrity
error.
Fix this by requring that a message has a series _and_ a number
before we try to add it to the series.
Add a test to verify correctness.
Reported-by: Daniel Wagner <wagi@monom.org> Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sat, 29 Oct 2016 13:13:40 +0000 (14:13 +0100)]
templates: Integrate series view into patches
Patches for related series are listed on the patch page - this
provides a way to quickly grok a given patches location in a
series hierarchy.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andy Doan <andy.doan@linaro.org> Reviewed-by: Daniel Axtens <dja@axtens.net> Tested-by: Russell Currey <ruscur@russell.cc>
Stephen Finucane [Sat, 29 Oct 2016 13:13:39 +0000 (14:13 +0100)]
templates: Integrate series support
Integrate support for series in the web UI. This is rather
straightforward, the only significant change being the addition of a
filter for series filtering.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andy Doan <andy.doan@linaro.org> Tested-by: Russell Currey <ruscur@russell.cc> Reviewed-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Stephen Finucane [Sat, 29 Oct 2016 13:13:38 +0000 (14:13 +0100)]
filters: Handle invalid ids
Two filters - SubmitterFilter and DelegateFilter - don't attempt to
handle invalid id values and will bubble an exception up as a 5xx
error. Correct this.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Stephen Finucane [Sat, 29 Oct 2016 13:13:35 +0000 (14:13 +0100)]
parser: Add series parsing
It is now possible to parse and store series, so do just that.
The parsing at the moment is based on both RFC822 headers and
subject lines.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andy Doan <andy.doan@linaro.org> Reviewed-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Tested-by: Russell Currey <ruscur@russell.cc>
Stephen Finucane [Sat, 29 Oct 2016 13:13:34 +0000 (14:13 +0100)]
models: Add 'Series' model
Add a series model. This model is expected to act like a collection for
patches, similar to bundles but thread-orientated.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andy Doan <andy.doan@linaro.org> Reviewed-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Tested-by: Russell Currey <ruscur@russell.cc>
Stephen Finucane [Sat, 29 Oct 2016 13:13:33 +0000 (14:13 +0100)]
models: Convert functions to properties
A number of models contain functions that are, semantically speaking,
actually properties. Mark them as such.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andy Doan <andy.doan@linaro.org> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Stephen Finucane [Sat, 22 Oct 2016 11:23:27 +0000 (12:23 +0100)]
views: Update how patch counts are retrieved
It's no longer possible to access 'Project.patch_set' as
'Project.submission_set' has replaced it. This was missed when the
cover letter feature was merged, so resolve it now.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Closes-bug: #54
Stephen Finucane [Thu, 20 Oct 2016 07:08:39 +0000 (08:08 +0100)]
docs: Add getmail documentation
It seems a lot of people are having success using tools like fetchmail
in combination with IMAP/POP-capable email accounts like Gmail. While
fetchmail itself is rather decrepit, the Python-based getmail seems
actively supported and pretty easy to configure. Document this process.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Stephen Finucane [Thu, 20 Oct 2016 07:28:53 +0000 (08:28 +0100)]
models: Make use of aggregates
We're well past Django 1.1 now, so resolve a TODO to use aggregate
support introduced in this version. As part of this change, replace
the use of 'count' to check for presence of matching objects with
'exists'.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Stephen Finucane [Tue, 11 Oct 2016 17:14:36 +0000 (18:14 +0100)]
docs: Rework XML-RPC docs
Add a barebones usage guide for pwclient and simplify the existing doc
for this API. This entails move the existing doc from the deployment
section to development section, which makes more sense given its focus.
Signed-off-by: Stephen Finucane <stephen@that.guru>
The 'future.cycle' template tag is removed in Django 1.10. Since
Django 1.7 is still supported by Patchwork, it is necessary to
provide a custom 'cycle' wrapper to prevent the deprecation
warnings resolved by '9cab078' being reintroduced.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Thu, 22 Sep 2016 02:15:09 +0000 (12:15 +1000)]
parsemail: Convert to a management command
Management comands allow applications to register their own actions
with 'manage.py'. This provides some advantages, like automatically
configuring Django (removing the need for 'django.setup' calls) and
removing the need to set the PYTHON_PATH. The 'parsemail' script is a
natural fit for this type of application. Migrate 'parsemail' to a
management command.
This includes some extensive work on logging configuration, as logging
is moved from code into settings. In addition, it removes a lot of the
customizable logging previously introduced in the parsemail command, in
favour of modifications to the settings files.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephen@that.guru>
Partial-bug: #17
Closes-bug: #15
Daniel Axtens [Tue, 20 Sep 2016 08:29:06 +0000 (18:29 +1000)]
parser tests: test parsing of interesting mail
We have 4 new tests:
- One with a header with a character that is neither 7-bit ASCII
nor UTF-8.
- One with UTF-8 in the body.
- One with un-encoded UTF-8 in the headers.
- One with a header that claims to be UTF-8 but isn't.
This also requires us to change how we open files on Python 3. Use
the binary file and message_from_binary_file setup under Python 3.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Wed, 28 Sep 2016 04:06:07 +0000 (14:06 +1000)]
parser: subject_check needs to call clean_header
When testing with headers with unencoded characters on Py3,
subject_check would fail. It was getting an Header class, rather
than a str/bytes, and so the regex was failing.
Run clean_header to decode everything fully first.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Mon, 19 Sep 2016 14:23:58 +0000 (00:23 +1000)]
parser: parse headers containing invalid characters or codings
If there is a non-ascii character in a header, parsing fails,
even on Py27.
This has huge Py2/Py3 complexities. The Py3 email package has tools
to handle this - we just need to use them. Py2, on the other hand,
needs a lot of hand-holding, as explained in the comments.
Additionally, support headers that claim an encoding, but fail to
decode with that encoding.
This is handy for mails with malformed headers containing weird
bytes.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reported-by: Thomas Monjalon <thomas.monjalon@6wind.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Tue, 20 Sep 2016 06:43:24 +0000 (16:43 +1000)]
parsemail.sh: make testing easier
- Take PW_PYTHON, allowing the python version to be set
- Only override DJANGO_SETTINGS_MODULE if it's not in the env
- Pass parameters to parsemail.py
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sun, 25 Sep 2016 21:00:47 +0000 (22:00 +0100)]
Update mailmap with new email
Hotmail is unfortunately not standards compliant and has a bad habit of
rewriting message-ids. This breaks threading for both Mailman and
Patchwork. Clearly it's time to switch to another email address.
Signed-off-by: Stephen Finucane <stephen@that.guru>
The 'with' statement does not start an exception block like 'try' would,
so we get an error executing pwclient:
File "/home/ymorin/bin/pwclient", line 300
except IOError:
^
SyntaxError: invalid syntax
Instead, don't try to catch any exception: any failure will be
propagated back up the stack and eventually cause an abort, which is
anyway what the except clause was trying to do.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Stephen Finucane [Mon, 19 Sep 2016 22:15:26 +0000 (23:15 +0100)]
trivial: noqa any 'assert' functions
There are advantages to sticking with unittest's use of camel case in
function names, but some code tools don't like this. noqa these lines
to hide the warnings.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Daniel Axtens <dja@axtens.net>
* The behavior of 'Bundle.add_patch' is changed. Previously this would
raise an exception if the provided patch already existed in the
bundle. Since this code was only used in one location, change this to
return the BundlePatch if valid else None and change the calling code
to check return value instead of catching the exception.
* Use a context manager to open the config file in pwclient. This loses
a little granularity in error messaging, but this is a worthy
compromise.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Thu, 15 Sep 2016 08:10:29 +0000 (18:10 +1000)]
Optimise querying of checks in patch list view
tl;dr: with about 300 mails from the patchwork list, according to
django-debug-toolbar, to render '/project/patchwork/list/'
Without this patch:
- ~1.35 seconds of CPU time
- 110 SQL queries, taking ~70ms
With this patch:
- < 0.3 seconds of CPU time
- 10 SQL queries, taking <20ms
How? Replace an .exclude() on a QuerySet with a list comprehension.
Yes, that's normally a pessimisation. Surprisingly, it's an
optimisation here. Why? Where we're looking at patches in anything
that uses a generic_list() in the view, we do a prefetch_related. But,
if we then do a .filter or a .exclude, that throws out the existing,
cached information, and does another query. (See the Django docs on
prefetch_related)
So, do it 'by hand' in Python instead.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Aspects of both Patchwork and Django, such as the 'makemigrations'
management command, function differently under Python 3. It should be
possible to run these commands under Python 2. Install dependencies in
the Python 2 environment to make this a possibility.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Acked-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Tue, 9 Aug 2016 04:27:41 +0000 (14:27 +1000)]
Add .travis.yml file
This automatically runs the tests (except the browser ones) via tox.
Tested on my github account. This will require some setup on the GitHub
end: Travis will need to be turned on for the project, and ideally a
badge added to the readme so we can see the build status.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Thomas Monjalon [Tue, 6 Sep 2016 20:37:03 +0000 (22:37 +0200)]
lib: Refresh nginx/uwsgi configuration
As stated in http://uwsgi-docs.readthedocs.io/en/latest/Nginx.html:
"
Unfortunately nginx is not able to rewrite PATH_INFO accordingly to
SCRIPT_NAME. For such reason you need to instruct uWSGI to map specific
apps in the so called “mountpoint” and rewrite SCRIPT_NAME and PATH_INFO
automatically
[...]
ancient uWSGI versions used to support the so called “uwsgi_modifier1 30”
approach. Do not do it. it is a really ugly hack
"
Replacing the uwsgi_modifier1 hack by a mount point seems to work.
Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
In '8e58c29', code was added to delete '.pyc' files. However, this was
later reverted in '9fd161' due to the increase in run time that this
incurred. However, the need to avoid stale '.pyc' files still exists.
To resolve this, add the 'PYTHONDONTWRITEBYTECODE' environment option
to disable generation of these files. This includes a slight
performance increase.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Sat, 3 Sep 2016 07:07:20 +0000 (17:07 +1000)]
xmlrpc: massively increase test coverage
XMLRPC defines a bunch of very similar methods for getting and
listing a bunch of different models. So create a mixin that allows
us to test an arbitrary model, then use that to create test classes
to test the various models.
Add other tests as well.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Sat, 3 Sep 2016 07:07:17 +0000 (17:07 +1000)]
xmlrpc: catch possible exceptions in patch filtering
Currently, filtering by project, submitter, delegate or state uses a
filter(id=filt[key])[0]. This will throw an exception when something
isn't found, as filter will return [], and getting the first element of
that will fail.
Convert them to explicit get()s, so it's clearer that they can throw an
exception, then catch the 3 possible types of DoesNotExists exceptions.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
projects / thirdparty / patchwork.git / log
