Stephen Finucane [Sat, 22 Oct 2016 11:23:27 +0000 (12:23 +0100)]
views: Update how patch counts are retrieved
It's no longer possible to access 'Project.patch_set' as
'Project.submission_set' has replaced it. This was missed when the
cover letter feature was merged, so resolve it now.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Closes-bug: #54
Stephen Finucane [Thu, 20 Oct 2016 07:08:39 +0000 (08:08 +0100)]
docs: Add getmail documentation
It seems a lot of people are having success using tools like fetchmail
in combination with IMAP/POP-capable email accounts like Gmail. While
fetchmail itself is rather decrepit, the Python-based getmail seems
actively supported and pretty easy to configure. Document this process.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Stephen Finucane [Thu, 20 Oct 2016 07:28:53 +0000 (08:28 +0100)]
models: Make use of aggregates
We're well past Django 1.1 now, so resolve a TODO to use aggregate
support introduced in this version. As part of this change, replace
the use of 'count' to check for presence of matching objects with
'exists'.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Stephen Finucane [Tue, 11 Oct 2016 17:14:36 +0000 (18:14 +0100)]
docs: Rework XML-RPC docs
Add a barebones usage guide for pwclient and simplify the existing doc
for this API. This entails move the existing doc from the deployment
section to development section, which makes more sense given its focus.
Signed-off-by: Stephen Finucane <stephen@that.guru>
The 'future.cycle' template tag is removed in Django 1.10. Since
Django 1.7 is still supported by Patchwork, it is necessary to
provide a custom 'cycle' wrapper to prevent the deprecation
warnings resolved by '9cab078' being reintroduced.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Thu, 22 Sep 2016 02:15:09 +0000 (12:15 +1000)]
parsemail: Convert to a management command
Management comands allow applications to register their own actions
with 'manage.py'. This provides some advantages, like automatically
configuring Django (removing the need for 'django.setup' calls) and
removing the need to set the PYTHON_PATH. The 'parsemail' script is a
natural fit for this type of application. Migrate 'parsemail' to a
management command.
This includes some extensive work on logging configuration, as logging
is moved from code into settings. In addition, it removes a lot of the
customizable logging previously introduced in the parsemail command, in
favour of modifications to the settings files.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephen@that.guru>
Partial-bug: #17
Closes-bug: #15
Daniel Axtens [Tue, 20 Sep 2016 08:29:06 +0000 (18:29 +1000)]
parser tests: test parsing of interesting mail
We have 4 new tests:
- One with a header with a character that is neither 7-bit ASCII
nor UTF-8.
- One with UTF-8 in the body.
- One with un-encoded UTF-8 in the headers.
- One with a header that claims to be UTF-8 but isn't.
This also requires us to change how we open files on Python 3. Use
the binary file and message_from_binary_file setup under Python 3.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Wed, 28 Sep 2016 04:06:07 +0000 (14:06 +1000)]
parser: subject_check needs to call clean_header
When testing with headers with unencoded characters on Py3,
subject_check would fail. It was getting an Header class, rather
than a str/bytes, and so the regex was failing.
Run clean_header to decode everything fully first.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Mon, 19 Sep 2016 14:23:58 +0000 (00:23 +1000)]
parser: parse headers containing invalid characters or codings
If there is a non-ascii character in a header, parsing fails,
even on Py27.
This has huge Py2/Py3 complexities. The Py3 email package has tools
to handle this - we just need to use them. Py2, on the other hand,
needs a lot of hand-holding, as explained in the comments.
Additionally, support headers that claim an encoding, but fail to
decode with that encoding.
This is handy for mails with malformed headers containing weird
bytes.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reported-by: Thomas Monjalon <thomas.monjalon@6wind.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Tue, 20 Sep 2016 06:43:24 +0000 (16:43 +1000)]
parsemail.sh: make testing easier
- Take PW_PYTHON, allowing the python version to be set
- Only override DJANGO_SETTINGS_MODULE if it's not in the env
- Pass parameters to parsemail.py
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sun, 25 Sep 2016 21:00:47 +0000 (22:00 +0100)]
Update mailmap with new email
Hotmail is unfortunately not standards compliant and has a bad habit of
rewriting message-ids. This breaks threading for both Mailman and
Patchwork. Clearly it's time to switch to another email address.
Signed-off-by: Stephen Finucane <stephen@that.guru>
The 'with' statement does not start an exception block like 'try' would,
so we get an error executing pwclient:
File "/home/ymorin/bin/pwclient", line 300
except IOError:
^
SyntaxError: invalid syntax
Instead, don't try to catch any exception: any failure will be
propagated back up the stack and eventually cause an abort, which is
anyway what the except clause was trying to do.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Stephen Finucane [Mon, 19 Sep 2016 22:15:26 +0000 (23:15 +0100)]
trivial: noqa any 'assert' functions
There are advantages to sticking with unittest's use of camel case in
function names, but some code tools don't like this. noqa these lines
to hide the warnings.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Daniel Axtens <dja@axtens.net>
* The behavior of 'Bundle.add_patch' is changed. Previously this would
raise an exception if the provided patch already existed in the
bundle. Since this code was only used in one location, change this to
return the BundlePatch if valid else None and change the calling code
to check return value instead of catching the exception.
* Use a context manager to open the config file in pwclient. This loses
a little granularity in error messaging, but this is a worthy
compromise.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Thu, 15 Sep 2016 08:10:29 +0000 (18:10 +1000)]
Optimise querying of checks in patch list view
tl;dr: with about 300 mails from the patchwork list, according to
django-debug-toolbar, to render '/project/patchwork/list/'
Without this patch:
- ~1.35 seconds of CPU time
- 110 SQL queries, taking ~70ms
With this patch:
- < 0.3 seconds of CPU time
- 10 SQL queries, taking <20ms
How? Replace an .exclude() on a QuerySet with a list comprehension.
Yes, that's normally a pessimisation. Surprisingly, it's an
optimisation here. Why? Where we're looking at patches in anything
that uses a generic_list() in the view, we do a prefetch_related. But,
if we then do a .filter or a .exclude, that throws out the existing,
cached information, and does another query. (See the Django docs on
prefetch_related)
So, do it 'by hand' in Python instead.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Aspects of both Patchwork and Django, such as the 'makemigrations'
management command, function differently under Python 3. It should be
possible to run these commands under Python 2. Install dependencies in
the Python 2 environment to make this a possibility.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Acked-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Tue, 9 Aug 2016 04:27:41 +0000 (14:27 +1000)]
Add .travis.yml file
This automatically runs the tests (except the browser ones) via tox.
Tested on my github account. This will require some setup on the GitHub
end: Travis will need to be turned on for the project, and ideally a
badge added to the readme so we can see the build status.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Thomas Monjalon [Tue, 6 Sep 2016 20:37:03 +0000 (22:37 +0200)]
lib: Refresh nginx/uwsgi configuration
As stated in http://uwsgi-docs.readthedocs.io/en/latest/Nginx.html:
"
Unfortunately nginx is not able to rewrite PATH_INFO accordingly to
SCRIPT_NAME. For such reason you need to instruct uWSGI to map specific
apps in the so called “mountpoint” and rewrite SCRIPT_NAME and PATH_INFO
automatically
[...]
ancient uWSGI versions used to support the so called “uwsgi_modifier1 30”
approach. Do not do it. it is a really ugly hack
"
Replacing the uwsgi_modifier1 hack by a mount point seems to work.
Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
In '8e58c29', code was added to delete '.pyc' files. However, this was
later reverted in '9fd161' due to the increase in run time that this
incurred. However, the need to avoid stale '.pyc' files still exists.
To resolve this, add the 'PYTHONDONTWRITEBYTECODE' environment option
to disable generation of these files. This includes a slight
performance increase.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Sat, 3 Sep 2016 07:07:20 +0000 (17:07 +1000)]
xmlrpc: massively increase test coverage
XMLRPC defines a bunch of very similar methods for getting and
listing a bunch of different models. So create a mixin that allows
us to test an arbitrary model, then use that to create test classes
to test the various models.
Add other tests as well.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Sat, 3 Sep 2016 07:07:17 +0000 (17:07 +1000)]
xmlrpc: catch possible exceptions in patch filtering
Currently, filtering by project, submitter, delegate or state uses a
filter(id=filt[key])[0]. This will throw an exception when something
isn't found, as filter will return [], and getting the first element of
that will fail.
Convert them to explicit get()s, so it's clearer that they can throw an
exception, then catch the 3 possible types of DoesNotExists exceptions.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Sat, 3 Sep 2016 07:07:14 +0000 (17:07 +1000)]
xmlrpc: Change filter(...)[0] to get(...)
filter() returns a list or [], and get() returns an object or raises an
exception.
Currently, we use filter()[0] inside a try/except model.DoesNotExist
block. This doesn't work: if filter returns [] we'll get an exception
about list indices, not a DoesNotExist exception.
Switch these over to get(), allowing them to use the existing exception
flow.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Stephen Finucane [Mon, 18 Apr 2016 10:21:09 +0000 (11:21 +0100)]
parser: Correct empty email value check
The check for empty emails in 'find_author' checked for None, but it
was not possible to ever return None unless the 'From:' header was
missing altogether. Seeing as, per RFC822 and its revisions, this is
not possible, check for the empty string instead.
Signed-off-by: Stephen Finucane <stephen.finucane@intel.com> Reviewed-by: Andy Doan <andy.doan@linaro.org>
Stephen Finucane [Thu, 14 Apr 2016 16:38:57 +0000 (17:38 +0100)]
parser: Put more specific 'From' regex first
Multiple different 'From:' header styles are supported by the parser,
but some of these are less specific than others. Attempt to parse using
a more specific style before going with the more generic one.
Signed-off-by: Stephen Finucane <stephen.finucane@intel.com> Reviewed-by: Andy Doan <andy.doan@linaro.org>
Stephen Finucane [Sun, 10 Apr 2016 12:45:13 +0000 (13:45 +0100)]
parsemail: Move parsing to 'parser'
Separate the parsing of mails from the CLI aspects of parsemail. Since
there is already a 'parser' module, it makes sense to place the parsing
functionality here. This will allow for additional uses of this parsing
functionality in the future.
Signed-off-by: Stephen Finucane <stephen.finucane@intel.com> Reviewed-by: Andy Doan <andy.doan@linaro.org>
Stephen Finucane [Sun, 10 Apr 2016 12:45:13 +0000 (13:45 +0100)]
trivial: Cleanup of 'parser'
Do some cleanup of the file by removing excess whitespace, adding some
documentation, removing shadowing of keywords and renaming some
functions to more accurately reflect their purpose.
Signed-off-by: Stephen Finucane <stephen.finucane@intel.com> Reviewed-by: Andy Doan <andy.doan@linaro.org>
Stephen Finucane [Sun, 21 Aug 2016 17:06:55 +0000 (18:06 +0100)]
requirements: Update django debug toolbar to 1.5
This is the latest version. This breaks compatibility with Django 1.7,
but this version is unsupported and should not be used for development.
We do, however, continue to test support for Django 1.7 and 1.6.
This also prevents the need to manually track sqlparse dependency [1].
Daniel Axtens [Tue, 9 Aug 2016 04:55:23 +0000 (14:55 +1000)]
tox: Add Python 3.5
Python 3.5 is shipped with Ubuntu 16.04, and it's the new stable
release. Add support for this runtime in tox. Note that Django 1.6 and
1.7 don't work with Python 3.5, so these are excluded.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Fri, 26 Aug 2016 06:55:59 +0000 (16:55 +1000)]
docker: Install Python 3.4
Currently, we tox test against Python 3.4. Python 3.4 is included with
Ubuntu 14.04, which is supported for several years to come, so we want
to keep supporting it. However, Python 3.4 isn't included with Ubuntu
16.04, which is what the Docker image is based on.
We could downgrade the container to 14.04, but then we'd struggle to
get Python 3.5 into the container. Python 3.5 is the most recent in the
Python 3 series, so we should also be supporting it.
Add the apt sources for Trusty and pull in Python 3.4 from there.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Florian Fainelli [Tue, 30 Aug 2016 01:12:15 +0000 (18:12 -0700)]
docs: Add a paragraph about the django admin console
Provide a few hints about how to get initial access to the django admin
console as well as a few essential configuration hints regarding
projects and sites setup.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Reviewed-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Florian Fainelli [Tue, 30 Aug 2016 01:12:14 +0000 (18:12 -0700)]
pwclient: Unhardcode prints to ~/.pwclientrc
Even though the default location is ~/.pwclientrc, we can change it, so
use CONFIG_FILE wherever we can to make the prints match the actual file
location.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Stephen Finucane [Sat, 13 Aug 2016 22:43:04 +0000 (23:43 +0100)]
tox: Disable F405
Recent versions of tox throw a lot of errors when using star-imports.
While these warnings are generally justified, the settings files
require this. Since 'noqa'-ing the offending lines would be rather
arduous, simply disable the warning instead.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
projects / thirdparty / patchwork.git / log
