The 'future.cycle' template tag is removed in Django 1.10. Since
Django 1.7 is still supported by Patchwork, it is necessary to
provide a custom 'cycle' wrapper to prevent the deprecation
warnings resolved by '9cab078' being reintroduced.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Thu, 22 Sep 2016 02:15:09 +0000 (12:15 +1000)]
parsemail: Convert to a management command
Management comands allow applications to register their own actions
with 'manage.py'. This provides some advantages, like automatically
configuring Django (removing the need for 'django.setup' calls) and
removing the need to set the PYTHON_PATH. The 'parsemail' script is a
natural fit for this type of application. Migrate 'parsemail' to a
management command.
This includes some extensive work on logging configuration, as logging
is moved from code into settings. In addition, it removes a lot of the
customizable logging previously introduced in the parsemail command, in
favour of modifications to the settings files.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephen@that.guru>
Partial-bug: #17
Closes-bug: #15
Daniel Axtens [Tue, 20 Sep 2016 08:29:06 +0000 (18:29 +1000)]
parser tests: test parsing of interesting mail
We have 4 new tests:
- One with a header with a character that is neither 7-bit ASCII
nor UTF-8.
- One with UTF-8 in the body.
- One with un-encoded UTF-8 in the headers.
- One with a header that claims to be UTF-8 but isn't.
This also requires us to change how we open files on Python 3. Use
the binary file and message_from_binary_file setup under Python 3.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Wed, 28 Sep 2016 04:06:07 +0000 (14:06 +1000)]
parser: subject_check needs to call clean_header
When testing with headers with unencoded characters on Py3,
subject_check would fail. It was getting an Header class, rather
than a str/bytes, and so the regex was failing.
Run clean_header to decode everything fully first.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Mon, 19 Sep 2016 14:23:58 +0000 (00:23 +1000)]
parser: parse headers containing invalid characters or codings
If there is a non-ascii character in a header, parsing fails,
even on Py27.
This has huge Py2/Py3 complexities. The Py3 email package has tools
to handle this - we just need to use them. Py2, on the other hand,
needs a lot of hand-holding, as explained in the comments.
Additionally, support headers that claim an encoding, but fail to
decode with that encoding.
This is handy for mails with malformed headers containing weird
bytes.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reported-by: Thomas Monjalon <thomas.monjalon@6wind.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Tue, 20 Sep 2016 06:43:24 +0000 (16:43 +1000)]
parsemail.sh: make testing easier
- Take PW_PYTHON, allowing the python version to be set
- Only override DJANGO_SETTINGS_MODULE if it's not in the env
- Pass parameters to parsemail.py
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sun, 25 Sep 2016 21:00:47 +0000 (22:00 +0100)]
Update mailmap with new email
Hotmail is unfortunately not standards compliant and has a bad habit of
rewriting message-ids. This breaks threading for both Mailman and
Patchwork. Clearly it's time to switch to another email address.
Signed-off-by: Stephen Finucane <stephen@that.guru>
The 'with' statement does not start an exception block like 'try' would,
so we get an error executing pwclient:
File "/home/ymorin/bin/pwclient", line 300
except IOError:
^
SyntaxError: invalid syntax
Instead, don't try to catch any exception: any failure will be
propagated back up the stack and eventually cause an abort, which is
anyway what the except clause was trying to do.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Stephen Finucane [Mon, 19 Sep 2016 22:15:26 +0000 (23:15 +0100)]
trivial: noqa any 'assert' functions
There are advantages to sticking with unittest's use of camel case in
function names, but some code tools don't like this. noqa these lines
to hide the warnings.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Daniel Axtens <dja@axtens.net>
* The behavior of 'Bundle.add_patch' is changed. Previously this would
raise an exception if the provided patch already existed in the
bundle. Since this code was only used in one location, change this to
return the BundlePatch if valid else None and change the calling code
to check return value instead of catching the exception.
* Use a context manager to open the config file in pwclient. This loses
a little granularity in error messaging, but this is a worthy
compromise.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Reviewed-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Thu, 15 Sep 2016 08:10:29 +0000 (18:10 +1000)]
Optimise querying of checks in patch list view
tl;dr: with about 300 mails from the patchwork list, according to
django-debug-toolbar, to render '/project/patchwork/list/'
Without this patch:
- ~1.35 seconds of CPU time
- 110 SQL queries, taking ~70ms
With this patch:
- < 0.3 seconds of CPU time
- 10 SQL queries, taking <20ms
How? Replace an .exclude() on a QuerySet with a list comprehension.
Yes, that's normally a pessimisation. Surprisingly, it's an
optimisation here. Why? Where we're looking at patches in anything
that uses a generic_list() in the view, we do a prefetch_related. But,
if we then do a .filter or a .exclude, that throws out the existing,
cached information, and does another query. (See the Django docs on
prefetch_related)
So, do it 'by hand' in Python instead.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Aspects of both Patchwork and Django, such as the 'makemigrations'
management command, function differently under Python 3. It should be
possible to run these commands under Python 2. Install dependencies in
the Python 2 environment to make this a possibility.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Acked-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Tue, 9 Aug 2016 04:27:41 +0000 (14:27 +1000)]
Add .travis.yml file
This automatically runs the tests (except the browser ones) via tox.
Tested on my github account. This will require some setup on the GitHub
end: Travis will need to be turned on for the project, and ideally a
badge added to the readme so we can see the build status.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Thomas Monjalon [Tue, 6 Sep 2016 20:37:03 +0000 (22:37 +0200)]
lib: Refresh nginx/uwsgi configuration
As stated in http://uwsgi-docs.readthedocs.io/en/latest/Nginx.html:
"
Unfortunately nginx is not able to rewrite PATH_INFO accordingly to
SCRIPT_NAME. For such reason you need to instruct uWSGI to map specific
apps in the so called “mountpoint” and rewrite SCRIPT_NAME and PATH_INFO
automatically
[...]
ancient uWSGI versions used to support the so called “uwsgi_modifier1 30”
approach. Do not do it. it is a really ugly hack
"
Replacing the uwsgi_modifier1 hack by a mount point seems to work.
Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
In '8e58c29', code was added to delete '.pyc' files. However, this was
later reverted in '9fd161' due to the increase in run time that this
incurred. However, the need to avoid stale '.pyc' files still exists.
To resolve this, add the 'PYTHONDONTWRITEBYTECODE' environment option
to disable generation of these files. This includes a slight
performance increase.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Sat, 3 Sep 2016 07:07:20 +0000 (17:07 +1000)]
xmlrpc: massively increase test coverage
XMLRPC defines a bunch of very similar methods for getting and
listing a bunch of different models. So create a mixin that allows
us to test an arbitrary model, then use that to create test classes
to test the various models.
Add other tests as well.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Sat, 3 Sep 2016 07:07:17 +0000 (17:07 +1000)]
xmlrpc: catch possible exceptions in patch filtering
Currently, filtering by project, submitter, delegate or state uses a
filter(id=filt[key])[0]. This will throw an exception when something
isn't found, as filter will return [], and getting the first element of
that will fail.
Convert them to explicit get()s, so it's clearer that they can throw an
exception, then catch the 3 possible types of DoesNotExists exceptions.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Sat, 3 Sep 2016 07:07:14 +0000 (17:07 +1000)]
xmlrpc: Change filter(...)[0] to get(...)
filter() returns a list or [], and get() returns an object or raises an
exception.
Currently, we use filter()[0] inside a try/except model.DoesNotExist
block. This doesn't work: if filter returns [] we'll get an exception
about list indices, not a DoesNotExist exception.
Switch these over to get(), allowing them to use the existing exception
flow.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Stephen Finucane [Mon, 18 Apr 2016 10:21:09 +0000 (11:21 +0100)]
parser: Correct empty email value check
The check for empty emails in 'find_author' checked for None, but it
was not possible to ever return None unless the 'From:' header was
missing altogether. Seeing as, per RFC822 and its revisions, this is
not possible, check for the empty string instead.
Signed-off-by: Stephen Finucane <stephen.finucane@intel.com> Reviewed-by: Andy Doan <andy.doan@linaro.org>
Stephen Finucane [Thu, 14 Apr 2016 16:38:57 +0000 (17:38 +0100)]
parser: Put more specific 'From' regex first
Multiple different 'From:' header styles are supported by the parser,
but some of these are less specific than others. Attempt to parse using
a more specific style before going with the more generic one.
Signed-off-by: Stephen Finucane <stephen.finucane@intel.com> Reviewed-by: Andy Doan <andy.doan@linaro.org>
Stephen Finucane [Sun, 10 Apr 2016 12:45:13 +0000 (13:45 +0100)]
parsemail: Move parsing to 'parser'
Separate the parsing of mails from the CLI aspects of parsemail. Since
there is already a 'parser' module, it makes sense to place the parsing
functionality here. This will allow for additional uses of this parsing
functionality in the future.
Signed-off-by: Stephen Finucane <stephen.finucane@intel.com> Reviewed-by: Andy Doan <andy.doan@linaro.org>
Stephen Finucane [Sun, 10 Apr 2016 12:45:13 +0000 (13:45 +0100)]
trivial: Cleanup of 'parser'
Do some cleanup of the file by removing excess whitespace, adding some
documentation, removing shadowing of keywords and renaming some
functions to more accurately reflect their purpose.
Signed-off-by: Stephen Finucane <stephen.finucane@intel.com> Reviewed-by: Andy Doan <andy.doan@linaro.org>
Stephen Finucane [Sun, 21 Aug 2016 17:06:55 +0000 (18:06 +0100)]
requirements: Update django debug toolbar to 1.5
This is the latest version. This breaks compatibility with Django 1.7,
but this version is unsupported and should not be used for development.
We do, however, continue to test support for Django 1.7 and 1.6.
This also prevents the need to manually track sqlparse dependency [1].
Daniel Axtens [Tue, 9 Aug 2016 04:55:23 +0000 (14:55 +1000)]
tox: Add Python 3.5
Python 3.5 is shipped with Ubuntu 16.04, and it's the new stable
release. Add support for this runtime in tox. Note that Django 1.6 and
1.7 don't work with Python 3.5, so these are excluded.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Fri, 26 Aug 2016 06:55:59 +0000 (16:55 +1000)]
docker: Install Python 3.4
Currently, we tox test against Python 3.4. Python 3.4 is included with
Ubuntu 14.04, which is supported for several years to come, so we want
to keep supporting it. However, Python 3.4 isn't included with Ubuntu
16.04, which is what the Docker image is based on.
We could downgrade the container to 14.04, but then we'd struggle to
get Python 3.5 into the container. Python 3.5 is the most recent in the
Python 3 series, so we should also be supporting it.
Add the apt sources for Trusty and pull in Python 3.4 from there.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Florian Fainelli [Tue, 30 Aug 2016 01:12:15 +0000 (18:12 -0700)]
docs: Add a paragraph about the django admin console
Provide a few hints about how to get initial access to the django admin
console as well as a few essential configuration hints regarding
projects and sites setup.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Reviewed-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Florian Fainelli [Tue, 30 Aug 2016 01:12:14 +0000 (18:12 -0700)]
pwclient: Unhardcode prints to ~/.pwclientrc
Even though the default location is ~/.pwclientrc, we can change it, so
use CONFIG_FILE wherever we can to make the prints match the actual file
location.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Stephen Finucane [Sat, 13 Aug 2016 22:43:04 +0000 (23:43 +0100)]
tox: Disable F405
Recent versions of tox throw a lot of errors when using star-imports.
While these warnings are generally justified, the settings files
require this. Since 'noqa'-ing the offending lines would be rather
arduous, simply disable the warning instead.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Tue, 9 Aug 2016 04:38:45 +0000 (14:38 +1000)]
Add Docker IP to whitelist for django-debug-toolbar
Yay for debugging with Docker.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Tue, 9 Aug 2016 04:27:40 +0000 (14:27 +1000)]
Allow use of Docker for development
This makes it possible to use Docker and docker-compose for development
as an alternative to Vagrant.
I quite liked vagrant a couple of years ago, but currently:
* Trying to install VirtualBox on Ubuntu wants me to disable
Secure Boot, and I don't want to do that.
* Trying to use the libvirt plugin for vagrant requires I pick
from a very small set of possible images, and requires that I
install the upstream vagrant rather than the vagrant shipped
with Ubuntu 16.04
* I find docker containers faster to work with and more transparent.
So I've done the work to make docker work for Patchwork development.
This doesn't break or in any way interfere with using Vagrant, it just
provides an alternative.
It includes support for headless selenium tests using Chromium.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Tue, 9 Aug 2016 04:27:39 +0000 (14:27 +1000)]
Put test db host and port into env variables
This is preparation for a Docker-based dev environment.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Tue, 9 Aug 2016 04:27:38 +0000 (14:27 +1000)]
Ignore *~ and #*# files.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Tue, 9 Aug 2016 04:27:37 +0000 (14:27 +1000)]
Fix test_login in test_user_browser.py
It tried to use the pbkdf2 hash as the password. Use the username instead, as that
is what create_user sets.
Then it compared the test user username to testuser, rather than the username, which
is dynamically generated. Compare to the generated username.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
Daniel Axtens [Tue, 9 Aug 2016 04:27:36 +0000 (14:27 +1000)]
Fix failure to start with uninitalised database
An OptionalModelChoiceField will attempt to query the database to get choices
in its __init__ method. This fails if the database hasn't been initialised
yet. So, put that in a try/catch block. This lets things work through the
migration and loading of data from fixtures.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephenfinucane@hotmail.com>
projects / thirdparty / patchwork.git / log
