Daniel Axtens [Wed, 28 Jun 2017 07:48:48 +0000 (17:48 +1000)]
parser: better date parsing
It turns out that there is a lot that can go wrong in parsing a
date. OverflowError, ValueError and OSError have all been observed.
If these go wrong, substitute the current datetime.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Daniel Axtens [Wed, 28 Jun 2017 07:48:47 +0000 (17:48 +1000)]
parser: deal with headers entirely failing to parse
It turns out that the attempts in clean_header() to convert
headers to strings are not guaranteed to work: you can end up with,
for example, a base64 decoding error which makes it impossible
to determine any header content.
In this case, sanitise_header() should return None, and thus
clean_header() should return None. We then need to plumb that
through.
Signed-off-by: Daniel Axtens <dja@axtens.net> Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Daniel Axtens [Wed, 28 Jun 2017 07:48:46 +0000 (17:48 +1000)]
parser: catch failures in decoding headers
Headers can fail to decode:
- if a part cannot be encoded as ascii
- if the coding hint names a codec that doesn't exist
- if there's a null byte in the codec name
Catch these errors.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Wed, 28 Jun 2017 07:48:44 +0000 (17:48 +1000)]
parser: don't assume headers are strings
In python3, mail.get() can return either a string, or an
email.header.Header type.
clean_header() is designed to clean headers into strings,
so make sure we use that everywhere.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Wed, 28 Jun 2017 07:48:43 +0000 (17:48 +1000)]
parser: fix charset 'guessing' algorithm
The charset guessing algorithm doesn't work if it has to guess
multiple charsets, as it overwrites the payload with None.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Wed, 28 Jun 2017 06:55:09 +0000 (16:55 +1000)]
Remove ResourceWarnings under Py3
This is just a matter of correctly closing files we open.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Aaron Conole [Wed, 14 Jun 2017 18:14:23 +0000 (14:14 -0400)]
events-api: allow filtering by date
This commit allows users of the REST API to query for events based on
the date field. This will allow utility writers to select a smaller
subset of events when polling.
Signed-off-by: Aaron Conole <aconole@bytheb.org> Reviewed-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Wed, 28 Jun 2017 19:54:03 +0000 (20:54 +0100)]
parser: Trivial fix of test docstring
Based on the 'References' and 'In-Reply-To' headers of the used mbox,
the docstring for the 'test_reply_nocover_noversion' test is incorrect.
Fix this.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Sean Farley <sean@farley.io>
Stephen Finucane [Wed, 28 Jun 2017 19:53:58 +0000 (20:53 +0100)]
parser: Support single-patch "series"
There are merits to supporting single-patch series, not least the
ability to provide two consistent interfaces that show _all_ patches in
the '/patches' endpoint and the '/series' endpoint.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Sean Farley <sean@farley.io>
Stephen Finucane [Sun, 25 Jun 2017 20:55:54 +0000 (21:55 +0100)]
parser: Use a series even if the version differs
Currently, once we've found a series for a new submission, we check to
ensure that the version of that series matches that of the submission,
and we discard the series if not. The original intention for this was
given in commit 'c21b30525', where this was added:
There are some things you probably shouldn't do on public mailing
lists, but which people do anyway.
- [PATCH 1/2] test: Add some lorem ipsum
- [PATCH 2/2] test: Convert to Markdown
- [PATCH v2 1/2] test: Add some lorem ipsum
- [PATCH v2 2/2] test: Convert to Markdown
We should correctly parse these...
This is unnecessary for two reasons:
- If the series is using proper references, then the mechanism that
we use to prevent this issue with unversioned follow ups (also added
in that patch) will work here: namely, we don't add submissions if an
submission with a given number already exists in that series.
- If the series is not using references, we already have a check for
version.
Seeing as this check is actually causing issues for legitimate typos, we
should just remove this check. Do so, adding a check to ensure we don't
regress.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Closes-bug: #105
Token authentication is generally viewed as a more secure option for API
authentication than storing a username and password.
Django REST Framework gives us a TokenAuthentication class and an authtoken
app that we can use to generate random tokens and authenticate to API
endpoints. Enable this support and add some tests to validate correct
behavior.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Signed-off-by: Stephen Finucane <stephen@that.guru>
We provide our own, much smaller implementation of this currently.
However, we want to be able to implement slightly different variants of
this elsewhere and using an existing library helps avoid reinventing the
wheel and lets us use already battle-tested code.
Signed-off-by: Stephen Finucane <stephen@that.guru> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Stephen Finucane [Tue, 30 May 2017 20:33:58 +0000 (21:33 +0100)]
models: Centralize generation of filenames
Move filename generation to a mixin. This allows us to reuse the code
for other items like cover letters. Some unncessary 'strip' calls are
removed as their unnecessary.
This allows us to change the file extension for diffs to 'diff', which
is a little more accurate.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Tue, 23 May 2017 14:08:13 +0000 (15:08 +0100)]
parser: Handle multiple reference headers
It's possible to duplicate message headers multiple times. One common
case is the 'Received' header, but it appears that multiple
'In-Reply-To' and 'References' headers are also a thing.
Handle these cases through the use of the 'Message.get_all' function,
which returns all matching headers, instead of the 'Message.get'
function previously used.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Wed, 24 May 2017 05:54:55 +0000 (06:54 +0100)]
parser: Strip whitespace from references
Some mail, particularly those generated with older versions of
git-send-email or written by hand, include some extra whitespace in the
'References' and 'In-Reply-To' lines. Ensure we always strip this,
preventing mismatches between this and 'Message-ID', which is already
stripped of whitespace, when looking up SeriesReference's.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Tue, 23 May 2017 11:41:58 +0000 (12:41 +0100)]
parser: Extend series heuristics to include other metadata
Not every series will include the reference headers necessary to do
proper series-ification, particularly those generated without the help
of 'git-send-email' or similar.
Make life a little easier for these folks by attempting to match on
other heuristics of the series: submitter, version, number of patches,
project (mailing list) and date. The last of these is particularly
important to prevent duplicate series getting munged together.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Andrew Donnellan [Thu, 25 May 2017 08:05:10 +0000 (18:05 +1000)]
docs/api: change POST to PATCH in REST API parameters example
api/rest.rst gives an example of how to POST parameters to the PatchDetail
view at api/patches/<patch_id>. However, the endpoint in question doesn't
support POST - you need to use PUT or PATCH. Change it to PATCH.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Andrew Donnellan [Thu, 25 May 2017 07:38:04 +0000 (17:38 +1000)]
bundle: Fix use of basic auth for bundle mboxes
Commit 0b4f508a8438 ("views: Allow use of basic auth for bundle mboxes")
added support for using Django REST Framework's BasicAuthentication to
authenticate when accessing the bundle-mbox view.
To check the user's credentials, we call
BasicAuthentication.authenticate(), however, we don't check whether
the returned user is actually the bundle owner. This means that any user
can access any private bundle if they authenticate using basic
authentication.
Additionally, if invalid credentials are provided via a basic
authentication header, BasicAuthentication.authenticate() will throw an
AuthenticationFailed exception. We currently don't catch this, resulting in
an exception page being displayed rather than a 404.
Add a new helper, rest_auth(), that takes a request and returns a user.
Call this in bundle_mbox() and save the result into request.user before we
check whether request.user is actually the bundle owner.
Found by code inspection.
Fixes: 0b4f508a8438 ("views: Allow use of basic auth for bundle mboxes") Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Andrew Donnellan [Thu, 25 May 2017 05:36:18 +0000 (15:36 +1000)]
docker: increase database connection timeout
When starting the Docker environment, if the web container can't see the
database immediately, it waits 5 seconds, tries again, then waits 15
seconds more to account for first-time start-ups where it takes a bit
longer for the database to be initialised.
Some of us, unfortunately, have slow computers with slow mechanical hard
drives which take just a bit longer. Increase the second timeout from 15
seconds to 60 seconds, testing every 5 seconds.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> Acked-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Robin Jarry [Tue, 23 May 2017 13:24:07 +0000 (15:24 +0200)]
pwclient: Force xmlrpc client to return unicode strings
On python 2, the reference implementation of the XML-RPC unmarshaller
decodes strings to unicode with the selected encoding (utf-8 by default)
but it tries to re-encode the unicode strings to ascii bytes before
returning the values. If it fails, it leaves the value as unicode.
Monkey-patch the internal xmlrpclib._stringify() function only on python
2 to force it to preserve unicode strings. This allows to have similar
behaviour in both python 2 and python 3.
Signed-off-by: Robin Jarry <robin.jarry@6wind.com> Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Mon, 15 May 2017 23:13:37 +0000 (00:13 +0100)]
REST: Resolve issues with filters
Turns out filtering patches using a series string wasn't as easy as we
thought. We need to slugify State.name, but unfortunately that isn't
stored in the database. The tests were hiding this fact as State objects
created by 'tests.utils.create_state' don't have spaces in them.
Override custom versions of both django-filter's 'Filter' class and
the Django 'Form' required by this, and update the tests to prevent a
regression.
Signed-off-by: Stephen Finucane <stephen@that.guru> Fixes: 6222574 ("REST: filter patches by state name") Tested-by: Philippe Pepiot <philippe.pepiot@logilab.fr>
Stephen Finucane [Mon, 15 May 2017 23:13:34 +0000 (00:13 +0100)]
REST: Embed nested element bodies instead of URLs
In developing a client for the Patchwork REST API, git-pw, it was noted
that it should be possible to embed some information about nested
resources in order to prevent the need for additional requests [1]. It
was seen that this would be particularly beneficial for list operations,
where each element in the N sized list could theoretically require an
additional request for each of the M nested fields, resulting in N * (M
+ 1) total requests.
Upon experimenting with the 2.0 RC1 API, this optimization was found to
be less of a nice-to-have (and possibly something for the 2.1 release)
and more of a must-have, particularly once one took network latency for
each request into account. During testing with 'git-pw', simple list
operations were found to take an average of 31 requests per operation,
of which only one for was the resource endpoint itself ('GET
/api/series'). As each of these requests took ~2 seconds a piece,
listing was essentially broken.
While local caching could be used to offset some of this demand, this
will result in (a) significantly larger, more complex clients or (b)
instances that strain under the load of dumb clients making multiple
requests per operation. Instead, the server should be smarter about
embedding the data that would actually be required by clients.
Resolve the issue by embedding summarized versions of various nested
fields instead of merely linking to them. Nesting is only a single level
deep, to avoid large/complex database queries and with the expectation
that only these basic fields (resource names, dates, etc.) would be
required. These summary serializers are kept in their own module, to
encourage consistent results throughout the API and to prevent circular
import errors.
This will have the side effect of slightly increasing load on the server
due to the additional serialization required. However, this load is
largely mitigated through the avoidance of deeper nesting as noted
above. In addition, any increase in load seen will be a fraction of the
demand that repeat requests will incur. While it would be possible to
make nesting optional (by way of an 'embed' or 'expand' parameter), it
is expected that this would be an atypical request and would result in
far more complicated serialization code.
Stephen Finucane [Mon, 15 May 2017 23:13:29 +0000 (00:13 +0100)]
REST: Stop including 'tags' in '/patches'
While this is a very helpful field to include, doing so significantly
increases the number of DB queries necessary for listing patches (from
~14 to ~46). Stop including this information until the model itself is
reworked to prevent this issue.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Mon, 15 May 2017 23:13:27 +0000 (00:13 +0100)]
REST: Correct some prefetch, select_related
There were two issues here:
- The 'get_queryset' function, rather than the 'queryset' attribute,
must be overriden when using either the 'prefetch_related' or
'select_related' functions
- A couple of endpoints contained a 'project' attribute, but this wasn't
being prefetched. This didn't cause issues in a single-project
deployment, as used in testing, but will for larger deployments
Resolve both issues.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Thu, 18 May 2017 20:17:48 +0000 (21:17 +0100)]
tox: Turn deprecation notices into warnings
Django does an excellent job of marking what features are going to
change in upcoming releases through extensive use of the 'warnings' module.
Pass the '-Werror' flag to 'manage.py' in tests, ensuring that any
warnings will result in exceptions instead. This should make upgrades a
mostly painless process going forward.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Thu, 18 May 2017 20:17:40 +0000 (21:17 +0100)]
urls: Don't "include" admin URLs
This raises a warning in Django 1.9 and will cause an error in Django
2.0. Per the documentation [1] it is not even necessary so it can simply
be removed.
This raises warnings for Django 1.8 and is mandatory in Django 1.10. It
provides a helpful feature, invalidating a user's session when their
password is changed, and can/should be enabled.
This resolves all issues with Django 1.8.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Thu, 18 May 2017 13:42:19 +0000 (14:42 +0100)]
docs: Don't mention 'default_project' in deployment guide
The deployment guide currently suggests using the 'default_project'
fixture when deploying a production installation of Patchwork. While one
_could_ use this, it's generally unnecessary given that most people care
about their own projects and not Patchwork.
Resolve this by simply removing any references. The references are
retained for the development installation guide, as they're likely
useful here.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Florian Fainelli [Thu, 11 May 2017 23:25:30 +0000 (16:25 -0700)]
lib/apache2: Update location to wsgi python script
Commit 8fe68d96f18e ("wsgi: Move wsgi file to expected location")
relocated lib/apache2/patchwork.wsgi to patchwork/wsgi.py but did not
update the Apache2 example configuration file under
lib/apache2/patchwork.wsgi.conf.
Fixes: 8fe68d96f18e ("wsgi: Move wsgi file to expected location") Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
doc: The REST API can now be used with Django < 1.8
The release notes stated that the REST API was only compatible with
Django 1.8. However, with the merge of commit '646366cc', this is no
longer the case.
We would rather people didn't use these older versions of Django, but
let's not lie.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Use the latest LTS version of Ubuntu. This is mostly a simplification of
the guide, which reverts back to single-node configuration and increases
the emphasis on installing system packages rather than using 'pip'.
There are also a series of corrections, mostly around using the Python 3
variants of packages.
Signed-off-by: Stephen Finucane <stephen@that.guru>
The nginx file was a replacement for '/etc/nginx/nginx.conf' instead of
a "site" file, while the uWSGI file referenced the Python 2 plugin
despite the sample deployment guide, which uses this, being Python
3-based. Correct both issues.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Robin Jarry [Tue, 2 May 2017 14:51:51 +0000 (16:51 +0200)]
pwclient: Fix silent crash on python 2
Replacing sys.stdout and sys.stderr can cause obscure crashes when
trying to write non unicode data. The interpreter is terminated with
SIGINT without any specific error writen on the console.
This happens easily when there is an untrapped exception which should
lead to printing a traceback on stderr.
The only way to prevent UnicodeEncodeErrors is to make sure that one of
the locale-related environment variables (LC_ALL, LANG, LANGUAGE, etc.)
is set. Python will use the correct encoding accordingly.
Add a note about this on `pwclient --help`. Also, display a help message
when an encoding error occurs.
Fixes: 046419a3bf8f ("pwclient: Fix encoding problems") Signed-off-by: Robin Jarry <robin.jarry@6wind.com> Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sat, 29 Apr 2017 00:13:06 +0000 (01:13 +0100)]
docs: Emphasise the cron job
Having talked to a few folks deploying Patchwork, it appears not
everyone is aware of/enabling the Patchwork cron job. Emphasise this
feature by moving it to its own section. This section is marked as
optional, given that it's not truly required but is helpful.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Wed, 26 Apr 2017 23:58:03 +0000 (00:58 +0100)]
REST: Fix versioning
One of the few remaining warts in the API is versioning. There is some
basic versioning there, but it doesn't work properly as-is. Fix this by
correcting the index endpoint ('/') to use Django REST Framework's
'reverse' function [1], which handles versioning for us [2] and
switching from 'NamespaceVersioning' to 'URLPathVersioning', the latter
of which does the same thing but in a different way.
In commit febad055, we attempted to start testing older versions of
Django REST Framework for use with older versions of Django. This work
was later reverted in commit 6f2ddab6 due to issues with 'django-filter'
versions.
There are additional reasons for validating older versions of DRF, such
as supporting the versions provide as part of distributions. It turns
out that those issues aren't such a big deal and can be handled with a
shim. All in all, this means we can and should support these older
versions of Django REST Framework, and that is what we'll do.
Note that the minimum version supported in requirements.txt is not
changed as it's good practice to use the latest available version.
However, that doesn't make it any less supported.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Older versions of Django REST Framework's 'RelatedField' did not check
for the presence 'get_queryset' before checking for a 'queryset'
attribute. This was fixed in 3.4.0 [1] and backported to the 3.3 series
in 3.3.3 [2]. However, the fix was not backported to the last release of
3.2 series [3], which is the last release to support Django 1.6. As
such, we must set a 'queryset' attribute, even if it's set to a garbage
value.
Recent versions of Django REST Framework (DRF) have deprecated the
'DjangoFilterBackend' filter found in-tree, in favour of an equivalent
implementation found in django-filter. However, we need to support older
versions of DRF for users who want to use system packages.
Seeing as the two implementations are, for all intents and purposes,
essentially the same thing, provide a shim that will allow us to use
both together.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Partial-bug: #94
Stephen Finucane [Wed, 26 Apr 2017 21:45:33 +0000 (22:45 +0100)]
docs: Split API docs into their own section
Third time lucky. There are two changes:
- Add a new 'clients' section to the usage doc, allowing us to remove a
lot of the API nitty gritty stuff from the users guide. This makes
more sense as users don't really care what API method they're using -
only what application).
- Change the API docs from the developers guide into a quick start
section, allowing us to greatly expand the REST API docs to include
information on pagination, authentication, etc.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Wed, 22 Mar 2017 23:17:08 +0000 (23:17 +0000)]
docs: Add skeleton for Sphinx docs
This is mostly the output of 'sphinx-quickstart' with all non-HTML build
cruft removed and Sphinx minimum version set to 1.5. A tox target is
included and the output of the docs build ignored.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Mon, 27 Mar 2017 05:42:52 +0000 (16:42 +1100)]
Display count of patches
Minor UI tweak - show the number of patches left in the current
view.
Suggested-by: Stewart Smith <stewart@linux.vnet.ibm.com> Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
projects / thirdparty / patchwork.git / log
