]> git.ipfire.org Git - thirdparty/patchwork.git/log
thirdparty/patchwork.git
7 years ago4x performance improvement for viewing patch with many comments
Stewart Smith [Fri, 10 Aug 2018 08:00:57 +0000 (18:00 +1000)] 
4x performance improvement for viewing patch with many comments

Using the example of id:20180720035941.6844-1-khandual@linux.vnet.ibm.com
with my test dataset of a chunk of a variety of mailing lists, has
this cover letter have 67 comments from a variety of people. Thus,
it's on the larger side of things.

Originally, displaying the /patch/550/ for this (redirected to /cover)
would take 81 SQL queries in ~60ms on my laptop.

After this optimisation, it's down to 14 queries in 14ms.

When the cache is cold, it's down to 32ms from 83ms.

The effect of this patch is to execute a join in the database to
get the submitter information for each comment at the same time as
getting all the comments rather than doing a one-by-one lookup after
the fact.

Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agoImprove patch listing performance (~3x)
Stewart Smith [Fri, 10 Aug 2018 08:00:56 +0000 (18:00 +1000)] 
Improve patch listing performance (~3x)

There's two main bits that are really expensive when composing the list
of patches for a project: the query getting the list, and the query
finding the series for each patch.

If we look at the query getting the list, it gets a lot of unnecessary
fields such as 'headers' and 'content', even though we tell Django not
to. It turns out that Django seems to ignore the Submission relationship
and I have no idea how to force it to ignore that thing (defer doesn't
work) but if we go only, then it works okay.

From my import of ~8000 messages for a few projects, my laptop query
time (MySQL, as setup by whatever the docker-compose things do) goes
from:

  http://localhost:8000/project/linux-kernel/list/
  FROM:
    342ms SQL queries cold cache, 268ms warm cache
  TO:
    118ms SQL queries cold cache, 88ms warm cache

Which is... non trivial to say the least.

The big jump is the patches.only change, and the removal of ordering
on the patchseries takes a further 10ms off. For some strange reason, it
seems rather hard to tell Django that you don't care what order the
results come back in for that query (if we do, then the db server has to
do a sort rather than just return each row)

Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
[stephenfin: Add missing migration that had been squashed into a later
 migration]
Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocker: Use heredocs where possible
Stephen Finucane [Wed, 29 Aug 2018 10:10:20 +0000 (11:10 +0100)] 
docker: Use heredocs where possible

This was suggested in a recent review [1]. Make it happen.

[1] http://patchwork.ozlabs.org/patch/933979/#1941584

Signed-off-by: Stephen Finucane <stephen@that.guru>
Suggested-by: Petr Vorel <petr.vorel@gmail.com>
Reviewed-by: Petr Vorel <pvorel@suse.cz>
7 years agodocs: Fix documentation of REST_RESULTS_PER_PAGE setting
Andrew Donnellan [Sun, 26 Aug 2018 10:56:05 +0000 (20:56 +1000)] 
docs: Fix documentation of REST_RESULTS_PER_PAGE setting

In 8fe11180a1a5 ("REST: Add new setting for maximum API page size") I
accidentally deleted the versionadded information for
REST_RESULTS_PER_PAGE. Restore it.

Fixes: 8fe11180a1a5 ("REST: Add new setting for maximum API page size")
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agopwclient: Fix pwclient am output formatting
Petr Vorel [Thu, 26 Jul 2018 07:11:14 +0000 (09:11 +0200)] 
pwclient: Fix pwclient am output formatting

repr() print unicode prefix for string:
$ pwclient git-am N
Applying patch #N using u'git am'

Remove it:
$ pwclient git-am N
Applying patch #918868 using "git am"

git mixes single and double quotes, use double quotes which are more
frequently used.

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Add new setting for maximum API page size
Andrew Donnellan [Tue, 24 Jul 2018 05:10:51 +0000 (15:10 +1000)] 
REST: Add new setting for maximum API page size

In 41790caf59ad ("REST: Limit max page size") we limited the maximum page
size to the default page size in the settings.

This turns out to be rather restrictive, as we usually want to keep the
default page size low, but an administrator may want to allow API clients
to fetch more than that per request.

Add a new setting, MAX_REST_RESULTS_PER_PAGE, to set the maximum page size.

Closes: #202 ("Separate max API page size and default API page size into different settings")
Suggested-by: Stewart Smith <stewart@linux.ibm.com>
Suggested-by: Joel Stanley <joel@jms.id.au>
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
[dja: set to 250 as per mailing list discussion]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agodocker: Don't require rebuilding if unnecessary
Stephen Finucane [Sun, 24 Jun 2018 19:55:57 +0000 (20:55 +0100)] 
docker: Don't require rebuilding if unnecessary

Now that we're pinning versions, we're going to see more frequent
dependency version changes. Requiring a rebuild after every one of these
is tiresome so don't force it and instead display a helpful message
merely suggesting that a rebuild may be necessary.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Cc: Daniel Axtens <dja@axtens.net>
Acked-by: Daniel Axtens <dja@axtens.net>
[dja: we're not currently pinning versions because I dropped that
 patch as it broke postgres. But this can stay - sfin has been
 asking for it for ages.]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agotox: Rework warning infrastructure
Stephen Finucane [Sun, 24 Jun 2018 19:55:56 +0000 (20:55 +0100)] 
tox: Rework warning infrastructure

Python 3.5's xmlrpc spews lots of ResourceWarnings that go
away in 3.6, so silence them.

We also see some warnings from inside the import machinery,
which we also silence.

Signed-off-by: Stephen Finucane <stephen@that.guru>
[dja: make slightly more restrictive, reword commit message]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoResolve Python 3.6 warnings
Stephen Finucane [Sun, 24 Jun 2018 19:55:55 +0000 (20:55 +0100)] 
Resolve Python 3.6 warnings

DeprecationWarning: invalid escape sequence \d

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reviewed-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoAdd support for Django 2.0
Stephen Finucane [Sun, 24 Jun 2018 19:55:53 +0000 (20:55 +0100)] 
Add support for Django 2.0

Nothing too complicated here except for the addition of a new compat
wrapper, which will be removed again shortly. According to the Django
release notes, Django should function with Python 3.4. However, it was
not possible to get this functioning due to the below error:

  Traceback (most recent call last):
    File ".../patchwork/manage.py", line 11, in <module>
    ...
    File ".../django/db/models/fields/related.py", line 313, in contribute_to_class
      'app_label': cls._meta.app_label.lower(),
  TypeError: unsupported operand type(s) for %: 'bytes' and 'dict'

This does not appear to be an issue with Patchwork but the exact root
cause has not been identified. As a result, only Python 3.5 and 3.6 are
marked as supported for this Django version.

As this is the first Python 3-only dependency we have, we need to start
making use of the 'python_version' environment marker.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoPrepare for Django 2.0
Stephen Finucane [Sun, 24 Jun 2018 19:55:52 +0000 (20:55 +0100)] 
Prepare for Django 2.0

Two changes necessary.

- Remove 'SessionAuthenticationMiddleware' from settings

  https://docs.djangoproject.com/en/2.0/releases/2.0/#miscellaneous

- Replace 'render_to_response' with 'render'

  https://docs.djangoproject.com/en/2.0/releases/2.0/#miscellaneous

Both of these are supported by Django 1.11 so there's no need for compat
wrappers.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoRemove compat wrapper for Django < 1.11
Stephen Finucane [Sun, 24 Jun 2018 19:55:51 +0000 (20:55 +0100)] 
Remove compat wrapper for Django < 1.11

Deleting code is fun. We no longer need to carry these so don't.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoRemove support for Django 1.8, 1.9, 1.10
Stephen Finucane [Sun, 24 Jun 2018 19:55:50 +0000 (20:55 +0100)] 
Remove support for Django 1.8, 1.9, 1.10

These are now all EOL and Debian Testing supports Django 1.11 (LTS). We
can and should drop them.

This change does not remove the many compat wrappers. These will be
removed separately (there are a lot of them).

This leaves 1.11 as the only supported version. This will be remedied
shortly with the inclusion of Django 2.0 support.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoAdd support for django-filter 1.1
Stephen Finucane [Sun, 24 Jun 2018 19:55:49 +0000 (20:55 +0100)] 
Add support for django-filter 1.1

There's one warning to handle here.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoAdd support for Django REST Framework 3.7, 3.8
Stephen Finucane [Sun, 24 Jun 2018 19:55:48 +0000 (20:55 +0100)] 
Add support for Django REST Framework 3.7, 3.8

No breaking changes that concern us here.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reviewed-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Check.user is not read-only
Stephen Finucane [Sun, 24 Jun 2018 19:55:47 +0000 (20:55 +0100)] 
REST: Check.user is not read-only

We only support 'Check' creation - not check updating. As a result,
there's no real reason that the 'Check.user' field should be read-only
and this is causing an issue with Django REST Framework 3.7. Simply
remove the attribute and extend the tests to validate things are working
as expected.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reviewed-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agotravis: add mariadb-10.3 as test
Daniel Black [Mon, 6 Aug 2018 06:56:39 +0000 (16:56 +1000)] 
travis: add mariadb-10.3 as test

Signed-off-by: Daniel Black <daniel@linux.ibm.com>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agotravis: test against postgresql 10 and 11
Daniel Black [Mon, 6 Aug 2018 06:56:37 +0000 (16:56 +1000)] 
travis: test against postgresql 10 and 11

postgresql 10 is the most recent major version, and it would be
good to keep an eye on postgres 11 compatibility also, so test
against both of those.

Specify PGPORT as a way to differentiate the new installations
from the postgres 9.6 server running on the default port (5432).
Use a local unix socket to work around the fact that by default
postgres only allows passwordless auth on local sockets - which
is tweaked for postgres 9.6 but not for later versions that aren't
in the usual Travis image.

Also print the current db version and user for validation.

Signed-off-by: Daniel Black <daniel@linux.ibm.com>
[cleanup, squash, commit message]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoparser: fix parsing of patches with headings
Jiri Benc [Thu, 28 Jun 2018 19:42:11 +0000 (15:42 -0400)] 
parser: fix parsing of patches with headings

Some people tend to use lines full of '=' as a fancy way to format headings
in their commit messages in a rst-like style. However, the current parser
treats such lines as a beginning of a diff.

The only currently used tool that produces diffs with '=' lines is quilt in
the default configuration. However, even with quilt, the diff looks this
way:

    Index: dir/file
    ===================================================================
    --- dir.orig/file
    +++ dir/file
    @@ ...etc...

It's enough to match on the "Index:" line. The state of the state machine is
kept at 1 when it encounters the '=' line, thus it's safe to remove the
match on '=' completely.

[This prevents us from properly parsing metadata out of the changelog. -dcz ]

Signed-off-by: Jiri Benc <jbenc@redhat.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agotests: Add tests for multipart emails
Stephen Finucane [Wed, 20 Jun 2018 14:25:33 +0000 (15:25 +0100)] 
tests: Add tests for multipart emails

Ensure HTML is dropped as expected.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoparsemail: ignore html part of multi-part comments
Yuri Volchkov [Wed, 20 Jun 2018 12:21:42 +0000 (14:21 +0200)] 
parsemail: ignore html part of multi-part comments

Currently an html-protection present only for patch-emails. If a
multi-part comment-email arrives, it messes up patchwork. In my case,
the symptom was a non intended 'Signed-off-by' in the downloaded
patches, with html-like junk.

This patch makes parsemail skip all parts of comment which are not
text/plain.

Of course, this will drop html-only emails completely. But they can
not be parsed anyways.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Update reno for stable/2.1
Stephen Finucane [Tue, 19 Jun 2018 21:20:05 +0000 (22:20 +0100)] 
docs: Update reno for stable/2.1

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoPost-release version bump
Stephen Finucane [Tue, 19 Jun 2018 21:21:16 +0000 (22:21 +0100)] 
Post-release version bump

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoRelease 2.1.0 v2.1.0
Stephen Finucane [Tue, 19 Jun 2018 21:11:48 +0000 (22:11 +0100)] 
Release 2.1.0

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agopwclient/get: Add suffix to created patch
Petr Vorel [Fri, 15 Jun 2018 13:52:41 +0000 (15:52 +0200)] 
pwclient/get: Add suffix to created patch

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agoREST: Add 'web_url' link to API responses
Stephen Finucane [Thu, 7 Jun 2018 14:14:14 +0000 (15:14 +0100)] 
REST: Add 'web_url' link to API responses

This provides an easy way for clients to navigate to the web view. The
URL is added to four resources: bundles, comments, cover letters and
series. We could also extend this to projects and users in the future,
but the latter would require renaming an existing property while the
latter would require a public "user" page which does not currently
exists.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reviewed-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Add missing tests for '/series'
Stephen Finucane [Tue, 12 Jun 2018 15:48:03 +0000 (16:48 +0100)] 
REST: Add missing tests for '/series'

I'm not sure how these were missed but they're definitely needed as we
continue adding functionality here.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reviewed-by: Daniel Axtens <dja@axtens.net>
7 years agoDon't discard values from 'archive' filter
Stephen Finucane [Wed, 13 Jun 2018 21:53:22 +0000 (22:53 +0100)] 
Don't discard values from 'archive' filter

The pagination functionality used in the 'patchwork.view.generic_list'
generates the filter querystring from scratch. To do this, it calls the
'patchwork.filters.Filters.params' function, which in turn calls the
'patchwork.filters.Filter.key' function for each filter. If any of these
'key' functions return None, the relevant filter is not included in the
querystring. This ensures we don't end up with a load of filters like
the below:

    ?submitter=&state=&series=&q=&delegate=&archive=both

which would be functionally equivalent to:

    ?archive=both

There is one exception to this rule, however: ArchiveFilter. This is a
little unusual in that it is active by default, excluding patches that
are "archived" from the list. As a result, the 'key' function should
return None for this active state, not for the disabled state. This has
been the case up until commit d848f046 which falsely equated 'is False'
with 'is None'. This small typo resulted in the filter being ignored
when generating pagination links and essentially broke pagination for
some use cases.

Fix this up. We could probably simplify this thing greatly by not
recalculating filters for pagination at least or, better yet, by using
django-filter here too. That is a change for another day though.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reported-by: John McNamara <john.mcnamara@intel.com>
Reported-by: Eli Schwartz <eschwartz93@gmail.com>
Fixes: d848f046 ("trivial: Don't shadow built-ins")
Closes: #184
7 years agoviews: Raise 404 if downloading non-existent dependencies
Stephen Finucane [Thu, 7 Jun 2018 13:04:47 +0000 (14:04 +0100)] 
views: Raise 404 if downloading non-existent dependencies

If a patch was processed by Patchwork before series support was added,
it will not have a series associated with it. As a result, it is not
possible to extract the dependencies for that patch from the series and
a 404 should be raised. This was not previously handled correctly.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reviewed-by: Daniel Axtens <dja@axtens.net>
Reported-by: John McNamara <john.mcnamara@intel.com>
Fixes: e2dfd490 ("views: Add 'series' parameter to '/mbox' endpoint")
Closes: #189
7 years agoparsemail.sh: don't set the python version
Ali Alnubani [Wed, 6 Jun 2018 13:11:54 +0000 (15:11 +0200)] 
parsemail.sh: don't set the python version

This is to fix using the wrong python version when inside a virtualenv.

Signed-off-by: Ali Alnubani <alialnu@mellanox.com>
Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Update deployment installation guide for v2.1
Stephen Finucane [Sat, 2 Jun 2018 20:55:29 +0000 (21:55 +0100)] 
docs: Update deployment installation guide for v2.1

There are no actual changes needed from the Patchwork side so this is
mostly a cleanup.

- Use Ubuntu 18.04 (including package names)
- Resolve some minor issues with commands
- Remove use of "trust" authentication for PostgreSQL
- Minor style changes

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agosql: Update 'grant-all.mysql' script with missing tables
Stephen Finucane [Wed, 30 May 2018 10:57:59 +0000 (11:57 +0100)] 
sql: Update 'grant-all.mysql' script with missing tables

These were all introduced in 2.0 and while the postgreSQL script was
fixed in commit 234bc7c3, the MySQL one was not. This suggests either
(a) no one is using this or (b) people are carrying local changes but
for now we just resolve the issues.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agosql: Sort 'grant-all' scripts alphabetically
Stephen Finucane [Wed, 30 May 2018 10:56:39 +0000 (11:56 +0100)] 
sql: Sort 'grant-all' scripts alphabetically

This makes it easier to compare to the output of 'show tables'.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoPatchwork v2.1.0-rc2 v2.1.0-rc2
Stephen Finucane [Wed, 16 May 2018 13:42:41 +0000 (14:42 +0100)] 
Patchwork v2.1.0-rc2

Signed-off-by: Stephen Finucane <stephen@that.guru>
Acked-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Update copyright and remove dead code
Stephen Finucane [Thu, 17 May 2018 09:34:04 +0000 (10:34 +0100)] 
docs: Update copyright and remove dead code

We're firmly in 2018 now and we're also using sphinx_rtd_theme 0.3.0.
This means we can bump the copyright and remove what is now dead code.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Acked-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Update release, contributing guides
Stephen Finucane [Wed, 16 May 2018 13:29:34 +0000 (14:29 +0100)] 
docs: Update release, contributing guides

Document the requirement to send an email to the list upon a release and
to always send patches via email.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Acked-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Resolve typos in release notes
Stephen Finucane [Thu, 17 May 2018 09:38:26 +0000 (10:38 +0100)] 
docs: Resolve typos in release notes

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoREST: Disable control for filtering patches by series in web view
Daniel Axtens [Fri, 11 May 2018 17:00:44 +0000 (03:00 +1000)] 
REST: Disable control for filtering patches by series in web view

As with the events view, creating and rendering the control for
filtering patches by series creates a massive slowdown. It's a little
sad not to be able to do this in the web UI as filtering patches
by series does make sense, but hopefully people figure out you can
still do it, just not from the web view.

Signed-off-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Add 'local' argument to 'contents'
Stephen Finucane [Wed, 9 May 2018 17:02:00 +0000 (18:02 +0100)] 
docs: Add 'local' argument to 'contents'

This means the page title won't be included in the table of contents,
which makes sense. I'm not really sure why this isn't the default, to be
honest.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Add additional information about API versions
Stephen Finucane [Wed, 9 May 2018 17:01:59 +0000 (18:01 +0100)] 
docs: Add additional information about API versions

As we're soon going to be supporting a v1.1 API, we should document what
versions are available and whether they're supported still (hint: we
support both v1.0 and v1.1 at present).

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoRevert "api: Only provide JSON version of events list"
Stephen Finucane [Thu, 10 May 2018 14:45:06 +0000 (15:45 +0100)] 
Revert "api: Only provide JSON version of events list"

This reverts commit 90d9ee14e73e8ec9248e89c788d64867c4a4bb74.

The root cause of this performance issue was not the use of the
JSONRenderer but rather the population of filter forms. The latter is
now disabled, meaning we can start using the original renderer.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Cc: Daniel Axtens <dja@axtens.net>
Acked-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Resolve performance issues with '/events' web view
Stephen Finucane [Thu, 10 May 2018 14:45:05 +0000 (15:45 +0100)] 
REST: Resolve performance issues with '/events' web view

The dropdown select-based filters in the web view of the REST API have
stung us a few times. In this case, populating these filters for the
'/events' endpoint results in a huge query that hammers the database and
results in seriously laggy responses.

The root cause of this performance issues was erroneously identified as
an issue with the JSON renderer so that particular patch can now be
reverted. This will be done separately.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Cc: Daniel Axtens <dja@axtens.net>
Tested-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Use DRF-specific filterset
Stephen Finucane [Wed, 11 Apr 2018 16:13:38 +0000 (17:13 +0100)] 
REST: Use DRF-specific filterset

This variant of 'FilterSet' will convert a 'django.forms.ValidationError',
which wouldn't be handled by DRF, to a
'rest_framework.exceptions.ValidationError', which would be.

Signed-off-by: Stephen Finucane <stephen@that.guru>
[dja: commit message]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Use ModelMultipleChoiceField for other fields
Stephen Finucane [Wed, 11 Apr 2018 16:13:37 +0000 (17:13 +0100)] 
REST: Use ModelMultipleChoiceField for other fields

There's benefit to being able to do stuff like select multiple patches.
Let's do that.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Rename Filter -> FilterSet
Stephen Finucane [Wed, 11 Apr 2018 16:13:36 +0000 (17:13 +0100)] 
REST: Rename Filter -> FilterSet

FilterSets are to Forms as Filters are to Fields: the former is
made up of the latter. We have a FilterSet for each resource that we
wish to support filtering on (i.e. all of them).

Rename our "Filters" to the more appropriate FilterSets.
The old name was confusing and will conflict with some forthcoming changes.

Signed-off-by: Stephen Finucane <stephen@that.guru>
[dja: commit message]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Remove ProjectFilterMixin
Stephen Finucane [Wed, 11 Apr 2018 16:13:35 +0000 (17:13 +0100)] 
REST: Remove ProjectFilterMixin

Whatever benefits this was giving us in the past are no more and it
simply confuses matters now.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reviewed-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Base filters on Django's ModelMultipleChoiceField
Stephen Finucane [Wed, 11 Apr 2018 16:13:34 +0000 (17:13 +0100)] 
REST: Base filters on Django's ModelMultipleChoiceField

Introduce a modified version of Django's ModelMultipleChoiceField
that allows us to query on multiple fields.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Fixes: #156
[dja: commit message, expand docs]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Simplify ModelMultiChoiceField
Stephen Finucane [Wed, 11 Apr 2018 16:13:33 +0000 (17:13 +0100)] 
REST: Simplify ModelMultiChoiceField

We're actually going to remove this shortly but the new technique works
for both the current approach and the approach we adopt in future patches.

Signed-off-by: Stephen Finucane <stephen@that.guru>
[dja: commit message, drop dead code]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Use UserFilter for patch delegates
Stephen Finucane [Wed, 11 Apr 2018 16:13:32 +0000 (17:13 +0100)] 
REST: Use UserFilter for patch delegates

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Bump API version in docs to 1.1
Stephen Finucane [Tue, 8 May 2018 08:18:22 +0000 (09:18 +0100)] 
docs: Bump API version in docs to 1.1

Signed-off-by: Stephen Finucane <stephen@that.guru>
Fixes: 81e6f8e4 ("docs: Prepare for 2.1.0-rc1")
Cc: Daniel Axtens <dja@axtens.net>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoskip original Content-Transfer-Encoding for mbox
Yuri Volchkov [Mon, 7 May 2018 15:57:55 +0000 (01:57 +1000)] 
skip original Content-Transfer-Encoding for mbox

In the commit 01b9cbb9 all original mail headers are copied into the
resulted mbox file. This means that some headers are going to be
present twice in the generated mbox. That is fine unless the original
email arrived in base64 encoding.

Apparently git relies on the latest Content-Transfer-Encoding key. And
since downloaded patch's actual encoding is '7bit', git fails to apply
it with the message 'Patch is empty'.

Since patchwork adds a proper 'Content-Transfer-Encoding' anyways,
let's skip this field while copying headers from the original mail

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
Acked-by: Veronika Kabatova <vkabatov@redhat.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agoPatchwork v2.1.0-rc1 v2.1.0-rc1
Daniel Axtens [Thu, 5 Apr 2018 16:31:16 +0000 (02:31 +1000)] 
Patchwork v2.1.0-rc1

Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Prepare for 2.1.0-rc1
Daniel Axtens [Thu, 5 Apr 2018 16:26:53 +0000 (02:26 +1000)] 
docs: Prepare for 2.1.0-rc1

Patchwork v2.1.0 will be Eolienne.

Move the "unreleased" notes to eolienne. We'll add a new 'unreleased'
right after tagging the final 2.1.0

Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoExplicitly distinguish between comments on patch and cover
Veronika Kabatova [Thu, 3 May 2018 10:55:16 +0000 (12:55 +0200)] 
Explicitly distinguish between comments on patch and cover

reverse() gets confused when the same view name and kwargs are passed to
it, ignoring what endpoint the request originated from. Fix this by
using different view names for cover letter and patch comments views.

Reported-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Veronika Kabatova <vkabatov@redhat.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agoparsemail: Clarify exit codes
Daniel Axtens [Mon, 30 Apr 2018 15:44:59 +0000 (01:44 +1000)] 
parsemail: Clarify exit codes

jk reports that the patchwork error codes are really unhelpful for
correct integration with an MDA. In particular they make sorting out
failures into a separate queue very difficult. Make this better and
clearer: only return 1 on a genuinely unexpected case that requires
adminstrator intervention.

Update the comment for parse_mail regarding return values and exceptions
to line up with how the function actually works and how we use the
results.

Update the tests. None of the existing tests should exit 1; they're
all 'expected' failures: unknown project etc. Also we removed the
exit(0) from the success path, so stop expecting that exception to
be raised.

Add a test for duplicates. That should also succeed without raising
an exception: dups are part of life.

Update parsearchive to deal with the fact that we can no longer
differentiate duplicates.

Reported-by: Jeremy Kerr <jk@ozlabs.org>
Fixes: #171
Signed-off-by: Daniel Axtens <dja@axtens.net>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agodocker-compose: Re-add 'PGPASSWORD' to correct section
Stephen Finucane [Wed, 2 May 2018 15:06:25 +0000 (16:06 +0100)] 
docker-compose: Re-add 'PGPASSWORD' to correct section

This was moved in commit '1590c21d' but should not have been. Restore it
to its rightful place.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Fixes: 1590c21d ("docker-compose: Remove 'links' section")
Tested-by: Daniel Axtens <dja@axtens.net>
7 years agoREST: Show 'comments' in '/patches', '/comments'
Stephen Finucane [Fri, 27 Apr 2018 15:33:21 +0000 (16:33 +0100)] 
REST: Show 'comments' in '/patches', '/comments'

This link is only shown for individual resources at the moment. Modify
this behavior.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Acked-by: Veronika Kabatova <vkabatov@redhat.com>
7 years agoREST: Add comments to patch and cover endpoints
Veronika Kabatova [Wed, 25 Apr 2018 17:33:27 +0000 (19:33 +0200)] 
REST: Add comments to patch and cover endpoints

Signed-off-by: Veronika Kabatova <vkabatov@redhat.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agoUpdate Docker IP to whitelist for django-debug-toolbar
Stephen Finucane [Thu, 26 Apr 2018 23:19:32 +0000 (00:19 +0100)] 
Update Docker IP to whitelist for django-debug-toolbar

This has changed with the recent move to the 3.0 syntax.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocker: Add 'depends_on'
Stephen Finucane [Thu, 26 Apr 2018 13:47:40 +0000 (14:47 +0100)] 
docker: Add 'depends_on'

We need this for the PostgreSQL variant too.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Fixes: 1590c21d ("docker-compose: Remove 'links' section")
7 years agodocker: Add 'depends_on'
Stephen Finucane [Thu, 26 Apr 2018 12:45:50 +0000 (13:45 +0100)] 
docker: Add 'depends_on'

This is required now that we're no longer using links. Without it,
'docker-compose up' works but 'docker-compose run web' does not.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Fixes: 1590c21d ("docker-compose: Remove 'links' section")
7 years agodocker: Bump to bionic
Stephen Finucane [Tue, 17 Apr 2018 10:17:55 +0000 (11:17 +0100)] 
docker: Bump to bionic

As noted in commit 94dd1d411, we can and should start using Bionic once
it's out. This is now the case.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocker-compose: Remove 'links' section
Stephen Finucane [Tue, 17 Apr 2018 09:32:13 +0000 (10:32 +0100)] 
docker-compose: Remove 'links' section

This is no longer needed as the service name is also the host names in
v2 syntax [1].

[1] https://docs.docker.com/compose/networking/

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocker-compose: Switch to 3.0 syntax
Stephen Finucane [Tue, 17 Apr 2018 08:58:00 +0000 (09:58 +0100)] 
docker-compose: Switch to 3.0 syntax

Ubuntu 18.04 (Bionic Beaver) providers 'docker-compose' 1.17.1 [1] at
release which supports the 3.0 syntax [2]. Using this allows some users
(me) to resolve a long standing issue caused by a UID that's not 1000.

[1] https://packages.ubuntu.com/bionic/docker-compose
[2] https://docs.docker.com/compose/compose-file/

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocker: Remove bash aliases
Stephen Finucane [Tue, 17 Apr 2018 09:20:28 +0000 (10:20 +0100)] 
docker: Remove bash aliases

These seem to be a hangover from the Vagrant days. I don't personally
use them anymore and I'm not sure anyone else does. I think they can go.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Cc: Daniel Axtens <dja@axtens.net>
7 years agotests: Remove Selenium tests
Stephen Finucane [Tue, 17 Apr 2018 08:50:44 +0000 (09:50 +0100)] 
tests: Remove Selenium tests

These were added quite some time ago in order to allow some level of UI
testing. However, I've personally never used them, they're not used by
the CI, and no one has shown any desire in extending them in their time
here. It is time to bid these tests adieu.

Removing these allows us to remove a whole load of wiring that existed
just to enable these. Some of this, like the '--quick-tox' option for
the Dockerfile, is retained so we don't need to use different commands
for various versions of Patchwork, but the majority is just stripped
out.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Cc: Daniel Axtens <dja@axtens.net>
7 years agodocker: Replace tabs with spaces
Stephen Finucane [Tue, 17 Apr 2018 08:45:02 +0000 (09:45 +0100)] 
docker: Replace tabs with spaces

Maintain your chill, people.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoUpdate django-debug-toolbar from 1.8 to 1.9.1
pyup-bot [Wed, 25 Apr 2018 21:39:42 +0000 (22:39 +0100)] 
Update django-debug-toolbar from 1.8 to 1.9.1

7 years agoUpdate sqlparse from 0.2.3 to 0.2.4
pyup-bot [Wed, 27 Sep 2017 07:55:45 +0000 (08:55 +0100)] 
Update sqlparse from 0.2.3 to 0.2.4

7 years agoUpdate sphinx_rtd_theme from 0.2.4 to 0.3.0
pyup-bot [Fri, 6 Apr 2018 15:25:09 +0000 (16:25 +0100)] 
Update sphinx_rtd_theme from 0.2.4 to 0.3.0

7 years agoResolve pep8 issues
Stephen Finucane [Wed, 25 Apr 2018 20:48:29 +0000 (21:48 +0100)] 
Resolve pep8 issues

Signed-off-by: Stephen Finucane <stephen@that.guru>
Fixes: 26f22a87 ("models: Add a backreference for a user's bundles")
7 years agomodels: Add a backreference for a user's bundles
Stephen Finucane [Thu, 12 Apr 2018 12:01:31 +0000 (13:01 +0100)] 
models: Add a backreference for a user's bundles

This is more intuitive.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Add release note for changes to 'headers' field
Stephen Finucane [Wed, 25 Apr 2018 20:17:53 +0000 (21:17 +0100)] 
docs: Add release note for changes to 'headers' field

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoapi: Show all headers with the same key
Veronika Kabatova [Fri, 6 Apr 2018 11:24:49 +0000 (13:24 +0200)] 
api: Show all headers with the same key

While the code on our side returns all (key, value) pairs for email
headers, Django's REST framework probably uses dictionaries behind the
scenes. This means that having multiple headers with same key (eg
'Received', which is totally valid and common situation), only one of
these headers is visible in the REST API.

Let's hack around this by returning a list of values in case the key is
present multiple times.

Signed-off-by: Veronika Kabatova <vkabatov@redhat.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Random fixes
Stephen Finucane [Sun, 15 Apr 2018 22:18:08 +0000 (23:18 +0100)] 
docs: Random fixes

Remove an unnecessary 'toctree' from the index page and fix some
definition lists.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agotests: Replace incorrect tests
Stephen Finucane [Tue, 17 Apr 2018 08:37:44 +0000 (09:37 +0100)] 
tests: Replace incorrect tests

In commit 683792d1, the 'test_api.py' was split into multiple
'api/test_xyz.py' files. As part of this change, the tests for cover
letter were mistakenly included in place of tests for checks. Correct
this oversight.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Reported-by: Veronika Kabatova <vkabatov@redhat.com>
Acked-by: Veronika Kabatova <vkabatov@redhat.com>
Fixes: 683792d1 ("tests: Split 'test_rest_api'")
7 years agogitignore: Ignore '.venv'
Stephen Finucane [Wed, 11 Apr 2018 16:36:19 +0000 (17:36 +0100)] 
gitignore: Ignore '.venv'

I use this to validate stuff quite frequently. Ignore it so it's not
polluting my git-status.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agotox: Add 'docs' to default environments
Stephen Finucane [Wed, 11 Apr 2018 16:36:18 +0000 (17:36 +0100)] 
tox: Add 'docs' to default environments

I'd simply run 'tox' (via docker) to validate some previous patches.
Sadly that didn't catch a release note issue. Make sure this doesn't
happen again by always running 'docs'.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Fix a release note issue
Stephen Finucane [Wed, 11 Apr 2018 16:36:17 +0000 (17:36 +0100)] 
docs: Fix a release note issue

You can't have a space before the closing double backticks of a literal
block. This was generating the following (unhelpful) error message:

  patchwork/docs/releases/unreleased.rst:82:Inline literal start-string
  without end-string.

reno should have better error messages but that's a problem for another
day.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Read version from 'patchwork.VERSION'
Stephen Finucane [Wed, 11 Apr 2018 16:36:16 +0000 (17:36 +0100)] 
docs: Read version from 'patchwork.VERSION'

Because this isn't an installable package we need to do some path
hackery. Not the end of the world though.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoFix stuff around mbox header changes
Veronika Kabatova [Tue, 10 Apr 2018 14:30:44 +0000 (16:30 +0200)] 
Fix stuff around mbox header changes

Bundle tests got broken after the subject in mbox was changed from the
parsed version to the original one because the tests checked for the
presence of patch's name in the response. Fixing this turned out to be
a bit tricky since the tests check the mbox attachment and HTML
responses separately, so we need a string that would be present in both
(the intuitive idea of checking X-Patchwork-Id won't work well).

Add the patch's name to the content of the test patch so we can continue
testing things the same way, checking for the presence of patch's name.

Also add a releasenote notifying about the inclusion of the original
headers.

Reverts: b2a25342 ("Use parsed subject for mboxes")
Fixes: 01b9cbb9 ("Include all email headers in mboxes")
Signed-off-by: Veronika Kabatova <vkabatov@redhat.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agogitignore: Ignore sql files
Stephen Finucane [Mon, 9 Apr 2018 21:04:36 +0000 (22:04 +0100)] 
gitignore: Ignore sql files

These are useful as backups.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Add note on restoring the docker database
Stephen Finucane [Mon, 9 Apr 2018 14:59:36 +0000 (15:59 +0100)] 
docs: Add note on restoring the docker database

If you back something up, you'd probably want to restore it soon enough
too.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agotrivial: Remove additional Django < 1.8 code
Stephen Finucane [Sat, 7 Apr 2018 16:57:33 +0000 (17:57 +0100)] 
trivial: Remove additional Django < 1.8 code

Yet more stuff that was missed in the previous changes.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoREST: Order 'filters' code
Stephen Finucane [Sun, 25 Mar 2018 18:28:23 +0000 (19:28 +0100)] 
REST: Order 'filters' code

Group custom filters and fields together followed by the actual filter
sets. This makes the file a little easier to comprehend.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Add note on backing up the docker database
Stephen Finucane [Sun, 25 Mar 2018 18:28:22 +0000 (19:28 +0100)] 
docs: Add note on backing up the docker database

I'm sick of waiting for 'parsearchive' to finish.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Add information on REST API versioning
Stephen Finucane [Sun, 25 Mar 2018 18:28:21 +0000 (19:28 +0100)] 
docs: Add information on REST API versioning

This isn't too prescriptive, given that so far we've only dealt with
adding new fields. However, it should serve as a guide to alert devs
that this stuff exists and should be a concern.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoREST: Use versioning for modified responses
Stephen Finucane [Sun, 25 Mar 2018 18:28:20 +0000 (19:28 +0100)] 
REST: Use versioning for modified responses

This ensures clients are getting a consistent response if they request
the old version of the API. We do this by way of extensions to the
'HyperlinkedModelSerializer' class rather than duplicating the
serializers as it results in far less duplication. This approach won't
work for a MAJOR version bump but, all going well, it will be a while
before we have to deal with one of these.

The only two fields added since API 1.0 was released, 'cover.mbox' and
'project.subject_match', are handled accordingly.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agotests: Split 'test_rest_api'
Stephen Finucane [Sun, 25 Mar 2018 18:28:19 +0000 (19:28 +0100)] 
tests: Split 'test_rest_api'

Lay the tests out per the main code.

Signed-off-by: Stephen Finucane <stephen@that.guru>
7 years agoUse parsed subject for mboxes
Stephen Finucane [Sat, 7 Apr 2018 16:41:07 +0000 (17:41 +0100)] 
Use parsed subject for mboxes

With a recent change, we started using the original subject header
instead of the one we had already cleaned up at the parsing stage.
Revert this aspect of that change.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Fixes: 01b9cbb9 ("Include all email headers in mboxes")
7 years agoInclude all email headers in mboxes
Veronika Kabatova [Thu, 5 Apr 2018 15:51:58 +0000 (17:51 +0200)] 
Include all email headers in mboxes

Solves issue #165 (Exported mboxes should include In-Reply-To,
References, etc headers). Instead of including only a few chosen ones,
all received headers are added to mboxes.

Signed-off-by: Veronika Kabatova <vkabatov@redhat.com>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agoFix incorrect autodelegation documentation
Veronika Kabatova [Tue, 3 Apr 2018 16:18:36 +0000 (18:18 +0200)] 
Fix incorrect autodelegation documentation

The docs suggested to account for git prefixes (a/, b/) using eg.
?/patchwork/views/*. My rules didn't work so I tried bare path
(patchwork/views/*) instead. Looking at the code, the prefix really is
striped away (filename = '/'.join(filename.split('/')[1:])). Fix the
documentation to reflect on what is really happening.

Signed-off-by: Veronika Kabatova <vkabatov@redhat.com>
[dja: see 7bb0ebd78ff7 ("parser: Add patch_get_filenames()")]
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agodocker: set timezone to Australia/Canberra
Daniel Axtens [Tue, 20 Mar 2018 22:34:15 +0000 (09:34 +1100)] 
docker: set timezone to Australia/Canberra

The tzinfo package isn't installed in docker, which makes the
default timezone UTC. This is unfortunate: the Django TZ in
settings/base.py is Australia/Canberra, and having a non-UTC
TZ is good for exposing faulty assumptions about what is and
isn't UTC.

Signed-off-by: Daniel Axtens <dja@axtens.net>
Reviewed-by: Stephen Finucane <stephen@that.guru>
7 years agodocs: Update reference to kernel documentation
Ali Alnubani [Sat, 31 Mar 2018 13:56:00 +0000 (16:56 +0300)] 
docs: Update reference to kernel documentation

The referenced url was moved to Documentation/process.

Signed-off-by: Ali Alnubani <alialnu@mellanox.com>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agodocs: Fix package name
Ali Alnubani [Sat, 31 Mar 2018 13:55:59 +0000 (16:55 +0300)] 
docs: Fix package name

Fixed a typo that instructed to install tox instead
of reno.

Signed-off-by: Ali Alnubani <alialnu@mellanox.com>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoapi: Only provide JSON version of events list
Daniel Axtens [Mon, 12 Mar 2018 12:08:54 +0000 (23:08 +1100)] 
api: Only provide JSON version of events list

Something is very, very slow in the d-r-f browsable API events renderer.

In my MySQL test (~33k patches), the CPU time to render the events list
is ~11s, and the time taken by SQL queries is only ~3s. If the JSON
renderer is used, that drops to 0.2s for the entire page (because less
CPU is used, and - for some as yet unknown reason - a *very* expensive
db query is dropped.)

In my PostgreSQL test (~100k patches), the results are even more stark:
30s of CPU time and 0.2s of DB time goes to 0.25s for the entire page.

Something is seriously, seriously wrong with whatever d-r-f is doing.
So, simply render the event list as unlinked JSON for now.

There are a few followups we should do, but this is an important start -
no-one should be able to DoS a patchwork server by just enumerating the
events!

In particular, we should find out:
 - why postgres and mysql behaviour is so different.
 - what on earth d-r-f is doing that makes rendering the pretty-printed
   version so incredibly slow.

Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoapi: EventList: change select_related() to prefetch_related()
Daniel Axtens [Mon, 12 Mar 2018 11:07:15 +0000 (22:07 +1100)] 
api: EventList: change select_related() to prefetch_related()

select_related() creates a single giant query that JOINs the required
tables together in the DB. prefetch_related() does a similar thing,
but at the Django layer - for all referenced models, it makes a
separate query to the DB to fetch them.

This massively, massively simplifies the job the DB has to do:
instead of creating a massive, sparse results table with many
columns, we do 1 query for the events, and then query for only
patches/cover letters/series/projects etc referenced in those 30
events.

Tested with cURL + JSON renderer + Postgres w/ ~100k patches,
request time went from 1.5s to 0.25s, a 6x speedup.

Tested with cURL + JSON renderer + MySQL w/ ~33k patches,
request time went from ~2.2s to ~0.20s, an ~11x speedup.

Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agomigrations: Add missing migration
Stephen Finucane [Sun, 25 Mar 2018 16:14:53 +0000 (17:14 +0100)] 
migrations: Add missing migration

Add a migration that was missed in an earlier change.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Fixes: 0f25d8a15 ("Add validation for regular expressions")
7 years agoFix slow Patch counting query
Daniel Axtens [Thu, 8 Mar 2018 01:28:22 +0000 (12:28 +1100)] 
Fix slow Patch counting query

Stephen Rothwell noticed (way back in September - sorry Stephen!) that
the following query is really slow on OzLabs:

SELECT COUNT(*) AS "__count" FROM "patchwork_patch"
    INNER JOIN "patchwork_submission" ON
        ("patchwork_patch"."submission_ptr_id" = "patchwork_submission"."id")
    WHERE ("patchwork_submission"."project_id" = 14 AND
           "patchwork_patch"."state_id" IN
       (SELECT U0."id" AS Col1 FROM "patchwork_state" U0
                WHERE U0."action_required" = true
ORDER BY U0."ordering" ASC));

I think this is really slow because we have to join the patch and
submission table to get the project id, which we need to filter the
patches.

Duplicate the project id in the patch table itself, which allows us to
avoid the JOIN.

The new query reads as:
SELECT COUNT(*) AS "__count" FROM "patchwork_patch"
    WHERE ("patchwork_patch"."patch_project_id" = 1 AND
           "patchwork_patch"."state_id" IN
       (SELECT U0."id" AS Col1 FROM "patchwork_state" U0
        WHERE U0."action_required" = true
ORDER BY U0."ordering" ASC));

Very simple testing on a small, artifical Postgres instance (3
projects, 102711 patches), shows speed gains of ~1.5-5x for this
query. Looking at Postgres' cost estimates (EXPLAIN) of the first
query vs the second query, we see a ~1.75x improvement there too.

I suspect the gains will be bigger on OzLabs.

Reported-by: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Daniel Axtens <dja@axtens.net>
7 years agoAdd validation for regular expressions
Veronika Kabatova [Fri, 16 Mar 2018 19:10:54 +0000 (20:10 +0100)] 
Add validation for regular expressions

Make sure entered regexes compile before saving them.

Signed-off-by: Veronika Kabatova <vkabatov@redhat.com>
Signed-off-by: Daniel Axtens <dja@axtens.net>