Raxel Gutierrez [Fri, 20 Aug 2021 04:50:29 +0000 (04:50 +0000)]
patch-detail: add label and button for comment addressed status
Add new label to patch and cover comments to show the status of whether
they are addressed or not and add an adjacent button to allow users to
change the status of the comment. Only users that can edit the patch
(i.e. patch author, delegate, project maintainers) as well as comment
authors can change the status of a patch comment. For cover comments,
there are no delegates, so only maintainers and cover/cover comment
authors can edit the status of the cover comment. Before [1] and after
[2] images for reference.
Use new comment detail REST API endpoint to update the addressed field
value when "Addressed" or "Unaddressed" buttons are clicked. After a
successful request is made, the appearance of the comment status label
and buttons are toggled appropriately. For unsuccessful requests (e.g.
network errors prevents reaching the server), the error message is
populated to the page. A future improvement on this behavior is to add
a spinner to the button to provide a feedback that the request is in a
pending state until it's handled.
Raxel Gutierrez [Fri, 20 Aug 2021 04:50:27 +0000 (04:50 +0000)]
api: add comments detail endpoint
Add new endpoint for patch and cover comments at api/.../comments/<comment_id>.
This involves updating the API version to v1.3 to reflect the new
endpoints as a minor change, following the usual semantic versioning
convention.
The endpoint will make it possible to use the REST API to update the new
`addressed` field for individual patch and cover comments with JavaScript
on the client side. In the process of these changes, clean up the use of
the CurrentPatchDefault context so that it exists in base.py and can be
used throughout the API (e.g. Check and Comment REST endpoints).
The tests cover retrieval and update requests and also handle calls from
the various API versions. Also, they cover permissions for update
requests and handle invalid update values for the new `addressed` field.
Signed-off-by: Raxel Gutierrez <raxel@google.com>
[dja: changes to not conflict with, and to adopt the changes in, fecf7c86c2c5
various other minor changes as described on list] Signed-off-by: Daniel Axtens <dja@axtens.net>
Raxel Gutierrez [Fri, 20 Aug 2021 04:50:26 +0000 (04:50 +0000)]
models: change edit permissions for comments
Change patch comments' edit permissions to match that of the patch
associated with the comment (i.e. patch author, project maintainers, and
delegate) and add permissions for both patch and cover comment authors
to be able to change the `addressed` status of comments as well. For
cover comments, add permissions to edit for cover submitter and project
maintainers.
Signed-off-by: Raxel Gutierrez <raxel@google.com> Signed-off-by: Daniel Axtens <dja@axtens.net>
Raxel Gutierrez [Fri, 20 Aug 2021 04:50:25 +0000 (04:50 +0000)]
models: add addressed field
Currently, there is no state or status associated with comments. In
particular, knowing whether a comment on a patch or cover letter is
addressed or not is useful for transparency and accountability in the
patch review and contribution process. This patch is backend setup for
tracking the state of patch and cover comments.
Add `addressed` boolean field to patch and cover comments to be able to
distinguish between unaddressed and addressed comments in the
patch-detail page.
Signed-off-by: Raxel Gutierrez <raxel@google.com> Reviewed-by: Daniel Axtens <dja@axtens.net>
[dja: give the migration a more meaningful name] Signed-off-by: Daniel Axtens <dja@axtens.net>
Raxel Gutierrez [Fri, 20 Aug 2021 04:50:24 +0000 (04:50 +0000)]
templatetags: add utils template filters and tags
Add utils.py file to create template filters and tags that can be used
by most if not all objects in Patchwork. In particular, add a template
filter to get the plural verbose name of a model and add a template tag
that returns whether an object is editable by the current user. These
utilities will be used in an upcoming patch that adds the `addressed`
status label to patch and cover comments.
Signed-off-by: Raxel Gutierrez <raxel@google.com> Reviewed-by: Stephen Finucane <stephen@that.guru> Signed-off-by: Daniel Axtens <dja@axtens.net>
Raxel Gutierrez [Fri, 20 Aug 2021 04:50:23 +0000 (04:50 +0000)]
api: change <pk> parameter to <cover_id> for cover comments endpoint
Rename cover lookup parameter `pk` to `cover_id` for the cover comments
list endpoints to disambiguate from the lookup parameter `comment_id` in
upcoming patches which introduces the cover comments detail endpoint.
This doesn't affect the user-facing API.
Signed-off-by: Raxel Gutierrez <raxel@google.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
[dja: fixup to apply on top of fecf7c86c2c5 and 0ed2feb26c84] Signed-off-by: Daniel Axtens <dja@axtens.net>
Raxel Gutierrez [Fri, 20 Aug 2021 04:50:22 +0000 (04:50 +0000)]
patch-detail: left align message headers
Change both of the message containers in the "Commit Message" and
"Comments" to have their content be left-aligned which improves the
proximity of items which boosts the efficiency of gathering information
and performing actions. Before [1] and after [2] images for reference.
Daniel Axtens [Mon, 23 Aug 2021 07:29:53 +0000 (17:29 +1000)]
REST: squish final open-codings of get_object_or_404
Basically, finish the job of commit fecf7c86c2c5 ("urls: Add missing path
converters for REST APIs"). With this commit, there are no more uses of
Http404 in patchwork/api
Stephen Finucane [Fri, 20 Aug 2021 21:57:51 +0000 (22:57 +0100)]
urls: Add missing path converters for REST APIs
Almost all of the API endpoints expect numerical resource IDs, with
'/projects' being the sole exception. However, we were not actually
enforcing this anywhere. Instead, we were relying on the custom
'get_object_or_404' implementation used by 'GenericAPIView.retrieve' via
'GenericAPIView.get_object'. Unfortunately we weren't using this
everywhere, most notably in our handler for 'GET /patches/{id}/checks'.
The end result was a HTTP 500 due to a ValueError.
Resolve this by adding the path converters for all REST API paths in
'patchwork.urls', along with tests to prevent regressions going forward.
We also switch to the DRF variant of 'get_object_or_404' in some places
to provide additional protection.
Signed-off-by: Stephen Finucane <stephen@that.guru> Cc: Daniel Axtens <dja@axtens.net>
[dja: s/TypeError/ValueError/; dropped an unrelated change] Signed-off-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Thu, 19 Aug 2021 03:04:26 +0000 (13:04 +1000)]
migrations: ignore flake8 on 0041_python3
commit 3a979ed8bfc6 ("migrations: don't go to the db for 0041_python3 migration")
made a bunch of strings go past 79 characters, breaking flake8 checks.
`black` doesn't seem to fix this and reflowing the strings manually is
error-prone.
We're not really expecting future changes to this file so just don't run
flake8 against it.
Fixes: 3a979ed8bfc6 ("migrations: don't go to the db for 0041_python3 migration") Signed-off-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Fri, 20 Aug 2021 14:57:58 +0000 (00:57 +1000)]
REST: Don't error if a versioned field we would remove is absent
We remove fields that shouldn't be seen on old versions of the API.
This was done with `pop(field name)`, which will throw an exception
if the named field is absent from the data. However, sometimes if
a patch request is via an old API version, we hit this line without
ever having the field present.
This is odd, but not harmful and we definitely shouldn't 500.
Signed-off-by: Daniel Axtens <dja@axtens.net>
[stephenfin: Merge test into bug fix] Signed-off-by: Stephen Finucane <stephen@that.guru> Tested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Fixes: d944f17ec059 ("REST: Use versioning for modified responses") Closes: #423
Raxel Gutierrez [Thu, 19 Aug 2021 04:20:23 +0000 (04:20 +0000)]
static: add rest.js to handle PATCH requests & respective responses
Add `rest.js` file to have a utilities JavaScript module that can be
reused by any Patchwork JS files that make requests to the REST API. The
comments for each function follow the Google JS Style guide [1] which is
something that would be nice to have for better documented frontend code,
especially for JS modules that export functions like rest.js. In
particular, this patch provides the following function:
- `updateProperty`: make PATCH requests that partially update the
fields of an object given it's REST API endpoint specified by the
caller. Also, the caller can specify the field(s) to modify and the
associated content for update messages in the case of both failed
successful requests that render to the current webpage. The caller
receives whether the request was successful or not.
The `rest.js` module can be further expanded to support and provide
functions that allow for other requests (e.g. GET, POST, PUT) to the
REST API.
Also, add functions that handle update & error messages for these PATCH
requests that match the Django messages framework format and form error
styling. These functions are internal to the module and aren't exposed
outside of the `rest.js` file.
Error and accompanying failed update messages are replaced by successful
update messages and vice versa. Consecutive successful update messages
add to a counter of updated objects. Consecutive error messages stack up.
Signed-off-by: Raxel Gutierrez <raxel@google.com> Reviewed-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
Raxel Gutierrez [Tue, 17 Aug 2021 21:31:40 +0000 (21:31 +0000)]
messages: clean up messages and errors containers
Refactor the messages container to make use of message.tags [1] which
allows for more customization for each level (e.g. success, warning,
error, etc.) of a message through CSS selectors. As basic proof of
concept, customize the text color of each existing message level. Also,
this addition resolves a TODO by stephenfin in the previous code.
Move the errors container after the messages container in the DOM in the
base.html template so that every template can share the same errors
container. Also, add a background color to the errors container so that
both containers blend in as a uniform block. Add bullet points to each
error item in the list of errors.
Change both the messages and errors containers to always exist in
the DOM. With this, the addition of update and error messages is simpler
because it eliminates the need to create the containers if they don't
exist. These changes will be useful in a following patch that introduces
an internal JS module to make client-side requests to the REST API.
These were previously symlinked, for reasons that I cannot fathom. Just
move them to where they are moved. In the future, we will probably want
to manage these with a package manager.
A README is removed as it mostly duplicated content already found in
'htdocs/README.rst'.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Raxel Gutierrez [Tue, 17 Aug 2021 21:33:49 +0000 (21:33 +0000)]
static: add JS Cookie library to get csrftoken for client-side requests
Currently in Patchwork, requests are made only through older methods via
form submissions, which means the UI is rendered strictly server-side.
This lags behind more modern and versatile approaches that use
JavaScript to send requests and dynamically update the UI based on the
respective responses.
In order to make REST API requests on the client-side secure from CSRF
attacks, add the JS Cookie library which allows the CSRF token to be
passed in the request header. A following patch that introduces the
`rest.js` module will make use of the JS Cookie library in this patch.
The library is a recommendation from Django docs [1]. The files for the
library can be downloaded in the releases page of the GitHub [2].
Raxel Gutierrez [Fri, 13 Aug 2021 05:31:21 +0000 (05:31 +0000)]
patch-detail: change patch info toggles from links to buttons
Change toggle links (i.e. show/hide and expand/collapse) in patch info
section to buttons because links normally suggest movement to another
page whereas buttons indicate an action on the current page. Also,
buttons have a bigger clickable area which boosts the efficiency of
the toggle actions.
Raxel Gutierrez [Fri, 13 Aug 2021 05:31:20 +0000 (05:31 +0000)]
patch-detail: refactor JS code into submission.js
Move submission.html script code for toggling header info to a separate
submission.js file that makes the code easy to read and ready for change
in one place.
The listener is moved from the 'href' to separate, explicit click
listener.
Signed-off-by: Raxel Gutierrez <raxel@google.com>
[stephenfin: Removed existing JS href added in previous change and
update commit message to reflect this] Signed-off-by: Stephen Finucane <stephen@that.guru>
Raxel Gutierrez [Fri, 13 Aug 2021 05:31:18 +0000 (05:31 +0000)]
api: change <pk> parameter to <patch_id> for comments endpoint
Refactor patch lookup parameter `pk` to `patch_id` for the comments list
endpoint to disambiguate from the lookup parameter `comment_id` in an
upcoming patch which introduces the comments detail endpoint. This
doesn't affect the user-facing API.
Signed-off-by: Raxel Gutierrez <raxel@google.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Daniel Axtens [Tue, 22 Jun 2021 07:59:37 +0000 (17:59 +1000)]
docs: clarify and improve docker dev docs
The docker dev docs:
- had the links for docker and docker-compose swapped.
- had a old docker install link.
- lacked an up-front explanation of the requirement that a regular user
be able to manage the docker daemon (and a fairly unhelpful reference
link in the most appropriate note block.)
Fix it all.
Reported-by: Emily Shaffer <emilyshaffer@google.com> Signed-off-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Wed, 23 Jun 2021 01:19:36 +0000 (11:19 +1000)]
docker: install Python 3.9
Since commit 9a54bf4bfc54 ("Add Python 3.9 support"), Python 3.9 is tested
by tox, so currently `docker-compose run web --tox` fails due to missing
Py3.9 binaries. Fix it.
Fixes: 9a54bf4bfc54 ("Add Python 3.9 support") Signed-off-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Tue, 22 Jun 2021 15:03:19 +0000 (01:03 +1000)]
tests: fix parallel tests
Parallel tests require:
- the % wildcard to be in a token enclosed by backticks, not single
quotes
- that the user still be able to use 'test_patchwork' (so we don't want
the \_ before the %)
Presumably this was skipped because if you get permissions working
manually but you miss part of the required permissions in the automated
script, you need to delete the old db data in order to observe the
issue.
Amusingly postgres worked the whole time.
Fixes: 6025f0e2533f ("Add parallel testing") Signed-off-by: Daniel Axtens <dja@axtens.net>
Commit e5c641fc4 optimized fetching of checks when displaying a patch by
prefetching these checks ahead of time. Unfortunately we missed that
this should exclude older versions of checks for a given context. Make
the code used to generate the unique checks generic and allow us to use
that as a filter to the checks provided to the template, restoring the
correct behavior.
Signed-off-by: Stephen Finucane <stephen@that.guru> Closes: #398 Fixes: e5c641fc4 ("Optimise fetching checks when displaying a patch")
Stephen Finucane [Wed, 24 Feb 2021 15:08:45 +0000 (15:08 +0000)]
api: do not fetch every patch in a patch detail view 404 (v2)
Commit 08c5856 fixed an issue whereby a 404 on the aforementioned URL
could result in a large DB query due to DRF attempting to populate the
'related' list box with all patches on the instance. That was
accidentally reverted in commit fe07f30. "Unrevert" this change.
Signed-off-by: Stephen Finucane <stephen@that.guru> Fixes: fe07f30 ("Remove 'PatchRelationSerializer'") Closes: #397
Stephen Finucane [Sat, 20 Feb 2021 14:35:07 +0000 (14:35 +0000)]
models: Use parent model to get comment's 'list_archive_url'
We were attempting to retrieve the 'list_archive_url' attribute from the
'PatchComment' or 'CoverComment' instances, rather than the parent
'Patch' and 'Cover' object, respectively. Correct this and add plenty of
tests to prevent this regressing.
Signed-off-by: Stephen Finucane <stephen@that.guru> Fixes: 02ffb1315 ("models: Add list archive lookup") Closes: #391
Stephen Finucane [Sat, 20 Feb 2021 12:22:08 +0000 (12:22 +0000)]
urls: Support sha256-based tokens
Django 3.1 changed the default hashing algorithm used for things like
password reset tokens from SHA-1 to SHA-256. As noted in the release
notes [1], this is configurable via the 'DEFAULT_HASHING_ALGORITHM'
transitional setting, but that's only intended to allow upgrades of
multiple instances in a HA deployment and shouldn't be used post
upgrade. Instead, we need to fix our URLs to support the longer tokens
generated by SHA-256.
Long term, we want to replace these regex-based routes with the simpler
flask-style template string routes. That's not really backportable so
we'll do that separately.
Stephen Finucane [Sat, 20 Feb 2021 13:27:08 +0000 (13:27 +0000)]
tests: Fix compatibility with openapi_core 0.13.7
It seems the 'openapi_core.schema.schemas.models.Format' mechanism of
defining custom formatters was deprecated in openapi_core 0.12.0 but we
never noticed. They've finally broken it in 0.13.7. Switch to the new
thing, 'openapi_core.unmarshalling.schemas.formatters.Formatter', which
expects a slightly different format.
Signed-off-by: Stephen Finucane <stephen@that.guru> Closes: #395
Stephen Finucane [Sun, 13 Dec 2020 19:44:53 +0000 (19:44 +0000)]
requirements: Switch to openapi-core 0.13.4
In commit b7f3c3d34 ("tests: Switch to openapi-core 0.13.x") we added
support for 'openapi-core' 0.13.x. However, we needed to use a commit
from master since an important fix [1] was not included in the latest
release at the time, 0.13.3. 0.13.4 has since been released so let's
move on and use that.
Stephen Finucane [Sun, 29 Nov 2020 12:50:22 +0000 (12:50 +0000)]
REST: Null out previous, current relation info
These fields don't work like we expect them to. Because we're linking to
a non-idempotent entity, an instance of 'relation', what we're storing
in either of these fields is subject to change as patches are added and
removed. This makes the information pretty much useless after the fact.
It's best to just state the patch and request that people query the
information themselves if necessary. We don't want to remove the field
entirely from the API - that would be a truly breaking change - so
instead we null it out like we do for patch tags. In a v2 API (i.e. a
major version bump) we can remove this entirely.
A small bug with the schema generation is corrected.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Related: #379
This wasn't writeable for reasons I haven't been able to figure out.
However, it's not actually needed: the 'PatchSerializer' can do the job
just fine, given enough information. This exposes a bug in DRF, which
has been reported upstream [1]. While we wait for that fix, or some
variant of it, to be merged, we must monkey patch the library.
tests: Add tests for 'patch-relation-changed' events
This event is rather odd. If you have two patches then the way a
relation is created is by creating a 'PatchRelation' instance and then
setting the 'related' attribute on the first patch followed by the
second patch. Because the event uses the 'Patch' model's 'pre_save'
signal, we'll only see events for the patch being currently saved. This
means no event will be raised for the first patch and only one event,
the one for the second patch, will be raised when the second patch is
being added to the relationship.
In hindsight, the structure of the event is off. We should have had
something like a 'patch-added-to-relationship' and a
'patch-removed-from-relationship' event, both with the same fields:
'project', 'actor', 'patch' and 'related', the latter of which would
have listed all of the _other_ patches in the relationship. Sadly, this
is an API change which means we can't do it now. We may well wish to do
so in the future though.
Signed-off-by: Stephen Finucane <stephen@that.guru>
This wasn't actually creating just a patch relation object - it was also
creating patches, which is something we already have an explicit helper
for. Clean this thing up.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sat, 28 Nov 2020 17:32:32 +0000 (17:32 +0000)]
lib: Grant SELECT on auth_user
If a mail arrives with the 'X-Patchwork-Delegate' hint header, the
'patchwork.parser' script will need to index the users table to find the
appropriate user. This should be okay from a security perspective since
passwords are hashed and salted and the rest of the information is
mostly accessible publicly via the web UI and REST API.
Signed-off-by: Stephen Finucane <stephen@that.guru> Suggested-by: Ali Alnubani <alialnu@mellanox.com> Closes: #365
The patch_mbox view returns text/plain data without specifying a character
set, which means clients will assume the default of iso-8559-1 as defined
in the HTTP/1.1 standard. Since the data being returned is in fact utf-8
encoded, set the encoding accordingly in the HTTP Content-Type header.
Reported-by: Jakub Kicinski <kuba@kernel.org> Suggested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Signed-off-by: Toke Høiland-Jørgensen <toke@toke.dk>
Stephen Finucane [Sat, 28 Nov 2020 16:34:01 +0000 (16:34 +0000)]
parser: Update reference to PatchComment
Commit 0686a736fbf6d869bd31bd135ba38080ac96de22 split out 'CoverLetter'
from the old 'Submission' model, removing the common 'Comment' model in
favour of distinct 'CoverComment' and 'PatchComment' models in the
process. Unfortunately we misssed some references to the old model in
the 'patchwork.parser' module. Correct these now, adding unit tests to
prevent regressions.
Signed-off-by: Stephen Finucane <stephen@that.guru> Fixes: 0686a736 ("models: Split 'CoverLetter' from 'Submission'") Closes: #384
models: Validate Project.linkname does not contain forward slash
I started by creating a project that contained a forward slash
(importing patches from https://lists.sr.ht/~sircmpwn/sr.ht-dev/) and
it fails to render the "projects" main page.
The specific error reads:
NoReverseMatch at /
Reverse for 'patch-list' with keyword arguments
'{'project_id': 'foo/bar'}' not found. 1 pattern(s) tried:
['project/(?P<project_id>[^/]+)/list/$']
which appears to explicitly disallow forward slashes.
So I think it makes sense to validate that project linkname doesn't
contain forward slahes.
This implementation uses the validate_unicode_slug validator instead of just
rejecting inputs that contain forward slashes.
Signed-off-by: Thomas Bracht Laumann Jespersen <t@laumann.xyz> Signed-off-by: Stephen Finucane <stephen@that.guru> Closes: #380
Django 3.1 adds a new admin sidebar feature that requires the
django.template.context_processors.request context processor to be enabled
in the settings.
Signed-off-by: Andrew Donnellan <ajd@linux.ibm.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Andrew Donnellan [Thu, 27 Aug 2020 14:14:03 +0000 (00:14 +1000)]
urls: Update url pattern functions
Django 3.1 deprecates django.conf.urls.url() as an alias for
django.urls.re_path(). Also switch to using django.urls.include() rather
than django.conf.urls.include().
Signed-off-by: Andrew Donnellan <ajd@linux.ibm.com> Signed-off-by: Stephen Finucane <stephen@that.guru>
Rohit Sarkar [Sat, 1 Aug 2020 06:24:15 +0000 (11:54 +0530)]
management: introduce replacerelations command
The replacerelations script is used to ingest relations into Patchwork's
patch database. A patch groups file is taken as input, which on each
line contains a space separated list of patchwork ids denoting a
relation. All the existing relations in Patchwork's database are removed
and the relations read from the patch groups file are ingested.
Signed-off-by: Rohit Sarkar <rohitsarkar5398@gmail.com>
[dja: pep8, drop relations directory as empty dirs don't get stored by git,
comment about how lines are generated.] Signed-off-by: Daniel Axtens <dja@axtens.net>
Stephen Finucane [Thu, 30 Apr 2020 21:22:10 +0000 (22:22 +0100)]
tox: Add default Django version
I occasionally forget myself and run e.g. 'tox -e pyNN' when I want to
sanity check something instead of 'tox -e pyNN-djangoMM'. Add fallback
Django versions so that this doesn't crash and burn. It's less succict
than it could be since tox doesn't seem to support '!django{22,30}'
(yet!).
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sun, 26 Apr 2020 13:17:57 +0000 (14:17 +0100)]
migrations: Resolve issues with other DB backends for 0042, 0043
0042 was using MySQL-specific SQL to delete entries in the
'patchwork_comment' table that were associated with entries in the
'patchwork_coverletter' table, while 0043 only considered MySQL and
PostgrSQL when attempting to copy fields from 'patchwork_patch' to
'patchwork_submission'. Both issues are resolved.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Fri, 10 Apr 2020 11:08:32 +0000 (12:08 +0100)]
migrations: Squash 0001 to 0040
Now that we're moving a new major version, we can squash the migrations
we have to date. This coexists alongside the old migrations, which we
can remove and replace entirely when we release 4.0, per the advice of
the 'squashmigrations' tool.
You should commit this migration but leave the old ones in place; the
new migration will be used for new installs. Once you are sure all
instances of the codebase have applied the migrations you squashed,
you can delete them.
The 'squashmigrations' tool can't parse the output of 'RunPython' blocks
so the output of the tool was less optimized than it could be. As a
result, we've manually modified this change to remove the 'RunPython'
block and unnecessary 'AlterField' entries. This was done by removing
all migrations and generating a new "initial" migration, which was then
modified to mark all strings as byte strings (as they were when we were
using Python 2 to generate these migrations) so that 0041 would apply
cleanly.
The main benefit of this change is that it significantly reduces the
startup time for unit tests. Executed on my host, the run time for a
single test goes from ~ 22 seconds to ~ 14 seconds. This is obviously
reduced for additional tests.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Oh, the follies of youth. Time to undo the damage of 2.0.0, specifically
commit 86172ccc16, which split Patch into two separate models using
concrete inheritance. As noted previously, this introduced a large
number of unavoidable JOINs across large tables and the performance
impacts these introduce are blocking other features we want, such as
improved tagging functionality. To combine these two models, we must do
the following:
- Update any references to the 'Patch' model to point to the
'Submission' model instead
- Move everything from 'Patch' to 'Submission', including both fields
and options
- Delete the 'Patch' model
- Rename the 'Submission' model to 'Patch'
With this change, our model "hierarchy" goes from:
Submission
Patch
PatchComment
Cover
CoverComment
To a nice, flat:
Patch
PatchComment
Cover
CoverComment
I expect this migration to be intensive, particularly for MySQL users
who will see their entire tables rewritten. Unfortunately I don't see
any way to resolve this in an easier manner.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sat, 29 Feb 2020 17:20:50 +0000 (17:20 +0000)]
models: Split 'CoverLetter' from 'Submission'
We want to get rid of the split between 'Patch' and 'Submission' because
of the cost of using JOINs basically everywhere we use 'Patch'. Before
we do that, we need to move the other users of 'Submission' to other
models and other models that rely on these users sharing the common
'Submission' base. For the former, there is only one user,
'CoverLetter', while for the latter there is only the 'Comment' model.
As a result, we must do the following:
- Create a new 'Cover' model
- Create a new 'CoverComment' model
- Move everything from 'CoverLetter' to 'Cover' and all entries
associated with a 'CoverLetter' from 'Comment' to 'CoverComment'
- Delete the 'CoverLetter' model
- Rename the 'Comment' model to 'PatchComment'
This means our model "hierarchy" goes from:
Submission
Patch
CoverLetter
Comment
To:
Submission
Patch
PatchComment
Cover
CoverComment
A future change will flatten the 'Submission' and 'Patch' model.
Note that this actually highlighted a bug in Django, which has since
been reported upstream [1]. As noted there, the issue stems from MySQL's
refusal to remove an index from a foreign key when DB constraints are
used and the workaround is to remove the foreign key constraint before
altering the indexes and then re-add the constraint after.
[1] https://code.djangoproject.com/ticket/31335
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sat, 18 Apr 2020 12:21:13 +0000 (13:21 +0100)]
travis: Resolve warnings, info messages from Travis
The following were reported by Travis' build config validation:
- root: deprecated key 'sudo' (The key `sudo` has no effect anymore.)
- env: key 'matrix' is an alias for 'jobs', using 'jobs'
- root: key 'matrix' is an alias for 'jobs', using 'jobs'
- root: missing 'os', using the default 'linux'
Resolve all of the above.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Jeremy Kerr [Thu, 16 Apr 2020 01:29:25 +0000 (09:29 +0800)]
tests: ensure we don't see database errors during duplicate insert
Currently, the parser causes IntegrityErrors while inserting duplicate
patches; these tend to pollute database logs.
This change adds a check, which currently fails, to ensure we do not
cause errors during a duplicate patch parse.
Signed-off-by: Jeremy Kerr <jk@ozlabs.org> Signed-off-by: Stephen Finucane <stephen@that.guru>
[stephenfin: Add 'expectedFailure' marker to keep all tests green]
Andrew Donnellan [Wed, 15 Apr 2020 09:06:56 +0000 (19:06 +1000)]
parser: Don't crash when From: is list email but has weird mangle format
get_original_sender() tries to demangle DMARC-mangled From headers, in
the case where the email's From address is the list address. It knows how
to handle Google Groups and Mailman style mangling, where the original
submitter's name will be turned into e.g. "Andrew Donnellan via
linuxppc-dev".
If an email has the From header set to the list address but has a name that
doesn't include " via ", we'll throw an exception because stripped_name
hasn't been set. Sometimes this is because the list name is seemingly
empty, resulting in a mangled name like "Andrew Donnellan via"
without the space after "via" that we detect. Handle this as well as we can
instead, and add a test.
Fixes: 8279a84238c10 ("parser: Unmangle From: headers that have been mangled for DMARC purposes") Reported-by: Jeremy Kerr <jk@ozlabs.org> Signed-off-by: Andrew Donnellan <ajd@linux.ibm.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Wed, 15 Apr 2020 15:52:08 +0000 (16:52 +0100)]
tests: Switch to openapi-core 0.13.x
We've done the necessary work here already so this is a relatively easy
switchover. However, we do have to work around an issue whereby the
first possible matching route is used rather than the best one [1]. In
addition, we have to install from master since there are fixes missing
from the latest release, 0.13.3. Hopefully both issues will be resolved
in a future release.
Stephen Finucane [Wed, 15 Apr 2020 15:44:32 +0000 (16:44 +0100)]
tests: Drop Django 1.x support
openapi-core 0.13.x has added support for Django validation. Before we
migrate to that version and presumably remove most of this code, remove
the stuff that is *definitely* dead.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Thu, 16 Apr 2020 13:03:54 +0000 (14:03 +0100)]
docs: Resolve issues with 'relations'
Two issues here:
- 'PATCH /patches/{id}' and 'PUT /patches/{id}' expect a list of
integers on the 'related' field - not strings
- 'GET /patches' and 'GET /patches/{id}' return a list of embedded patch
objects on the 'related' field - not strings
Signed-off-by: Stephen Finucane <stephen@that.guru>
projects / thirdparty / patchwork.git / log
