Add tests for the recent changes we made to how we parse multiple series
received at once. These tests actually highlighted what appeared to be
the test failure that's been intermittently breaking our CI for years
now, so the 'expectedFailure' marker has been removed in the hope that
this is actually the case.
Signed-off-by: Stephen Finucane <stephen@that.guru>
parser: Use a second query to weed out duplicate series
Annoyingly, not all email clients properly thread emails using the
message ID fields originally specified in RFC 822 [1]. Worse, some MTAs
(cough, outlook.com, cough) actually override what the client
configures, breaking the world in the process. Realising this is an
issue, Patchwork supports threading using arbitrary metadata in addition
to the RFC 822 metadata. Specifically, it uses a combination of
submitter and list-id extracted from the headers along with the series
version and total count metadata extracted from the subject. In addition
to this, we timebox things so that two or more series that match on all
of this metadata but which are sent some time apart from each other
aren't combined by accident. This does leave one edge case - duplicate
series received within the timebox will be combined. We've resigned
ourselves to this fact on the basis that it's extremely unlikely for all
of these things to go wrong at once.
Given all the above, there should be no reason that attempting to find
series by series markers should return more than one series. The
timeboxing will prevent us grouping similar looking series by accident
and the only other reason for this to happen is because we lost a race
and we should try again.
[1] https://tools.ietf.org/html/rfc822
Signed-off-by: Stephen Finucane <stephen@that.guru> Cc: Daniel Axtens <dja@axtens.net>
models: Use database constraints to prevent split Series
Currently, the 'SeriesReference' object has a unique constraint on the
two fields it has, 'series', which is a foreign key to 'Series', and
'msgid'. This is the wrong constraint. What we actually want to enforce
is that a patch, cover letter or comment is referenced by a single
series, or rather a single series per project the submission appears on.
As such, we should be enforcing uniqueness on the msgid and the project
that the patch, cover letter or comment belongs to.
This requires adding a new field to the object, 'project', since it's
not possible to do something like the following:
unique_together = [('msgid', 'series__project')]
This is detailed here [1]. In addition, the migration needs a precursor
migration step to merge any broken series.
These are failing due to differences in behavior of the backend. Since
this will never be used for production, we can simply skip these unit
tests and rely on the CI to catch potential issues.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Fri, 20 Dec 2019 10:49:21 +0000 (10:49 +0000)]
models: Add State.slug field
This reduces a lot of the tech debt we had built up around this. This
field is populated from a slugified representation of State.name and has
a uniqueness constraint. As a result, we also need to a uniqueness
constraint to the State.name field. This shouldn't be an issue and
probably should have been used from day one.
Note that there is no REST API impact for this since we've been using
state slugs all along.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sat, 30 Nov 2019 18:48:35 +0000 (18:48 +0000)]
tests: Provide a way to disable API schema
The API schema validation is strict, in that it will error out with
invalid keys in either the request or response. Unfortunately the API
itself is not. We're hopefully going to fix this in a distant v2.0, but
for now we need a way to ensure that the API does what it's supposed to,
namely not set fields that don't exist or that the user isn't allowed to
set, even if proper error codes aren't raised.
This isn't actually used yet. That will come later.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Mete Polat [Sat, 7 Dec 2019 16:46:20 +0000 (17:46 +0100)]
docs: Add missing series index schema
Fixes: 7d8e24bc84bd ("docs: Start documenting API using OpenAPI") Signed-off-by: Mete Polat <metepolat2000@gmail.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
requirements: Add pyup markers to prevent dumb PRs
Until [1] is merged, we're going to have to override what these markers
are doing. Perhaps it would be easier to just specify the markers in the
comments as the actual marker, but I like using pip's features and the
comments *should* be temporary.
[1] https://github.com/pyupio/pyup/pull/367
Signed-off-by: Stephen Finucane <stephen@that.guru>
Johan Herland [Sun, 1 Dec 2019 01:49:53 +0000 (02:49 +0100)]
Include the responsible actor in applicable events
We want to use the events as an audit log. An important part of this is
recording _who_ made the changes that the events represent.
To accomplish this, we need to know the current user (aka. request.user)
at the point where we create the Event instance. Event creation is
currently triggered by signals, but neither the signal handlers, nor the
model classes themselves have easy access to request.user.
For some Patch-based events (patch-state-changed, patch-delegated), we
can do the following hack: The relevant events are created in signal
handlers that are all hooked up to either the pre_save or post_save
signals sent by Patch.save(). But before calling Patch.save(),
Patchwork must naturally query Patch.is_editable() to ascertain whether
the patch can in fact be changed by the current user. Thus, we only
need a way to communicate the current user from Patch.is_editable()
to the signal handlers that create the resulting Events. The Patch
object itself is available in both places, so we simply add an
'_edited_by' attribute to the instance (which fortunately is not
detected as a persistent db field by Django).
For the check-created event the current user always happens to be the
same as the 'user' field recorded in the Check object itself.
For the other Patch-based events (patch-created, patch-completed, and
series-completed), although they are also triggered by Patch.save(),
they are triggered as a result of incoming emails, hence have no real
actor as such, so we simply leave the actor as None/NULL. The same
argument also applies to the cover-created and series-created events.
Signed-off-by: Johan Herland <johan@herland.net> Reviewed-by: Stephen Finucane <stephen@that.guru> Cc: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Stephen Finucane [Sat, 30 Nov 2019 17:11:13 +0000 (17:11 +0000)]
templates: Use 'en' locale
As discussed at [1], the UI was originally written in Australian English
but as it's been through a couple of pairs of hands since the chances
are things are more than a little messed up. Just use 'en' as our locale
rather than 'en-US', 'en-AU' or anything else.
Jeremy Cline [Tue, 15 Oct 2019 21:30:11 +0000 (17:30 -0400)]
Allow ordering events by date
By default, the events API orders events by date in descending order
(newest first). However, it's useful to be able to order the events by
oldest events first. For example, when a client is polling the events
API for new events since a given date and wishes to process them in
chronological order.
Signed-off-by: Jeremy Cline <jcline@redhat.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Stephen Finucane [Sat, 30 Nov 2019 16:24:36 +0000 (16:24 +0000)]
REST: Exclude filters added in later version
If a person requests API version 1.1, they should get the exact same
behavior regardless of the base Patchwork version. We already do this
for fields in the output, so now extend this to filters in the
querystring.
Signed-off-by: Stephen Finucane <stephen@that.guru> Cc: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Wed, 23 Oct 2019 14:33:42 +0000 (01:33 +1100)]
api: support filtering patches by hash
This is a feature that the XML-RPC API has, and which is used in
the wild [1], so support it in the REST API.
I tried to version the new filter field, but it's not at all clear
how to do this with django-filters. The best way I could find
requires manually manipulating request.GET, which seems to defeat
the point of django-filters. So document it for 1.2, and have it
work on older versions as an undocumented feature.
Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Signed-off-by: Daniel Axtens <dja@axtens.net> Acked-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Ali Alnubani [Mon, 21 Oct 2019 15:06:24 +0000 (15:06 +0000)]
docs: Fix note about the required Postfix rights
The permissions for the user running the postfix process are
not the ones used for external file or command delivery by default.
The ones defined by default_privs are (in case the aliases(5) file
that is owned by root was being used). A privileged user or the
postfix owner should not be used in this case.
See http://www.postfix.org/postconf.5.html#default_privs and
local(8).
Signed-off-by: Ali Alnubani <alialnu@mellanox.com> Reviewed-by: Stephen Finucane <stephen@that.guru>
Two issues here. Firstly, the use of the 'USE_I18N'. The Django docs
describe this as such:
A boolean that specifies whether Django’s translation system should
be enabled. This provides an easy way to turn it off, for performance.
If this is set to False, Django will make some optimizations so as not
to load the translation machinery.
We don't do translations and won't until such a time as someone comes
asking for them. Optimize things accordingly by setting 'USE_I18N' to
False and removing the now-unnecessary 'LANGUAGE_CODE' setting.
Secondly, the use of en-AU is a bit of a lie since our UI is actually
written in US English (or should be). The primary reason for a lang tag
to be present is to assist screenreaders and other accessibility tools,
so make their lives easier by reflecting the truth.
Signed-off-by: Stephen Finucane <stephen@that.guru>
When git-request-pull output is pasted into a mail client instead of
mailed directly, the ref part of the pull URL may end up wrapped to the
next line.
Daniel Axtens [Tue, 29 Oct 2019 07:04:14 +0000 (18:04 +1100)]
README: stop trying to track supported versions
We're not doing a good job of it, the versions are out of date and
we keep forgetting to update the README. We are a bit better at
making release notes, so just point people there.
Reviewed-by: Stephen Finucane <stephen@that.guru> Signed-off-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Tue, 29 Oct 2019 07:10:44 +0000 (18:10 +1100)]
README: fix .env
The .env setup didn't do GID. It's a bit of a chore to do because
there doesn't seem to be a GID shell variable and because we need
to do a bit more work to get a multi-line thing, but this should
work.
While we're at it, change the docker-compose info, it's hopelessly
out of date.
Reviewed-by: Stephen Finucane <stephen@that.guru> Signed-off-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Thu, 24 Oct 2019 03:17:04 +0000 (14:17 +1100)]
travis, tox: only do coverage testing in py27
Currently Travis calls `tox -e coverage` unconditionally. However,
the environment has py27 basepython, so all the runs only generate
py27 coverage!
Rather than try to untangle that, just run the coverage when run
in a py27 travis environment. This makes things faster for no
loss of coverage. It means that codecov has nothing to submit for
the py3x environments, but that's no real loss: it would otherwise
submit lots of duplicate data.
We could try to improve coverage by running coverage for 27 and 3x,
but given that 27 is going away, don't stress at this point.
Daniel Axtens [Wed, 23 Oct 2019 22:12:50 +0000 (09:12 +1100)]
travis: fix codecov
In e017f69376da ("travis: run pep8/flake8 tests"), codecov
was removed from the install step, on the basis that tox-travis
would pull it in automatically.
This, it turns out, isn't entirely true: it is pulled in to the
tox environment, but the data is actually sent in an after_success
step. That is outside the tox environment, as it should be - if it
were part of the tox environment, running `tox -e coverage` on a
developer laptop would try to send data to the web. But, as codecov
now isn't present outside the tox environment, we see in the logs:
$ codecov
codecov: command not found
We don't get any reporting of success/failure in the after_success
step, so we didn't notice.
Restore the installation in the travis environment.
Fixes: e017f69376da ("travis: run pep8/flake8 tests") Signed-off-by: Daniel Axtens <dja@axtens.net>
Andrew Donnellan [Wed, 16 Oct 2019 04:23:07 +0000 (15:23 +1100)]
parser: Unmangle From: headers that have been mangled for DMARC purposes
To avoid triggering spam filters due to failed signature validation, many
mailing lists mangle the From header to change the From address to be the
address of the list, typically where the sender's domain has a strict DMARC
policy enabled.
In this case, we should try to unmangle the From header.
Add support for using the X-Original-From or Reply-To headers, as used by
Google Groups and Mailman respectively, to unmangle the From header when
necessary and associate the patch with the correct submitter based on the
unmangled email address.
When downloading mboxes, rewrite the From header using the unmangled
address, and preserve the original header as X-Patchwork-Original-From in
case someone needs it for some reason. The original From header will still
be stored in the database and exposed via the API, as we want to keep
messages as close to the original received format as possible.
Closes: #64 ("Incorrect submitter when using googlegroups") Reported-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Reported-by: Stephen Rothwell <sfr@canb.auug.org.au> Signed-off-by: Andrew Donnellan <ajd@linux.ibm.com> Tested-by: Daniel Axtens <dja@axtens.net> # mailman only
[dja: add release note] Signed-off-by: Daniel Axtens <dja@axtens.net>
Jeremy Cline [Wed, 9 Oct 2019 19:03:45 +0000 (15:03 -0400)]
Use secrets and fall back to random.SystemRandom for keys
The random module uses the Mersenne Twister pseudorandom number
generator and is not a cryptographically secure random number
generator[0]. The secrets[1] module is intended for generating
cryptographically strong random numbers, so recommend using that to
generate the secret key. It's new in Python 3, so if it's unavailable
fall back to using the ``os.urandom()`` backed implementation of random.
NOTE(stephenfin): Modified to include change to 'config.yaml'. Also
renamed reno to just stick with hyphens for filenames.
Stephen Finucane [Tue, 15 Oct 2019 12:21:41 +0000 (13:21 +0100)]
docker: Rely on caching
It seems less likely that tox and tox-pyenv will change than our
requirements. Split up the 'RUN' steps so we don't have to reinstall the
former every time the latter change.
Signed-off-by: Stephen Finucane <stephen@that.guru>
The usual sub-resources (mbox, raw) hang off those URLs.
The old style URLs (/patch/NNN/*, /cover/NNN/*) redirect appropriately.
I haven't attempted to do anything meaningful with series, and I
have dropped any attempt to provide a generic message-id lookup
or search functionality. One step at a time.
Our database still stores message ids as with angle brackets; we
just work around that rather than trying to migrate. That too can
come later if we think the pain is justified.
Partially-closes: #106 Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Reported-by-but-I-don't-want-to-spam: Linus Torvalds <torvalds@linux-foundation.org> Reported-by: Stephen Finucane <stephen@that.guru> Signed-off-by: Daniel Axtens <dja@axtens.net>
Stephen Finucane [Sat, 21 Sep 2019 18:06:14 +0000 (19:06 +0100)]
Fix issue with delegation of patch via REST API
There have been reports of people being unable to delegate patches to
themselves, despite being a maintainer or the project to which the patch
is associated.
The issue is a result of how we do a check for whether the user is a
maintainer of the patch's project [1]. This check is checking if a given
'User.id' is in the list of items referenced by
'Project.maintainer_project'. However, 'Project.maintainer_project' is a
backref to 'UserProfile.maintainer_projects'. This means we're comparing
'User.id' and 'UserProfile.id'. Boo.
This wasn't seen in testing since we've had a post-save callback [2] for some
time that ensures we always create a 'UserProfile' object whenever we create a
'User' object. This also means we won't have an issue on deployments initially
deployed after that post-save callback was added, a 'User' with id=N will
always have a corresponding 'UserProfile' with id=N. However, that's not true
for older deployments such as the ozlabs.org one.
Daniel Axtens [Wed, 18 Sep 2019 06:17:28 +0000 (16:17 +1000)]
parsearchive, mail: use repr() to get a human readable exception
Currently if we have particular types of error in mail parsing
in parsearchive or parsemail, we print exc.message, which doesn't
always work:
Traceback (most recent call last):
File ".../patchwork/management/commands/parsearchive.py", line 90, in handle
obj = parse_mail(msg, options['list_id'])
File ".../patchwork/parser.py", line 961, in parse_mail
raise ValueError("Missing 'Message-Id' header")
ValueError: Missing 'Message-Id' header
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File ".../django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File ".../django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File ".../django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File ".../django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File ".../patchwork/management/commands/parsearchive.py", line 100, in handle
logger.warning('Invalid mail: %s', exc.message)
AttributeError: 'ValueError' object has no attribute 'message'
repr(exc) will work. Use it.
Signed-off-by: Daniel Axtens <dja@axtens.net> Reviewed-by: Stephen Finucane <stephen@that.guru>
In commit ab0c443691, we switched from using commit ranges to fixed
ranges. This was a good idea in so far as it ensures we're providing an
application with dependencies that are guaranteed to work. However,
Patchwork as a project isn't active enough to warrant the continued busy
work effort necessary to keep bumping these versions and it's probably
about time to abandon the experiment. However, rather than switching
back to version ranges, use the compatible releases feature introduced
in PEP 440 [1]. This gives us most of the benefits of ranges but with a
nicer syntax.
This is slightly slower to initially configure but requires less hacking
to get the same environment and should be a lot more maintainable (just
a simple modification to change the Python version).
Signed-off-by: Stephen Finucane <stephen@that.guru>
This is failing since the update to Xenial [1] with the following
warning:
$ sudo -u postgres psql -c "grant ALL on DATABASE postgres to travis WITH GRANT OPTION;"
could not change directory to "/home/travis/build/getpatchwork/patchwork": Permission denied
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5433"?
Bump the distro version to xenial, as this is the first version to
include Python 3.7 support. Bionic is also available but it doesn't
support Python 3.5 [1].
In addition, skip a test that was valid on Python 3.4 - 3.6 but does not
appear to be an issue for Python 3.7.
Ali Alnubani [Wed, 4 Sep 2019 13:40:31 +0000 (13:40 +0000)]
sql: Fix table lists
The patch adds missing commas in the table lists where missing, and
removes where unnecessary. This fixes errors such as the following when
feeding the script to psql:
psql:lib/sql/grant-all.postgres.sql:37: ERROR: syntax error at or near "patchwork_emailconfirmation"
LINE 19: patchwork_emailconfirmation,
...
Signed-off-by: Ali Alnubani <alialnu@mellanox.com> Reviewed-by: Stephen Finucane <stephen@that.guru> Fixes: ca0e79d4db34 ("sql: Sort 'grant-all' scripts alphabetically")
migrations: Correct 'unique_together' order in '0015'
This was resulting in exceptions like the following when used with MySQL
8.0:
Traceback (most recent call last):
File "../patchwork/manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
...
File "../.tox/py27-django111/lib/python2.7/site-packages/django/db/backends/mysql/schema.py", line 88, in _delete_composed_index
return super(DatabaseSchemaEditor, self)._delete_composed_index(model, fields, *args)
File "../.tox/py27-django111/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 394, in _delete_composed_index
", ".join(columns),
ValueError: Found wrong number (0) of constraints for patchwork_seriespatch(series_id, number)
This error was being raised by the following lines in the 0033
migration:
It appears that this is because of a mismatch between the order of
fields in a 'unique_together' constraint [1]. Correct the order in the
original migration and see the issue disappear.
Signed-off-by: Stephen Finucane <stephen@that.guru> Fixes: d67d859f40f ("models: Add 'Series' model")
As was designed, starting the interpreter would cause the State model
and its entries to be evaluated. This was an issue if, for example, the
model had been modified and you were attempting to apply the migration.
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
...
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 1199, in _set_queryset
self.widget.choices = self.choices
File "/home/patchwork/patchwork/patchwork/forms.py", line 157, in _get_choices
super(OptionalModelChoiceField, self)._get_choices())
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 1143, in __len__
return (len(self.queryset) + (1 if self.field.empty_label is not None else 0))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 232, in __len__
self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
raise original_exception
django.db.utils.OperationalError: (1054, "Unknown column 'patchwork_state.slug' in 'field list'")
Resolve this by moving the evaluation into '__init__', meaning it will
only occur when a new form is created.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Michael Ellerman [Tue, 27 Aug 2019 06:13:13 +0000 (16:13 +1000)]
models: Add commit_url_format to Project
Add a new field to Project, commit_url_format, which specifies a
format string that can be used to generate a link to a particular
commit for a project.
This is used in the display of a patch, to render the patch's commit
as a clickable link back to the commit on the SCM website.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Daniel Axtens <dja@axtens.net>
Daniel Axtens [Wed, 21 Aug 2019 05:15:44 +0000 (15:15 +1000)]
mbox: do not copy Content-Type into exported mbox
Daniel reports a patch + comment combination that breaks in
git am. The patch reports a Content-Type of US-ASCII, while
the comment adds a Ack with UTF-8 characters. The exported
mbox contains both the original Content-Type, and a UTF-8
Content-Type that we set. However, because the US-ASCII one
occurs later, git am honours it instead of ours, and chokes
on the UTF-8 characters.
Strip out any subsequent Content-Type:s. We normalise things
to UTF-8 and should not allow it to be overridden.
Add a test for this, based on the original report.
Reported-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Axtens <dja@axtens.net>
docker: *Actually* don't require rebuilding if unnecessary
Because we were using 'set -e', we were erroring out as soon as
something - a diff in this case - failed. Temporarily disable it for
this one check.
Signed-off-by: Stephen Finucane <stephen@that.guru> Fixes: 0b5b4e8c ("docker: Don't require rebuilding if unnecessary") Cc: Daniel Axtens <dja@axtens.net> Signed-off-by: Daniel Axtens <dja@axtens.net>
projects / thirdparty / patchwork.git / log
