Stephen Finucane [Fri, 21 Aug 2015 14:32:07 +0000 (15:32 +0100)]
docs: Rewrite documentation
The INSTALL and HACKING documents are an important guide for new
patchwork users and developers and should be as informative as
possible. A number of changes were needed to these documents owing
to the out-of-date or incomplete information they contained. These
changes include:
* Removing references to the dead mod_python/flup projects
* Adding references to Gunicorn+nginx, which a credible modern
alternative to Apache+mod_wsgi
* Providing explanatory links to concepts/tools like ident-based
authentication and tox
* Referencing the newer tools available to developers, like tox
and the 'requirements.txt' files
* Integration with mkdocs, with eye on eventual publishing of
documentation to ReadTheDocs or equivalent.
These changes result in a significant rewrite which should hopefully
lower the barrier to entry for people wishing to use or develop
patchwork.
Stephen Finucane [Fri, 21 Aug 2015 14:32:06 +0000 (15:32 +0100)]
tox: Update versions of Django to be tested
Since patchwork now supports Django 1.8 and no longer support 1.5, it
is necessary to update the tox targets to validate this new subset of
supported Django versions.
In addition, the paths to the requirements.txt files were not updated
as part of a rebase. Correct this oversight.
Stephen Finucane [Fri, 21 Aug 2015 14:32:05 +0000 (15:32 +0100)]
Add missing explicit state fixture load
The 'patchwork: Explicitly load states fixtures' change introduced
explicit state loading into many of the tests, but missed one.
Correct this oversight.
Jeremy Kerr [Fri, 5 Jun 2015 06:53:07 +0000 (14:53 +0800)]
patchwork/templates: Suppress cycle template tag warning
Running on django 1.7 gives a warning:
RemovedInDjango18Warning: 'The `cycle` template tag is changing to escape its arguments; the non-autoescaping version is deprecated. Load it from the `future` tag library to start using the new behavior.
Although this change doesn't affect this template, suppress the warning
by loading from the future library.
Jeremy Kerr [Fri, 5 Jun 2015 06:33:13 +0000 (14:33 +0800)]
patchwork/parser: Adapt for new unsaved-foreign-key behaviour in django 1.8
Django 1.8 no longer supports assignment of unsaved models to
ForeignKey fields:
File "/home/jk/devel/patchwork/patchwork/tests/test_patchparser.py", line 75, in setUp
(self.patch, self.comment) = find_content(self.project, email)
File "/home/jk/devel/patchwork/patchwork/bin/parsemail.py", line 231, in find_content
headers = mail_headers(mail))
File "/home/jk/devel/patchwork/lib/python/django/db/models/base.py", line 468, in __init__
setattr(self, field.name, rel_obj)
File "/home/jk/devel/patchwork/lib/python/django/db/models/fields/related.py", line 668, in __set__
(value, self.field.rel.to._meta.object_name)
ValueError: Cannot assign "<Patch: Test Subject>": "Patch" instance isn't saved in the database.
Even though we'd be guaranteed to save the patch before the comment, we
need to avoid this error.
This change defers the assigment of Comment.patch until we know we have
a saved Patch instance.
Jeremy Kerr [Thu, 4 Jun 2015 10:13:13 +0000 (18:13 +0800)]
patchwork: Add initial migrations
Add migrations for the patchwork app; these will be required for
django-1.9, and are required to express the test dependencies (to the
auth models) on 1.8.
Jeremy Kerr [Fri, 5 Jun 2015 06:32:26 +0000 (14:32 +0800)]
patchwork: Explicitly load states fixtures
Apps with migrations will no longer load the initial_data fixtures by
default. In order to prepare to add migrations to patchwork, rename the
initial_data fixture to default_states (to match the default_tags
fixture), and explicitly load them in tests that require them.
Jeremy Kerr [Thu, 4 Jun 2015 09:58:51 +0000 (17:58 +0800)]
patchwork/views: prefetch delegate in generic_list
If patches have a delegate, the patch list template will cause a query
to render the delegate username. Instead, we should include the delegate
in the select_related query.
Jeremy Kerr [Thu, 28 May 2015 05:44:14 +0000 (13:44 +0800)]
tests: Remove old-style test runner module
We get the following warning on django 1.7:
System check identified some issues:
WARNINGS:
?: (1_6.W001) Some project unittests may not execute as expected.
HINT: Django 1.6 introduced a new default test runner. It looks like this project was generated using Django 1.5 or earlier. You should ensure your tests are all running & behaving as expected. See https://docs.djangoproject.com/en/dev/releases/1.6/#new-test-runner for more information.
This change removes the unneeded base test module, and moves the
patchparser doctests into a proper test module.
Jeremy Kerr [Wed, 27 May 2015 01:56:36 +0000 (09:56 +0800)]
Add patch tag infrastructure
This change add patch 'tags', eg 'Acked-by' / 'Reviewed-by', etc., to
patchwork.
Tag parsing is implemented in the patch parser's extract_tags function,
which returns a Counter object of the tags in a comment. These are
stored in the PatchTag (keyed to Tag) objects associated with each
patch.
We need to ensure that the main patch lists do not cause per-patch
queries on the Patch.tags ManyToManyField (this would result in ~500
queries per page), so we introduce a new QuerySet (and Manager) for
Patch, adding a with_tag_counts() method to populate the tag counts in a
single query.
As users may be migrating from previous patchwork versions (ie, with no
tag counts in the database), we add a 'retag' management command.
Jeremy Kerr [Sun, 24 May 2015 09:50:33 +0000 (17:50 +0800)]
Update documentation and default settings to suit patchwork deployment model
We've always allowed configuration without altering any of the
version-controlled files. With the recent settings changes, we have an
extra level of indirection with the dev/prod settings modules.
Since we have to edit a config file anyway, this change moves the
prod.py settings file to a template, which is then used directly by
mange.py (and the wsgi application).
Jeremy Kerr [Sun, 24 May 2015 08:57:33 +0000 (16:57 +0800)]
Move to a more recent django project structure
This change updates patchwor to the newer project struture: we've moved
the actual application out of the apps/ directory, and the
patchwork-specific templates to under the patchwork application.
This gives us the manage.py script in the top-level now.
Currently this contains calls to execute the following on the code
base:
* Unit tests (for all currently supported versions of Django). This
requires the addition of a "test" 'local_settings' file
* PEP8 (or, rather, flake8)
* PyLint
* Coverage (based on unit tests)
These are designed in such a way that it should be possible to easily
add additional environment for testing (like Python3 or Django 1.8).
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
As a result, one should not encourage people to develop and/or deploy
against these versions - the latest version of Django supported by
patchwork (currently 1.7) should be used for both.
In addition, rather than duplicating shared requirements for 'dev' and
'prod' environments, move all shared requirements to a "base" file.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Resolve remaining Django 1.7 warnings with the exception of a single
deprecation warning. However, this remaining issue is really a
non-issue due to the limited (and non-offending) use of the code.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Passing 'mimetype' to 'HttpResponse' is deprecated in 1.6 and
removed in Django 1.7. Among other things, this causes some unit
tests to fail when using Django 1.7. Its replacement - 'content_type'
- is available in Django 1.5+. This can be seen here:
The 'AUTH_PROFILE_MODULE' setting, and the 'get_profile()' method on
the 'User' model are removed in Django 1.7. This causes errors when
using Patchwork with Django 1.7+.
There are three changes necessary:
* Replace profile model's 'ForeignKey' with a 'OneToOneField'
* Remove all 'get_profile()' calls
* Delete 'AUTH_PROFILE_MODULE' settings from 'settings.py'
Django 1.6 also introduces two other notable changes:
* The 'XViewMiddleware' module has been moved
* A new test runner has been introduced
It is not possible to fix these issues without breaking compatibility
with Django 1.5. As a result they have been ignored and must be
resolved in a future release.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Michael Ellerman [Mon, 23 Mar 2015 06:56:22 +0000 (17:56 +1100)]
Make the submitter name link to a query for that submitter
Currently the submitter name is rendered as a mailto: link. This is
possibly useful in some circumstances, but in my experience is not
usually what I want. Although it opens a mail to the submitter, it
doesn't include any of the patch context, so is not very helpful.
Instead the submitter link can be a link to a query for patches by that
submitter. In my experience that is more useful, ie. when looking at a
single patch for a submitter you can then quickly get the list of all
patches by them.
So do that conversion.
In order to do it we need to know the current project, so that becomes a
parameter to personify. I believe the url reversal is correct, though
it's not pretty, and pulling SubmitterFilter.param out feels a little
wrong, but is the best solution I could come up with.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Michael Ellerman [Wed, 18 Mar 2015 03:39:24 +0000 (14:39 +1100)]
parser: Fix parsing of patches with a trailing no-newline marker
If a patch ends with a "No newline at end of file" marker, it is
incorrectly considered part of the comment.
Add a testcase which shows the bug, and then fix the parser. The parser
fix is hopefully sufficiently specific so as to not break any other
unrelated case. But ..
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Damien Lespiau [Sat, 15 Nov 2014 01:58:25 +0000 (01:58 +0000)]
Always use #!/usr/bin/env python
Hardcoding the path to the python binary breaks virtualenv. virtualenv
creates a special python binary in the path, and specifically using
/usr/bin/python breaks that.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Sun, 23 Nov 2014 05:49:53 +0000 (21:49 -0800)]
pwclient: support 'archived' filtering and updating
Examples:
# Mark patch as uperseded and archived
pwclient update -s Superseded -a yes <ID>
# List all archived patches
pwclient list -a yes
Notably, we still leave the '-s' option as required for 'pwclient
update'; so you can't *just* archive a patch without setting its state.
I couldn't quite figure out the right argparse usage to represent that
the user must include one or both of '-s' and '-a'.
And of course, the server must have an updated xmlrpc that supports the
'archived' field for list filtering (recently patched), otherwise you'll
just get an empty list.
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Sun, 23 Nov 2014 04:38:46 +0000 (20:38 -0800)]
pwclient: exit on first patch which fails to apply
When run with more than one patch ID, the 'apply' and 'git-am' commands
should not continue to process other patches if an earlier one failed.
We should stop so the user can address the situation.
Future work: it'd be nice to just pipe all the patches at once to
git-am, so that git's nice handling of fixup-and-continue workflow can
be used.
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Sun, 23 Nov 2014 04:38:45 +0000 (20:38 -0800)]
pwclient: add project option to single-patch commands
Previously, we could not target a particular server instance with the
apply, git-am, info, and view subcommands. Under a single-server design,
we never needed to target a particular project for a patch. We just
targeted the patch itself. But with the advent of multi-server
.pwclientrc configurations, we should allow pwclient to specify the
project for these commands.
This adds the '-p PROJECT' option to:
* apply
* git-am
* info
* view
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Sun, 23 Nov 2014 04:38:42 +0000 (20:38 -0800)]
xmlrpc: include UNIX mbox 'From ' header in patch_get_mbox()
This function is misleadingly named 'mbox'; the contents do not begin
with a proper 'From xxx <date>' separator line. (Notably, the mbox
format does not have an authoritative standard, but at least this basic
'separator' construct is noted in http://tools.ietf.org/html/rfc4155.)
The Message.as_string() function takes an optional [unixfrom] boolean
argument. Let's use it, like we do everywhere else (including in the web
interface 'mbox' link).
Among other things, this means that we can straightforwardly concatenate
the output of patch_get_mbox(), and more tools can use this output
as-is.
Example header:
>From patchwork Fri Nov 21 18:24:29 2014
...
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Johannes Berg [Tue, 6 Jan 2015 10:21:37 +0000 (11:21 +0100)]
views/xmlrpc: fix xmlrpc delegate filtering
Trying to use
pwclient list -d 'johannes@sipsolutions.net'
doesn't result in any patches listed - it seems that the filter is
constructed wrongly on the xmlrpc server side (going by how the
submitter filter is done.)
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Damien Lespiau [Sat, 25 Oct 2014 21:17:22 +0000 (22:17 +0100)]
parsemail: Honour existing PYTHONPATH
For setups where the patchwork dependencies are not part of the system
packages, one needs to indicate where to find them. parsemail.sh was
overriding PYTHONPATH without leaving us a chance to join in.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jani Nikula [Tue, 21 Oct 2014 08:18:53 +0000 (11:18 +0300)]
pwclient: add output format option to list/search command
Make scripting easier by letting the user specify the output format. The
format string may contain tag references to fields, such as %{id} and
%{msgid}.
Signed-off-by: Jani Nikula <jani.nikula@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
HACKING: Add some documentation about using virtualenv with patchwork
virtualenv is super nice to isolate devevelopment from the python
packages installed in the sytem. It also allows to have parallel
configurations to run tests againsts.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Damien Lespiau [Sun, 31 Aug 2014 00:02:37 +0000 (01:02 +0100)]
INSTALL: Sprinkle a few UTF-8 in the mysql documentation
When not specifying the charset/collation, I managed to create a latin1
database where all strings were encoded in latin1. That's really not
ideal. Adding 'CHARACTER SET utf8' when creating the DB fixes it. Then:
$ ./manage.py syncdb
will correctly create tables with UTF-8 encoded rows.
However, for some reason, when django creates the test tables, that
default, DB wide, encoding is not respected and one needs to provide an
additional TEST_CHARSET entry in the config dictionary.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Damien Lespiau [Sun, 31 Aug 2014 00:02:36 +0000 (01:02 +0100)]
INSTALL: Update the database configuration instructions
That's the "new" (django 1.5+) way of defining databases, strictly
following the instructions wasn't working. This should save the next
user a bit of time.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Sun, 7 Sep 2014 12:05:18 +0000 (20:05 +0800)]
pwclient: preserve ordering of patch ids
Currently, we use a frozenset to remove duplicates in the patch id
list. However, this means we lose the patch ordering, which is important
for a git-am.
This change drops the frozenset, so we preserve the ordering from the
arguments.
Allow commands that take an ID to operate on multiple IDs.
E.g.:
update -s Superseded 1 2 3 4 5
apply 2 4 6
Reject update -c COMMIT-REF on multiple IDs though as that does not
make sense.
Implementation note:
nargs='*' instead of '?' results in (wrong/inconvenient):
mutually exclusive arguments must be optional
So remove mutual exclusive handling via argparse and instead do it by
hand. This might be implemented more conveniently in later python but we
(have to) stick with 2.7.x for the time being.
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Thu, 12 Jun 2014 01:54:59 +0000 (18:54 -0700)]
post-receive: exclude commits from the patch update step
When merging upstream work related to other projects into your own
project repository, you probably don't want to check for (and try to
update) the status on every change-set in the merge. So add a list of
references (branches, tags, commits, etc.) whose commits should be
ignored in the patch update step.
This could be used, for example, to set:
EXCLUDE="refs/heads/upstream"
Then when you're ready to merge in new upstream code, you first update
the 'upstream' branch before pushing your own.
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
pwclient: accept more than one project in ~/.pwclientrc
Currently, given the format of ~/.pwclientrc, pwclient can only
really act on a single project, as ~/pwclientrc can only contain
the configuration for a single project.
Although the -p options comes in handy to specify a project
different from the one configured in ~/.pwclientrc, this only works
if it is hosted on the same server. As soon as one needs to switch
server, it is necessary to edit ~/pwclientrc.
This can be quite inefficient when dealing with many projects, hosted
on different servers.
Change the format of ~/.pwclientrc so it is possible to define more
than one project, and for each project, specify an URL and credentials.
This has the advantage of not changing the options to pwclient, so
the user experience is unmodified.
If a ~/.pwclentrc exists in the old format, it is automatically
converted over to the new format, and the previous one is saved as
~/.pwclientrc.orig. Upon conversion, no action is made, pwclient just
exits (with return-code 1) to inform the user to review the conversion.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Thu, 8 May 2014 05:53:09 +0000 (13:53 +0800)]
filters: fix exception in filter querystring generation
We get a silent (as it's during template render) exception when
generating filter querystrings, as we're passing a list to the string
format operator rather than a tuple.
This change removes the map and explicitly applies sanitise to the
(name, value) pair.
Jeremy Kerr [Wed, 7 May 2014 12:24:17 +0000 (20:24 +0800)]
views/generic_list: reduce number of queries in list rendering
Currently, we do two database queries per patch on a list view; one to
retrieve the submitter, and one for the state.
This patch adds a .select_related to fetch for the submitter and state,
and a .defer() to prevent loading large amounts of text data from the
patch content and headers.
This gives a significant reduction in the work per request. For a
paginated list view (ie 100 patches per page):
before after
User 1344 ms 228 ms
System 170 ms 25 ms
Total 1514 ms 253 ms
Elapsed 1605 ms 274 ms
Jeremy Kerr [Wed, 7 May 2014 02:03:44 +0000 (10:03 +0800)]
tests: Add XMLRPC interface test
This change adds a simple test for the XMLRPC interface, essentially to
ensure that configuration changes haven't broken accessibility to the
interface.
Jeremy Kerr [Tue, 22 Apr 2014 13:19:52 +0000 (21:19 +0800)]
Fix django-1.6 incompatibilities
We're seeing a couple of final quirks running the testsuite on django
1.6:
Traceback (most recent call last):
File "patchwork/apps/patchwork/tests/notifications.py", line 182, in testNotificationEscaping
errors = send_notifications()
File "patchwork/apps/patchwork/utils.py", line 227, in send_notifications
delete_notifications()
File "patchwork/apps/patchwork/utils.py", line 197, in delete_notifications
pk__in = notifications).delete()
File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 163, in filter
return self.get_queryset().filter(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 590, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 608, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1198, in add_q
clause = self._add_q(where_part, used_aliases)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1232, in _add_q
current_negated=current_negated)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1122, in build_filter
lookup_type, value)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 1107, in get_lookup_constraint
values = [get_normalized_value(value) for value in raw_value]
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 1084, in get_normalized_value
value_list.append(getattr(value, source.attname))
AttributeError: 'PatchChangeNotification' object has no attribute 'id'
- we're specifying our own pk here, so the PatchChangeNotification has
no id attribute; it looks like the pk__in syntax is expecting IDs.
We also need a default value for BooleanField, as we're getting
integrity errors when creating rows with no explicit send_notifications
set.
projects / patchwork / log