As a result, one should not encourage people to develop and/or deploy
against these versions - the latest version of Django supported by
patchwork (currently 1.7) should be used for both.
In addition, rather than duplicating shared requirements for 'dev' and
'prod' environments, move all shared requirements to a "base" file.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Resolve remaining Django 1.7 warnings with the exception of a single
deprecation warning. However, this remaining issue is really a
non-issue due to the limited (and non-offending) use of the code.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Passing 'mimetype' to 'HttpResponse' is deprecated in 1.6 and
removed in Django 1.7. Among other things, this causes some unit
tests to fail when using Django 1.7. Its replacement - 'content_type'
- is available in Django 1.5+. This can be seen here:
The 'AUTH_PROFILE_MODULE' setting, and the 'get_profile()' method on
the 'User' model are removed in Django 1.7. This causes errors when
using Patchwork with Django 1.7+.
There are three changes necessary:
* Replace profile model's 'ForeignKey' with a 'OneToOneField'
* Remove all 'get_profile()' calls
* Delete 'AUTH_PROFILE_MODULE' settings from 'settings.py'
Django 1.6 also introduces two other notable changes:
* The 'XViewMiddleware' module has been moved
* A new test runner has been introduced
It is not possible to fix these issues without breaking compatibility
with Django 1.5. As a result they have been ignored and must be
resolved in a future release.
Signed-off-by: Stephen Finucane <stephenfinucane@hotmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Michael Ellerman [Mon, 23 Mar 2015 06:56:22 +0000 (17:56 +1100)]
Make the submitter name link to a query for that submitter
Currently the submitter name is rendered as a mailto: link. This is
possibly useful in some circumstances, but in my experience is not
usually what I want. Although it opens a mail to the submitter, it
doesn't include any of the patch context, so is not very helpful.
Instead the submitter link can be a link to a query for patches by that
submitter. In my experience that is more useful, ie. when looking at a
single patch for a submitter you can then quickly get the list of all
patches by them.
So do that conversion.
In order to do it we need to know the current project, so that becomes a
parameter to personify. I believe the url reversal is correct, though
it's not pretty, and pulling SubmitterFilter.param out feels a little
wrong, but is the best solution I could come up with.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Michael Ellerman [Wed, 18 Mar 2015 03:39:24 +0000 (14:39 +1100)]
parser: Fix parsing of patches with a trailing no-newline marker
If a patch ends with a "No newline at end of file" marker, it is
incorrectly considered part of the comment.
Add a testcase which shows the bug, and then fix the parser. The parser
fix is hopefully sufficiently specific so as to not break any other
unrelated case. But ..
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Damien Lespiau [Sat, 15 Nov 2014 01:58:25 +0000 (01:58 +0000)]
Always use #!/usr/bin/env python
Hardcoding the path to the python binary breaks virtualenv. virtualenv
creates a special python binary in the path, and specifically using
/usr/bin/python breaks that.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Sun, 23 Nov 2014 05:49:53 +0000 (21:49 -0800)]
pwclient: support 'archived' filtering and updating
Examples:
# Mark patch as uperseded and archived
pwclient update -s Superseded -a yes <ID>
# List all archived patches
pwclient list -a yes
Notably, we still leave the '-s' option as required for 'pwclient
update'; so you can't *just* archive a patch without setting its state.
I couldn't quite figure out the right argparse usage to represent that
the user must include one or both of '-s' and '-a'.
And of course, the server must have an updated xmlrpc that supports the
'archived' field for list filtering (recently patched), otherwise you'll
just get an empty list.
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Sun, 23 Nov 2014 04:38:46 +0000 (20:38 -0800)]
pwclient: exit on first patch which fails to apply
When run with more than one patch ID, the 'apply' and 'git-am' commands
should not continue to process other patches if an earlier one failed.
We should stop so the user can address the situation.
Future work: it'd be nice to just pipe all the patches at once to
git-am, so that git's nice handling of fixup-and-continue workflow can
be used.
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Sun, 23 Nov 2014 04:38:45 +0000 (20:38 -0800)]
pwclient: add project option to single-patch commands
Previously, we could not target a particular server instance with the
apply, git-am, info, and view subcommands. Under a single-server design,
we never needed to target a particular project for a patch. We just
targeted the patch itself. But with the advent of multi-server
.pwclientrc configurations, we should allow pwclient to specify the
project for these commands.
This adds the '-p PROJECT' option to:
* apply
* git-am
* info
* view
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Sun, 23 Nov 2014 04:38:42 +0000 (20:38 -0800)]
xmlrpc: include UNIX mbox 'From ' header in patch_get_mbox()
This function is misleadingly named 'mbox'; the contents do not begin
with a proper 'From xxx <date>' separator line. (Notably, the mbox
format does not have an authoritative standard, but at least this basic
'separator' construct is noted in http://tools.ietf.org/html/rfc4155.)
The Message.as_string() function takes an optional [unixfrom] boolean
argument. Let's use it, like we do everywhere else (including in the web
interface 'mbox' link).
Among other things, this means that we can straightforwardly concatenate
the output of patch_get_mbox(), and more tools can use this output
as-is.
Example header:
>From patchwork Fri Nov 21 18:24:29 2014
...
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Johannes Berg [Tue, 6 Jan 2015 10:21:37 +0000 (11:21 +0100)]
views/xmlrpc: fix xmlrpc delegate filtering
Trying to use
pwclient list -d 'johannes@sipsolutions.net'
doesn't result in any patches listed - it seems that the filter is
constructed wrongly on the xmlrpc server side (going by how the
submitter filter is done.)
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Damien Lespiau [Sat, 25 Oct 2014 21:17:22 +0000 (22:17 +0100)]
parsemail: Honour existing PYTHONPATH
For setups where the patchwork dependencies are not part of the system
packages, one needs to indicate where to find them. parsemail.sh was
overriding PYTHONPATH without leaving us a chance to join in.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jani Nikula [Tue, 21 Oct 2014 08:18:53 +0000 (11:18 +0300)]
pwclient: add output format option to list/search command
Make scripting easier by letting the user specify the output format. The
format string may contain tag references to fields, such as %{id} and
%{msgid}.
Signed-off-by: Jani Nikula <jani.nikula@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
HACKING: Add some documentation about using virtualenv with patchwork
virtualenv is super nice to isolate devevelopment from the python
packages installed in the sytem. It also allows to have parallel
configurations to run tests againsts.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Damien Lespiau [Sun, 31 Aug 2014 00:02:37 +0000 (01:02 +0100)]
INSTALL: Sprinkle a few UTF-8 in the mysql documentation
When not specifying the charset/collation, I managed to create a latin1
database where all strings were encoded in latin1. That's really not
ideal. Adding 'CHARACTER SET utf8' when creating the DB fixes it. Then:
$ ./manage.py syncdb
will correctly create tables with UTF-8 encoded rows.
However, for some reason, when django creates the test tables, that
default, DB wide, encoding is not respected and one needs to provide an
additional TEST_CHARSET entry in the config dictionary.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Damien Lespiau [Sun, 31 Aug 2014 00:02:36 +0000 (01:02 +0100)]
INSTALL: Update the database configuration instructions
That's the "new" (django 1.5+) way of defining databases, strictly
following the instructions wasn't working. This should save the next
user a bit of time.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Sun, 7 Sep 2014 12:05:18 +0000 (20:05 +0800)]
pwclient: preserve ordering of patch ids
Currently, we use a frozenset to remove duplicates in the patch id
list. However, this means we lose the patch ordering, which is important
for a git-am.
This change drops the frozenset, so we preserve the ordering from the
arguments.
Allow commands that take an ID to operate on multiple IDs.
E.g.:
update -s Superseded 1 2 3 4 5
apply 2 4 6
Reject update -c COMMIT-REF on multiple IDs though as that does not
make sense.
Implementation note:
nargs='*' instead of '?' results in (wrong/inconvenient):
mutually exclusive arguments must be optional
So remove mutual exclusive handling via argparse and instead do it by
hand. This might be implemented more conveniently in later python but we
(have to) stick with 2.7.x for the time being.
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Thu, 12 Jun 2014 01:54:59 +0000 (18:54 -0700)]
post-receive: exclude commits from the patch update step
When merging upstream work related to other projects into your own
project repository, you probably don't want to check for (and try to
update) the status on every change-set in the merge. So add a list of
references (branches, tags, commits, etc.) whose commits should be
ignored in the patch update step.
This could be used, for example, to set:
EXCLUDE="refs/heads/upstream"
Then when you're ready to merge in new upstream code, you first update
the 'upstream' branch before pushing your own.
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
pwclient: accept more than one project in ~/.pwclientrc
Currently, given the format of ~/.pwclientrc, pwclient can only
really act on a single project, as ~/pwclientrc can only contain
the configuration for a single project.
Although the -p options comes in handy to specify a project
different from the one configured in ~/.pwclientrc, this only works
if it is hosted on the same server. As soon as one needs to switch
server, it is necessary to edit ~/pwclientrc.
This can be quite inefficient when dealing with many projects, hosted
on different servers.
Change the format of ~/.pwclientrc so it is possible to define more
than one project, and for each project, specify an URL and credentials.
This has the advantage of not changing the options to pwclient, so
the user experience is unmodified.
If a ~/.pwclentrc exists in the old format, it is automatically
converted over to the new format, and the previous one is saved as
~/.pwclientrc.orig. Upon conversion, no action is made, pwclient just
exits (with return-code 1) to inform the user to review the conversion.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Thu, 8 May 2014 05:53:09 +0000 (13:53 +0800)]
filters: fix exception in filter querystring generation
We get a silent (as it's during template render) exception when
generating filter querystrings, as we're passing a list to the string
format operator rather than a tuple.
This change removes the map and explicitly applies sanitise to the
(name, value) pair.
Jeremy Kerr [Wed, 7 May 2014 12:24:17 +0000 (20:24 +0800)]
views/generic_list: reduce number of queries in list rendering
Currently, we do two database queries per patch on a list view; one to
retrieve the submitter, and one for the state.
This patch adds a .select_related to fetch for the submitter and state,
and a .defer() to prevent loading large amounts of text data from the
patch content and headers.
This gives a significant reduction in the work per request. For a
paginated list view (ie 100 patches per page):
before after
User 1344 ms 228 ms
System 170 ms 25 ms
Total 1514 ms 253 ms
Elapsed 1605 ms 274 ms
Jeremy Kerr [Wed, 7 May 2014 02:03:44 +0000 (10:03 +0800)]
tests: Add XMLRPC interface test
This change adds a simple test for the XMLRPC interface, essentially to
ensure that configuration changes haven't broken accessibility to the
interface.
Jeremy Kerr [Tue, 22 Apr 2014 13:19:52 +0000 (21:19 +0800)]
Fix django-1.6 incompatibilities
We're seeing a couple of final quirks running the testsuite on django
1.6:
Traceback (most recent call last):
File "patchwork/apps/patchwork/tests/notifications.py", line 182, in testNotificationEscaping
errors = send_notifications()
File "patchwork/apps/patchwork/utils.py", line 227, in send_notifications
delete_notifications()
File "patchwork/apps/patchwork/utils.py", line 197, in delete_notifications
pk__in = notifications).delete()
File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 163, in filter
return self.get_queryset().filter(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 590, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 608, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1198, in add_q
clause = self._add_q(where_part, used_aliases)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1232, in _add_q
current_negated=current_negated)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1122, in build_filter
lookup_type, value)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 1107, in get_lookup_constraint
values = [get_normalized_value(value) for value in raw_value]
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 1084, in get_normalized_value
value_list.append(getattr(value, source.attname))
AttributeError: 'PatchChangeNotification' object has no attribute 'id'
- we're specifying our own pk here, so the PatchChangeNotification has
no id attribute; it looks like the pk__in syntax is expecting IDs.
We also need a default value for BooleanField, as we're getting
integrity errors when creating rows with no explicit send_notifications
set.
Jeremy Kerr [Tue, 22 Apr 2014 12:48:19 +0000 (20:48 +0800)]
Defer Person creation/linkage until registration is confirmed
We currently create Person objects when a registration is submitted, not
when it is confirmed. This can lead to stale Person objects for
unconfirmed registrations.
Jeremy Kerr [Tue, 22 Apr 2014 12:08:27 +0000 (20:08 +0800)]
Add unconfirmed registration expiry
Currently, unconfirmed registrations remain in the database. Although we
have an expiry for the registrations, we don't actually remove rows from
the database. This can clog the admin interface up with unnecessary
registration spam.
We currently have a patchwork cron script to send notifications on patch
changes, so hook this into a new do_expiry function.
Carl Worth [Fri, 13 Dec 2013 04:28:02 +0000 (20:28 -0800)]
post-receive: Fix error message when pushing to an unmapped branch
Previously, if STATE_MAP contained only refs/heads/master and a user
pushed to a branch such as refs/heads/stable, the hook emitted a
misleading error message:
E: no mapping for refname refs/heads/master
Fix this by correctly printing the reference being looked for, rather
than the reference most recently found in STATE_MAP. Also, reword the
message slightly to point the user to STATE_MAP and to use an actual
word "branch" instead of the non-word "refname".
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:28 +0000 (10:46 -0800)]
docs: Add a pointer to the git post-receive hook
I had to stumble upon this feature before I realized it was here
inside the tools directory all along. This documentation should make
it easier for the next person coming along looking for a patchwork git
hook.
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:27 +0000 (10:46 -0800)]
Make the post-receive hook more efficient (using pwclient info, not view)
Older versions of patchwork shipped with a version of pwclient that
had the view command, but not info command. Now that info exists, it's
inefficient to use it to obtain the patchwork ID value, (since it
requires the server to send the entire patch content in response to an
XML RPC request).
So be kind to the server by using info instead, (which has a small,
fixed-length response).
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:26 +0000 (10:46 -0800)]
tools/post-receive.hook: don't update the previously pushed commit
Previously, the post-receive hook would always examine one commit that
had been previously pushed, (when the purpose of the hook is only to
example newly-pushed commits). We fix this by simply dropping the '^'
in the commit-range specification.
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
This patch fixes following error when redirecting 'pwclient info' output:
---8<---
andreas@dockstar % pwclient info 295268 > /tmp/test.out
Traceback (most recent call last):
File "/home/andreas/bin/pwclient", line 508, in <module>
main()
File "/home/andreas/bin/pwclient", line 471, in main
action_info(rpc, patch_id)
File "/home/andreas/bin/pwclient", line 242, in action_info
print("- %- 14s: %s" % (key, value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 29: ordinal not in range(128)
--->8---
Signed-off-by: Andreas Bießmann <andreas@biessmann.de> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
In situations where SSL is terminated at the load-balancer, we cannot
rely on guessing the scheme based on whether patchwork itself was
accessed via http or https, since the last-leg is always going to be
done over http.
Unfortunately, wrongly using http:// URLs results in unusable
.pwclientrc files, since xmlrpc does not handle http->https redirects
and instead displays a traceback.
This change introduces a FORCE_HTTPS_LINKS option, which forces
pwclientrc links to always return "https" regardless of how the project
itself is accessed.
It appears that the http/https check is currently only used for
generating pwclientrc -- a lot of other places seem to hardcode
"http://" and rely on the server to transparently upgrade the
connection. This is not a secure approach (it allows for MITM and
SSL-Strip attacks) and therefore all places currently hardcoding
http://{{site.domain}} and similar should be switched to using the
"sheme" variable, the same as done for generating pwclientrc files.
lib/sql/migration: Fix column names for MySQL/MariaDB
The first issue is that patchwork was no longer accepting new patches via
the apps/patchwork/bin/parsemail.sh script. When I was trying to invoke
it manually, it only printed "no project found". I was able to figure
out that this was caused by changes to the database scheme which are taken
care of by the two SQL scripts:
The catch - these first script don't run with MySQL - or rather MariaDB that
Fedora has switched to. MariaDB doesn't like the quotes around the table
and column names. Patch below.
This change fixes the issue by adding a new state for patch 'meta'
headers: header test which is not parsed as a hunk, but has the same
behaviour as a hunk in that it will cause the patchbuf to be populated.
Jeremy Kerr [Sun, 13 Oct 2013 04:48:21 +0000 (12:48 +0800)]
views/base: Include email addresses in submitter autocomplete search
Currently, we only search names, which is a problem for Person objects
with only an email address set. This change includes the email addresses
in the search.
docs: also mention that ROOT_DIR needs to be updated
When the patchwork installation resides in the non-default location
(/srv/patchwork) make sure that local_settings.py also gets updated with
a proper ROOT_DIR value otherwise various problems will happen.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Django is no longer accessible using the old Subversion URL, update that
to the Github repository. Besides, the command-line used to perform the
"syncdb" operation does not work because we are off by one directory in
the filesystem, fix that.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
The total_seconds function was added to datetime in python-2.7. For
compatibility with previous versions of python, use its suggested
equivalent (except drop microseconds, since we don't care about them in
this context).
Signed-off-by: Konstantin Ryabitsev <mricon@kernel.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Mon, 10 Jun 2013 03:50:21 +0000 (11:50 +0800)]
views/mbox: Use Date: header from original message
Since we use UTC for internal date storage, we lose the timestamp info
from the original patch submissions. This means that the mbox views
(which are typically passed through to SCM commit logs) lose the
timezone information.
This change uses the Date header from the original message, if possible.
projects / patchwork / log