Jeremy Kerr [Wed, 7 May 2014 02:03:44 +0000 (10:03 +0800)]
tests: Add XMLRPC interface test
This change adds a simple test for the XMLRPC interface, essentially to
ensure that configuration changes haven't broken accessibility to the
interface.
Jeremy Kerr [Tue, 22 Apr 2014 13:19:52 +0000 (21:19 +0800)]
Fix django-1.6 incompatibilities
We're seeing a couple of final quirks running the testsuite on django
1.6:
Traceback (most recent call last):
File "patchwork/apps/patchwork/tests/notifications.py", line 182, in testNotificationEscaping
errors = send_notifications()
File "patchwork/apps/patchwork/utils.py", line 227, in send_notifications
delete_notifications()
File "patchwork/apps/patchwork/utils.py", line 197, in delete_notifications
pk__in = notifications).delete()
File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 163, in filter
return self.get_queryset().filter(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 590, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 608, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1198, in add_q
clause = self._add_q(where_part, used_aliases)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1232, in _add_q
current_negated=current_negated)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1122, in build_filter
lookup_type, value)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 1107, in get_lookup_constraint
values = [get_normalized_value(value) for value in raw_value]
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 1084, in get_normalized_value
value_list.append(getattr(value, source.attname))
AttributeError: 'PatchChangeNotification' object has no attribute 'id'
- we're specifying our own pk here, so the PatchChangeNotification has
no id attribute; it looks like the pk__in syntax is expecting IDs.
We also need a default value for BooleanField, as we're getting
integrity errors when creating rows with no explicit send_notifications
set.
Jeremy Kerr [Tue, 22 Apr 2014 12:48:19 +0000 (20:48 +0800)]
Defer Person creation/linkage until registration is confirmed
We currently create Person objects when a registration is submitted, not
when it is confirmed. This can lead to stale Person objects for
unconfirmed registrations.
Jeremy Kerr [Tue, 22 Apr 2014 12:08:27 +0000 (20:08 +0800)]
Add unconfirmed registration expiry
Currently, unconfirmed registrations remain in the database. Although we
have an expiry for the registrations, we don't actually remove rows from
the database. This can clog the admin interface up with unnecessary
registration spam.
We currently have a patchwork cron script to send notifications on patch
changes, so hook this into a new do_expiry function.
Carl Worth [Fri, 13 Dec 2013 04:28:02 +0000 (20:28 -0800)]
post-receive: Fix error message when pushing to an unmapped branch
Previously, if STATE_MAP contained only refs/heads/master and a user
pushed to a branch such as refs/heads/stable, the hook emitted a
misleading error message:
E: no mapping for refname refs/heads/master
Fix this by correctly printing the reference being looked for, rather
than the reference most recently found in STATE_MAP. Also, reword the
message slightly to point the user to STATE_MAP and to use an actual
word "branch" instead of the non-word "refname".
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:28 +0000 (10:46 -0800)]
docs: Add a pointer to the git post-receive hook
I had to stumble upon this feature before I realized it was here
inside the tools directory all along. This documentation should make
it easier for the next person coming along looking for a patchwork git
hook.
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:27 +0000 (10:46 -0800)]
Make the post-receive hook more efficient (using pwclient info, not view)
Older versions of patchwork shipped with a version of pwclient that
had the view command, but not info command. Now that info exists, it's
inefficient to use it to obtain the patchwork ID value, (since it
requires the server to send the entire patch content in response to an
XML RPC request).
So be kind to the server by using info instead, (which has a small,
fixed-length response).
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:26 +0000 (10:46 -0800)]
tools/post-receive.hook: don't update the previously pushed commit
Previously, the post-receive hook would always examine one commit that
had been previously pushed, (when the purpose of the hook is only to
example newly-pushed commits). We fix this by simply dropping the '^'
in the commit-range specification.
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
This patch fixes following error when redirecting 'pwclient info' output:
---8<---
andreas@dockstar % pwclient info 295268 > /tmp/test.out
Traceback (most recent call last):
File "/home/andreas/bin/pwclient", line 508, in <module>
main()
File "/home/andreas/bin/pwclient", line 471, in main
action_info(rpc, patch_id)
File "/home/andreas/bin/pwclient", line 242, in action_info
print("- %- 14s: %s" % (key, value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 29: ordinal not in range(128)
--->8---
Signed-off-by: Andreas Bießmann <andreas@biessmann.de> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
In situations where SSL is terminated at the load-balancer, we cannot
rely on guessing the scheme based on whether patchwork itself was
accessed via http or https, since the last-leg is always going to be
done over http.
Unfortunately, wrongly using http:// URLs results in unusable
.pwclientrc files, since xmlrpc does not handle http->https redirects
and instead displays a traceback.
This change introduces a FORCE_HTTPS_LINKS option, which forces
pwclientrc links to always return "https" regardless of how the project
itself is accessed.
It appears that the http/https check is currently only used for
generating pwclientrc -- a lot of other places seem to hardcode
"http://" and rely on the server to transparently upgrade the
connection. This is not a secure approach (it allows for MITM and
SSL-Strip attacks) and therefore all places currently hardcoding
http://{{site.domain}} and similar should be switched to using the
"sheme" variable, the same as done for generating pwclientrc files.
lib/sql/migration: Fix column names for MySQL/MariaDB
The first issue is that patchwork was no longer accepting new patches via
the apps/patchwork/bin/parsemail.sh script. When I was trying to invoke
it manually, it only printed "no project found". I was able to figure
out that this was caused by changes to the database scheme which are taken
care of by the two SQL scripts:
The catch - these first script don't run with MySQL - or rather MariaDB that
Fedora has switched to. MariaDB doesn't like the quotes around the table
and column names. Patch below.
This change fixes the issue by adding a new state for patch 'meta'
headers: header test which is not parsed as a hunk, but has the same
behaviour as a hunk in that it will cause the patchbuf to be populated.
Jeremy Kerr [Sun, 13 Oct 2013 04:48:21 +0000 (12:48 +0800)]
views/base: Include email addresses in submitter autocomplete search
Currently, we only search names, which is a problem for Person objects
with only an email address set. This change includes the email addresses
in the search.
docs: also mention that ROOT_DIR needs to be updated
When the patchwork installation resides in the non-default location
(/srv/patchwork) make sure that local_settings.py also gets updated with
a proper ROOT_DIR value otherwise various problems will happen.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Django is no longer accessible using the old Subversion URL, update that
to the Github repository. Besides, the command-line used to perform the
"syncdb" operation does not work because we are off by one directory in
the filesystem, fix that.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
The total_seconds function was added to datetime in python-2.7. For
compatibility with previous versions of python, use its suggested
equivalent (except drop microseconds, since we don't care about them in
this context).
Signed-off-by: Konstantin Ryabitsev <mricon@kernel.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Mon, 10 Jun 2013 03:50:21 +0000 (11:50 +0800)]
views/mbox: Use Date: header from original message
Since we use UTC for internal date storage, we lose the timestamp info
from the original patch submissions. This means that the mbox views
(which are typically passed through to SCM commit logs) lose the
timezone information.
This change uses the Date header from the original message, if possible.
Jeremy Kerr [Mon, 10 Jun 2013 03:37:25 +0000 (11:37 +0800)]
views: Move mbox handling from models to views
Mbox generation is related to views, not the models themselves. This
change creates a patch_to_mbox function in the views/ directory, which
the actual view handlers use instead of Patch.mbox().
Jeremy Kerr [Mon, 10 Jun 2013 02:16:24 +0000 (10:16 +0800)]
models: Fix invalid dates in patch mbox
Currently, the Date header in mbox views is incorrect;
datetime.datetime.utctimetuple assumes that the date is in the local
timezone, but we keep all dates as UTC.
This change replaces utctimetuple with a manual calculation of the
seconds-since-epoch timestamp, which email.utils.formatdate requires. We
add a testcase for this too.
Reported-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Andreas Bießmann [Fri, 10 May 2013 00:44:26 +0000 (00:44 +0000)]
models: fix From header in mbox view
The From header in mbox view was corrupt for patch submitters with non-ASCII
characters in their names. RFC 2822 requires to retain the mail as ASCII in
braces '<' and '>' while the name should be coded. Before we coded the whole
string which led to some MUA misinterpret the From header (while git could
manage it).
Fix this by coding just the name and let the email untouched.
Signed-off-by: Andreas Bießmann <andreas.devel@googlemail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Sat, 20 Apr 2013 13:45:40 +0000 (21:45 +0800)]
bundles: Remove separate public bundle views
Having two views for bundles (public and non-public) can cause confusion
when bundle owners want to share a URL; it's not obvious that the
private URL isn't shareable.
This change removes the private URLs, and puts all bundles under the
/bundle/<username>/<bundlename>/ URL space. For non-public bundles, this
will just 404 for non-owners.
Jeremy Kerr [Sun, 30 Dec 2012 05:57:18 +0000 (13:57 +0800)]
forms: Only pass post data to multiplepatchform if that was what was submitted
When updating bundles, we get a spurious error message in the
MultiplePatchForm; it's performing validation on the blank form data.
This change only passes the post data to the MuliplePatchForm if we
detect (through data['form']) that it was the MultiplePatchForm that was
submitted.
Jeremy Kerr [Sun, 30 Dec 2012 04:31:50 +0000 (12:31 +0800)]
templates/bundle: Unify public bundle meta information
Currently, the public bundle meta information (the download link before
the list of patches) differs from that of the private bundle view. This
change unifies this information with the private view.
Doug Anderson [Fri, 21 Dec 2012 13:42:43 +0000 (13:42 +0000)]
xmlrpc: Avoid trying to marshall None for people with no name
If a person sent patches with no real name, they may have None for
their name. In this case just use their email address as their name.
The previous pwclient error that would show up looks like:
$ pwclient list -w anonymous@example.com
Traceback (most recent call last):
File ".../pwclient", line 631, in <module>
main()
File ".../pwclient", line 570, in main
action_list(rpc, filt, submitter_str, delegate_str, series_str)
File ".../pwclient", line 316, in action_list
ids = person_ids_by_name(rpc, submitter_str)
File ".../pwclient", line 241, in person_ids_by_name
people = rpc.person_list(name, 0)
File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1297, in single_request
return self.parse_response(response)
File "/usr/lib/python2.7/xmlrpclib.py", line 1473, in parse_response
return u.close()
File "/usr/lib/python2.7/xmlrpclib.py", line 793, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
Signed-off-by: Doug Anderson <dianders@chromium.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Simo Sorce [Fri, 26 Oct 2012 12:51:42 +0000 (12:51 +0000)]
Add URL and SCM data to projects
This allows us to display friendly links to the project website,
web SCM UI and SCM URL.
For example for the patchwork project these could be set to:
web_url: http://jk.ozlabs.org/projects/patchwork/
scm_url: git://ozlabs.org/home/jk/git/patchwork
webscm_url: http://git.ozlabs.org/?p=patchwork;a=tree
Requires a DB schema upgrade
Signed-off-by: Simo Sorce <idra@samba.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Stefan Hajnoczi [Sun, 15 Jan 2012 09:49:24 +0000 (09:49 +0000)]
notifications: do not HTML-escape patch change notifications
Patch change notifications are text/plain emails and should not
HTML-escape patch names. This means quotes and ampersands in patch
names should not be turned into " and & HTML entities.
Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
This will make editing states through the admin UI less error-prone, and
will facilitate parsing patches when relying on a default state, as well as
testing.
Use the explicit default state when parsing mails.
Signed-off-by: Dirk Wallenstein <halsmit@t-online.de> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Thu, 8 Dec 2011 13:42:26 +0000 (21:42 +0800)]
filters: handle utf-8 characters when building filter querystring
Currently, we build the filter querystring by passing the raw string to
urllib.quote, which expects an encoded string. When the raw string (in
this case, a unicode object) contains a unicode character, quote()
raises a KeyError.
This fixes the problem by encoding the parameter name and values to
UTF-8 first.
Reported-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
david decotigny [Tue, 22 Nov 2011 09:12:48 +0000 (09:12 +0000)]
patchwork: new pwclient git-am action
This commit adds a new "pwclient git-am" action which applies given
patch ID on the current git branch using "git am". It's convenient to
keep track and authorship of patches applied locally from a patchwork
server.
Signed-off-by: David Decotigny <david.decotigny@google.com>
Jeremy Kerr [Mon, 19 Sep 2011 02:33:51 +0000 (10:33 +0800)]
models: Update Person & UserProfile object whenever User is saved
Currently, UserProfiles are only saved on creation of the User object,
before the first_name and last_name attributes are set. This means that
we fallback to using the User.username value as the new Person.name.
This change modifies User's post-save signal to always update the Person
object, both on creation and updates. This means we get the proper name
appearing on Person instances.
Jeremy Kerr [Mon, 18 Jul 2011 13:46:08 +0000 (21:46 +0800)]
xmlrpc: do slice before patch_to_dict
Currently, we map patch_to_dict before we slice the results (to only
return max_count patches). This means that we hacve to retrieve all
patches, then throw away most of the results of the map.
This change does the slice on the patches before the map, letting django
do a LIMIT-ed query instead.
Jeremy Kerr [Tue, 19 Apr 2011 05:18:45 +0000 (13:18 +0800)]
settings: Use multiple-DB-style format for default settings
The DATABASE_* settings have been deprecated in django 1.3, replaced by
the multiple-database style format (introduced in 1.2, which patchwork
requires anyway).
This change modifies the default settings.py to use the new format.
Jeremy Kerr [Tue, 29 Mar 2011 14:18:54 +0000 (22:18 +0800)]
notifications: Add code to send notifications
Add a function (patchwork.utils.send_notifications) to process the
PatchChangeNotification queue. We try to group mail to the same sender,
by waiting settings.NOTIFICATION_DELAY_MINUTES to allow other
notifications to arrive.
Jeremy Kerr [Wed, 11 Aug 2010 06:16:28 +0000 (14:16 +0800)]
Add email opt-out system
We're going to start generating emails on patchwork updates, so firstly
allow people to opt-out of all patchwork communications.
We do this with a 'mail settings' interface, allowing non-registered
users to set preferences on their email address. Logged-in users can do
this through the user profile view.
Jeremy Kerr [Thu, 12 Aug 2010 04:15:48 +0000 (12:15 +0800)]
Use generic email confirmation object
Rather than having a UserPerson-specific confirmation, add an
EmailConfirmation object to allow multiple types of confirmations (eg,
opt-out requests in future).
To do this, we use a view (patchwork.views.confirm) that will call the
type-specific view with the confirmation object.
Also, add tests to check that the User/Person linkage system works.
forms: Fix MultipleBooleanField.to_python when the field value is missing
If you write a test for, say, the bundle form of a patch list, you'd
still have to specify the 'no change' value to other form (e.g. the
multiple update one) fields using MultipleBooleanField or else it'd
raise a ValueError when field.clean() is called as part of
form._get_errors().
Signed-off-by: Guilherme Salgado <guilherme.salgado@linaro.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
projects / patchwork / log