Jani Nikula [Tue, 21 Oct 2014 08:18:53 +0000 (11:18 +0300)]
pwclient: add output format option to list/search command
Make scripting easier by letting the user specify the output format. The
format string may contain tag references to fields, such as %{id} and
%{msgid}.
Signed-off-by: Jani Nikula <jani.nikula@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
HACKING: Add some documentation about using virtualenv with patchwork
virtualenv is super nice to isolate devevelopment from the python
packages installed in the sytem. It also allows to have parallel
configurations to run tests againsts.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Damien Lespiau [Sun, 31 Aug 2014 00:02:37 +0000 (01:02 +0100)]
INSTALL: Sprinkle a few UTF-8 in the mysql documentation
When not specifying the charset/collation, I managed to create a latin1
database where all strings were encoded in latin1. That's really not
ideal. Adding 'CHARACTER SET utf8' when creating the DB fixes it. Then:
$ ./manage.py syncdb
will correctly create tables with UTF-8 encoded rows.
However, for some reason, when django creates the test tables, that
default, DB wide, encoding is not respected and one needs to provide an
additional TEST_CHARSET entry in the config dictionary.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Damien Lespiau [Sun, 31 Aug 2014 00:02:36 +0000 (01:02 +0100)]
INSTALL: Update the database configuration instructions
That's the "new" (django 1.5+) way of defining databases, strictly
following the instructions wasn't working. This should save the next
user a bit of time.
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Sun, 7 Sep 2014 12:05:18 +0000 (20:05 +0800)]
pwclient: preserve ordering of patch ids
Currently, we use a frozenset to remove duplicates in the patch id
list. However, this means we lose the patch ordering, which is important
for a git-am.
This change drops the frozenset, so we preserve the ordering from the
arguments.
Allow commands that take an ID to operate on multiple IDs.
E.g.:
update -s Superseded 1 2 3 4 5
apply 2 4 6
Reject update -c COMMIT-REF on multiple IDs though as that does not
make sense.
Implementation note:
nargs='*' instead of '?' results in (wrong/inconvenient):
mutually exclusive arguments must be optional
So remove mutual exclusive handling via argparse and instead do it by
hand. This might be implemented more conveniently in later python but we
(have to) stick with 2.7.x for the time being.
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Brian Norris [Thu, 12 Jun 2014 01:54:59 +0000 (18:54 -0700)]
post-receive: exclude commits from the patch update step
When merging upstream work related to other projects into your own
project repository, you probably don't want to check for (and try to
update) the status on every change-set in the merge. So add a list of
references (branches, tags, commits, etc.) whose commits should be
ignored in the patch update step.
This could be used, for example, to set:
EXCLUDE="refs/heads/upstream"
Then when you're ready to merge in new upstream code, you first update
the 'upstream' branch before pushing your own.
Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
pwclient: accept more than one project in ~/.pwclientrc
Currently, given the format of ~/.pwclientrc, pwclient can only
really act on a single project, as ~/pwclientrc can only contain
the configuration for a single project.
Although the -p options comes in handy to specify a project
different from the one configured in ~/.pwclientrc, this only works
if it is hosted on the same server. As soon as one needs to switch
server, it is necessary to edit ~/pwclientrc.
This can be quite inefficient when dealing with many projects, hosted
on different servers.
Change the format of ~/.pwclientrc so it is possible to define more
than one project, and for each project, specify an URL and credentials.
This has the advantage of not changing the options to pwclient, so
the user experience is unmodified.
If a ~/.pwclentrc exists in the old format, it is automatically
converted over to the new format, and the previous one is saved as
~/.pwclientrc.orig. Upon conversion, no action is made, pwclient just
exits (with return-code 1) to inform the user to review the conversion.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Thu, 8 May 2014 05:53:09 +0000 (13:53 +0800)]
filters: fix exception in filter querystring generation
We get a silent (as it's during template render) exception when
generating filter querystrings, as we're passing a list to the string
format operator rather than a tuple.
This change removes the map and explicitly applies sanitise to the
(name, value) pair.
Jeremy Kerr [Wed, 7 May 2014 12:24:17 +0000 (20:24 +0800)]
views/generic_list: reduce number of queries in list rendering
Currently, we do two database queries per patch on a list view; one to
retrieve the submitter, and one for the state.
This patch adds a .select_related to fetch for the submitter and state,
and a .defer() to prevent loading large amounts of text data from the
patch content and headers.
This gives a significant reduction in the work per request. For a
paginated list view (ie 100 patches per page):
before after
User 1344 ms 228 ms
System 170 ms 25 ms
Total 1514 ms 253 ms
Elapsed 1605 ms 274 ms
Jeremy Kerr [Wed, 7 May 2014 02:03:44 +0000 (10:03 +0800)]
tests: Add XMLRPC interface test
This change adds a simple test for the XMLRPC interface, essentially to
ensure that configuration changes haven't broken accessibility to the
interface.
Jeremy Kerr [Tue, 22 Apr 2014 13:19:52 +0000 (21:19 +0800)]
Fix django-1.6 incompatibilities
We're seeing a couple of final quirks running the testsuite on django
1.6:
Traceback (most recent call last):
File "patchwork/apps/patchwork/tests/notifications.py", line 182, in testNotificationEscaping
errors = send_notifications()
File "patchwork/apps/patchwork/utils.py", line 227, in send_notifications
delete_notifications()
File "patchwork/apps/patchwork/utils.py", line 197, in delete_notifications
pk__in = notifications).delete()
File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 163, in filter
return self.get_queryset().filter(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 590, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 608, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1198, in add_q
clause = self._add_q(where_part, used_aliases)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1232, in _add_q
current_negated=current_negated)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1122, in build_filter
lookup_type, value)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 1107, in get_lookup_constraint
values = [get_normalized_value(value) for value in raw_value]
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 1084, in get_normalized_value
value_list.append(getattr(value, source.attname))
AttributeError: 'PatchChangeNotification' object has no attribute 'id'
- we're specifying our own pk here, so the PatchChangeNotification has
no id attribute; it looks like the pk__in syntax is expecting IDs.
We also need a default value for BooleanField, as we're getting
integrity errors when creating rows with no explicit send_notifications
set.
Jeremy Kerr [Tue, 22 Apr 2014 12:48:19 +0000 (20:48 +0800)]
Defer Person creation/linkage until registration is confirmed
We currently create Person objects when a registration is submitted, not
when it is confirmed. This can lead to stale Person objects for
unconfirmed registrations.
Jeremy Kerr [Tue, 22 Apr 2014 12:08:27 +0000 (20:08 +0800)]
Add unconfirmed registration expiry
Currently, unconfirmed registrations remain in the database. Although we
have an expiry for the registrations, we don't actually remove rows from
the database. This can clog the admin interface up with unnecessary
registration spam.
We currently have a patchwork cron script to send notifications on patch
changes, so hook this into a new do_expiry function.
Carl Worth [Fri, 13 Dec 2013 04:28:02 +0000 (20:28 -0800)]
post-receive: Fix error message when pushing to an unmapped branch
Previously, if STATE_MAP contained only refs/heads/master and a user
pushed to a branch such as refs/heads/stable, the hook emitted a
misleading error message:
E: no mapping for refname refs/heads/master
Fix this by correctly printing the reference being looked for, rather
than the reference most recently found in STATE_MAP. Also, reword the
message slightly to point the user to STATE_MAP and to use an actual
word "branch" instead of the non-word "refname".
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:28 +0000 (10:46 -0800)]
docs: Add a pointer to the git post-receive hook
I had to stumble upon this feature before I realized it was here
inside the tools directory all along. This documentation should make
it easier for the next person coming along looking for a patchwork git
hook.
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:27 +0000 (10:46 -0800)]
Make the post-receive hook more efficient (using pwclient info, not view)
Older versions of patchwork shipped with a version of pwclient that
had the view command, but not info command. Now that info exists, it's
inefficient to use it to obtain the patchwork ID value, (since it
requires the server to send the entire patch content in response to an
XML RPC request).
So be kind to the server by using info instead, (which has a small,
fixed-length response).
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Carl Worth [Wed, 11 Dec 2013 18:46:26 +0000 (10:46 -0800)]
tools/post-receive.hook: don't update the previously pushed commit
Previously, the post-receive hook would always examine one commit that
had been previously pushed, (when the purpose of the hook is only to
example newly-pushed commits). We fix this by simply dropping the '^'
in the commit-range specification.
Signed-off-by: Carl Worth <cworth@cworth.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
This patch fixes following error when redirecting 'pwclient info' output:
---8<---
andreas@dockstar % pwclient info 295268 > /tmp/test.out
Traceback (most recent call last):
File "/home/andreas/bin/pwclient", line 508, in <module>
main()
File "/home/andreas/bin/pwclient", line 471, in main
action_info(rpc, patch_id)
File "/home/andreas/bin/pwclient", line 242, in action_info
print("- %- 14s: %s" % (key, value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 29: ordinal not in range(128)
--->8---
Signed-off-by: Andreas Bießmann <andreas@biessmann.de> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
In situations where SSL is terminated at the load-balancer, we cannot
rely on guessing the scheme based on whether patchwork itself was
accessed via http or https, since the last-leg is always going to be
done over http.
Unfortunately, wrongly using http:// URLs results in unusable
.pwclientrc files, since xmlrpc does not handle http->https redirects
and instead displays a traceback.
This change introduces a FORCE_HTTPS_LINKS option, which forces
pwclientrc links to always return "https" regardless of how the project
itself is accessed.
It appears that the http/https check is currently only used for
generating pwclientrc -- a lot of other places seem to hardcode
"http://" and rely on the server to transparently upgrade the
connection. This is not a secure approach (it allows for MITM and
SSL-Strip attacks) and therefore all places currently hardcoding
http://{{site.domain}} and similar should be switched to using the
"sheme" variable, the same as done for generating pwclientrc files.
lib/sql/migration: Fix column names for MySQL/MariaDB
The first issue is that patchwork was no longer accepting new patches via
the apps/patchwork/bin/parsemail.sh script. When I was trying to invoke
it manually, it only printed "no project found". I was able to figure
out that this was caused by changes to the database scheme which are taken
care of by the two SQL scripts:
The catch - these first script don't run with MySQL - or rather MariaDB that
Fedora has switched to. MariaDB doesn't like the quotes around the table
and column names. Patch below.
This change fixes the issue by adding a new state for patch 'meta'
headers: header test which is not parsed as a hunk, but has the same
behaviour as a hunk in that it will cause the patchbuf to be populated.
Jeremy Kerr [Sun, 13 Oct 2013 04:48:21 +0000 (12:48 +0800)]
views/base: Include email addresses in submitter autocomplete search
Currently, we only search names, which is a problem for Person objects
with only an email address set. This change includes the email addresses
in the search.
docs: also mention that ROOT_DIR needs to be updated
When the patchwork installation resides in the non-default location
(/srv/patchwork) make sure that local_settings.py also gets updated with
a proper ROOT_DIR value otherwise various problems will happen.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Django is no longer accessible using the old Subversion URL, update that
to the Github repository. Besides, the command-line used to perform the
"syncdb" operation does not work because we are off by one directory in
the filesystem, fix that.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
The total_seconds function was added to datetime in python-2.7. For
compatibility with previous versions of python, use its suggested
equivalent (except drop microseconds, since we don't care about them in
this context).
Signed-off-by: Konstantin Ryabitsev <mricon@kernel.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Mon, 10 Jun 2013 03:50:21 +0000 (11:50 +0800)]
views/mbox: Use Date: header from original message
Since we use UTC for internal date storage, we lose the timestamp info
from the original patch submissions. This means that the mbox views
(which are typically passed through to SCM commit logs) lose the
timezone information.
This change uses the Date header from the original message, if possible.
Jeremy Kerr [Mon, 10 Jun 2013 03:37:25 +0000 (11:37 +0800)]
views: Move mbox handling from models to views
Mbox generation is related to views, not the models themselves. This
change creates a patch_to_mbox function in the views/ directory, which
the actual view handlers use instead of Patch.mbox().
Jeremy Kerr [Mon, 10 Jun 2013 02:16:24 +0000 (10:16 +0800)]
models: Fix invalid dates in patch mbox
Currently, the Date header in mbox views is incorrect;
datetime.datetime.utctimetuple assumes that the date is in the local
timezone, but we keep all dates as UTC.
This change replaces utctimetuple with a manual calculation of the
seconds-since-epoch timestamp, which email.utils.formatdate requires. We
add a testcase for this too.
Reported-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Andreas Bießmann [Fri, 10 May 2013 00:44:26 +0000 (00:44 +0000)]
models: fix From header in mbox view
The From header in mbox view was corrupt for patch submitters with non-ASCII
characters in their names. RFC 2822 requires to retain the mail as ASCII in
braces '<' and '>' while the name should be coded. Before we coded the whole
string which led to some MUA misinterpret the From header (while git could
manage it).
Fix this by coding just the name and let the email untouched.
Signed-off-by: Andreas Bießmann <andreas.devel@googlemail.com> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Jeremy Kerr [Sat, 20 Apr 2013 13:45:40 +0000 (21:45 +0800)]
bundles: Remove separate public bundle views
Having two views for bundles (public and non-public) can cause confusion
when bundle owners want to share a URL; it's not obvious that the
private URL isn't shareable.
This change removes the private URLs, and puts all bundles under the
/bundle/<username>/<bundlename>/ URL space. For non-public bundles, this
will just 404 for non-owners.
Jeremy Kerr [Sun, 30 Dec 2012 05:57:18 +0000 (13:57 +0800)]
forms: Only pass post data to multiplepatchform if that was what was submitted
When updating bundles, we get a spurious error message in the
MultiplePatchForm; it's performing validation on the blank form data.
This change only passes the post data to the MuliplePatchForm if we
detect (through data['form']) that it was the MultiplePatchForm that was
submitted.
Jeremy Kerr [Sun, 30 Dec 2012 04:31:50 +0000 (12:31 +0800)]
templates/bundle: Unify public bundle meta information
Currently, the public bundle meta information (the download link before
the list of patches) differs from that of the private bundle view. This
change unifies this information with the private view.
Doug Anderson [Fri, 21 Dec 2012 13:42:43 +0000 (13:42 +0000)]
xmlrpc: Avoid trying to marshall None for people with no name
If a person sent patches with no real name, they may have None for
their name. In this case just use their email address as their name.
The previous pwclient error that would show up looks like:
$ pwclient list -w anonymous@example.com
Traceback (most recent call last):
File ".../pwclient", line 631, in <module>
main()
File ".../pwclient", line 570, in main
action_list(rpc, filt, submitter_str, delegate_str, series_str)
File ".../pwclient", line 316, in action_list
ids = person_ids_by_name(rpc, submitter_str)
File ".../pwclient", line 241, in person_ids_by_name
people = rpc.person_list(name, 0)
File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1297, in single_request
return self.parse_response(response)
File "/usr/lib/python2.7/xmlrpclib.py", line 1473, in parse_response
return u.close()
File "/usr/lib/python2.7/xmlrpclib.py", line 793, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">
Signed-off-by: Doug Anderson <dianders@chromium.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Simo Sorce [Fri, 26 Oct 2012 12:51:42 +0000 (12:51 +0000)]
Add URL and SCM data to projects
This allows us to display friendly links to the project website,
web SCM UI and SCM URL.
For example for the patchwork project these could be set to:
web_url: http://jk.ozlabs.org/projects/patchwork/
scm_url: git://ozlabs.org/home/jk/git/patchwork
webscm_url: http://git.ozlabs.org/?p=patchwork;a=tree
Requires a DB schema upgrade
Signed-off-by: Simo Sorce <idra@samba.org> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>