Rusty Russell [Fri, 25 Sep 2009 05:03:08 +0000 (14:33 +0930)]
ccanlint: compile and run tests.
This means we should skip building if there are no C files in module: running tests requires building the module, but not necessarily that it has any C files.
Rusty Russell [Fri, 25 Sep 2009 03:16:08 +0000 (12:46 +0930)]
tdb: allow reads after prepare commit
(Imported from SAMBA commit 46c99ec2a3781d53138245182345c6c2ddd9a258:
From: Andrew Tridgell <tridge@samba.org>
Date: Tue, 15 Sep 2009 14:04:22 -0700
We previously only allowed a commit to happen after a prepare
commit. It is in fact safe to allow reads between a prepare and a
commit, and the s4 replication code can make use of that, so allow it.
Rusty Russell [Sat, 12 Sep 2009 01:27:44 +0000 (10:57 +0930)]
Build tests for ccan.
More sophisticated skipping: skip dependencies when one fails as well.
Allow tests to change their total_score; only access it after running (other than to check it's non-zero).
Rusty Russell [Tue, 18 Aug 2009 04:07:47 +0000 (13:37 +0930)]
tdb: locking cleanup
Explicitly split lock and unlock, for future replacement with user-supplied functions.
Use explicit flags enum for marking, blocking and probing.
Indicate upon unlock whether we had a read or write lock (POSIX doesn't care, but
wrappers might, and it's nice to be explicit).
Rusty Russell [Tue, 4 Aug 2009 03:52:28 +0000 (13:22 +0930)]
Wean off TDB_ERRCODE.
It was a regrettable hack to reduce line count in tdb; in fact it caused confusion as can be seen in this patch.
In particular, ecode now needs to be set before TDB_LOG.
Also, we should never set errno, as io.c was doing.
This greatly reduces the fragmentation of databases where records
tend to grow slowly by a small amount each time. The case where this
is most seen is the ldb index records. Adding this overallocation
reduced the size of the resulting database by more than 20x when
running a test that adds 10k users.
The idea behind this is to recover from badly fragmented free
lists. Choosing the point where the file expands is fairly arbitrary,
but seems to work well.
The tdb_repack() function repacks a TDB so that it has a single
freelist entry. The file doesn't shrink, but it does remove all
freelist fragmentation. This code originated in the CTDB vacuuming
code, but will now be used in ldb to cope with fragmentation from
re-indexing
make TDB_NOSYNC affect all the fsync/msync calls in transactions
During a transaction commit tdb normally uses fsync/msync calls to
make it crash safe. This can be disabled using the TDB_NOSYNC flag,
but it wasn't disabling all the code paths that caused a fsync/msync.
Using tdb_transaction_prepare_commit() gives us 2-phase commits. This
allows us to safely commit across multiple tdb databases at once, with
reasonable transaction semantics
Rusty Russell [Fri, 17 Jul 2009 07:29:58 +0000 (16:59 +0930)]
Cleanup variable names.
struct op: serial -> seqnum (and drop use of term "serial" everywhere).
struct op: op -> type
struct op_desc: general replacement for file/op_num pair.
Change add_dependency to take two const op_desc *: means reshuffle since we can't adjust the args any more.
Rusty Russell [Thu, 16 Jul 2009 01:58:33 +0000 (11:28 +0930)]
Implement timeout for the deadlock of traverse & transactions.
This has proven to be intractible: various attempts to eliminate have failed, so detect at runtime and cease the traversal (and do the remaining ops outside a traverse).
Rusty Russell [Wed, 15 Jul 2009 05:27:49 +0000 (14:57 +0930)]
Handle transactions!
Note: we can still deadlock on traversal vs transaction corner cases.
We handle transactions as single operation, which it logically is.
Rusty Russell [Wed, 15 Jul 2009 03:49:35 +0000 (13:19 +0930)]
Insert (implied) transaction cancel on tdb_close/EOF.
Also changes first member of transaction to have valid start_group field,
and fix outdated comment.
Rusty Russell [Tue, 14 Jul 2009 12:34:36 +0000 (22:04 +0930)]
Fix early transaction unlock when traverse done inside transaction.
Generalizes traverse in traverse fix from rusty@rustcorp.com.au-20090629073630-3eduhyypx2tp6u80
Rusty Russell [Mon, 13 Jul 2009 06:19:52 +0000 (15:49 +0930)]
More general solution for serial number misorders.
Make sort_deps more efficient, and also only alter order when necessary. This means by default we run in serial number order, only going outside when we detect a dependency.
Maintain trace file order in original sort, so sort_deps doesn't mess it up.
We still need serial numbers: sort_deps can have multiple solutions for a single key, but these may deadlock with the ordering requirements of other keys. By sticking close to the actual order (ie. serial order), we minimize the chance of this happening.
Rusty Russell [Mon, 13 Jul 2009 02:14:40 +0000 (11:44 +0930)]
Optimize to reduce extraneous dependencies.
In my tdbtorture -n 4 example trace, this reduces from 14493 to 3210 dependencies, but doesn't make any measurable improvement in the time. Still, it's simple to do and might make a difference for larger sets.
Rusty Russell [Mon, 13 Jul 2009 01:48:05 +0000 (11:18 +0930)]
Simplify dependencies by passing pointers over the pipe: avoid O(n^2) behaviour for searching.
Also, using a single structure makes talloc_free convenient for when we do optimization.
Rusty Russell [Sun, 12 Jul 2009 12:48:09 +0000 (22:18 +0930)]
Get more sophisticated with resolving duplicate serial numbers.
Still deadlocks in one case, due to spurious dependencies inside traversals. See next commit.