9 years agotlist: implement tlist_for_each_rev
Rusty Russell [Tue, 29 Nov 2011 22:33:11 +0000 (09:03 +1030)]
tlist: implement tlist_for_each_rev

9 years agolist: implement list_for_each_rev()
Rusty Russell [Tue, 29 Nov 2011 22:32:11 +0000 (09:02 +1030)]
list: implement list_for_each_rev()

9 years agotdb2: consolidate testing failtest suppression routines.
Rusty Russell [Tue, 29 Nov 2011 22:31:11 +0000 (09:01 +1030)]
tdb2: consolidate testing failtest suppression routines.

Less cut & paste means less patching as failtest changes.

9 years agofailtest: fix internal cut & paste bug
Rusty Russell [Tue, 29 Nov 2011 22:30:11 +0000 (09:00 +1030)]
failtest: fix internal cut & paste bug

failtest_malloc should use p->u.malloc not p->u.calloc.  The layouts
are identical, so it doesn't matter, but it's confusing and leaves us
open to weird bugs in future should one change.

9 years agoccanlint: fix depends_build_without_features
Rusty Russell [Tue, 29 Nov 2011 22:29:11 +0000 (08:59 +1030)]
ccanlint: fix depends_build_without_features

This depends on reduce_features; currently it tends to run before
that, so it doesn't think there are any features to reduce and doesn't
build or run the tests with reduced features.

9 years agoMakefile-web: push to github repo too.
Rusty Russell [Wed, 23 Nov 2011 02:36:00 +0000 (13:06 +1030)]
Makefile-web: push to github repo too.

Saves me doing it manually.

9 years agolist: fix list element counting in provided example inside _info file
Vladimir Zapolskiy [Tue, 22 Nov 2011 23:58:15 +0000 (01:58 +0200)]
list: fix list element counting in provided example inside _info file

This change initializes a counter of children, otherwise is may contain
arbitrary value.

Signed-off-by: Vladimir Zapolskiy <>
9 years agocast: fix warnings with -Wextra (specifically -Wmissing-field-initializers)
Rusty Russell [Mon, 21 Nov 2011 05:17:00 +0000 (15:47 +1030)]
cast: fix warnings with -Wextra (specifically -Wmissing-field-initializers)

As noted by Jan Engelhardt; libHX fixed this already.

9 years agolist: simplify tests a little.
Rusty Russell [Mon, 21 Nov 2011 02:02:30 +0000 (12:32 +1030)]
list: simplify tests a little.

The two tests are identical with one line different, so make it clear.

9 years agoccanlint: fix --test-dep-graph
Rusty Russell [Wed, 16 Nov 2011 07:44:56 +0000 (18:14 +1030)]
ccanlint: fix --test-dep-graph

Broken by 758ab1f402b4 "ccanlint: drop -d, interpret any arguments as
directories, and allow multiple."

9 years agoccanlint: make fewer tests compulsory.
Rusty Russell [Tue, 15 Nov 2011 02:12:25 +0000 (12:42 +1030)]
ccanlint: make fewer tests compulsory.

Compulsory means "malformed", we might get rid of it altogether, since
any test can mark "fail" and make ccanlint exit with non-zero status.

Now we only have four compulsory tests:
   info_exists               Module has _info file
   depends_exist             Module's CCAN dependencies can be found
   objects_build             Module object files can be built
   module_builds             Module can be built from object files

9 years agoccanlint: fix ccanlint -l
Rusty Russell [Tue, 15 Nov 2011 02:10:37 +0000 (12:40 +1030)]
ccanlint: fix ccanlint -l

9 years agotdb2: don't be fascist when TDB_VERSION1 is specified.
Rusty Russell [Tue, 1 Nov 2011 23:38:32 +0000 (10:08 +1030)]
tdb2: don't be fascist when TDB_VERSION1 is specified.

We currently insist that a tdb file be a version1 file if tdb_open() is
passed the TDB_VERSION1 flag; we fail if it's actually a tdb2.

But that makes generic wrappers harder, and is unlikely to be what the
user wants: if they do, they can check tdb_get_flags() & TDB_VERSION1
after opening.

9 years agostrset: set errno to ENOENT even if we return NULL.
Rusty Russell [Tue, 1 Nov 2011 23:06:29 +0000 (09:36 +1030)]
strset: set errno to ENOENT even if we return NULL.

It seems redundant: strset_test() and strset_clear() can only return NULL
when the string is not a member.  However, it became clear in writing
ccan/tsort that it's much more convenient for callers if we set errno
in this case too, so they can pass it up.

9 years agostrmap: set errno to ENOENT even if we return NULL.
Rusty Russell [Tue, 1 Nov 2011 23:05:29 +0000 (09:35 +1030)]
strmap: set errno to ENOENT even if we return NULL.

It seems redundant: strmap_get() and strmap_del() can only return NULL
when the string is not a member.  However, it became clear in writing
ccan/tsort that it's much more convenient for callers if we set errno
in this case too, so they can pass it up.

9 years agoccanlint: fix uninitialized variables in tests_pass_valgrind.
Rusty Russell [Tue, 1 Nov 2011 23:04:29 +0000 (09:34 +1030)]
ccanlint: fix uninitialized variables in tests_pass_valgrind.

We weren't initializing the leak_info field when the test was marked
FAIL in _info (as in tdb2).

9 years agoccanlint: fix test link when given no arguments.
Rusty Russell [Tue, 1 Nov 2011 23:01:25 +0000 (09:31 +1030)]
ccanlint: fix test link when given no arguments.

Fallout from commit 758ab1f402b4af3cccac09548b709deca17ef930
(ccanlint: drop -d, interpret any arguments as directories...)

9 years agocompiler, ilog: IDEMPOTENT "idempotent does not mean what you think it means"
Rusty Russell [Wed, 26 Oct 2011 06:26:22 +0000 (16:56 +1030)]
compiler, ilog: IDEMPOTENT "idempotent does not mean what you think it means"

Actually, I don't even think it means that.  But rename it to something
which is sane.

Thanks to David Gibson for reporting.

9 years agostrset: set errno on strset_add failures.
Rusty Russell [Wed, 26 Oct 2011 06:25:50 +0000 (16:55 +1030)]
strset: set errno on strset_add failures.

9 years agostrmap: set errno on strmap_add() failures.
Rusty Russell [Wed, 26 Oct 2011 06:25:38 +0000 (16:55 +1030)]
strmap: set errno on strmap_add() failures.

9 years agoccanlint: fix creation of example test/run.c
Rusty Russell [Mon, 24 Oct 2011 23:50:18 +0000 (10:20 +1030)]
ccanlint: fix creation of example test/run.c

Hasn't worked for a while now.  Oops.

9 years agoccanlint: drop -d, interpret any arguments as directories, and allow multiple.
Rusty Russell [Wed, 5 Oct 2011 05:23:35 +0000 (15:53 +1030)]
ccanlint: drop -d, interpret any arguments as directories, and allow multiple.

This gets us closer to 'ccanlint *' rather than relying on Makefiles
to test all modules.  Unfortunately, because of limited parallelism, it's
currently slower than make -j.

9 years agoccanlint: fix spurious warning errors.
Rusty Russell [Wed, 5 Oct 2011 03:40:29 +0000 (14:10 +1030)]
ccanlint: fix spurious warning errors.

Because we fork children to do compilations, and we use stdio, we need
to flush stdout before the fork otherwise the child will flush
afterwards.  The compile tests interpret this output as a compiler

This shows up if you redirect ccanlint output to a file.

9 years agotdb2: fix valgrind suppression in _info.
Rusty Russell [Tue, 4 Oct 2011 01:40:09 +0000 (12:10 +1030)]
tdb2: fix valgrind suppression in _info.

Commit c4ca9f54301c0367 renamed run-83-openhook.c to
api-83-openhook.c, but didn't update the suppression in _info.  My
system runs a patched valgrind, which doesn't have this problem.

9 years agoshort_types: fix example for 64 bit platforms
Rusty Russell [Tue, 4 Oct 2011 01:40:08 +0000 (12:10 +1030)]
short_types: fix example for 64 bit platforms

Use correct printf string.

9 years agoopt: increase testing coverage on 64 bit platforms
Rusty Russell [Tue, 4 Oct 2011 01:40:08 +0000 (12:10 +1030)]
opt: increase testing coverage on 64 bit platforms

Help the compiler eliminate untestable code.

9 years agonet: fix tests for systems without IPv6 entries in /etc/hosts.
Rusty Russell [Tue, 4 Oct 2011 01:40:08 +0000 (12:10 +1030)]
net: fix tests for systems without IPv6 entries in /etc/hosts.

We fake up a dual IPv4/IPv6 response to localhost for testing, but it
assumed localhost6 or ip6-localhost entries.  Fall back to "::1" which
should always work.

9 years agojmap: fix _info example for 64 bit systems.
Rusty Russell [Tue, 4 Oct 2011 01:40:08 +0000 (12:10 +1030)]
jmap: fix _info example for 64 bit systems.

9 years agoforeach: make test more accurate, fix for 64-bit gcc 4.4.3
Rusty Russell [Tue, 4 Oct 2011 01:39:45 +0000 (12:09 +1030)]
foreach: make test more accurate, fix for 64-bit gcc 4.4.3

gcc (Ubuntu 4.4.3-4ubuntu5) 4.4.3 on x86_64 uses a different stack
offset for the second call to test_ptr_recursion(), meaning it sees
new iterators.  Put it inside a loop to reduce the chance of that

9 years agoasearch: fix example on 64 bit platforms.
Rusty Russell [Tue, 4 Oct 2011 01:38:45 +0000 (12:08 +1030)]
asearch: fix example on 64 bit platforms.

9 years agoalloc: fix memory leak in tests.
Rusty Russell [Tue, 4 Oct 2011 01:37:45 +0000 (12:07 +1030)]
alloc: fix memory leak in tests.

9 years agotdb2: fix test printf for 64 bit.
Rusty Russell [Tue, 4 Oct 2011 01:36:45 +0000 (12:06 +1030)]
tdb2: fix test printf for 64 bit.

Very minor fix, since this won't happen unless the test fails.

9 years agoweb: add pointer to wiki/Contribute.
Rusty Russell [Thu, 29 Sep 2011 03:20:16 +0000 (12:50 +0930)]
web: add pointer to wiki/Contribute.

9 years agoNote new URL:, and point to new wiki on GitHub.
Rusty Russell [Thu, 29 Sep 2011 03:08:06 +0000 (12:38 +0930)]
Note new URL:, and point to new wiki on GitHub.

9 years agostrmap: new module for ordered map of strings using a critbit tree.
Rusty Russell [Wed, 28 Sep 2011 02:37:35 +0000 (12:07 +0930)]
strmap: new module for ordered map of strings using a critbit tree.

9 years agotcon: more help, for making strmap.
Rusty Russell [Wed, 28 Sep 2011 02:36:35 +0000 (12:06 +0930)]
tcon: more help, for making strmap.

9 years agohtable: benchmark against hsearch(3)
Rusty Russell [Tue, 27 Sep 2011 06:28:44 +0000 (15:58 +0930)]
htable: benchmark against hsearch(3)

Since that has a fixed hash table size and doesn't support delete, we can't
do a thorough comparison, but we can insert and search.

9 years agolikely: use htable_type
Rusty Russell [Tue, 27 Sep 2011 06:28:43 +0000 (15:58 +0930)]
likely: use htable_type

Also general cleanups:
(1) Don't assume that strings are folded by the compiler.
(2) Implement likely_stats_reset().
(3) Return non-const string from likely_stats(), as caller must free it.
(4) Don't use struct info indirection (that was from when we used callbacks?)
(5) Close memory leak in run-debug.c

9 years agohtable: HTABLE_INITIALIZER() for static initialization.
Rusty Russell [Tue, 27 Sep 2011 06:28:41 +0000 (15:58 +0930)]
htable: HTABLE_INITIALIZER() for static initialization.

Unfortunately it's a bit of a pain to use for typed hashtables, but it

9 years agohtable: clean up interface, document htable_type better.
Rusty Russell [Tue, 27 Sep 2011 05:07:01 +0000 (14:37 +0930)]
htable: clean up interface, document htable_type better.

We change from htable_new()/htable_free() to htable_init/htable_clear.
We also change HTABLE_DEFINE_TYPE() to be the full name, without automatically
prepending htable_.

9 years agohtable: start empty.
Rusty Russell [Tue, 27 Sep 2011 04:56:59 +0000 (14:26 +0930)]
htable: start empty.

There's no real reason to start with 128 entries.

9 years agotap: fix exit status when we have exactly 256 errors.
Rusty Russell [Tue, 27 Sep 2011 04:55:59 +0000 (14:25 +0930)]
tap: fix exit status when we have exactly 256 errors.

9 years agotcon: update and enhance documentation.
Rusty Russell [Tue, 27 Sep 2011 03:38:16 +0000 (13:08 +0930)]
tcon: update and enhance documentation.

9 years agohtable, strset: benchmarking tools.
Rusty Russell [Mon, 26 Sep 2011 11:10:42 +0000 (20:40 +0930)]
htable, strset: benchmarking tools.

This lets us compare hash table vs. strset vs. the example
implementation of critbit trees.

cbspeed 100 runs, min-max(avg):
#01: Initial insert:   236-245(237)
#02: Initial lookup (match):   180-186(180)
#03: Initial lookup (miss):   171-185(172)
#04: Initial lookup (random):   441-457(444)
#05: Initial delete all:   127-132(128)
#06: Initial re-inserting:   219-225(220)
#07: Deleting first half:   101-104(102)
#08: Adding (a different) half:   158-162(159)
#09: Lookup after half-change (match):   202-207(203)
#10: Lookup after half-change (miss):   217-222(218)
#11: Churn 1:   297-302(299)
#12: Churn 2:   297-305(300)
#13: Churn 3:   301-308(303)
#14: Post-Churn lookup (match):   189-195(190)
#15: Post-Churn lookup (miss):   189-193(190)
#16: Post-Churn lookup (random):   499-513(503)

speed 100 runs, min-max(avg):
#01: Initial insert:   211-218(212)
#02: Initial lookup (match):   161-166(162)
#03: Initial lookup (miss):   157-162(158)
#04: Initial lookup (random):   452-460(454)
#05: Initial delete all:   126-135(127)
#06: Initial re-inserting:   193-201(194)
#07: Deleting first half:   99-107(99)
#08: Adding (a different) half:   143-190(144)
#09: Lookup after half-change (match):   183-195(184)
#10: Lookup after half-change (miss):   197-203(198)
#11: Churn 1:   271-278(274)
#12: Churn 2:   280-287(282)
#13: Churn 3:   277-285(279)
#14: Post-Churn lookup (match):   171-175(171)
#15: Post-Churn lookup (miss):   174-178(175)
#16: Post-Churn lookup (random):   525-552(528)

stringspeed 100 runs, min-max(avg):
#01: Initial insert:   300-343(308)
#02: Initial lookup (match):   98-136(99)
#03: Initial lookup (miss):   73-102(75)
#04: Initial lookup (random):   230-282(233)
#05: Initial delete all:   66-102(69)
#06: Initial re-inserting:   62-99(64)
#07: Deleting first half:   43-52(43)
#08: Adding (a different) half:   101-156(106)
#09: Lookup after half-change (match):   114-156(120)
#10: Lookup after half-change (miss):   94-103(95)
#11: Churn 1:   98-105(99)
#12: Churn 2:   96-104(98)
#13: Churn 3:   174-184(176)
#14: Post-Churn lookup (match):   93-112(94)
#15: Post-Churn lookup (miss):   77-107(79)
#16: Post-Churn lookup (random):   229-265(232)

9 years agostrset: new module using critbit trees.
Rusty Russell [Mon, 26 Sep 2011 11:01:17 +0000 (20:31 +0930)]
strset: new module using critbit trees.

Not as fast as using htable, but simple and provides order and prefix

9 years agostr: fix warnings.
Rusty Russell [Mon, 26 Sep 2011 10:43:07 +0000 (20:13 +0930)]
str: fix warnings.

Firstly, -Wwrite-strings makes string literals const, secondly, we mustn't
define str_strstr etc in terms of themselves!

9 years agostr: relicense to public domain.
Rusty Russell [Mon, 26 Sep 2011 10:42:15 +0000 (20:12 +0930)]
str: relicense to public domain.

LGPL is overkill for trivial wrappers like this.

9 years agoccanlint: more tweaks to example mangling.
Rusty Russell [Mon, 26 Sep 2011 09:53:36 +0000 (19:23 +0930)]
ccanlint: more tweaks to example mangling.

Note where previous analysis said the code was outside a function, and
fix double new-line.  Also, treat end of comment as end of statement.

9 years agotypesafe_cb: fix example macro ordering in typesafe_cb_preargs documentation.
Rusty Russell [Sat, 24 Sep 2011 04:31:19 +0000 (14:01 +0930)]
typesafe_cb: fix example macro ordering in typesafe_cb_preargs documentation.

9 years agojmap: use ccan/tcon and always be typesafe.
Rusty Russell [Fri, 23 Sep 2011 02:30:49 +0000 (12:00 +0930)]
jmap: use ccan/tcon and always be typesafe.

This handles both pointer and integer types using ccan/tcon.

9 years agojbitset: rename to jset, use ccan/tcon
Rusty Russell [Fri, 23 Sep 2011 02:29:49 +0000 (11:59 +0930)]
jbitset: rename to jset, use ccan/tcon

It's now a completely generic set implementation, and uses ccan/tcon
to always be typesafe.  It handles both integer and pointer types.

9 years agotlist: use ccan/tcon
Rusty Russell [Fri, 23 Sep 2011 02:28:52 +0000 (11:58 +0930)]
tlist: use ccan/tcon

No real change for users.

9 years agotcon: routines for creating typesafe generic containers
Rusty Russell [Thu, 22 Sep 2011 06:30:30 +0000 (16:00 +0930)]
tcon: routines for creating typesafe generic containers

9 years agoccanlint: fix output for async commands.
Rusty Russell [Thu, 22 Sep 2011 04:30:21 +0000 (14:00 +0930)]
ccanlint: fix output for async commands.

Previously, it was always "", since we left the nul terminator at the

9 years agotdb2: allow readonly changes even while holding locks.
Rusty Russell [Tue, 20 Sep 2011 02:32:43 +0000 (12:02 +0930)]
tdb2: allow readonly changes even while holding locks.

This happens in SAMBA with the TDB_VERSION1, presumably due to a
read-only traverse nested inside a normal traverse (since it doesn't
occur without TDB_VERSION1).

Signed-off-by: Rusty Russell <>
9 years agotdb2: add error conversion functions.
Rusty Russell [Mon, 12 Sep 2011 04:51:40 +0000 (14:21 +0930)]
tdb2: add error conversion functions.

This clarifies the code a little, but also provides a more explicit
mechanism which can be used to debug error handling (by introducing
tdb_err_t and making it a pointer type).

9 years agoccanlint: run tests in parallel
Rusty Russell [Mon, 12 Sep 2011 04:50:40 +0000 (14:20 +0930)]
ccanlint: run tests in parallel

Measuring ccanlint -d ccan/tdb2 on my dual core hyperthread laptop gives:
Before: 5:30
After: 3:31

9 years agotdb2: make tests work in parallel.
Rusty Russell [Mon, 12 Sep 2011 04:50:40 +0000 (14:20 +0930)]
tdb2: make tests work in parallel.

Lazy cut & paste of tdb names causes failure when we run in parallel.

9 years agoccanlint: compile tests with coverage in parallel
Rusty Russell [Mon, 12 Sep 2011 04:50:39 +0000 (14:20 +0930)]
ccanlint: compile tests with coverage in parallel

Measuring ccanlint -d ccan/tdb2 on my dual core hyperthread laptop gives:
Before: 5:59
After: 5:30

9 years agoccanlint: compile tests in parallel
Rusty Russell [Mon, 12 Sep 2011 04:50:39 +0000 (14:20 +0930)]
ccanlint: compile tests in parallel

Measuring ccanlint -d ccan/tdb2 on my dual core hyperthread laptop gives:
Before: 6:47
After: 5:59

9 years agoccanlint: infrastructure to run commands async.
Rusty Russell [Mon, 12 Sep 2011 04:50:26 +0000 (14:20 +0930)]
ccanlint: infrastructure to run commands async.

We use the lbalance module to figure out how many to run in parallel.

9 years agotools: fix realloc bug.
Rusty Russell [Mon, 12 Sep 2011 03:54:34 +0000 (13:24 +0930)]
tools: fix realloc bug.

Reallocing smaller can move the pointer; certainly valgrind detects
it.  Found while chasing intermittant errors on "make fastcheck".

9 years agottxml: zero terminate the buffer after checking it was allocated
Daniel Burke [Fri, 9 Sep 2011 12:29:32 +0000 (21:59 +0930)]
ttxml: zero terminate the buffer after checking it was allocated

9 years agottxml: covering my shame
Daniel Burke [Mon, 5 Sep 2011 09:36:46 +0000 (19:06 +0930)]
ttxml: covering my shame

9 years agottxml: modified it to use a smaller buffer during testing
Daniel Burke [Mon, 5 Sep 2011 09:24:03 +0000 (18:54 +0930)]
ttxml: modified it to use a smaller buffer during testing

9 years agottxml: ccanlint fixes finished
Daniel Burke [Mon, 5 Sep 2011 09:18:04 +0000 (18:48 +0930)]
ttxml: ccanlint fixes finished

Meets all of the requirements of ccanlint.
Code coverage of the tests covers everything except for a malloc() failure.
Added some xml files for the tests to work on.

9 years agottxml: added drop static keyword
Daniel Burke [Mon, 5 Sep 2011 07:46:50 +0000 (17:16 +0930)]
ttxml: added drop static keyword

9 years agottxml: Passes Valgrind now
Daniel Burke [Mon, 5 Sep 2011 07:38:46 +0000 (17:08 +0930)]
ttxml: Passes Valgrind now

Removed the last references of the old buffering method. Should be
stable now. Time to write some proper testing.

9 years agottxml: Rplaced ratchet buffer with regular buffer, 20x speed increase
Daniel Burke [Mon, 5 Sep 2011 01:10:52 +0000 (10:40 +0930)]
ttxml: Rplaced ratchet buffer with regular buffer, 20x speed increase

9 years agotdb2: fix error handling for tdb1_transaction_commit.
Rusty Russell [Fri, 2 Sep 2011 06:18:17 +0000 (15:48 +0930)]
tdb2: fix error handling for tdb1_transaction_commit.

tdb_repack() returns an enum TDB_ERROR, whereas
tdb1_transaction_commit is expected to return 0 or -1.

9 years agotdb2: don't return -1 (ie. TDB_ERR_CORRUPT) on transaction write fail.
Rusty Russell [Fri, 2 Sep 2011 06:13:20 +0000 (15:43 +0930)]
tdb2: don't return -1 (ie. TDB_ERR_CORRUPT) on transaction write fail.

A left-over -1 return; should be returning ecode (probably TDB_ERR_IO).

9 years agotdb2: fix prototypes which return enum TDB_ERROR.
Rusty Russell [Fri, 2 Sep 2011 06:06:47 +0000 (15:36 +0930)]
tdb2: fix prototypes which return enum TDB_ERROR.

Since enums and ints are compatible in C, the compiler didn't warn
that our prototypes for these functions disagreed with the

9 years agotdb2: set tdb_error() on tdb_transaction_prepare_commit.
Rusty Russell [Fri, 2 Sep 2011 06:03:59 +0000 (15:33 +0930)]
tdb2: set tdb_error() on tdb_transaction_prepare_commit.

9 years agotdb2: enhance tdb1 porting document.
Rusty Russell [Wed, 31 Aug 2011 06:09:17 +0000 (15:39 +0930)]
tdb2: enhance tdb1 porting document.

9 years agotdb2: delete old tdb1 tests.
Rusty Russell [Wed, 31 Aug 2011 06:09:17 +0000 (15:39 +0930)]
tdb2: delete old tdb1 tests.

These two tdb1-specific tests have been superceded by the TDB_VERSION1
flags used in the main tests.

9 years agotdb2: test: Speed up run-01-new_database.
Rusty Russell [Wed, 31 Aug 2011 06:09:17 +0000 (15:39 +0930)]
tdb2: test: Speed up run-01-new_database.

Don't run tdb_check with failtest on, since it is very slow.  Do the
tdb_check w/ failtest in a new test, but skip MMAP suppression which
doesn't add much and slows down valgrind a lot.

Before this change run-01-new_database took 40 seconds (under
valgrind), after it takes 8 seconds, and run-12-check takes 3

9 years agotdb2: speed up testing.
Rusty Russell [Wed, 31 Aug 2011 06:09:17 +0000 (15:39 +0930)]
tdb2: speed up testing.

The time to run tests under valgrind has become excessive; particularly
the failure tests which fork().   Thus we cut down testing:

1) api-94-repack: reduce from 234 seconds to 2 seconds by cutting
   iterations, despite adding TDB_VERSION1 tests.

2) api-missing-entries: reduce from 17 seconds to under 1 second by
   not checking db inside loop, but at end.

This reduces the total ccanlint time from 729 to 489 seconds.

9 years agotdb2: test: convert (non-invasive) run tests to api tests.
Rusty Russell [Wed, 31 Aug 2011 06:09:16 +0000 (15:39 +0930)]
tdb2: test: convert (non-invasive) run tests to api tests.

This reduces compilation time, since these are merely linked with the
pre-built module, rather than recompiling it into the test (which
allows for fancy things like failtest).

This reduces the test compile time down from about 62 seconds to 45
seconds.  Since ccanlint compiles tests three times (once normally,
once with coverage, and once with reduced config.h) this makes a
difference: we go from 780 seconds to 729 seconds.

9 years agotdb2: test: fix run-57-die-during-transaction.c to be more efficient.
Rusty Russell [Wed, 31 Aug 2011 06:09:16 +0000 (15:39 +0930)]
tdb2: test: fix run-57-die-during-transaction.c to be more efficient.

We track malloc and free, but we didn't catch the free() inside
external_agent, which means that our list of allocations keeps
growing.  Particularly under valgrind, which re-uses memory less than
the glibc allocator.

9 years agotdb2: don't continue if tdb1_find fails.
Rusty Russell [Wed, 31 Aug 2011 06:08:53 +0000 (15:38 +0930)]
tdb2: don't continue if tdb1_find fails.

The TDB1 code's tdb1_find() returns 0 on error; the callers should
not assume that the error means that the entry wasn't found, but use
last_error to determine it.

This was found by looking at how long the failure path testing for
test/run-10-simple-store.c was taking under valgrind, ie:

valgrind -q ./run-10-simple-store --show-slowest

This change dropped the time for that test from 53 seconds to 19

9 years agotdb2: test: try (almost) all tests with TDB_VERSION1 flag.
Rusty Russell [Wed, 31 Aug 2011 06:01:08 +0000 (15:31 +0930)]
tdb2: test: try (almost) all tests with TDB_VERSION1 flag.

There are some minor changes required, in particular:

1) Make sure lockcheck understands tdb1 allrecord lock upgrades.
2) Handle tdb1 sequence number jumps: various operations increment the
   sequence number twice, especually tdb_append.
3) Don't test fail on unlock, since it gets triggered with traversal on the
   tdb1 backend (we didn't actually ever test this case for tdb2).
4) Move clear_if_first to offset 4, to match tdb1.

9 years agotdb2: log allocation failures in tdb1 backend.
Rusty Russell [Wed, 31 Aug 2011 06:01:08 +0000 (15:31 +0930)]
tdb2: log allocation failures in tdb1 backend.

The TDB2 tests are stricter about this; they want every error logged.

9 years agotdb2: fix bogus error from tdb_unlockall() after fork in tdb1 backend.
Rusty Russell [Wed, 31 Aug 2011 06:01:08 +0000 (15:31 +0930)]
tdb2: fix bogus error from tdb_unlockall() after fork in tdb1 backend.

We're always allowed to unlock after a fork; by setting the count to 0
before calling the generic unlock function we don't trigger the pid

9 years agotdb2: check lock owner in tdb1 backend.
Rusty Russell [Wed, 31 Aug 2011 06:01:08 +0000 (15:31 +0930)]
tdb2: check lock owner in tdb1 backend.

This reports errors if we fork() while holding a lock, or misuse a tdb
which we have dual-opened.

9 years agotdb2: return TDB_ERR_RDONLY from transactions on R/O databases, log nesting
Rusty Russell [Wed, 31 Aug 2011 06:01:08 +0000 (15:31 +0930)]
tdb2: return TDB_ERR_RDONLY from transactions on R/O databases, log nesting

Rather than rturning TDB_ERR_EINVAL, return TDB_ERR_RDONLY, and log
when we fail due to nesting.

9 years agotdb2: add stats to tdb1 backend.
Rusty Russell [Wed, 31 Aug 2011 06:01:08 +0000 (15:31 +0930)]
tdb2: add stats to tdb1 backend.

It's actually quite a good fit; we use compare_wrong_bucket for dead
records, which is kind of correct (they should be in the free list).

9 years agotdb2: increment seqnum on tdb_wipe_all in tdb1 backend.
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: increment seqnum on tdb_wipe_all in tdb1 backend.

9 years agotdb2: log an error when calling tdb_store() on read-only TDB in tdb1 backend.
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: log an error when calling tdb_store() on read-only TDB in tdb1 backend.

9 years agotdb2: catch errors in tdb1_needs_recovery()
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: catch errors in tdb1_needs_recovery()

The tdb1 backend simply returns "true" if there's an error determining
if a tdb needs recovery.  But this leads failtest down a rabbit hole;
it's better to return the error at this case (and makes for better for
diagnostics, since they will come from the first fault, not later in

9 years agotdb2: add tdb_attribute_tdb1_max_dead
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: add tdb_attribute_tdb1_max_dead

This allows us to simulate the old "volatile" flag for tdb1.  It's not
necessary for tdb2.

As this is the last function in tdb1.h, we remove that file.

9 years agotdb2: expose and document tdb1_incompatible_hash.
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: expose and document tdb1_incompatible_hash.

9 years agotdb2: make tdb1_null static.
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: make tdb1_null static.

It's only used in two files, make it static in one and do it manually
in the other.

9 years agotdb2: tdb_repack
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: tdb_repack

Move the tdb1_repack() code into the core, make it general, rename to

It's generic code: copy database into temporary, wipe it, copy back.

9 years agotdb2: unify tdb1_wipe into tdb_wipe.
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: unify tdb1_wipe into tdb_wipe.

Switch on the TDB_VERSION1 flag.

9 years agotdb2: unify tdb1_get_seqnum/tdb1_increment_seqnum into tdb_get_seqnum/tdb_inc_seqnum
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: unify tdb1_get_seqnum/tdb1_increment_seqnum into tdb_get_seqnum/tdb_inc_seqnum

Switch on the TDB_VERSION1 flag.

9 years agotdb2: unify tdb1_firstkey/tdb1_nextkey into tdb_firstkey/tdb_nextkey.
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: unify tdb1_firstkey/tdb1_nextkey into tdb_firstkey/tdb_nextkey.

Switch on the TDB_VERSION1 flag.

9 years agotdb2: unify tdb1_check and tdb1_summary into tdb_check and tdb_summary.
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: unify tdb1_check and tdb1_summary into tdb_check and tdb_summary.

Switch on the TDB_VERSION1 flag.  Also, change tdb1_check's checkfn argument
to return an error code (and set tdb->last_error accordingly).

9 years agotdb2: unify tdb1_transaction_start etc. into tdb_transaction_start.
Rusty Russell [Wed, 31 Aug 2011 06:01:07 +0000 (15:31 +0930)]
tdb2: unify tdb1_transaction_start etc. into tdb_transaction_start.

Switch on the TDB_VERSION1 flag.

9 years agotdb2: unify tdb1_parse_record into tdb_parse_record
Rusty Russell [Wed, 31 Aug 2011 06:01:06 +0000 (15:31 +0930)]
tdb2: unify tdb1_parse_record into tdb_parse_record

Switch on the TDB_VERSION1 flag.

9 years agotdb2: unify tdb1_chainlock et al. into tdb_chainlock
Rusty Russell [Wed, 31 Aug 2011 06:01:06 +0000 (15:31 +0930)]
tdb2: unify tdb1_chainlock et al. into tdb_chainlock

Switch on the TDB_VERSION1 flag.