ccan
9 years agotalloc: fixed a use after free error
Rusty Russell [Mon, 10 Jan 2011 05:32:03 +0000 (16:02 +1030)]
talloc: fixed a use after free error

(Import from SAMBA commit 6f51a1f45bf4de062cce7a562477e8140630a53d):

this is the minimal fix for the problem Rusty found. I previously
thought that the best fix would be to change tc->parent to be valid
for all pointers, but that is expensive for realloc with large numbers
of child pointers, which is much more commmon than I expected it to
be.

9 years agotalloc: use failtest to test failure paths.
Rusty Russell [Mon, 10 Jan 2011 03:31:47 +0000 (14:01 +1030)]
talloc: use failtest to test failure paths.

9 years agofailtest: new module.
Rusty Russell [Mon, 10 Jan 2011 04:12:38 +0000 (14:42 +1030)]
failtest: new module.

A module designed to help test "never fails" functions like malloc.

9 years agotap: add fail callback
Rusty Russell [Mon, 10 Jan 2011 05:30:48 +0000 (16:00 +1030)]
tap: add fail callback

This is useful for failtest, so we can abort on first failure.

9 years agoccanlint: fix and simplify depends-accurate (with strreg)
Rusty Russell [Sun, 9 Jan 2011 01:26:41 +0000 (11:56 +1030)]
ccanlint: fix and simplify depends-accurate (with strreg)

9 years agoccanlint: fix total score for running examples
Rusty Russell [Sat, 8 Jan 2011 09:39:25 +0000 (20:09 +1030)]
ccanlint: fix total score for running examples

Don't count examples which didn't compile (expected, as we mangle them in
various different ways).

9 years agoccanlint: fix uninitialized variable
Rusty Russell [Sat, 8 Jan 2011 10:13:59 +0000 (20:43 +1030)]
ccanlint: fix uninitialized variable

9 years agoccanlint: use strreg for section extraction.
Rusty Russell [Sat, 8 Jan 2011 08:07:19 +0000 (18:37 +1030)]
ccanlint: use strreg for section extraction.

Makes it simpler and clearer.

9 years agoMakefile: append git revision to "make scores"
Rusty Russell [Sat, 8 Jan 2011 08:06:56 +0000 (18:36 +1030)]
Makefile: append git revision to "make scores"

Good for testing ccanlint changes.

9 years agostr_talloc: strreg
Rusty Russell [Sat, 8 Jan 2011 02:45:35 +0000 (13:15 +1030)]
str_talloc: strreg

Useful wrapper for extended POSIX regular expressions.

9 years agostr: strcount
Rusty Russell [Fri, 7 Jan 2011 23:47:37 +0000 (10:17 +1030)]
str: strcount

Useful routine to count number of matches in a string.

9 years agostr: clean up tests so ccanlint doesn't complain about memory leaking.
Rusty Russell [Thu, 6 Jan 2011 03:50:26 +0000 (14:20 +1030)]
str: clean up tests so ccanlint doesn't complain about memory leaking.

9 years agoccanlint: use positive description for test_pass_valgrind_noleaks
Rusty Russell [Fri, 7 Jan 2011 11:51:29 +0000 (22:21 +1030)]
ccanlint: use positive description for test_pass_valgrind_noleaks

A bit more awkward, but more consistent with everything else.

9 years agoccanlint: print keys in output
Rusty Russell [Fri, 7 Jan 2011 11:50:44 +0000 (22:20 +1030)]
ccanlint: print keys in output

Since test keys are used for --target=, this is useful.

9 years agoccanlint: rename files to match keys
Rusty Russell [Fri, 7 Jan 2011 11:50:13 +0000 (22:20 +1030)]
ccanlint: rename files to match keys

9 years agoccanlint: rename structures to match keys
Rusty Russell [Fri, 7 Jan 2011 11:50:04 +0000 (22:20 +1030)]
ccanlint: rename structures to match keys

9 years agoccanlint: list dependencies by key
Rusty Russell [Fri, 7 Jan 2011 11:49:49 +0000 (22:19 +1030)]
ccanlint: list dependencies by key

Joey Adams also pointed out that we should use strings for the dependency
lists.  Moving them into the structure also somewhat simplifies it.

9 years agoccanlint: rename test keys
Rusty Russell [Fri, 7 Jan 2011 11:48:41 +0000 (22:18 +1030)]
ccanlint: rename test keys

Joey Adams rightly points out that the current keys are a mess: ideally the
filenames, test keys and structure names in ccanlint should be the same.

First step is to make the test names all regular, of basic form <noun>_<verb>
(eg "tests_exist" rather than "has-tests").

9 years agoAdd scores/ directory to .gitignore.
Rusty Russell [Fri, 7 Jan 2011 02:52:43 +0000 (13:22 +1030)]
Add scores/ directory to .gitignore.

9 years agoccanlint: added --test-dep-graph option
Joey Adams [Thu, 6 Jan 2011 20:35:51 +0000 (15:35 -0500)]
ccanlint: added --test-dep-graph option

This option prints the dependency graph of ccanlint's tests
in Graphviz .dot format.

Sample usage:

ccanlint --test-dep-graph | dot -Tpng > out.png && eog out.png

9 years agoccanlint: fix segfault caused by tests not depending on the "info" test.
Joey Adams [Thu, 6 Jan 2011 20:12:18 +0000 (15:12 -0500)]
ccanlint: fix segfault caused by tests not depending on the "info" test.

These tests:

"depends-exist"      (compulsory_tests/check_depends_exist.c)
"info-documentation" (tests/has_info_documentation.c)

used m->info_file without checking if it was NULL,
leading to a segfault when no _info file was present.

Some other tests also used m->info_file without depending on "info",
but are taken care of indirectly by this patch.

9 years agoopt: Fix warnings with gcc-4.5 (same approach as commit 6535bde)
Joey Adams [Thu, 6 Jan 2011 15:50:04 +0000 (10:50 -0500)]
opt: Fix warnings with gcc-4.5 (same approach as commit 6535bde)

&*ptr is used in some other macros, but at a glance, they look like
cases where the pointer shouldn't be NULL .  Didn't change those,
and if we get more warnings, we'll cross that bridge when we get to it.
For now, I suppose they are just free NULL checks.

9 years agodaemonize: set stderr to /dev/null.
Rusty Russell [Thu, 6 Jan 2011 01:54:05 +0000 (12:24 +1030)]
daemonize: set stderr to /dev/null.

9 years agodaemonize: make valgrind happy.
Rusty Russell [Thu, 6 Jan 2011 01:45:03 +0000 (12:15 +1030)]
daemonize: make valgrind happy.

9 years agodaemonize: use BSD-MIT as License: string in _info
Rusty Russell [Thu, 6 Jan 2011 01:37:37 +0000 (12:07 +1030)]
daemonize: use BSD-MIT as License: string in _info

The parenthesized thing is confusing and ccanlint dislikes it.

9 years agoccanlint: allow BSD-MIT for MIT license.
Rusty Russell [Thu, 6 Jan 2011 01:37:05 +0000 (12:07 +1030)]
ccanlint: allow BSD-MIT for MIT license.

There are a large number of BSD variants out there, be explicit.

9 years agoccanlint: fix parsing bug which believes lines starting with - are a section header.
Rusty Russell [Thu, 6 Jan 2011 00:56:47 +0000 (11:26 +1030)]
ccanlint: fix parsing bug which believes lines starting with - are a section header.

9 years agoccanlint: have valgrind fail with an error, always
Rusty Russell [Tue, 4 Jan 2011 10:41:47 +0000 (21:11 +1030)]
ccanlint: have valgrind fail with an error, always

The upcoming failtest module can only tell that a child failed when it
exits with a non-zero error.  So we need this, although it means for ccanlint
it still needs to look at output to distinguish a memory leak from a real
error.

9 years agoccanlint: compile modules required by examples.
Rusty Russell [Tue, 4 Jan 2011 10:41:47 +0000 (21:11 +1030)]
ccanlint: compile modules required by examples.

If an example #includes <ccan/foo/...> we assume it needs module foo,
but we would fail instead of building it if it isn't built.

9 years agoccanlint: make get_manifest cache manifests
Rusty Russell [Tue, 4 Jan 2011 10:41:47 +0000 (21:11 +1030)]
ccanlint: make get_manifest cache manifests

As we start doing more building of dependencies, this saves us effort.

9 years agocompiler: NORETURN
Rusty Russell [Tue, 4 Jan 2011 10:42:41 +0000 (21:12 +1030)]
compiler: NORETURN

9 years agotlist: typesafe variant of list module.
Rusty Russell [Sat, 1 Jan 2011 06:44:05 +0000 (17:14 +1030)]
tlist: typesafe variant of list module.

I chose not to do the "macro creates set of routines" approach, as
we can be almost as safe with a struct containing a pointer to the member
type.

9 years agolist: LIST_HEAD_INIT
Rusty Russell [Sat, 1 Jan 2011 06:43:04 +0000 (17:13 +1030)]
list: LIST_HEAD_INIT

I find hiding the declaration in LIST_HEAD() a bit weird.

9 years agolist: improve test coverage to 100%
Rusty Russell [Sat, 1 Jan 2011 06:42:40 +0000 (17:12 +1030)]
list: improve test coverage to 100%

In particular, we test the corruption cases.

9 years agolist: list_del_from()
Rusty Russell [Sat, 1 Jan 2011 06:41:15 +0000 (17:11 +1030)]
list: list_del_from()

Deletion from a specific list.

9 years agolist: test CCAN_LIST_DEBUG=1 case
Rusty Russell [Sat, 1 Jan 2011 06:45:10 +0000 (17:15 +1030)]
list: test CCAN_LIST_DEBUG=1 case

9 years agolist: rename debug_list to list_debug, use list_check_node in list_del
Rusty Russell [Sat, 1 Jan 2011 06:40:54 +0000 (17:10 +1030)]
list: rename debug_list to list_debug, use list_check_node in list_del

When CCAN_LIST_DEBUG was defined, we were previously calling
list_check() in list_del instead of list_check_node(), which caused a
warning.

We also should stick within the "list_" prefix namespace, so rename
"debug_list" to "list_debug".

9 years agolist: implement list_check_node to check list from node rather than head
Rusty Russell [Sat, 1 Jan 2011 06:38:58 +0000 (17:08 +1030)]
list: implement list_check_node to check list from node rather than head

9 years agoconfig.h: HAVE_FLEXIBLE_ARRAY_MEMBER
Rusty Russell [Sat, 1 Jan 2011 06:45:41 +0000 (17:15 +1030)]
config.h: HAVE_FLEXIBLE_ARRAY_MEMBER

9 years agoccanlint: give a bonus for 100% code coverage
Rusty Russell [Sat, 1 Jan 2011 06:42:00 +0000 (17:12 +1030)]
ccanlint: give a bonus for 100% code coverage

9 years agotdb: add test for tdb_summary
Rusty Russell [Thu, 23 Dec 2010 02:35:55 +0000 (13:05 +1030)]
tdb: add test for tdb_summary

9 years agotdb: remove tally dependency
Rusty Russell [Thu, 23 Dec 2010 02:14:48 +0000 (12:44 +1030)]
tdb: remove tally dependency

We lose the histograms, but this prepares it for upstream merge.

9 years agotdb: fix uncoalesced record count in tdb_summary.
Rusty Russell [Thu, 23 Dec 2010 02:14:25 +0000 (12:44 +1030)]
tdb: fix uncoalesced record count in tdb_summary.

One free record is not "uncoalesced".

9 years agotdb: combine dead_space helper from summary.c and check.c
Rusty Russell [Thu, 23 Dec 2010 02:13:54 +0000 (12:43 +1030)]
tdb: combine dead_space helper from summary.c and check.c

Remove code duplication, and allow unit tests (which #include both).

9 years agotypesafe_cb: Fix warnings with gcc-4.5:
Rusty Russell [Thu, 16 Dec 2010 05:33:41 +0000 (16:03 +1030)]
typesafe_cb: Fix warnings with gcc-4.5:

Test compiled with warnings:
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c: In function ‘main’:
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:18:2: warning: taking address of expression of type ‘void’
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:18:2: warning: taking address of expression of type ‘void’
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:18:2: warning: taking address of expression of type ‘void’
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:18:2: warning: taking address of expression of type ‘void’
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:18:2: warning: taking address of expression of type ‘void’
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:18:2: warning: taking address of expression of type ‘void’
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:18:2: warning: taking address of expression of type ‘void’
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/test/compile_ok-typesafe_cb-NULL.c:19:2: warning: taking address of expression of type ‘void’

9 years agoccanlint: use valgrind options when debugging too
Rusty Russell [Mon, 13 Dec 2010 08:15:24 +0000 (18:45 +1030)]
ccanlint: use valgrind options when debugging too

If they specify valgrind options in their _info, we should use them when
running with --db-attach=yes, too.

9 years agoccanlint: report valgrind "definite" leaks.
Rusty Russell [Mon, 13 Dec 2010 08:15:25 +0000 (18:45 +1030)]
ccanlint: report valgrind "definite" leaks.

This is complicated by valgrind's limited options, and our desire not to run
valgrind twice (it's already the slowest part of the tests).

Ideally I'd like a different error code for each kind of error.  I
could parse and pretty-print the XML output, but instead I just parse
the human-readable (which is fragile).

9 years agoccanlint: fix bogus strerror printing when unknown test handed to --target
Rusty Russell [Mon, 13 Dec 2010 08:15:25 +0000 (18:45 +1030)]
ccanlint: fix bogus strerror printing when unknown test handed to --target

9 years agoccanlint: fix error with --target=build
Rusty Russell [Mon, 13 Dec 2010 08:23:14 +0000 (18:53 +1030)]
ccanlint: fix error with --target=build

It requires that we build the objects first.

9 years agorbtree: remove unused variable in tests.
Rusty Russell [Wed, 8 Dec 2010 02:05:46 +0000 (12:35 +1030)]
rbtree: remove unused variable in tests.

9 years agorb_tree: fix trbt_delete
Ronnie Sahlberg [Wed, 8 Dec 2010 01:00:35 +0000 (12:00 +1100)]
rb_tree: fix trbt_delete

trbt_delete32() was broken and caused SEGV as soon as you tried to
delete an object from a tree.

Rework trbt_delete32() to instead just call talloc_free() instread of trying
to call delete_node() directly.
This makes the "from_destructor" argument to delete_node() redundant
so that parameter is removed too.

Signed-off-by: Ronnie Sahlberg <sahlberg@lenovo-laptop.(none)>
9 years agoconfigurator: warnings count as failures too.
Rusty Russell [Wed, 8 Dec 2010 00:44:21 +0000 (11:14 +1030)]
configurator: warnings count as failures too.

Unfortunately, gcc only warns if it sees an unknown attribute (in this case, gcc 4.1 vs "cold").

9 years agoccanlint: fix compile on x86-64
Rusty Russell [Wed, 8 Dec 2010 00:34:03 +0000 (11:04 +1030)]
ccanlint: fix compile on x86-64

cc -g -Wall -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -I. -MD -Werror   -c -o tools/ccanlint/tests/examples_run.o tools/ccanlint/tests/examples_run.c
cc1: warnings being treated as errors
tools/ccanlint/tests/examples_run.c: In function ‘scan_forv’:
tools/ccanlint/tests/examples_run.c:37: warning: passing argument 2 of ‘__builtin_va_copy’ discards qualifiers from pointer target type
tools/ccanlint/tests/examples_run.c:43: warning: passing argument 4 of ‘scan_forv’ from incompatible pointer type
tools/ccanlint/tests/examples_run.c:52: warning: passing argument 4 of ‘scan_forv’ from incompatible pointer type
tools/ccanlint/tests/examples_run.c:60: warning: passing argument 4 of ‘scan_forv’ from incompatible pointer type
tools/ccanlint/tests/examples_run.c: In function ‘scan_for’:
tools/ccanlint/tests/examples_run.c:78: warning: passing argument 4 of ‘scan_forv’ from incompatible pointer type
make: *** [tools/ccanlint/tests/examples_run.o] Error 1

It really doesn't like constifying a va_arg, so remove the const declaration.

9 years agorbtree: add tests (which currently fail)
Rusty Russell [Tue, 7 Dec 2010 05:19:08 +0000 (15:49 +1030)]
rbtree: add tests (which currently fail)

9 years agoidtree: fix overflow for v. large ids on allocation and removal
Rusty Russell [Mon, 6 Dec 2010 03:12:13 +0000 (13:42 +1030)]
idtree: fix overflow for v. large ids on allocation and removal

Chris Cowan tracked down a SEGV in sub_alloc: idp->level can actually
be equal to 7 (MAX_LEVEL) there, as it can be in sub_remove.

9 years agoidtree: fix right shift of signed ints
Rusty Russell [Mon, 6 Dec 2010 03:12:00 +0000 (13:42 +1030)]
idtree: fix right shift of signed ints

(Imported from SAMBA commit 2db1987f5a3a)

Right-shifting signed integers in undefined; indeed it seems that on
AIX with their compiler, doing a 30-bit shift on (INT_MAX-200) gives
0, not 1 as we might expect (THIS WAS WRONG, REAL FIX LATER).

The obvious fix is to make id and oid unsigned: l (level count) is also
logically unsigned.

(Note: Samba doesn't generally get to ids > 1 billion, but ctdb does)

Reported-by: Chris Cowan <cc@us.ibm.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoidtree: import fix from SAMBA source
Rusty Russell [Mon, 6 Dec 2010 03:00:59 +0000 (13:30 +1030)]
idtree: import fix from SAMBA source

(Imported from b53f8c187de8)

Author: Rusty Russell <rusty@rustorp.com.au>
Date:   Thu Jun 10 13:27:51 2010 -0700

Since idtree assigns sequentially, it rarely reaches high numbers.
But such numbers can be forced with idr_get_new_above(), and that
reveals two bugs:

1) Crash in sub_remove() caused by pa array being too short.
2) Shift by more than 32 in _idr_find(), which is undefined, causing
   the "outside the current tree" optimization to misfire and return NULL.

9 years agoidtree: add unit test for ids around INT_MAX
Rusty Russell [Mon, 6 Dec 2010 02:22:19 +0000 (12:52 +1030)]
idtree: add unit test for ids around INT_MAX

This causes a SEGV on my laptop.

9 years agoccanlint: ignore comments when parsing examples.
Rusty Russell [Mon, 6 Dec 2010 02:21:40 +0000 (12:51 +1030)]
ccanlint: ignore comments when parsing examples.

We insert comments when we massage or combine examples; don't let these
throw off our analysis (as happened for idtree.h).

9 years agoccanlint: show example we actually compiled when we report warnings.
Rusty Russell [Mon, 6 Dec 2010 02:20:45 +0000 (12:50 +1030)]
ccanlint: show example we actually compiled when we report warnings.

9 years agoccanlint: examples_compile depends on build, which depends on build_objs
Rusty Russell [Mon, 6 Dec 2010 02:19:31 +0000 (12:49 +1030)]
ccanlint: examples_compile depends on build, which depends on build_objs

We might as well use the compiled .o rather than all the little .o files.

9 years agoccanlint: fix usage of static funcs
Rusty Russell [Mon, 6 Dec 2010 02:18:18 +0000 (12:48 +1030)]
ccanlint: fix usage of static funcs

Out-by-one error had us using character prior to declaration, eg, in
"static int *foo" we use "*foo".  This seems to compile, but is weird.

9 years agoarray: fix type introduced in handling !HAVE_TYPEOF
Rusty Russell [Fri, 3 Dec 2010 12:23:52 +0000 (22:53 +1030)]
array: fix type introduced in handling !HAVE_TYPEOF

Commit da72623aec30 added a typo; ccanlint caught it, but doesn't consider
test compile failing to be fatal (it should!).

9 years agoMerge branch 'tdb2'
Rusty Russell [Wed, 1 Dec 2010 13:24:33 +0000 (23:54 +1030)]
Merge branch 'tdb2'

9 years agotdb2: update documentation
Rusty Russell [Wed, 1 Dec 2010 13:16:26 +0000 (23:46 +1030)]
tdb2: update documentation

Specifically the linked free tables, and reflect on the status of each
point of the design document.

9 years agotdb2: use separate magic constants for chain, htable and ftable entries
Rusty Russell [Wed, 1 Dec 2010 13:15:56 +0000 (23:45 +1030)]
tdb2: use separate magic constants for chain, htable and ftable entries

Rather than overloading TDB_USED_MAGIC and the hash value as we do now.
We also rename "free list" to the more-accurate "free table" everywhere.

9 years agotdb2: direct access during transactions.
Rusty Russell [Wed, 1 Dec 2010 13:14:54 +0000 (23:44 +1030)]
tdb2: direct access during transactions.

Currently we fall back to copying data during a transaction, but we don't
need to in many cases.  Grant direct access in those cases.

Before:
$ ./speed --transaction 1000000
Adding 1000000 records:  2409 ns (59916680 bytes)
Finding 1000000 records:  1156 ns (59916680 bytes)
Missing 1000000 records:  604 ns (59916680 bytes)
Missing 1000000 records:  604 ns (59916680 bytes)
Traversing 1000000 records:  1226 ns (59916680 bytes)
Deleting 1000000 records:  1556 ns (119361928 bytes)
Re-adding 1000000 records:  2326 ns (119361928 bytes)
Appending 1000000 records:  3269 ns (246656880 bytes)
Churning 1000000 records:  5613 ns (338235248 bytes)

After:
$ ./speed --transaction 1000000
Adding 1000000 records:  1902 ns (59916680 bytes)
Finding 1000000 records:  1032 ns (59916680 bytes)
Missing 1000000 records:  606 ns (59916680 bytes)
Missing 1000000 records:  606 ns (59916680 bytes)
Traversing 1000000 records:  741 ns (59916680 bytes)
Deleting 1000000 records:  1347 ns (119361928 bytes)
Re-adding 1000000 records:  1727 ns (119361928 bytes)
Appending 1000000 records:  2561 ns (246656880 bytes)
Churning 1000000 records:  4403 ns (338235248 bytes)

9 years agotdb2: add write flag to tdb_direct
Rusty Russell [Wed, 1 Dec 2010 13:14:14 +0000 (23:44 +1030)]
tdb2: add write flag to tdb_direct

This is a precursor to direct access during transactions: they care about
whether we are going to read or write to the file.

9 years agotdb2: hash chaining
Rusty Russell [Wed, 1 Dec 2010 13:13:37 +0000 (23:43 +1030)]
tdb2: hash chaining

If we get enough hash collisions, we can run out of hash bits; this almost
certainly is caused by a deliberate attempt to break the tdb (hash bombing).

Implement chained records for this case; they're slow but will keep the
rest of the database functioning.

9 years agotdb2: use magic freetable value rather than different magic for coalescing
Rusty Russell [Wed, 1 Dec 2010 13:11:26 +0000 (23:41 +1030)]
tdb2: use magic freetable value rather than different magic for coalescing

We have to unlock during coalescing, so we mark records specially to indicate
to tdb_check that they're not on any list, and to prevent other coalescers
from grabbing them.

Use a special free list number, rather than a new magic.

9 years agotdb2: compare the extra 11 hash bits in the header.
Rusty Russell [Wed, 1 Dec 2010 13:10:05 +0000 (23:40 +1030)]
tdb2: compare the extra 11 hash bits in the header.

We already have 10 hash bits encoded in the offset itself; we only get
here incorrectly about 1 time in 1000, so it's a pretty minor
optimization at best.

Nonetheless, we have the information, so let's check it before
accessing the key.  This reduces the probability of a false keycmp by
another factor of 2000.

9 years agotdb2: add comparison stats
Rusty Russell [Wed, 1 Dec 2010 12:53:50 +0000 (23:23 +1030)]
tdb2: add comparison stats

9 years agotdb2: clean up logging
Rusty Russell [Wed, 1 Dec 2010 12:41:51 +0000 (23:11 +1030)]
tdb2: clean up logging

Logged errors should always set tdb->ecode before they are called, and
there's little reason to have a sprintf-style logging function since
we can do the formatting internally.

Change the tdb_log attribute to just take a "const char *", and create
a new tdb_logerr() helper which sets ecode and calls it.  As a bonus,
mark it COLD so the compiler can optimize appropriately knowing that
it's unlikely to be invoked.

9 years agotdb2: remove truncated bit.
Rusty Russell [Wed, 1 Dec 2010 12:38:52 +0000 (23:08 +1030)]
tdb2: remove truncated bit.

There was an idea that we would use a bit to indicate that we didn't have
the full hash value; this would allow us to move records down when we
expanded a hash without rehashing them.

There's little evidence that rehashing in this case is particularly
expensive, so remove the bit.  We use that bit simply to indicate that
an offset refers to a subhash instead.

9 years agotdb2: use direct access for tdb_read_off/tdb_write_off
Rusty Russell [Wed, 1 Dec 2010 12:36:12 +0000 (23:06 +1030)]
tdb2: use direct access for tdb_read_off/tdb_write_off

This is one case where getting rid of tdb_get() cost us.  Also, we
add more read-only checks.

Before we removed tdb_get:
Adding 1000000 records:  6480 ns (59900296 bytes)
Finding 1000000 records:  2839 ns (59900296 bytes)
Missing 1000000 records:  2485 ns (59900296 bytes)
Traversing 1000000 records:  2598 ns (59900296 bytes)
Deleting 1000000 records:  5342 ns (59900296 bytes)
Re-adding 1000000 records:  5613 ns (59900296 bytes)
Appending 1000000 records:  12194 ns (93594224 bytes)
Churning 1000000 records:  14549 ns (93594224 bytes)

Now:
Adding 1000000 records:  6307 ns (59900296 bytes)
Finding 1000000 records:  2801 ns (59900296 bytes)
Missing 1000000 records:  2515 ns (59900296 bytes)
Traversing 1000000 records:  2579 ns (59900296 bytes)
Deleting 1000000 records:  5225 ns (59900296 bytes)
Re-adding 1000000 records:  5878 ns (59900296 bytes)
Appending 1000000 records:  12665 ns (93594224 bytes)
Churning 1000000 records:  16090 ns (93594224 bytes)

9 years agotdb2: remove tdb_get()
Rusty Russell [Wed, 1 Dec 2010 12:34:50 +0000 (23:04 +1030)]
tdb2: remove tdb_get()

We have four internal helpers for reading data from the database:
1) tdb_read_convert() - read (and convert) into a buffer.
2) tdb_read_off() - read (and convert) and offset.
3) tdb_access_read() - malloc or direct access to the database.
4) tdb_get() - copy into a buffer or direct access to the database.

The last one doesn't really buy us anything, so remove it (except for
tdb_read_off/tdb_write_off, see next patch).

Before:
Adding 1000000 records:  6480 ns (59900296 bytes)
Finding 1000000 records:  2839 ns (59900296 bytes)
Missing 1000000 records:  2485 ns (59900296 bytes)
Traversing 1000000 records:  2598 ns (59900296 bytes)
Deleting 1000000 records:  5342 ns (59900296 bytes)
Re-adding 1000000 records:  5613 ns (59900296 bytes)
Appending 1000000 records:  12194 ns (93594224 bytes)
Churning 1000000 records:  14549 ns (93594224 bytes)

After:
Adding 1000000 records:  6497 ns (59900296 bytes)
Finding 1000000 records:  2854 ns (59900296 bytes)
Missing 1000000 records:  2563 ns (59900296 bytes)
Traversing 1000000 records:  2735 ns (59900296 bytes)
Deleting 1000000 records:  11357 ns (59900296 bytes)
Re-adding 1000000 records:  8145 ns (59900296 bytes)
Appending 1000000 records:  10939 ns (93594224 bytes)
Churning 1000000 records:  18479 ns (93594224 bytes)

9 years agoccanlint: build depends if necessary
Rusty Russell [Tue, 23 Nov 2010 12:49:19 +0000 (23:19 +1030)]
ccanlint: build depends if necessary

Now it will build copies of other ccan deps if it can't find them.

9 years agotdb2: trivial optimization for free list
Rusty Russell [Tue, 23 Nov 2010 02:21:18 +0000 (12:51 +1030)]
tdb2: trivial optimization for free list

We currently only have one, so shortcut the case where we want our current
one.

9 years agotdb2: Add stats attribute.
Rusty Russell [Wed, 1 Dec 2010 13:06:14 +0000 (23:36 +1030)]
tdb2: Add stats attribute.

This is good for deep debugging.

9 years agotdb2: enable transactions in tdbtorture
Rusty Russell [Tue, 23 Nov 2010 01:44:31 +0000 (12:14 +1030)]
tdb2: enable transactions in tdbtorture

9 years agotdb2: stricter ordering on expansion lock
Rusty Russell [Tue, 23 Nov 2010 01:43:59 +0000 (12:13 +1030)]
tdb2: stricter ordering on expansion lock

It's problematic for transaction commit to get the expansion lock, but
in fact we always grab a hash lock before the transaction lock, so it
doesn't really need it (the transaction locks the entire database).

Assert that this is true, and fix up a few lowlevel tests where it wasn't.

9 years agotdb2: remove all the dead code
Rusty Russell [Tue, 23 Nov 2010 01:40:32 +0000 (12:10 +1030)]
tdb2: remove all the dead code

I left much tdb1 code in various files for inspiration, and in case I needed
it later.  Now we have all the major features implemented, remove it.

9 years agotdb2: transaction support
Rusty Russell [Tue, 23 Nov 2010 01:38:21 +0000 (12:08 +1030)]
tdb2: transaction support

This adds transactions to tdb2; the code is taken from tdb1 with minimal
modifications, as are the unit

9 years agotdb2: allow nesting of read locks on top of write locks.
Rusty Russell [Tue, 23 Nov 2010 01:37:29 +0000 (12:07 +1030)]
tdb2: allow nesting of read locks on top of write locks.

If we have a write lock and ask for a read lock, that's OK, but not the
other way around.  tdb_nest_lock() allowed both, tdb_allrecord_lock() allowed
neither.

9 years agoccanlint: fix -x core dump
Rusty Russell [Tue, 23 Nov 2010 01:36:31 +0000 (12:06 +1030)]
ccanlint: fix -x core dump

This wasn't fixed when we converted to ccan/opt in 8d70667887.

Unfortunately, unistd.h defines optarg, so the compiler didn't catch
this.

9 years agotdb2: relax locking to allow two free list locks at once
Rusty Russell [Mon, 22 Nov 2010 02:13:00 +0000 (12:43 +1030)]
tdb2: relax locking to allow two free list locks at once

As long as they are in descending order.  This prevents the common case of:

1) Grab lock for bucket.
2) Remove entry from bucket.
3) Drop lock for bucket.
4) Grab lock for bucket for leftover.
5) Add leftover entry to bucket.
6) Drop lock for leftover bucket.

In particular it's quite common for the leftover bucket to be the same
as the entry bucket (when it's the largest bucket); if it's not, we are
no worse than before.

Current results of speed test:
$ ./speed 1000000
Adding 1000000 records:  13194 ns (60083192 bytes)
Finding 1000000 records:  2438 ns (60083192 bytes)
Traversing 1000000 records:  2167 ns (60083192 bytes)
Deleting 1000000 records:  9265 ns (60083192 bytes)
Re-adding 1000000 records:  10241 ns (60083192 bytes)
Appending 1000000 records:  17692 ns (93879992 bytes)
Churning 1000000 records:  26077 ns (93879992 bytes)

Previous:
$ ./speed 1000000
Adding 1000000 records:  23210 ns (59193360 bytes)
Finding 1000000 records:  2387 ns (59193360 bytes)
Traversing 1000000 records:  2150 ns (59193360 bytes)
Deleting 1000000 records:  13392 ns (59193360 bytes)
Re-adding 1000000 records:  11546 ns (59193360 bytes)
Appending 1000000 records:  29327 ns (91193360 bytes)
Churning 1000000 records:  33026 ns (91193360 bytes)

9 years agotdb2: use expansion heuristics from tdb1
Rusty Russell [Mon, 22 Nov 2010 01:20:58 +0000 (11:50 +1030)]
tdb2: use expansion heuristics from tdb1

This reduces the amount of expansion we do.

Before:
./speed 1000000
Adding 1000000 records:  23210 ns (59193360 bytes)
Finding 1000000 records:  2387 ns (59193360 bytes)
Traversing 1000000 records:  2150 ns (59193360 bytes)
Deleting 1000000 records:  13392 ns (59193360 bytes)
Re-adding 1000000 records:  11546 ns (59193360 bytes)
Appending 1000000 records:  29327 ns (91193360 bytes)
Churning 1000000 records:  33026 ns (91193360 bytes)

After:
$ ./speed 1000000
Adding 1000000 records:  17480 ns (61472904 bytes)
Finding 1000000 records:  2431 ns (61472904 bytes)
Traversing 1000000 records:  2194 ns (61472904 bytes)
Deleting 1000000 records:  10948 ns (61472904 bytes)
Re-adding 1000000 records:  11247 ns (61472904 bytes)
Appending 1000000 records:  21826 ns (96051424 bytes)
Churning 1000000 records:  27242 ns (96051424 bytes)

9 years agotdb2: shrink free header from 32 to 24 bytes.
Rusty Russell [Wed, 1 Dec 2010 13:22:43 +0000 (23:52 +1030)]
tdb2: shrink free header from 32 to 24 bytes.

This reduces our minimum key+data length to 8 bytes; we do this by packing
the prev pointer where we used to put the flist pointer, and storing the
flist as an 8 bit index (meaning we can only have 256 free tables).

Note that this has a perverse result on the size of the database, as our
4-byte key and 4-byte data now fit perfectly in a minimal record, so
appeding causes us to allocate new records which are 50% larger,
since we detect growing.

Current results of speed test:
$ ./speed 1000000
Adding 1000000 records:  23210 ns (59193360 bytes)
Finding 1000000 records:  2387 ns (59193360 bytes)
Traversing 1000000 records:  2150 ns (59193360 bytes)
Deleting 1000000 records:  13392 ns (59193360 bytes)
Re-adding 1000000 records:  11546 ns (59193360 bytes)
Appending 1000000 records:  29327 ns (91193360 bytes)
Churning 1000000 records:  33026 ns (91193360 bytes)

Previous:
$ ./speed 1000000
Adding 1000000 records:  28324 ns (67232528 bytes)
Finding 1000000 records:  2468 ns (67232528 bytes)
Traversing 1000000 records:  2200 ns (67232528 bytes)
Deleting 1000000 records:  13083 ns (67232528 bytes)
Re-adding 1000000 records:  16433 ns (67232528 bytes)
Appending 1000000 records:  2511 ns (67232528 bytes)
Churning 1000000 records:  31068 ns (67570448 bytes)

9 years agotdb2: rename set_header to the more appropriate set_used_header.
Rusty Russell [Tue, 23 Nov 2010 03:10:59 +0000 (13:40 +1030)]
tdb2: rename set_header to the more appropriate set_used_header.

9 years agotdb2: Add speed test to tdb and tdb2
Rusty Russell [Wed, 1 Dec 2010 12:52:55 +0000 (23:22 +1030)]
tdb2: Add speed test to tdb and tdb2

Current results of speed test:
$ ./speed 1000000
Adding 1000000 records:  14726 ns (67244816 bytes)
Finding 1000000 records:  2844 ns (67244816 bytes)
Missing 1000000 records:  2528 ns (67244816 bytes)
Traversing 1000000 records:  2572 ns (67244816 bytes)
Deleting 1000000 records:  5358 ns (67244816 bytes)
Re-adding 1000000 records:  9176 ns (67244816 bytes)
Appending 1000000 records:  3035 ns (67244816 bytes)
Churning 1000000 records:  18139 ns (67565840 bytes)
$ ./speed 100000
Adding 100000 records:  13270 ns (14349584 bytes)
Finding 100000 records:  2769 ns (14349584 bytes)
Missing 100000 records:  2422 ns (14349584 bytes)
Traversing 100000 records:  2595 ns (14349584 bytes)
Deleting 100000 records:  5331 ns (14349584 bytes)
Re-adding 100000 records:  5875 ns (14349584 bytes)
Appending 100000 records:  2751 ns (14349584 bytes)
Churning 100000 records:  20666 ns (25771280 bytes)

vs tdb1 (with hashsize 100003):
$ ./speed 1000000
Adding 1000000 records:  8547 ns (44306432 bytes)
Finding 1000000 records:  5595 ns (44306432 bytes)
Missing 1000000 records:  3469 ns (44306432 bytes)
Traversing 1000000 records:  4571 ns (44306432 bytes)
Deleting 1000000 records:  12115 ns (44306432 bytes)
Re-adding 1000000 records:  10505 ns (44306432 bytes)
Appending 1000000 records:  10610 ns (44306432 bytes)
Churning 1000000 records:  28697 ns (44306432 bytes)
$ ./speed 100000
Adding 100000 records:  6030 ns (4751360 bytes)
Finding 100000 records:  3141 ns (4751360 bytes)
Missing 100000 records:  3143 ns (4751360 bytes)
Traversing 100000 records:  4659 ns (4751360 bytes)
Deleting 100000 records:  7891 ns (4751360 bytes)
Re-adding 100000 records:  5913 ns (4751360 bytes)
Appending 100000 records:  4242 ns (4751360 bytes)
Churning 100000 records:  15300 ns (4751360 bytes)

9 years agotdb2: fix tdb_check() return when free table entries missing.
Rusty Russell [Wed, 1 Dec 2010 12:36:56 +0000 (23:06 +1030)]
tdb2: fix tdb_check() return when free table entries missing.

It mistakenly returned -1 meaning "success".

9 years agotdb2: make tdb_check call check() function.
Rusty Russell [Wed, 1 Dec 2010 12:26:35 +0000 (22:56 +1030)]
tdb2: make tdb_check call check() function.

9 years agotdb2: make summary command handle recovery "dead zone"
Rusty Russell [Wed, 1 Dec 2010 13:21:41 +0000 (23:51 +1030)]
tdb2: make summary command handle recovery "dead zone"

We can run summary with a recovery area, or a dead zone.

9 years agotdb2: cancel transactions on tdb_close
Rusty Russell [Wed, 1 Dec 2010 13:18:42 +0000 (23:48 +1030)]
tdb2: cancel transactions on tdb_close

Otherwise we leak memory.

9 years agotdb2: handle chains of free tables
Rusty Russell [Wed, 17 Nov 2010 09:56:52 +0000 (20:26 +1030)]
tdb2: handle chains of free tables

This adds chains of free tables: we choose one at random at the start and
we iterate through them when they are exhausted.

Currently there is no code to actually add a new free table, but we test
that we can handle it if we add one in future.

9 years agotdb2: get rid of zones
Rusty Russell [Wed, 17 Nov 2010 09:55:41 +0000 (20:25 +1030)]
tdb2: get rid of zones

Zones were a bad idea.  They mean we can't simply add stuff to the end
of the file (which transactions relied upon), and there's no good heuristic
in how to size them.

This patch reverts us to a single free table.

9 years agotdb2: fix bucket search
Rusty Russell [Wed, 17 Nov 2010 10:00:07 +0000 (20:30 +1030)]
tdb2: fix bucket search

We were previously jumping straight from the first bucket to the end.

9 years agotdb2: only adjust size once when growing
Rusty Russell [Wed, 17 Nov 2010 09:52:19 +0000 (20:22 +1030)]
tdb2: only adjust size once when growing

We were adding 50% to datalen twice, so move it out of adjust_size and
make the callers do it.

We also add a test that the heuristic is working at all.

9 years agotdb2: remove tailer
Rusty Russell [Wed, 17 Nov 2010 09:50:35 +0000 (20:20 +1030)]
tdb2: remove tailer

We don't actually need it.

9 years agotdb2: fix tdb_chainlock
Rusty Russell [Mon, 15 Nov 2010 07:24:15 +0000 (17:54 +1030)]
tdb2: fix tdb_chainlock

We can't enlarge the lock without risking deadlock, so tdb_chainlock() can't
simply grab a one-byte lock; it needs to grab the lock we'd need to protect
the hash.

In theory, tdb_chainlock_read() could use a one-byte lock though.