ccan
9 years agotdb2: don't start again when we coalesce a record.
Rusty Russell [Wed, 27 Apr 2011 12:13:23 +0000 (21:43 +0930)]
tdb2: don't start again when we coalesce a record.

We currently start walking the free list again when we coalesce any record;
this is overzealous, as we only care about the next record being blatted,
or the record we currently consider "best".

We can also opportunistically try to add the coalesced record into the
new free list: if it fails, we go back to the old "mark record,
unlock, re-lock" code.

Before:
$ time ./growtdb-bench 250000 10 > /dev/null && ls -l /tmp/growtdb.tdb && time ./tdbtorture -s 0 && ls -l torture.tdb && ./speed --transaction 2000000
real 1m0.243s
user 0m13.677s
sys 0m4.336s
-rw------- 1 rusty rusty 683302864 2011-04-27 21:03 /tmp/growtdb.tdb
testing with 3 processes, 5000 loops, seed=0
OK

real 1m24.074s
user 0m0.344s
sys 0m0.468s
-rw------- 1 rusty rusty 836040 2011-04-27 21:04 torture.tdb
Adding 2000000 records:  1015 ns (110551992 bytes)
Finding 2000000 records:  641 ns (110551992 bytes)
Missing 2000000 records:  445 ns (110551992 bytes)
Traversing 2000000 records:  439 ns (110551992 bytes)
Deleting 2000000 records:  807 ns (199517112 bytes)
Re-adding 2000000 records:  851 ns (199517112 bytes)
Appending 2000000 records:  1301 ns (376542552 bytes)
Churning 2000000 records:  2423 ns (553641304 bytes)

After:
$ time ./growtdb-bench 250000 10 > /dev/null && ls -l /tmp/growtdb.tdb && time ./tdbtorture -s 0 && ls -l torture.tdb && ./speed --transaction 2000000
real 0m57.403s
user 0m11.361s
sys 0m4.056s
-rw------- 1 rusty rusty 689536976 2011-04-27 21:10 /tmp/growtdb.tdb
testing with 3 processes, 5000 loops, seed=0
OK

real 1m24.901s
user 0m0.380s
sys 0m0.512s
-rw------- 1 rusty rusty 655368 2011-04-27 21:12 torture.tdb
Adding 2000000 records:  941 ns (110551992 bytes)
Finding 2000000 records:  603 ns (110551992 bytes)
Missing 2000000 records:  428 ns (110551992 bytes)
Traversing 2000000 records:  416 ns (110551992 bytes)
Deleting 2000000 records:  741 ns (199517112 bytes)
Re-adding 2000000 records:  819 ns (199517112 bytes)
Appending 2000000 records:  1228 ns (376542552 bytes)
Churning 2000000 records:  2042 ns (553641304 bytes)

9 years agotdb2: make internal coalesce() function return length coalesced.
Rusty Russell [Fri, 25 Mar 2011 04:23:23 +0000 (14:53 +1030)]
tdb2: make internal coalesce() function return length coalesced.

This makes life easier for the next patch.

9 years agotdb2: expand more slowly.
Rusty Russell [Wed, 27 Apr 2011 12:09:27 +0000 (21:39 +0930)]
tdb2: expand more slowly.

We took the original expansion heuristic from TDB1, and they just
fixed theirs, so copy that.

Before:

After:
time ./growtdb-bench 250000 10 > /dev/null && ls -l /tmp/growtdb.tdb && time ./tdbtorture -s 0 && ls -l torture.tdb && ./speed --transaction 2000000
growtdb-bench.c: In function ‘main’:
growtdb-bench.c:74:8: warning: ignoring return value of ‘system’, declared with attribute warn_unused_result
growtdb-bench.c:108:9: warning: ignoring return value of ‘system’, declared with attribute warn_unused_result

real 1m0.243s
user 0m13.677s
sys 0m4.336s
-rw------- 1 rusty rusty 683302864 2011-04-27 21:03 /tmp/growtdb.tdb
testing with 3 processes, 5000 loops, seed=0
OK

real 1m24.074s
user 0m0.344s
sys 0m0.468s
-rw------- 1 rusty rusty 836040 2011-04-27 21:04 torture.tdb
Adding 2000000 records:  1015 ns (110551992 bytes)
Finding 2000000 records:  641 ns (110551992 bytes)
Missing 2000000 records:  445 ns (110551992 bytes)
Traversing 2000000 records:  439 ns (110551992 bytes)
Deleting 2000000 records:  807 ns (199517112 bytes)
Re-adding 2000000 records:  851 ns (199517112 bytes)
Appending 2000000 records:  1301 ns (376542552 bytes)
Churning 2000000 records:  2423 ns (553641304 bytes)

9 years agotdb2: use 64 bit file offsets on 32 bit systems if available.
Rusty Russell [Tue, 19 Apr 2011 07:24:41 +0000 (16:54 +0930)]
tdb2: use 64 bit file offsets on 32 bit systems if available.

And testing reveals a latent bug on 32 bit systems.

9 years agotdb2: test lock timeout plugin code.
Rusty Russell [Thu, 21 Apr 2011 04:04:13 +0000 (13:34 +0930)]
tdb2: test lock timeout plugin code.

9 years agotdb2: allow transaction to nest.
Rusty Russell [Thu, 7 Apr 2011 01:29:45 +0000 (10:59 +0930)]
tdb2: allow transaction to nest.

This is definitely a bad idea in general, but SAMBA uses nested transactions
in many and varied ways (some of them probably reflect real bugs) and it's
far easier to support them inside tdb2 with a flag.

We already have part of the TDB1 infrastructure in place, so this patch
just completes it and fixes one place where I'd messed it up.

9 years agotdb2: allow multiple chain locks.
Rusty Russell [Wed, 27 Apr 2011 11:18:39 +0000 (20:48 +0930)]
tdb2: allow multiple chain locks.

It's probably not a good idea, because it's a recipe for deadlocks if
anyone else grabs any *other* two chainlocks, or the allrecord lock,
but SAMBA definitely does it, so allow it as TDB1 does.

9 years agotdb2: TDB_ATTRIBUTE_STATS access via tdb_get_attribute.
Rusty Russell [Wed, 27 Apr 2011 13:51:32 +0000 (23:21 +0930)]
tdb2: TDB_ATTRIBUTE_STATS access via tdb_get_attribute.

Now we have tdb_get_attribute, it makes sense to make that the method
of accessing statistics.  That way they are always available, and it's
probably cheaper doing the direct increment than even the unlikely()
branch.

9 years agotdb2: make tdb_name() valid early in tdb_open()
Rusty Russell [Thu, 7 Apr 2011 11:18:33 +0000 (20:48 +0930)]
tdb2: make tdb_name() valid early in tdb_open()

Otherwise tdb_name() can be NULL in log functions.  And we might as
well allocate it with the tdb, as well.

9 years agotdb2: fix an error message misspelling.
Rusty Russell [Thu, 7 Apr 2011 11:18:27 +0000 (20:48 +0930)]
tdb2: fix an error message misspelling.

9 years agotdb2: tdb_set_attribute, tdb_unset_attribute and tdb_get_attribute
Rusty Russell [Thu, 7 Apr 2011 07:22:35 +0000 (16:52 +0930)]
tdb2: tdb_set_attribute, tdb_unset_attribute and tdb_get_attribute

It makes sense for some attributes to be manipulated after tdb_open,
so allow that.

9 years agotdb2: TDB_ATTRIBUTE_FLOCK support
Rusty Russell [Thu, 7 Apr 2011 07:18:43 +0000 (16:48 +0930)]
tdb2: TDB_ATTRIBUTE_FLOCK support

This allows overriding of low-level locking functions.  This allows
special effects such as non-blocking operations, and lock proxying.

We rename some local lock vars to l to avoid -Wshadow warnings.

9 years agotdb2: don't cancel transaction when tdb_transaction_prepare_commit fails
Rusty Russell [Wed, 6 Apr 2011 23:00:39 +0000 (08:30 +0930)]
tdb2: don't cancel transaction when tdb_transaction_prepare_commit fails

And don't double-log.  Both of these cause problems if we want to do
tdb_transaction_prepare_commit non-blocking (and have it fail so we can
try again).

9 years agotdb2: open hook for implementing TDB_CLEAR_IF_FIRST
Rusty Russell [Thu, 7 Apr 2011 04:21:54 +0000 (13:51 +0930)]
tdb2: open hook for implementing TDB_CLEAR_IF_FIRST

This allows the caller to implement clear-if-first semantics as per
TDB1.  The flag was removed for good reasons: performance and
unreliability, but SAMBA3 still uses it widely, so this allows them to
reimplement it themselves.

(There is no way to do it without help like this from tdb2, since it has
 to be done under the open lock).

9 years agotdb2: cleanups for tools/speed.c
Rusty Russell [Tue, 10 May 2011 01:45:04 +0000 (11:15 +0930)]
tdb2: cleanups for tools/speed.c

1) The logging function needs to append a \n.
2) The transaction start code should be after the comment and print.
3) We should run tdb_check to make sure the database is OK after each op.

9 years agotdb2: rearrange log function to put data arg at the end.
Rusty Russell [Thu, 7 Apr 2011 03:46:35 +0000 (13:16 +0930)]
tdb2: rearrange log function to put data arg at the end.

Also, rename private logfn to log_fn for consistency with other members.

9 years agotdb2: rename internal hashfn and logfn to hash_fn and log_fn.
Rusty Russell [Thu, 7 Apr 2011 05:35:58 +0000 (15:05 +0930)]
tdb2: rename internal hashfn and logfn to hash_fn and log_fn.

We use underscores everywhere else, so be consistent.

9 years agotdb2: shorten attribute members.
Rusty Russell [Thu, 7 Apr 2011 03:36:11 +0000 (13:06 +0930)]
tdb2: shorten attribute members.

It's redundant calling hash.hash_fn for example.  Standardize on fn
and data as names (private conflicts with C++).

9 years agotdb2: extend start of hash locks.
Rusty Russell [Tue, 29 Mar 2011 05:24:13 +0000 (15:54 +1030)]
tdb2: extend start of hash locks.

This gives us more locks for future use, plus allows a clear-if-first-style
hack to be implemented.

9 years agotdb2: implement tdb_chainlock_read/tdb_chainunlock_read.
Rusty Russell [Tue, 29 Mar 2011 05:23:36 +0000 (15:53 +1030)]
tdb2: implement tdb_chainlock_read/tdb_chainunlock_read.

9 years agotdb2: fix tdb_summary reports
Rusty Russell [Tue, 19 Apr 2011 13:34:13 +0000 (23:04 +0930)]
tdb2: fix tdb_summary reports

1) Fix the bogus reporting of uncoalesced runs: there has to be more than 1
   free record to make a "run", and any other record interrups the run.
2) Fix the bogus data count in the top line (which was number of records,
   not bytes).
3) Remove the count of free buckets: it's now a constant.

9 years agoconfigurator: make it BSD-MIT licensed.
Rusty Russell [Thu, 28 Apr 2011 04:19:32 +0000 (13:49 +0930)]
configurator: make it BSD-MIT licensed.

Douglas Bagnell noted that it didn't specify.

9 years agocompiler: don't override existing definitions.
Rusty Russell [Thu, 28 Apr 2011 03:52:07 +0000 (13:22 +0930)]
compiler: don't override existing definitions.

It's common when integrating CCAN into existing projects that they define
such constants for themselves.  In an ideal world, the entire project
switches to one set of definitions, but for large projects (eg SAMBA)
that's not realistic and conflicts with the aim of making CCAN modules
easy to "drop in" to existing code.

(This is a generalization of Douglas Bagnell's patch sent in Message-ID:
<4DB8D00D.8000800@paradise.net.nz>).

9 years agotools: don't unnecessarily redefine _GNU_SOURCE in config.h
Douglas Bagnall [Wed, 20 Apr 2011 14:50:23 +0000 (02:50 +1200)]
tools: don't unnecessarily redefine _GNU_SOURCE in config.h

Makes it easier to reuse this code in other projects.

9 years agoilog: credit Tim Terriberry as author in ccan/ilog/_info
Douglas Bagnall [Sat, 23 Apr 2011 13:55:26 +0000 (01:55 +1200)]
ilog: credit Tim Terriberry as author in ccan/ilog/_info

9 years agoMakefile: don't define -Werror for ccanlint.
Rusty Russell [Wed, 27 Apr 2011 07:05:07 +0000 (16:35 +0930)]
Makefile: don't define -Werror for ccanlint.

It's useful for developers, but not so much for casual users.  For example,
RHEL 5.6 has qsort_r, but no prototype, which causes a warning.

9 years agotools: always include config.h before anything else.
Rusty Russell [Wed, 27 Apr 2011 07:04:04 +0000 (16:34 +0930)]
tools: always include config.h before anything else.

Otherwise, _GNU_SOURCE isn't defined (for example) so prototypes such as
isblank can be missing.

9 years agostr: fix tests on unsigned chars, and !HAVE_ISBLANK.
Rusty Russell [Wed, 27 Apr 2011 06:35:43 +0000 (16:05 +0930)]
str: fix tests on unsigned chars, and !HAVE_ISBLANK.

9 years agociniparser: Add a check that len remains within bounds.
Andreas Schlick [Tue, 26 Apr 2011 15:07:49 +0000 (17:07 +0200)]
ciniparser: Add a check that len remains within bounds.

If the line is made solely of whitespaces, they get removed and we end up with len
set to -1. Therefore we need an additional check to avoid dereferencing line[-1].

9 years agoccanlint: Add more C++ keywords.
Andreas Schlick [Tue, 26 Apr 2011 13:19:04 +0000 (15:19 +0200)]
ccanlint: Add more C++ keywords.

9 years agostr_talloc: make strjoin much more efficient.
Rusty Russell [Tue, 19 Apr 2011 10:53:59 +0000 (20:23 +0930)]
str_talloc: make strjoin much more efficient.

Inspired by patch from Volker.

9 years agostr_talloc: avoid const warnings on test/run.c
Rusty Russell [Tue, 19 Apr 2011 10:53:35 +0000 (20:23 +0930)]
str_talloc: avoid const warnings on test/run.c

9 years agotypesafe_cb: simplify, preserve namespace.
Rusty Russell [Wed, 6 Apr 2011 22:54:41 +0000 (08:24 +0930)]
typesafe_cb: simplify, preserve namespace.

Get rid of many variants, which were just confusing for most people.
Keep typesafe_cb(), typesafe_cb_preargs() and typesafe_cb_postarts(),
and rework cast_if_type() into typesafe_cb_cast() so we stay in our
namespace.

I should have done this as soon as I discovered the limitation that
the types have to be defined if I want const-taking callbacks.

9 years agoccanlint: test for C++ reserved words in headers.
Rusty Russell [Tue, 19 Apr 2011 10:56:13 +0000 (20:26 +0930)]
ccanlint: test for C++ reserved words in headers.

Don't check the whole source, but it's nice for headers to be C++-clean.

9 years agoconfigurator: HAVE_FILE_OFFSET_BITS
Rusty Russell [Tue, 19 Apr 2011 07:23:39 +0000 (16:53 +0930)]
configurator: HAVE_FILE_OFFSET_BITS

Defines whether it's useful to do #define _FILE_OFFSET_BITS 64 to get a
larger off_t.

9 years agojunkcode: upload via website.
Rusty Russell [Tue, 19 Apr 2011 00:54:27 +0000 (10:24 +0930)]
junkcode: upload via website.

9 years agoconfigurator: Remove the now unneeded test for nested functions.
Andreas Schlick [Sat, 2 Apr 2011 01:31:51 +0000 (12:01 +1030)]
configurator: Remove the now unneeded test for nested functions.

9 years agoasort: Remove the trampoline version.
Andreas Schlick [Sat, 2 Apr 2011 01:30:36 +0000 (12:00 +1030)]
asort: Remove the trampoline version.

Most Linux systems use glibc and therefore have qsort_r() and for the others an
implementation is provided, therefore there is no need to have a third case in
asort.

9 years agofailtest: Change mode_t to int.
Andreas Schlick [Sat, 2 Apr 2011 01:13:52 +0000 (11:43 +1030)]
failtest: Change mode_t to int.

This avoids a warning on systems that have a 16bit wide mode_t.

9 years agohash: use config.h settings for endian.
Rusty Russell [Sat, 2 Apr 2011 01:36:49 +0000 (12:06 +1030)]
hash: use config.h settings for endian.

Leave old hacky #ifdef tests there for reference.

9 years agohash: Extend the byte order check to cover x86_64.
Andreas Schlick [Sat, 2 Apr 2011 01:13:52 +0000 (11:43 +1030)]
hash: Extend the byte order check to cover x86_64.

9 years agoTo compile ccan under DragonFly BSD some additional includes are required.
Andreas Schlick [Sat, 2 Apr 2011 01:13:52 +0000 (11:43 +1030)]
To compile ccan under DragonFly BSD some additional includes are required.

9 years agoasort: Use qsort_r if the system provides it.
Andreas Schlick [Sat, 2 Apr 2011 01:30:08 +0000 (12:00 +1030)]
asort: Use qsort_r if the system provides it.

tools/ccanlint/file_analysis.c needs to include config.h to set _GNU_SOURCE
before any other header file includes stdlib.h.

9 years agoconfigurator: Add test for glibc's qsort_r.
Andreas Schlick [Sat, 2 Apr 2011 01:30:51 +0000 (12:00 +1030)]
configurator: Add test for glibc's qsort_r.

9 years agofailtest: fix fascist warn_unused_result warnings
Rusty Russell [Mon, 28 Mar 2011 04:14:40 +0000 (14:44 +1030)]
failtest: fix fascist warn_unused_result warnings

9 years agofailtest: fix void * arithmetic in example.
Rusty Russell [Mon, 28 Mar 2011 04:00:56 +0000 (14:30 +1030)]
failtest: fix void * arithmetic in example.

9 years agofailtest: continue (without failing) if we run off end of --failpath=
Rusty Russell [Mon, 28 Mar 2011 03:58:22 +0000 (14:28 +1030)]
failtest: continue (without failing) if we run off end of --failpath=

This is important because we tell people to use --failpath to reproduce a
failure, but the fail path we list only goes up to the last failure injection
if the child dies, for example.

Thanks to David Gibson for prodding me to fix this...

9 years agotdb2: make head of free list point to last entry.
Rusty Russell [Mon, 28 Mar 2011 03:48:49 +0000 (14:18 +1030)]
tdb2: make head of free list point to last entry.

This allows for access to the end of the list, in case we need it
later (eg. moving free list entries to the end of the list after
failed coalescing?).

9 years agotdb2: speed up transaction code by making page size a constant.
Rusty Russell [Fri, 25 Mar 2011 05:42:03 +0000 (16:12 +1030)]
tdb2: speed up transaction code by making page size a constant.

Turning getpagesize() into a constant 4096 really helps gcc optimize, and
not just on 64 bit.

Here are the results of running "tools/speed --transaction --no-sync 2000000":

i386 with gcc 4.4.5:
Before:
$ ./speed --transaction --no-sync 2000000
Adding 2000000 records:  1195 ns (93594224 bytes)
Finding 2000000 records:  719 ns (93594224 bytes)
Missing 2000000 records:  429 ns (93594224 bytes)
Traversing 2000000 records:  523 ns (93594224 bytes)
Deleting 2000000 records:  901 ns (93594224 bytes)
Re-adding 2000000 records:  1032 ns (93594224 bytes)
Appending 2000000 records:  1711 ns (182801232 bytes)
Churning 2000000 records:  3233 ns (182801232 bytes)

After:
Adding 2000000 records:  868 ns (93594224 bytes)
Finding 2000000 records:  569 ns (93594224 bytes)
Missing 2000000 records:  369 ns (93594224 bytes)
Traversing 2000000 records:  406 ns (93594224 bytes)
Deleting 2000000 records:  674 ns (93594224 bytes)
Re-adding 2000000 records:  730 ns (93594224 bytes)
Appending 2000000 records:  1144 ns (182801232 bytes)
Churning 2000000 records:  2085 ns (182801232 bytes)

Here's the effect (average of 10) on x8664 with gcc 4.1.2, with an earlier
tree and 1000000:
Before:
Adding 830.1ns
Finding 428.3ns
Missing 250.3ns
Traversing 387.4ns
Deleting 588.8ns
Re-adding 737.2ns
Appending 1141.4ns
Churning 1792ns

After:
Adding 562.8ns
Finding 362.1ns
Missing 226.4ns
Traversing 276.5ns
Deleting 426.7ns
Re-adding 489.8ns
Appending 736.4ns
Churning 1112.7ns

9 years agotdb2: fix traversal bug in free list lock_and_alloc()
Rusty Russell [Fri, 25 Mar 2011 05:39:05 +0000 (16:09 +1030)]
tdb2: fix traversal bug in free list lock_and_alloc()

We keep looking even if the current best is exactly right.  This is
really bad, because our smaller free lists hold exactly one size: with
this bug we iterate to the end of the list before hitting the end and
deciding we can use it after all.

Before:
$ ./speed --transaction --no-sync 2000000
Adding 2000000 records:  1179 ns (93594224 bytes)
Finding 2000000 records:  694 ns (93594224 bytes)
Missing 2000000 records:  429 ns (93594224 bytes)
Traversing 2000000 records:  519 ns (93594224 bytes)
Deleting 2000000 records:  896 ns (93594224 bytes)
Re-adding 2000000 records:  1129 ns (93594224 bytes)
Appending 2000000 records:  1713 ns (182801232 bytes)
Churning 2000000 records:  32612 ns (182801232 bytes)

After:
$ ./speed --transaction --no-sync 2000000
Adding 2000000 records:  1195 ns (93594224 bytes)
Finding 2000000 records:  719 ns (93594224 bytes)
Missing 2000000 records:  429 ns (93594224 bytes)
Traversing 2000000 records:  523 ns (93594224 bytes)
Deleting 2000000 records:  901 ns (93594224 bytes)
Re-adding 2000000 records:  1032 ns (93594224 bytes)
Appending 2000000 records:  1711 ns (182801232 bytes)
Churning 2000000 records:  3233 ns (182801232 bytes)

9 years agotdb2: add --summary and logging to tools/speed.
Rusty Russell [Fri, 25 Mar 2011 05:21:08 +0000 (15:51 +1030)]
tdb2: add --summary and logging to tools/speed.

This helps us see where the problems are.

9 years agotdb2: add --no-sync argument to speed.
Rusty Russell [Mon, 28 Mar 2011 03:51:55 +0000 (14:21 +1030)]
tdb2: add --no-sync argument to speed.

Running speed with --transaction --no-sync means no locking or syncs
are done, so we can measure raw TDB speed.

9 years agotdb2: fix use after free on error message
Rusty Russell [Thu, 24 Mar 2011 03:40:22 +0000 (14:10 +1030)]
tdb2: fix use after free on error message

We use "r" after we call tdb_access_release() when we find corruption
in the free list.  "r" may be a pointer into malloced memory, freed
by tdb_access_release().

9 years agotdb2: fix two transaction bugs.
Rusty Russell [Thu, 24 Mar 2011 01:12:21 +0000 (11:42 +1030)]
tdb2: fix two transaction bugs.

One but were we didn't update the map_size if we expanded the
transaction but didn't create a new recovery area (most easily
reproduced by setting the TDB_NOSYNC flag).

Another out-by-one bug in transaction_direct where we would give
read-access to the underlying file despite the last block having been
modified.

Both these were found by tdbtorture.

9 years agotdb2: cleanups for tdbtorture, add more debugging and -S flag.
Rusty Russell [Wed, 23 Mar 2011 22:47:58 +0000 (09:17 +1030)]
tdb2: cleanups for tdbtorture, add more debugging and -S flag.

9 years agotdb2: actually implement tdb_error().
Rusty Russell [Thu, 24 Mar 2011 01:26:38 +0000 (11:56 +1030)]
tdb2: actually implement tdb_error().

I'm still not sure it'll last into the final version, but finish the
implementation.

9 years agotdb2: fix tools warnings with -Wshadow and -Wcast-qual.
Rusty Russell [Wed, 23 Mar 2011 05:02:01 +0000 (15:32 +1030)]
tdb2: fix tools warnings with -Wshadow and -Wcast-qual.

Samba uses these flags, so be friendly.

9 years agotdb2: avoid C++ reserved words (ie. private)
Rusty Russell [Wed, 23 Mar 2011 05:00:37 +0000 (15:30 +1030)]
tdb2: avoid C++ reserved words (ie. private)

9 years agotdb2: check PID if we are holding a lock.
Rusty Russell [Thu, 24 Mar 2011 01:24:43 +0000 (11:54 +1030)]
tdb2: check PID if we are holding a lock.

tdb1 has tdb_reopen/tdb_reopen_all, which you are supposed to call
after a fork().  But we can detect PID changes whenever we grab a lock,
which is to say, before we do anything.

We currently assume that pread and pwrite work, so the only problem
with fork() is if we are holding locks and expect them to still be
there in the child.  This is unusual, but most likely caused by a
fork() inside a transaction.  In such cases, allow the child to unlock
without complaint; they can then use the TDB as expected.

Performance before and after shows no measurable speed difference:

Total of 5 runs of tools/speed 100000:
Before:
Adding: 18899ns
Finding: 7040ns
Missing: 5802ns
Traversing: 6466ns
Deleting: 12607ns
Re-adding: 14185ns
Appending: 20770ns
Churning: 93845ns

After:
Adding: 18073ns
Finding: 6795ns
Missing: 5549ns
Traversing: 6333ns
Deleting: 12313ns
Re-adding: 13835ns
Appending: 20343ns
Churning: 92208ns

9 years agotdb2: tdb_lockall() and tdb_lockall_read() support.
Rusty Russell [Tue, 22 Mar 2011 07:51:27 +0000 (18:21 +1030)]
tdb2: tdb_lockall() and tdb_lockall_read() support.

We also change tdb_chainunlock not to return an error (as per the
other locking functions: we log a message, but don't return an error).

9 years agoconfigure.h: HAVE_ISBLANK
Rusty Russell [Mon, 28 Mar 2011 03:50:38 +0000 (14:20 +1030)]
configure.h: HAVE_ISBLANK

This isn't on RHEL4, for example: isascii() and isblank() are not in
C89.

9 years agofailtest: handle EINTR from poll.
Rusty Russell [Fri, 25 Mar 2011 01:16:17 +0000 (11:46 +1030)]
failtest: handle EINTR from poll.

I don't quite know why, but this started happening to me.  We should
handle it anyway.

9 years agoccanlint: fix listing of dependencies
Rusty Russell [Fri, 25 Mar 2011 03:21:20 +0000 (13:51 +1030)]
ccanlint: fix listing of dependencies

gcc gave a warning:
tools/ccanlint/ccanlint.c:230:19: error: ‘c’ may be used uninitialized in this function

Which indicated that test dependency printing was broken: we need to
loop through the tests!  Also, we haven't parsed options yet, so
verbose is never true: move it to later and make it depend on -vvv.

9 years agoAdd configurator test for memmem()
David Gibson [Fri, 18 Mar 2011 08:29:16 +0000 (19:29 +1100)]
Add configurator test for memmem()

memmem(), which searches for one bytestring in a longer
bytestring is one of those functions that should be standard, but
isn't.  This patch adds a ccan configurator test for the function
so that modules can use or replace it as necessary.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
9 years agoiscsi: fix warnings with -Wwrite-strings
Rusty Russell [Tue, 22 Mar 2011 01:51:39 +0000 (12:21 +1030)]
iscsi: fix warnings with -Wwrite-strings

Simplest fix is to use const where possible, and use a different var for
asprintf and string literals in login.c.

9 years agodarray: handle case where we don't have typeof.
Rusty Russell [Tue, 22 Mar 2011 04:19:40 +0000 (14:49 +1030)]
darray: handle case where we don't have typeof.

Tests don't compile otherwise, so fix that up (ccanlint now tests with
reduced features, so this became clear).

9 years agotap: ccanlint fixups
Rusty Russell [Tue, 22 Mar 2011 04:16:04 +0000 (14:46 +1030)]
tap: ccanlint fixups

Add a license tag (though it's a weird BSD variant), remove whitespace, and
avoid void pointer arithmetic.

9 years agoccanlint: fix compilation of API tests with coverage
Rusty Russell [Tue, 22 Mar 2011 03:52:28 +0000 (14:22 +1030)]
ccanlint: fix compilation of API tests with coverage

They expect to be linked against the module objects.

9 years agocrcsync: avoid arithmetic on void pointers
Rusty Russell [Tue, 22 Mar 2011 03:41:32 +0000 (14:11 +1030)]
crcsync: avoid arithmetic on void pointers

9 years agoccanlint: roughly handle multiple conditions on #if line.
Rusty Russell [Tue, 22 Mar 2011 01:13:00 +0000 (11:43 +1030)]
ccanlint: roughly handle multiple conditions on #if line.

Just grab the tokens for the simple case: this allows the ccanlint
reduce_features test to work on eg. ccan/cast.

9 years agoccanlint: exit with non-zero exit status if a test fails.
Rusty Russell [Tue, 22 Mar 2011 04:22:13 +0000 (14:52 +1030)]
ccanlint: exit with non-zero exit status if a test fails.

This means we change some minor tests to "never fail", eg. whitespace
or documentation tests.  Note that pass/fail is independent of the
score for a test.

9 years agocompiler: fix -Wwrite-strings warning in example.
Rusty Russell [Tue, 22 Mar 2011 02:12:43 +0000 (12:42 +1030)]
compiler: fix -Wwrite-strings warning in example.

9 years agoantithread: fix -Wwrite-strings warning in test, and trailing whitespace.
Rusty Russell [Tue, 22 Mar 2011 02:13:03 +0000 (12:43 +1030)]
antithread: fix -Wwrite-strings warning in test, and trailing whitespace.

9 years agoalloc: dont clash with libc's fls, avoid void pointer arithmetic
Rusty Russell [Tue, 22 Mar 2011 02:14:03 +0000 (12:44 +1030)]
alloc: dont clash with libc's fls, avoid void pointer arithmetic

9 years agoantithread: avoid arithmetic on void pointers
Rusty Russell [Tue, 22 Mar 2011 01:50:49 +0000 (12:20 +1030)]
antithread: avoid arithmetic on void pointers

9 years agotools: fix up warnings with -Wwrite-strings.
Rusty Russell [Tue, 22 Mar 2011 03:37:45 +0000 (14:07 +1030)]
tools: fix up warnings with -Wwrite-strings.

Be a little more careful with const.

9 years agoforeach: more const iteration fixes.
Rusty Russell [Tue, 22 Mar 2011 03:35:33 +0000 (14:05 +1030)]
foreach: more const iteration fixes.

Shown up by -Wwrite-strings.

9 years agoccan: add -Wpointer-arith and -Wwrite-strings by default
Rusty Russell [Tue, 22 Mar 2011 01:21:00 +0000 (11:51 +1030)]
ccan: add -Wpointer-arith and -Wwrite-strings by default

Wpointer-arith is useful for spotting GCC-isms which do arithmetic on
void pointers.  Wwrite-strings treats string constants as const char *,
which is sane (but bad for legacy code).

9 years agoalloc: avoid arithmetic on void pointers.
Rusty Russell [Tue, 22 Mar 2011 01:50:19 +0000 (12:20 +1030)]
alloc: avoid arithmetic on void pointers.

9 years agoread_write_all: avoid arithmetic on void pointers.
Rusty Russell [Tue, 22 Mar 2011 01:20:34 +0000 (11:50 +1030)]
read_write_all: avoid arithmetic on void pointers.

9 years agoopt: avoid function pointer arithmetic
Rusty Russell [Tue, 22 Mar 2011 01:20:08 +0000 (11:50 +1030)]
opt: avoid function pointer arithmetic

Same fix as we used for ccan/typesafe_cb.

9 years agoopt: fix memory leak in tests.
Rusty Russell [Tue, 22 Mar 2011 01:19:36 +0000 (11:49 +1030)]
opt: fix memory leak in tests.

No only does this give us one more ccanlint point, it clears the way
to see if we introduce a *real* memory leak later.

9 years agoopt: allow const arguments.
Rusty Russell [Tue, 22 Mar 2011 01:18:43 +0000 (11:48 +1030)]
opt: allow const arguments.

This need shows up most clearly with opt_usage_and_exit and gcc's
-Wwrite-strings, where string literals become "const char *".  Our
callbacks should take const void *: since we overload the arg field
already (to hold table size) it make sense to turn it into a proper
union.

9 years agoopt: define NULL
Rusty Russell [Tue, 22 Mar 2011 01:17:42 +0000 (11:47 +1030)]
opt: define NULL

Headers should stand alone; we use NULL, so we should make sure it's defined.

9 years agocast: fix cast of void * when we don't have GCC features.
Rusty Russell [Tue, 22 Mar 2011 01:23:17 +0000 (11:53 +1030)]
cast: fix cast of void * when we don't have GCC features.

I thought using sizeof() comparison to compare the types was clever,
but it doesn't work on void pointers, as sizeof(void) is illegal.

9 years agoarray_size: make sure tests work even without GCC features.
Rusty Russell [Tue, 22 Mar 2011 01:36:40 +0000 (12:06 +1030)]
array_size: make sure tests work even without GCC features.

In this case, "work" means "fail to compile cleanly".

9 years agotalloc: fix leak in test/run-set_allocator.c
Rusty Russell [Tue, 22 Mar 2011 02:44:23 +0000 (13:14 +1030)]
talloc: fix leak in test/run-set_allocator.c

9 years agofailtest: print complete history when we detect a leak.
Rusty Russell [Tue, 22 Mar 2011 02:44:04 +0000 (13:14 +1030)]
failtest: print complete history when we detect a leak.

9 years agotdb2: fix arithmetic on void * pointer.
Rusty Russell [Mon, 21 Mar 2011 11:48:48 +0000 (22:18 +1030)]
tdb2: fix arithmetic on void * pointer.

9 years agotypesafe: fix warnings with gcc's -Wcast-qual
Rusty Russell [Mon, 21 Mar 2011 10:32:24 +0000 (21:02 +1030)]
typesafe: fix warnings with gcc's -Wcast-qual

Don't implicitly cast away const in the tests.

9 years agotypesafe: fix warnings with gcc's -Wpointer-arith
Rusty Russell [Mon, 21 Mar 2011 10:31:35 +0000 (21:01 +1030)]
typesafe: fix warnings with gcc's -Wpointer-arith

In particular, avoid doing arithmetic on NULL; decay the function to a
pointer using a trinary conditional instead.

9 years agotdb2: tdb_mkdata
Rusty Russell [Mon, 21 Mar 2011 11:42:17 +0000 (22:12 +1030)]
tdb2: tdb_mkdata

It's common to want to create a struct tdb_data which contains a const
pointer, so add a helper to do so, and use it to clean up other -Wcast-qual
warnings.

9 years agotdb2: fix warnings with -Wcast-qual using ccan/cast.
Rusty Russell [Mon, 21 Mar 2011 11:34:30 +0000 (22:04 +1030)]
tdb2: fix warnings with -Wcast-qual using ccan/cast.

9 years agocast: fix compilation with GCC's -Wcast-qual
Rusty Russell [Mon, 21 Mar 2011 11:47:42 +0000 (22:17 +1030)]
cast: fix compilation with GCC's -Wcast-qual

Samba uses -Wcast-qual to warn about casting away const.  The kind of
coders who care about const correctness are likely to turn that option
on, so we should make sure it's compatible with cast_const().

9 years agotdb2: fix tools compilation.
Rusty Russell [Mon, 21 Mar 2011 10:20:22 +0000 (20:50 +1030)]
tdb2: fix tools compilation.

9 years agotdb2: fix compile of test/run-check-callback.c
Rusty Russell [Mon, 21 Mar 2011 10:19:49 +0000 (20:49 +1030)]
tdb2: fix compile of test/run-check-callback.c

9 years agoforeach: fix case where iterators are not on the stack.
Rusty Russell [Mon, 21 Mar 2011 03:14:22 +0000 (13:44 +1030)]
foreach: fix case where iterators are not on the stack.

The foreach garbage collection assumed that iterators were all on the
stack, but they could be on the heap or a global (or static) variable.
We can prevent the heap case by tricky use of macros to complain on
any iterator which isn't a single token, but we can't prevent
globals/statics.

So, if an iterator already seems to be "off" the stack, mark it as
such and simply never free it.

9 years agoconfigurator: HAVE_STACK_GROWS_UPWARDS
Rusty Russell [Mon, 21 Mar 2011 02:44:03 +0000 (13:14 +1030)]
configurator: HAVE_STACK_GROWS_UPWARDS

9 years agoforeach: allow a single argument to foreach_int() and foreach_ptr().
Rusty Russell [Mon, 21 Mar 2011 02:43:51 +0000 (13:13 +1030)]
foreach: allow a single argument to foreach_int() and foreach_ptr().

David Gibson noted that foreach requires more than one argument.
There's no particularly good reason for this, other than lack of
imagination on my part.

9 years agoforeach: fix HAVE_FOR_LOOP_DECLARATION case.
Rusty Russell [Mon, 21 Mar 2011 02:42:46 +0000 (13:12 +1030)]
foreach: fix HAVE_FOR_LOOP_DECLARATION case.