Rusty Russell [Fri, 24 Sep 2010 05:24:46 +0000 (14:54 +0930)]
tdb: TDB_INCOMPATIBLE_HASH, to allow safe changing of default hash.
This flag to tdb_open/tdb_open_ex effects creation of a new database:
1) Uses the Jenkins lookup3 hash instead of the old gdbm hash if none is
specified,
2) Places a non-zero field in header->rwlocks, so older versions of TDB will
refuse to open it.
This means that the caller (ie Samba) can set this flag to safely
change the hash function. Versions of TDB from this one on will either
use the correct hash or refuse to open (if a different hash is specified).
Older TDB versions will see the nonzero rwlocks field and refuse to open
it under any conditions.
Rusty Russell [Mon, 13 Sep 2010 10:12:34 +0000 (19:42 +0930)]
tdb: put example hashes into header, so we notice incorrect hash_fn.
This is Stefan Metzmacher <metze@samba.org>'s patch with minor
changes:
1) Use the TDB_MAGIC constant so both hashes aren't of strings.
2) Check the hash in tdb_check (paranoia, really).
3) Additional check in the (unlikely!) case where both examples hash to 0.
4) Cosmetic changes to var names and complaint message.
Fix tdb_check() to work with read-only tdb databases. The function tdb_lockall() uses F_WRLCK internally, which doesn't work on a fd opened with O_RDONLY. Use tdb_lockall_read() instead.
Rusty Russell [Sat, 11 Sep 2010 02:17:37 +0000 (11:47 +0930)]
tdb: rewrite external agent for testing.
For locking and transaction tests, we need an external process to probe the
tdb. This code was a mess, and unreliable (occasional failures). Rewrite
it so that instead of blocking and using SIGALRM, we mug fcntl and force
non-blocking locks. We use a special return to day "I couldn't because
the lock wasn't available".
I also made the operations clearer and more orthogonal rather than "what
we needed for each test" which was really hard to understand.
It turns out those occasional failures weren't spurious. Next patch!
Rusty Russell [Fri, 10 Sep 2010 03:59:54 +0000 (13:29 +0930)]
tdb: enforce hashing, via example hash in old rwlocks entry in header.
This means that older code will not be able to open new TDBs with
a non-default hash, *even* if they are using the correct hash.
Non-default hashes were unusual, but now SAMBA is considering using
a non-default hash, and avoiding corruption seems more important
than backwards compatibility for an obscure case.
Rusty Russell [Fri, 10 Sep 2010 03:20:16 +0000 (12:50 +0930)]
tdb: test that new-style TDBs with non-default hashes can't be opened.
We currently allow opening of a TDB without any idea if we're using the
right hash function. Since most people use the default, this hasn't been
a big issue.
The next change will put an example hash value in the rwlocks field in
the header: current TDB already refuses to open a TDB with this field
non-zero. This commit simply adds tests that that occurs.
Rusty Russell [Thu, 9 Sep 2010 09:29:18 +0000 (18:59 +0930)]
tdb2: change to using a hash tree.
As the locking issues with enlarging a hash were so nasty, we switch to a
tree structure for the entries. It's a hash which expands to point to
sub-hashes when it fills.
This means we no longer have a 'volatile' header: the top hash cannot move.
In fact, we no longer store a copy of the header in the tdb_context; we only
need hash_seed.
New helper functions for accessing writable areas and committing the results
(if they had to be copied). New debug test to make sure we don't hold access
while we're doing something which could cause us to unmap/remap.
Find becomes more complicated: we need to track where we found (or didn't
find) an entry so we can easily add/delete it.
Traverse becomes more complicated: we need to track where we were in the
hash tree.
Rusty Russell [Fri, 3 Sep 2010 12:39:40 +0000 (22:09 +0930)]
tdb2: use immobile free buckets, rename tests to show some ordering.
We put the free lists at the beginning of a zone; this means no record
can be larger than a zone, but means they cannot move. Once we change
hashes to be expanding, they won't move either and the result should be
simpler.
Rusty Russell [Thu, 26 Aug 2010 14:38:34 +0000 (00:08 +0930)]
tdb2: more fixes and tests for enlarging hash.
- Neaten I/O function
- Don't use fill in zero_out: it's only for low-level ops.
- Don't mangle arg in tdb_write_convert: it broke write_header.
- More use of tdb_access_read, make it optionally converting.
- Rename unlock_range to unlock_lists.
- Lots of fixes to enlarge_hash now it's being tested.
- More expansion cases tested.
Rusty Russell [Tue, 15 Jun 2010 10:02:55 +0000 (19:32 +0930)]
typesafe_cb: expose _exact and _def variants.
We can't allow NULL with the new variant (needed by talloc's set_destructor
for example), so document that and expose all three variants for different
uses.
Rusty Russell [Fri, 11 Jun 2010 03:36:40 +0000 (13:06 +0930)]
typesafe_cb: fix promotable types being incorrectly accepted by cast_if_type.
cast_if_type() should not try to degrade the expression using 1?(test):0,
as that promotes bool to int, as well as degrading functions to function
pointers: it should be done by the callers.
Rusty Russell [Wed, 9 Jun 2010 14:33:04 +0000 (00:03 +0930)]
alloc: reduce page header further, go down to 64k minimum.
This means we can't have more than 2^25 elements per page; that's
a maximum small page size of about 2^24 (with >8 objects per small page
we move to large pages), meaning a poolsize max of 4G.
We have a tighter limit at the moment anyway, but we should remove it
once we fix this. In particular count all-zero and all-one words in
the used field (that's what we care about: full or empty) would give us
another factor of 64 (we only care about larger pool sizes on 64-bit
platforms).
We can also restore the larger number of pages and greater inter-page
spacing once we implement the alternative tiny allocator.