Makefile: Add hacks for the nested modules for make check.

Merge commit 'add31d8'

sparse_bsearch: fixed the no ssize_t defined error.

siphash: Use uint64_t directly, remove typedef

siphash: Simplify helper function W64's signature

siphash: Use ~7 correctly

7 in ~7 needs to be size_t for it to be correct when len > INT_MAX.

siphash: Write siphash_epilogue in nicer style

siphash: Use simple but misalignment-correct implementation of W64

A simple implementation compiles very well with gcc and clang on ppc and
x86, and will handle any-aligned input.

siphash: A keyed hash designed by Aumasson and djb

ccanlint: fix stack smash.

"engla" on IRC reported that ccanlint on linux/powerpc would loop
infinitely calling dep_failed after all tests are run and the score is
output.

Reproduced, and discovered that cannot_run() takes a container_of(),
except our top node is not a struct ccanlint.  The result was harmless on
x86, but set dep_failed to the return address on powerpc, causing that
to be called repeatedly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

dgraph: add dgraph_check and CCAN_DGRAPH_DEBUG

This is good form for datastructures, so caller can track weird bugs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: handle when _info doesn't compile.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Merge branch 'rfc822'

rfc822: switch to ccan/tal.

We use TAL_USE_TALLOC to use libtalloc as the backend: you can test that with
ccanlint --compiler="cc -DTAL_USE_TALLOC".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: use tal/talloc backend #ifdef TAL_USE_TALLOC.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: ignore "unused" non-ccan dependencies.

We can't really tell if non-ccan dependencies are used, so don't try.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/talloc: new module for backending tal onto talloc.

David Gibson points out that a project may already be using talloc, so
this gives a simple adapter if one module uses tal.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: don't automatically register cleanup function.

It may interfere with other at_exit() calls, so let them call it manually.
Also, use memset to zero, which really does make valgrind notice any leaks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: fix single-arg case.

In this case, CCAN_CFLAGS should be "".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: append the length property to the initial allocation.

Since we never post-add a length property, we can save some cycles by
allocating it together with the object itself:

Before:
$ ./samba-allocs talloc.dump --tal
Tal time:                6058997-6215211(6102645)ns
Tal_free time:           4791112-4968964(4824814)ns
Single tal_free time:    3173647-3331584(3200183)ns
$ ./samba-allocs talloc.dump --tal-size
Virtual size = 9809920, RSS = 3952640

After:
$ ./samba-allocs talloc.dump --tal
Tal time:                5911994-6043808(5956914)ns
Tal_free time:           4676026-4834598(4719908)ns
Single tal_free time:    2888195-3067365(2922298)ns
$ ./samba-allocs talloc.dump --tal-size
Virtual size = 9809920, RSS = 3948544

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: don't always allocate an array in the samba benchmark.

Since we added the length property, this penalizes us with an extra
alloc.  Unfortunately, we can't tell from the talloc.dump which are
array tallocs and which are single tallocs, but grepping the sources
gives a rate of 1 in 27.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: fix skipping tal in samba-allocs benchmark.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: fix compilation of speed benchmark.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rfc822: fix sigaction test, wean off ccan-specific talloc_set_allocator()

The first ok() line was actually failing, because sigaction returns 0
on success, but because the program exit(0) out the abort handler,
that wasn't being seen.

So remove tap and just abort and assert our way through the test,
which I think is clearer.

We use macros to mug the talloc entry points, since talloc_set_allocator()
is not in mainline talloc.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rfc822: beef up _info example.

Now it's a useful little command line utility, with tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: fix --compiler and --cflags options to apply to _info files as well.

We weren't using the compiler and cflags options in tools/compile.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: manifest.c should use path_canon.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: remove redundant num_lines in struct ccan_file.

We can use tal_count.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: print module name properly for nested modules.

eg:
tal/str: Module's tests cover all the code (tests_coverage): PASS (+3/6)
tal/str: Total score: 55/58

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: use tal/path instead of opencoding most paths.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: use tal/path instead of writing own path handlers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: use tal instead of talloc.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: use rbuf instead of grab_file.

Slowly removing the talloc dependency.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rbuf: new module.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/link: new module for reference-count style coding for tal.

Tests and example copied from talloc_link, which I also wrote.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: add tal_append_fmt() and tal_append_vfmt() helpers.

They are far too convenient, despite their inefficiency.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: rename tal_asprintf/tal_vasprintf to tal_fmt/tal_vfmt.

Shorter, sweeter, and less legacy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: add tal_strcat().

Useful for joining two strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add tal_expand().

Helper to reallocate and append to an array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: move tal string functions here from tal.

They don't need anything internal to tal, they're just helpers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: use tal_ prefix.

It's short, and much clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: make tal_count() work for strsplit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add tal_count() and length properties for arrays.

Unlike talloc, we don't have a count on everything, just on array
allocations.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: adding or removing a notifier/destructor can be const.

You don't need write access to the context to attach a destructor;
it's meta.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: optimize case where no (non-destructor) notifiers are used.

$ for i in `seq 10`; do ./samba-allocs talloc.dump --tal; done | stats.c --trim-outliers
Before:
Read 25998 nodes
Tal time:                2857988-2981158(2904324)ns
Tal_free time:           2417170-2582088(2470931)ns
Single tal_free time:    1551654-1609583(1575984)ns

After:
Read 25998 nodes
Tal time:                2780974-2901323(2839352)ns
Tal_free time:           2241725-2355309(2285165)ns
Single tal_free time:    1549908-1612485(1566609)ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: use notifier infrastructure for destructors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add general notifiers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add del_destructor().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: check headers more carefully.

We sanity check tal headers by ensuring that the pointers are in the
bounds of things we've allocated.  But the first one we check is the
prop ptr, which may also be a literal: this is_literal() dereferences
the pointer, which means we usually crash here if it's not a tal
object.

Move that last, and we have far more success with our sanity checking.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: simplify.

All the effort trying to keep the header size down to 2 pointers turns
out to be wasted.  In addition, getting the parent of a tal pointer is
now much faster.

./samba-allocs talloc.dump --tal-size:
Before:
Virtual size = 9633792, RSS = 3952640
After:
Virtual size = 9793536, RSS = 3948544

And we're much faster now, esp. on free.

./samba-allocs talloc.dump --tal:
Before:
Tal time:                2718068ns
Tal_free time:           3360258ns
Single tal_free time:    1667412ns
After:
Tal time:                2788287ns
Tal_free time:           2290167ns
Single tal_free time:    1566998ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'tal-fix-set-destroying-bit.patch':

tal: fix destroying bit usage.

Actually looking at a destroying contexts' parent didn't work, and a couple
of places didn't filter out the destroying bit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: handle nested modules when mentioned in examples.

If we see an ccan include line in an example, we add that to the
dependencies to try to build the example, but we didn't handle nested
modules.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: don't abort on malformed documentation lines.

ccanlint would abort with 'Malformed line 53' if there was a bad header.
That's very poor, and deeply unhelpful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Makefile: score nested modules.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: fix benchmark/Makefile

We need ccan/take here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/path: new module

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'path-talloc-take.patch':

tal/path: accept take() for arguments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: test for HAVE_FCHDIR

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: accept take() for arguments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal/str: new module.

Tal-enhanced string routines, copied from str_talloc (which I also wrote).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

alloc: move into antithread/alloc.

Our first nested module; easy because noone else relies on it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

web: generate correct pages for submodules.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: don't assume modules are immediately below ccan/ dir.

Removing this assumption should allow nested modules.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Makefile: remove FASTTIMEOUT

Far more effective to just skip valgrind and gcov if you want fast
results; that's what we do for the summary already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Makefile: list modules explicitly.

Clumsier than using wildcards, but that breaks down with nested modules
anyway.  In the longer term, we want ccantool to do the building, but for
the moment make is a necessary evil.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: fix dependencies.

In particular, we might need the C files for foreach and err.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: fix _info documentation addition.

Firstly, we should always ask before altering files.  Secondly, we are
not in the module directory: we need to use m->info_file->fullname.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Makefile: generate config.h more carefully

Don't risk reading partially-constructed files.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Makefile: add -Wundef to the default CFLAGS

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

TAGS: new Makefile target.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: make tal_free() return NULL

This makes it convenient to do:

ptr = tal_free(ptr);

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: take implies NULL passthrough.

This makes error handling much more convenient, since take is usually
used for chaining functions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: remove TAL_TAKE in favor of ccan/take.

TAL_TAKE is awkward to use, particularly on functions which take multiple
arguments.  Instead, record annotations for pointers due to be freed in
the callee.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

take: new module for parameter ownership.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: tal_dup()

Sometimes you want to duplicate and enlarge in one step: particularly with
TAL_TAKE.  It's also typesafe, unlike tal_memdup().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: Make tal_resize() easier to use.

Instead of trying to force people to use the return value, pass a pointer.
This makes it easier if you want to handle failure: no overwriting the old
pointer!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: make sure tal_free() preserves errno.

Always good form to have cleanup functions preserve errno.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add test for group growth.

Rusty forgets a git add.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: tal_steal() fix.

We call remove_node() in tal_steal, then re-use the node.  So it's important
that we always unlink the group property from the node for that case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: tal_parent(NULL) should be NULL.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: neaten child property initialization.

Don't open-code the list_head_init: left over from performance
debugging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: bounds checking fixes.

First, we need to update the bounds for the group property allocated
with the children property, just in case it's at the end of the
allocations.

Second, we need to allow the tal_check() code to accept the pointer to
the null parent's group property.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add benchmark based on Samba4's talloc usage.

This is based on a very simple dump of S4's talloc tree, which we try
to duplicate using tal.  The benchmarks are simply to allocate all the
nodes, free all the nodes individually, and a top-level free.

Size results (32-bit x86):
$ ./samba-allocs talloc.dump --talloc-size
Read 25998 nodes
Virtual size = 10469376, RSS = 4759552
$ ./samba-allocs talloc.dump --tal-size
Read 25998 nodes
Virtual size = 9629696, RSS = 3948544

Speed results:
$ ./samba-allocs talloc.dump
Read 25998 nodes
Malloc time:             1912082ns
Free time:               1384892ns
Talloc time:             2851531ns
talloc_free time:        2133801ns
Single talloc_free time: 1696298ns
Tal time:                2686952ns
Tal_free time:           3316153ns
Single tal_free time:    1639407ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add talloc's speed benchmark.

We modify it a bit (to do more allocations before freeing).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add typenames by default.

The really size-conscious can override them, but it's great for
debugging to have names on the nodes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add TAL_TAKE.

TAL_TAKE provides a magic context meaning "consume my args and return
a replacement".  This is useful for writing convenience functions,
though not so useful in the standard routines here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: new module.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

likely: relicense to CC0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

typesafe_cb: relicense to CC0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

likely: make dependencies correctly conditional on CCAN_LIKELY_DEBUG.

Without this, it's a trivial header.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rfc822: Update rfc822 module to use testdepends

Several modules were only listed as dependencies because they were used in
the testcases.  That in turn gave some bogus incompatible license warnings
which we partially suppressed from _info.  Now that ccanlint supports
testdepends, use that to clean it up.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

talloc: move ccan/failtest to testdepends.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

str_talloc: remove unused ccan/noerr dependency.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

grab_file: move ccan/str_alloc to testdepends

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

crcsync: move array_size to testdepends.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

asort: move array_size to testdepends.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

asearch: move array_size to testdepends.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

alignof: remove unnecessary ccan/build_assert dependency.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: detect unused dependencies, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: add testdepends support.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: enhance get_libs to get libraries for tests, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools: add testdepends handling in _info.

This allows us to separate dependencies only needed for testing.
This matters: they don't have the same impact on licensing, nor necessarily
on end-users.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>