ccan
7 weeks agobitmap: Allow bitmap type to be forward declared master origin/master
Kirill Smelkov [Mon, 21 Oct 2019 15:09:20 +0000 (15:09 +0000)]
bitmap: Allow bitmap type to be forward declared

Currently bitmap type is defined via untagged struct which makes it
impossible to forward declare it. Forward-declaring is useful since all
bitmap functions only use bitmap* and in public user-visible
headers/datastructures it is enough to indicate that a data field with
bitmap pointer is there, whereas bitmap.h can be included only in
implementation.

Beside that some headers are included by both C and C++ parts of a
project, and when ccan/bitmap.h is processed by C++ compiler it gives:

    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_alloc(long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:201:15: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      return malloc(bitmap_sizeof(nbits));
             ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_realloc0(bitmap*, long unsigned int, long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:227:18: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      bitmap = realloc(bitmap, bitmap_sizeof(nbits));
               ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_realloc1(bitmap*, long unsigned int, long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:238:18: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      bitmap = realloc(bitmap, bitmap_sizeof(nbits));

-> Give to users ability not to force-include ccan/bitmap.h by
forward-declaring bitmaps like this:

    typedef struct bitmap bitmap;
    ...
    struct MyStruct {
        bitmap *my_bitmap;
    };

Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Message-Id: <20191021150903.25159-1-kirr@nexedi.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2 months agointmap: implement intmap_before()
Rusty Russell [Thu, 10 Oct 2019 05:06:00 +0000 (15:36 +1030)]
intmap: implement intmap_before()

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 months agofailtest, rbuf: fix up incorrect lseek arg order.
Rusty Russell [Fri, 4 Oct 2019 05:07:14 +0000 (14:37 +0930)]
failtest, rbuf: fix up incorrect lseek arg order.

SEEK_SET == 0, so it's not hurting anyone now, but when I cut & paste
it elsewhere...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 months agopipecmd: close fds in child.
Rusty Russell [Wed, 14 Aug 2019 03:43:46 +0000 (13:13 +0930)]
pipecmd: close fds in child.

This is usually what you want; I didn't even add a flag to stop it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
4 months agohtable: add htable_count().
Rusty Russell [Tue, 13 Aug 2019 03:02:42 +0000 (12:32 +0930)]
htable: add htable_count().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
4 months agoSuppress false positive warning "-Wmaybe-uninitialized" with GCC -O3
Chirimen-Jako [Wed, 31 Jul 2019 09:21:58 +0000 (18:21 +0900)]
Suppress false positive warning "-Wmaybe-uninitialized" with GCC -O3

4 months agoconfigurator: fix openmp test on some versions of clang.
Rusty Russell [Wed, 31 Jul 2019 06:56:30 +0000 (16:26 +0930)]
configurator: fix openmp test on some versions of clang.

Discovered on bionic's Travis install, it failed at *runtime* when
it couldn't find the shared library.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
5 months agobitmap: Fix some bugs on 32-bit platforms
David Gibson [Thu, 11 Jul 2019 02:07:33 +0000 (12:07 +1000)]
bitmap: Fix some bugs on 32-bit platforms

The bitmap_word type is an unsigned long.  However in some places we assign
it using -1ULL, a 64-bit value on many platforms.  We sometimes get away
with this because it masks correctly, but in other cases it breaks things.

To clean this up define a new BITMAP_WORD_1 constant, indicating a
bitmap_word with all bits set, and use that instead of explicit UL or ULL
qualifiers.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
6 months agoccan/crc: remove broken "crc32c" routines which gave wrong results.
Rusty Russell [Tue, 11 Jun 2019 04:56:02 +0000 (14:26 +0930)]
ccan/crc: remove broken "crc32c" routines which gave wrong results.

I copied them from the kernel, but obviously didn't find good test
vectors.  In particular, the CRC of all zeroes is not 0!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agoccan/crc32c: new module for accelerated CRC32 (on x86-64).
Rusty Russell [Tue, 11 Jun 2019 04:54:56 +0000 (14:24 +0930)]
ccan/crc32c: new module for accelerated CRC32 (on x86-64).

Note: the previous code in ccan/crc is wrong, so I started fresh with
actual test vectors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agoccan/compiler: add cpu_supports() macro.
Rusty Russell [Tue, 11 Jun 2019 04:54:11 +0000 (14:24 +0930)]
ccan/compiler: add cpu_supports() macro.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agoconfigurator: detect __builtin_cpu_supports.
Rusty Russell [Tue, 11 Jun 2019 04:53:52 +0000 (14:23 +0930)]
configurator: detect __builtin_cpu_supports.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agojson_out: pass through OOM failures.
Rusty Russell [Thu, 23 May 2019 05:29:44 +0000 (14:59 +0930)]
json_out: pass through OOM failures.

And fix escaping to work with addstr(), and assert in debug mode
if they don't specify quotes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agojson_out: new module for authoring JSON strings.
Rusty Russell [Tue, 21 May 2019 06:41:58 +0000 (16:11 +0930)]
json_out: new module for authoring JSON strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agojson_escape: add fast-path for when we don't need to escape.
Rusty Russell [Tue, 21 May 2019 04:34:49 +0000 (14:04 +0930)]
json_escape: add fast-path for when we don't need to escape.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agojson_escape: new module to escape JSON strings.
Rusty Russell [Tue, 21 May 2019 04:21:12 +0000 (13:51 +0930)]
json_escape: new module to escape JSON strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
8 months agohtable: avoid branch in calculating perfect bit.
Rusty Russell [Tue, 2 Apr 2019 01:19:28 +0000 (11:49 +1030)]
htable: avoid branch in calculating perfect bit.

Final results of tools/speed/10000000 (10 runs) shows a slight
slowdown in some tests, but it makes an empty htable smaller.

-Initial delete all: 96-98(96.4+/-0.66) ns
+Initial delete all: 97-99(98.2+/-0.75) ns
-Initial re-inserting: 117-124(121.4+/-1.9) ns
+Initial re-inserting: 124-131(126.4+/-2.4) ns
-Adding (a different) half: 49-50(49.3+/-0.46) ns
+Adding (a different) half: 50-52(51.2+/-0.75) ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
8 months agohtable: reduce size of htable by storing perfect bitnum, not mask.
Rusty Russell [Mon, 1 Apr 2019 23:49:19 +0000 (10:19 +1030)]
htable: reduce size of htable by storing perfect bitnum, not mask.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
8 months agohtable: reduce size of htable by calculating max every time.
Rusty Russell [Mon, 1 Apr 2019 23:20:37 +0000 (09:50 +1030)]
htable: reduce size of htable by calculating max every time.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
8 months agoio: fix another leak path for always array.
Rusty Russell [Sat, 16 Mar 2019 04:59:24 +0000 (15:29 +1030)]
io: fix another leak path for always array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
8 months agoio: don't leak memory on clean shutdown.
Rusty Russell [Sat, 16 Mar 2019 04:27:05 +0000 (14:57 +1030)]
io: don't leak memory on clean shutdown.

Free the internal "always" array like we free "fds".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
8 months agoio: typo fixes.
Rusty Russell [Sat, 16 Mar 2019 03:14:13 +0000 (13:44 +1030)]
io: typo fixes.

Suggested-by: Lisa Neigut <@niftynei>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agohtable: add allocator hooks.
Rusty Russell [Mon, 4 Mar 2019 10:25:34 +0000 (20:55 +1030)]
htable: add allocator hooks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agohtable: commit extra tests I had lying around.
Rusty Russell [Mon, 4 Mar 2019 10:25:54 +0000 (20:55 +1030)]
htable: commit extra tests I had lying around.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agotimer: clean up hook allocator API
Rusty Russell [Mon, 4 Mar 2019 11:06:59 +0000 (21:36 +1030)]
timer: clean up hook allocator API

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agotimer: add hook for allocation functions.
Rusty Russell [Mon, 4 Mar 2019 09:12:57 +0000 (19:42 +1030)]
timer: add hook for allocation functions.

malloc and free aren't for everyone.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agoccan/io: add io_conn_exclusive and io_conn_out_exclusive.
Rusty Russell [Fri, 1 Mar 2019 01:43:35 +0000 (12:13 +1030)]
ccan/io: add io_conn_exclusive and io_conn_out_exclusive.

There are cases where we want to suppress all activity except for a
single fd; we already have ugly io_flush_sync, but this is more
useful and more general.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agoccan: fix erroneous fd negation in poll.c.
Rusty Russell [Thu, 28 Feb 2019 05:26:10 +0000 (15:56 +1030)]
ccan: fix erroneous fd negation in poll.c.

We could just make them -1, since we don't rely on the values.
But simple negation doesn't make 0 invalid.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agoccan/io: keep always pointers to plans, not a linked list.
Rusty Russell [Thu, 28 Feb 2019 05:25:29 +0000 (15:55 +1030)]
ccan/io: keep always pointers to plans, not a linked list.

Duplex io_conns can be in the always list twice, and it was a source
of bugs, but I didn't want to add a second list_node.  Since there are
not many always at once, it's better (and more space-efficient) to use
a simple pointer array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agoccan/io: have io_plan mappable back to io_conn.
Rusty Russell [Thu, 28 Feb 2019 05:24:09 +0000 (15:54 +1030)]
ccan/io: have io_plan mappable back to io_conn.

io_conn contains two plans: by knowing what plan this is, it can be cast
back into the io_conn.  That helps for the next patch.

We also remove the strange logic where io_duplex() would return a
magic, invalid io_plan pointer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 months agocompiler: allow NON_NULL_ARGS with only a single argument.
Rusty Russell [Wed, 13 Feb 2019 01:47:56 +0000 (12:17 +1030)]
compiler: allow NON_NULL_ARGS with only a single argument.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 months agopipecmd: don't close stderr/stdout/stdin in parent when &pipecmd_preserve
Rusty Russell [Thu, 7 Feb 2019 03:56:58 +0000 (14:26 +1030)]
pipecmd: don't close stderr/stdout/stdin in parent when &pipecmd_preserve

This was reported and fixed by @m-schmoock for stderr, but I decided to
rework to make it clearer and cover the other cases:

https://github.com/ElementsProject/lightning/pull/2321/commits/67dad01549f876e5e620eeffe2f1e78c35bd0f03

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 months agoconfigurator: fix crash when more than one extra test provided
Rusty Russell [Tue, 5 Feb 2019 04:04:47 +0000 (14:34 +1030)]
configurator: fix crash when more than one extra test provided

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
11 months agotal: handle take() pointers more carefully.
Rusty Russell [Mon, 17 Dec 2018 00:16:32 +0000 (10:46 +1030)]
tal: handle take() pointers more carefully.

take() applies to the literal pointer value, so tal_free() means it may
randomly apply to a future allocation.  Similarly, tal_resize() should
preserve it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
13 months agopipecmd: reverse order of args to match documentation; add pipecmd_preserve.
Rusty Russell [Mon, 12 Nov 2018 05:41:18 +0000 (16:11 +1030)]
pipecmd: reverse order of args to match documentation; add pipecmd_preserve.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
13 months agosmall fix for ccan/take/take.c
Dmitry Petukhov [Sun, 4 Nov 2018 14:58:04 +0000 (19:58 +0500)]
small fix for ccan/take/take.c

Hello.

I've decided to take a look at c-lighting code, because we might
consider using it in the future.

I found a small problem in ccan/take/take.c that only concerns code
that runs with CCAN_TAKE_DEBUG set (DEVELOPER=1 for c-lighting).

It is a small issue, but I decided to notify you as the author of the
code, anyway.

the issue is:
  in take_() function, potential failure of realloc for labelarr is not
  handled.

I attached a diff with a fix.

I thought that making a pull request for c-lighting would not be right,
as ccan is a separate project, but I did not find a way to report this
at http://git.ozlabs.org/, where ccan repo resides.

Therefore I wrote to you directly.

[ Minor whitespace changes --RR ]

13 months agoopt: add new parse_early_args_incomplete.
Rusty Russell [Fri, 2 Nov 2018 01:27:57 +0000 (11:57 +1030)]
opt: add new parse_early_args_incomplete.

If we have plugins, and those can register args, we have a problem finding
the plugin dir!  So, do a best-effort incomplete parse.

Note that this can screw up in theory if we have "--unknown --foo" since we
don't know if unknown takes an argument (in which case, ignore --foo) or
not.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
13 months agocompiler: add more attributes.
Rusty Russell [Fri, 2 Nov 2018 01:01:39 +0000 (11:31 +1030)]
compiler: add more attributes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
13 months agoio: don't leave errno as a random value when we hit EOF.
Rusty Russell [Thu, 18 Oct 2018 00:29:00 +0000 (10:59 +1030)]
io: don't leave errno as a random value when we hit EOF.

It's used inside io_finish; setting to 0 allows that to know we hit
EOF on a read.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
14 months agohtable: add and integrate htable_check function.
Rusty Russell [Thu, 11 Oct 2018 23:54:10 +0000 (10:24 +1030)]
htable: add and integrate htable_check function.

Trying to debug what looked like an htable fail in my own code (but seems
it's not), I added htable_check a-la list_check et al.

This means we now depend on ccan/str (for stringify), which breaks the
_info example which also defined streq, and played nasty with const pointers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
14 months agorbuf, tools: clean up rbuf usage.
Rusty Russell [Fri, 12 Oct 2018 00:25:31 +0000 (10:55 +1030)]
rbuf, tools: clean up rbuf usage.

We should use rbuf primitives not reach inside to membuf.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
14 months agomembuf: add membuf_added and membuf_unadd APIs.
Rusty Russell [Fri, 12 Oct 2018 00:23:35 +0000 (10:53 +1030)]
membuf: add membuf_added and membuf_unadd APIs.

Clean up some whitespace while we're there too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
14 months agocrypto/shachain/tools: update to new rbuf API.
Rusty Russell [Fri, 21 Sep 2018 07:31:09 +0000 (17:01 +0930)]
crypto/shachain/tools: update to new rbuf API.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
14 months agostructeq: fix case where we mark padding as unknown.
Rusty Russell [Wed, 26 Sep 2018 23:39:50 +0000 (09:09 +0930)]
structeq: fix case where we mark padding as unknown.

And change semantics: a negative number means "up to this much padding".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
14 months agotools: fix compile after rbuf rewrite.
Rusty Russell [Wed, 26 Sep 2018 23:57:40 +0000 (09:27 +0930)]
tools: fix compile after rbuf rewrite.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
14 months agorbuf: adapt to work on ccan/membuf.
Rusty Russell [Mon, 17 Sep 2018 01:24:45 +0000 (10:54 +0930)]
rbuf: adapt to work on ccan/membuf.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
14 months agomembuf: new module for linear memory buffers.
Rusty Russell [Mon, 17 Sep 2018 01:17:42 +0000 (10:47 +0930)]
membuf: new module for linear memory buffers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
15 months agotal: allow notifiers on NULL.
Rusty Russell [Thu, 23 Aug 2018 02:41:13 +0000 (12:11 +0930)]
tal: allow notifiers on NULL.

A destructor on NULL doesn't make sense, but notifiers (eg. new children)
do.  We fix up a mistake in run-notifier (comparing ctx with itself) and
loose typing in tal.c's tal_add_notifier_ too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
16 months agoFix typos detected by github.com/client9/misspell
Kazuhiro Sera [Fri, 10 Aug 2018 06:54:10 +0000 (15:54 +0900)]
Fix typos detected by github.com/client9/misspell

16 months agotal/str: always create strings which have tal_count() == strlen() + 1.
Rusty Russell [Fri, 27 Jul 2018 06:25:50 +0000 (15:55 +0930)]
tal/str: always create strings which have tal_count() == strlen() + 1.

This is least-surprise, but also means callers can sometimes do faster
string handling by avoiding strstr().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
16 months agoccan/tal: always include a length field.
Rusty Russell [Fri, 27 Jul 2018 04:29:18 +0000 (13:59 +0930)]
ccan/tal: always include a length field.

The current semantics of tal_count() / tal_bytelen() are to return 0
for anything not allocated using tal_arr*.  This is because we tried
to save a native-length word in the header, but produces an awkward
API.

(To make it worse, defining CCAN_TAL_DEBUG turns length to always on,
and we enable that for c-lightning developer mode, which hides bugs!).

However, for c-lightning, just over half of allocations want a length:
these use 3 words each, so we're actually worse off overall.

The answer is to always have a length field in the header.  This also
simplfies the tal code.

samba-allocs stats before:
Tal time: 1237102-1305755(1.251e+06+/-2.1e+04)ns
Tal_free time: 1346871-1514514(1.37844e+06+/-5.2e+04)ns

After:
Tal time: 1115180-1180633(1.1351e+06+/-2.1e+04)ns
Tal_free time: 1334381-1465933(1.39148e+06+/-4.7e+04)ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
16 months agotal/str and tal/stack: use _label interfaces.
Rusty Russell [Fri, 27 Jul 2018 04:23:19 +0000 (13:53 +0930)]
tal/str and tal/stack: use _label interfaces.

In particular, tal/str now passes through the label from the caller,
so (in case of CCAN_TAL_DEBUG) you can actually see the file and line
where the caller was, not just inside ccan/str.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
16 months agotal: add _label interfaces.
Rusty Russell [Fri, 27 Jul 2018 04:22:20 +0000 (13:52 +0930)]
tal: add _label interfaces.

There are a number of other utilities which use the tal_alloc_ and
tal_dup_arr_ internal interfaces directly, because they want to set
the label themselves.  We're about to break them all by changing those
internal interfaces, so give them a mid-level interface to use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
16 months agotal: rename tal_len to tal_bytelen.
Rusty Russell [Fri, 27 Jul 2018 04:20:42 +0000 (13:50 +0930)]
tal: rename tal_len to tal_bytelen.

I had a bug caused by using tal_len instead of tal_count: let's make
it explicit. @jb55 commented "ha. I always forget which one does which... Ack"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
17 months agoccan/structeq: make it safe when there's padding.
Rusty Russell [Wed, 4 Jul 2018 04:07:28 +0000 (13:37 +0930)]
ccan/structeq: make it safe when there's padding.

ccan/cppmagic FTW!

The only issue is that we can't tell if there's padding or they've missed
a member, so we add a padding bytes count, so they'll get an error if it
(for example) the structure adds a new member later.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
17 months agoccan/utf8: new module.
Rusty Russell [Mon, 18 Jun 2018 10:42:46 +0000 (20:12 +0930)]
ccan/utf8: new module.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: add manual page.
Rusty Russell [Tue, 12 Jun 2018 02:37:51 +0000 (12:07 +0930)]
tools/configurator: add manual page.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: --extra-tests to read more test descriptions from stdin.
Rusty Russell [Tue, 12 Jun 2018 02:36:51 +0000 (12:06 +0930)]
tools/configurator: --extra-tests to read more test descriptions from stdin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: don't rely on size of tests_array.
Rusty Russell [Tue, 12 Jun 2018 02:35:51 +0000 (12:05 +0930)]
tools/configurator: don't rely on size of tests_array.

Groundwork for adding tests dynamically.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: better descriptions for --autotools-style.
Rusty Russell [Tue, 12 Jun 2018 02:34:51 +0000 (12:04 +0930)]
tools/configurator: better descriptions for --autotools-style.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: Better argument handling, particularly unknown arguments.
Rusty Russell [Tue, 12 Jun 2018 02:33:51 +0000 (12:03 +0930)]
tools/configurator: Better argument handling, particularly unknown arguments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: support --header-file if we don't want to write to stdout.
Rusty Russell [Tue, 12 Jun 2018 02:32:51 +0000 (12:02 +0930)]
tools/configurator: support --header-file if we don't want to write to stdout.

Works well with --autotools-style.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: support --autotools-style.
Rusty Russell [Tue, 12 Jun 2018 02:31:51 +0000 (12:01 +0930)]
tools/configurator: support --autotools-style.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: support --var-file for outputting VAR=VAL format.
Rusty Russell [Tue, 12 Jun 2018 02:30:51 +0000 (12:00 +0930)]
tools/configurator: support --var-file for outputting VAR=VAL format.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotools/configurator: convert flags to text in 'struct test'
Rusty Russell [Tue, 12 Jun 2018 02:29:51 +0000 (11:59 +0930)]
tools/configurator: convert flags to text in 'struct test'

This is the only non-text field.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
18 months agotimer: remove unnecessary dependency on likely
Eric Wong [Fri, 8 Jun 2018 02:45:42 +0000 (02:45 +0000)]
timer: remove unnecessary dependency on likely

I don't see likely/unlikely being used by ccan/timer or any of its
dependencies right now.

Running `tools/ccanlint/ccanlint ccan/timer' reveals no regressions

Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
19 months agoendian: fix compilation with musl libc.
Rusty Russell [Thu, 10 May 2018 02:25:59 +0000 (11:55 +0930)]
endian: fix compilation with musl libc.

It defines __BYTE_ORDER to __BYTE_ORDER__; gcc complains when we define
it to something else.  Let it be already defined, but check that the
value is what we expect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
19 months agoMakefile-web: any PHP version will do.
Rusty Russell [Wed, 9 May 2018 04:13:17 +0000 (13:43 +0930)]
Makefile-web: any PHP version will do.

php5 was finally removed from ozlabs.org, but php7.2 seems to work fine.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
19 months agoopt: print usage correctly for early args.
Rusty Russell [Wed, 9 May 2018 03:09:47 +0000 (12:39 +0930)]
opt: print usage correctly for early args.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agoFix HAVE_ASPRINTF detection
Kamil [Fri, 6 Apr 2018 23:00:42 +0000 (01:00 +0200)]
Fix HAVE_ASPRINTF detection

gcc6 introduceed a new warning switched with -Wmisleading-identation. This caused to generate a compilation warning for if statement misleadingly indented, which caused HAVE_ASPRINTF to be defined as 0. Adding newline after an if statement fixes the problem.

20 months agotal/grab_file: be robust against EINTR.
Rusty Russell [Fri, 6 Apr 2018 07:29:39 +0000 (16:59 +0930)]
tal/grab_file: be robust against EINTR.

Exracted (and slightly modified) from a MacOS PR for lightning.

Based-on-patch-by: https://github.com/conanoc
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agoptr_valid: fix spurious SIGINT under lldb on MacOS
Rusty Russell [Fri, 6 Apr 2018 07:26:26 +0000 (16:56 +0930)]
ptr_valid: fix spurious SIGINT under lldb on MacOS

(Taken from PR for lightning)

Patch-from: https://github.com/conanoc
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agobase32: add ability to substitute character set.
Rusty Russell [Thu, 5 Apr 2018 02:31:51 +0000 (12:01 +0930)]
base32: add ability to substitute character set.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agostr/base32: new module.
Rusty Russell [Thu, 5 Apr 2018 02:23:11 +0000 (11:53 +0930)]
str/base32: new module.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agointmap: add iterator-by-callback.
Rusty Russell [Mon, 26 Mar 2018 10:42:39 +0000 (21:12 +1030)]
intmap: add iterator-by-callback.

It's significantly faster because it assumes no deletion:

10000000,critbit iteration (nsec),316
10000000,critbit callback iteration (nsec),90
...
10000000,critbit consecutive iteration (nsec),308
10000000,critbit consecutive callback iteration (nsec),78

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agointmap: add exhaustive testcases for intmap_after
Rusty Russell [Mon, 26 Mar 2018 10:38:34 +0000 (21:08 +1030)]
intmap: add exhaustive testcases for intmap_after

We can't do the full range, but we can for a handful of bits (8).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agointmap: add test case which failed, extracted from real world usage.
Rusty Russell [Mon, 26 Mar 2018 10:37:34 +0000 (21:07 +1030)]
intmap: add test case which failed, extracted from real world usage.

Because intmap_after_() would simply examine the critbits to walk the
tree, it wouldn't realize that it might be in the completely wrong tree.

In this case:

         Bit 4:
         0   1
        /     \
       /       \
  100000011  100001011

When we ask for intmap_after_(011111111) we would check the critbit, it's
a 1, so we end up on the right leaf instead of the left.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agointmap: reimplement so that intmap_after works.
Rusty Russell [Mon, 26 Mar 2018 10:36:34 +0000 (21:06 +1030)]
intmap: reimplement so that intmap_after works.

A critbit tree is a binary tree which keeps branches for each bit
which differs in the leaves.  It's a simple data structure, but not
entirely simple to implement the primitives people expect, as this bus
shows.

The bug: I added an open iterator, and intmap_after_ for a random
value would sometimes return the wrong node.

Cause: we don't know what the prefix is as we iterate, so by only
testing the critbits in the tree, we can end up in the wrong place.
This is only a problem if the value isn't in the (sub)tree, but this
can easily happen even with contiguous trees should deletion occur.
You can see an example in the next patch, which adds a test.

After finding a bug in my intmap_after() routine, I went searching for
other implementations to see how they handled it.  Most didn't provide
an open-ended iterator like this, relying on callback iterators which
don't allow deletion.  Gah!

The exception was https://github.com/blynn/blt/blob/master/blt.c#L179
which implements blt_ceil() which does this (if you add one to the
key, at least).  However, it does it by effectively finding a node,
using that to derive the prefix, then walking down the tree again.
That's pretty suboptimal.

There are basically two choices if you want an efficient after()
operation: to reimplement this approach with some optimizations
(ie. keep branches as we descend, and when we get to the bottom and
know the prefix, we know which branch to go down), or keep the bits
which got to each node.

The latter is more optimal, but less generally useful: for bit
strings, for example, we could keep the bits in common on each node,
rather than storing the entire string at the bottom.  But in practice
you'd be doing allocations to re-create the index if the caller wanted
it.

However, in this implementation our keys are 64 bits only, and we
already use a u8 for the bit number: using a 64-bit value there
consumes no more space (thanks to alignment).  We can store the
critbit by using the prefix capped by a bit: 0b10000...0000 means
no prefix and highest bit is the critbit, and 0bxxxxx1000...000
means the prefix is xxxxxx and the critbit is the 6th highest bit.

The penalty is that iteration 70% slower.  It's still pretty fast
though.

Before:
$ for i in `seq 5`; do ./speed 10000000; done | stats
10000000,random generation (nsec),3-4(3.2+/-0.4)
10000000,critbit insert (nsec),1530-1751(1633.2+/-80)
10000000,critbit successful lookup (nsec),1723-1993(1806.8+/-97)
10000000,critbit failed lookup (nsec),1763-2104(1933.6+/-1.3e+02)
10000000,critbit iteration (nsec),208-266(242.2+/-19)
10000000,critbit memory (bytes),48
10000000,critbit delete (nsec),1747-1861(1803.8+/-42)
10000000,critbit consecutive iteration (nsec),182-228(210+/-18)

After:
10000000,random generation (nsec),3-4(3.2+/-0.4)
10000000,critbit insert (nsec),1533-1699(1628+/-65)
10000000,critbit successful lookup (nsec),1831-2104(1972.4+/-1e+02)
10000000,critbit failed lookup (nsec),1850-2152(2008.2+/-1.1e+02)
10000000,critbit iteration (nsec),304-324(312.8+/-7.5)
10000000,critbit memory (bytes),48
10000000,critbit delete (nsec),1617-1872(1752+/-99)
10000000,critbit consecutive iteration (nsec),303-318(311+/-5.4)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agointmap: add benchmarks.
Rusty Russell [Mon, 26 Mar 2018 10:27:12 +0000 (20:57 +1030)]
intmap: add benchmarks.

I wrote these a while ago, dig them out.

On my laptop, min-max(avg+/-stdev) of 5 runs:

make && for i in `seq 5`; do ./speed 10000000; done | stats
make: Nothing to be done for 'all'.
10000000,random generation (nsec),3-4(3.2+/-0.4)
10000000,critbit insert (nsec),1530-1751(1633.2+/-80)
10000000,critbit successful lookup (nsec),1723-1993(1806.8+/-97)
10000000,critbit failed lookup (nsec),1763-2104(1933.6+/-1.3e+02)
10000000,critbit iteration (nsec),208-266(242.2+/-19)
10000000,critbit memory (bytes),48
10000000,critbit delete (nsec),1747-1861(1803.8+/-42)
10000000,critbit consecutive iteration (nsec),182-228(210+/-18)
10000000,hash insert (nsec),396-424(412+/-9.6)
10000000,hash successful lookup (nsec),150-164(157.4+/-5.5)
10000000,hash failed lookup (nsec),163-178(170+/-5.5)
10000000,hash iteration (nsec),21-26(23.2+/-1.7)
10000000,hash memory (bytes),45
10000000,hash delete (nsec),179-194(183.6+/-5.3)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agobitops: new module.
Rusty Russell [Mon, 26 Mar 2018 05:03:11 +0000 (15:33 +1030)]
bitops: new module.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agoconfigurator: add tests for other popcount variants.
Rusty Russell [Mon, 26 Mar 2018 05:02:55 +0000 (15:32 +1030)]
configurator: add tests for other popcount variants.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agoMakefile: Fix asort multiple definition error
Jan Sarenik [Fri, 23 Feb 2018 05:44:25 +0000 (06:44 +0100)]
Makefile: Fix asort multiple definition error

Error I experienced on Alpine Linux without this patch:

    In file included from ccan/generator/generator.c:8:0:
    ./ccan/generator/generator.h:23:2: error: #error Generators require coroutines
     #error Generators require coroutines
      ^~~~~
    make: *** [Makefile:32: ccan/generator/generator.o] Error 1

21 months agofix misuse of typesafe_cb_cast() in example
Yubin Ruan [Wed, 14 Mar 2018 03:16:42 +0000 (11:16 +0800)]
fix misuse of typesafe_cb_cast() in example

From 32f86c701ab0e0ad0ad6981314a9bff2dc5ebb74 Mon Sep 17 00:00:00 2001
From: Yubin Ruan <ablacktshirt@gmail.com>
Date: Wed, 14 Mar 2018 11:14:54 +0800
Subject: [PATCH] fix misuse of typesafe_cb_cast() in example

Signed-off-by: Yubin Ruan <ablacktshirt@gmail.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
21 months agofix misspelling in the example of container_of
Yubin Ruan [Mon, 12 Mar 2018 03:24:14 +0000 (11:24 +0800)]
fix misspelling in the example of container_of

From 47c92fe951545e780ca31c598bbcbe5347059b27 Mon Sep 17 00:00:00 2001
From: Yubin Ruan <ablacktshirt@gmail.com>
Date: Mon, 12 Mar 2018 11:22:35 +0800
Subject: [PATCH] fix misspelling in the example of container_of

Signed-off-by: Yubin Ruan <ablacktshirt@gmail.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
21 months agotal: don't access freed memory on unusual case of redundant tal_free() in destructor.
Rusty Russell [Thu, 1 Mar 2018 06:25:51 +0000 (16:55 +1030)]
tal: don't access freed memory on unusual case of redundant tal_free() in destructor.

We already handle normal free traversal loops, just not ones caused by a
direct tal_free() call, such a calling tal_free() on one's own parent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
21 months agointmap: implement uintmap_last/sintmap_last.
Rusty Russell [Mon, 26 Feb 2018 04:33:28 +0000 (15:03 +1030)]
intmap: implement uintmap_last/sintmap_last.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
21 months agotools/configurator: allow overriding of which cc we will run.
Rusty Russell [Mon, 26 Feb 2018 02:26:18 +0000 (12:56 +1030)]
tools/configurator: allow overriding of which cc we will run.

This is for cross-configuring, where we might want to run
`qemu-user-... gcc` or even more exotic things.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
21 months agopath: expose separator constants.
Rusty Russell [Fri, 16 Feb 2018 00:06:11 +0000 (10:36 +1030)]
path: expose separator constants.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
22 months agoio: don't fail if we get a signal.
Rusty Russell [Sun, 4 Feb 2018 23:31:51 +0000 (10:01 +1030)]
io: don't fail if we get a signal.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
23 months agotal/path: handle weird case of path_join("")
Rusty Russell [Thu, 21 Dec 2017 23:41:09 +0000 (10:11 +1030)]
tal/path: handle weird case of path_join("")

It seems most sensible to make it a noop, but it definitely shouldn't
access out of bounds as it does.

Reported-by: Russ Dill
Fixes: #61
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoopt: fix libmusl compile.
Rusty Russell [Wed, 22 Nov 2017 05:44:02 +0000 (16:14 +1030)]
opt: fix libmusl compile.

Fixes: #63
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoio: query whether io_plan in/out have started.
Rusty Russell [Wed, 25 Oct 2017 05:39:47 +0000 (16:09 +1030)]
io: query whether io_plan in/out have started.

For lightning, we want to hand the socket off to another daemon, but we need
to be on a packet boundary.  This lets us check if we've part-read or
part-written.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoio: fix corner case in EPIPE handling.
Rusty Russell [Thu, 12 Oct 2017 06:33:09 +0000 (17:03 +1030)]
io: fix corner case in EPIPE handling.

If io_read is always called, we don't know that it will actually read,
so it might not notice error.  In that case, safest to fail immediately.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoio: fix duplex read on last bytes of closed pipe.
Rusty Russell [Thu, 12 Oct 2017 06:28:34 +0000 (16:58 +1030)]
io: fix duplex read on last bytes of closed pipe.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoccan/io: example failure
Rusty Russell [Mon, 4 Sep 2017 00:51:45 +0000 (10:21 +0930)]
ccan/io: example failure

So many bugs in one example program!

There was an unrelated but which strace revealed (trying to write -7
bytes), but I think your issue was more prosaic: failing to zero the
from buffer.

Reported-by: Ian Zimmerman <itz@very.loosely.org>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoendian: Add Glibc like endianess check
Akshay Adiga [Tue, 12 Sep 2017 05:23:13 +0000 (10:53 +0530)]
endian: Add Glibc like endianess check

An application built using glibc would expect __BYTE_ORDER to tell if
it should be compiled for BIG_ENDIAN or LITTLE_ENDIAN, whereas ccan uses
HAVE_LITTLE_ENDIAN and HAVE_BIG_ENDIAN for the same purpose.

Hence setting __BYTE_ORDER based on what CCAN provides will no longer
break the applications which check endianness the glibc way.

Signed-off-by: Akshay Adiga <akshay.adiga@linux.vnet.ibm.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agotal: add xor into child_parent pointer.
Rusty Russell [Mon, 4 Sep 2017 06:57:03 +0000 (16:27 +0930)]
tal: add xor into child_parent pointer.

I had a case where I was handing a sub-object (not a tal object!) to
tal_steal() and it wasn't detected, because the pointers looked correct.

This should help.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agotal: fix up benchmarks for interface changes.
Rusty Russell [Mon, 4 Sep 2017 04:36:46 +0000 (14:06 +0930)]
tal: fix up benchmarks for interface changes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agodarray: Fix bug in the darray_remove() macro
Damien Grassart [Tue, 29 Aug 2017 10:08:42 +0000 (12:08 +0200)]
darray: Fix bug in the darray_remove() macro

The memmove() call should be using the index argument to determine the
number of bytes to copy. To be consistent with the rest of the code,
we should also not evaluate the index parameter multiple
times. Calling this with rand() % arr.size would otherwise generally
segfault.

Finally, we want to avoid using "index" as an identifier so as to not
shadow index(3) in the C library.

Signed-off-by: Damien Grassart <damien@grassart.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2 years agodarray: Rename identifiers starting with an underscore
Damien Grassart [Tue, 29 Aug 2017 10:08:41 +0000 (12:08 +0200)]
darray: Rename identifiers starting with an underscore

Identifiers starting with underscores are technically reserved for
system use, so rename all of them to end with one instead.

Signed-off-by: Damien Grassart <damien@grassart.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>