ccan
5 months agotal: add tal_dup_or_null and tal_dup_talarr helpers. master origin/master
Rusty Russell [Wed, 15 Dec 2021 01:35:08 +0000 (12:05 +1030)]
tal: add tal_dup_or_null and tal_dup_talarr helpers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
5 months agocrc32c, crypto/xtea: check in LICENSE symlinks.
Rusty Russell [Mon, 13 Dec 2021 23:35:39 +0000 (10:05 +1030)]
crc32c, crypto/xtea: check in LICENSE symlinks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
5 months agographql: latest changes from @rl-d
Rusty Russell [Mon, 13 Dec 2021 23:32:47 +0000 (10:02 +1030)]
graphql: latest changes from @rl-d

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
5 months agoccan: make tal_dump() format more regular.
Rusty Russell [Wed, 24 Nov 2021 02:06:52 +0000 (12:36 +1030)]
ccan: make tal_dump() format more regular.

Putting a \n in there was a pain: I had to do:

awk '/CHILDREN/ { SAVED=$0 } !/CHILDREN/ { print SAVED$0; SAVED="" }' < lightningd-tal_dump > lightningd-tal_dump-children-on-same-line

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agopipecmd: Use closefrom instead of iterating directly.
ZmnSCPxj jxPCSnmZ [Tue, 19 Oct 2021 01:55:48 +0000 (09:55 +0800)]
pipecmd: Use closefrom instead of iterating directly.

Signed-off-by: ZmnSCPxj jxPCSnmZ <ZmnSCPxj@protonmail.com>
6 months agoclosefrom: fix minor issues found by ccanlint
Rusty Russell [Thu, 21 Oct 2021 04:31:18 +0000 (15:01 +1030)]
closefrom: fix minor issues found by ccanlint

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
6 months agoclosefrom: Close all file descriptors above a certain value.
ZmnSCPxj jxPCSnmZ [Tue, 19 Oct 2021 01:51:43 +0000 (09:51 +0800)]
closefrom: Close all file descriptors above a certain value.

For more information: https://github.com/ElementsProject/lightning/issues/4868

Signed-off-by: ZmnSCPxj jxPCSnmZ <ZmnSCPxj@protonmail.com>
8 months agoweb: create "versioned" tarballs.
Rusty Russell [Wed, 8 Sep 2021 07:29:02 +0000 (16:59 +0930)]
web: create "versioned" tarballs.

Fixes: https://github.com/rustyrussell/ccan/issues/90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
8 months agobase64: fix assignment to wrong type, use ssize_t
Martin Milata [Thu, 11 May 2017 16:28:24 +0000 (18:28 +0200)]
base64: fix assignment to wrong type, use ssize_t

Functions that returned int have been modified to return ssize_t
instead.

Found by gcc:

base64.c:196:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  if (more == -1) {
           ^

Signed-off-by: Martin Milata <martin@martinmilata.cz>
8 months agonit: daemonize.c: fix Stevens' in comment
Jan Sarenik [Tue, 27 Oct 2020 07:21:25 +0000 (08:21 +0100)]
nit: daemonize.c: fix Stevens' in comment

8 months agoSetting the same name as at the implementation
Renat Idrisov [Tue, 4 Jun 2019 21:05:40 +0000 (14:05 -0700)]
Setting the same name as at the implementation

8 months agoconfigurator: always true branch
Niklas Claesson [Wed, 5 Feb 2020 21:03:10 +0000 (22:03 +0100)]
configurator: always true branch

8 months agoFix "for loop initial declarations are only allowed in C99 mode" compile errors.
Christopher McGeorge [Mon, 22 Mar 2021 22:24:16 +0000 (16:24 -0600)]
Fix "for loop initial declarations are only allowed in C99 mode" compile errors.

8 months agodocs: fix simple typo, unchaged -> unchanged
Tim Gates [Fri, 11 Dec 2020 18:59:26 +0000 (05:59 +1100)]
docs: fix simple typo, unchaged -> unchanged

There is a small typo in ccan/objset/objset.h.

Should read `unchanged` rather than `unchaged`.

8 months agofix warnings
rl-d [Wed, 8 Sep 2021 04:07:28 +0000 (00:07 -0400)]
fix warnings

8 months agoGraphQL lexer and parser (without optional type system)
rl-d [Wed, 8 Sep 2021 02:03:07 +0000 (22:03 -0400)]
GraphQL lexer and parser (without optional type system)

8 months agotal: tal_dump() now prints to stderr.
Rusty Russell [Mon, 23 Aug 2021 06:40:10 +0000 (16:10 +0930)]
tal: tal_dump() now prints to stderr.

This is more widely available than stdout, which can interfere with
other things.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
11 months agocompiler: RETURNS_NONNULL macro
Rusty Russell [Wed, 9 Jun 2021 05:42:28 +0000 (15:12 +0930)]
compiler: RETURNS_NONNULL macro

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
11 months agoconfigurator: HAVE_ATTRIBUTE_RETURNS_NONNULL
Rusty Russell [Wed, 9 Jun 2021 05:42:22 +0000 (15:12 +0930)]
configurator: HAVE_ATTRIBUTE_RETURNS_NONNULL

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
13 months agotimer: handle time going backwards.
Rusty Russell [Tue, 23 Mar 2021 03:06:50 +0000 (13:36 +1030)]
timer: handle time going backwards.

As seen in https://github.com/ElementsProject/lightning/issues/4401

OpenBSD 6.8 inside VirtualBox.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
17 months agoutf8: don't allow NUL in decoded strings.
Rusty Russell [Tue, 1 Dec 2020 00:35:48 +0000 (11:05 +1030)]
utf8: don't allow NUL in decoded strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months agojson_out: make json_out_finished finish buffer.
Rusty Russell [Mon, 14 Sep 2020 05:30:55 +0000 (15:00 +0930)]
json_out: make json_out_finished finish buffer.

For JSON RPC, we can actually write multiple JSON objects back
to back, and we don't want a comma between them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
20 months ago*: Fix build with recent gcc/binutils
Kirill Smelkov [Thu, 10 Sep 2020 19:16:19 +0000 (19:16 +0000)]
*: Fix build with recent gcc/binutils

With current Debian testing I get:

tools/ccanlint/ccanlint -s --deps-fail-ignore   ccan/compiler/ && touch ccan/compiler/.ok
compiler: Module tests compile (tests_compile): FAIL
/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/compiler/test/run-is_compile_constant.c:Compile failed:
/usr/bin/ld: /tmp/ccanlint-129350.1804289383/tap.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/tap/tap.h:249: multiple definition of `tap_fail_callback'; /tmp/ccrAwqbl.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/tap/tap.h:249: first defined here
collect2: error: ld returned 1 exit status
...

cc -g3 -ggdb -Wall -Wstrict-prototypes -Wold-style-definition -Wundef -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wwrite-strings -DCCAN_STR_DEBUG=1 -I.  -MMD -MP -MFtools/ccanlint/ccanlint.d -MTtools/ccanlint/ccanlint tools/ccanlint/ccanlint.c tools/ccanlint/async.o tools/ccanlint/file_analysis.o tools/ccanlint/licenses.o tools/ccanlint/tests/avoids_cpp_reserved.o tools/ccanlint/tests/depends_accurate.o tools/ccanlint/tests/depends_build.o tools/ccanlint/tests/depends_build_without_features.o tools/ccanlint/tests/depends_exist.o tools/ccanlint/tests/examples_compile.o tools/ccanlint/tests/examples_exist.o tools/ccanlint/tests/examples_relevant.o tools/ccanlint/tests/examples_run.o tools/ccanlint/tests/hash_if.o tools/ccanlint/tests/headers_idempotent.o tools/ccanlint/tests/info_compiles.o tools/ccanlint/tests/info_documentation_exists.o tools/ccanlint/tests/info_exists.o tools/ccanlint/tests/info_ported.o tools/ccanlint/tests/info_summary_single_line.o tools/ccanlin
 t/tests/license_comment.o tools/ccanlint/tests/license_depends_compat.o tools/ccanlint/tests/license_exists.o tools/ccanlint/tests/license_file_compat.o tools/ccanlint/tests/main_header_compiles.o tools/ccanlint/tests/main_header_exists.o tools/ccanlint/tests/module_builds.o tools/ccanlint/tests/module_links.o tools/ccanlint/tests/no_trailing_whitespace.o tools/ccanlint/tests/objects_build.o tools/ccanlint/tests/objects_build_with_stringchecks.o tools/ccanlint/tests/objects_build_without_features.o tools/ccanlint/tests/reduce_features.o tools/ccanlint/tests/tests_compile.o tools/ccanlint/tests/tests_compile_coverage.o tools/ccanlint/tests/tests_coverage.o tools/ccanlint/tests/tests_exist.o tools/ccanlint/tests/tests_helpers_compile.o tools/ccanlint/tests/tests_pass.o tools/ccanlint/tests/tests_pass_valgrind.o tools/ccanlint/tests/tests_pass_without_features.o ccan/autodata/autodata.o ccan/dgraph/dgraph.o ccan/ilog/ilog.o ccan/lbalance/lbalance.o ccan/ptr_valid/ptr_valid.o ccan/strma
 p/strmap.o tools/ccan_dir.o tools/compi
le.o tools/depends.o tools/doc_extract-core.o tools/gcov.o tools/manifest.o tools/read_config_header.o tools/tools.o ccan/asort/asort.o ccan/err/err.o ccan/foreach/foreach.o ccan/hash/hash.o ccan/htable/htable.o ccan/list/list.o ccan/membuf/membuf.o ccan/noerr/noerr.o ccan/opt/helpers.o ccan/opt/opt.o ccan/opt/parse.o ccan/opt/usage.o ccan/rbuf/rbuf.o ccan/read_write_all/read_write_all.o ccan/str/debug.o ccan/str/str.o ccan/take/take.o ccan/tal/tal.o ccan/tal/grab_file/grab_file.o ccan/tal/link/link.o ccan/tal/path/path.o ccan/tal/str/str.o ccan/time/time.o -lm -o tools/ccanlint/ccanlint
/usr/bin/ld: tools/ccanlint/async.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/file_analysis.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/licenses.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/avoids_cpp_reserved.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined
here
/usr/bin/ld: tools/ccanlint/tests/depends_accurate.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/depends_build.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/depends_build_without_features.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
...

The errors are valid becuase symbols are declared without extern in .h
files and those .h files are included from multiple translation units.

-> Fix it. I'm not sure why it used to compile ok before.

Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Message-Id: <20200910191610.131510-1-kirr@nexedi.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
22 months agotools/configurator: fix compile error with -O2
Rusty Russell [Fri, 17 Jul 2020 01:56:51 +0000 (11:26 +0930)]
tools/configurator: fix compile error with -O2

In file included from /usr/include/string.h:495,
                 from configuratortest.c:2:
In function ‘strncpy’,
    inlined from ‘main’ at configuratortest.c:6:2:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: warning: ‘__builtin_strncpy’ specified bound 8 equals destination size [-Wstringop-truncation]
  106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
22 months agotal: document tal_steal(ctx, NULL) as OK.
Rusty Russell [Fri, 19 Jun 2020 01:45:21 +0000 (11:15 +0930)]
tal: document tal_steal(ctx, NULL) as OK.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agohtable: add htable_type _delval.
Rusty Russell [Thu, 20 Feb 2020 02:31:19 +0000 (13:01 +1030)]
htable: add htable_type _delval.

To match the raw delval

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agohtable: fix type of htable_type _pick return.
Rusty Russell [Thu, 20 Feb 2020 02:23:11 +0000 (12:53 +1030)]
htable: fix type of htable_type _pick return.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agohtable: htable_pick helper to select a random entry.
Rusty Russell [Wed, 19 Feb 2020 23:48:10 +0000 (10:18 +1030)]
htable: htable_pick helper to select a random entry.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agotal: don't defer-after-free if a destructor deletes itself.
Rusty Russell [Mon, 17 Feb 2020 00:12:46 +0000 (10:42 +1030)]
tal: don't defer-after-free if a destructor deletes itself.

==10868==    at 0x109A96: notify (tal.c:220)
==10868==    by 0x109F7E: del_tree (tal.c:397)
==10868==    by 0x10A31A: tal_free (tal.c:481)
==10868==    by 0x10BE73: main (run-destructor.c:75)
==10868==  Address 0x4a60bd8 is 8 bytes inside a block of size 32 free'd
==10868==    at 0x483BA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==10868==    by 0x109D4F: del_notifier_property (tal.c:340)
==10868==    by 0x10A610: tal_del_notifier_ (tal.c:564)
==10868==    by 0x10A687: tal_del_destructor_ (tal.c:576)
==10868==    by 0x10B653: remove_own_destructor (run-destructor.c:35)
==10868==    by 0x109A67: notify (tal.c:235)
==10868==    by 0x109F7E: del_tree (tal.c:397)
==10868==    by 0x10A31A: tal_free (tal.c:481)
==10868==    by 0x10BE73: main (run-destructor.c:75)
==10868==  Block was alloc'd at
==10868==    at 0x483A7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==10868==    by 0x109AD5: allocate (tal.c:245)
==10868==    by 0x109C3E: add_notifier_property (tal.c:300)
==10868==    by 0x10A467: tal_add_destructor_ (tal.c:516)
==10868==    by 0x10BDFE: main (run-destructor.c:74)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoopt: fix opt_unregister.
Rusty Russell [Tue, 11 Feb 2020 04:09:43 +0000 (14:39 +1030)]
opt: fix opt_unregister.

Instead of memmoving N structs, we were memmoving N bytes.

But why did the test pass then?  It was doing memmove(..., 1)
instead of memmove(..., sizeof(struct opt_table)!

Because the two structures were really similar; the main difference
was the first entry, which points to the name.  But they were allocated
consecutively, and Intel being little-endian, the only difference was
the first byte!  Thus memmove(1) was enough to make it "work".

Change two options in the test to be sufficiently different, and
the bug shows up.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoxtea: new module.
Rusty Russell [Tue, 4 Feb 2020 02:38:02 +0000 (13:08 +1030)]
xtea: new module.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoopt: add opt_unregister.
Rusty Russell [Tue, 4 Feb 2020 01:10:36 +0000 (11:40 +1030)]
opt: add opt_unregister.

Sometimes we all make mistakes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agotools/configurator: simplify wrapper code a little.
Rusty Russell [Tue, 4 Feb 2020 00:38:35 +0000 (11:08 +1030)]
tools/configurator: simplify wrapper code a little.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoconfigurator: Enable running configurator in wrapper
Niklas Claesson [Tue, 28 Jan 2020 23:50:15 +0000 (00:50 +0100)]
configurator: Enable running configurator in wrapper

Since the probing binaries compiled by the configurator needs to run on
the host machine we provide a variable CONFIGURATOR_WRAPPER that can be
set to anything that you want to wrap the calls with.

One example is `qemu-aarch64-static`.

2 years agobitmap: Allow bitmap type to be forward declared
Kirill Smelkov [Mon, 21 Oct 2019 15:09:20 +0000 (15:09 +0000)]
bitmap: Allow bitmap type to be forward declared

Currently bitmap type is defined via untagged struct which makes it
impossible to forward declare it. Forward-declaring is useful since all
bitmap functions only use bitmap* and in public user-visible
headers/datastructures it is enough to indicate that a data field with
bitmap pointer is there, whereas bitmap.h can be included only in
implementation.

Beside that some headers are included by both C and C++ parts of a
project, and when ccan/bitmap.h is processed by C++ compiler it gives:

    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_alloc(long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:201:15: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      return malloc(bitmap_sizeof(nbits));
             ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_realloc0(bitmap*, long unsigned int, long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:227:18: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      bitmap = realloc(bitmap, bitmap_sizeof(nbits));
               ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_realloc1(bitmap*, long unsigned int, long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:238:18: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      bitmap = realloc(bitmap, bitmap_sizeof(nbits));

-> Give to users ability not to force-include ccan/bitmap.h by
forward-declaring bitmaps like this:

    typedef struct bitmap bitmap;
    ...
    struct MyStruct {
        bitmap *my_bitmap;
    };

Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Message-Id: <20191021150903.25159-1-kirr@nexedi.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2 years agointmap: implement intmap_before()
Rusty Russell [Thu, 10 Oct 2019 05:06:00 +0000 (15:36 +1030)]
intmap: implement intmap_before()

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agofailtest, rbuf: fix up incorrect lseek arg order.
Rusty Russell [Fri, 4 Oct 2019 05:07:14 +0000 (14:37 +0930)]
failtest, rbuf: fix up incorrect lseek arg order.

SEEK_SET == 0, so it's not hurting anyone now, but when I cut & paste
it elsewhere...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agopipecmd: close fds in child.
Rusty Russell [Wed, 14 Aug 2019 03:43:46 +0000 (13:13 +0930)]
pipecmd: close fds in child.

This is usually what you want; I didn't even add a flag to stop it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agohtable: add htable_count().
Rusty Russell [Tue, 13 Aug 2019 03:02:42 +0000 (12:32 +0930)]
htable: add htable_count().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoSuppress false positive warning "-Wmaybe-uninitialized" with GCC -O3
Chirimen-Jako [Wed, 31 Jul 2019 09:21:58 +0000 (18:21 +0900)]
Suppress false positive warning "-Wmaybe-uninitialized" with GCC -O3

2 years agoconfigurator: fix openmp test on some versions of clang.
Rusty Russell [Wed, 31 Jul 2019 06:56:30 +0000 (16:26 +0930)]
configurator: fix openmp test on some versions of clang.

Discovered on bionic's Travis install, it failed at *runtime* when
it couldn't find the shared library.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agobitmap: Fix some bugs on 32-bit platforms
David Gibson [Thu, 11 Jul 2019 02:07:33 +0000 (12:07 +1000)]
bitmap: Fix some bugs on 32-bit platforms

The bitmap_word type is an unsigned long.  However in some places we assign
it using -1ULL, a 64-bit value on many platforms.  We sometimes get away
with this because it masks correctly, but in other cases it breaks things.

To clean this up define a new BITMAP_WORD_1 constant, indicating a
bitmap_word with all bits set, and use that instead of explicit UL or ULL
qualifiers.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2 years agoccan/crc: remove broken "crc32c" routines which gave wrong results.
Rusty Russell [Tue, 11 Jun 2019 04:56:02 +0000 (14:26 +0930)]
ccan/crc: remove broken "crc32c" routines which gave wrong results.

I copied them from the kernel, but obviously didn't find good test
vectors.  In particular, the CRC of all zeroes is not 0!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoccan/crc32c: new module for accelerated CRC32 (on x86-64).
Rusty Russell [Tue, 11 Jun 2019 04:54:56 +0000 (14:24 +0930)]
ccan/crc32c: new module for accelerated CRC32 (on x86-64).

Note: the previous code in ccan/crc is wrong, so I started fresh with
actual test vectors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoccan/compiler: add cpu_supports() macro.
Rusty Russell [Tue, 11 Jun 2019 04:54:11 +0000 (14:24 +0930)]
ccan/compiler: add cpu_supports() macro.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agoconfigurator: detect __builtin_cpu_supports.
Rusty Russell [Tue, 11 Jun 2019 04:53:52 +0000 (14:23 +0930)]
configurator: detect __builtin_cpu_supports.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agojson_out: pass through OOM failures.
Rusty Russell [Thu, 23 May 2019 05:29:44 +0000 (14:59 +0930)]
json_out: pass through OOM failures.

And fix escaping to work with addstr(), and assert in debug mode
if they don't specify quotes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agojson_out: new module for authoring JSON strings.
Rusty Russell [Tue, 21 May 2019 06:41:58 +0000 (16:11 +0930)]
json_out: new module for authoring JSON strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agojson_escape: add fast-path for when we don't need to escape.
Rusty Russell [Tue, 21 May 2019 04:34:49 +0000 (14:04 +0930)]
json_escape: add fast-path for when we don't need to escape.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2 years agojson_escape: new module to escape JSON strings.
Rusty Russell [Tue, 21 May 2019 04:21:12 +0000 (13:51 +0930)]
json_escape: new module to escape JSON strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agohtable: avoid branch in calculating perfect bit.
Rusty Russell [Tue, 2 Apr 2019 01:19:28 +0000 (11:49 +1030)]
htable: avoid branch in calculating perfect bit.

Final results of tools/speed/10000000 (10 runs) shows a slight
slowdown in some tests, but it makes an empty htable smaller.

-Initial delete all: 96-98(96.4+/-0.66) ns
+Initial delete all: 97-99(98.2+/-0.75) ns
-Initial re-inserting: 117-124(121.4+/-1.9) ns
+Initial re-inserting: 124-131(126.4+/-2.4) ns
-Adding (a different) half: 49-50(49.3+/-0.46) ns
+Adding (a different) half: 50-52(51.2+/-0.75) ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agohtable: reduce size of htable by storing perfect bitnum, not mask.
Rusty Russell [Mon, 1 Apr 2019 23:49:19 +0000 (10:19 +1030)]
htable: reduce size of htable by storing perfect bitnum, not mask.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agohtable: reduce size of htable by calculating max every time.
Rusty Russell [Mon, 1 Apr 2019 23:20:37 +0000 (09:50 +1030)]
htable: reduce size of htable by calculating max every time.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoio: fix another leak path for always array.
Rusty Russell [Sat, 16 Mar 2019 04:59:24 +0000 (15:29 +1030)]
io: fix another leak path for always array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoio: don't leak memory on clean shutdown.
Rusty Russell [Sat, 16 Mar 2019 04:27:05 +0000 (14:57 +1030)]
io: don't leak memory on clean shutdown.

Free the internal "always" array like we free "fds".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoio: typo fixes.
Rusty Russell [Sat, 16 Mar 2019 03:14:13 +0000 (13:44 +1030)]
io: typo fixes.

Suggested-by: Lisa Neigut <@niftynei>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agohtable: add allocator hooks.
Rusty Russell [Mon, 4 Mar 2019 10:25:34 +0000 (20:55 +1030)]
htable: add allocator hooks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agohtable: commit extra tests I had lying around.
Rusty Russell [Mon, 4 Mar 2019 10:25:54 +0000 (20:55 +1030)]
htable: commit extra tests I had lying around.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotimer: clean up hook allocator API
Rusty Russell [Mon, 4 Mar 2019 11:06:59 +0000 (21:36 +1030)]
timer: clean up hook allocator API

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotimer: add hook for allocation functions.
Rusty Russell [Mon, 4 Mar 2019 09:12:57 +0000 (19:42 +1030)]
timer: add hook for allocation functions.

malloc and free aren't for everyone.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoccan/io: add io_conn_exclusive and io_conn_out_exclusive.
Rusty Russell [Fri, 1 Mar 2019 01:43:35 +0000 (12:13 +1030)]
ccan/io: add io_conn_exclusive and io_conn_out_exclusive.

There are cases where we want to suppress all activity except for a
single fd; we already have ugly io_flush_sync, but this is more
useful and more general.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoccan: fix erroneous fd negation in poll.c.
Rusty Russell [Thu, 28 Feb 2019 05:26:10 +0000 (15:56 +1030)]
ccan: fix erroneous fd negation in poll.c.

We could just make them -1, since we don't rely on the values.
But simple negation doesn't make 0 invalid.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoccan/io: keep always pointers to plans, not a linked list.
Rusty Russell [Thu, 28 Feb 2019 05:25:29 +0000 (15:55 +1030)]
ccan/io: keep always pointers to plans, not a linked list.

Duplex io_conns can be in the always list twice, and it was a source
of bugs, but I didn't want to add a second list_node.  Since there are
not many always at once, it's better (and more space-efficient) to use
a simple pointer array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoccan/io: have io_plan mappable back to io_conn.
Rusty Russell [Thu, 28 Feb 2019 05:24:09 +0000 (15:54 +1030)]
ccan/io: have io_plan mappable back to io_conn.

io_conn contains two plans: by knowing what plan this is, it can be cast
back into the io_conn.  That helps for the next patch.

We also remove the strange logic where io_duplex() would return a
magic, invalid io_plan pointer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agocompiler: allow NON_NULL_ARGS with only a single argument.
Rusty Russell [Wed, 13 Feb 2019 01:47:56 +0000 (12:17 +1030)]
compiler: allow NON_NULL_ARGS with only a single argument.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agopipecmd: don't close stderr/stdout/stdin in parent when &pipecmd_preserve
Rusty Russell [Thu, 7 Feb 2019 03:56:58 +0000 (14:26 +1030)]
pipecmd: don't close stderr/stdout/stdin in parent when &pipecmd_preserve

This was reported and fixed by @m-schmoock for stderr, but I decided to
rework to make it clearer and cover the other cases:

https://github.com/ElementsProject/lightning/pull/2321/commits/67dad01549f876e5e620eeffe2f1e78c35bd0f03

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoconfigurator: fix crash when more than one extra test provided
Rusty Russell [Tue, 5 Feb 2019 04:04:47 +0000 (14:34 +1030)]
configurator: fix crash when more than one extra test provided

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotal: handle take() pointers more carefully.
Rusty Russell [Mon, 17 Dec 2018 00:16:32 +0000 (10:46 +1030)]
tal: handle take() pointers more carefully.

take() applies to the literal pointer value, so tal_free() means it may
randomly apply to a future allocation.  Similarly, tal_resize() should
preserve it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agopipecmd: reverse order of args to match documentation; add pipecmd_preserve.
Rusty Russell [Mon, 12 Nov 2018 05:41:18 +0000 (16:11 +1030)]
pipecmd: reverse order of args to match documentation; add pipecmd_preserve.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agosmall fix for ccan/take/take.c
Dmitry Petukhov [Sun, 4 Nov 2018 14:58:04 +0000 (19:58 +0500)]
small fix for ccan/take/take.c

Hello.

I've decided to take a look at c-lighting code, because we might
consider using it in the future.

I found a small problem in ccan/take/take.c that only concerns code
that runs with CCAN_TAKE_DEBUG set (DEVELOPER=1 for c-lighting).

It is a small issue, but I decided to notify you as the author of the
code, anyway.

the issue is:
  in take_() function, potential failure of realloc for labelarr is not
  handled.

I attached a diff with a fix.

I thought that making a pull request for c-lighting would not be right,
as ccan is a separate project, but I did not find a way to report this
at http://git.ozlabs.org/, where ccan repo resides.

Therefore I wrote to you directly.

[ Minor whitespace changes --RR ]

3 years agoopt: add new parse_early_args_incomplete.
Rusty Russell [Fri, 2 Nov 2018 01:27:57 +0000 (11:57 +1030)]
opt: add new parse_early_args_incomplete.

If we have plugins, and those can register args, we have a problem finding
the plugin dir!  So, do a best-effort incomplete parse.

Note that this can screw up in theory if we have "--unknown --foo" since we
don't know if unknown takes an argument (in which case, ignore --foo) or
not.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agocompiler: add more attributes.
Rusty Russell [Fri, 2 Nov 2018 01:01:39 +0000 (11:31 +1030)]
compiler: add more attributes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoio: don't leave errno as a random value when we hit EOF.
Rusty Russell [Thu, 18 Oct 2018 00:29:00 +0000 (10:59 +1030)]
io: don't leave errno as a random value when we hit EOF.

It's used inside io_finish; setting to 0 allows that to know we hit
EOF on a read.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agohtable: add and integrate htable_check function.
Rusty Russell [Thu, 11 Oct 2018 23:54:10 +0000 (10:24 +1030)]
htable: add and integrate htable_check function.

Trying to debug what looked like an htable fail in my own code (but seems
it's not), I added htable_check a-la list_check et al.

This means we now depend on ccan/str (for stringify), which breaks the
_info example which also defined streq, and played nasty with const pointers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agorbuf, tools: clean up rbuf usage.
Rusty Russell [Fri, 12 Oct 2018 00:25:31 +0000 (10:55 +1030)]
rbuf, tools: clean up rbuf usage.

We should use rbuf primitives not reach inside to membuf.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agomembuf: add membuf_added and membuf_unadd APIs.
Rusty Russell [Fri, 12 Oct 2018 00:23:35 +0000 (10:53 +1030)]
membuf: add membuf_added and membuf_unadd APIs.

Clean up some whitespace while we're there too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agocrypto/shachain/tools: update to new rbuf API.
Rusty Russell [Fri, 21 Sep 2018 07:31:09 +0000 (17:01 +0930)]
crypto/shachain/tools: update to new rbuf API.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agostructeq: fix case where we mark padding as unknown.
Rusty Russell [Wed, 26 Sep 2018 23:39:50 +0000 (09:09 +0930)]
structeq: fix case where we mark padding as unknown.

And change semantics: a negative number means "up to this much padding".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools: fix compile after rbuf rewrite.
Rusty Russell [Wed, 26 Sep 2018 23:57:40 +0000 (09:27 +0930)]
tools: fix compile after rbuf rewrite.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agorbuf: adapt to work on ccan/membuf.
Rusty Russell [Mon, 17 Sep 2018 01:24:45 +0000 (10:54 +0930)]
rbuf: adapt to work on ccan/membuf.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agomembuf: new module for linear memory buffers.
Rusty Russell [Mon, 17 Sep 2018 01:17:42 +0000 (10:47 +0930)]
membuf: new module for linear memory buffers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotal: allow notifiers on NULL.
Rusty Russell [Thu, 23 Aug 2018 02:41:13 +0000 (12:11 +0930)]
tal: allow notifiers on NULL.

A destructor on NULL doesn't make sense, but notifiers (eg. new children)
do.  We fix up a mistake in run-notifier (comparing ctx with itself) and
loose typing in tal.c's tal_add_notifier_ too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoFix typos detected by github.com/client9/misspell
Kazuhiro Sera [Fri, 10 Aug 2018 06:54:10 +0000 (15:54 +0900)]
Fix typos detected by github.com/client9/misspell

3 years agotal/str: always create strings which have tal_count() == strlen() + 1.
Rusty Russell [Fri, 27 Jul 2018 06:25:50 +0000 (15:55 +0930)]
tal/str: always create strings which have tal_count() == strlen() + 1.

This is least-surprise, but also means callers can sometimes do faster
string handling by avoiding strstr().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoccan/tal: always include a length field.
Rusty Russell [Fri, 27 Jul 2018 04:29:18 +0000 (13:59 +0930)]
ccan/tal: always include a length field.

The current semantics of tal_count() / tal_bytelen() are to return 0
for anything not allocated using tal_arr*.  This is because we tried
to save a native-length word in the header, but produces an awkward
API.

(To make it worse, defining CCAN_TAL_DEBUG turns length to always on,
and we enable that for c-lightning developer mode, which hides bugs!).

However, for c-lightning, just over half of allocations want a length:
these use 3 words each, so we're actually worse off overall.

The answer is to always have a length field in the header.  This also
simplfies the tal code.

samba-allocs stats before:
Tal time: 1237102-1305755(1.251e+06+/-2.1e+04)ns
Tal_free time: 1346871-1514514(1.37844e+06+/-5.2e+04)ns

After:
Tal time: 1115180-1180633(1.1351e+06+/-2.1e+04)ns
Tal_free time: 1334381-1465933(1.39148e+06+/-4.7e+04)ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotal/str and tal/stack: use _label interfaces.
Rusty Russell [Fri, 27 Jul 2018 04:23:19 +0000 (13:53 +0930)]
tal/str and tal/stack: use _label interfaces.

In particular, tal/str now passes through the label from the caller,
so (in case of CCAN_TAL_DEBUG) you can actually see the file and line
where the caller was, not just inside ccan/str.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotal: add _label interfaces.
Rusty Russell [Fri, 27 Jul 2018 04:22:20 +0000 (13:52 +0930)]
tal: add _label interfaces.

There are a number of other utilities which use the tal_alloc_ and
tal_dup_arr_ internal interfaces directly, because they want to set
the label themselves.  We're about to break them all by changing those
internal interfaces, so give them a mid-level interface to use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotal: rename tal_len to tal_bytelen.
Rusty Russell [Fri, 27 Jul 2018 04:20:42 +0000 (13:50 +0930)]
tal: rename tal_len to tal_bytelen.

I had a bug caused by using tal_len instead of tal_count: let's make
it explicit. @jb55 commented "ha. I always forget which one does which... Ack"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoccan/structeq: make it safe when there's padding.
Rusty Russell [Wed, 4 Jul 2018 04:07:28 +0000 (13:37 +0930)]
ccan/structeq: make it safe when there's padding.

ccan/cppmagic FTW!

The only issue is that we can't tell if there's padding or they've missed
a member, so we add a padding bytes count, so they'll get an error if it
(for example) the structure adds a new member later.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agoccan/utf8: new module.
Rusty Russell [Mon, 18 Jun 2018 10:42:46 +0000 (20:12 +0930)]
ccan/utf8: new module.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: add manual page.
Rusty Russell [Tue, 12 Jun 2018 02:37:51 +0000 (12:07 +0930)]
tools/configurator: add manual page.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: --extra-tests to read more test descriptions from stdin.
Rusty Russell [Tue, 12 Jun 2018 02:36:51 +0000 (12:06 +0930)]
tools/configurator: --extra-tests to read more test descriptions from stdin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: don't rely on size of tests_array.
Rusty Russell [Tue, 12 Jun 2018 02:35:51 +0000 (12:05 +0930)]
tools/configurator: don't rely on size of tests_array.

Groundwork for adding tests dynamically.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: better descriptions for --autotools-style.
Rusty Russell [Tue, 12 Jun 2018 02:34:51 +0000 (12:04 +0930)]
tools/configurator: better descriptions for --autotools-style.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: Better argument handling, particularly unknown arguments.
Rusty Russell [Tue, 12 Jun 2018 02:33:51 +0000 (12:03 +0930)]
tools/configurator: Better argument handling, particularly unknown arguments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: support --header-file if we don't want to write to stdout.
Rusty Russell [Tue, 12 Jun 2018 02:32:51 +0000 (12:02 +0930)]
tools/configurator: support --header-file if we don't want to write to stdout.

Works well with --autotools-style.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: support --autotools-style.
Rusty Russell [Tue, 12 Jun 2018 02:31:51 +0000 (12:01 +0930)]
tools/configurator: support --autotools-style.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: support --var-file for outputting VAR=VAL format.
Rusty Russell [Tue, 12 Jun 2018 02:30:51 +0000 (12:00 +0930)]
tools/configurator: support --var-file for outputting VAR=VAL format.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotools/configurator: convert flags to text in 'struct test'
Rusty Russell [Tue, 12 Jun 2018 02:29:51 +0000 (11:59 +0930)]
tools/configurator: convert flags to text in 'struct test'

This is the only non-text field.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3 years agotimer: remove unnecessary dependency on likely
Eric Wong [Fri, 8 Jun 2018 02:45:42 +0000 (02:45 +0000)]
timer: remove unnecessary dependency on likely

I don't see likely/unlikely being used by ccan/timer or any of its
dependencies right now.

Running `tools/ccanlint/ccanlint ccan/timer' reveals no regressions

Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>