rune: add helpers for users doing DIY parsing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rune: avoid shadowing var in test/run.c

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rune: fix encoding of alternates which need escapes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

net: fix gcc warning about mismatch.

Header can't use pollfds[2] because type is unknown.  And gcc complains:

ccan/net/net.c:78:70: warning: argument 2 of type ‘struct pollfd[2]’ with mismatched bound [-Warray-parameter=]

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

alloc: fix shadowed variable

Thanks -Wshadow=local!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: fix shadowed variable name.

Thanks -Wshadow=local!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Makefile: add -Wshadow=local, fix configurator!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/base64: fix GCC warning.

```
ccan/ccan/base64/base64.c:146:71: error: argument 2 of type ‘char[3]’ with mismatched bound [-Werror=array-parameter=]
  146 | ssize_t base64_decode_tail_using_maps(const base64_maps_t *maps, char dest[3],
      |                                                                  ~~~~~^~~~~~~
In file included from ccan/ccan/base64/base64.c:2:
ccan/ccan/base64/base64.h:119:72: note: previously declared as ‘char *’
  119 | ssize_t base64_decode_tail_using_maps(const base64_maps_t *maps, char *dest,
      |                                                                  ~~~~~~^~~~
cc1: all warnings being treated as errors
make: *** [Makefile:817: ccan-base64.o] Error 1
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/rune: compile without warnings on -O3.

```
ccan/ccan/rune/rune.c: In function ‘rune_alt_single_int’:
ccan/ccan/rune/rune.c:257:5: error: ‘runeval_int’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  257 |  if (cond)
      |     ^
ccan/ccan/rune/rune.c:305:6: note: ‘runeval_int’ was declared here
  305 |  s64 runeval_int;
      |      ^~~~~~~~~~~
cc1: all warnings being treated as errors
make: *** [Makefile:919: ccan-rune-rune.o] Error 1
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/rune: simplify check helper interfaces, allow explicit string lengths.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/rune: add explicit-length variants.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/rune: fix take() logic which caused memory leaks.

You have to tal_steal() if you want to keep it!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/io: add debugging helper for seeing if module is in charge of an fd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

strmap: add strmap_getn for non-terminated string search.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: fix typo in example.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rune: new module to implement runes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: opportunistically avoid delete marker.

Fairly cheap test, we can see a drop in initial re-inserting, probably
because it has just been cleaned at this point with the old code, but
generally we do better with churn.

I also tried a variant which only checked if next bucket was on same
cacheline, but no significant difference.

5 runs of tools/speed 10000000:

```
-Initial re-inserting: 88-99(94.6+/-5) ns
+Initial re-inserting: 104-116(107.2+/-4.4) ns
-Adding (a different) half: 94-101(98.6+/-3) ns
+Adding (a different) half: 38-44(39.8+/-2.1) ns
-Lookup after half-change (miss): 90-105(96.4+/-5.7) ns
+Lookup after half-change (miss): 112-118(113.4+/-2.3) ns
-Churning third time: 301-323(314.4+/-9.8) ns
+Churning third time: 233-259(239+/-10) ns
-Details: worst run 146 (24 deleted)
+Details: worst run 115 (2 deleted)
-Lookup after churn & spread (match): 78-87(83.6+/-4.2) ns
+Lookup after churn & spread (match): 71-81(73.6+/-3.8) ns
-Lookup after churn & spread (miss): 118-137(124.2+/-6.9) ns
+Lookup after churn & spread (miss): 93-104(95.4+/-4.3) ns
-Adding (a different) half after churn & spread: 81-89(85.4+/-3.6) ns
+Adding (a different) half after churn & spread: 38-43(39.6+/-1.7) ns
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: increase density from 75% to 87.5%

Across all benchmarks (on various sizes) this actually drops
performance by 3%.  However, density increasing by 15% is probably a
worthwhile tradeoff.

```
-Initial insert: 227-237(230+/-3.7) ns
+Initial insert: 266-281(273.4+/-6.5) ns
-Initial lookup (match): 65-66(65.4+/-0.49) ns
+Initial lookup (match): 67-78(70.6+/-3.8) ns
-Initial lookup (miss): 79-80(79.4+/-0.49) ns
+Initial lookup (miss): 82-93(84.8+/-4.2) ns
-Initial lookup (random): 111-115(112.6+/-1.6) ns
+Initial lookup (random): 118-138(123.4+/-7.4) ns
-Initial delete all: 66-68(66.6+/-0.8) ns
+Initial delete all: 79-90(82+/-4.1) ns
-Initial re-inserting: 87-90(88+/-1.3) ns
+Initial re-inserting: 104-116(107.2+/-4.4) ns
-Deleting first half: 32-33(32.4+/-0.49) ns
+Deleting first half: 37-43(39+/-2.1) ns
-Lookup after half-change (match): 67-69(68+/-0.89) ns
+Lookup after half-change (match): 72-81(74.2+/-3.4) ns
-Churning second time: 272-276(274.2+/-1.5) ns
+Churning second time: 304-332(310.2+/-11) ns
-Churning third time: 290-296(293+/-2) ns
+Churning third time: 233-259(239+/-10) ns
-Churning fourth time: 289-295(292+/-2.2) ns
+Churning fourth time: 309-337(315.6+/-11) ns
-Churning fifth time: 289-294(291.8+/-1.9) ns
+Churning fifth time: 307-337(315+/-11) ns
-Details: worst run 146 (24 deleted)
+Details: worst run 115 (2 deleted)
-Lookup after churn & spread (miss): 110-111(110.6+/-0.49) ns
+Lookup after churn & spread (miss): 93-104(95.4+/-4.3) ns
-Deleting half after churn & spread: 36-37(36.4+/-0.49) ns
+Deleting half after churn & spread: 39-45(40.4+/-2.3) ns
-Adding (a different) half after churn & spread: 81-82(81.6+/-0.49) ns
+Adding (a different) half after churn & spread: 38-43(39.6+/-1.7) ns
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: tools/density

Everyone loves stats: this looks at different levels of table density
and lookup speed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: fix vanishing entries properly.

A hash table entry consists of the pointer, but we mask out bits which
are common to all pointers, and replace them with hash bits.

The entry values 0 and 1 are special, meaning "empty" and "deleted"
respectively.

However, if a hash has mainly zero bits, and an pointer's non-shared
bits are all zero, we can create an entry which looks empty or
deleted.

The solution in this case is to share another bit (there must be some
bit which is not zero, since we don't allow 0 and 1 as pointers in the
hash table).

It's unusual, so I put this in a cold function: it shares code with
the existing "oh, we need to reduce the common bits, since this new
pointer doesn't look like the previous ones" case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Partial revert "htable: handle v. unlikely case where entries look deleted/empty."

This reverts commit 7218ee2476d8a7e4f9413781e2bedcd2daaa8190.

Except for:
 - Leaves the test behind though (which fail!
 - Leaves the reversion of 0b93bd102aad6b61f1e569fb12aabc6352a1d7cd, since
   we're about to fix it, and properly!

Revert "htable: optimize a little more."

This reverts commit 0f144bfe0ca3cc06564e19a7f3ea40177d1a3c2c.

htable: optimize a little more.

tools/speed (10 runs) vs before fix:

-Lookup after half-change (match): 53-61(54.8+/-2.3) ns
+Lookup after half-change (match): 63-97(72.4+/-9) ns
-Churning fifth time: 171-181(175.4+/-2.9) ns
+Churning fifth time: 179-198(183.9+/-5.2) ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: handle v. unlikely case where entries look deleted/empty.

If the hash function doesn't set any bits we use, and the common
bits are all zero the slot will look empty (or, just the lower bit is
set: the slow looks deleted).

However, each bucket is distinct since there are no duplicates, so
at worse there can be two "looks invalid but actually is valid"
buckets.  Keep them at the end.

Lookup suffers in raw tools/speed though:

-Lookup after half-change (match): 53-61(54.8+/-2.3) ns
+Lookup after half-change (match): 61-113(83.3+/-17) ns
-Lookup after churn & spread (match): 54-90(59.9+/-11) ns
+Lookup after churn & spread (match): 70-108(85.2+/-14) ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: add new io_sock_shutdown helper.

Turns out that there's no good way to flush a socket:

https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add tal_dup_or_null and tal_dup_talarr helpers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

crc32c, crypto/xtea: check in LICENSE symlinks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

graphql: latest changes from @rl-d

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan: make tal_dump() format more regular.

Putting a \n in there was a pain: I had to do:

awk '/CHILDREN/ { SAVED=$0 } !/CHILDREN/ { print SAVED$0; SAVED="" }' < lightningd-tal_dump > lightningd-tal_dump-children-on-same-line

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

pipecmd: Use closefrom instead of iterating directly.

Signed-off-by: ZmnSCPxj jxPCSnmZ <ZmnSCPxj@protonmail.com>

closefrom: fix minor issues found by ccanlint

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

closefrom: Close all file descriptors above a certain value.

For more information: https://github.com/ElementsProject/lightning/issues/4868

Signed-off-by: ZmnSCPxj jxPCSnmZ <ZmnSCPxj@protonmail.com>

web: create "versioned" tarballs.

Fixes: https://github.com/rustyrussell/ccan/issues/90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

base64: fix assignment to wrong type, use ssize_t

Functions that returned int have been modified to return ssize_t
instead.

Found by gcc:

base64.c:196:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  if (more == -1) {
           ^

Signed-off-by: Martin Milata <martin@martinmilata.cz>

nit: daemonize.c: fix Stevens' in comment

Setting the same name as at the implementation

configurator: always true branch

Fix "for loop initial declarations are only allowed in C99 mode" compile errors.

docs: fix simple typo, unchaged -> unchanged

There is a small typo in ccan/objset/objset.h.

Should read `unchanged` rather than `unchaged`.

fix warnings

GraphQL lexer and parser (without optional type system)

tal: tal_dump() now prints to stderr.

This is more widely available than stdout, which can interfere with
other things.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

compiler: RETURNS_NONNULL macro

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: HAVE_ATTRIBUTE_RETURNS_NONNULL

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

timer: handle time going backwards.

As seen in https://github.com/ElementsProject/lightning/issues/4401

OpenBSD 6.8 inside VirtualBox.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

utf8: don't allow NUL in decoded strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

json_out: make json_out_finished finish buffer.

For JSON RPC, we can actually write multiple JSON objects back
to back, and we don't want a comma between them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

*: Fix build with recent gcc/binutils

With current Debian testing I get:

tools/ccanlint/ccanlint -s --deps-fail-ignore   ccan/compiler/ && touch ccan/compiler/.ok
compiler: Module tests compile (tests_compile): FAIL
/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/compiler/test/run-is_compile_constant.c:Compile failed:
/usr/bin/ld: /tmp/ccanlint-129350.1804289383/tap.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/tap/tap.h:249: multiple definition of `tap_fail_callback'; /tmp/ccrAwqbl.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/tap/tap.h:249: first defined here
collect2: error: ld returned 1 exit status
...

cc -g3 -ggdb -Wall -Wstrict-prototypes -Wold-style-definition -Wundef -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wwrite-strings -DCCAN_STR_DEBUG=1 -I.  -MMD -MP -MFtools/ccanlint/ccanlint.d -MTtools/ccanlint/ccanlint tools/ccanlint/ccanlint.c tools/ccanlint/async.o tools/ccanlint/file_analysis.o tools/ccanlint/licenses.o tools/ccanlint/tests/avoids_cpp_reserved.o tools/ccanlint/tests/depends_accurate.o tools/ccanlint/tests/depends_build.o tools/ccanlint/tests/depends_build_without_features.o tools/ccanlint/tests/depends_exist.o tools/ccanlint/tests/examples_compile.o tools/ccanlint/tests/examples_exist.o tools/ccanlint/tests/examples_relevant.o tools/ccanlint/tests/examples_run.o tools/ccanlint/tests/hash_if.o tools/ccanlint/tests/headers_idempotent.o tools/ccanlint/tests/info_compiles.o tools/ccanlint/tests/info_documentation_exists.o tools/ccanlint/tests/info_exists.o tools/ccanlint/tests/info_ported.o tools/ccanlint/tests/info_summary_single_line.o tools/ccanlin
 t/tests/license_comment.o tools/ccanlint/tests/license_depends_compat.o tools/ccanlint/tests/license_exists.o tools/ccanlint/tests/license_file_compat.o tools/ccanlint/tests/main_header_compiles.o tools/ccanlint/tests/main_header_exists.o tools/ccanlint/tests/module_builds.o tools/ccanlint/tests/module_links.o tools/ccanlint/tests/no_trailing_whitespace.o tools/ccanlint/tests/objects_build.o tools/ccanlint/tests/objects_build_with_stringchecks.o tools/ccanlint/tests/objects_build_without_features.o tools/ccanlint/tests/reduce_features.o tools/ccanlint/tests/tests_compile.o tools/ccanlint/tests/tests_compile_coverage.o tools/ccanlint/tests/tests_coverage.o tools/ccanlint/tests/tests_exist.o tools/ccanlint/tests/tests_helpers_compile.o tools/ccanlint/tests/tests_pass.o tools/ccanlint/tests/tests_pass_valgrind.o tools/ccanlint/tests/tests_pass_without_features.o ccan/autodata/autodata.o ccan/dgraph/dgraph.o ccan/ilog/ilog.o ccan/lbalance/lbalance.o ccan/ptr_valid/ptr_valid.o ccan/strma
 p/strmap.o tools/ccan_dir.o tools/compi
le.o tools/depends.o tools/doc_extract-core.o tools/gcov.o tools/manifest.o tools/read_config_header.o tools/tools.o ccan/asort/asort.o ccan/err/err.o ccan/foreach/foreach.o ccan/hash/hash.o ccan/htable/htable.o ccan/list/list.o ccan/membuf/membuf.o ccan/noerr/noerr.o ccan/opt/helpers.o ccan/opt/opt.o ccan/opt/parse.o ccan/opt/usage.o ccan/rbuf/rbuf.o ccan/read_write_all/read_write_all.o ccan/str/debug.o ccan/str/str.o ccan/take/take.o ccan/tal/tal.o ccan/tal/grab_file/grab_file.o ccan/tal/link/link.o ccan/tal/path/path.o ccan/tal/str/str.o ccan/time/time.o -lm -o tools/ccanlint/ccanlint
/usr/bin/ld: tools/ccanlint/async.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/file_analysis.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/licenses.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/avoids_cpp_reserved.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined
here
/usr/bin/ld: tools/ccanlint/tests/depends_accurate.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/depends_build.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/depends_build_without_features.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
...

The errors are valid becuase symbols are declared without extern in .h
files and those .h files are included from multiple translation units.

-> Fix it. I'm not sure why it used to compile ok before.

Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Message-Id: <20200910191610.131510-1-kirr@nexedi.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

tools/configurator: fix compile error with -O2

In file included from /usr/include/string.h:495,
                 from configuratortest.c:2:
In function ‘strncpy’,
    inlined from ‘main’ at configuratortest.c:6:2:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: warning: ‘__builtin_strncpy’ specified bound 8 equals destination size [-Wstringop-truncation]
  106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: document tal_steal(ctx, NULL) as OK.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: add htable_type _delval.

To match the raw delval

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: fix type of htable_type _pick return.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: htable_pick helper to select a random entry.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: don't defer-after-free if a destructor deletes itself.

==10868==    at 0x109A96: notify (tal.c:220)
==10868==    by 0x109F7E: del_tree (tal.c:397)
==10868==    by 0x10A31A: tal_free (tal.c:481)
==10868==    by 0x10BE73: main (run-destructor.c:75)
==10868==  Address 0x4a60bd8 is 8 bytes inside a block of size 32 free'd
==10868==    at 0x483BA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==10868==    by 0x109D4F: del_notifier_property (tal.c:340)
==10868==    by 0x10A610: tal_del_notifier_ (tal.c:564)
==10868==    by 0x10A687: tal_del_destructor_ (tal.c:576)
==10868==    by 0x10B653: remove_own_destructor (run-destructor.c:35)
==10868==    by 0x109A67: notify (tal.c:235)
==10868==    by 0x109F7E: del_tree (tal.c:397)
==10868==    by 0x10A31A: tal_free (tal.c:481)
==10868==    by 0x10BE73: main (run-destructor.c:75)
==10868==  Block was alloc'd at
==10868==    at 0x483A7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==10868==    by 0x109AD5: allocate (tal.c:245)
==10868==    by 0x109C3E: add_notifier_property (tal.c:300)
==10868==    by 0x10A467: tal_add_destructor_ (tal.c:516)
==10868==    by 0x10BDFE: main (run-destructor.c:74)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

opt: fix opt_unregister.

Instead of memmoving N structs, we were memmoving N bytes.

But why did the test pass then?  It was doing memmove(..., 1)
instead of memmove(..., sizeof(struct opt_table)!

Because the two structures were really similar; the main difference
was the first entry, which points to the name.  But they were allocated
consecutively, and Intel being little-endian, the only difference was
the first byte!  Thus memmove(1) was enough to make it "work".

Change two options in the test to be sufficiently different, and
the bug shows up.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

xtea: new module.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

opt: add opt_unregister.

Sometimes we all make mistakes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tools/configurator: simplify wrapper code a little.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: Enable running configurator in wrapper

Since the probing binaries compiled by the configurator needs to run on
the host machine we provide a variable CONFIGURATOR_WRAPPER that can be
set to anything that you want to wrap the calls with.

One example is `qemu-aarch64-static`.

bitmap: Allow bitmap type to be forward declared

Currently bitmap type is defined via untagged struct which makes it
impossible to forward declare it. Forward-declaring is useful since all
bitmap functions only use bitmap* and in public user-visible
headers/datastructures it is enough to indicate that a data field with
bitmap pointer is there, whereas bitmap.h can be included only in
implementation.

Beside that some headers are included by both C and C++ parts of a
project, and when ccan/bitmap.h is processed by C++ compiler it gives:

    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_alloc(long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:201:15: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      return malloc(bitmap_sizeof(nbits));
             ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_realloc0(bitmap*, long unsigned int, long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:227:18: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      bitmap = realloc(bitmap, bitmap_sizeof(nbits));
               ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ./3rdparty/ccan/ccan/bitmap/bitmap.h: In function ‘bitmap* bitmap_realloc1(bitmap*, long unsigned int, long unsigned int)’:
    ./3rdparty/ccan/ccan/bitmap/bitmap.h:238:18: error: invalid conversion from ‘void*’ to ‘bitmap*’ [-fpermissive]
      bitmap = realloc(bitmap, bitmap_sizeof(nbits));

-> Give to users ability not to force-include ccan/bitmap.h by
forward-declaring bitmaps like this:

    typedef struct bitmap bitmap;
    ...
    struct MyStruct {
        bitmap *my_bitmap;
    };

Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Message-Id: <20191021150903.25159-1-kirr@nexedi.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

intmap: implement intmap_before()

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

failtest, rbuf: fix up incorrect lseek arg order.

SEEK_SET == 0, so it's not hurting anyone now, but when I cut & paste
it elsewhere...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

pipecmd: close fds in child.

This is usually what you want; I didn't even add a flag to stop it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: add htable_count().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Suppress false positive warning "-Wmaybe-uninitialized" with GCC -O3

configurator: fix openmp test on some versions of clang.

Discovered on bionic's Travis install, it failed at *runtime* when
it couldn't find the shared library.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

bitmap: Fix some bugs on 32-bit platforms

The bitmap_word type is an unsigned long.  However in some places we assign
it using -1ULL, a 64-bit value on many platforms.  We sometimes get away
with this because it masks correctly, but in other cases it breaks things.

To clean this up define a new BITMAP_WORD_1 constant, indicating a
bitmap_word with all bits set, and use that instead of explicit UL or ULL
qualifiers.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

ccan/crc: remove broken "crc32c" routines which gave wrong results.

I copied them from the kernel, but obviously didn't find good test
vectors.  In particular, the CRC of all zeroes is not 0!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/crc32c: new module for accelerated CRC32 (on x86-64).

Note: the previous code in ccan/crc is wrong, so I started fresh with
actual test vectors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/compiler: add cpu_supports() macro.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: detect __builtin_cpu_supports.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

json_out: pass through OOM failures.

And fix escaping to work with addstr(), and assert in debug mode
if they don't specify quotes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

json_out: new module for authoring JSON strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

json_escape: add fast-path for when we don't need to escape.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

json_escape: new module to escape JSON strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: avoid branch in calculating perfect bit.

Final results of tools/speed/10000000 (10 runs) shows a slight
slowdown in some tests, but it makes an empty htable smaller.

-Initial delete all: 96-98(96.4+/-0.66) ns
+Initial delete all: 97-99(98.2+/-0.75) ns
-Initial re-inserting: 117-124(121.4+/-1.9) ns
+Initial re-inserting: 124-131(126.4+/-2.4) ns
-Adding (a different) half: 49-50(49.3+/-0.46) ns
+Adding (a different) half: 50-52(51.2+/-0.75) ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: reduce size of htable by storing perfect bitnum, not mask.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: reduce size of htable by calculating max every time.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: fix another leak path for always array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: don't leak memory on clean shutdown.

Free the internal "always" array like we free "fds".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: typo fixes.

Suggested-by: Lisa Neigut <@niftynei>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: add allocator hooks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: commit extra tests I had lying around.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

timer: clean up hook allocator API

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

timer: add hook for allocation functions.

malloc and free aren't for everyone.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/io: add io_conn_exclusive and io_conn_out_exclusive.

There are cases where we want to suppress all activity except for a
single fd; we already have ugly io_flush_sync, but this is more
useful and more general.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan: fix erroneous fd negation in poll.c.

We could just make them -1, since we don't rely on the values.
But simple negation doesn't make 0 invalid.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/io: keep always pointers to plans, not a linked list.

Duplex io_conns can be in the always list twice, and it was a source
of bugs, but I didn't want to add a second list_node.  Since there are
not many always at once, it's better (and more space-efficient) to use
a simple pointer array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/io: have io_plan mappable back to io_conn.

io_conn contains two plans: by knowing what plan this is, it can be cast
back into the io_conn.  That helps for the next patch.

We also remove the strange logic where io_duplex() would return a
magic, invalid io_plan pointer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

compiler: allow NON_NULL_ARGS with only a single argument.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

pipecmd: don't close stderr/stdout/stdin in parent when &pipecmd_preserve

This was reported and fixed by @m-schmoock for stderr, but I decided to
rework to make it clearer and cover the other cases:

https://github.com/ElementsProject/lightning/pull/2321/commits/67dad01549f876e5e620eeffe2f1e78c35bd0f03

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: fix crash when more than one extra test provided

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: handle take() pointers more carefully.

take() applies to the literal pointer value, so tal_free() means it may
randomly apply to a future allocation.  Similarly, tal_resize() should
preserve it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

pipecmd: reverse order of args to match documentation; add pipecmd_preserve.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

small fix for ccan/take/take.c

Hello.

I've decided to take a look at c-lighting code, because we might
consider using it in the future.

I found a small problem in ccan/take/take.c that only concerns code
that runs with CCAN_TAKE_DEBUG set (DEVELOPER=1 for c-lighting).

It is a small issue, but I decided to notify you as the author of the
code, anyway.

the issue is:
  in take_() function, potential failure of realloc for labelarr is not
  handled.

I attached a diff with a fix.

I thought that making a pull request for c-lighting would not be right,
as ccan is a separate project, but I did not find a way to report this
at http://git.ozlabs.org/, where ccan repo resides.

Therefore I wrote to you directly.

[ Minor whitespace changes --RR ]

opt: add new parse_early_args_incomplete.

If we have plugins, and those can register args, we have a problem finding
the plugin dir!  So, do a best-effort incomplete parse.

Note that this can screw up in theory if we have "--unknown --foo" since we
don't know if unknown takes an argument (in which case, ignore --foo) or
not.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

compiler: add more attributes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: don't leave errno as a random value when we hit EOF.

It's used inside io_finish; setting to 0 allows that to know we hit
EOF on a read.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: add and integrate htable_check function.

Trying to debug what looked like an htable fail in my own code (but seems
it's not), I added htable_check a-la list_check et al.

This means we now depend on ccan/str (for stringify), which breaks the
_info example which also defined streq, and played nasty with const pointers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rbuf, tools: clean up rbuf usage.

We should use rbuf primitives not reach inside to membuf.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>