global: avoid using sprintf.

snprintf has different footguns, but MacOS started complaining, a-la:

Core Lightning fails to compile on macOS with the latest Command Line Tools (macOS 15 SDK) due to sprintf being marked as deprecated and -Werror treating the deprecation as a compilation error.

Environment: Sequoia 15.6.1 (24G90)
SDK Version: MacOSX15.sdk

```
ccan/ccan/closefrom/closefrom.c:83:2: error: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Werror,-Wdeprecated-declarations]
   83 |         sprintf(dnam, "/proc/%ld/fd", (long) getpid());
      |         ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:278:1: note: 'sprintf' has been explicitly marked deprecated here
  278 | deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
      | ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro 'deprecated_msg'
  218 |         #define deprecated_msg(_msg) attribute((deprecated(_msg)))
      |                                                       ^
ccan/ccan/closefrom/closefrom.c:162:2: error: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Werror,-Wdeprecated-declarations]
  162 |         sprintf(dnam, "/proc/%ld/fd", (long) getpid());
      |         ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:278:1: note: 'sprintf' has been explicitly marked deprecated here
  278 | deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
      | ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro 'deprecated_msg'
  218 |         #define deprecated_msg(_msg) attribute((deprecated(_msg)))
      |                                                       ^
2 errors generated.
make: *** [ccan-closefrom.o] Error 1
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: fix bad code in HAVE_UNALIGNED_ACCESS detection.

Reported on armhf, but applies to any 32-bit platform (detected at -O2 or above)

```
$ arm-linux-gnueabihf-gcc-13 -O2 -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard -c /tmp/foo.c -Wall
/tmp/foo.c: In function ‘main’:
/tmp/foo.c:4:23: error: expected expression before ‘int’
    4 |      char pad[sizeof(*int) * 1];
      |                       ^~~
/tmp/foo.c:4:11: warning: unused variable ‘pad’ [-Wunused-variable]
    4 |      char pad[sizeof(*int) * 1];
      |           ^~~
```

Reported-by: https://github.com/Raimo33
Fixes: https://github.com/ElementsProject/lightning/issues/8501
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/io: fix misleading comment in previous commit.

We don't need to be duplex to have this problem: the write side
does it too in the example I found.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/io: fix occasional immediate close after io_shutdown()

If we're doing a duplex connection, we could be still trying to write:
that will fail (since we called shutdown on the write side), and we'll
respond by closing the socket.

The reason we're doing shutdown() is to ensure the final bytes are
written: doing a close() breaks this, and so the result is lost final
bytes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

json_escape: add json_escape_unescape_len()

Lacking a variant of json_escape_unescape() that takes a sized span,
the common case of unescaping a JSON string that is embedded inside a
larger buffer is forced to make a copy of the escaped string so as to
terminate it with a NUL character. This extra copy can be avoided if
we can pass an explicit length to the unescaping function.

Also, a drive-by fix: shrink the returned unescaped string to fit its
contents, which can save the caller a subsequent call to strlen().

Remove ccodearchive.net references

Our former website ccodearchive.net has been taken over by domain
squatters, who've put up a near-duplicate of the former site with sneeaky
advertising links inserted.  No idea if they've also subverted the actual
code, but it's possible.

There is, alas, not much we can do about that.  We can, at least, stop
linking to it in the repository though.  So, remove all references to the
now untrustworthy website.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

fdpass: fix complilation on FreeBSD.

```
cc ccan/ccan/fdpass/fdpass.c
ccan/ccan/fdpass/fdpass.c:16:12: error: use of undeclared identifier 'u_long'; did you mean 'long'?
                char buf[CMSG_SPACE(sizeof(fd))];
                         ^
/usr/include/sys/socket.h:577:25: note: expanded from macro 'CMSG_SPACE'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:16:12: error: use of undeclared identifier 'u_long'
/usr/include/sys/socket.h:577:25: note: expanded from macro 'CMSG_SPACE'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:16:12: error: use of undeclared identifier 'u_long'; did you mean 'long'?
/usr/include/sys/socket.h:577:58: note: expanded from macro 'CMSG_SPACE'
                                                                  ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:16:12: error: use of undeclared identifier 'u_long'
/usr/include/sys/socket.h:577:58: note: expanded from macro 'CMSG_SPACE'
                                                                  ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:26:19: error: use of undeclared identifier 'u_long'; did you mean 'long'?
        cmsg->cmsg_len = CMSG_LEN(sizeof(fd));
                         ^
/usr/include/sys/socket.h:578:23: note: expanded from macro 'CMSG_LEN'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:26:19: error: use of undeclared identifier 'u_long'
        cmsg->cmsg_len = CMSG_LEN(sizeof(fd));
                         ^
/usr/include/sys/socket.h:578:23: note: expanded from macro 'CMSG_LEN'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:26:19: error: use of undeclared identifier 'u_long'
/usr/include/sys/socket.h:578:23: note: expanded from macro 'CMSG_LEN'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:27:9: error: use of undeclared identifier 'u_long'; did you mean 'long'?
        memcpy(CMSG_DATA(cmsg), &fd, sizeof(fd));
               ^
/usr/include/sys/socket.h:554:6: note: expanded from macro 'CMSG_DATA'
                                 _ALIGN(sizeof(struct cmsghdr)))
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:27:9: error: use of undeclared identifier 'u_long'
/usr/include/sys/socket.h:554:6: note: expanded from macro 'CMSG_DATA'
                                 _ALIGN(sizeof(struct cmsghdr)))
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:53:12: error: use of undeclared identifier 'u_long'; did you mean 'long'?
                char buf[CMSG_SPACE(sizeof(fd))];
                         ^
/usr/include/sys/socket.h:577:25: note: expanded from macro 'CMSG_SPACE'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:53:12: error: use of undeclared identifier 'u_long'
/usr/include/sys/socket.h:577:25: note: expanded from macro 'CMSG_SPACE'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:53:12: error: use of undeclared identifier 'u_long'; did you mean 'long'?
/usr/include/sys/socket.h:577:58: note: expanded from macro 'CMSG_SPACE'
                                                                  ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:74:27: error: use of undeclared identifier 'u_long'; did you mean 'long'?
            || cmsg->cmsg_len != CMSG_LEN(sizeof(fd))
                                 ^
/usr/include/sys/socket.h:578:23: note: expanded from macro 'CMSG_LEN'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:74:27: error: use of undeclared identifier 'u_long'
/usr/include/sys/socket.h:578:23: note: expanded from macro 'CMSG_LEN'
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:81:14: error: use of undeclared identifier 'u_long'; did you mean 'long'?
        memcpy(&fd, CMSG_DATA(cmsg), sizeof(fd));
                    ^
/usr/include/sys/socket.h:554:6: note: expanded from macro 'CMSG_DATA'
                                 _ALIGN(sizeof(struct cmsghdr)))
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
ccan/ccan/fdpass/fdpass.c:81:14: error: use of undeclared identifier 'u_long'
/usr/include/sys/socket.h:554:6: note: expanded from macro 'CMSG_DATA'
                                 _ALIGN(sizeof(struct cmsghdr)))
                                 ^
/usr/include/machine/_align.h:45:22: note: expanded from macro '_ALIGN'
                           ^
16 errors generated.
gmake: *** [Makefile:994: ccan-fdpass.o] Error 1
```

Reported-by: https://github.com/21M4TW
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

time: add timemono_after, fix example compilation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

time: flesh out more timemono functions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: separate htable_type constructors to clarify duplicate keys.

Most hash tables do not want duplicates, so specify this
at declaration time.  This way you crash should such a thing
be inserted, not have some subtle behavior you didn't expect!

Also disable the functions depending on the type.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

opt: add more user-available bits.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tcon: fix warning from gcc when tcon_check_ptr arg is constant.

e.g. `&msg`:

```
src/nostrdb.c: In function ‘ndb_ingester_queue_event’:
ccan/ccan/tcon/tcon.h:150:24: error: the address of ‘msg’ will always evaluate as ‘true’ [-Werror=address]
  150 |         (sizeof((expr) ? (expr) : &(x)->_tcon[0].canary) ? (x) : (x))
      |                        ^
ccan/ccan/tcon/tcon.h:116:30: note: in definition of macro ‘tcon_unwrap’
  116 | #define tcon_unwrap(ptr) (&((ptr)->_base))
      |                              ^~~
src/threadpool.h:91:42: note: in expansion of macro ‘tcon_check_ptr’
   91 |         threadpool_dispatch_(tcon_unwrap(tcon_check_ptr((tp), tp_canary, (msg))), (msg))
      |                                          ^~~~~~~~~~~~~~
src/nostrdb.c:4114:16: note: in expansion of macro ‘threadpool_dispatch’
 4114 |         return threadpool_dispatch(&ingester->tp, &msg);
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io/fdpass: don't leak fd if conn closes before we write it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan: call io routines repeatedly until EAGAIN.

Benchmark time halved.

```
Finished: 68697129usec
```

perf shows 75% of time in libc_write.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: add benchmark for large poll.

This simulates a performance issue reported in `connectd` for
lightningd with 900 fds: it's consuming 70% of a CPU, and strace shows
it getting a single fd (or two) from poll, doing a write (gossip store
streaming), then going back into poll.

I tested using 1000 x netcat from a machine on the same LAN:

```
remote$ for i in `seq 1000`; do sleep 0.1; nc 10.88.9.17 8888 </dev/null > /dev/null & done
```

```
local$ ./run-stream-many 8888 980 20 100
rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrwwwwwwwwwwwwwwwwwwwwStarting!
Finished: 122852940usec
```

Perf record shows 93.5% of time spent in poll.  strace (separate run) shows us
servicing all 20 fds at once, because LAN is too fast.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: clean up compiler warning.

```
/home/rusty/devel/cvs/ccan/ccan/io/test/run-08-read-after-hangup.c: In function ‘main’:
/home/rusty/devel/cvs/ccan/ccan/io/io.h:62:58: warning: declaration of ‘conn’ shadows a previous local [-Wshadow=compatible-local]
   62 |                                          struct io_conn *conn),         \
      |                                          ~~~~~~~~~~~~~~~~^~~~
/home/rusty/devel/cvs/ccan/ccan/typesafe_cb/typesafe_cb.h:35:46: note: in definition of macro ‘typesafe_cb_cast’
   35 |                                              oktype),                   \
      |                                              ^~~~~~
/home/rusty/devel/cvs/ccan/ccan/io/io.h:60:22: note: in expansion of macro ‘typesafe_cb_preargs’
   60 |                      typesafe_cb_preargs(struct io_plan *, void *,      \
      |                      ^~~~~~~~~~~~~~~~~~~
/home/rusty/devel/cvs/ccan/ccan/io/test/run-08-read-after-hangup.c:42:9: note: in expansion of macro ‘io_new_conn’
   42 |         io_new_conn(conn, fds[1], init_writer, conn);
      |         ^~~~~~~~~~~
/home/rusty/devel/cvs/ccan/ccan/io/test/run-08-read-after-hangup.c:36:25: note: shadowed declaration is here
   36 |         struct io_conn *conn;
      |                         ^~~~

```

ccan/io/fdpass: tell caller that fd recv failed if they enabled extended errors

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/io: extended errors API so caller can be told if we fail accept()

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: fix clang complaining we return NULL if we promised not to.

Since really, the user is promising their alloc fn won't return NULL.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal, tal/str: allow annotation with returns_nonnull if they opt in.

This is true if you don't override the error callback, and helps with
optimization (and warnings!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rune: make error message better when integer parameter is missing.

We say "is not an integer field" when a field is missing entirely,
which is technically true but confusing.  Say "not present" like
every other condition does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

base64: really fix.

Playing games with chars is always a bad idea, but this time for sure!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

bitmap: Avoid shadowing type name with parameter name

'bitmap' is a typedef name in this module, and we also use it as a
parameter name in a bunch of places.  Although that's possible, because
types and variables live in different namespaces, it's probably not that
wise.  It also appears to trip warnings with at least some options and
compiler versions.

Reported-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

base64: fix for unsigned chars (e.g. ARM).

```
ccan/ccan/base64/base64.c:34:10: error: result of comparison of constant 255 with expression of type 'int8_t' (aka 'signed char') is always false [-Werror,-Wtautological-constant-out-of-range-compare]
        if (ret == (char)0xff) {
            ~~~ ^  ~~~~~~~~~~
ccan/ccan/base64/base64.c:44:57: error: result of comparison of constant 255 with expression of type 'const signed char' is always true [-Werror,-Wtautological-constant-out-of-range-compare]
        return (maps->decode_map[(const unsigned char)b64char] != (char)0xff);
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~
```

Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

opt: allow show callbacks to return false.

Sometimes, arguments are optional, so it's useful to decide at runtime
whether to print a default in --help.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan: update show callback to take explicit length.

For now we still use 80, but we could vary that in future.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan: allow user to set some bits in opt_table.type.

In particular, Core Lightning wants to use this to flag arguments
which can be specified multiple times (without overriding).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan: break out routine to parse a single long argument.

Core Lightning (ab)uses opt for config file parsing; this makes
that quite a bit neater!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan: add test that second arg in multiple arg string is correctly reported.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: increase stack size in HAVE_POINTER_SAFE_MAKECONTEXT test

This was *hanging* on GitHub's CI (Ubuntu 22.04) with clang.  The increase
seems to have fixed it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

runes: allow underscores in field names, as per Python runes 0.6

Suggested-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: remove undefined behavior.

In particular, we were casting between struct prop_hdr ** and char **:
fixing this properly was quite a rework.

And in one case we were using a pre-realloc pointer in tests: though to be
fair, we tested it for equality.  But ubsan got upset anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ungraph: new module

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tcon: fix warning when it's used with NULL on some gcc versions.

Seems to be happening with gcc 12.2.0-3ubuntu1:

```
lightningd/jsonrpc.c: In function ‘destroy_json_command’:
lightningd/jsonrpc.c:1180:63: error: the comparison will always evaluate as ‘false’ for the address of ‘canary’ will never be NULL [-Werror=address]
lightningd/jsonrpc.c:108:53: note: ‘canary’ declared here
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

fdpass: avoid memory leak in example.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

stringbuilder: avoid overwriting end of buffer with nul terminator.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tally: avoid complaints about bitshifts on giant bucket sizes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

objset: avoid memleak report in example.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rbuf: avoid doing memchr(NULL, ..., 0).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

opt: avoid bad access on empty usage message, free all mem in tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

membuf: avoid buffer overrun in example code, memleak.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

idtree: make sanitizer happier about shifts.

```
/home/rusty/devel/cvs/ccan/ccan/idtree/test/run-wrap.c:/home/rusty/devel/cvs/ccan/ccan/idtree/idtree.c:76:21: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

version: fix version() function.

```
/home/rusty/devel/cvs/ccan/ccan/version/test/run.c:/home/rusty/devel/cvs/ccan/ccan/version/version.h:58:35: runtime error: left shift of 65535 by 16 places cannot be represented in type 'int'
```

configurator: fix bitshift tests (-fsanitize=undefined)

```
configuratortest.c:5:24: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

antithread, htable, ilog: more misc sanitizer complaints.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

failtest: avoid complaints about copying uninitializes memory.

True, we don't actually use the "call", so omit it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

bytestring, crypto/hmac_sha256, darray, mem, tal: avoid mem* with NULL.

I swear the C standards committee likes compilers more than they like
programmers :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

bitops: fix -fsanitize=undefined warning:

```
/home/rusty/devel/cvs/ccan/ccan/bitops/test/run.c:28:3: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'

bitops: Module's run and api tests pass (tests_pass): FAIL (+1/2)
/home/rusty/devel/cvs/ccan/ccan/bitops/test/run.c:1..632
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tcon: fix overzealous gcc 12.2.0 warning.

At least with -fsanitize=address -fno-sanitize-recover=address -fsanitize=undefined -fno-sanitize-recover=undefined:

```
/home/rusty/devel/cvs/ccan/ccan/tcon/test/compile_fail-container1.c:Compile gave warnings without -DFAIL:
In file included from /home/rusty/devel/cvs/ccan/ccan/tcon/test/compile_fail-container1.c:3:
/home/rusty/devel/cvs/ccan/ccan/tcon/tcon.h: In function ‘main’:
/home/rusty/devel/cvs/ccan/ccan/tcon/tcon.h:332:25: warning: ‘innerp’ may be used uninitialized [-Wmaybe-uninitialized]
  332 |                 canary, tcon_container_of_((member_ptr),                \
      |                         ^~~~~~~~~~~~~~~~~~
/home/rusty/devel/cvs/ccan/ccan/tcon/tcon.h:335:21: note: by argument 1 of type ‘const void *’ to ‘tcon_container_of_’ declared here
  335 | static inline void *tcon_container_of_(const void *member_ptr, size_t offset)
      |                     ^~~~~~~~~~~~~~~~~~
/home/rusty/devel/cvs/ccan/ccan/tcon/test/compile_fail-container1.c:28:22: note: ‘ovar’ declared here
   28 |         struct outer ovar;
      |                      ^~~~

tcon: Total score: 33/41 FAIL!
```

ccanlint: do fewer reallocs, to speed up under -fsanitize=address

This is just a hack, but it makes the speed managable for ccan/utf8
which produces a lot of output!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccanlint: fix use-after-free in verbose mode.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rune: add helpers for users doing DIY parsing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rune: avoid shadowing var in test/run.c

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rune: fix encoding of alternates which need escapes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

net: fix gcc warning about mismatch.

Header can't use pollfds[2] because type is unknown.  And gcc complains:

ccan/net/net.c:78:70: warning: argument 2 of type ‘struct pollfd[2]’ with mismatched bound [-Warray-parameter=]

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

alloc: fix shadowed variable

Thanks -Wshadow=local!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: fix shadowed variable name.

Thanks -Wshadow=local!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Makefile: add -Wshadow=local, fix configurator!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/base64: fix GCC warning.

```
ccan/ccan/base64/base64.c:146:71: error: argument 2 of type ‘char[3]’ with mismatched bound [-Werror=array-parameter=]
  146 | ssize_t base64_decode_tail_using_maps(const base64_maps_t *maps, char dest[3],
      |                                                                  ~~~~~^~~~~~~
In file included from ccan/ccan/base64/base64.c:2:
ccan/ccan/base64/base64.h:119:72: note: previously declared as ‘char *’
  119 | ssize_t base64_decode_tail_using_maps(const base64_maps_t *maps, char *dest,
      |                                                                  ~~~~~~^~~~
cc1: all warnings being treated as errors
make: *** [Makefile:817: ccan-base64.o] Error 1
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/rune: compile without warnings on -O3.

```
ccan/ccan/rune/rune.c: In function ‘rune_alt_single_int’:
ccan/ccan/rune/rune.c:257:5: error: ‘runeval_int’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  257 |  if (cond)
      |     ^
ccan/ccan/rune/rune.c:305:6: note: ‘runeval_int’ was declared here
  305 |  s64 runeval_int;
      |      ^~~~~~~~~~~
cc1: all warnings being treated as errors
make: *** [Makefile:919: ccan-rune-rune.o] Error 1
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/rune: simplify check helper interfaces, allow explicit string lengths.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/rune: add explicit-length variants.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/rune: fix take() logic which caused memory leaks.

You have to tal_steal() if you want to keep it!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan/io: add debugging helper for seeing if module is in charge of an fd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

strmap: add strmap_getn for non-terminated string search.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: fix typo in example.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

rune: new module to implement runes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: opportunistically avoid delete marker.

Fairly cheap test, we can see a drop in initial re-inserting, probably
because it has just been cleaned at this point with the old code, but
generally we do better with churn.

I also tried a variant which only checked if next bucket was on same
cacheline, but no significant difference.

5 runs of tools/speed 10000000:

```
-Initial re-inserting: 88-99(94.6+/-5) ns
+Initial re-inserting: 104-116(107.2+/-4.4) ns
-Adding (a different) half: 94-101(98.6+/-3) ns
+Adding (a different) half: 38-44(39.8+/-2.1) ns
-Lookup after half-change (miss): 90-105(96.4+/-5.7) ns
+Lookup after half-change (miss): 112-118(113.4+/-2.3) ns
-Churning third time: 301-323(314.4+/-9.8) ns
+Churning third time: 233-259(239+/-10) ns
-Details: worst run 146 (24 deleted)
+Details: worst run 115 (2 deleted)
-Lookup after churn & spread (match): 78-87(83.6+/-4.2) ns
+Lookup after churn & spread (match): 71-81(73.6+/-3.8) ns
-Lookup after churn & spread (miss): 118-137(124.2+/-6.9) ns
+Lookup after churn & spread (miss): 93-104(95.4+/-4.3) ns
-Adding (a different) half after churn & spread: 81-89(85.4+/-3.6) ns
+Adding (a different) half after churn & spread: 38-43(39.6+/-1.7) ns
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: increase density from 75% to 87.5%

Across all benchmarks (on various sizes) this actually drops
performance by 3%.  However, density increasing by 15% is probably a
worthwhile tradeoff.

```
-Initial insert: 227-237(230+/-3.7) ns
+Initial insert: 266-281(273.4+/-6.5) ns
-Initial lookup (match): 65-66(65.4+/-0.49) ns
+Initial lookup (match): 67-78(70.6+/-3.8) ns
-Initial lookup (miss): 79-80(79.4+/-0.49) ns
+Initial lookup (miss): 82-93(84.8+/-4.2) ns
-Initial lookup (random): 111-115(112.6+/-1.6) ns
+Initial lookup (random): 118-138(123.4+/-7.4) ns
-Initial delete all: 66-68(66.6+/-0.8) ns
+Initial delete all: 79-90(82+/-4.1) ns
-Initial re-inserting: 87-90(88+/-1.3) ns
+Initial re-inserting: 104-116(107.2+/-4.4) ns
-Deleting first half: 32-33(32.4+/-0.49) ns
+Deleting first half: 37-43(39+/-2.1) ns
-Lookup after half-change (match): 67-69(68+/-0.89) ns
+Lookup after half-change (match): 72-81(74.2+/-3.4) ns
-Churning second time: 272-276(274.2+/-1.5) ns
+Churning second time: 304-332(310.2+/-11) ns
-Churning third time: 290-296(293+/-2) ns
+Churning third time: 233-259(239+/-10) ns
-Churning fourth time: 289-295(292+/-2.2) ns
+Churning fourth time: 309-337(315.6+/-11) ns
-Churning fifth time: 289-294(291.8+/-1.9) ns
+Churning fifth time: 307-337(315+/-11) ns
-Details: worst run 146 (24 deleted)
+Details: worst run 115 (2 deleted)
-Lookup after churn & spread (miss): 110-111(110.6+/-0.49) ns
+Lookup after churn & spread (miss): 93-104(95.4+/-4.3) ns
-Deleting half after churn & spread: 36-37(36.4+/-0.49) ns
+Deleting half after churn & spread: 39-45(40.4+/-2.3) ns
-Adding (a different) half after churn & spread: 81-82(81.6+/-0.49) ns
+Adding (a different) half after churn & spread: 38-43(39.6+/-1.7) ns
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: tools/density

Everyone loves stats: this looks at different levels of table density
and lookup speed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: fix vanishing entries properly.

A hash table entry consists of the pointer, but we mask out bits which
are common to all pointers, and replace them with hash bits.

The entry values 0 and 1 are special, meaning "empty" and "deleted"
respectively.

However, if a hash has mainly zero bits, and an pointer's non-shared
bits are all zero, we can create an entry which looks empty or
deleted.

The solution in this case is to share another bit (there must be some
bit which is not zero, since we don't allow 0 and 1 as pointers in the
hash table).

It's unusual, so I put this in a cold function: it shares code with
the existing "oh, we need to reduce the common bits, since this new
pointer doesn't look like the previous ones" case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Partial revert "htable: handle v. unlikely case where entries look deleted/empty."

This reverts commit 7218ee2476d8a7e4f9413781e2bedcd2daaa8190.

Except for:
 - Leaves the test behind though (which fail!
 - Leaves the reversion of 0b93bd102aad6b61f1e569fb12aabc6352a1d7cd, since
   we're about to fix it, and properly!

Revert "htable: optimize a little more."

This reverts commit 0f144bfe0ca3cc06564e19a7f3ea40177d1a3c2c.

htable: optimize a little more.

tools/speed (10 runs) vs before fix:

-Lookup after half-change (match): 53-61(54.8+/-2.3) ns
+Lookup after half-change (match): 63-97(72.4+/-9) ns
-Churning fifth time: 171-181(175.4+/-2.9) ns
+Churning fifth time: 179-198(183.9+/-5.2) ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

htable: handle v. unlikely case where entries look deleted/empty.

If the hash function doesn't set any bits we use, and the common
bits are all zero the slot will look empty (or, just the lower bit is
set: the slow looks deleted).

However, each bucket is distinct since there are no duplicates, so
at worse there can be two "looks invalid but actually is valid"
buckets.  Keep them at the end.

Lookup suffers in raw tools/speed though:

-Lookup after half-change (match): 53-61(54.8+/-2.3) ns
+Lookup after half-change (match): 61-113(83.3+/-17) ns
-Lookup after churn & spread (match): 54-90(59.9+/-11) ns
+Lookup after churn & spread (match): 70-108(85.2+/-14) ns

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

io: add new io_sock_shutdown helper.

Turns out that there's no good way to flush a socket:

https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

tal: add tal_dup_or_null and tal_dup_talarr helpers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

crc32c, crypto/xtea: check in LICENSE symlinks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

graphql: latest changes from @rl-d

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ccan: make tal_dump() format more regular.

Putting a \n in there was a pain: I had to do:

awk '/CHILDREN/ { SAVED=$0 } !/CHILDREN/ { print SAVED$0; SAVED="" }' < lightningd-tal_dump > lightningd-tal_dump-children-on-same-line

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

pipecmd: Use closefrom instead of iterating directly.

Signed-off-by: ZmnSCPxj jxPCSnmZ <ZmnSCPxj@protonmail.com>

closefrom: fix minor issues found by ccanlint

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

closefrom: Close all file descriptors above a certain value.

For more information: https://github.com/ElementsProject/lightning/issues/4868

Signed-off-by: ZmnSCPxj jxPCSnmZ <ZmnSCPxj@protonmail.com>

web: create "versioned" tarballs.

Fixes: https://github.com/rustyrussell/ccan/issues/90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

base64: fix assignment to wrong type, use ssize_t

Functions that returned int have been modified to return ssize_t
instead.

Found by gcc:

base64.c:196:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  if (more == -1) {
           ^

Signed-off-by: Martin Milata <martin@martinmilata.cz>

nit: daemonize.c: fix Stevens' in comment

Setting the same name as at the implementation

configurator: always true branch

Fix "for loop initial declarations are only allowed in C99 mode" compile errors.

docs: fix simple typo, unchaged -> unchanged

There is a small typo in ccan/objset/objset.h.

Should read `unchanged` rather than `unchaged`.

fix warnings

GraphQL lexer and parser (without optional type system)

tal: tal_dump() now prints to stderr.

This is more widely available than stdout, which can interfere with
other things.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

compiler: RETURNS_NONNULL macro

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

configurator: HAVE_ATTRIBUTE_RETURNS_NONNULL

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

timer: handle time going backwards.

As seen in https://github.com/ElementsProject/lightning/issues/4401

OpenBSD 6.8 inside VirtualBox.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

utf8: don't allow NUL in decoded strings.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

json_out: make json_out_finished finish buffer.

For JSON RPC, we can actually write multiple JSON objects back
to back, and we don't want a comma between them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

*: Fix build with recent gcc/binutils

With current Debian testing I get:

tools/ccanlint/ccanlint -s --deps-fail-ignore   ccan/compiler/ && touch ccan/compiler/.ok
compiler: Module tests compile (tests_compile): FAIL
/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/compiler/test/run-is_compile_constant.c:Compile failed:
/usr/bin/ld: /tmp/ccanlint-129350.1804289383/tap.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/tap/tap.h:249: multiple definition of `tap_fail_callback'; /tmp/ccrAwqbl.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/ccan/tap/tap.h:249: first defined here
collect2: error: ld returned 1 exit status
...

cc -g3 -ggdb -Wall -Wstrict-prototypes -Wold-style-definition -Wundef -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wwrite-strings -DCCAN_STR_DEBUG=1 -I.  -MMD -MP -MFtools/ccanlint/ccanlint.d -MTtools/ccanlint/ccanlint tools/ccanlint/ccanlint.c tools/ccanlint/async.o tools/ccanlint/file_analysis.o tools/ccanlint/licenses.o tools/ccanlint/tests/avoids_cpp_reserved.o tools/ccanlint/tests/depends_accurate.o tools/ccanlint/tests/depends_build.o tools/ccanlint/tests/depends_build_without_features.o tools/ccanlint/tests/depends_exist.o tools/ccanlint/tests/examples_compile.o tools/ccanlint/tests/examples_exist.o tools/ccanlint/tests/examples_relevant.o tools/ccanlint/tests/examples_run.o tools/ccanlint/tests/hash_if.o tools/ccanlint/tests/headers_idempotent.o tools/ccanlint/tests/info_compiles.o tools/ccanlint/tests/info_documentation_exists.o tools/ccanlint/tests/info_exists.o tools/ccanlint/tests/info_ported.o tools/ccanlint/tests/info_summary_single_line.o tools/ccanlin
 t/tests/license_comment.o tools/ccanlint/tests/license_depends_compat.o tools/ccanlint/tests/license_exists.o tools/ccanlint/tests/license_file_compat.o tools/ccanlint/tests/main_header_compiles.o tools/ccanlint/tests/main_header_exists.o tools/ccanlint/tests/module_builds.o tools/ccanlint/tests/module_links.o tools/ccanlint/tests/no_trailing_whitespace.o tools/ccanlint/tests/objects_build.o tools/ccanlint/tests/objects_build_with_stringchecks.o tools/ccanlint/tests/objects_build_without_features.o tools/ccanlint/tests/reduce_features.o tools/ccanlint/tests/tests_compile.o tools/ccanlint/tests/tests_compile_coverage.o tools/ccanlint/tests/tests_coverage.o tools/ccanlint/tests/tests_exist.o tools/ccanlint/tests/tests_helpers_compile.o tools/ccanlint/tests/tests_pass.o tools/ccanlint/tests/tests_pass_valgrind.o tools/ccanlint/tests/tests_pass_without_features.o ccan/autodata/autodata.o ccan/dgraph/dgraph.o ccan/ilog/ilog.o ccan/lbalance/lbalance.o ccan/ptr_valid/ptr_valid.o ccan/strma
 p/strmap.o tools/ccan_dir.o tools/compi
le.o tools/depends.o tools/doc_extract-core.o tools/gcov.o tools/manifest.o tools/read_config_header.o tools/tools.o ccan/asort/asort.o ccan/err/err.o ccan/foreach/foreach.o ccan/hash/hash.o ccan/htable/htable.o ccan/list/list.o ccan/membuf/membuf.o ccan/noerr/noerr.o ccan/opt/helpers.o ccan/opt/opt.o ccan/opt/parse.o ccan/opt/usage.o ccan/rbuf/rbuf.o ccan/read_write_all/read_write_all.o ccan/str/debug.o ccan/str/str.o ccan/take/take.o ccan/tal/tal.o ccan/tal/grab_file/grab_file.o ccan/tal/link/link.o ccan/tal/path/path.o ccan/tal/str/str.o ccan/time/time.o -lm -o tools/ccanlint/ccanlint
/usr/bin/ld: tools/ccanlint/async.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/file_analysis.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/licenses.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/avoids_cpp_reserved.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined
here
/usr/bin/ld: tools/ccanlint/tests/depends_accurate.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/depends_build.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
/usr/bin/ld: tools/ccanlint/tests/depends_build_without_features.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/./tools/ccanlint/../ccanlint/licenses.h:33: multiple definition of `license_compatible'; /tmp/cc72h7qq.o:/home/kirr/src/wendelin/wendelin.core/3rdparty/ccan/tools/ccanlint/../ccanlint/licenses.h:33: first defined here
...

The errors are valid becuase symbols are declared without extern in .h
files and those .h files are included from multiple translation units.

-> Fix it. I'm not sure why it used to compile ok before.

Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Message-Id: <20200910191610.131510-1-kirr@nexedi.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

tools/configurator: fix compile error with -O2

In file included from /usr/include/string.h:495,
                 from configuratortest.c:2:
In function ‘strncpy’,
    inlined from ‘main’ at configuratortest.c:6:2:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: warning: ‘__builtin_strncpy’ specified bound 8 equals destination size [-Wstringop-truncation]
  106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>