discover/boot: Download resources in parallel

Now that we can download asynchronously, we can download boot resources
in parallel. This also means we don't need a state machine; we're either
not done (loads are still pending) or done.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: Add struct load_url_result and load_task

Currently, load_url and load_url_async return a filename parameter and a
tempfile flag (indicating whether the file needs to be cleaned after
use).

Instead, encapsulate this data in a struct load_url_result, which the
caller (and async callbacks) can read the status, filename and clean
parameters.

For internal use in load_url and helpers, we add a struct load_task to
hold a pointer to the load_url_result and async data.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: Don't depend on tftp failure for type detection

Rather than always trying both TFTP client types, do a runtime detection
on first invocation. This can be fixed at build-time with
--with-tftp=TYPE.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/boot: Store clean_* in the boot task

... otherwise we won't clean up local files from earlier boot stages.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/boot: Check for failed loads in boot()

Use the status argument to the URL load callback to check for load
failures.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: load_url_async callback should take an int status

We don't need a pointer here, just the status value.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/boot: Implement asynchronous boot() process

The current boot process handled by boot() function is implemented such
that it blocks on the boot resources being downloaded which blocks
the discover process until the download completes. This patch
re-implements the boot() function using callback mechanism and
asynchronous notifications so that boot() returns to continue the
discover process while the download is in progress.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Signed-off-by: Neelesh Gupta <neelegup@linux.vnet.ibm.com>

discover: Add an asynchronous version of load_url

A new load_url_async() function handles retrieving the resources by
forking an external process which currently blocks the parent waiting
for the process to complete. This patch modifies the load_*() functions
to handle it asynchronously by providing the exit callback and returning
without waiting for the child process to exit.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Signed-off-by: Neelesh Gupta <neelegup@linux.vnet.ibm.com>

discover/grub2: Reimplement default options

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: Check for devices with duplicate serial properties

If we have multipath devices present in the system, we'll end up with
duplicate mounts, parse results and boot options. This change adds a
check to see if we've encountered a device with this serial number
previously.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

pb-udhcpc: use plain kernel names for dhcp events

No need to prefix with /net/

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Fix uninitialised op variable

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/parser: Add test-grub2-single-line-if test

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Support var=value assignments

We want to allow assignments outside of the 'set' builtin.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/parser: Allow double-quotes in embedded config

We need to escape any double-quotes.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add true & false builtins

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/parser: Allow checks for NULL boot_args

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/parser: Add variable expansion in Ubuntu conf file test

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2/lexer: Require 1+ spaces for a TOKEN_DELIM

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add linux16 command

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Allow both delimiters and EOLs after a "then" reserved word

Putting an if-statement on one line should be possible:

 if true; then echo "true"; else echo "false"; fi

To do this, we'll need to allow both EOLs and TOKEN_DELIM after then and
else.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/parser: Fix spacing in grub2-f18-ppc64 test

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement test builtin

A faily simple implementation now, and could do with some testing...

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add nops for unnecessary builtins

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement statement_block_execute

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: unknown commands should fail

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Populate option ids

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Allow EOF as a statement terminator

Files may not have a trailing newline.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add print_token

For better trace output.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: fix comment handling

Remove '#' from the list of word chars. Also, don't need the '$' char in
the comment regex

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement 'elif'

Rather than just having one conditional in an if statement, we use a
list of conditionals instead. This allows us to implement elif.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Create 'conditional' statements

Rather than holding the condition and conditional-statements in struct
grub2_statment_if, create a new conditional type that contains these. We
can then use this to implement elif statements.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Allow empty statements

Re-arrange the script/statements parse rules to allow empty statements.
We also move the EOL token out of the 'statement' definition, and use
it to separate 'statements'.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement submenu

Just recurse into the submenu items, as we flatten everything into the
one menu.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add function support

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Improve error handling

Only run the script if the parse succeeded, and and improve the
error reporting by enabling verbose errors and line numbers.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement 'else' blocks

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement 'search' command

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement 'linux' & 'initrd' commands

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Function infrastructure improvements

For user-defined functions, we'll need a data pointer to the function's
execution callback. Add this as a void *, and change references from
'command' to 'function'.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add option state in menuentry processing

This will allow menuentry-specific commands to populate boot option
data.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add a reference to the context from grub2_script

We'll need the context to add boot options.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Hook up flex/bison parser to discover server

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Move parser-api definitions to parser.y

No need for a separate file here.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement multiple-arv variable splitting

If we expand a variable containing word-delimiter chars, we need to
create new argv items.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add initial command execution

.. with a simple 'set' command to update the environment

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add menuentry execution

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Fix word append

The parser was setting word->next on the top-level word, resulting in
word chains of a maximum length of 2. Instead, use a last pointer to
always append to the last word.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Handle var tokens in lexer

Rather than post-processing to expand variables, use the lexer to
identify variable tokens as a type of grub2_word.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Remove debug printfs

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Initial environment handling

A simple linked-list implementation of string pairs.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add initial execution code

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Fix statement reduce rule

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add var expansion code

Still todo: splitting.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add initial script infrastructure

Now that we can parse scripts, we want some infrastructure for
execution.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Use subclass-style structs for AST

Use embedded 'struct grub2_statement' data to create a subclass-style
syntax tree of statements.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement statement blocks

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement reduce rules

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Add structures & API for parser

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Implement parser

Add our simple parser to handle a grub2 script. Since we're not building
a parse tree at the moment (we have no reduce rules), we just have a
simple word-based %union for our tokens.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/grub2: Initial grub2 lexer

Add a simple flex lexer to tokenise our grub2 config files.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib: Add array-size header

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/boot: Add booting status message

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/boot: Allow printf formatting in update_status

It'd be nice to include arbitrary format strings here, so do a
taloc_vasprintf in update_status.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/boot: Fix log message with no newline

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: clear default_timeout waiter on timeout

We don't want cancel_default to try to remove an already-removed waiter.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: Don't register multiple boot_timeout waiters

Rather than adding another timeout waiter, just override
handler->default_boot_option and extend the timeout a little.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/parser: check for multiple default options

Update check_boot_option_count to check if a parser returned more than
one boot option with is_default set.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/pxe: Implement default options

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: Implement device priorities

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: register network devices with the device handler

We'd like to correlate incoming network boot options with a device, so
register the interface with the device hander.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/network: Manage network->interfaces with {add,remove}_interface

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/network: ignore devices with zero MTU

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: Remove unnecessary event passing

Currently, we pass "events" between the udev, user-event and
device-handler layers. These events all get sent through
device_handler_event, then de-multiplexed to an appropriate handler,
depending on their source.

Instead, just export relevant device_handler functions, and have the
(old) event sources call these functions directly.

This also means we can include a lot more of the device hander code in
the parser tests.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/boot: Fix use-after-free in boot()

We need boot_task for kexec_reboot.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/pb-config: Initialise DNS server info

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/log: Cleanup log API

Rather than exposing log internals (through always_flush and
set_stream), do all logging init through pb_log_init(). If pb_log_init()
hasn't been called, pb_log will drop messages.

Also, add a pb_debug() function, specifically for debugging information.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: Consolidate device path, name and ID.

This change cleans up our usage of device path, names and IDs.

Device ID is the kernel name for the device. We also expose this through
lookup_by_name.

Device path is the path to the dev node (ie, always starts with /dev/),
and is only used for mounting.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover: populate udev device types

Now that we have device types, populate from the udev info.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

types: Add device_type to struct device

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/pb-config/powerpc-nvram: Add petitboot,timeout nvram property

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/pb-config: Add autoboot timeout to configuration vars

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

pb-udhcpc: Fix string returned from resolve_url

In the 'as-is' case, we were echoing both URLs.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/pxe: check for a valid boot option before adding

If we didn't find any valid boot options in the pxe buffer, we'll call
discover_context_add_boot_option with a NULL boot option.

This change adds a check before we try to add the boot option, and a
test to verify this situation.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

ui/ncurses: Allow both DEL and ^H to map to backspace

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/parser: add 'partition' directive overriding test for yaboot conf file

Check the overriding of 'partition' directive from yaboot conf file for
'petitboot', then verify the device name on which the resource resides

Signed-off-by: Neelesh Gupta <neelegup@linux.vnet.ibm.com>
Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/yaboot-parser: Handle 'partition=' directive override

In a yaboot conf file, we may see a device= directive that actually
specifies a partition (eg, sda1) rather than the underlying block device
(sda). If we then encounter a partition= directive, we don't handle the
resolution of the partition correctly, as we simply append the
partition number to the device= string.

This change implements a smarter handling of the partition= directive,
where we strip away any partition information from the device=
parameter first.

Signed-off-by: Neelesh Gupta <neelegup@linux.vnet.ibm.com>
Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test: add 'partition=directive' test for yaboot conf file

Signed-off-by: Neelesh Gupta <neelegup@linux.vnet.ibm.com>
Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/lib: Implement process_init change in testcases

We need to provide the dry_run argument.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/process: handle EINTR from waitpid

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/process: Add dry_run argument to process_init

Implement dry-run behaviour on the discover server by passing a bool to
process_init. UIs don't need to support dry runs.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/process: replace pb_run_cmd

This change replaces the pb_run_cmd() function with proper usage of the
process API.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/process: replace pb_run_cmd_pipe

Replace pb_run_cmd_pipe with process_create / process_run_sync.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/lib: add process tests

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/waiter: handle -EINTR

Now that we're handing non-fatal signals (i.e., SIGCHLD in the process
lib), we need to gracefully handle -EINTR from poll().

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

lib/process: Add process helpers

We've grown-out of pb_run_cmd a little, as we have a number of different
process types:

   boot():
    - kexec: short-running process, run synchronously
    - boot hooks: short-running, run sync, need exit code & stdout

   network init:
    - interface configuration: short running, run sync
    - udhcp processes are long running, we may want completion, but
      doesn't block other actions

   downloads:
    - potentially long-running, block parse progress

   config nvram:
    - read: short running, can block, need stdout
    - write: short running, can block

We'd like to introduce proper asynchronous processes, to allow config &
boot-option downloads without blocking the discover server.

This change introduces a new type for processes, 'struct process'. These
structures are created with process_create, and run with
process_run_sync or process_run_async. The latter reports completion
through a callback member of struct process.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test: Add --enable-test-valgrind

Add an --enable-test-valgrind argument to configure, to run all tests
under valgrind, configured to fail on leaked memory

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/lib: Use talloc in list test

.. so we can free at exit

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

test/lib: build with -DEBUG

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>

discover/boot: talloc struct boot_task

Rather than using the stack for struct boot_task, talloc one instead.
This gives us a short-lived context (active for the boot() process
only), and we don't need to use the externally-provided context
directly.

Signed-off-by: Jeremy Kerr <jk@ozlabs.org>