tdb2: change API to return the error value.

Mostly a fairly simple transformation, since 0 still means success.

One new twist is that tdb_nextkey now frees the .dptr of the key; this
us usually what we want but does cause issues for our weird test code.