lib/hexdump.c: reduce stack variable size and cleanups