Clean up format_hex_ex()

Clean up format_hex_ex()

Cherry-pick of commit 29404010 from master, slightly reworked to match the
2.3 codebase (no flags inside space_break here), and pulled in the new
static_assert() fallback we also have in master now.

Fix a potential null-pointer dereference, and make the code a bit more
readable while doing so.

The NULL dereference could not be triggered, because the current code
never called format_hex_ex() with maxouput == 0 and separator == NULL.
But it's nicer to not depend on that.

Our use of int vs size_t for lengths needs some attention too, but I'm
not pulling that into this patch.  Instead I decided to just make the
(previously existing) assumption that INT_MAX <= SIZE_MAX explicit by
adding a static_assert().

Signed-off-by: Steffan Karger <steffan.karger@fox-it.com>
Acked-by: Gert Doering <gert@greenie.muc.de>
Message-Id: <1480352107-19652-1-git-send-email-steffan.karger@fox-it.com>
URL: https://www.mail-archive.com/openvpn-devel@lists.sourceforge.net/msg13269.html
Signed-off-by: Gert Doering <gert@greenie.muc.de>