Fix KMAC bounds checks.

Fix KMAC bounds checks.

Setting an output length higher than 8191 was causing a buffer overflow.
This was reported by Acumen (FIPS lab).

The max output size has increased to ~2M and it now checks this during set_parameters.

The encoder related functions now pass in the maximum size of the output buffer so they
can correctly check their size. kmac_bytepad_encode_key() calls bytepad twice in
order to calculate and check the length before encoding.

Note that right_encode() is currently only used in one place but this
may change if other algorithms are supported (such as TupleHash).

Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15106)