[9.20] chg: dev: Use enum rather than numbers for isc_base64_tobuffer and isc_hex_tob...

Use isc_one_or_more and isc_zero_or_more rather than (-2) and
(-1) when calling isc_base64_tobuffer. Similarly for
isc_hex_tobuffer. This should help reduce the probability
that the wrong number is used and it makes the intent clearer.

Closes #5713

Backport of MR !11479

Merge branch 'backport-5713-use-macros-with-isc_base64_tobuffer-and-isc_hex_tobuffer-9.20' into 'bind-9.20'

See merge request isc-projects/bind9!11498