chg: dev: Use enum rather than numbers for isc_base64_tobuffer and isc_hex_tobuffer

Use isc_one_or_more and isc_zero_or_more rather than (-2) and
(-1) when calling isc_base64_tobuffer. Similarly for
isc_hex_tobuffer. This should help reduce the probability
that the wrong number is used and it makes the intent clearer.

Closes #5713

Merge branch '5713-use-macros-with-isc_base64_tobuffer-and-isc_hex_tobuffer' into 'main'

See merge request isc-projects/bind9!11479