[9.20] fix: dev: Validate nsec3hash arguments instead of relying on atoi()

[9.20] fix: dev: Validate nsec3hash arguments instead of relying on atoi()

The nsec3hash tool parsed its algorithm, flags, and iterations
arguments with atoi(), then range-checked the result. For values
that overflow int during digit-by-digit accumulation, atoi() is
undefined; in practice on musl libc the modular wrap leaves
n == 0, which silently passes the "iterations > 0xffffU" check.
On Alpine Linux this made nsec3hash succeed with iterations
treated as 0 for inputs like 4294967296 (2^32).

The latent bug only surfaced when the recent image rebuild pulled
in Hypothesis 6.152.9 (2026-05-19), which unified the distribution
used for bounded and unbounded integers() strategies. The new
smoother distribution explores the 2^32 boundary on unbounded
ranges like integers(min_value=65536); earlier versions did not
reach there, so test_nsec3hash_too_many_iterations only started
failing on Alpine after the image refresh.

Replace the three atoi() calls with isc_parse_uint8 /
isc_parse_uint16, which uniformly reject overflow, trailing
garbage, leading sign, and non-numeric input across libc
implementations. As a side effect, error messages now include
the offending argument and a specific reason ("out of range" vs
"not a valid number").

Assisted-by: Claude:claude-opus-4-7
Closes #6013

Backport of MR !12062

Merge branch 'backport-6013-nsec3hash-iterations-overflow-9.20' into 'bind-9.20'

See merge request isc-projects/bind9!12074