Values such as q=1 or priv=0 caused infinite loops when calling
DSA_sign() without these changes.
There are other cases where bad domain parameters may have caused
infinite loops where the retry counter has been added. The simpler case
of priv=0 also hits this case. q=1 caused an infinite loop in the setup.
The max retry value has been set to an arbitrary value of 8 (it is
unlikely to ever do a single retry for valid values).
The minimum q bits was set to an arbitrary value of 128 (160 is still
used for legacy reasons when using 512 bit keys).
Thanks @guidovranken for detecting this, and @davidben for his
insightful analysis.
Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20384)