Enhance documentation for `BN_mask_bits()`

Enhance documentation for `BN_mask_bits()`

Fixes #5537

Added a note that the error check for `BN_mask_bits()` depends
on the internal representation that depends on the platform's word size.
Included a reference to the `BN_num_bits()` function for precise bit checking.

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24812)