Fix DB2 hash bitmap page count validation

Fix DB2 hash bitmap page count validation

In __kdb2_hash_open(), bpages is computed from the hash file header
and then used as the size argument when clearing hashp->mapp.  The
mapp array has only NCACHED entries, so a malformed hash database can
cause memset() to write past the end of the array.  Return EFTYPE if
the computed bitmap page count is negative or greater then NCACHED.

Found by Linux Verification Center (linuxtesting.org) with SVACE.

ticket: 9215