tests: shell: add test case for checkentry hook validations

tests: shell: add test case for checkentry hook validations

A few matches/targets reject based on the calling hook mask
from their checkentry functions.  Some are cosmetic (reject
nonsensical rule that would not work, but others are mandatory
rejects, in particular TCPMSS which may need skb_dst()
depending on the requested mode of operation.

For -legacy this yields:
xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks
xt_addrtype: output interface limitation not valid in PREROUTING and INPUT
xt_addrtype: input interface limitation not valid in POSTROUTING and OUTPUT
xt_physdev: --physdev-out and --physdev-is-out only supported in the FORWARD and POSTROUTING chains with bridged traffic
xt_physdev: --physdev-out and --physdev-is-out only supported in the FORWARD and POSTROUTING chains with bridged traffic
xt_policy: input policy not valid in POSTROUTING and OUTPUT
xt_policy: output policy not valid in PREROUTING and INPUT

... in dmesg.  -j SET is currently missing, could be added
later (needs an existing ipset).

Signed-off-by: Florian Westphal <fw@strlen.de>