evaluate: convert expr_rt byteorder when evaluating statment arg

evaluate: convert expr_rt byteorder when evaluating statment arg

expr_rt might write data in host byte order, so make sure to
convert if needed.

This makes 'tcp option maxseg size rt mtu' actually work, right now such rules
are no-ops because nft_exthdr never increases the mss.

While at it, extend the example to not bother testing non-syn packets.

Reported-by: Matteo Croce <technoboy85@gmail.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
Acked-by: Pablo Neira Ayuso <pablo@netfilter.org>