kernel-netlink: Change priority calculation for compatibility with Linux 5.7

kernel-netlink: Change priority calculation for compatibility with Linux 5.7

Already since a patch from 2013 the Linux kernel would allow duplicate
policies with different priorities when installing.  However, the code
actually didn't work as intended and policies with the same mark/mask were
always considered equal no matter what the priority was.  This changed
recently with the 5.7 kernel.  Now policies with the same mark are only
considered equal if their policy matches too.  This means when we update
policies and the priority changes duplicate policies are created in the
kernel.  These are then later not deleted as we only delete the policies
once.  And because we can't specify the priority when deleting/querying,
the kernel implementation is actually quite problematic anyway.

To workaround this issue, we now calculate the priority based on the
selector only.  The type (and whether it's a trap policy) is just used
when we sort the policies ourselves to decide which type of policy to
install.  This means bypass, drop, and regular IPsec policies are now all
in the same priority range (trap policies were already in the same class
for a while).

The additional high-level priority class (policy_priority_t) is only
used to distinguish between the two types of POLICY_IPSEC polices.  So
maybe we could change this somehow (e.g. add a new type to policy_type_t,
or make it an ORable flag there).