Cleanup libnettle detection (#1144)

Cleanup libnettle detection (#1144)

... to use pkg-config compatible variable naming and break out
the code test to a dedicated macro per the style used by other
libraries.

Also, add support for pkg-config library auto-detection.

Also, our libcompat provides local replacement for Base64 on
builds with older libnettle. Fix the library link order to
ensure our functions get linked properly.