Allow re-run of the shotgun jobs to reduce false positive

Allow re-run of the shotgun jobs to reduce false positive

The false positive rate is about 10-20 % when evaluating shotgun results
from a single run. Attempt to reduce the false positive rate by allowing
a re-run of failed jobs.

While there is a slight risk that barely noticable decreases in
performance might slip by more easily in MRs, they'd still likely pop up
during nightly or pre-release testing.

Also increase the tolerance threshold for DoH latency comparisons, as
those tests often experience increased jitter in the tail end latencies.

(cherry picked from commit 5eab352478623eef57008a274c3a6505d9c76390)