Set stricter respdiff:recent-named threshold

This job is testing the current BIND implementation against the latest
released version. Unless there has been a behaviour change, there should
be no difference.

In practice, there is a small number of differences caused by upstream
discrepencies. Some of those cause "upstream unstable" answers which are
excluded from the results, but statistically, some of those will end up
being detected as differences on the resolver under test.

Currently, there seems to be about 300 upstream unstable answers with
typically around 50-60 differences. Setting the threshold to 0.1 should
be stable enough to pass if there are no changes, yet sensitive enough
to detect even fairly small changes to behaviour.

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 90b2b9d834891a8f62c3391c83ccaa38a24d21ca..196ce3f807af1a681dd885f3269d5d96ea622814 100644 (file)
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -2027,7 +2027,7 @@ respdiff:recent-named:
     CC: gcc
     CFLAGS: "${CFLAGS_COMMON} -DISC_TRACK_PTHREADS_OBJECTS"
     EXTRA_CONFIGURE: "-Doptimization=g"
-    MAX_DISAGREEMENTS_PERCENTAGE: "0.3"
+    MAX_DISAGREEMENTS_PERCENTAGE: "0.1"
 
 # Performance tests