From: Dan Fandrich <dan@coneharvesters.com>
Date: Fri, 8 Sep 2023 06:48:33 +0000 (-0700)
Subject: tests: add the timing-dependent keyword on several tests
X-Git-Tag: curl-8_4_0~211
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=381792dfbf11ceb5a6f83217f7b3ee751209ece5;p=thirdparty%2Fcurl.git

tests: add the timing-dependent keyword on several tests

These are ones likely to fail on heavily-loaded machines that alter the
normal test timing. Most of these tests already had the flaky keyword
since this condition makes them more likely to fail on CI.
---

diff --git a/tests/FILEFORMAT.md b/tests/FILEFORMAT.md
index 57bfb4f5bb..665d93eb85 100644
--- a/tests/FILEFORMAT.md
+++ b/tests/FILEFORMAT.md
@@ -190,7 +190,9 @@ tests. Try to use already used keywords. These keywords will be used for
 statistical/informational purposes and for choosing or skipping classes of
 tests. Keywords must begin with an alphabetic character, `-`, `[` or `{` and
 may actually consist of multiple words separated by spaces which are treated
-together as a single identifier.
+together as a single identifier. Most keywords are only there to provide a way
+for users to skip certain classes of tests, if desired, but a few are treated
+specially by the test harness or build system.
 
 When using curl built with Hyper, the keywords must include `HTTP` or `HTTPS`
 for 'hyper mode' to kick in and make line ending checks work for tests.
@@ -199,6 +201,12 @@ When running a unit test and the keywords include `unittest`, the `<tool>`
 section can be left empty to use the standard unit test tool name `unitN` where
 `N` is the test number.
 
+The `text-ci` make target automatically skips test with the `flaky` keyword.
+
+Tests that have strict timing dependencies have the `timing-dependent` keyword.
+These are intended to eventually be treated specially on CI builds which are
+often run on overloaded machines with unpredictable timing.
+
 ## `<reply>`
 
 ### `<data [nocheck="yes"] [sendzero="yes"] [base64="yes"] [hex="yes"] [nonewline="yes"] [crlf="yes"]>`
diff --git a/tests/data/test1112 b/tests/data/test1112
index eb4d6f02f8..0efcf42ba6 100644
--- a/tests/data/test1112
+++ b/tests/data/test1112
@@ -1,4 +1,5 @@
 <testcase>
+# This is the ftps equivalent of test1086
 <info>
 <keywords>
 FTPS
@@ -7,6 +8,7 @@ RETR
 timeout
 FAILURE
 SLOWDOWN
+timing-dependent
 </keywords>
 </info>
 
diff --git a/tests/data/test1206 b/tests/data/test1206
index b192592700..453b991084 100644
--- a/tests/data/test1206
+++ b/tests/data/test1206
@@ -46,6 +46,7 @@ SIZE %TESTNUMBER
 RETR %TESTNUMBER
 QUIT
 </protocol>
+10 is CURLE_FTP_ACCEPT_FAILED
 <errorcode>
 10
 </errorcode>
diff --git a/tests/data/test1208 b/tests/data/test1208
index 2de619d90d..57937a9672 100644
--- a/tests/data/test1208
+++ b/tests/data/test1208
@@ -5,9 +5,10 @@ FTP
 PORT
 RETR
 NODATACONN150
-flaky
 timeout
 FAILURE
+flaky
+timing-dependent
 </keywords>
 </info>
 # Server-side
@@ -53,6 +54,7 @@ TYPE I
 SIZE %TESTNUMBER
 RETR %TESTNUMBER
 </protocol>
+# 28 is CURLE_OPERATION_TIMEDOUT
 <errorcode>
 28
 </errorcode>
diff --git a/tests/data/test1474 b/tests/data/test1474
index 8ca2cb8cc3..848f15211b 100644
--- a/tests/data/test1474
+++ b/tests/data/test1474
@@ -32,6 +32,7 @@ HTTP
 HTTP PUT
 Expect
 flaky
+timing-dependent
 </keywords>
 </info>
 # Server-side
diff --git a/tests/data/test1592 b/tests/data/test1592
index 8de07341ef..6da96dc492 100644
--- a/tests/data/test1592
+++ b/tests/data/test1592
@@ -6,6 +6,7 @@ multi
 resolve
 speedcheck
 flaky
+timing-dependent
 </keywords>
 </info>
 
diff --git a/tests/data/test2032 b/tests/data/test2032
index 72c7cc4d21..d3d199c14c 100644
--- a/tests/data/test2032
+++ b/tests/data/test2032
@@ -5,8 +5,9 @@ HTTP
 HTTP GET
 HTTP Basic auth
 HTTP NTLM auth
-flaky
 NTLM
+flaky
+timing-dependent
 </keywords>
 </info>
 # Server-side
diff --git a/tests/data/test2600 b/tests/data/test2600
index d6d07256d7..829ea0c6fe 100644
--- a/tests/data/test2600
+++ b/tests/data/test2600
@@ -3,6 +3,7 @@
 <keywords>
 unittest
 cfilter
+timing-dependent
 </keywords>
 </info>