fuzzit: make the submit phase a bit more robust

The submit phase of the Fuzzit Travis job has been spuriously failing
for some time with various (and usually pretty hidden) errors, like:

```
./fuzzit create job --type regression ...
2020/04/23 17:02:12 please set env variable FUZZIT_API_KEY or pass --api-key. API Key for you account: ...
```

```
./fuzzit create job --type regression ...
2020/04/23 11:36:53 Creating job...
2020/04/23 11:36:54 Uploading fuzzer...
2020/04/23 11:36:54 Job created successfully
2020/04/23 11:36:54 Get https://...&action=create: read tcp x.x.x.x:39674->x.x.x.x:443: read: connection reset by peer
```

```
./fuzzit create job --type regression ...
2020/04/22 18:09:15 Creating job...
2020/04/22 18:09:16 Uploading fuzzer...
2020/04/22 18:09:37 Job created successfully
2020/04/22 18:09:37 500 Internal Server Error
```

etc.

Let's retry each submit job up to three times to (hopefully) mitigate this.

diff --git a/travis-ci/managers/fuzzit.sh b/travis-ci/managers/fuzzit.sh
index 3e78b2ef9abdc0f5ce4680ceeb4ac844551093c0..a4bb5b143fa67b6f843c0a02e9dd0c0c69825f9e 100755 (executable)
--- a/travis-ci/managers/fuzzit.sh
+++ b/travis-ci/managers/fuzzit.sh
@@ -48,10 +48,30 @@ FUZZIT_ARGS="--type ${FUZZING_TYPE} --branch ${FUZZIT_BRANCH} --revision ${TRAVI
 wget -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/latest/download/fuzzit_Linux_x86_64
 chmod +x fuzzit
 
-find out/ -maxdepth 1 -name 'fuzz-*' -executable -type f -exec basename '{}' \; | xargs --verbose -n1 -I%FUZZER% ./fuzzit create job ${FUZZIT_ARGS} %FUZZER%-asan-ubsan out/%FUZZER% ${FUZZIT_ADDITIONAL_FILES}
+# Simple wrapper which retries given command up to three times if it fails
+_retry() {
+    local EC=1
+
+    for _ in {0..2}; do
+        if "$@"; then
+            EC=0
+            break
+        fi
+
+        sleep 1
+    done
+
+    return $EC
+}
+
+find out/ -maxdepth 1 -name 'fuzz-*' -executable -type f -exec basename '{}' \; | while read -r fuzzer; do
+    _retry ./fuzzit create job ${FUZZIT_ARGS} ${fuzzer}-asan-ubsan out/${fuzzer} ${FUZZIT_ADDITIONAL_FILES}
+done
 
 export SANITIZER="memory -fsanitize-memory-track-origins"
 FUZZIT_ARGS="--type ${FUZZING_TYPE} --branch ${FUZZIT_BRANCH} --revision ${TRAVIS_COMMIT}"
 tools/oss-fuzz.sh
 
-find out/ -maxdepth 1 -name 'fuzz-*' -executable -type f -exec basename '{}' \; | xargs --verbose -n1 -I%FUZZER% ./fuzzit create job ${FUZZIT_ARGS} %FUZZER%-msan out/%FUZZER% ${FUZZIT_ADDITIONAL_FILES}
+find out/ -maxdepth 1 -name 'fuzz-*' -executable -type f -exec basename '{}' \; | while read -r fuzzer; do
+    _retry ./fuzzit create job ${FUZZIT_ARGS} ${fuzzer}-msan out/${fuzzer} ${FUZZIT_ADDITIONAL_FILES}
+done