HTTP: Warn if a download exceeds 50% of the file size limit

Requested by Ties de Kock:

> Since some RPs now includes an upper limit on object size (some use
> 2MB if I recall correctly) I would appreciate a warning if an object
> goes over "a large fraction" of this limit (and a sample of the
> warning in the changelog and metrics if possible) - so people know
> what they need to alert on. In this situation operators can monitor
> for "natural growth" of an object and intervene, while the case that
> this check prevents (maliciously large objects) is still covered.
>
> The largest object I could find in the wild is 1.2MB (APNIC AS0 ROA).
> The RIPE NCC's largest object is smaller at the moment (but the CRL
> grows quickly if we do member CA keyrolls - since it adds all object
> on it).
>
> In summary:
>
> - I would recommend a warning (and preferably a metric) when an object
>   of 50% of the object size limit is encountered.
> - I would like it if the hard limit is "safe" - especially CRLs can
>   grow in some cases.

The metric will be added later, as part of #50. The warning is eg.

	File size exceeds 50% of the configured limit (10/20 bytes).

50% is hardcoded at the moment.

Notice that this is an HTTP-only patch. rsync does not warn.

diff --git a/src/http/http.c b/src/http/http.c
index 035678bab8c791072a11134924cd632ead13e2ee..87312e370fc57aff417fbe3efb31d3c029d6ca96 100644 (file)
--- a/src/http/http.c
+++ b/src/http/http.c
@@ -172,6 +172,26 @@ curl_err_string(struct http_handler *handler, CURLcode res)
            handler->errbuf : curl_easy_strerror(res);
 }
 
+static int
+validate_file_size(char const *uri, struct write_callback_arg *args)
+{
+       float ratio;
+
+       if (args->error == -EFBIG) {
+               pr_val_err("File too big (read: %zu bytes). Rejecting.",
+                   args->total_bytes);
+               return -EFBIG;
+       }
+
+       ratio = args->total_bytes / (float) config_get_http_max_file_size();
+       if (ratio > 0.5f) {
+               pr_op_warn("File size exceeds 50%% of the configured limit (%zu/%ld bytes).",
+                   args->total_bytes, config_get_http_max_file_size());
+       }
+
+       return 0;
+}
+
 static int
 get_http_response_code(struct http_handler *handler, long *http_code,
     char const *uri)
@@ -224,11 +244,9 @@ http_fetch(struct http_handler *handler, char const *uri, long *response_code,
        res = curl_easy_perform(handler->curl);
        pr_val_debug("Done. Total bytes transferred: %zu", args.total_bytes);
 
-       if (args.error == -EFBIG) {
-               pr_val_err("The file '%s' is too big (read: %zu bytes). Rejecting.",
-                   uri, args.total_bytes);
-               return -EFBIG;
-       }
+       args.error = validate_file_size(uri, &args);
+       if (args.error)
+               return args.error;
 
        args.error = get_http_response_code(handler, &http_code, uri);
        if (args.error)