doc/man3/OSSL_HTTP_transfer.pod

   1 =pod
   2
   3 =head1 NAME
   4
   5 OSSL_HTTP_open,
   6 OSSL_HTTP_bio_cb_t,
   7 OSSL_HTTP_proxy_connect,
   8 OSSL_HTTP_set1_request,
   9 OSSL_HTTP_exchange,
  10 OSSL_HTTP_get,
  11 OSSL_HTTP_transfer,
  12 OSSL_HTTP_close
  13 -  HTTP client high-level functions
  14
  15 =head1 SYNOPSIS
  16
  17  #include <openssl/http.h>
  18
  19  typedef BIO *(*OSSL_HTTP_bio_cb_t)(BIO *bio, void *arg,
  20                                     int connect, int detail);
  21  OSSL_HTTP_REQ_CTX *OSSL_HTTP_open(const char *server, const char *port,
  22                                    const char *proxy, const char *no_proxy,
  23                                    int use_ssl, BIO *bio, BIO *rbio,
  24                                    OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  25                                    int buf_size, int overall_timeout);
  26  int OSSL_HTTP_proxy_connect(BIO *bio, const char *server, const char *port,
  27                              const char *proxyuser, const char *proxypass,
  28                              int timeout, BIO *bio_err, const char *prog);
  29  int OSSL_HTTP_set1_request(OSSL_HTTP_REQ_CTX *rctx, const char *path,
  30                             const STACK_OF(CONF_VALUE) *headers,
  31                             const char *content_type, BIO *req,
  32                             const char *expected_content_type, int expect_asn1,
  33                             size_t max_resp_len, int timeout, int keep_alive);
  34  BIO *OSSL_HTTP_exchange(OSSL_HTTP_REQ_CTX *rctx, char **redirection_url);
  35  BIO *OSSL_HTTP_get(const char *url, const char *proxy, const char *no_proxy,
  36                     BIO *bio, BIO *rbio,
  37                     OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  38                     int buf_size, const STACK_OF(CONF_VALUE) *headers,
  39                     const char *expected_content_type, int expect_asn1,
  40                     size_t max_resp_len, int timeout);
  41  BIO *OSSL_HTTP_transfer(OSSL_HTTP_REQ_CTX **prctx,
  42                          const char *server, const char *port,
  43                          const char *path, int use_ssl,
  44                          const char *proxy, const char *no_proxy,
  45                          BIO *bio, BIO *rbio,
  46                          OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  47                          int buf_size, const STACK_OF(CONF_VALUE) *headers,
  48                          const char *content_type, BIO *req,
  49                          const char *expected_content_type, int expect_asn1,
  50                          size_t max_resp_len, int timeout, int keep_alive);
  51  int OSSL_HTTP_close(OSSL_HTTP_REQ_CTX *rctx, int ok);
  52
  53 =head1 DESCRIPTION
  54
  55 OSSL_HTTP_open() initiates an HTTP session using the I<bio> argument if not
  56 NULL, else by connecting to a given I<server> optionally via a I<proxy>.
  57
  58 Typically the OpenSSL build supports sockets and the I<bio> parameter is NULL.
  59 In this case I<rbio> must be NULL as well and the I<server> must be non-NULL.
  60 The function creates a network BIO internally using L<BIO_new_connect(3)>
  61 for connecting to the given server and the optionally given I<port>,
  62 defaulting to 80 for HTTP or 443 for HTTPS.
  63 Then this internal BIO is used for setting up a connection
  64 and for exchanging one or more request and response.
  65
  66 If I<bio> is given and I<rbio> is NULL then this I<bio> is used instead.
  67 If both I<bio> and I<rbio> are given (which may be memory BIOs for instance)
  68 then no explicit connection is set up, but
  69 I<bio> is used for writing requests and I<rbio> for reading responses.
  70 As soon as the client has flushed I<bio> the server must be ready to provide
  71 a response or indicate a waiting condition via I<rbio>.
  72
  73 If I<bio> is given,
  74 it is an error to provide non-NULL I<proxy> or I<no_proxy> arguments,
  75 while I<server> and I<port> arguments may be given to support diagnostic output.
  76 If I<bio> is NULL the optional I<proxy> parameter can be used to set an
  77 HTTP(S) proxy to use (unless overridden by "no_proxy" settings).
  78 If TLS is not used this defaults to the environment variable C<http_proxy>
  79 if set, else C<HTTP_PROXY>.
  80 If I<use_ssl> != 0 it defaults to C<https_proxy> if set, else C<HTTPS_PROXY>.
  81 An empty proxy string C<""> forbids using a proxy.
  82 Otherwise, the format is
  83 C<[http[s]://][userinfo@]host[:port][/path][?query][#fragment]>,
  84 where any userinfo, path, query, and fragment given is ignored.
  85 If the host string is an IPv6 address, it must be enclosed in C<[> and C<]>.
  86 The default proxy port number is 80, or 443 in case "https:" is given.
  87 The HTTP client functions connect via the given proxy unless the I<server>
  88 is found in the optional list I<no_proxy> of proxy hostnames or IP addresses
  89 separated by C<,> and/or whitespace (if not NULL;
  90 default is the environment variable C<no_proxy> if set, else C<NO_PROXY>).
  91 Proxying plain HTTP is supported directly,
  92 while using a proxy for HTTPS connections requires a suitable callback function
  93 such as OSSL_HTTP_proxy_connect(), described below.
  94
  95 If I<use_ssl> is nonzero a TLS connection is requested
  96 and the I<bio_update_fn> parameter must be provided.
  97
  98 The parameter I<bio_update_fn>, which is optional if I<use_ssl> is 0,
  99 may be used to modify the connection BIO used by the HTTP client,
 100 but cannot be used when both I<bio> and I<rbio> are given.
 101 I<bio_update_fn> is a BIO connect/disconnect callback function with prototype
 102
 103  BIO *(*OSSL_HTTP_bio_cb_t)(BIO *bio, void *arg, int connect, int detail)
 104
 105 The callback function may modify the BIO provided in the I<bio> argument,
 106 whereby it may use an optional custom defined argument I<arg>,
 107 which can for instance point to an B<SSL_CTX> structure.
 108 During connection establishment, just after calling BIO_do_connect_retry(), the
 109 callback function is invoked with the I<connect> argument being 1 and
 110 I<detail> being 1 if I<use_ssl> is nonzero (i.e., HTTPS is requested), else 0.
 111 On disconnect I<connect> is 0 and I<detail> is 1 if no error occurred, else 0.
 112 For instance, on connect the callback may push an SSL BIO to implement HTTPS;
 113 after disconnect it may do some diagnostic output and pop and free the SSL BIO.
 114
 115 The callback function must return either the potentially modified BIO I<bio>
 116 or NULL to indicate failure, in which case it should not modify the BIO.
 117
 118 Here is a simple example that supports TLS connections (but not via a proxy):
 119
 120  BIO *http_tls_cb(BIO *bio, void *arg, int connect, int detail)
 121  {
 122      if (connect && detail) { /* connecting with TLS */
 123          SSL_CTX *ctx = (SSL_CTX *)arg;
 124          BIO *sbio = BIO_new_ssl(ctx, 1);
 125
 126          bio = sbio != NULL ? BIO_push(sbio, bio) : NULL;
 127      } else if (!connect) { /* disconnecting */
 128          BIO *hbio;
 129
 130          if (!detail) { /* an error has occurred */
 131              /* optionally add diagnostics here */
 132          }
 133          BIO_ssl_shutdown(bio);
 134          hbio = BIO_pop(bio);
 135          BIO_free(bio); /* SSL BIO */
 136          bio = hbio;
 137      }
 138      return bio;
 139  }
 140
 141 After disconnect the modified BIO will be deallocated using BIO_free_all().
 142 The optional callback function argument I<arg> is not consumed,
 143 so must be freed by the caller when not needed any more.
 144
 145 The I<buf_size> parameter specifies the response header maximum line length.
 146 A value <= 0 means that the B<OSSL_HTTP_DEFAULT_MAX_LINE_LEN> (4KiB) is used.
 147 I<buf_size> is also used as the number of content bytes that are read at a time.
 148
 149 If the I<overall_timeout> parameter is > 0 this indicates the maximum number of
 150 seconds the overall HTTP transfer (i.e., connection setup if needed,
 151 sending requests, and receiving responses) is allowed to take until completion.
 152 A value <= 0 enables waiting indefinitely, i.e., no timeout.
 153
 154 OSSL_HTTP_proxy_connect() may be used by an above BIO connect callback function
 155 to set up an SSL/TLS connection via an HTTPS proxy.
 156 It promotes the given BIO I<bio> representing a connection
 157 pre-established with a TLS proxy using the HTTP CONNECT method,
 158 optionally using proxy client credentials I<proxyuser> and I<proxypass>,
 159 to connect with TLS protection ultimately to I<server> and I<port>.
 160 If the I<port> argument is NULL or the empty string it defaults to "443".
 161 If the I<timeout> parameter is > 0 this indicates the maximum number of
 162 seconds the connection setup is allowed to take.
 163 A value <= 0 enables waiting indefinitely, i.e., no timeout.
 164 Since this function is typically called by applications such as
 165 L<openssl-s_client(1)> it uses the I<bio_err> and I<prog> parameters (unless
 166 NULL) to print additional diagnostic information in a user-oriented way.
 167
 168 OSSL_HTTP_set1_request() sets up in I<rctx> the request header and content data
 169 and expectations on the response using the following parameters.
 170 If <rctx> indicates using a proxy for HTTP (but not HTTPS), the server host
 171 (and optionally port) needs to be placed in the header; thus it must be present
 172 in I<rctx>.
 173 For backward compatibility, the server (and optional port) may also be given in
 174 the I<path> argument beginning with C<http://> (thus giving an absoluteURI).
 175 If I<path> is NULL it defaults to "/".
 176 If I<req> is NULL the HTTP GET method will be used to send the request
 177 else HTTP POST with the contents of I<req> and optional I<content_type>, where
 178 the length of the data in I<req> does not need to be determined in advance: the
 179 BIO will be read on-the-fly while sending the request, which supports streaming.
 180 The optional list I<headers> may contain additional custom HTTP header lines.
 181 The I<max_resp_len> parameter specifies the maximum allowed
 182 response content length, where the value 0 indicates no limit.
 183 For the meaning of the I<expected_content_type>, I<expect_asn1>, I<timeout>,
 184 and I<keep_alive> parameters, see L<OSSL_HTTP_REQ_CTX_set_expected(3)>.
 185
 186 OSSL_HTTP_exchange() exchanges any form of HTTP request and response
 187 as specified by I<rctx>, which must include both connection and request data,
 188 typically set up using OSSL_HTTP_open() and OSSL_HTTP_set1_request().
 189 It implements the core of the functions described below.
 190 If the HTTP method is GET and I<redirection_url>
 191 is not NULL the latter pointer is used to provide any new location that
 192 the server may return with HTTP code 301 (MOVED_PERMANENTLY) or 302 (FOUND).
 193 In this case the function returns NULL and the caller is
 194 responsible for deallocating the URL with L<OPENSSL_free(3)>.
 195 If the response header contains one or more C<Content-Length> lines and/or
 196 an ASN.1-encoded response is expected, which should include a total length,
 197 the length indications received are checked for consistency
 198 and for not exceeding any given maximum response length.
 199 If an ASN.1-encoded response is expected, the function returns on success
 200 the contents buffered in a memory BIO, which does not support streaming.
 201 Otherwise it returns directly the read BIO that holds the response contents,
 202 which allows a response of indefinite length and may support streaming.
 203 The caller is responsible for freeing the BIO pointer obtained.
 204
 205 OSSL_HTTP_get() uses HTTP GET to obtain data from I<bio> if non-NULL,
 206 else from the server contained in the I<url>, and returns it as a BIO.
 207 It supports redirection via HTTP status code 301 or 302.  It is meant for
 208 transfers with a single round trip, so does not support persistent connections.
 209 If I<bio> is non-NULL, any host and port components in the I<url> are not used
 210 for connecting but the hostname is used, as usual, for the C<Host> header.
 211 Any userinfo and fragment components in the I<url> are ignored.
 212 Any query component is handled as part of the path component.
 213 If the scheme component of the I<url> is C<https> a TLS connection is requested
 214 and the I<bio_update_fn>, as described for OSSL_HTTP_open(), must be provided.
 215 Also the remaining parameters are interpreted as described for OSSL_HTTP_open()
 216 and OSSL_HTTP_set1_request(), respectively.
 217 The caller is responsible for freeing the BIO pointer obtained.
 218
 219 OSSL_HTTP_transfer() exchanges an HTTP request and response
 220 over a connection managed via I<prctx> without supporting redirection.
 221 It combines OSSL_HTTP_open(), OSSL_HTTP_set1_request(), OSSL_HTTP_exchange(),
 222 and OSSL_HTTP_close().
 223 If I<prctx> is not NULL it reuses any open connection represented by a non-NULL
 224 I<*prctx>.  It keeps the connection open if a persistent connection is requested
 225 or required and this was granted by the server, else it closes the connection
 226 and assigns NULL to I<*prctx>.
 227 The remaining parameters are interpreted as described for OSSL_HTTP_open()
 228 and OSSL_HTTP_set1_request(), respectively.
 229 The caller is responsible for freeing the BIO pointer obtained.
 230
 231 OSSL_HTTP_close() closes the connection and releases I<rctx>.
 232 The I<ok> parameter is passed to any BIO update function
 233 given during setup as described above for OSSL_HTTP_open().
 234 It must be 1 if no error occurred during the HTTP transfer and 0 otherwise.
 235
 236 =head1 NOTES
 237
 238 The names of the environment variables used by this implementation:
 239 C<http_proxy>, C<HTTP_PROXY>, C<https_proxy>, C<HTTPS_PROXY>, C<no_proxy>, and
 240 C<NO_PROXY>, have been chosen for maximal compatibility with
 241 other HTTP client implementations such as wget, curl, and git.
 242
 243 When built with tracing enabled, OSSL_HTTP_transfer() and all functions using it
 244 may be traced using B<OSSL_TRACE_CATEGORY_HTTP>.
 245 See also L<OSSL_trace_enabled(3)> and L<openssl-env(7)>.
 246
 247 =head1 RETURN VALUES
 248
 249 OSSL_HTTP_open() returns on success a B<OSSL_HTTP_REQ_CTX>, else NULL.
 250
 251 OSSL_HTTP_proxy_connect() and OSSL_HTTP_set1_request()
 252 return 1 on success, 0 on error.
 253
 254 On success, OSSL_HTTP_exchange(), OSSL_HTTP_get(), and OSSL_HTTP_transfer()
 255 return a memory BIO that buffers all the data received if an ASN.1-encoded
 256 response is expected, otherwise a BIO that may support streaming.
 257 The BIO must be freed by the caller.
 258 On failure, they return NULL.
 259 Failure conditions include connection/transfer timeout, parse errors, etc.
 260 The caller is responsible for freeing the BIO pointer obtained.
 261
 262 OSSL_HTTP_close() returns 0 if anything went wrong while disconnecting, else 1.
 263
 264 =head1 SEE ALSO
 265
 266 L<OSSL_HTTP_parse_url(3)>, L<BIO_new_connect(3)>,
 267 L<ASN1_item_i2d_mem_bio(3)>, L<ASN1_item_d2i_bio(3)>,
 268 L<OSSL_HTTP_REQ_CTX_set_expected(3)>,
 269 L<OSSL_HTTP_is_alive(3)>,
 270 L<OSSL_trace_enabled(3)>, and L<openssl-env(7)>.
 271
 272 =head1 HISTORY
 273
 274 All the functions described here were added in OpenSSL 3.0.
 275
 276 =head1 COPYRIGHT
 277
 278 Copyright 2019-2025 The OpenSSL Project Authors. All Rights Reserved.
 279
 280 Licensed under the Apache License 2.0 (the "License").  You may not use
 281 this file except in compliance with the License.  You can obtain a copy
 282 in the file LICENSE in the source distribution or at
 283 L<https://www.openssl.org/source/license.html>.
 284
 285 =cut