doc/man3/OSSL_HTTP_transfer.pod

   1 =pod
   2
   3 =head1 NAME
   4
   5 OSSL_HTTP_open,
   6 OSSL_HTTP_bio_cb_t,
   7 OSSL_HTTP_proxy_connect,
   8 OSSL_HTTP_set1_request,
   9 OSSL_HTTP_exchange,
  10 OSSL_HTTP_get,
  11 OSSL_HTTP_transfer,
  12 OSSL_HTTP_close
  13 -  HTTP client high-level functions
  14
  15 =head1 SYNOPSIS
  16
  17  #include <openssl/http.h>
  18
  19  typedef BIO *(*OSSL_HTTP_bio_cb_t)(BIO *bio, void *arg,
  20                                     int connect, int detail);
  21  OSSL_HTTP_REQ_CTX *OSSL_HTTP_open(const char *server, const char *port,
  22                                    const char *proxy, const char *no_proxy,
  23                                    int use_ssl, BIO *bio, BIO *rbio,
  24                                    OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  25                                    int buf_size, int overall_timeout);
  26  int OSSL_HTTP_proxy_connect(BIO *bio, const char *server, const char *port,
  27                              const char *proxyuser, const char *proxypass,
  28                              int timeout, BIO *bio_err, const char *prog);
  29  int OSSL_HTTP_set1_request(OSSL_HTTP_REQ_CTX *rctx, const char *path,
  30                             const STACK_OF(CONF_VALUE) *headers,
  31                             const char *content_type, BIO *req,
  32                             const char *expected_content_type, int expect_asn1,
  33                             size_t max_resp_len, int timeout, int keep_alive);
  34  BIO *OSSL_HTTP_exchange(OSSL_HTTP_REQ_CTX *rctx, char **redirection_url);
  35  BIO *OSSL_HTTP_get(const char *url, const char *proxy, const char *no_proxy,
  36                     BIO *bio, BIO *rbio,
  37                     OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  38                     int buf_size, const STACK_OF(CONF_VALUE) *headers,
  39                     const char *expected_content_type, int expect_asn1,
  40                     size_t max_resp_len, int timeout);
  41  BIO *OSSL_HTTP_transfer(OSSL_HTTP_REQ_CTX **prctx,
  42                          const char *server, const char *port,
  43                          const char *path, int use_ssl,
  44                          const char *proxy, const char *no_proxy,
  45                          BIO *bio, BIO *rbio,
  46                          OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  47                          int buf_size, const STACK_OF(CONF_VALUE) *headers,
  48                          const char *content_type, BIO *req,
  49                          const char *expected_content_type, int expect_asn1,
  50                          size_t max_resp_len, int timeout, int keep_alive);
  51  int OSSL_HTTP_close(OSSL_HTTP_REQ_CTX *rctx, int ok);
  52
  53 =head1 DESCRIPTION
  54
  55 OSSL_HTTP_open() initiates an HTTP session using the I<bio> argument if not
  56 NULL, else by connecting to a given I<server> optionally via a I<proxy>.
  57
  58 Typically the OpenSSL build supports sockets and the I<bio> parameter is NULL.
  59 In this case I<rbio> must be NULL as well, and the
  60 library creates a network BIO internally for connecting to the given I<server>
  61 and the optionally given I<port>, defaulting to 80 for HTTP or 443 for HTTPS.
  62 Then this internal BIO is used for setting up a connection
  63 and for exchanging one or more request and response.
  64 If I<bio> is given and I<rbio> is NULL then this I<bio> is used instead.
  65 If both I<bio> and I<rbio> are given (which may be memory BIOs for instance)
  66 then no explicit connection is set up, but
  67 I<bio> is used for writing requests and I<rbio> for reading responses.
  68 As soon as the client has flushed I<bio> the server must be ready to provide
  69 a response or indicate a waiting condition via I<rbio>.
  70
  71 If I<bio> is given, it is an error to provide I<proxy> or I<no_proxy> arguments,
  72 while I<server> and I<port> arguments may be given to support diagnostic output.
  73 If I<bio> is NULL the optional I<proxy> parameter can be used to set an
  74 HTTP(S) proxy to use (unless overridden by "no_proxy" settings).
  75 If TLS is not used this defaults to the environment variable C<http_proxy>
  76 if set, else C<HTTP_PROXY>.
  77 If I<use_ssl> != 0 it defaults to C<https_proxy> if set, else C<HTTPS_PROXY>.
  78 An empty proxy string C<""> forbids using a proxy.
  79 Else the format is
  80 C<[http[s]://][userinfo@]host[:port][/path][?query][#fragment]>,
  81 where any userinfo, path, query, and fragment given is ignored.
  82 The default proxy port number is 80, or 443 in case "https:" is given.
  83 The HTTP client functions connect via the given proxy unless the I<server>
  84 is found in the optional list I<no_proxy> of proxy hostnames (if not NULL;
  85 default is the environment variable C<no_proxy> if set, else C<NO_PROXY>).
  86 Proxying plain HTTP is supported directly,
  87 while using a proxy for HTTPS connections requires a suitable callback function
  88 such as OSSL_HTTP_proxy_connect(), described below.
  89
  90 If I<use_ssl> is nonzero a TLS connection is requested
  91 and the I<bio_update_fn> parameter must be provided.
  92
  93 The parameter I<bio_update_fn>, which is optional if I<use_ssl> is 0,
  94 may be used to modify the connection BIO used by the HTTP client,
  95 but cannot be used when both I<bio> and I<rbio> are given.
  96 I<bio_update_fn> is a BIO connect/disconnect callback function with prototype
  97
  98  BIO *(*OSSL_HTTP_bio_cb_t)(BIO *bio, void *arg, int connect, int detail)
  99
 100 The callback function may modify the BIO provided in the I<bio> argument,
 101 whereby it may make use of a custom defined argument I<arg>,
 102 which may for instance point to an B<SSL_CTX> structure.
 103 During connection establishment, just after calling BIO_do_connect_retry(), the
 104 callback function is invoked with the I<connect> argument being 1 and the I<detail>
 105 argument being 1 if HTTPS is requested, i.e., SSL/TLS should be enabled, else 0.
 106 On disconnect I<connect> is 0 and I<detail> is 1 if no error occurred, else 0.
 107 For instance, on connect the callback may push an SSL BIO to implement HTTPS;
 108 after disconnect it may do some diagnostic output and pop and free the SSL BIO.
 109
 110 The callback function must return either the potentially modified BIO I<bio>.
 111 or NULL to indicate failure, in which case it should not modify the BIO.
 112
 113 Here is a simple example that supports TLS connections (but not via a proxy):
 114
 115  BIO *http_tls_cb(BIO *hbio, void *arg, int connect, int detail)
 116  {
 117      if (connect && detail) { /* connecting with TLS */
 118          SSL_CTX *ctx = (SSL_CTX *)arg;
 119          BIO *sbio = BIO_new_ssl(ctx, 1);
 120
 121          hbio = sbio != NULL ? BIO_push(sbio, hbio) : NULL;
 122      } else if (!connect && !detail) { /* disconnecting after error */
 123          /* optionally add diagnostics here */
 124      }
 125      return hbio;
 126  }
 127
 128 After disconnect the modified BIO will be deallocated using BIO_free_all().
 129
 130 The I<buf_size> parameter specifies the response header maximum line length.
 131 A value <= 0 means that the B<OSSL_HTTP_DEFAULT_MAX_LINE_LEN> (4KiB) is used.
 132 I<buf_size> is also used as the number of content bytes that are read at a time.
 133
 134 If the I<overall_timeout> parameter is > 0 this indicates the maximum number of
 135 seconds the overall HTTP transfer (i.e., connection setup if needed,
 136 sending requests, and receiving responses) is allowed to take until completion.
 137 A value <= 0 enables waiting indefinitely, i.e., no timeout.
 138
 139 OSSL_HTTP_proxy_connect() may be used by an above BIO connect callback function
 140 to set up an SSL/TLS connection via an HTTPS proxy.
 141 It promotes the given BIO I<bio> representing a connection
 142 pre-established with a TLS proxy using the HTTP CONNECT method,
 143 optionally using proxy client credentials I<proxyuser> and I<proxypass>,
 144 to connect with TLS protection ultimately to I<server> and I<port>.
 145 If the I<port> argument is NULL or the empty string it defaults to "443".
 146 If the I<timeout> parameter is > 0 this indicates the maximum number of
 147 seconds the connection setup is allowed to take.
 148 A value <= 0 enables waiting indefinitely, i.e., no timeout.
 149 Since this function is typically called by applications such as
 150 L<openssl-s_client(1)> it uses the I<bio_err> and I<prog> parameters (unless
 151 NULL) to print additional diagnostic information in a user-oriented way.
 152
 153 OSSL_HTTP_set1_request() sets up in I<rctx> the request header and content data
 154 and expectations on the response using the following parameters.
 155 If <rctx> indicates using a proxy for HTTP (but not HTTPS), the server hostname
 156 (and optionally port) needs to be placed in the header and thus must be present.
 157 If I<path> is NULL it defaults to "/".
 158 If I<req> is NULL the HTTP GET method will be used to send the request
 159 else HTTP POST with the contents of I<req> and optional I<content_type>, where
 160 the length of the data in I<req> does not need to be determined in advance: the
 161 BIO will be read on-the-fly while sending the request, which supports streaming.
 162 The optional list I<headers> may contain additional custom HTTP header lines.
 163 If the parameter I<expected_content_type>
 164 is not NULL then the client will check that the given content type string
 165 is included in the HTTP header of the response and return an error if not.
 166 If the I<expect_asn1> parameter is nonzero,
 167 a structure in ASN.1 encoding will be expected as response content.
 168 The I<max_resp_len> parameter specifies the maximum allowed
 169 response content length, where the value 0 indicates no limit.
 170 If the I<timeout> parameter is > 0 this indicates the maximum number of seconds
 171 the subsequent HTTP transfer (sending the request and receiving a response)
 172 is allowed to take.
 173 A value of 0 enables waiting indefinitely, i.e., no timeout.
 174 A value < 0 indicates that the I<overall_timeout> parameter value given
 175 when opening the HTTP transfer will be used instead.
 176 If I<keep_alive> is 0 the connection is not kept open
 177 after receiving a response, which is the default behavior for HTTP 1.0.
 178 If the value is 1 or 2 then a persistent connection is requested.
 179 If the value is 2 then a persistent connection is required,
 180 i.e., an error occurs in case the server does not grant it.
 181
 182 OSSL_HTTP_exchange() exchanges any form of HTTP request and response
 183 as specified by I<rctx>, which must include both connection and request data,
 184 typically set up using OSSL_HTTP_open() and OSSL_HTTP_set1_request().
 185 It implements the core of the functions described below.
 186 If the HTTP method is GET and I<redirection_url>
 187 is not NULL the latter pointer is used to provide any new location that
 188 the server may return with HTTP code 301 (MOVED_PERMANENTLY) or 302 (FOUND).
 189 In this case the function returns NULL and the caller is
 190 responsible for deallocating the URL with L<OPENSSL_free(3)>.
 191 If the response header contains one or more "Content-Length" header lines and/or
 192 an ASN.1-encoded response is expected, which should include a total length,
 193 the length indications received are checked for consistency
 194 and for not exceeding any given maximum response length.
 195 If an ASN.1-encoded response is expected, the function returns on success
 196 the contents as a memory BIO, which does not support streaming.
 197 Otherwise it returns directly the read BIO that holds the response contents,
 198 which allows a response of indefinite length and may support streaming.
 199 The caller is responsible for freeing the BIO pointer obtained.
 200
 201 OSSL_HTTP_get() uses HTTP GET to obtain data from I<bio> if non-NULL,
 202 else from the server contained in the I<url>, and returns it as a BIO.
 203 It supports redirection via HTTP status code 301 or 302.  It is meant for
 204 transfers with a single round trip, so does not support persistent connections.
 205 If I<bio> is non-NULL, any host and port components in the I<url> are not used
 206 for connecting but the hostname is used, as usual, for the C<Host> header.
 207 Any userinfo and fragment components in the I<url> are ignored.
 208 Any query component is handled as part of the path component.
 209 If the scheme component of the I<url> is C<https> a TLS connection is requested
 210 and the I<bio_update_fn>, as described for OSSL_HTTP_open(), must be provided.
 211 Also the remaining parameters are interpreted as described for OSSL_HTTP_open()
 212 and OSSL_HTTP_set1_request(), respectively.
 213 The caller is responsible for freeing the BIO pointer obtained.
 214
 215 OSSL_HTTP_transfer() exchanges an HTTP request and response
 216 over a connection managed via I<prctx> without supporting redirection.
 217 It combines OSSL_HTTP_open(), OSSL_HTTP_set1_request(), OSSL_HTTP_exchange(),
 218 and OSSL_HTTP_close().
 219 If I<prctx> is not NULL it reuses any open connection represented by a non-NULL
 220 I<*prctx>.  It keeps the connection open if a persistent connection is requested
 221 or required and this was granted by the server, else it closes the connection
 222 and assigns NULL to I<*prctx>.
 223 The remaining parameters are interpreted as described for OSSL_HTTP_open()
 224 and OSSL_HTTP_set1_request(), respectively.
 225 The caller is responsible for freeing the BIO pointer obtained.
 226
 227 OSSL_HTTP_close() closes the connection and releases I<rctx>.
 228 The I<ok> parameter is passed to any BIO update function
 229 given during setup as described above for OSSL_HTTP_open().
 230 It must be 1 if no error occurred during the HTTP transfer and 0 otherwise.
 231
 232 =head1 NOTES
 233
 234 The names of the environment variables used by this implementation:
 235 C<http_proxy>, C<HTTP_PROXY>, C<https_proxy>, C<HTTPS_PROXY>, C<no_proxy>, and
 236 C<NO_PROXY>, have been chosen for maximal compatibility with
 237 other HTTP client implementations such as wget, curl, and git.
 238
 239 =head1 RETURN VALUES
 240
 241 OSSL_HTTP_open() returns on success a B<OSSL_HTTP_REQ_CTX>, else NULL.
 242
 243 OSSL_HTTP_proxy_connect() and OSSL_HTTP_set1_request()
 244 return 1 on success, 0 on error.
 245
 246 On success, OSSL_HTTP_exchange(), OSSL_HTTP_get(), and OSSL_HTTP_transfer()
 247 return a memory BIO containing the data received if an ASN.1-encoded response
 248 is expected, else a BIO that may support streaming.
 249 The BIO must be freed by the caller.
 250 On failure, they return NULL.
 251 Failure conditions include connection/transfer timeout, parse errors, etc.
 252 The caller is responsible for freeing the BIO pointer obtained.
 253
 254 OSSL_HTTP_close() returns 0 if anything went wrong while disconnecting, else 1.
 255
 256 =head1 SEE ALSO
 257
 258 L<OSSL_HTTP_parse_url(3)>, L<BIO_set_conn_port(3)>
 259 L<ASN1_item_i2d_mem_bio(3)>, L<ASN1_item_d2i_bio(3)>,
 260 L<OSSL_HTTP_is_alive(3)>
 261
 262 =head1 HISTORY
 263
 264 All the functions described here were added in OpenSSL 3.0.
 265
 266 =head1 COPYRIGHT
 267
 268 Copyright 2019-2021 The OpenSSL Project Authors. All Rights Reserved.
 269
 270 Licensed under the Apache License 2.0 (the "License").  You may not use
 271 this file except in compliance with the License.  You can obtain a copy
 272 in the file LICENSE in the source distribution or at
 273 L<https://www.openssl.org/source/license.html>.
 274
 275 =cut