doc/man3/OSSL_HTTP_transfer.pod

   1 =pod
   2
   3 =head1 NAME
   4
   5 OSSL_HTTP_open,
   6 OSSL_HTTP_bio_cb_t,
   7 OSSL_HTTP_proxy_connect,
   8 OSSL_HTTP_set_request,
   9 OSSL_HTTP_exchange,
  10 OSSL_HTTP_get,
  11 OSSL_HTTP_transfer,
  12 OSSL_HTTP_close
  13 -  HTTP client high-level functions
  14
  15 =head1 SYNOPSIS
  16
  17  #include <openssl/http.h>
  18
  19  typedef BIO *(*OSSL_HTTP_bio_cb_t)(BIO *bio, void *arg,
  20                                     int connect, int detail);
  21  OSSL_HTTP_REQ_CTX *OSSL_HTTP_open(const char *server, const char *port,
  22                                    const char *proxy, const char *no_proxy,
  23                                    int use_ssl, BIO *bio, BIO *rbio,
  24                                    OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  25                                    int buf_size, int overall_timeout);
  26  int OSSL_HTTP_proxy_connect(BIO *bio, const char *server, const char *port,
  27                              const char *proxyuser, const char *proxypass,
  28                              int timeout, BIO *bio_err, const char *prog);
  29  int OSSL_HTTP_set_request(OSSL_HTTP_REQ_CTX *rctx, const char *path,
  30                            const STACK_OF(CONF_VALUE) *headers,
  31                            const char *content_type, BIO *req,
  32                            const char *expected_content_type, int expect_asn1,
  33                            size_t max_resp_len, int timeout, int keep_alive);
  34  BIO *OSSL_HTTP_exchange(OSSL_HTTP_REQ_CTX *rctx, char **redirection_url);
  35  BIO *OSSL_HTTP_get(const char *url, const char *proxy, const char *no_proxy,
  36                     BIO *bio, BIO *rbio,
  37                     OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  38                     int buf_size, const STACK_OF(CONF_VALUE) *headers,
  39                     const char *expected_content_type, int expect_asn1,
  40                     size_t max_resp_len, int timeout);
  41  BIO *OSSL_HTTP_transfer(OSSL_HTTP_REQ_CTX **prctx,
  42                          const char *server, const char *port,
  43                          const char *path, int use_ssl,
  44                          const char *proxy, const char *no_proxy,
  45                          BIO *bio, BIO *rbio,
  46                          OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
  47                          int buf_size, const STACK_OF(CONF_VALUE) *headers,
  48                          const char *content_type, BIO *req,
  49                          const char *expected_content_type, int expect_asn1,
  50                          size_t max_resp_len, int timeout, int keep_alive);
  51  int OSSL_HTTP_close(OSSL_HTTP_REQ_CTX *rctx, int ok);
  52
  53 =head1 DESCRIPTION
  54
  55 OSSL_HTTP_open() initiates an HTTP session using I<bio> if not NULL, else
  56 by connecting to a given I<server> optionally via a I<proxy>.
  57
  58 Typically the OpenSSL build supports sockets and the I<bio> parameter is NULL.
  59 In this case I<rbio> must be NULL as well, and the
  60 library creates a network BIO internally for connecting to the given I<server>
  61 at the specified I<port> if any, defaulting to 80 for HTTP or 443 for HTTPS.
  62 Then this internal BIO is used for setting up a connection
  63 and for exchanging one or more request and response.
  64 If I<bio> is given and I<rbio> is NULL then this I<bio> is used instead.
  65 If both I<bio> and I<rbio> are given (which may be memory BIOs for instance)
  66 then no explicit connection is set up, but
  67 I<bio> is used for writing requests and I<rbio> for reading responses.
  68 As soon as the client has flushed I<bio> the server must be ready to provide
  69 a response or indicate a waiting condition via I<rbio>.
  70
  71 If I<bio> is NULL the optional I<proxy> parameter can be used to set an
  72 HTTP(S) proxy to use (unless overridden by "no_proxy" settings).
  73 If TLS is not used this defaults to the environment variable C<http_proxy>
  74 if set, else C<HTTP_PROXY>.
  75 If I<use_ssl> != 0 it defaults to C<https_proxy> if set, else C<HTTPS_PROXY>.
  76 An empty proxy string C<""> forbids using a proxy.
  77 Else the format is
  78 C<[http[s]://][userinfo@]host[:port][/path][?query][#fragment]>,
  79 where any userinfo, path, query, and fragment given is ignored.
  80 The default proxy port number is 80, or 443 in case "https:" is given.
  81 The HTTP client functions connect via the given proxy unless the I<server>
  82 is found in the optional list I<no_proxy> of proxy hostnames (if not NULL;
  83 default is the environment variable C<no_proxy> if set, else C<NO_PROXY>).
  84 Proxying plain HTTP is supported directly,
  85 while using a proxy for HTTPS connections requires a suitable callback function
  86 such as OSSL_HTTP_proxy_connect(), described below.
  87
  88 If I<use_ssl> is nonzero a TLS connection is requested
  89 and the I<bio_update_fn> parameter must be provided.
  90
  91 The parameter I<bio_update_fn>, which is optional if I<use_ssl> is 0,
  92 may be used to modify the connection BIO used by the HTTP client,
  93 but cannot be used when both I<bio> and I<rbio> are given.
  94 I<bio_update_fn> is a BIO connect/disconnect callback function with prototype
  95
  96  BIO *(*OSSL_HTTP_bio_cb_t)(BIO *bio, void *arg, int connect, int detail)
  97
  98 The callback may modify the HTTP BIO provided in the I<bio> argument,
  99 whereby it may make use of a custom defined argument I<arg>,
 100 which may for instance refer to an I<SSL_CTX> structure.
 101 During connection establishment, just after calling BIO_do_connect_retry(),
 102 the function is invoked with the I<connect> argument being 1 and the I<detail>
 103 argument being 1 if HTTPS is requested, i.e., SSL/TLS should be enabled, else 0.
 104 On disconnect I<connect> is 0 and I<detail> is 1 if no error occurred, else 0.
 105 For instance, on connect the function may prepend a TLS BIO to implement HTTPS;
 106 after disconnect it may do some diagnostic output and/or specific cleanup.
 107 The function should return NULL to indicate failure.
 108 Here is a simple example that supports TLS connections (but not via a proxy):
 109
 110  BIO *http_tls_cb(BIO *hbio, void *arg, int connect, int detail)
 111  {
 112      if (connect && detail) { /* connecting with TLS */
 113          SSL_CTX *ctx = (SSL_CTX *)arg;
 114          BIO *sbio = BIO_new_ssl(ctx, 1);
 115
 116          hbio = sbio != NULL ? BIO_push(sbio, hbio) : NULL;
 117      } else if (!connect && !detail) { /* disconnecting after error */
 118          /* optionally add diagnostics here */
 119      }
 120      return hbio;
 121  }
 122
 123 After disconnect the modified BIO will be deallocated using BIO_free_all().
 124
 125 The I<buf_size> parameter specifies the response header maximum line length.
 126 A value <= 0 indicates that
 127 the B<HTTP_DEFAULT_MAX_LINE_LENGTH> of 4KiB should be used.
 128 This length is also used as the number of content bytes that are read at a time.
 129 The I<max_resp_len> specifies the maximum allowed response content length.
 130 The value 0 indicates B<HTTP_DEFAULT_MAX_RESP_LEN>, which currently is 100 KiB.
 131
 132 If the I<overall_timeout> parameter is > 0 this indicates the maximum number of
 133 seconds the overall HTTP transfer (i.e., connection setup if needed,
 134 sending requests, and receiving responses) is allowed to take until completion.
 135 A value <= 0 enables waiting indefinitely, i.e., no timeout.
 136
 137 OSSL_HTTP_proxy_connect() may be used by an above BIO connect callback function
 138 to set up an SSL/TLS connection via an HTTPS proxy.
 139 It promotes the given BIO I<bio> representing a connection
 140 pre-established with a TLS proxy using the HTTP CONNECT method,
 141 optionally using proxy client credentials I<proxyuser> and I<proxypass>,
 142 to connect with TLS protection ultimately to I<server> and I<port>.
 143 If the I<port> argument is NULL or the empty string it defaults to "443".
 144 If the I<timeout> parameter is > 0 this indicates the maximum number of
 145 seconds the connection setup is allowed to take.
 146 A value <= 0 enables waiting indefinitely, i.e., no timeout.
 147 Since this function is typically called by applications such as
 148 L<openssl-s_client(1)> it uses the I<bio_err> and I<prog> parameters (unless
 149 NULL) to print additional diagnostic information in a user-oriented way.
 150
 151
 152 OSSL_HTTP_set_request() sets up in I<rctx> the request header and content data
 153 and expectations on the response using the following parameters.
 154 If I<path> is NULL it defaults to "/".
 155 If I<req_mem> is NULL the HTTP GET method will be used to send the request
 156 else HTTP POST with the contents of I<req_mem> and optional I<content_type>.
 157 The optional list I<headers> may contain additional custom HTTP header lines.
 158 If the parameter I<expected_content_type>
 159 is not NULL then the client will check that the given content type string
 160 is included in the HTTP header of the response and return an error if not.
 161 If the I<expect_asn1> parameter is nonzero,
 162 a structure in ASN.1 encoding will be expected as response content.
 163 If the I<timeout> parameter is > 0 this indicates the maximum number of seconds
 164 the subsequent HTTP transfer (sending the request and receiving a response)
 165 is allowed to take.
 166 A value of 0 enables waiting indefinitely, i.e., no timeout.
 167 A value < 0 indicates that the I<overall_timeout> parameter value given
 168 when opening the HTTP transfer will be used instead.
 169 If I<keep_alive> is 0 the connection is not kept open
 170 after receiving a response, which is the default behavior for HTTP 1.0.
 171 If the value is 1 or 2 then a persistent connection is requested.
 172 If the value is 2 then a persistent connection is required,
 173 i.e., an error occurs in case the server does not grant it.
 174
 175 OSSL_HTTP_transfer() exchanges any form of HTTP request and response
 176 as specified by I<rctx>, which must include both connection and request data,
 177 typically set up using OSSL_HTTP_open() and OSSL_HTTP_set_request().
 178 It implements the core of the functions described below.
 179 If the HTTP method is GET and I<redirection_url>
 180 is not NULL the latter pointer is used to provide any new location that
 181 the server may return with HTTP code 301 (MOVED_PERMANENTLY) or 302 (FOUND).
 182 In this case the caller is responsible for deallocating this URL with
 183 L<OPENSSL_free(3)>.
 184 If the response header contains one or more "Content-Length" header lines and/or
 185 an ASN.1-encoded response is expected, which should include a total length,
 186 the length indications received are checked for consistency
 187 and for not exceeding any given maximum response length.
 188 On receiving a response, the function returns the contents as a memory BIO,
 189 which does not support streaming, in case an ASN.1-encoded response is expected.
 190 Else it returns directly the read BIO that holds the response contents,
 191 which allows a response of indefinite length and may support streaming.
 192
 193 OSSL_HTTP_get() uses HTTP GET to obtain data from I<bio> if non-NULL,
 194 else from the server contained in the I<url>, and returns it as a BIO.
 195 It supports redirection via HTTP status code 301 or 302.  It is meant for
 196 transfers with a single round trip, so does not support persistent connections.
 197 If I<bio> is non-NULL, any host and port components in the I<url> are not used
 198 for connecting but the hostname is used, as usual, for the C<Host> header.
 199 Any userinfo and fragment components in the I<url> are ignored.
 200 Any query component is handled as part of the path component.
 201 If the scheme component of the I<url> is C<https> a TLS connection is requested
 202 and the I<bio_update_fn>, as described for OSSL_HTTP_open(), must be provided.
 203 Also the remaining parameters are interpreted as described for OSSL_HTTP_open()
 204 and OSSL_HTTP_set_request(), respectively.
 205
 206 OSSL_HTTP_close() closes the connection and releases I<rctx>.
 207 The I<ok> parameter is passed to any BIO update function
 208 given during setup as described above for OSSL_HTTP_open().
 209
 210 =head1 NOTES
 211
 212 The names of the environment variables used by this implementation:
 213 C<http_proxy>, C<HTTP_PROXY>, C<https_proxy>, C<HTTPS_PROXY>, C<no_proxy>, and
 214 C<NO_PROXY>, have been chosen for maximal compatibility with
 215 other HTTP client implementations such as wget, curl, and git.
 216
 217 =head1 RETURN VALUES
 218
 219 OSSL_HTTP_open() returns on success a B<OSSL_HTTP_REQ_CTX>, else NULL.
 220
 221 OSSL_HTTP_proxy_connect() and OSSL_HTTP_set_request()
 222 return 1 on success, 0 on error.
 223
 224 On success, OSSL_HTTP_exchange(), OSSL_HTTP_get(), and OSSL_HTTP_transfer()
 225 return a memory BIO containing the data received if an ASN.1-encoded response
 226 is expected, else a BIO that may support streaming.
 227 The BIO must be freed by the caller.
 228 On failure, they return NULL.
 229 Failure conditions include connection/transfer timeout, parse errors, etc.
 230
 231 OSSL_HTTP_close() returns 0 if anything went wrong while disconnecting, else 1.
 232
 233 =head1 SEE ALSO
 234
 235 L<OSSL_HTTP_parse_url(3)>, L<BIO_set_conn_port(3)>
 236 L<ASN1_item_i2d_mem_bio(3)>, L<ASN1_item_d2i_bio(3)>,
 237 L<OSSL_HTTP_is_alive(3)>
 238
 239 =head1 HISTORY
 240
 241 All the functions described here were added in OpenSSL 3.0.
 242
 243 =head1 COPYRIGHT
 244
 245 Copyright 2019-2021 The OpenSSL Project Authors. All Rights Reserved.
 246
 247 Licensed under the Apache License 2.0 (the "License").  You may not use
 248 this file except in compliance with the License.  You can obtain a copy
 249 in the file LICENSE in the source distribution or at
 250 L<https://www.openssl.org/source/license.html>.
 251
 252 =cut