From: Amos Jeffries <squid3@treenet.co.nz>
Date: Tue, 12 Aug 2014 15:57:20 +0000 (-0700)
Subject: Merge from trunk rev.13532
X-Git-Tag: SQUID_3_5_0_1~75^2~8
X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=a07a354fd0c528b6ce429fd39fdf2f8ae1283f17;p=thirdparty%2Fsquid.git

Merge from trunk rev.13532
---

a07a354fd0c528b6ce429fd39fdf2f8ae1283f17
diff --cc doc/release-notes/release-3.5.sgml
index c403c1b6c8,ed3af7edd5..acb91142ba
--- a/doc/release-notes/release-3.5.sgml
+++ b/doc/release-notes/release-3.5.sgml
@@@ -43,7 -43,7 +43,8 @@@ The 3.5 change history can be <url url=
  	<item>Support named services
  	<item>Upgraded squidclient tool
  	<item>Helper support for concurrency channels
+ 	<item>Native FTP Relay
 +	<item>Receive PROXY protocol, Versions 1 & 2
  </itemize>
  
  Most user-facing changes are reflected in squid.conf (see below).
@@@ -164,41 -164,47 +165,82 @@@
     With these helpers concurrency may now be set to 0 or any higher number as desired.
  
  
+ <sect1>Native FTP Relay
+ <p>Details at <url url="http://wiki.squid-cache.org/Features/FtpRelay">.
+ 
+ <p>Squid is now capable of accepting native FTP commands and relaying native
+    FTP messages between FTP clients and FTP servers. Native FTP commands
+    accepted at ftp_port are internally converted or wrapped into HTTP-like
+    messages. The same happens to Native FTP responses received from FTP origin
+    servers. Those HTTP-like messages are shoveled through regular access
+    control and adaptation layers between the FTP client and the FTP origin
+    server. This allows Squid to examine, adapt, block, and log FTP exchanges.
+    Squid reuses most HTTP mechanisms when shoveling wrapped FTP messages. For
+    example, http_access and adaptation_access directives are used.
+ 
+ <p>FTP Relay is a new, experimental, complex feature that has seen limited
+    production exposure. Some Squid modules (e.g., caching) do not currently
+    work with native FTP proxying, and many features have not even been tested
+    for compatibility. Test well before deploying!
+ 
+ <p>Native FTP proxying differs substantially from proxying HTTP requests with
+    <em>ftp://</em> URIs because Squid works as an FTP server and receives
+    actual FTP commands (rather than HTTP requests with FTP URLs).
+ 
+ <p>FTP Relay highlights:</p>
+ 
+ <itemize>
+     <item>Added ftp_port directive telling Squid to relay native FTP commands.
+     <item>Active and passive FTP support on the user-facing side; require
+ 	  passive connections to come from the control connection source IP
+ 	  address.
+     <item>IPv6 support (EPSV and, on the user-facing side, EPRT).
+     <item>Intelligent adaptation of relayed FTP FEAT responses.
+     <item>Relaying of multi-line FTP control responses using various formats.
+     <item>Support relaying of FTP MLSD and MLST commands (RFC 3659).
+     <item>Several Microsoft FTP server compatibility features.
+     <item>ICAP/eCAP support (at individual FTP command/response level).
+     <item>Optional "current FTP directory" tracking with the assistance of
+ 	  injected (by Squid) PWD commands (cannot be 100% reliable due to
+ 	  symbolic links and such, but is helpful in some common use cases).
+     <item>No caching support -- no reliable Request URIs for that (see above).
+ </itemize>
+ 
 +<sect1>Receive PROXY protocol, Versions 1 & 2
 +<p>More info at <url url="http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt">
 +
 +<p>PROXY protocol provides a simple way for proxies and tunnels of any kind to
 +   relay the original client source details without having to alter or understand
 +   the protocol being relayed on the connection.
 +
 +<p>Squid currently supports receiving HTTP traffic from a client proxy using this protocol.
 +   An http_port which has been configured to receive this protocol may only be used to
 +   receive traffic from client software sending in this protocol.
 +   HTTP traffic without the PROXY header is not accepted on such a port.
 +
 +<p>The <em>accel</em> and <em>intercept</em> options are still used to identify the
 +   traffic syntax being delivered by the client proxy.
 +
 +<p>Squid can be configured by adding an <em>http_port</em>
 +   with the <em>require-proxy-header</em> mode flag. The <em>proxy_protocol_access</em>
 +   must also be configured with <em>src</em> ACLs to whitelist proxies which are
 +   trusted to send correct client details.
 +
 +<p>Forward-proxy traffic from a client proxy:
 +<verbatim>
 + http_port 3128 require-proxy-header
 + proxy_protocol_access allow localhost
 +</verbatim>
 +
 +<p>Intercepted traffic from a client proxy or tunnel:
 +<verbatim>
 + http_port 3128 intercept require-proxy-header
 + proxy_protocol_access allow localhost
 +</verbatim>
 +
 +<p><em>Known Issue:</em>
 +   Use of <em>require-proxy-header</em> on <em>https_port</em> is not supported.
 +
  
  <sect>Changes to squid.conf since Squid-3.4
  <p>
diff --cc src/cache_cf.cc
index 3f6ea48f46,a96c3269fe..dcbe03f26a
--- a/src/cache_cf.cc
+++ b/src/cache_cf.cc
@@@ -3598,26 -3620,14 +3620,26 @@@ parse_port_option(AnyP::PortCfgPointer 
          /* Log information regarding the port modes under transparency. */
          debugs(3, DBG_IMPORTANT, "Disabling Authentication on port " << s->s << " (TPROXY enabled)");
  
 +        if (s->flags.proxySurrogate) {
 +            debugs(3, DBG_IMPORTANT, "Disabling TPROXY Spoofing on port " << s->s << " (require-proxy-header enabled)");
 +        }
 +
          if (!Ip::Interceptor.ProbeForTproxy(s->s)) {
-             debugs(3, DBG_CRITICAL, "FATAL: http(s)_port: TPROXY support in the system does not work.");
+             debugs(3, DBG_CRITICAL, "FATAL: " << cfg_directive << ": TPROXY support in the system does not work.");
              self_destruct();
          }
  
 +    } else if (strcmp(token, "require-proxy-header") == 0) {
 +        s->flags.proxySurrogate = true;
 +        if (s->flags.tproxyIntercept) {
 +            // receiving is still permitted, so we do not unset the TPROXY flag
 +            // spoofing access control override takes care of the spoof disable later
 +            debugs(3, DBG_IMPORTANT, "Disabling TPROXY Spoofing on port " << s->s << " (require-proxy-header enabled)");
 +        }
 +
      } else if (strncmp(token, "defaultsite=", 12) == 0) {
          if (!s->flags.accelSurrogate) {
-             debugs(3, DBG_CRITICAL, "FATAL: http(s)_port: defaultsite option requires Acceleration mode flag.");
+             debugs(3, DBG_CRITICAL, "FATAL: " << cfg_directive << ": defaultsite option requires Acceleration mode flag.");
              self_destruct();
          }
          safe_free(s->defaultsite);
@@@ -3824,12 -3838,14 +3850,23 @@@ parsePortCfg(AnyP::PortCfgPointer *head
              debugs(3, DBG_CRITICAL, "FATAL: tproxy/intercept on https_port requires ssl-bump which is missing.");
              self_destruct();
          }
 +#endif
-         if (s->transport.protocol == AnyP::PROTO_HTTPS) {
++        if (s->flags.proxySurrogate) {
 +            debugs(3,DBG_CRITICAL, "FATAL: https_port: require-proxy-header option is not supported on HTTPS ports.");
 +            self_destruct();
 +        }
+     } else if (protoName.cmp("FTP") == 0) {
+         /* ftp_port does not support ssl-bump */
+         if (s->flags.tunnelSslBumping) {
+             debugs(3, DBG_CRITICAL, "FATAL: ssl-bump is not supported for ftp_port.");
+             self_destruct();
+         }
++        if (s->flags.proxySurrogate) {
++            // Passive FTP data channel does not work without deep protocol inspection in the frontend.
++            debugs(3,DBG_CRITICAL, "FATAL: require-proxy-header option is not supported on ftp_port.");
++            self_destruct();
++        }
      }
 -#endif
  
      if (Ip::EnableIpv6&IPV6_SPECIAL_SPLITSTACK && s->s.isAnyAddr()) {
          // clone the port options from *s to *(s->next)
diff --cc src/client_side.cc
index 07cdca53fd,d8966a220e..4e20a9eda7
--- a/src/client_side.cc
+++ b/src/client_side.cc
@@@ -119,9 -119,9 +119,10 @@@
  #include "MemBuf.h"
  #include "MemObject.h"
  #include "mime_header.h"
 +#include "parser/Tokenizer.h"
  #include "profiler/Profiler.h"
  #include "rfc1738.h"
+ #include "servers/forward.h"
  #include "SquidConfig.h"
  #include "SquidTime.h"
  #include "StatCounters.h"
@@@ -3196,19 -2919,8 +3188,12 @@@ ConnStateData::clientParseRequests(
          if (concurrentRequestQueueFilled())
              break;
  
-         /* Begin the parsing */
-         PROF_start(parseHttpRequest);
- 
 +        // try to parse the PROXY protocol header magic bytes
 +        if (needProxyProtocolHeader_ && !parseProxyProtocolHeader())
 +            break;
 +
-         HttpParserInit(&parser_, in.buf.c_str(), in.buf.length());
- 
-         /* Process request */
          Http::ProtocolVersion http_ver;
-         ClientSocketContext *context = parseHttpRequest(this, &parser_, &method, &http_ver);
-         PROF_stop(parseHttpRequest);
+         ClientSocketContext *context = parseOneRequest(http_ver);
  
          /* partial or incomplete request */
          if (!context) {
@@@ -3553,6 -3252,18 +3525,17 @@@ ConnStateData::ConnStateData(const Mast
      port = xact->squidPort;
      log_addr = xact->tcpClient->remote;
      log_addr.applyMask(Config.Addrs.client_netmask);
 -
+     flags.readMore = true; // kids may overwrite
+ }
+ 
+ void
+ ConnStateData::start()
+ {
+     BodyProducer::start();
+     HttpControlMsgSink::start();
+ 
+     // ensure a buffer is present for this connection
+     in.maybeMakeSpaceAvailable();
  
      if (port->disable_pmtu_discovery != DISABLE_PMTU_OFF &&
              (transparent() || port->disable_pmtu_discovery == DISABLE_PMTU_ALWAYS)) {
@@@ -3596,57 -3300,8 +3579,14 @@@ ConnStateData::start(
  
      clientdbEstablished(clientConnection->remote, 1);
  
 +    needProxyProtocolHeader_ = port->flags.proxySurrogate;
 +    if (needProxyProtocolHeader_) {
 +        if (!proxyProtocolValidateClient()) // will close the connection on failure
 +            return;
 +    }
 +
-     // prepare any child API state that is needed
-     BodyProducer::start();
-     HttpControlMsgSink::start();
- 
-     // if all is well, start reading
-     flags.readMore = true;
-     readSomeData();
- }
- 
- /** Handle a new connection on HTTP socket. */
- void
- httpAccept(const CommAcceptCbParams &params)
- {
-     MasterXaction::Pointer xact = params.xaction;
-     AnyP::PortCfgPointer s = xact->squidPort;
- 
-     // NP: it is possible the port was reconfigured when the call or accept() was queued.
- 
-     if (params.flag != Comm::OK) {
-         // Its possible the call was still queued when the client disconnected
-         debugs(33, 2, "httpAccept: " << s->listenConn << ": accept failure: " << xstrerr(params.xerrno));
-         return;
-     }
- 
-     debugs(33, 4, HERE << params.conn << ": accepted");
-     fd_note(params.conn->fd, "client http connect");
- 
-     if (s->tcp_keepalive.enabled) {
-         commSetTcpKeepalive(params.conn->fd, s->tcp_keepalive.idle, s->tcp_keepalive.interval, s->tcp_keepalive.timeout);
-     }
- 
-     ++ incoming_sockets_accepted;
- 
-     // Socket is ready, setup the connection manager to start using it
-     ConnStateData *connState = new ConnStateData(xact);
- 
-     typedef CommCbMemFunT<ConnStateData, CommTimeoutCbParams> TimeoutDialer;
-     AsyncCall::Pointer timeoutCall =  JobCallback(33, 5,
-                                       TimeoutDialer, connState, ConnStateData::requestTimeout);
-     commSetConnTimeout(params.conn, Config.Timeout.request, timeoutCall);
- 
-     AsyncJob::Start(connState);
- 
  #if USE_DELAY_POOLS
-     fd_table[params.conn->fd].clientInfo = NULL;
+     fd_table[clientConnection->fd].clientInfo = NULL;
  
      if (Config.onoff.client_db) {
          /* it was said several times that client write limiter does not work if client_db is disabled */
diff --cc src/client_side.h
index 6b30a9c314,238c684cc1..a0bf4f2438
--- a/src/client_side.h
+++ b/src/client_side.h
@@@ -400,16 -435,8 +435,18 @@@ private
      void clientAfterReadingRequests();
      bool concurrentRequestQueueFilled() const;
  
+     void pinNewConnection(const Comm::ConnectionPointer &pinServer, HttpRequest *request, CachePeer *aPeer, bool auth);
+ 
 +    /* PROXY protocol functionality */
 +    bool proxyProtocolValidateClient();
 +    bool parseProxyProtocolHeader();
 +    bool parseProxy1p0();
 +    bool parseProxy2p0();
 +    bool proxyProtocolError(const char *reason);
 +
 +    /// whether PROXY protocol header is still expected
 +    bool needProxyProtocolHeader_;
 +
  #if USE_AUTH
      /// some user details that can be used to perform authentication on this connection
      Auth::UserRequest::Pointer auth_;