+++ /dev/null
-2011/04/20 - List of keep-alive / close options with associated behaviours.
-
-PK="http-pretend-keepalive", HC="httpclose", SC="http-server-close",
-
-0 = option not set
-1 = option is set
-* = option doesn't matter
-
-Options can be split between frontend and backend, so some of them might have
-a meaning only when combined by associating a frontend to a backend. Some forms
-are not the normal ones and provide a behaviour compatible with another normal
-form. Those are considered alternate forms and are marked "(alt)".
-
-SC HC PK Behaviour
- 0 0 X tunnel mode
- 0 1 0 passive close, only set headers then tunnel
- 0 1 1 forced close with keep-alive announce (alt)
- 1 0 0 server close
- 1 0 1 server close with keep-alive announce
- 1 1 0 forced close (alt)
- 1 1 1 forced close with keep-alive announce (alt)
-
-At this point this results in 4 distinct effective modes for a request being
-processed :
- - tunnel mode : Connection header is left untouched and body is ignored
- - passive close : Connection header is changed and body is ignored
- - server close : Connection header set, body scanned, client-side keep-alive
- is made possible regardless of server-side capabilities
- - forced close : Connection header set, body scanned, connection closed.
-
-The "close" modes may be combined with a fake keep-alive announce to the server
-in order to workaround buggy servers that disable chunked encoding and content
-length announces when the client does not ask for keep-alive.
-
-Note: "http-pretend-keepalive" alone has no effect. However, if it is set in a
- backend while a frontend is in "http-close" mode, then the combination of
- both will result in a forced close with keep-alive announces for requests
- passing through both.
-
+++ /dev/null
-1 type générique "entité", avec les attributs suivants :
-
- - frontend *f
- - l7switch *s
- - backend *b
-
-des types spécifiques sont simplement des entités avec certains
-de ces champs remplis et pas forcément tous :
-
- listen = f [s] b
- frontend = f [s]
- l7switch = s
- backend = [s] b
-
-Ensuite, les traitements sont évalués dans l'ordre :
- - listen -> s'il a des règles de l7, on les évalue, et potentiellement on branche vers d'autres listen, l7 ou back, ou on travaille avec le back local.
- - frontend -> s'il a des règles de l7, on les évalue, et potentiellement on branche vers d'autres listen, l7 ou back
- - l7switch -> on évalue ses règles, potentiellement on branche vers d'autres listen, l7 ou backends
- - backend -> s'il a des règles l7, on les évalue (quitte à changer encore de backend) puis on traite.
-
-Les requêtes sont traitées dans l'ordre des chaînages f->s*->b, et les réponses doivent être
-traitées dans l'ordre inverse b->s*->f. Penser aux réécritures de champs Host à l'aller et
-Location en retour.
-
-D'autre part, prévoir des "profils" plutôt que des blocs de nouveaux paramètres par défaut.
-Ca permettra d'avoir plein de jeux de paramètres par défaut à utiliser dans chacun de ces
-types.
+++ /dev/null
-There has been a lot of confusion during the development because of the
-backends and frontends.
-
-What we want :
-
-- being able to still use a listener as it has always been working
-
-- being able to write a rule stating that we will *change* the backend when we
- match some pattern. Only one jump is allowed.
-
-- being able to write a "use_filters_from XXX" line stating that we will ignore
- any filter in the current listener, and that those from XXX will be borrowed
- instead. A warning would be welcome for options which will silently get
- masked. This is used to factor configuration.
-
-- being able to write a "use_backend_from XXX" line stating that we will ignore
- any server and timeout config in the current listener, and that those from
- XXX will be borrowed instead. A warning would be welcome for options which
- will silently get masked. This is used to factor configuration.
-
-
-
-Example :
----------
-
- | # frontend HTTP/80
- | listen fe_http 1.1.1.1:80
- | use_filters_from default_http
- | use_backend_from appli1
- |
- | # frontend HTTPS/443
- | listen fe_https 1.1.1.1:443
- | use_filters_from default_https
- | use_backend_from appli1
- |
- | # frontend HTTP/8080
- | listen fe_http-dev 1.1.1.1:8080
- | reqadd "X-proto: http"
- | reqisetbe "^Host: www1" appli1
- | reqisetbe "^Host: www2" appli2
- | reqisetbe "^Host: www3" appli-dev
- | use_backend_from appli1
- |
- |
- | # filters default_http
- | listen default_http
- | reqadd "X-proto: http"
- | reqisetbe "^Host: www1" appli1
- | reqisetbe "^Host: www2" appli2
- |
- | # filters default_https
- | listen default_https
- | reqadd "X-proto: https"
- | reqisetbe "^Host: www1" appli1
- | reqisetbe "^Host: www2" appli2
- |
- |
- | # backend appli1
- | listen appli1
- | reqidel "^X-appli1:.*"
- | reqadd "Via: appli1"
- | balance roundrobin
- | cookie app1
- | server srv1
- | server srv2
- |
- | # backend appli2
- | listen appli2
- | reqidel "^X-appli2:.*"
- | reqadd "Via: appli2"
- | balance roundrobin
- | cookie app2
- | server srv1
- | server srv2
- |
- | # backend appli-dev
- | listen appli-dev
- | reqadd "Via: appli-dev"
- | use_backend_from appli2
- |
- |
-
-
-Now we clearly see multiple things :
-------------------------------------
-
- - a frontend can EITHER have filters OR reference a use_filter
-
- - a backend can EITHER have servers OR reference a use_backend
-
- - we want the evaluation to cross TWO levels per request. When a request is
- being processed, it keeps track of its "frontend" side (where it came
- from), and of its "backend" side (where the server-side parameters have
- been found).
-
- - the use_{filters|backend} have nothing to do with how the request is
- decomposed.
-
-
-Conclusion :
-------------
-
- - a proxy is always its own frontend. It also has 2 parameters :
- - "fi_prm" : pointer to the proxy holding the filters (itself by default)
- - "be_prm" : pointer to the proxy holding the servers (itself by default)
-
- - a request has a frontend (fe) and a backend (be). By default, the backend
- is initialized to the frontend. Everything related to the client side is
- accessed through ->fe. Everything related to the server side is accessed
- through ->be.
-
- - request filters are first called from ->fe then ->be. Since only the
- filters can change ->be, it is possible to iterate the filters on ->be
- only and stop when ->be does not change anymore.
-
- - response filters are first called from ->be then ->fe IF (fe != be).
-
-
-When we parse the configuration, we immediately configure ->fi and ->be for
-all proxies.
-
-Upon session creation, s->fe and s->be are initialized to the proxy. Filters
-are executed via s->fe->fi_prm and s->be->fi_prm. Servers are found in
-s->be->be_prm.
-
+++ /dev/null
-- PR_O_TRANSP => FE !!! devra peut-être changer vu que c'est un complément du mode dispatch.
-- PR_O_NULLNOLOG => FE
-- PR_O_HTTP_CLOSE => FE. !!! mettre BE aussi !!!
-- PR_O_TCP_CLI_KA => FE
-
-- PR_O_FWDFOR => BE. FE aussi ?
-- PR_O_FORCE_CLO => BE
-- PR_O_PERSIST => BE
-- PR_O_COOK_RW, PR_O_COOK_INS, PR_O_COOK_PFX, PR_O_COOK_POST => BE
-- PR_O_COOK_NOC, PR_O_COOK_IND => BE
-- PR_O_ABRT_CLOSE => BE
-- PR_O_REDISP => BE
-- PR_O_BALANCE, PR_O_BALANCE_RR, PR_O_BALANCE_SH => BE
-- PR_O_CHK_CACHE => BE
-- PR_O_TCP_SRV_KA => BE
-- PR_O_BIND_SRC => BE
-- PR_O_TPXY_MASK => BE
-
-
-- PR_MODE_TCP : BE côté serveur, FE côté client
-
-- nbconn -> fe->nbconn, be->nbconn.
- Pb: rendre impossible le fait que (fe == be) avant de faire ça,
- sinon on va compter les connexions en double. Ce ne sera possible
- que lorsque les FE et BE seront des entités distinctes. On va donc
- commencer par laisser uniquement fe->nbconn (vu que le fe ne change
- pas), et modifier ceci plus tard, ne serait-ce que pour prendre en
- compte correctement les minconn/maxconn.
- => solution : avoir beconn et feconn dans chaque proxy.
-
-- failed_conns, failed_secu (réponses bloquées), failed_resp... : be
- Attention: voir les cas de ERR_SRVCL, il semble que parfois on
- indique ça alors qu'il y a un write error côté client (ex: ligne
- 2044 dans proto_http).
-
- => be et pas be->beprm
-
-- logs du backup : ->be (idem)
-
-- queue : be
-
-- logs/debug : srv toujours associé à be (ex: proxy->id:srv->id). Rien
- pour le client pour le moment. D'une manière générale, les erreurs
- provoquées côté serveur vont sur BE et celles côté client vont sur
- FE.
-- logswait & LW_BYTES : FE (puisqu'on veut savoir si on logue tout de suite)
-
-- messages d'erreurs personnalisés (errmsg, ...) -> fe
-
-- monitor_uri -> fe
-- uri_auth -> (fe->firpm puis be->fiprm). Utilisation de ->be
-
-- req_add, req_exp => fe->fiprm, puis be->fiprm
-- req_cap, rsp_cap -> fe->fiprm
-- rsp_add, rsp_exp => be->fiprm, devrait être fait ensuite aussi sur fe->fiprm
-- capture_name, capture_namelen : fe->fiprm
-
- Ce n'est pas la solution idéale, mais au moins la capture et configurable
- par les filtres du FE et ne bouge pas lorsque le BE est réassigné. Cela
- résoud aussi un pb d'allocation mémoire.
-
-
-- persistance (appsessions, cookiename, ...) -> be
-- stats:scope "." = fe (celui par lequel on arrive)
- !!!ERREUR!!! => utiliser be pour avoir celui qui a été validé par
- l'uri_auth.
-
-
---------- corrections à effectuer ---------
-
-- remplacement de headers : parser le header et éventuellement le supprimer puis le(les) rajouter.
-- session->proto.{l4state,l7state,l7substate} pour CLI et SRV
-- errorloc : si définie dans backend, la prendre, sinon dans front.
-- logs : faire be sinon fe.
+++ /dev/null
-Prévoir des commandes en plusieurs mots clés.
-Par exemple :
-
- timeout connection XXX
- connection scale XXX
-
-On doit aussi accepter les préfixes :
-
- tim co XXX
- co sca XXX
-
-Prévoir de ranger les combinaisons dans un tableau. On doit même
-pouvoir effectuer un mapping simplifiant le parseur.
-
-
-Pour les filtres :
-
-
- <direction> <where> <what> <operator> <pattern> <action> [ <args>* ]
-
- <direction> = [ req | rsp ]
- <where> = [ in | out ]
- <what> = [ line | LINE | METH | URI | h(hdr) | H(hdr) | c(cookie) | C(cookie) ]
- <operator> = [ == | =~ | =* | =^ | =/ | != | !~ | !* | !^ | !/ ]
- <pattern> = "<string>"
- <action> = [ allow | permit | deny | delete | replace | switch | add | set | redir ]
- <args> = optional action args
-
- examples:
-
- req in URI =^ "/images" switch images
- req in h(host) =* ".mydomain.com" switch mydomain
- req in h(host) =~ "localhost(.*)" replace "www\1"
-
- alternative :
-
- <direction> <where> <action> [not] <what> [<operator> <pattern> [ <args>* ]]
-
- req in switch URI =^ "/images" images
- req in switch h(host) =* ".mydomain.com" mydomain
- req in replace h(host) =~ "localhost(.*)" "www\1"
- req in delete h(Connection)
- req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
- req out set h(Connection) "close"
- req out add line "Server: truc"
-
-
- <direction> <action> <where> [not] <what> [<operator> <pattern> [ <args>* ]] ';' <action2> <what2>
-
- req in switch URI =^ "/images/" images ; replace "/"
- req in switch h(host) =* ".mydomain.com" mydomain
- req in replace h(host) =~ "localhost(.*)" "www\1"
- req in delete h(Connection)
- req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
- req out set h(Connection) "close"
- req out add line == "Server: truc"
-
-
-Extension avec des ACL :
-
- req in acl(meth_valid) METH =~ "(GET|POST|HEAD|OPTIONS)"
- req in acl(meth_options) METH == "OPTIONS"
- req in acl(uri_slash) URI =^ "/"
- req in acl(uri_star) URI == "*"
-
- req in deny acl !(meth_options && uri_star || meth_valid && uri_slash)
-
-Peut-être plus simplement :
-
- acl meth_valid METH =~ "(GET|POST|HEAD|OPTIONS)"
- acl meth_options METH == "OPTIONS"
- acl uri_slash URI =^ "/"
- acl uri_star URI == "*"
-
- req in deny not acl(meth_options uri_star, meth_valid uri_slash)
-
- req in switch URI =^ "/images/" images ; replace "/"
- req in switch h(host) =* ".mydomain.com" mydomain
- req in replace h(host) =~ "localhost(.*)" "www\1"
- req in delete h(Connection)
- req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
- req out set h(Connection) "close"
- req out add line == "Server: truc"
-
-Prévoir le cas du "if" pour exécuter plusieurs actions :
-
- req in if URI =^ "/images/" then replace "/" ; switch images
-
-Utiliser les noms en majuscules/minuscules pour indiquer si on veut prendre
-en compte la casse ou non :
-
- if uri =^ "/watch/" setbe watch rebase "/watch/" "/"
- if uri =* ".jpg" setbe images
- if uri =~ ".*dll.*" deny
- if HOST =* ".mydomain.com" setbe mydomain
- etc...
-
-Another solution would be to have a dedicated keyword to URI remapping. It
-would both rewrite the URI and optionally switch to another backend.
-
- uriremap "/watch/" "/" watch
- uriremap "/chat/" "/" chat
- uriremap "/event/" "/event/" event
-
-Or better :
-
- uriremap "/watch/" watch "/"
- uriremap "/chat/" chat "/"
- uriremap "/event/" event
-
-For the URI, using a regex is sometimes useful (eg: providing a set of possible prefixes.
-
-
-Sinon, peut-être que le "switch" peut prendre un paramètre de mapping pour la partie matchée :
-
- req in switch URI =^ "/images/" images:"/"
-
-
-2007/03/31 - Besoins plus précis.
-
-1) aucune extension de branchement ou autre dans les "listen", c'est trop complexe.
-
-Distinguer les données entrantes (in) et sortantes (out).
-
-Le frontend ne voit que les requetes entrantes et les réponses sortantes.
-Le backend voir les requêtes in/out et les réponses in/out.
-Le frontend permet les branchements d'ensembles de filtres de requêtes vers
-d'autres. Le frontend et les ensembles de filtres de requêtes peuvent brancher
-vers un backend.
-
------------+--------+----------+----------+---------+----------+
- \ Where | | | | | |
- \______ | Listen | Frontend | ReqRules | Backend | RspRules |
- \| | | | | |
-Capability | | | | | |
------------+--------+----------+----------+---------+----------+
-Frontend | X | X | | | |
------------+--------+----------+----------+---------+----------+
-FiltReqIn | X | X | X | X | |
------------+--------+----------+----------+---------+----------+
-JumpFiltReq| X | X | X | | | \
------------+--------+----------+----------+---------+----------+ > = ReqJump
-SetBackend | X | X | X | | | /
------------+--------+----------+----------+---------+----------+
-FiltReqOut | | | | X | |
------------+--------+----------+----------+---------+----------+
-FiltRspIn | X | | | X | X |
------------+--------+----------+----------+---------+----------+
-JumpFiltRsp| | | | X | X |
------------+--------+----------+----------+---------+----------+
-FiltRspOut | | X | | X | X |
------------+--------+----------+----------+---------+----------+
-Backend | X | | | X | |
------------+--------+----------+----------+---------+----------+
-
-En conclusion
--------------
-
-Il y a au moins besoin de distinguer 8 fonctionnalités de base :
- - capacité à recevoir des connexions (frontend)
- - capacité à filtrer les requêtes entrantes
- - capacité à brancher vers un backend ou un ensemble de règles de requêtes
- - capacité à filtrer les requêtes sortantes
- - capacité à filtrer les réponses entrantes
- - capacité à brancher vers un autre ensemble de règles de réponses
- - capacité à filtrer la réponse sortante
- - capacité à gérer des serveurs (backend)
-
-Remarque
---------
- - on a souvent besoin de pouvoir appliquer un petit traitement sur un ensemble
- host/uri/autre. Le petit traitement peut consister en quelques filtres ainsi
- qu'une réécriture du couple (host,uri).
-
-
-Proposition : ACL
-
-Syntaxe :
----------
-
- acl <name> <what> <operator> <value> ...
-
-Ceci créera une acl référencée sous le nom <name> qui sera validée si
-l'application d'au moins une des valeurs <value> avec l'opérateur <operator>
-sur le sujet <what> est validée.
-
-Opérateurs :
-------------
-
-Toujours 2 caractères :
-
- [=!][~=*^%/.]
-
-Premier caractère :
- '=' : OK si test valide
- '!' : OK si test échoué.
-
-Second caractère :
- '~' : compare avec une regex
- '=' : compare chaîne à chaîne
- '*' : compare la fin de la chaîne (ex: =* ".mydomain.com")
- '^' : compare le début de la chaîne (ex: =^ "/images/")
- '%' : recherche une sous-chaîne
- '/' : compare avec un mot entier en acceptant le '/' comme délimiteur
- '.' : compare avec un mot entier en acceptant el '.' comme délimiteur
-
-Ensuite on exécute une action de manière conditionnelle si l'ensemble des ACLs
-mentionnées sont validées (ou invalidées pour celles précédées d'un "!") :
-
- <what> <where> <action> on [!]<aclname> ...
-
-
-Exemple :
----------
-
- acl www_pub host =. www www01 dev preprod
- acl imghost host =. images
- acl imgdir uri =/ img
- acl imagedir uri =/ images
- acl msie h(user-agent) =% "MSIE"
-
- set_host "images" on www_pub imgdir
- remap_uri "/img" "/" on www_pub imgdir
- remap_uri "/images" "/" on www_pub imagedir
- setbe images on imghost
- reqdel "Cookie" on all
-
-
-
-Actions possibles :
-
- req {in|out} {append|delete|rem|add|set|rep|mapuri|rewrite|reqline|deny|allow|setbe|tarpit}
- resp {in|out} {append|delete|rem|add|set|rep|maploc|rewrite|stsline|deny|allow}
-
- req in append <line>
- req in delete <line_regex>
- req in rem <header>
- req in add <header> <new_value>
- req in set <header> <new_value>
- req in rep <header> <old_value> <new_value>
- req in mapuri <old_uri_prefix> <new_uri_prefix>
- req in rewrite <old_uri_regex> <new_uri>
- req in reqline <old_req_regex> <new_req>
- req in deny
- req in allow
- req in tarpit
- req in setbe <backend>
-
- resp out maploc <old_location_prefix> <new_loc_prefix>
- resp out stsline <old_sts_regex> <new_sts_regex>
-
-Les chaînes doivent être délimitées par un même caractère au début et à la fin,
-qui doit être échappé s'il est présent dans la chaîne. Tout ce qui se trouve
-entre le caractère de fin et les premiers espace est considéré comme des
-options passées au traitement. Par exemple :
-
- req in rep host /www/i /www/
- req in rep connection /keep-alive/i "close"
-
-Il serait pratique de pouvoir effectuer un remap en même temps qu'un setbe.
-
-Captures: les séparer en in/out. Les rendre conditionnelles ?
+++ /dev/null
-2014/10/28 - Server connection sharing
-
-For HTTP/2 we'll have to use multiplexed connections to the servers and to
-share them between multiple streams. We'll also have to do this for H/1, but
-with some variations since H1 doesn't offer connection status verification.
-
-In order to validate that an idle connection is still usable, it is desirable
-to periodically send health checks over it. Normally, idle connections are
-meant to be heavily used, so there is no reason for having them idle for a long
-time. Thus we have two possibilities :
-
- - either we time them out after some inactivity, this saves server resources ;
- - or we check them after some inactivity. For this we can send the server-
- side HTTP health check (only when the server uses HTTP checks), and avoid
- using that to mark the server down, and instead consider the connection as
- dead.
-
-For HTTP/2 we'll have to send pings periodically over these connections, so
-it's worth considering a per-connection task to validate that the channel still
-works.
-
-In the current model, a connection necessarily belongs to a session, so it's
-not really possible to share them, at best they can be exchanged, but that
-doesn't make much sense as it means that it could disturb parallel traffic.
-
-Thus we need to have a per-server list of idle connections and a max-idle-conn
-setting to kill them when there are too many. In the case of H/1 it is also
-advisable to consider that if a connection was created to pass a first non-
-idempotent request while other idle connections were still existing, then a
-connection will have to be killed in order not to exceed the limit.
-
+++ /dev/null
-2014/10/30 - dynamic buffer management
-
-Since HTTP/2 processing will significantly increase the need for buffering, it
-becomes mandatory to be able to support dynamic buffer allocation. This also
-means that at any moment some buffer allocation will fail and that a task or an
-I/O operation will have to be paused for the time needed to allocate a buffer.
-
-There are 3 places where buffers are needed :
-
- - receive side of a stream interface. A connection notifies about a pending
- recv() and the SI calls the receive function to put the data into a buffer.
- Here the buffer will have to be picked from a pool first, and if the
- allocation fails, the I/O will have to temporarily be disabled, the
- connection will have to subscribe for buffer release notification to be
- woken up once a buffer is available again. It's important to keep in mind
- that buffer availability doesn't necessarily mean a desire to enable recv
- again, just that recv is not paused anymore for resource reasons.
-
- - receive side of a stream interface when the other end point is an applet.
- The applet wants to write into the buffer and for this the buffer needs to
- be allocated as well. It is the same as above except that it is the applet
- which is put to a pause. Since the applet might be at the core of the task
- itself, it could become tricky to handle the situation correctly. Stats and
- peers are in this situation.
-
- - Tx of a task : some tasks perform spontaneous writes to a buffer. Checks
- are an example of this. The checks will have to be able to sleep while a
- buffer is being awaited.
-
-One important point is that such pauses must not prevent the task from timing
-out. There it becomes difficult because in the case of a time out, we could
-want to emit a timeout error message and for this, require a buffer. So it is
-important to keep the ability not to send messages upon error processing, and
-to be able to give up and stop waiting for buffers.
-
-The refill mechanism needs to be designed in a thread-safe way because this
-will become one of the rare cases of inter-task activity. Thus it is important
-to ensure that checking the state of the task and passing of the freshly
-released buffer are performed atomically, and that in case the task doesn't
-want it anymore, it is responsible for passing it to the next one.
-
+++ /dev/null
-2012/07/05 - Connection layering and sequencing
-
-
-An FD has a state :
- - CLOSED
- - READY
- - ERROR (?)
- - LISTEN (?)
-
-A connection has a state :
- - CLOSED
- - ACCEPTED
- - CONNECTING
- - ESTABLISHED
- - ERROR
-
-A stream interface has a state :
- - INI, REQ, QUE, TAR, ASS, CON, CER, EST, DIS, CLO
-
-Note that CON and CER might be replaced by EST if the connection state is used
-instead. CON might even be more suited than EST to indicate that a connection
-is known.
-
-
-si_shutw() must do :
-
- data_shutw()
- if (shutr) {
- data_close()
- ctrl_shutw()
- ctrl_close()
- }
-
-si_shutr() must do :
- data_shutr()
- if (shutw) {
- data_close()
- ctrl_shutr()
- ctrl_close()
- }
-
-Each of these steps may fail, in which case the step must be retained and the
-operations postponed in an asynchronous task.
-
-The first asynchronous data_shut() might already fail so it is mandatory to
-save the other side's status with the connection in order to let the async task
-know whether the 3 next steps must be performed.
-
-The connection (or perhaps the FD) needs to know :
- - the desired close operations : DSHR, DSHW, CSHR, CSHW
- - the completed close operations : DSHR, DSHW, CSHR, CSHW
-
-
-On the accept() side, we probably need to know :
- - if a header is expected (eg: accept-proxy)
- - if this header is still being waited for
- => maybe both info might be combined into one bit
-
- - if a data-layer accept() is expected
- - if a data-layer accept() has been started
- - if a data-layer accept() has been performed
- => possibly 2 bits, to indicate the need to free()
-
-On the connect() side, we need to know :
- - the desire to send a header (eg: send-proxy)
- - if this header has been sent
- => maybe both info might be combined
-
- - if a data-layer connect() is expected
- - if a data-layer connect() has been started
- - if a data-layer connect() has been completed
- => possibly 2 bits, to indicate the need to free()
-
-On the response side, we also need to know :
- - the desire to send a header (eg: health check response for monitor-net)
- - if this header was sent
- => might be the same as sending a header over a new connection
-
-Note: monitor-net has precedence over proxy proto and data layers. Same for
- health mode.
-
-For multi-step operations, use 2 bits :
- 00 = operation not desired, not performed
- 10 = operation desired, not started
- 11 = operation desired, started but not completed
- 01 = operation desired, started and completed
-
- => X != 00 ==> operation desired
- X & 01 ==> operation at least started
- X & 10 ==> operation not completed
-
-Note: no way to store status information for error reporting.
-
-Note2: it would be nice if "tcp-request connection" rules could work at the
-connection level, just after headers ! This means support for tracking stick
-tables, possibly not too much complicated.
-
-
-Proposal for incoming connection sequence :
-
-- accept()
-- if monitor-net matches or if mode health => try to send response
-- if accept-proxy, wait for proxy request
-- if tcp-request connection, process tcp rules and possibly keep the
- pointer to stick-table
-- if SSL is enabled, switch to SSL handshake
-- then switch to DATA state and instantiate a session
-
-We just need a map of handshake handlers on the connection. They all manage the
-FD status themselves and set the callbacks themselves. If their work succeeds,
-they remove themselves from the list. If it fails, they remain subscribed and
-enable the required polling until they are woken up again or the timeout strikes.
-
-Identified handshake handlers for incoming connections :
- - HH_HEALTH (tries to send OK and dies)
- - HH_MONITOR_IN (matches src IP and adds/removes HH_SEND_OK/HH_SEND_HTTP_OK)
- - HH_SEND_OK (tries to send "OK" and dies)
- - HH_SEND_HTTP_OK (tries to send "HTTP/1.0 200 OK" and dies)
- - HH_ACCEPT_PROXY (waits for PROXY line and parses it)
- - HH_TCP_RULES (processes TCP rules)
- - HH_SSL_HS (starts SSL handshake)
- - HH_ACCEPT_SESSION (instantiates a session)
-
-Identified handshake handlers for outgoing connections :
- - HH_SEND_PROXY (tries to build and send the PROXY line)
- - HH_SSL_HS (starts SSL handshake)
-
-For the pollers, we could check that handshake handlers are not 0 and decide to
-call a generic connection handshake handler instead of usual callbacks. Problem
-is that pollers don't know connections, they know fds. So entities which manage
-handlers should update change the FD callbacks accordingly.
-
-With a bit of care, we could have :
- - HH_SEND_LAST_CHUNK (sends the chunk pointed to by a pointer and dies)
- => merges HEALTH, SEND_OK and SEND_HTTP_OK
-
-It sounds like the ctrl vs data state for the connection are per-direction
-(eg: support an async ctrl shutw while still reading data).
-
-Also support shutr/shutw status at L4/L7.
-
-In practice, what we really need is :
-
-shutdown(conn) =
- conn.data.shut()
- conn.ctrl.shut()
- conn.fd.shut()
-
-close(conn) =
- conn.data.close()
- conn.ctrl.close()
- conn.fd.close()
-
-With SSL over Remote TCP (RTCP + RSSL) to reach the server, we would have :
-
- HTTP -> RTCP+RSSL connection <-> RTCP+RRAW connection -> TCP+SSL connection
-
-The connection has to be closed at 3 places after a successful response :
- - DATA (RSSL over RTCP)
- - CTRL (RTCP to close connection to server)
- - SOCK (FD to close connection to second process)
-
-Externally, the connection is seen with very few flags :
- - SHR
- - SHW
- - ERR
-
-We don't need a CLOSED flag as a connection must always be detached when it's closed.
-
-The internal status doesn't need to be exposed :
- - FD allocated (Y/N)
- - CTRL initialized (Y/N)
- - CTRL connected (Y/N)
- - CTRL handlers done (Y/N)
- - CTRL failed (Y/N)
- - CTRL shutr (Y/N)
- - CTRL shutw (Y/N)
- - DATA initialized (Y/N)
- - DATA connected (Y/N)
- - DATA handlers done (Y/N)
- - DATA failed (Y/N)
- - DATA shutr (Y/N)
- - DATA shutw (Y/N)
-
-(note that having flags for operations needing to be completed might be easier)
---------------
-
-Maybe we need to be able to call conn->fdset() and conn->fdclr() but it sounds
-very unlikely since the only functions manipulating this are in the code of
-the data/ctrl handlers.
-
-FDSET/FDCLR cannot be directly controlled by the stream interface since it also
-depends on the DATA layer (WANT_READ/WANT_WRITE).
-
-But FDSET/FDCLR is probably controlled by who owns the connection (eg: DATA).
-
-Example: an SSL conn relies on an FD. The buffer is full, and wants the conn to
-stop reading. It must not stop the FD itself. It is the read function which
-should notice that it has nothing to do with a read wake-up, which needs to
-disable reading.
-
-Conversely, when calling conn->chk_rcv(), the reader might get a WANT_READ or
-even WANT_WRITE and adjust the FDs accordingly.
-
-------------------------
-
-OK, the problem is simple : we don't manipulate the FD at the right level.
-We should have :
- ->connect(), ->chk_snd(), ->chk_rcv(), ->shutw(), ->shutr() which are
- called from the upper layer (buffer)
- ->recv(), ->send(), called from the lower layer
-
-Note that the SHR is *reported* by lower layer but can be forced by upper
-layer. In this case it's like a delayed abort. The difficulty consists in
-knowing the output data were correctly read. Probably we'd need to drain
-incoming data past the active shutr().
-
-The only four purposes of the top-down shutr() call are :
- - acknowledge a shut read report : could probably be done better
- - read timeout => disable reading : it's a delayed abort. We want to
- report that the buffer is SHR, maybe even the connection, but the
- FD clearly isn't.
- - read abort due to error on the other side or desire to close (eg:
- http-server-close) : delayed abort
- - complete abort
-
-The active shutr() is problematic as we can't disable reading if we expect some
-exchanges for data acknowledgement. We probably need to drain data only until
-the shutw() has been performed and ACKed.
-
-A connection shut down for read would behave like this :
-
- 1) bidir exchanges
-
- 2) shutr() => read_abort_pending=1
-
- 3) drain input, still send output
-
- 4) shutw()
-
- 5) drain input, wait for read0 or ack(shutw)
-
- 6) close()
-
---------------------- 2012/07/05 -------------------
-
-Communications must be performed this way :
-
- connection <-> channel <-> connection
-
-A channel is composed of flags and stats, and may store data in either a buffer
-or a pipe. We need low-layer operations between sockets and buffers or pipes.
-Right now we only support sockets, but later we might support remote sockets
-and maybe pipes or shared memory segments.
-
-So we need :
-
- - raw_sock_to_buf() => receive raw data from socket into buffer
- - raw_sock_to_pipe => receive raw data from socket into pipe (splice in)
- - raw_sock_from_buf() => send raw data from buffer to socket
- - raw_sock_from_pipe => send raw data from pipe to socket (splice out)
-
- - ssl_sock_to_buf() => receive ssl data from socket into buffer
- - ssl_sock_to_pipe => receive ssl data from socket into a pipe (NULL)
- - ssl_sock_from_buf() => send ssl data from buffer to socket
- - ssl_sock_from_pipe => send ssl data from pipe to socket (NULL)
-
-These functions should set such status flags :
-
-#define ERR_IN 0x01
-#define ERR_OUT 0x02
-#define SHUT_IN 0x04
-#define SHUT_OUT 0x08
-#define EMPTY_IN 0x10
-#define FULL_OUT 0x20
-
+++ /dev/null
-How it works ? (unfinished and inexact)
-
-For TCP and HTTP :
-
-- listeners create listening sockets with a READ callback pointing to the
- protocol-specific accept() function.
-
-- the protocol-specific accept() function then accept()'s the connection and
- instantiates a "server TCP socket" (which is dedicated to the client side),
- and configures it (non_block, get_original_dst, ...).
-
-For TCP :
-- in case of pure TCP, a request buffer is created, as well as a "client TCP
- socket", which tries to connect to the server.
-
-- once the connection is established, the response buffer is allocated and
- connected to both ends.
-
-- both sockets are set to "autonomous mode" so that they only wake up their
- supervising session when they encounter a special condition (error or close).
-
-
-For HTTP :
-- in case of HTTP, a request buffer is created with the "HOLD" flag set and
- a read limit to support header rewriting (may be this one will be removed
- eventually because it's better to limit only to the buffer size and report
- an error when rewritten data overflows)
-
-- a "flow analyzer" is attached to the buffer (or possibly multiple flow
- analyzers). For the request, the flow analyzer is "http_lb_req". The flow
- analyzer is a function which gets called when new data is present and
- blocked. It has a timeout (request timeout). It can also be bypassed on
- demand.
-
-- when the "http_lb_req" has received the whole request, it creates a client
- socket with all the parameters needed to try to connect to the server. When
- the connection establishes, the response buffer is allocated on the fly,
- put to HOLD mode, and a an "http_lb_resp" flow analyzer is attached to the
- buffer.
-
-
-For client-side HTTPS :
-
-- the accept() function must completely instantiate a TCP socket + an SSL
- reader. It is when the SSL session is complete that we call the
- protocol-specific accept(), and create its buffer.
-
-
-
-
-Conclusions
------------
-
-- we need a generic TCP accept() function with a lot of flags set by the
- listener, to tell it what info we need to get at the accept() time, and
- what flags will have to be set on the socket.
-
-- once the TCP accept() function ends, it wakes up the protocol supervisor
- which is in charge of creating the buffers, etc, switch states, etc...
-
+++ /dev/null
-2014/10/23 - design thoughts for HTTP/2
-
-- connections : HTTP/2 depends a lot more on a connection than HTTP/1 because a
- connection holds a compression context (headers table, etc...). We probably
- need to have an h2_conn struct.
-
-- multiple transactions will be handled in parallel for a given h2_conn. They
- are called streams in HTTP/2 terminology.
-
-- multiplexing : for a given client-side h2 connection, we can have multiple
- server-side h2 connections. And for a server-side h2 connection, we can have
- multiple client-side h2 connections. Streams circulate in N-to-N fashion.
-
-- flow control : flow control will be applied between multiple streams. Special
- care must be taken so that an H2 client cannot block some H2 servers by
- sending requests spread over multiple servers to the point where one server
- response is blocked and prevents other responses from the same server from
- reaching their clients. H2 connection buffers must always be empty or nearly
- empty. The per-stream flow control needs to be respected as well as the
- connection's buffers. It is important to implement some fairness between all
- the streams so that it's not always the same which gets the bandwidth when
- the connection is congested.
-
-- some clients can be H1 with an H2 server (is this really needed ?). Most of
- the initial use case will be H2 clients to H1 servers. It is important to keep
- in mind that H1 servers do not do flow control and that we don't want them to
- block transfers (eg: post upload).
-
-- internal tasks : some H2 clients will be internal tasks (eg: health checks).
- Some H2 servers will be internal tasks (eg: stats, cache). The model must be
- compatible with this use case.
-
-- header indexing : headers are transported compressed, with a reference to a
- static or a dynamic header, or a literal, possibly huffman-encoded. Indexing
- is specific to the H2 connection. This means there is no way any binary data
- can flow between both sides, headers will have to be decoded according to the
- incoming connection's context and re-encoded according to the outgoing
- connection's context, which can significantly differ. In order to avoid the
- parsing trouble we currently face, headers will have to be clearly split
- between name and value. It is worth noting that neither the incoming nor the
- outgoing connections' contexts will be of any use while processing the
- headers. At best we can have some shortcuts for well-known names that map
- well to the static ones (eg: use the first static entry with same name), and
- maybe have a few special cases for static name+value as well. Probably we can
- classify headers in such categories :
-
- - static name + value
- - static name + other value
- - dynamic name + other value
-
- This will allow for better processing in some specific cases. Headers
- supporting a single value (:method, :status, :path, ...) should probably
- be stored in a single location with a direct access. That would allow us
- to retrieve a method using hdr[METHOD]. All such indexing must be performed
- while parsing. That also means that HTTP/1 will have to be converted to this
- representation very early in the parser and possibly converted back to H/1
- after processing.
-
- Header names/values will have to be placed in a small memory area that will
- inevitably get fragmented as headers are rewritten. An automatic packing
- mechanism must be implemented so that when there's no more room, headers are
- simply defragmented/packet to a new table and the old one is released. Just
- like for the static chunks, we need to have a few such tables pre-allocated
- and ready to be swapped at any moment. Repacking must not change any index
- nor affect the way headers are compressed so that it can happen late after a
- retry (send-name-header for example).
-
-- header processing : can still happen on a (header, value) basis. Reqrep/
- rsprep completely disappear and will have to be replaced with something else
- to support renaming headers and rewriting url/path/...
-
-- push_promise : servers can push dummy requests+responses. They advertise
- the stream ID in the push_promise frame indicating the associated stream ID.
- This means that it is possible to initiate a client-server stream from the
- information coming from the server and make the data flow as if the client
- had made it. It's likely that we'll have to support two types of server
- connections: those which support push and those which do not. That way client
- streams will be distributed to existing server connections based on their
- capabilities. It's important to keep in mind that PUSH will not be rewritten
- in responses.
-
-- stream ID mapping : since the stream ID is per H2 connection, stream IDs will
- have to be mapped. Thus a given stream is an entity with two IDs (one per
- side). Or more precisely a stream has two end points, each one carrying an ID
- when it ends on an HTTP2 connection. Also, for each stream ID we need to
- quickly find the associated transaction in progress. Using a small quick
- unique tree seems indicated considering the wide range of valid values.
-
-- frame sizes : frame have to be remapped between both sides as multiplexed
- connections won't always have the same characteristics. Thus some frames
- might be spliced and others will be sliced.
-
-- error processing : care must be taken to never break a connection unless it
- is dead or corrupt at the protocol level. Stats counter must exist to observe
- the causes. Timeouts are a great problem because silent connections might
- die out of inactivity. Ping frames should probably be scheduled a few seconds
- before the connection timeout so that an unused connection is verified before
- being killed. Abnormal requests must be dealt with using RST_STREAM.
-
-- ALPN : ALPN must be observed on the client side, and transmitted to the server
- side.
-
-- proxy protocol : proxy protocol makes little to no sense in a multiplexed
- protocol. A per-stream equivalent will surely be needed if implementations
- do not quickly generalize the use of Forward.
-
-- simplified protocol for local devices (eg: haproxy->varnish in clear and
- without handshake, and possibly even with splicing if the connection's
- settings are shared)
-
-- logging : logging must report a number of extra information such as the
- stream ID, and whether the transaction was initiated by the client or by the
- server (which can be deduced from the stream ID's parity). In case of push,
- the number of the associated stream must also be reported.
-
-- memory usage : H2 increases memory usage by mandating use of 16384 bytes
- frame size minimum. That means slightly more than 16kB of buffer in each
- direction to process any frame. It will definitely have an impact on the
- deployed maxconn setting in places using less than this (4..8kB are common).
- Also, the header list is persistent per connection, so if we reach the same
- size as the request, that's another 16kB in each direction, resulting in
- about 48kB of memory where 8 were previously used. A more careful encoder
- can work with a much smaller set even if that implies evicting entries
- between multiple headers of the same message.
-
-- HTTP/1.0 should very carefully be transported over H2. Since there's no way
- to pass version information in the protocol, the server could use some
- features of HTTP/1.1 that are unsafe in HTTP/1.0 (compression, trailers,
- ...).
-
-- host / :authority : ":authority" is the norm, and "host" will be absent when
- H2 clients generate :authority. This probably means that a dummy Host header
- will have to be produced internally from :authority and removed when passing
- to H2 behind. This can cause some trouble when passing H2 requests to H1
- proxies, because there's no way to know if the request should contain scheme
- and authority in H1 or not based on the H2 request. Thus a "proxy" option
- will have to be explicitly mentioned on HTTP/1 server lines. One of the
- problem that it creates is that it's not longer possible to pass H/1 requests
- to H/1 proxies without an explicit configuration. Maybe a table of the
- various combinations is needed.
-
- :scheme :authority host
- HTTP/2 request present present absent
- HTTP/1 server req absent absent present
- HTTP/1 proxy req present present present
-
- So in the end the issue is only with H/2 requests passed to H/1 proxies.
-
-- ping frames : they don't indicate any stream ID so by definition they cannot
- be forwarded to any server. The H2 connection should deal with them only.
-
-There's a layering problem with H2. The framing layer has to be aware of the
-upper layer semantics. We can't simply re-encode HTTP/1 to HTTP/2 then pass
-it over a framing layer to mux the streams, the frame type must be passed below
-so that frames are properly arranged. Header encoding is connection-based and
-all streams using the same connection will interact in the way their headers
-are encoded. Thus the encoder *has* to be placed in the h2_conn entity, and
-this entity has to know for each stream what its headers are.
-
-Probably that we should remove *all* headers from transported data and move
-them on the fly to a parallel structure that can be shared between H1 and H2
-and consumed at the appropriate level. That means buffers only transport data.
-Trailers have to be dealt with differently.
-
-So if we consider an H1 request being forwarded between a client and a server,
-it would look approximately like this :
-
- - request header + body land into a stream's receive buffer
- - headers are indexed and stripped out so that only the body and whatever
- follows remain in the buffer
- - both the header index and the buffer with the body stay attached to the
- stream
- - the sender can rebuild the whole headers. Since they're found in a table
- supposed to be stable, it can rebuild them as many times as desired and
- will always get the same result, so it's safe to build them into the trash
- buffer for immediate sending, just as we do for the PROXY protocol.
- - the upper protocol should probably provide a build_hdr() callback which
- when called by the socket layer, builds this header block based on the
- current stream's header list, ready to be sent.
- - the socket layer has to know how many bytes from the headers are left to be
- forwarded prior to processing the body.
- - the socket layer needs to consume only the acceptable part of the body and
- must not release the buffer if any data remains in it (eg: pipelining over
- H1). This is already handled by channel->o and channel->to_forward.
- - we could possibly have another optional callback to send a preamble before
- data, that could be used to send chunk sizes in H1. The danger is that it
- absolutely needs to be stable if it has to be retried. But it could
- considerably simplify de-chunking.
-
-When the request is sent to an H2 server, an H2 stream request must be made
-to the server, we find an existing connection whose settings are compatible
-with our needs (eg: tls/clear, push/no-push), and with a spare stream ID. If
-none is found, a new connection must be established, unless maxconn is reached.
-
-Servers must have a maxstream setting just like they have a maxconn. The same
-queue may be used for that.
-
-The "tcp-request content" ruleset must apply to the TCP layer. But with HTTP/2
-that becomes impossible (and useless). We still need something like the
-"tcp-request session" hook to apply just after the SSL handshake is done.
-
-It is impossible to defragment the body on the fly in HTTP/2. Since multiple
-messages are interleaved, we cannot wait for all of them and block the head of
-line. Thus if body analysis is required, it will have to use the stream's
-buffer, which necessarily implies a copy. That means that with each H2 end we
-necessarily have at least one copy. Sometimes we might be able to "splice" some
-bytes from one side to the other without copying into the stream buffer (same
-rules as for TCP splicing).
-
-In theory, only data should flow through the channel buffer, so each side's
-connector is responsible for encoding data (H1: linear/chunks, H2: frames).
-Maybe the same mechanism could be extrapolated to tunnels / TCP.
-
-Since we'd use buffers only for data (and for receipt of headers), we need to
-have dynamic buffer allocation.
-
-Thus :
-- Tx buffers do not exist. We allocate a buffer on the fly when we're ready to
- send something that we need to build and that needs to be persistent in case
- of partial send. H1 headers are built on the fly from the header table to a
- temporary buffer that is immediately sent and whose amount of sent bytes is
- the only information kept (like for PROXY protocol). H2 headers are more
- complex since the encoding depends on what was successfully sent. Thus we
- need to build them and put them into a temporary buffer that remains
- persistent in case send() fails. It is possible to have a limited pool of
- Tx buffers and refrain from sending if there is no more buffer available in
- the pool. In that case we need a wake-up mechanism once a buffer is
- available. Once the data are sent, the Tx buffer is then immediately recycled
- in its pool. Note that no tx buffer being used (eg: for hdr or control) means
- that we have to be able to serialize access to the connection and retry with
- the same stream. It also means that a stream that times out while waiting for
- the connector to read the second half of its request has to stay there, or at
- least needs to be handled gracefully. However if the connector cannot read
- the data to be sent, it means that the buffer is congested and the connection
- is dead, so that probably means it can be killed.
-
-- Rx buffers have to be pre-allocated just before calling recv(). A connection
- will first try to pick a buffer and disable reception if it fails, then
- subscribe to the list of tasks waiting for an Rx buffer.
-
-- full Rx buffers might sometimes be moved around to the next buffer instead of
- experiencing a copy. That means that channels and connectors must use the
- same format of buffer, and that only the channel will have to see its
- pointers adjusted.
-
-- Tx of data should be made as much as possible without copying. That possibly
- means by directly looking into the connection buffer on the other side if
- the local Tx buffer does not exist and the stream buffer is not allocated, or
- even performing a splice() call between the two sides. One of the problem in
- doing this is that it requires proper ordering of the operations (eg: when
- multiple readers are attached to a same buffer). If the splitting occurs upon
- receipt, there's no problem. If we expect to retrieve data directly from the
- original buffer, it's harder since it contains various things in an order
- which does not even indicate what belongs to whom. Thus possibly the only
- mechanism to implement is the buffer permutation which guarantees zero-copy
- and only in the 100% safe case. Also it's atomic and does not cause HOL
- blocking.
-
-It makes sense to chose the frontend_accept() function right after the
-handshake ended. It is then possible to check the ALPN, the SNI, the ciphers
-and to accept to switch to the h2_conn_accept handler only if everything is OK.
-The h2_conn_accept handler will have to deal with the connection setup,
-initialization of the header table, exchange of the settings frames and
-preparing whatever is needed to fire new streams upon receipt of unknown
-stream IDs. Note: most of the time it will not be possible to splice() because
-we need to know in advance the amount of bytes to write the header, and here it
-will not be possible.
-
-H2 health checks must be seen as regular transactions/streams. The check runs a
-normal client which seeks an available stream from a server. The server then
-finds one on an existing connection or initiates a new H2 connection. The H2
-checks will have to be configurable for sharing streams or not. Another option
-could be to specify how many requests can be made over existing connections
-before insisting on getting a separate connection. Note that such separate
-connections might end up stacking up once released. So probably that they need
-to be recycled very quickly (eg: fix how many unused ones can exist max).
-
+++ /dev/null
-2010/01/24 - Design of multi-criteria request rate shaping.
-
-We want to be able to rate-shape traffic on multiple cirteria. For instance, we
-may want to support shaping of per-host header requests, as well as per source.
-
-In order to achieve this, we will use checkpoints, one per criterion. Each of
-these checkpoints will consist in a test, a rate counter and a queue.
-
-A request reaches the checkpoint and checks the counter. If the counter is
-below the limit, it is updated and the request continues. If the limit is
-reached, the request attaches itself into the queue and sleeps. The sleep time
-is computed from the queue status, and updates the queue status.
-
-A task is dedicated to each queue. Its sole purpose is to be woken up when the
-next task may wake up, to check the frequency counter, wake as many requests as
-possible and update the counter. All the woken up requests are detached from
-the queue. Maybe the task dedicated to the queue can be avoided and replaced
-with all queued tasks's sleep counters, though this looks tricky. Or maybe it's
-just the first request in the queue that should be responsible for waking up
-other tasks, and not to forget to pass on this responsibility to next tasks if
-it leaves the queue.
-
-The woken up request then goes on evaluating other criteria and possibly sleeps
-again on another one. In the end, the task will have waited the amount of time
-required to pass all checkpoints, and all checkpoints will be able to maintain
-a permanent load of exactly their limit if enough streams flow through them.
-
-Since a request can only sleep in one queue at a time, it makes sense to use a
-linked list element in each session to attach it to any queue. It could very
-well be shared with the pendconn hooks which could then be part of the session.
-
-This mechanism could be used to rate-shape sessions and requests per backend
-and per server.
-
-When rate-shaping on dynamic criteria, such as the source IP address, we have
-to first extract the data pattern, then look it up in a table very similar to
-the stickiness tables, but with a frequency counter. At the checkpoint, the
-pattern is looked up, the entry created or refreshed, and the frequency counter
-updated and checked. Then the request either goes on or sleeps as described
-above, but if it sleeps, it's still in the checkpoint's queue, but with a date
-computed from the criterion's status.
-
-This means that we need 3 distinct features :
-
- - optional pattern extraction
- - per-pattern or per-queue frequency counter
- - time-ordered queue with a task
-
-Based on past experiences with frequency counters, it does not appear very easy
-to exactly compute sleep delays in advance for multiple requests. So most
-likely we'll have to run per-criterion queues too, with only the head of the
-queue holding a wake-up timeout.
-
-This finally leads us to the following :
-
- - optional pattern extraction
- - per-pattern or per-queue frequency counter
- - per-frequency counter queue
- - head of the queue serves as a global queue timer.
-
-This brings us to a very flexible architecture :
- - 1 list of rule-based checkpoints per frontend
- - 1 list of rule-based checkpoints per backend
- - 1 list of rule-based checkpoints per server
-
-Each of these lists have a lot of rules conditioned by ACLs, just like the
-use-backend rules, except that all rules are evaluated in turn.
-
-Since we might sometimes just want to enable that without setting any limit and
-just for enabling control in ACLs (or logging ?), we should probably try to
-find a flexible way of declaring just a counter without a queue.
-
-These checkpoints could be of two types :
- - rate-limit (described here)
- - concurrency-limit (very similar with the counter and no timer). This
- feature would require to keep track of all accounted criteria in a
- request so that they can be released upon request completion.
-
-It should be possible to define a max of requests in the queue, above which a
-503 is returned. The same applies for the max delay in the queue. We could have
-it per-task (currently it's the connection timeout) and abort tasks with a 503
-when the delay is exceeded.
-
-Per-server connection concurrency could be converted to use this mechanism
-which is very similar.
-
-The construct should be flexible enough so that the counters may be checked
-from ACLs. That would allow to reject connections or switch to an alternate
-backend when some limits are reached.
-
+++ /dev/null
-Graphe des nombres de traitements par seconde unité de temps avec
- - un algo linéaire et très peu coûteux unitairement (0.01 ut)
- - un algo en log(2) et 5 fois plus coûteux (0.05 ut)
-
-set yrange [0:1]
-plot [0:1000] 1/(1+0.01*x), 1/(1+0.05*log(x+1)/log(2))
-
-Graphe de la latence induite par ces traitements en unités de temps :
-
-set yrange [0:1000]
-plot [0:1000] x/(1+0.01*x), x/(1+0.05*log(x+1)/log(2))
-
-
+++ /dev/null
-An FD has a state :
- - CLOSED
- - READY
- - ERROR (?)
- - LISTEN (?)
-
-A connection has a state :
- - CLOSED
- - ACCEPTED
- - CONNECTING
- - ESTABLISHED
- - ERROR
-
-A stream interface has a state :
- - INI, REQ, QUE, TAR, ASS, CON, CER, EST, DIS, CLO
-
-Note that CON and CER might be replaced by EST if the connection state is used
-instead. CON might even be more suited than EST to indicate that a connection
-is known.
-
-
-si_shutw() must do :
-
- data_shutw()
- if (shutr) {
- data_close()
- ctrl_shutw()
- ctrl_close()
- }
-
-si_shutr() must do :
- data_shutr()
- if (shutw) {
- data_close()
- ctrl_shutr()
- ctrl_close()
- }
-
-Each of these steps may fail, in which case the step must be retained and the
-operations postponed in an asynchronous task.
-
-The first asynchronous data_shut() might already fail so it is mandatory to
-save the other side's status with the connection in order to let the async task
-know whether the 3 next steps must be performed.
-
-The connection (or perhaps the FD) needs to know :
- - the desired close operations : DSHR, DSHW, CSHR, CSHW
- - the completed close operations : DSHR, DSHW, CSHR, CSHW
-
-
-On the accept() side, we probably need to know :
- - if a header is expected (eg: accept-proxy)
- - if this header is still being waited for
- => maybe both info might be combined into one bit
-
- - if a data-layer accept() is expected
- - if a data-layer accept() has been started
- - if a data-layer accept() has been performed
- => possibly 2 bits, to indicate the need to free()
-
-On the connect() side, we need to know :
- - the desire to send a header (eg: send-proxy)
- - if this header has been sent
- => maybe both info might be combined
-
- - if a data-layer connect() is expected
- - if a data-layer connect() has been started
- - if a data-layer connect() has been completed
- => possibly 2 bits, to indicate the need to free()
-
-On the response side, we also need to know :
- - the desire to send a header (eg: health check response for monitor-net)
- - if this header was sent
- => might be the same as sending a header over a new connection
-
-Note: monitor-net has precedence over proxy proto and data layers. Same for
- health mode.
-
-For multi-step operations, use 2 bits :
- 00 = operation not desired, not performed
- 10 = operation desired, not started
- 11 = operation desired, started but not completed
- 01 = operation desired, started and completed
-
- => X != 00 ==> operation desired
- X & 01 ==> operation at least started
- X & 10 ==> operation not completed
-
-Note: no way to store status information for error reporting.
-
-Note2: it would be nice if "tcp-request connection" rules could work at the
-connection level, just after headers ! This means support for tracking stick
-tables, possibly not too much complicated.
-
-
-Proposal for incoming connection sequence :
-
-- accept()
-- if monitor-net matches or if mode health => try to send response
-- if accept-proxy, wait for proxy request
-- if tcp-request connection, process tcp rules and possibly keep the
- pointer to stick-table
-- if SSL is enabled, switch to SSL handshake
-- then switch to DATA state and instantiate a session
-
-We just need a map of handshake handlers on the connection. They all manage the
-FD status themselves and set the callbacks themselves. If their work succeeds,
-they remove themselves from the list. If it fails, they remain subscribed and
-enable the required polling until they are woken up again or the timeout strikes.
-
-Identified handshake handlers for incoming connections :
- - HH_HEALTH (tries to send OK and dies)
- - HH_MONITOR_IN (matches src IP and adds/removes HH_SEND_OK/HH_SEND_HTTP_OK)
- - HH_SEND_OK (tries to send "OK" and dies)
- - HH_SEND_HTTP_OK (tries to send "HTTP/1.0 200 OK" and dies)
- - HH_ACCEPT_PROXY (waits for PROXY line and parses it)
- - HH_TCP_RULES (processes TCP rules)
- - HH_SSL_HS (starts SSL handshake)
- - HH_ACCEPT_SESSION (instantiates a session)
-
-Identified handshake handlers for outgoing connections :
- - HH_SEND_PROXY (tries to build and send the PROXY line)
- - HH_SSL_HS (starts SSL handshake)
-
-For the pollers, we could check that handshake handlers are not 0 and decide to
-call a generic connection handshake handler instead of usual callbacks. Problem
-is that pollers don't know connections, they know fds. So entities which manage
-handlers should update change the FD callbacks accordingly.
-
-With a bit of care, we could have :
- - HH_SEND_LAST_CHUNK (sends the chunk pointed to by a pointer and dies)
- => merges HEALTH, SEND_OK and SEND_HTTP_OK
-
-It sounds like the ctrl vs data state for the connection are per-direction
-(eg: support an async ctrl shutw while still reading data).
-
-Also support shutr/shutw status at L4/L7.
-
-In practice, what we really need is :
-
-shutdown(conn) =
- conn.data.shut()
- conn.ctrl.shut()
- conn.fd.shut()
-
-close(conn) =
- conn.data.close()
- conn.ctrl.close()
- conn.fd.close()
-
-With SSL over Remote TCP (RTCP + RSSL) to reach the server, we would have :
-
- HTTP -> RTCP+RSSL connection <-> RTCP+RRAW connection -> TCP+SSL connection
-
-The connection has to be closed at 3 places after a successful response :
- - DATA (RSSL over RTCP)
- - CTRL (RTCP to close connection to server)
- - SOCK (FD to close connection to second process)
-
-Externally, the connection is seen with very few flags :
- - SHR
- - SHW
- - ERR
-
-We don't need a CLOSED flag as a connection must always be detached when it's closed.
-
-The internal status doesn't need to be exposed :
- - FD allocated (Y/N)
- - CTRL initialized (Y/N)
- - CTRL connected (Y/N)
- - CTRL handlers done (Y/N)
- - CTRL failed (Y/N)
- - CTRL shutr (Y/N)
- - CTRL shutw (Y/N)
- - DATA initialized (Y/N)
- - DATA connected (Y/N)
- - DATA handlers done (Y/N)
- - DATA failed (Y/N)
- - DATA shutr (Y/N)
- - DATA shutw (Y/N)
-
-(note that having flags for operations needing to be completed might be easier)
---------------
-
-Maybe we need to be able to call conn->fdset() and conn->fdclr() but it sounds
-very unlikely since the only functions manipulating this are in the code of
-the data/ctrl handlers.
-
-FDSET/FDCLR cannot be directly controlled by the stream interface since it also
-depends on the DATA layer (WANT_READ/wANT_WRITE).
-
-But FDSET/FDCLR is probably controlled by who owns the connection (eg: DATA).
-
+++ /dev/null
-2011/02/25 - Description of the different entities in haproxy - w@1wt.eu
-
-
-1) Definitions
---------------
-
-Listener
---------
-
-A listener is the entity which is part of a frontend and which accepts
-connections. There are as many listeners as there are ip:port couples.
-There is at least one listener instantiated for each "bind" entry, and
-port ranges will lead to as many listeners as there are ports in the
-range. A listener just has a listening file descriptor ready to accept
-incoming connections and to dispatch them to upper layers.
-
-
-Initiator
----------
-
-An initiator is instantiated for each incoming connection on a listener. It may
-also be instantiated by a task pretending to be a client. An initiator calls
-the next stage's accept() callback to present it with the parameters of the
-incoming connection.
-
-
-Session
--------
-
-A session is the only entity located between an initiator and a connector.
-This is the last stage which offers an accept() callback, and all of its
-processing will continue with the next stage's connect() callback. It holds
-the buffers needed to forward the protocol data between each side. This entity
-sees the native protocol, and is able to call analysers on these buffers. As it
-is used in both directions, it always has two buffers.
-
-When transformations are required, some of them may be done on the initiator
-side and other ones on the connector side. If additional buffers are needed for
-such transforms, those buffers cannot replace the session's buffers, but they
-may complete them.
-
-A session only needs to be instantiated when forwarding of data is required
-between two sides. Accepting and filtering on layer 4 information only does not
-require a session.
-
-For instance, let's consider the case of a proxy which receives and decodes
-HTTPS traffic, processes it as HTTP and recodes it as HTTPS before forwarding
-it. We'd have 3 layers of buffers, where the middle ones are used for
-forwarding of the protocol data (HTTP here) :
-
- <-- ssl dec --> <-forwarding-> <-- ssl enc -->
-
- ,->[||||]--. ,->[||||]--. ,->[||||]--.
- client (|) (|) (|) (|) server
- ^--[||||]<-' ^--[||||]<-' ^--[||||]<-'
-
- HTTPS HTTP HTTPS
-
-The session handling code is only responsible for monitoring the forwarding
-buffers here. It may declare the end of the session once those buffers are
-closed and no analyser wants to re-open them. The session is also the entity
-which applies the load balancing algorithm and decides the server to use.
-
-The other sides are responsible for propagating the state up to the session
-which takes decisions.
-
-
-Connector
----------
-
-A connector is the entity which permits to instantiate a connection to a known
-destination. It presents a connect() callback, and as such appears on the right
-side of diagrams.
-
-
-Connection
-----------
-
-A connection is the entity instantiated by a connector. It may be composed of
-multiple stages linked together. Generally it is the part of the stream
-interface holding a file descriptor, but it can also be a processing block or a
-transformation block terminated by a connection. A connection presents a
-server-side interface.
-
-
-2) Sequencing
--------------
-
-Upon startup, listeners are instantiated by the configuration. When an incoming
-connection reaches a listening file descriptor, its read() callback calls the
-corresponding listener's accept() function which instantiates an initiator and
-in turn recursively calls upper layers' accept() callbacks until
-accept_session() is called. accept_session() instantiates a new session which
-starts protocol analysis via process_session(). When all protocol analysis is
-done, process_session() calls the connect() callback of the connector in order
-to get a connection.
+++ /dev/null
-TEST 3:
-
- printf "GET /\r\nbla: truc\r\n\r\n"
-
-
-NO SPEEDUP :
-
-WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x8071094
-WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x8071094
-WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071086, r=0x8071094
-WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071087, r=0x8071094
-WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071091, r=0x8071094
-WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071092, r=0x8071094
-WHL: hdr_st=0x34, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071093, r=0x8071094
-WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071093, r=0x8071094
-END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071094, r=0x8071094
-=> 9 trans
-
-
-FULL SPEEDUP :
-
-WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806a770, lr=0x806a770, r=0x806a784
-WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806a770, lr=0x806a776, r=0x806a784
-WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806a777, lr=0x806a777, r=0x806a784
-WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806a777, lr=0x806a781, r=0x806a784
-WHL: hdr_st=0x26, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806a782, lr=0x806a783, r=0x806a784
-END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806a782, lr=0x806a784, r=0x806a784
-=> 6 trans
-
-
-
-TEST 4:
-
-
- printf "GET /\nbla: truc\n\n"
-
-
-NO SPEEDUP :
-
-WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d0, r=0x80750e1
-WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d0, r=0x80750e1
-WHL: hdr_st=0x02, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d5, r=0x80750e1
-WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x80750d6, lr=0x80750d6, r=0x80750e1
-WHL: hdr_st=0x04, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x80750d6, lr=0x80750df, r=0x80750e1
-WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1
-WHL: hdr_st=0x04, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1
-WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1
-END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e1, r=0x80750e1
-=> 9 trans
-
-
-FULL SPEEDUP :
-
-WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8072010, lr=0x8072010, r=0x8072021
-WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8072016, lr=0x8072016, r=0x8072021
-END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8072020, lr=0x8072021, r=0x8072021
-=> 3 trans
-
-
-TEST 5:
-
-
- printf "GET /\r\nbla: truc\r\n truc2\r\n\r\n"
-
-
-NO SPEEDUP :
-
-WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x807109d
-WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x807109d
-WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071086, r=0x807109d
-WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071087, r=0x807109d
-WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071091, r=0x807109d
-WHL: hdr_st=0x05, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071092, r=0x807109d
-WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071094, r=0x807109d
-WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x807109a, r=0x807109d
-WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109b, r=0x807109d
-WHL: hdr_st=0x34, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109c, r=0x807109d
-WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109c, r=0x807109d
-END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109d, r=0x807109d
-=> 12 trans
-
-
-FULL SPEEDUP :
-
-WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806dfc0, lr=0x806dfc0, r=0x806dfdd
-WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806dfc0, lr=0x806dfc6, r=0x806dfdd
-WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfc7, r=0x806dfdd
-WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfd1, r=0x806dfdd
-WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfda, r=0x806dfdd
-WHL: hdr_st=0x26, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806dfdb, lr=0x806dfdc, r=0x806dfdd
-END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806dfdb, lr=0x806dfdd, r=0x806dfdd
-=> 7 trans
+++ /dev/null
-2007/03/30 - Header storage in trees
-
-This documentation describes how to store headers in radix trees, providing
-fast access to any known position, while retaining the ability to grow/reduce
-any arbitrary header without having to recompute all positions.
-
-Principle :
- We have a radix tree represented in an integer array, which represents the
- total number of bytes used by all headers whose position is below it. This
- ensures that we can compute any header's position in O(log(N)) where N is
- the number of headers.
-
-Example with N=16 :
-
- +-----------------------+
- | |
- +-----------+ +-----------+
- | | | |
- +-----+ +-----+ +-----+ +-----+
- | | | | | | | |
- +--+ +--+ +--+ +--+ +--+ +--+ +--+ +--+
- | | | | | | | | | | | | | | | |
-
- 0 1 2 3 4 5 6 7 8 9 A B C D E F
-
- To reach header 6, we have to compute hdr[0]+hdr[4]+hdr[6]
-
- With this method, it becomes easy to grow any header and update the array.
- To achieve this, we have to replace one after the other all bits on the
- right with one 1 followed by zeroes, and update the position if it's higher
- than current position, and stop when it's above number of stored headers.
-
- For instance, if we want to grow hdr[6], we proceed like this :
-
- 6 = 0110 (BIN)
-
- Let's consider the values to update :
-
- (bit 0) : (0110 & ~0001) | 0001 = 0111 = 7 > 6 => update
- (bit 1) : (0110 & ~0011) | 0010 = 0110 = 6 <= 6 => leave it
- (bit 2) : (0110 & ~0111) | 0100 = 0100 = 4 <= 6 => leave it
- (bit 4) : (0110 & ~1111) | 1000 = 1000 = 8 > 6 => update
- (bit 5) : larger than array size, stop.
-
-
-It's easy to walk through the tree too. We only have one iteration per bit
-changing from X to the ancestor, and one per bit from the ancestor to Y.
-The ancestor is found while walking. To go from X to Y :
-
- pos = pos(X)
-
- while (Y != X) {
- if (Y > X) {
- // walk from Y to ancestor
- pos += hdr[Y]
- Y &= (Y - 1)
- } else {
- // walk from X to ancestor
- pos -= hdr[X]
- X &= (X - 1)
- }
- }
-
-However, it is not trivial anymore to linearly walk the tree. We have to move
-from a known place to another known place, but a jump to next entry costs the
-same as a jump to a random place.
-
-Other caveats :
- - it is not possible to remove a header, it is only possible to empty it.
- - it is not possible to insert a header, as that would imply a renumbering.
- => this means that a "defrag" function is required. Headers should preferably
- be added, then should be stuffed on top of destroyed ones, then only
- inserted if absolutely required.
-
-
-When we have this, we can then focus on a 32-bit header descriptor which would
-look like this :
-
-{
- unsigned line_len :13; /* total line length, including CRLF */
- unsigned name_len :6; /* header name length, max 63 chars */
- unsigned sp1 :5; /* max spaces before value : 31 */
- unsigned sp2 :8; /* max spaces after value : 255 */
-}
-
-Example :
-
- Connection: close \r\n
- <---------+-----+-----+-------------> line_len
- <-------->| | | name_len
- <-----> | sp1
- <-------------> sp2
-Rem:
- - if there are more than 31 spaces before the value, the buffer will have to
- be moved before being registered
-
- - if there are more than 255 spaces after the value, the buffer will have to
- be moved before being registered
-
- - we can use the empty header name as an indicator for a deleted header
-
- - it would be wise to format a new request before sending lots of random
- spaces to the servers.
-
- - normal clients do not send such crap, so those operations *may* reasonably
- be more expensive than the rest provided that other ones are very fast.
-
-It would be handy to have the following macros :
-
- hdr_eon(hdr) => end of name
- hdr_sov(hdr) => start of value
- hdr_eof(hdr) => end of value
- hdr_vlen(hdr) => length of value
- hdr_hlen(hdr) => total header length
-
-
-A 48-bit encoding would look like this :
-
- Connection: close \r\n
- <---------+------+---+--------------> eoh = 16 bits
- <-------->| | | eon = 8 bits
- <--------------->| | sov = 8 bits
- <---> vlen = 16 bits
-
+++ /dev/null
-2010/08/31 - HTTP Cookies - Theory and reality
-
-HTTP cookies are not uniformly supported across browsers, which makes it very
-hard to build a widely compatible implementation. At least four conflicting
-documents exist to describe how cookies should be handled, and browsers
-generally don't respect any but a sensibly selected mix of them :
-
- - Netscape's original spec (also mirrored at Curl's site among others) :
- http://web.archive.org/web/20070805052634/http://wp.netscape.com/newsref/std/cookie_spec.html
- http://curl.haxx.se/rfc/cookie_spec.html
-
- Issues: uses an unquoted "Expires" field that includes a comma.
-
- - RFC 2109 :
- http://www.ietf.org/rfc/rfc2109.txt
-
- Issues: specifies use of "Max-Age" (not universally implemented) and does
- not talk about "Expires" (generally supported). References quoted
- strings, not generally supported (eg: MSIE). Stricter than browsers
- about domains. Ambiguous about allowed spaces in values and attrs.
-
- - RFC 2965 :
- http://www.ietf.org/rfc/rfc2965.txt
-
- Issues: same as RFC2109 + describes Set-Cookie2 which only Opera supports.
-
- - Current internet draft :
- https://datatracker.ietf.org/wg/httpstate/charter/
-
- Issues: as of -p10, does not explain how the Set-Cookie2 header must be
- emitted/handled, while suggesting a stricter approach for Cookie.
- Documents reality and as such reintroduces the widely used unquoted
- "Expires" attribute with its error-prone syntax. States that a
- server should not emit more than one cookie per Set-Cookie header,
- which is incompatible with HTTP which says that multiple headers
- are allowed only if they can be folded.
-
-See also the following URL for a browser * feature matrix :
- http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy_for_cookies
-
-In short, MSIE and Safari neither support quoted strings nor max-age, which
-make it mandatory to continue to send an unquoted Expires value (maybe the
-day of week could be omitted though). Only Safari supports comma-separated
-lists of Set-Cookie headers. Support for cross-domains is not uniform either.
-
+++ /dev/null
-Many interesting RFC and drafts linked to from this site :
-
- http://www.web-cache.com/Writings/protocols-standards.html
-
-
+++ /dev/null
---- Relevant portions of RFC2616 ---
-
-OCTET = <any 8-bit sequence of data>
-CHAR = <any US-ASCII character (octets 0 - 127)>
-UPALPHA = <any US-ASCII uppercase letter "A".."Z">
-LOALPHA = <any US-ASCII lowercase letter "a".."z">
-ALPHA = UPALPHA | LOALPHA
-DIGIT = <any US-ASCII digit "0".."9">
-CTL = <any US-ASCII control character (octets 0 - 31) and DEL (127)>
-CR = <US-ASCII CR, carriage return (13)>
-LF = <US-ASCII LF, linefeed (10)>
-SP = <US-ASCII SP, space (32)>
-HT = <US-ASCII HT, horizontal-tab (9)>
-<"> = <US-ASCII double-quote mark (34)>
-CRLF = CR LF
-LWS = [CRLF] 1*( SP | HT )
-TEXT = <any OCTET except CTLs, but including LWS>
-HEX = "A" | "B" | "C" | "D" | "E" | "F"
- | "a" | "b" | "c" | "d" | "e" | "f" | DIGIT
-separators = "(" | ")" | "<" | ">" | "@"
- | "," | ";" | ":" | "\" | <">
- | "/" | "[" | "]" | "?" | "="
- | "{" | "}" | SP | HT
-token = 1*<any CHAR except CTLs or separators>
-
-quoted-pair = "\" CHAR
-ctext = <any TEXT excluding "(" and ")">
-qdtext = <any TEXT except <">>
-quoted-string = ( <"> *(qdtext | quoted-pair ) <"> )
-comment = "(" *( ctext | quoted-pair | comment ) ")"
-
-
-
-
-
-4 HTTP Message
-4.1 Message Types
-
-HTTP messages consist of requests from client to server and responses from
-server to client. Request (section 5) and Response (section 6) messages use the
-generic message format of RFC 822 [9] for transferring entities (the payload of
-the message). Both types of message consist of :
-
- - a start-line
- - zero or more header fields (also known as "headers")
- - an empty line (i.e., a line with nothing preceding the CRLF) indicating the
- end of the header fields
- - and possibly a message-body.
-
-
-HTTP-message = Request | Response
-
-start-line = Request-Line | Status-Line
-generic-message = start-line
- *(message-header CRLF)
- CRLF
- [ message-body ]
-
-In the interest of robustness, servers SHOULD ignore any empty line(s) received
-where a Request-Line is expected. In other words, if the server is reading the
-protocol stream at the beginning of a message and receives a CRLF first, it
-should ignore the CRLF.
-
-
-4.2 Message headers
-
-- Each header field consists of a name followed by a colon (":") and the field
- value.
-- Field names are case-insensitive.
-- The field value MAY be preceded by any amount of LWS, though a single SP is
- preferred.
-- Header fields can be extended over multiple lines by preceding each extra
- line with at least one SP or HT.
-
-
-message-header = field-name ":" [ field-value ]
-field-name = token
-field-value = *( field-content | LWS )
-field-content = <the OCTETs making up the field-value and consisting of
- either *TEXT or combinations of token, separators, and
- quoted-string>
-
-
-The field-content does not include any leading or trailing LWS occurring before
-the first non-whitespace character of the field-value or after the last
-non-whitespace character of the field-value. Such leading or trailing LWS MAY
-be removed without changing the semantics of the field value. Any LWS that
-occurs between field-content MAY be replaced with a single SP before
-interpreting the field value or forwarding the message downstream.
-
-
-=> format des headers = 1*(CHAR & !ctl & !sep) ":" *(OCTET & (!ctl | LWS))
-=> les regex de matching de headers s'appliquent sur field-content, et peuvent
- utiliser field-value comme espace de travail (mais de préférence après le
- premier SP).
-
-(19.3) The line terminator for message-header fields is the sequence CRLF.
-However, we recommend that applications, when parsing such headers, recognize
-a single LF as a line terminator and ignore the leading CR.
-
-
-
-
-
-message-body = entity-body
- | <entity-body encoded as per Transfer-Encoding>
-
-
-
-5 Request
-
-Request = Request-Line
- *(( general-header
- | request-header
- | entity-header ) CRLF)
- CRLF
- [ message-body ]
-
-
-
-5.1 Request line
-
-The elements are separated by SP characters. No CR or LF is allowed except in
-the final CRLF sequence.
-
-Request-Line = Method SP Request-URI SP HTTP-Version CRLF
-
-(19.3) Clients SHOULD be tolerant in parsing the Status-Line and servers
-tolerant when parsing the Request-Line. In particular, they SHOULD accept any
-amount of SP or HT characters between fields, even though only a single SP is
-required.
-
-4.5 General headers
-Apply to MESSAGE.
-
-general-header = Cache-Control
- | Connection
- | Date
- | Pragma
- | Trailer
- | Transfer-Encoding
- | Upgrade
- | Via
- | Warning
-
-General-header field names can be extended reliably only in combination with a
-change in the protocol version. However, new or experimental header fields may
-be given the semantics of general header fields if all parties in the
-communication recognize them to be general-header fields. Unrecognized header
-fields are treated as entity-header fields.
-
-
-
-
-5.3 Request Header Fields
-
-The request-header fields allow the client to pass additional information about
-the request, and about the client itself, to the server. These fields act as
-request modifiers, with semantics equivalent to the parameters on a programming
-language method invocation.
-
-request-header = Accept
- | Accept-Charset
- | Accept-Encoding
- | Accept-Language
- | Authorization
- | Expect
- | From
- | Host
- | If-Match
- | If-Modified-Since
- | If-None-Match
- | If-Range
- | If-Unmodified-Since
- | Max-Forwards
- | Proxy-Authorization
- | Range
- | Referer
- | TE
- | User-Agent
-
-Request-header field names can be extended reliably only in combination with a
-change in the protocol version. However, new or experimental header fields MAY
-be given the semantics of request-header fields if all parties in the
-communication recognize them to be request-header fields. Unrecognized header
-fields are treated as entity-header fields.
-
-
-
-7.1 Entity header fields
-
-Entity-header fields define metainformation about the entity-body or, if no
-body is present, about the resource identified by the request. Some of this
-metainformation is OPTIONAL; some might be REQUIRED by portions of this
-specification.
-
-entity-header = Allow
- | Content-Encoding
- | Content-Language
- | Content-Length
- | Content-Location
- | Content-MD5
- | Content-Range
- | Content-Type
- | Expires
- | Last-Modified
- | extension-header
-extension-header = message-header
-
-The extension-header mechanism allows additional entity-header fields to be
-defined without changing the protocol, but these fields cannot be assumed to be
-recognizable by the recipient. Unrecognized header fields SHOULD be ignored by
-the recipient and MUST be forwarded by transparent proxies.
-
-----------------------------------
-
-The format of Request-URI is defined by RFC3986 :
-
- URI = scheme ":" hier-part [ "?" query ] [ "#" fragment ]
-
- hier-part = "//" authority path-abempty
- / path-absolute
- / path-rootless
- / path-empty
-
- URI-reference = URI / relative-ref
-
- absolute-URI = scheme ":" hier-part [ "?" query ]
-
- relative-ref = relative-part [ "?" query ] [ "#" fragment ]
-
- relative-part = "//" authority path-abempty
- / path-absolute
- / path-noscheme
- / path-empty
-
- scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
-
- authority = [ userinfo "@" ] host [ ":" port ]
- userinfo = *( unreserved / pct-encoded / sub-delims / ":" )
- host = IP-literal / IPv4address / reg-name
- port = *DIGIT
-
- IP-literal = "[" ( IPv6address / IPvFuture ) "]"
-
- IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
-
- IPv6address = 6( h16 ":" ) ls32
- / "::" 5( h16 ":" ) ls32
- / [ h16 ] "::" 4( h16 ":" ) ls32
- / [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
- / [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
- / [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32
- / [ *4( h16 ":" ) h16 ] "::" ls32
- / [ *5( h16 ":" ) h16 ] "::" h16
- / [ *6( h16 ":" ) h16 ] "::"
-
- h16 = 1*4HEXDIG
- ls32 = ( h16 ":" h16 ) / IPv4address
- IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet
- dec-octet = DIGIT ; 0-9
- / %x31-39 DIGIT ; 10-99
- / "1" 2DIGIT ; 100-199
- / "2" %x30-34 DIGIT ; 200-249
- / "25" %x30-35 ; 250-255
-
- reg-name = *( unreserved / pct-encoded / sub-delims )
-
- path = path-abempty ; begins with "/" or is empty
- / path-absolute ; begins with "/" but not "//"
- / path-noscheme ; begins with a non-colon segment
- / path-rootless ; begins with a segment
- / path-empty ; zero characters
-
- path-abempty = *( "/" segment )
- path-absolute = "/" [ segment-nz *( "/" segment ) ]
- path-noscheme = segment-nz-nc *( "/" segment )
- path-rootless = segment-nz *( "/" segment )
- path-empty = 0<pchar>
-
- segment = *pchar
- segment-nz = 1*pchar
- segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
- ; non-zero-length segment without any colon ":"
-
- pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
-
- query = *( pchar / "/" / "?" )
-
- fragment = *( pchar / "/" / "?" )
-
- pct-encoded = "%" HEXDIG HEXDIG
-
- unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
- reserved = gen-delims / sub-delims
- gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
- sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
- / "*" / "+" / "," / ";" / "="
-
-=> so the list of allowed characters in a URI is :
-
- uri-char = unreserved / gen-delims / sub-delims / "%"
- = ALPHA / DIGIT / "-" / "." / "_" / "~"
- / ":" / "/" / "?" / "#" / "[" / "]" / "@"
- / "!" / "$" / "&" / "'" / "(" / ")" /
- / "*" / "+" / "," / ";" / "=" / "%"
-
-Note that non-ascii characters are forbidden ! Spaces and CTL are forbidden.
-Unfortunately, some products such as Apache allow such characters :-/
-
----- The correct way to do it ----
-
-- one http_session
- It is basically any transport session on which we talk HTTP. It may be TCP,
- SSL over TCP, etc... It knows a way to talk to the client, either the socket
- file descriptor or a direct access to the client-side buffer. It should hold
- information about the last accessed server so that we can guarantee that the
- same server can be used during a whole session if needed. A first version
- without optimal support for HTTP pipelining will have the client buffers tied
- to the http_session. It may be possible that it is not sufficient for full
- pipelining, but this will need further study. The link from the buffers to
- the backend should be managed by the http transaction (http_txn), provided
- that they are serialized. Each http_session, has 0 to N http_txn. Each
- http_txn belongs to one and only one http_session.
-
-- each http_txn has 1 request message (http_req), and 0 or 1 response message
- (http_rtr). Each of them has 1 and only one http_txn. An http_txn holds
- information such as the HTTP method, the URI, the HTTP version, the
- transfer-encoding, the HTTP status, the authorization, the req and rtr
- content-length, the timers, logs, etc... The backend and server which process
- the request are also known from the http_txn.
-
-- both request and response messages hold header and parsing information, such
- as the parsing state, start of headers, start of message, captures, etc...
-
+++ /dev/null
-Naming rules for manipulated objects and structures.
-
-Previously, there were ambiguities between sessions, transactions and requests,
-as well as in the way responses are noted ("resp", "rep", "rsp").
-
-Here is a proposal for a better naming scheme.
-
-The "session" is above the transport level, which means at ISO layer 5.
-We can talk about "http sessions" when we consider the entity which lives
-between the accept() and the close(), or the connect() and the close().
-
-=> This demonstrates that it is not possible to have the same http session from
- the client to the server.
-
-A session can carry one or multiple "transactions", which are each composed of
-one "request" and zero or one "response". Both "request" and "response" are
-described in RFC2616 as "HTTP messages". RFC2616 also seldom references the
-word "transaction" without explicitly defining it.
-
-An "HTTP message" is composed of a "start line" which can be either a
-"request line" or a "status line", followed by a number of "message headers"
-which can be either "request headers" or "response headers", and an "entity",
-itself composed of "entity headers" and an "entity body".Most probably,
-"message headers" and "entity headers" will always be processed together as
-"headers", while the "entity body" will design the payload.
-
-We must try to always use the same abbreviations when naming objects. Here are
-a few common ones :
-
- - txn : transaction
- - req : request
- - rtr : response to request
- - msg : message
- - hdr : header
- - ent : entity
- - bdy : body
- - sts : status
- - stt : state
- - idx : index
- - cli : client
- - srv : server
- - svc : service
- - ses : session
- - tsk : task
-
-Short names for unions or cascaded structs :
- - sl : start line
- - sl.rq : request line
- - sl.st : status line
- - cl : client
- - px : proxy
- - sv : server
- - st : state / status
-
+++ /dev/null
-- session : ajouter ->fiprm et ->beprm comme raccourcis
-- px->maxconn: ne s'applique qu'au FE. Pour le BE, on utilise fullconn,
- initialisé par défaut à la même chose que maxconn.
-
-
- \ from: proxy session server actuellement
-field \
-rules px->fiprm sess->fiprm -
-srv,cookies px->beprm sess->beprm srv->px
-options(log) px-> sess->fe -
-options(fe) px-> sess->fe -
-options(be) px->beprm sess->beprm srv->px
-captures px-> sess->fe - ->fiprm
-
-
-logs px-> sess->fe srv->px
-errorloc px-> sess->beprm|fe -
-maxconn px-> sess->fe - ->be
-fullconn px-> sess->beprm srv->px -
-
projects / thirdparty / haproxy.git / commitdiff
