mod_proxy_uwsgi: Fix PATH_INFO setting for generic worker.
When the generic "proxy:reverse" worker is selected for an uwsgi scheme, the
worker name is irrelevant so uwscgi_handler() should point to the PATH_INFO
directly from the given URL.
Stefan Eissing [Wed, 1 Sep 2021 13:16:03 +0000 (13:16 +0000)]
mod_md:
* Certificate/keys pairs are verified as matching before a renewal is accepted
as successful or a staged renewal is replacing the existing certificates.
This avoid potential mess ups in the md store file system to render the active
certificates non-working. [@mkauf]
Yann Ylavic [Mon, 30 Aug 2021 18:04:20 +0000 (18:04 +0000)]
mod_proxy: Fix potential tunneling infinite loop and spurious timeout.
PRs 65521 and 65519.
* modules/proxy/proxy_util.c(ap_proxy_tunnel_run):
Avoid an infinite loop by shutting down the connection for write when poll()
returns POLLHUP and read is already down. PR 65521.
* modules/proxy/proxy_util.c(ap_proxy_tunnel_run):
When write completion is finished don't check for ap_filter_input_pending()
before proxy_tunnel_forward() to flush input data, this is a nonblocking read
already which will do the same thing implicitely. ap_filter_input_pending()
is broken in 2.4.x without the whole pending data mechanism (not backported
yet), so let's align here. PR 65519.
Stefan Eissing [Mon, 30 Aug 2021 10:05:06 +0000 (10:05 +0000)]
* mod_deflate: refrain from reading buckets of known length, just
to get their length. This may transform buckets unwanted (e.g. file
to mmap) and prevent optimization down the filter chain.
Stefan Eissing [Thu, 26 Aug 2021 11:10:06 +0000 (11:10 +0000)]
test/modules/http2:
* removing unneeded python imports for smaller footprint
* fixing bug in waiting for stopped server to no longer
accept new connections.
Stefan Eissing [Wed, 25 Aug 2021 14:59:03 +0000 (14:59 +0000)]
mod_http2: fixes a use-after-read of an integer value when
passing a stream identifier for further IO checking. A
non-issue since an int value matching no active stream
will lead to no action.
Stefan Eissing [Wed, 25 Aug 2021 14:31:12 +0000 (14:31 +0000)]
* test/modules/http2: using stop/start instead of reload when changing apache configs
to give reliable results. The new reload behaviour keeps old children around until
very late and may answer on old configurations.
Yann Ylavic [Tue, 24 Aug 2021 22:22:40 +0000 (22:22 +0000)]
mpm_{event,worker,prefork}: late stop of children processes on restart.
Change how the main process handles restarts, from:
0. <restart signal>
1. stop old generation of children processes (graceful or not)
2. reload new configuration
3. start new generation of children processes
to:
0. <restart signal>
1. reload new configuration
2. stop old generation of children processes (graceful or not)
3. start new generation of children processes
The delay between stop and start is now very short and does not depend on the
reload time (which can be quite long with many vhosts and/or complex setups
with regexps or whatever third party components to compile).
Also, while reloading, the old generation of children processes keeps accepting
and handling incoming connections until the new generation is up to take over.
* os/unix/unixd.c (sig_term, sig_restart):
Set AP_MPMQ_STOPPING only once.
* server/listen.c (ap_duplicate_listeners):
Use ap_log_error() the main server instead of ap_log_perror().
* server/mpm/{event,worker,prefork}/{event,worker,prefork}.c
({event,worker,prefork}_retained_data):
Save the generation pool pointer (gen_pool) and all the buckets here, they
won't be cleared before the reload like pconf so they need a persitent
storage accross restarts (i.e. retained->gen_pool).
* server/mpm/{event,worker,prefork}/{event,worker,prefork}.c
(perform_idle_server_maintenance, child_main, make_child):
Change usage of all_buckets (previously with global/static scope) to the new
retained->buckets array.
Stefan Eissing [Fri, 20 Aug 2021 16:07:44 +0000 (16:07 +0000)]
* test/module/http2: test suite from github mod_h2 repository
slightly adapted to run in a source build. usage:
> make install # to have httpd, apxs etc. in place
> cd test
> pytest
Yann Ylavic [Fri, 20 Aug 2021 09:36:19 +0000 (09:36 +0000)]
core: follow up to r1891148: WC bucket defaulting to FLUSH bucket.
Define WC bucket semantics as:
/**
* @brief Write Completion (WC) bucket
*
* A WC bucket is a FLUSH bucket with special ->data == &ap_bucket_wc_data,
* still both AP_BUCKET_IS_WC() and APR_BUCKET_IS_FLUSH() hold for them so
* they have the same semantics for most filters, namely:
* Everything produced before shall be passed to the next filter, including
* the WC/FLUSH bucket itself.
* The distinction between WC and FLUSH buckets is only for filters that care
* about write completion (calling ap_filter_reinstate_brigade() with non-NULL
* flush_upto), those can setaside WC buckets and the preceding data provided
* they have first determined that the next filter(s) have pending data
* already, usually by calling ap_filter_should_yield(f->next).
*/
The only filters that care about write completion for now are
ap_core_output_filter() and ssl_io_filter_output(), which try to fill
in the pipe as much as possible, using ap_filter_reinstate_brigade(&flush_upto)
to determine whether they should flush (blocking) or setaside their remaining
data.
So ap_filter_reinstate_brigade() is made to not treat WC as FLUSH buckets and
keep the above filters working as before (and correctly w.r.t. above WC bucket
semantics).
* include/ap_mmn.h, include/util_filter.h:
Axe specific ap_bucket_type_wc and define global &ap_bucket_wc_data address to
mark WC buckets checked by AP_BUCKET_IS_WC().
* server/util_filter.c (ap_filter_reinstate_brigade):
Don't treat WC buckets as FLUSH buckets.
Yann Ylavic [Thu, 19 Aug 2021 16:04:45 +0000 (16:04 +0000)]
core: core output filter to cleanup empty/meta buckets following data, ASAP.
* server/core_filters.c (delete_meta_bucket):
Add delete_meta_bucket() helper to set r->flush on EOR.
* server/core_filters.c (send_brigade_nonblocking):
Use delete_meta_bucket() to factorize code.
* server/core_filters.c (writev_nonblocking):
Use delete_meta_bucket() to factorize code.
Cleanup trailing empty/meta buckets immediately after data have been sent to
get more accurate EOR timestamp (otherwise trailing buckets could be deleted
only after the next data or with a final send_brigade_nonblocking() call in
the output filter).
Improve fix to please a fuzzer which reports:
util.c:2713:26: runtime error: signed integer overflow: 9999999999999999 * 1000 cannot be represented in type 'long'
Compute the maximum limit for each case 's', 'h', 'ms' and 'mi' and make sure that the input is below this value.
While at it, move a comment to make things more consistent and use 'apr_time_from_msec() instead of hand writing it.
Stefan Eissing [Sat, 31 Jul 2021 13:36:19 +0000 (13:36 +0000)]
* core/mpm: add hook 'child_stopping` that gets called when the MPM is
stopping a child process. The additional `graceful` parameter allows
registered hooks to free resources early during a graceful shutdown.
* server/mpm/event/event.c (listener_thread): EVENT_FUDGE_FACTOR is used to
limit wakeups but still expiring timers should be compared with the real
apr_time_now().
mpm_event: clear APR_INCOMPLETE_READ on lingering close.
* server/mpm/event/event.c (process_lingering_close): APR_INCOMPLETE_READ may
have been set anytime while processing the connection, clear it before
nonblocking apr_socket_recv() to prevent internal apr_poll().
* server/connection.c (ap_lingering_close): Do apr_socket_close() on
ap_start_lingering_close() failure.
On failure ap_start_lingering_close() did not consistently close the socket,
so the caller had to call apr_socket_close() too with possible/unreliable
EBADF. The only upstream callers of ap_start_lingering_close() are MPM event
and ap_start_lingering_close(), but any third-party user has this issue so the
change should't break anyone.
mpm_event: handle lingering close fully in process_lingering_close().
* server/mpm/event/event.c (start_lingering_close_blocking): Axe the function
and put the code directly in process_lingering_close() on the first call when
cs->pub.state == CONN_STATE_LINGER, this simplifies lingering close handling
from process_socket() and event_resume_suspended().
mpm_event: Fix queues' maintenance and linger timeouts on graceful restart/stop
* server/mpm/event/event.c (abort_socket_nonblocking): Renamed to
close_socket_nonblocking_() with close_socket_nonblocking() as a macro to
APLOG_TRACE8 the caller __FUNCTION__ and __LINE__. Do nothing if the socket
is closed already, e.g. from close_worker_sockets().
* server/mpm/event/event.c (close_worker_sockets): Let defer_linger_chain be
cleaned by the terminating workers.
* server/mpm/event/event.c (wakeup_listener): Add a trace8 of the call.
* server/mpm/event/event.c (decrement_connection_count): Add a trace8 of the
call, decrement lingering_count from CONN_STATE_LINGER too which may happen
from start_lingering_close_blocking() on ap_start_lingering_close() failure,
and when dying call ap_queue_interrupt_one() from there instead of
close_connection() so that we don't miss a worker_thread_should_exit_early()
case.
* server/mpm/event/event.c (start_lingering_close_blocking): Don't increment
lingering_count when it's already done by defer_lingering_close() (i.e.
cs->deferred_linger is set), set socket timeout to 2s (SECONDS_TO_LINGER)
before pre_close hooks and then after let socket timeout handling to
process_lingering_close().
* server/mpm/event/event.c (start_lingering_close_nonblocking): Rename to
defer_lingering_close(), add trace6 of the call, set CONN_STATE_LINGER so
that the connection is accounted for until handled by a worker, and let
start_lingering_close_blocking() know by setting cs->deferred_linger.
* server/mpm/event/event.c (stop_lingering_close): Rename to close_connection()
and log the call at trace6 level instead of trace4 (for consistency), and use
it everywhere apr_socket_close()+ap_queue_info_push_pool() is open coded.
* server/mpm/event/event.c (shutdown_connection): This helper can be called
anywhere where nonblocking is required to either initiate short lingering
close or directly close if the connection is lingering already. It's used
as a generic process_timeout_queue() callback to terminate expired entries
in any state.
* server/mpm/event/event.c (update_reqevents_from_sense): Allow to specify the
sense as argument where -1 means cs->pub.state as before, and call it from
everywhere before adding to the pollset such that APR_POLL* events are set
fully/appropritely regardless of the underlying pollset implementation.
* server/mpm/event/event.c (process_socket): Gracefuly stop the process on
pollset errors, and don't forcibly close the connection if listener_may_exit
when ap_run_input_pending() determines that the next request is already here.
* server/mpm/event/event.c (close_listeners): Add trace6 of the call, and
return whether it's the first close or not.
* server/mpm/event/event.c (push2worker): Call shutdown_connection() on failure
when cs is available so that pre_close hooks (h2 flush, ssl close notify) are
still called.
* server/mpm/event/event.c (process_lingering_close): Add trace6 of the call,
always set timeout to zero (hence nonblocking) before reading pending data,
and terminate the process gracefuly on pollset errors.
* server/mpm/event/event.c (listener_thread): Add some trace7 messages around
poll()ing for debug, round poll()ing timeout to the upper millisecond to work
around the rounding down done by some pollset implementations, and on the
first call to close_listeners() when listener_may_exit->dying let's force the
maintenance of the queues to kill keep alive connections and apply shorter
lingering close timeout to free worker threads faster for the next generation.
* server/mpm/event/event.c (join_workers): Rewrite the !dying loop to avoid
"listener has not stopped accepting yet" if sleeping 500ms once did it.
Stefan Eissing [Tue, 20 Jul 2021 12:44:31 +0000 (12:44 +0000)]
*) mod_md:
- Domain names in `<MDomain ...>` can now appear in quoted form.
- Fixed a failure in ACME challenge selection that aborted further searches
when the tls-alpn-01 method did not seem to be suitable.
- Changed the tls-alpn-01 setup to only become unsuitable when none of the
dns names showed support for a configured 'Protocols ... acme-tls/1'. This
allows use of tls-alpn-01 for dns names that are not mapped to a VirtualHost.
Stefan Eissing [Thu, 15 Jul 2021 08:29:19 +0000 (08:29 +0000)]
* mod_http2:
- Added a timeout to h2 worker cleanup to exit latest after 5 seconds of
waiting on idle workers to terminate. This happens after all connections
have been processed. a WARNING is logged in case workers lagged behind.
mod_proxy: Fix icomplete initialization of BalancerMember(s) from the manager.
Clear the workers created in ap_proxy_sync_balancer(), notably ->local_status
for below ap_proxy_initialize_worker() to initialize all the child structures
like ->cp and ->cp->reslist, avoiding a possible crash when the workers are
used at runtime.
Stefan Eissing [Tue, 6 Jul 2021 13:06:00 +0000 (13:06 +0000)]
*) mod_http2:
- Aborting requests via RST_STREAM no longer affect the available
resources of a connection when the first chunk of the response
body has been sent.
- H2Min/MaxWorkers behave as intended again. The module will initially
create H2MinWorkers threads and add up to H2MaxWorkers when needed. These
additional workers time out when idle after H2MaxWorkerIdleSeconds and
disappear again.
- When the shutdown of a child is detected (e.g. graceful shutdown), the
module will terminate all idle workers above H2MinWorkers right away.
This detection currently only happens when a HTTP/2 connection is active.
mod_proxy: Avoid confusion of prefix/regex matching workers at loading. PR 65429.
ap_proxy_get_worker() needs to know whether it should lookup for prefix or
match or both matching workers, depending on the context.
For instance <Proxy[Match]> or ProxyPass[Match] directives need to lookup for
an existing worker with the same type as the directive (*Match or not), because
they will define one with that matching type if none exists.
On the contrary, "ProxySet <url>" at load time or ap_proxy_pre_request() at run
time need to find a worker matching an url whether it's by prefix or by regex.
So this commit adds ap_proxy_get_worker_ex() which takes a bitmask for the
matching type and calls it appropriately where needed.
For consistency, ap_proxy_define_worker_ex() is also added, using the same
bitmask flags, deprecating ap_proxy_define_match_worker().
mpm_proxy: Fix possible reuse/merging of Proxy(Pass)Match workers. PR 65419.
We can't truncate ProxyMatch's worker name/url to the first '$' substitution
without possibly colliding with other workers. This also makes the matching
done at runtime by ap_proxy_strcmp_ematch() completely pointless.
To fix this and still address r1878467 (i.e. make http://host:port$1 a "valid"
URL), we need to remove '$' substitutions from the :port part of the URL only
since it's allowed anywhere else by apr_uri_parse().
So let's strip them before apr_uri_parse() and prepend them back in the path
before apr_uri_unparse() to restore the original URL. Non-matchable workers are
not concerned so ap_proxy_define_worker() is made a local helper (w/o the ap_
prefix) which takes "matchable" as argument and can then be called by both
ap_proxy_define_[match_]worker() functions.
Yann Ylavic [Tue, 29 Jun 2021 21:16:21 +0000 (21:16 +0000)]
core: Write Completion (WC) bucket type.
A WC bucket is meant to prevent buffering/coalescing filters from retaining
data, but unlike a FLUSH bucket it won't cause the core output filter to
block trying to flush anything before.
It can be passed by async handlers which want to never block, followed by
ap_filter_should_yield() to check for pending data and eventually suspend
processing until MPM/asynchronous write completion finishes.
In this commit it's used that way by the tunneling loop of mod_proxy to
prevent SSL coaslescing.
Graham Leggett [Thu, 24 Jun 2021 10:27:49 +0000 (10:27 +0000)]
dbm: Split the loading of a dbm driver from the opening of a dbm file. When
an attempt to load a dbm driver fails, log clearly which driver triggered
the error (not "default"), and what the error was.
Stefan Eissing [Fri, 11 Jun 2021 10:45:25 +0000 (10:45 +0000)]
*) mod_ssl: tighten the handling of ALPN for outgoing (proxy)
connections. If ALPN protocols are provided and sent to the
remote server, the received protocol selected is inspected
and checked for a match. Without match, the peer handshake
fails.
An exception is the proposal of "http/1.1" where it is
accepted if the remote server did not answer ALPN with
a selected protocol. This accomodates for hosts that do
not observe/support ALPN and speak http/1.x be default.
Stefan Eissing [Tue, 8 Jun 2021 14:37:44 +0000 (14:37 +0000)]
*) core/mod_proxy/mod_ssl:
Adding `outgoing` flag to conn_rec, indicating a connection is
initiated by the server to somewhere, in contrast to incoming
connections from clients.
Adding 'ap_ssl_bind_outgoing()` function that marks a connection
as outgoing and is used by mod_proxy instead of the previous
optional function `ssl_engine_set`. This enables other SSL
module to secure proxy connections.
The optional functions `ssl_engine_set`, `ssl_engine_disable` and
`ssl_proxy_enable` are now provided by the core to have backward
compatibility with non-httpd modules that might use them. mod_ssl
itself no longer registers these functions, but keeps them in its
header for backward compatibility.
The core provided optional function wrap any registered function
like it was done for `ssl_is_ssl`.
Yann Ylavic [Fri, 4 Jun 2021 13:21:28 +0000 (13:21 +0000)]
mpm_prefork: mask signals during ap_run_child_init().
This prevents threads potentially created from the child_init hooks (e.g.
mod_watchdog workers) to catch signals needed by the MPM, like here:
https://travis-ci.com/github/apache/httpd/jobs/510821148#L5356.
projects / thirdparty / apache / httpd.git / log
