From: Tom Christie Date: Thu, 4 Apr 2019 17:28:38 +0000 (+0100) Subject: Stuff X-Git-Tag: 0.0.1~3 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=30530d446d1ae2f0c3df9b25a67b919c2c00ba1a;p=thirdparty%2Fhttpx.git Stuff --- diff --git a/README.md b/README.md new file mode 100644 index 00000000..55687f8b --- /dev/null +++ b/README.md @@ -0,0 +1,82 @@ +# HTTPCore + +I started to dive into implementation and API design here. + +I know this isn't what you were suggesting with `requests-core`, but it'd be +worth you taking a slow look at this and seeing if there's anything that you +think is a no-go. + +`httpcore` provides the same proposed *functionality* as requests-core, but at a slightly +lower abstraction level. + +Rather than returning `Response` models, it returns the minimal possible +interface. There's no `response.text` or any other cleverness, `response.headers` +are plain byte-pair lists, rather than a headers datastructure etc... + +**The proposal here is that `httpcore` would be a silent-partner dependency of `requests3`, +taking the place of the existing `urllib3` dependency.** + +--- + +The benefits to my mind of this level of abstraction are that it is as +agnostic as possible to whatever request/response models are built on top +of it, and exposes only plain datastructures that reflect the network response. + +* An `encode/httpcore` package would be something I'd gladly maintain. The naming + makes sense to me, as there's no strictly implied relationship to `requests`, + although it would fulfil all the requirements for `requests3` to build on, + and would have a strict semver policy. +* An `encode/httpcore` package is something that would play in well to the + collaboratively sponsored OSS story that Encode is pitching. It'd provide what + you need for `requests3` without encroaching on the `requests` brand. + We'd position it similarly to how `urllib3` is positioned to `requests` now. + A focused, low-level networking library, that `requests` then builds the + developer-focused API on top of. +* The current implementation includes all the async API points. + The `PoolManger.request()` and `PoolManager.close()` methods are currently + stubbed-out. All the remaining implementation hangs off of those two points. +* Take a quick look over the test cases or the package itself to get a feel + for it. It's all type annotated, and should be easy to find your way around. +* I've not yet added corresponding sync API points to the implementation, but + they will come. +* There's [a chunk of work towards connection pooling here](https://github.com/encode/requests-async/blob/5ec2aa80bd4499997fa744f3be19a0bdeccbaeed/requests_async/connections.py). I've not had enough time to nail it yet, but it's got the broad brush-strokes. +* We would absolutely want to implement HTTP/2 support. +* Trio support is something that could *potentially* come later, but it needs to + be a secondary consideration. +* I think all the functionality required is stubbed out in the API, with two exceptions. + 1. I've not yet added any proxy configuration API. Haven't looked into that enough + yet. 2. I've not yet added any retry configuration API, since I havn't really + looked enough into which side of requests vs. urllib3 that sits on, or exactly how + urllib3 tackles retries, etc. +* I'd be planning to prioritize working on this from Mon 15th April. I don't think + it'd take too long to get it to a feature complete and API stable state. + (With the exception of the later HTTP/2 work, which I can't really assess yet.) + I probably don't have any time left before then - need to focus on what I'm + delivering to DjangoCon Europe over the rest of this week. +* To my mind the killer app for `requests3`/`httpcore` is a high-performance + proxy server / gateway service in Python. Pitching the growing ASGI ecosystem + is an important part of that story. +* I think there's enough headroom before PyCon to have something ready to pitch by then. + I could be involved in sprints remotely if there's areas we still need to fill in, + anyplace. + +```python +import httpcore + +response = await httpcore.request('GET', 'http://example.com') +assert response.status_code == 200 +assert response.body == b'Hello, world' +``` + +API... + +```python +response = await httpcore.request(method, url, [headers], [body], [stream]) +``` + +Explicit PoolManager... + +```python +async with httpcore.PoolManager([ssl], [timeout], [limits]) as pool: + response = await pool.request(method, url, [headers], [body], [stream]) +``` diff --git a/httpcore/__init__.py b/httpcore/__init__.py index f102a9ca..44bb53a4 100644 --- a/httpcore/__init__.py +++ b/httpcore/__init__.py @@ -1 +1,5 @@ +from .api import PoolManager, Response, request +from .config import PoolLimits, SSLConfig, TimeoutConfig +from .exceptions import ResponseClosed, StreamConsumed + __version__ = "0.0.1" diff --git a/httpcore/api.py b/httpcore/api.py new file mode 100644 index 00000000..c270d569 --- /dev/null +++ b/httpcore/api.py @@ -0,0 +1,132 @@ +import typing +from types import TracebackType + +from .config import ( + DEFAULT_POOL_LIMITS, + DEFAULT_SSL_CONFIG, + DEFAULT_TIMEOUT_CONFIG, + PoolLimits, + SSLConfig, + TimeoutConfig, +) +from .exceptions import ResponseClosed, StreamConsumed + + +async def request( + method: str, + url: str, + *, + headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (), + body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"", + stream: bool = False, + ssl: SSLConfig = DEFAULT_SSL_CONFIG, + timeout: TimeoutConfig = DEFAULT_TIMEOUT_CONFIG, +) -> "Response": + async with PoolManager(ssl=ssl, timeout=timeout) as pool: + return await pool.request( + method=method, url=url, headers=headers, body=body, stream=stream + ) + + +class PoolManager: + def __init__( + self, + *, + ssl: SSLConfig = DEFAULT_SSL_CONFIG, + timeout: TimeoutConfig = DEFAULT_TIMEOUT_CONFIG, + limits: PoolLimits = DEFAULT_POOL_LIMITS, + ): + self.ssl = ssl + self.timeout = timeout + self.limits = limits + self.is_closed = False + + async def request( + self, + method: str, + url: str, + *, + headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (), + body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"", + stream: bool = False, + ) -> "Response": + if stream: + async def streaming_body(): + yield b"Hello, " + yield b"world!" + return Response(200, body=streaming_body) + return Response(200, body=b"Hello, world!") + + async def close(self) -> None: + self.is_closed = True + + async def __aenter__(self) -> "PoolManager": + return self + + async def __aexit__( + self, + exc_type: typing.Type[BaseException] = None, + exc_value: BaseException = None, + traceback: TracebackType = None, + ) -> None: + await self.close() + + +class Response: + def __init__( + self, + status_code: int, + *, + headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (), + body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"", + on_close: typing.Callable = None, + ): + self.status_code = status_code + self.headers = list(headers) + self.on_close = on_close + self.is_closed = False + self.is_streamed = False + if isinstance(body, bytes): + self.is_closed = True + self.body = body + else: + self.body_aiter = body + + async def read(self) -> bytes: + if not hasattr(self, "body"): + body = b"" + async for part in self.stream(): + body += part + self.body = body + return self.body + + async def stream(self) -> typing.AsyncIterator[bytes]: + if hasattr(self, "body"): + yield self.body + else: + if self.is_streamed: + raise StreamConsumed() + if self.is_closed: + raise ResponseClosed() + self.is_streamed = True + async for part in self.body_aiter(): + yield part + await self.close() + + async def close(self) -> None: + if not self.is_closed: + self.is_closed = True + if self.on_close is not None: + await self.on_close() + + async def __aenter__(self) -> "Response": + return self + + async def __aexit__( + self, + exc_type: typing.Type[BaseException] = None, + exc_value: BaseException = None, + traceback: TracebackType = None, + ) -> None: + if not self.is_closed: + await self.close() diff --git a/httpcore/config.py b/httpcore/config.py new file mode 100644 index 00000000..aa0c0718 --- /dev/null +++ b/httpcore/config.py @@ -0,0 +1,54 @@ +import typing + + +class SSLConfig: + """ + SSL Configuration. + """ + + def __init__(self, *, cert: typing.Optional[str], verify: typing.Union[str, bool]): + self.cert = cert + self.verify = verify + + +class TimeoutConfig: + """ + Timeout values. + """ + + def __init__( + self, + timeout: float = None, + *, + connect_timeout: float = None, + read_timeout: float = None, + pool_timeout: float = None + ): + if timeout is not None: + # Specified as a single timeout value + assert connect_timeout is None + assert read_timeout is None + assert pool_timeout is None + connect_timeout = timeout + read_timeout = timeout + pool_timeout = timeout + + self.connect_timeout = connect_timeout + self.read_timeout = read_timeout + self.pool_timeout = pool_timeout + + +class PoolLimits: + """ + Limits on the number of connections in a connection pool. + """ + + def __init__(self, *, max_hosts: int, conns_per_host: int, hard_limit: bool): + self.max_hosts = max_hosts + self.conns_per_host = conns_per_host + self.hard_limit = hard_limit + + +DEFAULT_SSL_CONFIG = SSLConfig(cert=None, verify=True) +DEFAULT_TIMEOUT_CONFIG = TimeoutConfig(timeout=5.0) +DEFAULT_POOL_LIMITS = PoolLimits(max_hosts=10, conns_per_host=10, hard_limit=False) diff --git a/httpcore/exceptions.py b/httpcore/exceptions.py new file mode 100644 index 00000000..69892afe --- /dev/null +++ b/httpcore/exceptions.py @@ -0,0 +1,42 @@ +class Timeout(Exception): + """ + A base class for all timeouts. + """ + + +class ConnectTimeout(Timeout): + """ + Timeout while establishing a connection. + """ + + +class ReadTimeout(Timeout): + """ + Timeout while reading response data. + """ + + +class PoolTimeout(Timeout): + """ + Timeout while waiting to acquire a connection from the pool. + """ + + +class BadResponse(Exception): + """ + A malformed HTTP response. + """ + + +class StreamConsumed(Exception): + """ + Attempted to read or stream response content, but the content has already + been streamed. + """ + + +class ResponseClosed(Exception): + """ + Attempted to read or stream response content, but the request has been + closed without loading the body. + """ diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 00000000..16c77063 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,11 @@ +h11 + +# Testing +autoflake +black +codecov +isort +mypy +pytest +pytest-asyncio +pytest-cov diff --git a/scripts/install b/scripts/install new file mode 100755 index 00000000..d263b44f --- /dev/null +++ b/scripts/install @@ -0,0 +1,24 @@ +#!/bin/sh -e + +# Use the Python executable provided from the `-p` option, or a default. +[[ $1 = "-p" ]] && PYTHON=$2 || PYTHON="python3" + +MIN_VERSION="(3, 6)" +VERSION_OK=`"$PYTHON" -c "import sys; print(sys.version_info[0:2] >= $MIN_VERSION and '1' or '');"` + +if [[ -z "$VERSION_OK" ]] ; then + PYTHON_VERSION=`"$PYTHON" -c "import sys; print('%s.%s' % sys.version_info[0:2]);"` + DISP_MIN_VERSION=`"$PYTHON" -c "print('%s.%s' % $MIN_VERSION)"` + echo "ERROR: Python $PYTHON_VERSION detected, but $DISP_MIN_VERSION+ is required." + echo "Please upgrade your Python distribution to install Databases." + exit 1 +fi + +REQUIREMENTS="requirements.txt" +VENV="venv" +PIP="$VENV/bin/pip" + +set -x +"$PYTHON" -m venv "$VENV" +"$PIP" install -r "$REQUIREMENTS" +"$PIP" install -e . diff --git a/scripts/lint b/scripts/lint new file mode 100755 index 00000000..3ce97239 --- /dev/null +++ b/scripts/lint @@ -0,0 +1,15 @@ +#!/bin/sh -e + +export PREFIX="" +if [ -d 'venv' ] ; then + export PREFIX="venv/bin/" +fi + +set -x + +${PREFIX}autoflake --in-place --recursive httpcore tests +${PREFIX}black httpcore tests +${PREFIX}isort --multi-line=3 --trailing-comma --force-grid-wrap=0 --combine-as --line-width 88 --recursive --apply httpcore tests +${PREFIX}mypy httpcore --ignore-missing-imports --disallow-untyped-defs + +scripts/clean diff --git a/scripts/test b/scripts/test new file mode 100755 index 00000000..8a72089e --- /dev/null +++ b/scripts/test @@ -0,0 +1,12 @@ +#!/bin/sh -e + +export PACKAGE="httpcore" +export PREFIX="" +if [ -d 'venv' ] ; then + export PREFIX="venv/bin/" +fi + +set -x + +PYTHONPATH=. ${PREFIX}pytest --ignore venv --cov tests --cov ${PACKAGE} --cov-report= ${@} +${PREFIX}coverage report diff --git a/tests/test_api.py b/tests/test_api.py new file mode 100644 index 00000000..2d5df0f3 --- /dev/null +++ b/tests/test_api.py @@ -0,0 +1,96 @@ +import pytest + +import httpcore + + +@pytest.mark.asyncio +async def test_request(): + response = await httpcore.request("GET", "http://example.com") + assert response.status_code == 200 + assert response.body == b"Hello, world!" + assert response.is_closed + + +@pytest.mark.asyncio +async def test_read_response(): + response = await httpcore.request("GET", "http://example.com") + + assert response.status_code == 200 + assert response.body == b"Hello, world!" + assert response.is_closed + + body = await response.read() + + assert body == b"Hello, world!" + assert response.body == b"Hello, world!" + assert response.is_closed + + +@pytest.mark.asyncio +async def test_stream_response(): + response = await httpcore.request("GET", "http://example.com") + + assert response.status_code == 200 + assert response.body == b"Hello, world!" + assert response.is_closed + + body = b'' + async for part in response.stream(): + body += part + + assert body == b"Hello, world!" + assert response.body == b"Hello, world!" + assert response.is_closed + + +@pytest.mark.asyncio +async def test_read_streaming_response(): + response = await httpcore.request("GET", "http://example.com", stream=True) + + assert response.status_code == 200 + assert not hasattr(response, 'body') + assert not response.is_closed + + body = await response.read() + + assert body == b"Hello, world!" + assert response.body == b"Hello, world!" + assert response.is_closed + + +@pytest.mark.asyncio +async def test_stream_streaming_response(): + response = await httpcore.request("GET", "http://example.com", stream=True) + + assert response.status_code == 200 + assert not hasattr(response, 'body') + assert not response.is_closed + + body = b'' + async for part in response.stream(): + body += part + + assert body == b"Hello, world!" + assert not hasattr(response, 'body') + assert response.is_closed + + +@pytest.mark.asyncio +async def test_cannot_read_after_stream_consumed(): + response = await httpcore.request("GET", "http://example.com", stream=True) + + body = b'' + async for part in response.stream(): + body += part + + with pytest.raises(httpcore.StreamConsumed): + await response.read() + +@pytest.mark.asyncio +async def test_cannot_read_after_response_closed(): + response = await httpcore.request("GET", "http://example.com", stream=True) + + await response.close() + + with pytest.raises(httpcore.ResponseClosed): + await response.read()