--- /dev/null
+# HTTPCore
+
+I started to dive into implementation and API design here.
+
+I know this isn't what you were suggesting with `requests-core`, but it'd be
+worth you taking a slow look at this and seeing if there's anything that you
+think is a no-go.
+
+`httpcore` provides the same proposed *functionality* as requests-core, but at a slightly
+lower abstraction level.
+
+Rather than returning `Response` models, it returns the minimal possible
+interface. There's no `response.text` or any other cleverness, `response.headers`
+are plain byte-pair lists, rather than a headers datastructure etc...
+
+**The proposal here is that `httpcore` would be a silent-partner dependency of `requests3`,
+taking the place of the existing `urllib3` dependency.**
+
+---
+
+The benefits to my mind of this level of abstraction are that it is as
+agnostic as possible to whatever request/response models are built on top
+of it, and exposes only plain datastructures that reflect the network response.
+
+* An `encode/httpcore` package would be something I'd gladly maintain. The naming
+ makes sense to me, as there's no strictly implied relationship to `requests`,
+ although it would fulfil all the requirements for `requests3` to build on,
+ and would have a strict semver policy.
+* An `encode/httpcore` package is something that would play in well to the
+ collaboratively sponsored OSS story that Encode is pitching. It'd provide what
+ you need for `requests3` without encroaching on the `requests` brand.
+ We'd position it similarly to how `urllib3` is positioned to `requests` now.
+ A focused, low-level networking library, that `requests` then builds the
+ developer-focused API on top of.
+* The current implementation includes all the async API points.
+ The `PoolManger.request()` and `PoolManager.close()` methods are currently
+ stubbed-out. All the remaining implementation hangs off of those two points.
+* Take a quick look over the test cases or the package itself to get a feel
+ for it. It's all type annotated, and should be easy to find your way around.
+* I've not yet added corresponding sync API points to the implementation, but
+ they will come.
+* There's [a chunk of work towards connection pooling here](https://github.com/encode/requests-async/blob/5ec2aa80bd4499997fa744f3be19a0bdeccbaeed/requests_async/connections.py). I've not had enough time to nail it yet, but it's got the broad brush-strokes.
+* We would absolutely want to implement HTTP/2 support.
+* Trio support is something that could *potentially* come later, but it needs to
+ be a secondary consideration.
+* I think all the functionality required is stubbed out in the API, with two exceptions.
+ 1. I've not yet added any proxy configuration API. Haven't looked into that enough
+ yet. 2. I've not yet added any retry configuration API, since I havn't really
+ looked enough into which side of requests vs. urllib3 that sits on, or exactly how
+ urllib3 tackles retries, etc.
+* I'd be planning to prioritize working on this from Mon 15th April. I don't think
+ it'd take too long to get it to a feature complete and API stable state.
+ (With the exception of the later HTTP/2 work, which I can't really assess yet.)
+ I probably don't have any time left before then - need to focus on what I'm
+ delivering to DjangoCon Europe over the rest of this week.
+* To my mind the killer app for `requests3`/`httpcore` is a high-performance
+ proxy server / gateway service in Python. Pitching the growing ASGI ecosystem
+ is an important part of that story.
+* I think there's enough headroom before PyCon to have something ready to pitch by then.
+ I could be involved in sprints remotely if there's areas we still need to fill in,
+ anyplace.
+
+```python
+import httpcore
+
+response = await httpcore.request('GET', 'http://example.com')
+assert response.status_code == 200
+assert response.body == b'Hello, world'
+```
+
+API...
+
+```python
+response = await httpcore.request(method, url, [headers], [body], [stream])
+```
+
+Explicit PoolManager...
+
+```python
+async with httpcore.PoolManager([ssl], [timeout], [limits]) as pool:
+ response = await pool.request(method, url, [headers], [body], [stream])
+```
+from .api import PoolManager, Response, request
+from .config import PoolLimits, SSLConfig, TimeoutConfig
+from .exceptions import ResponseClosed, StreamConsumed
+
__version__ = "0.0.1"
--- /dev/null
+import typing
+from types import TracebackType
+
+from .config import (
+ DEFAULT_POOL_LIMITS,
+ DEFAULT_SSL_CONFIG,
+ DEFAULT_TIMEOUT_CONFIG,
+ PoolLimits,
+ SSLConfig,
+ TimeoutConfig,
+)
+from .exceptions import ResponseClosed, StreamConsumed
+
+
+async def request(
+ method: str,
+ url: str,
+ *,
+ headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (),
+ body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"",
+ stream: bool = False,
+ ssl: SSLConfig = DEFAULT_SSL_CONFIG,
+ timeout: TimeoutConfig = DEFAULT_TIMEOUT_CONFIG,
+) -> "Response":
+ async with PoolManager(ssl=ssl, timeout=timeout) as pool:
+ return await pool.request(
+ method=method, url=url, headers=headers, body=body, stream=stream
+ )
+
+
+class PoolManager:
+ def __init__(
+ self,
+ *,
+ ssl: SSLConfig = DEFAULT_SSL_CONFIG,
+ timeout: TimeoutConfig = DEFAULT_TIMEOUT_CONFIG,
+ limits: PoolLimits = DEFAULT_POOL_LIMITS,
+ ):
+ self.ssl = ssl
+ self.timeout = timeout
+ self.limits = limits
+ self.is_closed = False
+
+ async def request(
+ self,
+ method: str,
+ url: str,
+ *,
+ headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (),
+ body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"",
+ stream: bool = False,
+ ) -> "Response":
+ if stream:
+ async def streaming_body():
+ yield b"Hello, "
+ yield b"world!"
+ return Response(200, body=streaming_body)
+ return Response(200, body=b"Hello, world!")
+
+ async def close(self) -> None:
+ self.is_closed = True
+
+ async def __aenter__(self) -> "PoolManager":
+ return self
+
+ async def __aexit__(
+ self,
+ exc_type: typing.Type[BaseException] = None,
+ exc_value: BaseException = None,
+ traceback: TracebackType = None,
+ ) -> None:
+ await self.close()
+
+
+class Response:
+ def __init__(
+ self,
+ status_code: int,
+ *,
+ headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (),
+ body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"",
+ on_close: typing.Callable = None,
+ ):
+ self.status_code = status_code
+ self.headers = list(headers)
+ self.on_close = on_close
+ self.is_closed = False
+ self.is_streamed = False
+ if isinstance(body, bytes):
+ self.is_closed = True
+ self.body = body
+ else:
+ self.body_aiter = body
+
+ async def read(self) -> bytes:
+ if not hasattr(self, "body"):
+ body = b""
+ async for part in self.stream():
+ body += part
+ self.body = body
+ return self.body
+
+ async def stream(self) -> typing.AsyncIterator[bytes]:
+ if hasattr(self, "body"):
+ yield self.body
+ else:
+ if self.is_streamed:
+ raise StreamConsumed()
+ if self.is_closed:
+ raise ResponseClosed()
+ self.is_streamed = True
+ async for part in self.body_aiter():
+ yield part
+ await self.close()
+
+ async def close(self) -> None:
+ if not self.is_closed:
+ self.is_closed = True
+ if self.on_close is not None:
+ await self.on_close()
+
+ async def __aenter__(self) -> "Response":
+ return self
+
+ async def __aexit__(
+ self,
+ exc_type: typing.Type[BaseException] = None,
+ exc_value: BaseException = None,
+ traceback: TracebackType = None,
+ ) -> None:
+ if not self.is_closed:
+ await self.close()
--- /dev/null
+import typing
+
+
+class SSLConfig:
+ """
+ SSL Configuration.
+ """
+
+ def __init__(self, *, cert: typing.Optional[str], verify: typing.Union[str, bool]):
+ self.cert = cert
+ self.verify = verify
+
+
+class TimeoutConfig:
+ """
+ Timeout values.
+ """
+
+ def __init__(
+ self,
+ timeout: float = None,
+ *,
+ connect_timeout: float = None,
+ read_timeout: float = None,
+ pool_timeout: float = None
+ ):
+ if timeout is not None:
+ # Specified as a single timeout value
+ assert connect_timeout is None
+ assert read_timeout is None
+ assert pool_timeout is None
+ connect_timeout = timeout
+ read_timeout = timeout
+ pool_timeout = timeout
+
+ self.connect_timeout = connect_timeout
+ self.read_timeout = read_timeout
+ self.pool_timeout = pool_timeout
+
+
+class PoolLimits:
+ """
+ Limits on the number of connections in a connection pool.
+ """
+
+ def __init__(self, *, max_hosts: int, conns_per_host: int, hard_limit: bool):
+ self.max_hosts = max_hosts
+ self.conns_per_host = conns_per_host
+ self.hard_limit = hard_limit
+
+
+DEFAULT_SSL_CONFIG = SSLConfig(cert=None, verify=True)
+DEFAULT_TIMEOUT_CONFIG = TimeoutConfig(timeout=5.0)
+DEFAULT_POOL_LIMITS = PoolLimits(max_hosts=10, conns_per_host=10, hard_limit=False)
--- /dev/null
+class Timeout(Exception):
+ """
+ A base class for all timeouts.
+ """
+
+
+class ConnectTimeout(Timeout):
+ """
+ Timeout while establishing a connection.
+ """
+
+
+class ReadTimeout(Timeout):
+ """
+ Timeout while reading response data.
+ """
+
+
+class PoolTimeout(Timeout):
+ """
+ Timeout while waiting to acquire a connection from the pool.
+ """
+
+
+class BadResponse(Exception):
+ """
+ A malformed HTTP response.
+ """
+
+
+class StreamConsumed(Exception):
+ """
+ Attempted to read or stream response content, but the content has already
+ been streamed.
+ """
+
+
+class ResponseClosed(Exception):
+ """
+ Attempted to read or stream response content, but the request has been
+ closed without loading the body.
+ """
--- /dev/null
+h11
+
+# Testing
+autoflake
+black
+codecov
+isort
+mypy
+pytest
+pytest-asyncio
+pytest-cov
--- /dev/null
+#!/bin/sh -e
+
+# Use the Python executable provided from the `-p` option, or a default.
+[[ $1 = "-p" ]] && PYTHON=$2 || PYTHON="python3"
+
+MIN_VERSION="(3, 6)"
+VERSION_OK=`"$PYTHON" -c "import sys; print(sys.version_info[0:2] >= $MIN_VERSION and '1' or '');"`
+
+if [[ -z "$VERSION_OK" ]] ; then
+ PYTHON_VERSION=`"$PYTHON" -c "import sys; print('%s.%s' % sys.version_info[0:2]);"`
+ DISP_MIN_VERSION=`"$PYTHON" -c "print('%s.%s' % $MIN_VERSION)"`
+ echo "ERROR: Python $PYTHON_VERSION detected, but $DISP_MIN_VERSION+ is required."
+ echo "Please upgrade your Python distribution to install Databases."
+ exit 1
+fi
+
+REQUIREMENTS="requirements.txt"
+VENV="venv"
+PIP="$VENV/bin/pip"
+
+set -x
+"$PYTHON" -m venv "$VENV"
+"$PIP" install -r "$REQUIREMENTS"
+"$PIP" install -e .
--- /dev/null
+#!/bin/sh -e
+
+export PREFIX=""
+if [ -d 'venv' ] ; then
+ export PREFIX="venv/bin/"
+fi
+
+set -x
+
+${PREFIX}autoflake --in-place --recursive httpcore tests
+${PREFIX}black httpcore tests
+${PREFIX}isort --multi-line=3 --trailing-comma --force-grid-wrap=0 --combine-as --line-width 88 --recursive --apply httpcore tests
+${PREFIX}mypy httpcore --ignore-missing-imports --disallow-untyped-defs
+
+scripts/clean
--- /dev/null
+#!/bin/sh -e
+
+export PACKAGE="httpcore"
+export PREFIX=""
+if [ -d 'venv' ] ; then
+ export PREFIX="venv/bin/"
+fi
+
+set -x
+
+PYTHONPATH=. ${PREFIX}pytest --ignore venv --cov tests --cov ${PACKAGE} --cov-report= ${@}
+${PREFIX}coverage report
--- /dev/null
+import pytest
+
+import httpcore
+
+
+@pytest.mark.asyncio
+async def test_request():
+ response = await httpcore.request("GET", "http://example.com")
+ assert response.status_code == 200
+ assert response.body == b"Hello, world!"
+ assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_read_response():
+ response = await httpcore.request("GET", "http://example.com")
+
+ assert response.status_code == 200
+ assert response.body == b"Hello, world!"
+ assert response.is_closed
+
+ body = await response.read()
+
+ assert body == b"Hello, world!"
+ assert response.body == b"Hello, world!"
+ assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_stream_response():
+ response = await httpcore.request("GET", "http://example.com")
+
+ assert response.status_code == 200
+ assert response.body == b"Hello, world!"
+ assert response.is_closed
+
+ body = b''
+ async for part in response.stream():
+ body += part
+
+ assert body == b"Hello, world!"
+ assert response.body == b"Hello, world!"
+ assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_read_streaming_response():
+ response = await httpcore.request("GET", "http://example.com", stream=True)
+
+ assert response.status_code == 200
+ assert not hasattr(response, 'body')
+ assert not response.is_closed
+
+ body = await response.read()
+
+ assert body == b"Hello, world!"
+ assert response.body == b"Hello, world!"
+ assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_stream_streaming_response():
+ response = await httpcore.request("GET", "http://example.com", stream=True)
+
+ assert response.status_code == 200
+ assert not hasattr(response, 'body')
+ assert not response.is_closed
+
+ body = b''
+ async for part in response.stream():
+ body += part
+
+ assert body == b"Hello, world!"
+ assert not hasattr(response, 'body')
+ assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_cannot_read_after_stream_consumed():
+ response = await httpcore.request("GET", "http://example.com", stream=True)
+
+ body = b''
+ async for part in response.stream():
+ body += part
+
+ with pytest.raises(httpcore.StreamConsumed):
+ await response.read()
+
+@pytest.mark.asyncio
+async def test_cannot_read_after_response_closed():
+ response = await httpcore.request("GET", "http://example.com", stream=True)
+
+ await response.close()
+
+ with pytest.raises(httpcore.ResponseClosed):
+ await response.read()
projects / thirdparty / httpx.git / commitdiff
