From: Senthil Kumaran Date: Sun, 31 Jul 2016 06:39:06 +0000 (-0700) Subject: [merge from 3.4] - Prevent HTTPoxy attack (CVE-2016-1000110) X-Git-Tag: v3.6.0a4~85^2^2 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=17742f2d45c9dd7ca777e33601a26e80576fdbf6;p=thirdparty%2FPython%2Fcpython.git [merge from 3.4] - Prevent HTTPoxy attack (CVE-2016-1000110) Ignore the HTTP_PROXY variable when REQUEST_METHOD environment is set, which indicates that the script is in CGI mode. Issue #27568 Reported and patch contributed by Rémi Rampin. --- 17742f2d45c9dd7ca777e33601a26e80576fdbf6 diff --cc Doc/howto/urllib2.rst index 24a415604fd1,4a67b30d899f..d2c799196bf6 --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@@ -538,6 -538,11 +538,11 @@@ setting up a `Basic Authentication`_ ha through a proxy. However, this can be enabled by extending urllib.request as shown in the recipe [#]_. + .. note:: + - `HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; see - the documentation on :func:`~urllib.request.getproxies`. ++ ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; see ++ the documentation on :func:`~urllib.request.getproxies`. + Sockets and Layers ================== diff --cc Doc/library/urllib.request.rst index 6c1bfb8b1fab,44db3e132555..1291aebcd231 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@@ -170,9 -165,15 +170,19 @@@ The :mod:`urllib.request` module define in a case insensitive approach, for all operating systems first, and when it cannot find it, looks for proxy information from Mac OSX System Configuration for Mac OS X and Windows Systems Registry for Windows. + If both lowercase and uppercase environment variables exist (and disagree), + lowercase is preferred. - .. note:: ++ .. note:: ++ ++ If the environment variable ``REQUEST_METHOD`` is set, which usually ++ indicates your script is running in a CGI environment, the environment ++ variable ``HTTP_PROXY`` (uppercase ``_PROXY``) will be ignored. This is ++ because that variable can be injected by a client using the "Proxy:" HTTP ++ header. If you need to use an HTTP proxy in a CGI environment, either use ++ ``ProxyHandler`` explicitly, or make sure the variable name is in ++ lowercase (or at least the ``_proxy`` suffix). + - If the environment variable ``REQUEST_METHOD`` is set, which usually - indicates your script is running in a CGI environment, the environment - variable ``HTTP_PROXY`` (uppercase ``_PROXY``) will be ignored. This is - because that variable can be injected by a client using the "Proxy:" HTTP - header. If you need to use an HTTP proxy in a CGI environment use - ``ProxyHandler`` explicitly. The following classes are provided: @@@ -275,11 -276,11 +285,16 @@@ To disable autodetected proxy pass an empty dictionary. - .. note:: + The :envvar:`no_proxy` environment variable can be used to specify hosts + which shouldn't be reached via proxy; if set, it should be a comma-separated + list of hostname suffixes, optionally with ``:port`` appended, for example + ``cern.ch,ncsa.uiuc.edu,some.host:8080``. - ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; - see the documentation on :func:`~urllib.request.getproxies`. ++ .. note:: ++ ++ ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; ++ see the documentation on :func:`~urllib.request.getproxies`. + .. class:: HTTPPasswordMgr() diff --cc Lib/test/test_urllib.py index 5d05f8d7d26d,87171e9b7b17..c26c52a6c5a8 --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@@ -227,59 -219,21 +227,71 @@@ class ProxyTests(unittest.TestCase) # getproxies_environment use lowered case truncated (no '_proxy') keys self.assertEqual('localhost', proxies['no']) # List of no_proxies with space. - self.env.set('NO_PROXY', 'localhost, anotherdomain.com, newdomain.com') + self.env.set('NO_PROXY', 'localhost, anotherdomain.com, newdomain.com:1234') self.assertTrue(urllib.request.proxy_bypass_environment('anotherdomain.com')) + self.assertTrue(urllib.request.proxy_bypass_environment('anotherdomain.com:8888')) + self.assertTrue(urllib.request.proxy_bypass_environment('newdomain.com:1234')) + def test_proxy_cgi_ignore(self): + try: + self.env.set('HTTP_PROXY', 'http://somewhere:3128') + proxies = urllib.request.getproxies_environment() + self.assertEqual('http://somewhere:3128', proxies['http']) + self.env.set('REQUEST_METHOD', 'GET') + proxies = urllib.request.getproxies_environment() + self.assertNotIn('http', proxies) + finally: + self.env.unset('REQUEST_METHOD') + self.env.unset('HTTP_PROXY') + + def test_proxy_bypass_environment_host_match(self): + bypass = urllib.request.proxy_bypass_environment + self.env.set('NO_PROXY', + 'localhost, anotherdomain.com, newdomain.com:1234') + self.assertTrue(bypass('localhost')) + self.assertTrue(bypass('LocalHost')) # MixedCase + self.assertTrue(bypass('LOCALHOST')) # UPPERCASE + self.assertTrue(bypass('newdomain.com:1234')) + self.assertTrue(bypass('anotherdomain.com:8888')) + self.assertTrue(bypass('www.newdomain.com:1234')) + self.assertFalse(bypass('prelocalhost')) + self.assertFalse(bypass('newdomain.com')) # no port + self.assertFalse(bypass('newdomain.com:1235')) # wrong port + +class ProxyTests_withOrderedEnv(unittest.TestCase): + + def setUp(self): + # We need to test conditions, where variable order _is_ significant + self._saved_env = os.environ + # Monkey patch os.environ, start with empty fake environment + os.environ = collections.OrderedDict() + + def tearDown(self): + os.environ = self._saved_env + + def test_getproxies_environment_prefer_lowercase(self): + # Test lowercase preference with removal + os.environ['no_proxy'] = '' + os.environ['No_Proxy'] = 'localhost' + self.assertFalse(urllib.request.proxy_bypass_environment('localhost')) + self.assertFalse(urllib.request.proxy_bypass_environment('arbitrary')) + os.environ['http_proxy'] = '' + os.environ['HTTP_PROXY'] = 'http://somewhere:3128' + proxies = urllib.request.getproxies_environment() + self.assertEqual({}, proxies) + # Test lowercase preference of proxy bypass and correct matching including ports + os.environ['no_proxy'] = 'localhost, noproxy.com, my.proxy:1234' + os.environ['No_Proxy'] = 'xyz.com' + self.assertTrue(urllib.request.proxy_bypass_environment('localhost')) + self.assertTrue(urllib.request.proxy_bypass_environment('noproxy.com:5678')) + self.assertTrue(urllib.request.proxy_bypass_environment('my.proxy:1234')) + self.assertFalse(urllib.request.proxy_bypass_environment('my.proxy')) + self.assertFalse(urllib.request.proxy_bypass_environment('arbitrary')) + # Test lowercase preference with replacement + os.environ['http_proxy'] = 'http://somewhere:3128' + os.environ['Http_Proxy'] = 'http://somewhereelse:3128' + proxies = urllib.request.getproxies_environment() + self.assertEqual('http://somewhere:3128', proxies['http']) class urlopen_HttpTests(unittest.TestCase, FakeHTTPMixin, FakeFTPMixin): """Test urlopen() opening a fake http connection.""" diff --cc Lib/urllib/request.py index 1731fe3df10b,f769386e0e47..3be327dd0063 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@@ -2412,29 -2337,22 +2412,35 @@@ def getproxies_environment() name = name.lower() if value and name[-6:] == '_proxy': proxies[name[:-6]] = value - + # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY + # (non-all-lowercase) as it may be set from the web server by a "Proxy:" + # header from the client ++ # If "proxy" is lowercase, it will still be used thanks to the next block + if 'REQUEST_METHOD' in os.environ: + proxies.pop('http', None) - + for name, value in os.environ.items(): + if name[-6:] == '_proxy': + name = name.lower() + if value: + proxies[name[:-6]] = value + else: + proxies.pop(name[:-6], None) return proxies -def proxy_bypass_environment(host): +def proxy_bypass_environment(host, proxies=None): """Test if proxies should not be used for a particular host. - Checks the environment for a variable named no_proxy, which should - be a list of DNS suffixes separated by commas, or '*' for all hosts. + Checks the proxy dict for the value of no_proxy, which should + be a list of comma separated DNS suffixes, or '*' for all hosts. + """ - no_proxy = os.environ.get('no_proxy', '') or os.environ.get('NO_PROXY', '') + if proxies is None: + proxies = getproxies_environment() + # don't bypass, if no_proxy isn't specified + try: + no_proxy = proxies['no'] + except KeyError: + return 0 # '*' is special case for always bypass if no_proxy == '*': return 1 diff --cc Misc/NEWS index 6b2b41921fec,362b6baf35b3..efe9b28874d6 --- a/Misc/NEWS +++ b/Misc/NEWS @@@ -34,91 -13,65 +34,95 @@@ Core and Builtin Library ------- + - Issue #27568: Prevent HTTPoxy attack (CVE-2016-1000110). Ignore the + HTTP_PROXY variable when REQUEST_METHOD environment is set, which indicates + that the script is in CGI mode. + -Tests ------ +- Issue #27130: In the "zlib" module, fix handling of large buffers + (typically 4 GiB) when compressing and decompressing. Previously, inputs + were limited to 4 GiB, and compression and decompression operations did not + properly handle results of 4 GiB. -- Issue #27369: In test_pyexpat, avoid testing an error message detail that - changed in Expat 2.2.0. +- Issue #27533: Release GIL in nt._isdir +- Issue #17711: Fixed unpickling by the persistent ID with protocol 0. + Original patch by Alexandre Vassalotti. -What's New in Python 3.4.5? -=========================== +- Issue #27522: Avoid an unintentional reference cycle in email.feedparser. -Release date: 2016-06-26 +- Issue #26844: Fix error message for imp.find_module() to refer to 'path' + instead of 'name'. Patch by Lev Maximov. -Tests ------ +- Issue #23804: Fix SSL zero-length recv() calls to not block and not raise + an error about unclean EOF. -- Issue #26867: Ubuntu's openssl OP_NO_SSLv3 is forced on by default; fix test. +- Issue #27466: Change time format returned by http.cookie.time2netscape, + confirming the netscape cookie format and making it consistent with + documentation. +- Issue #26664: Fix activate.fish by removing mis-use of ``$``. -What's New in Python 3.4.5rc1? -============================== +- Issue #22115: Fixed tracing Tkinter variables: trace_vdelete() with wrong + mode no longer break tracing, trace_vinfo() now always returns a list of + pairs of strings, tracing in the "u" mode now works. -Release date: 2016-06-11 +- Fix a scoping issue in importlib.util.LazyLoader which triggered an + UnboundLocalError when lazy-loading a module that was already put into + sys.modules. -Core and Builtins ------------------ +- Issue #27079: Fixed curses.ascii functions isblank(), iscntrl() and ispunct(). -- Issue #26478: Fix semantic bugs when using binary operators with dictionary - views and tuples. +- Issue #26754: Some functions (compile() etc) accepted a filename argument + encoded as an iterable of integers. Now only strings and byte-like objects + are accepted. -- Issue #26171: Fix possible integer overflow and heap corruption in - zipimporter.get_data(). +- Issue #27048: Prevents distutils failing on Windows when environment + variables contain non-ASCII characters -Library -------- +- Issue #27330: Fixed possible leaks in the ctypes module. -- Issue #26556: Update expat to 2.1.1, fixes CVE-2015-1283. +- Issue #27238: Got rid of bare excepts in the turtle module. Original patch + by Jelle Zijlstra. -- Fix TLS stripping vulnerability in smptlib, CVE-2016-0772. Reported by Team - Oststrom +- Issue #27122: When an exception is raised within the context being managed + by a contextlib.ExitStack() and one of the exit stack generators + catches and raises it in a chain, do not re-raise the original exception + when exiting, let the new chained one through. This avoids the PEP 479 + bug described in issue25782. -- Issue #25939: On Windows open the cert store readonly in ssl.enum_certificates. +- [Security] Issue #27278: Fix os.urandom() implementation using getrandom() on Linux. + Truncate size to INT_MAX and loop until we collected enough random bytes, + instead of casting a directly Py_ssize_t to int. -- Issue #26012: Don't traverse into symlinks for ** pattern in - pathlib.Path.[r]glob(). +- Issue #26386: Fixed ttk.TreeView selection operations with item id's + containing spaces. -- Issue #24120: Ignore PermissionError when traversing a tree with - pathlib.Path.[r]glob(). Patch by Ulrich Petri. +- [Security] Issue #22636: Avoid shell injection problems with + ctypes.util.find_library(). -- Skip getaddrinfo if host is already resolved. - Patch by A. Jesse Jiryu Davis. +- Issue #16182: Fix various functions in the "readline" module to use the + locale encoding, and fix get_begidx() and get_endidx() to return code point + indexes. -- Add asyncio.timeout() context manager. +- Issue #26930: Update Windows builds to use OpenSSL 1.0.2h. -- Issue #26050: Add asyncio.StreamReader.readuntil() method. - Patch by Марк Коренберг. +- Issue #27392: Add loop.connect_accepted_socket(). + Patch by Jim Fulton. + +IDLE +---- + +- Issue #27365: Allow non-ascii chars in IDLE NEWS.txt, for contributor names. + +- Issue #27245: IDLE: Cleanly delete custom themes and key bindings. + Previously, when IDLE was started from a console or by import, a cascade + of warnings was emitted. Patch by Serhiy Storchaka. + +C API +----- + +- Issue #26754: PyUnicode_FSDecoder() accepted a filename argument encoded as + an iterable of integers. Now only strings and bytes-like objects are accepted. Tests -----