values are ``n``, ``no``, ``f``, ``false``, ``off`` and ``0``. Raises
:exc:`ValueError` if *val* is anything else.
-.. TODO Add :term: markup to bytecode when merging into the stdlib
.. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None])
If *force* is true, all files are recompiled regardless of timestamps.
- The source filename encoded in each bytecode file defaults to the filenames
+ The source filename encoded in each :term:`bytecode` file defaults to the filenames
listed in *py_files*; you can modify these with *prefix* and *basedir*.
*prefix* is a string that will be stripped off of each source filename, and
*base_dir* is a directory name that will be prepended (after *prefix* is
figure out to use direct compilation or not (see the source for details).
The *direct* flag is used by the script generated in indirect mode; unless
you know what you're doing, leave it set to ``None``.
+
+ This function is independent from the running Python's :option:`-O` or
+ :option:`-B` options; it is fully controlled by the parameters passed in.
have to create a separate installer for every Python version you want to
support.
-.. TODO Add :term: markup to bytecode when merging into the stdlib
-
-The installer will try to compile pure modules into bytecode after installation
+The installer will try to compile pure modules into :term:`bytecode` after installation
on the target system in normal and optimizing mode. If you don't want this to
happen for some reason, you can run the :command:`bdist_wininst` command with
the :option:`--no-target-compile` and/or the :option:`--no-target-optimize`
-------------------
Build the Python modules (just copy them to the build directory) and
-byte-compile them to .pyc files.
+:term:`byte-compile <bytecode>` them to :file:`.pyc` and/or :file:`.pyo` files.
+
+The byte compilation is controlled by two sets of options:
+
+- ``--compile`` and ``--no-compile`` are used to control the creation of
+ :file:`.pyc` files; the default is ``--no-compile``.
+
+- ``--optimize N`` (or ``-ON``) is used to control the creation of :file:`.pyo`
+ files: ``-O1`` turns on basic optimizations, ``-O2`` also discards docstrings,
+ ``-O0`` does not create :file:`.pyo` files; the default is ``-O0``.
+
+You can mix and match these options: for example, ``--no-compile --optimize 2``
+will create :file:`.pyo` files but no :file:`.pyc` files.
+
+.. XXX these option roles do not work
+
+Calling Python with :option:`-O` or :option:`-B` does not control the creation
+of bytecode files, only the options described above do.
:command:`build_scripts`
:command:`install_lib`
----------------------
-Install C library files.
+Install all modules (extensions and pure Python).
+
+.. XXX what about C libraries created with build_clib?
+
+Similarly to ``build_py``, there are options to control the compilation of
+Python code to :term:`bytecode` files (see above). By default, :file:`.pyc`
+files will be created (``--compile``) and :file:`.pyo` files will not
+(``--optimize 0``).
:command:`install_scripts`
import os
import imp
-import sys
from glob import glob
from packaging import logger
# marking public APIs
__all__ = ['build_py']
+
class build_py(Command, Mixin2to3):
description = "build pure Python modules (copy to build directory)"
+ # The options for controlling byte compilations are two independent sets;
+ # more info in install_lib or the reST docs
+
user_options = [
('build-lib=', 'd', "directory to build (copy) to"),
('compile', 'c', "compile .py to .pyc"),
]
boolean_options = ['compile', 'force']
- negative_opt = {'no-compile' : 'compile'}
+
+ negative_opt = {'no-compile': 'compile'}
def initialize_options(self):
self.build_lib = None
def get_data_files(self):
"""Generate list of '(package,src_dir,build_dir,filenames)' tuples.
- Helper function for `finalize_options()`.
+ Helper function for finalize_options.
"""
data = []
if not self.packages:
# Length of path to strip from found files
plen = 0
if src_dir:
- plen = len(src_dir)+1
+ plen = len(src_dir) + 1
# Strip directory from globbed filenames
filenames = [
def find_data_files(self, package, src_dir):
"""Return filenames for package's data files in 'src_dir'.
- Helper function for `get_data_files()`.
+ Helper function for get_data_files.
"""
globs = (self.package_data.get('', [])
+ self.package_data.get(package, []))
def build_package_data(self):
"""Copy data files into build directory.
- Helper function for `run()`.
+ Helper function for run.
"""
# FIXME add tests for this method
for package, src_dir, build_dir, filenames in self.data_files:
self.mkpath(os.path.dirname(target))
outf, copied = self.copy_file(srcfile,
target, preserve_mode=False)
- if copied and srcfile in self.distribution.convert_2to3.doctests:
+ doctests = self.distribution.convert_2to3_doctests
+ if copied and srcfile in doctests:
self._doctests_2to3.append(outf)
# XXX - this should be moved to the Distribution class as it is not
# only needed for build_py. It also has no dependencies on this class.
def get_package_dir(self, package):
"""Return the directory, relative to the top of the source
- distribution, where package 'package' should be found
- (at least according to the 'package_dir' option, if any)."""
-
+ distribution, where package 'package' should be found
+ (at least according to the 'package_dir' option, if any).
+ """
path = package.split('.')
if self.package_dir is not None:
path.insert(0, self.package_dir)
return ''
def check_package(self, package, package_dir):
- """Helper function for `find_package_modules()` and `find_modules()'.
- """
+ """Helper function for find_package_modules and find_modules."""
# Empty dir name means current directory, which we can probably
# assume exists. Also, os.path.exists and isdir don't know about
# my "empty string means current dir" convention, so we have to
if os.path.isfile(init_py):
return init_py
else:
- logger.warning(("package init file '%s' not found " +
- "(or not a regular file)"), init_py)
+ logger.warning("package init file %r not found "
+ "(or not a regular file)", init_py)
# Either not in a package at all (__init__.py not expected), or
# __init__.py doesn't exist -- so don't return the filename.
def check_module(self, module, module_file):
if not os.path.isfile(module_file):
- logger.warning("file %s (for module %s) not found",
+ logger.warning("file %r (for module %r) not found",
module_file, module)
return False
else:
module = os.path.splitext(os.path.basename(f))[0]
modules.append((package, module, f))
else:
- logger.debug("excluding %s", setup_script)
+ logger.debug("excluding %r", setup_script)
return modules
def find_modules(self):
outputs.append(filename)
if include_bytecode:
if self.compile:
- outputs.append(imp.cache_from_source(filename))
+ outputs.append(imp.cache_from_source(filename,
+ debug_override=True))
if self.optimize > 0:
outputs.append(imp.cache_from_source(filename,
debug_override=False))
def build_modules(self):
modules = self.find_modules()
for package, module, module_file in modules:
-
# Now "build" the module -- ie. copy the source file to
# self.build_lib (the build directory for Python source).
# (Actually, it gets copied to the directory for this package
def build_packages(self):
for package in self.packages:
-
# Get list of (package, module, module_file) tuples based on
# scanning the package directory. 'package' is only included
# in the tuple so that 'find_modules()' and
self.build_module(module, module_file, package)
def byte_compile(self, files):
- if sys.dont_write_bytecode:
- logger.warning('%s: byte-compiling is disabled, skipping.',
- self.get_command_name())
- return
-
from packaging.util import byte_compile # FIXME use compileall
prefix = self.build_lib
if prefix[-1] != os.sep:
# XXX this code is essentially the same as the 'byte_compile()
# method of the "install_lib" command, except for the determination
# of the 'prefix' string. Hmmm.
-
if self.compile:
byte_compile(files, optimize=0,
force=self.force, prefix=prefix, dry_run=self.dry_run)
import os
import imp
-import sys
import logging
from packaging import logger
# Extension for Python source files.
+# XXX dead code? most of the codebase checks for literal '.py'
if hasattr(os, 'extsep'):
PYTHON_SOURCE_EXTENSION = os.extsep + "py"
else:
PYTHON_SOURCE_EXTENSION = ".py"
+
class install_lib(Command):
description = "install all modules (extensions and pure Python)"
- # The byte-compilation options are a tad confusing. Here are the
- # possible scenarios:
- # 1) no compilation at all (--no-compile --no-optimize)
- # 2) compile .pyc only (--compile --no-optimize; default)
- # 3) compile .pyc and "level 1" .pyo (--compile --optimize)
- # 4) compile "level 1" .pyo only (--no-compile --optimize)
- # 5) compile .pyc and "level 2" .pyo (--compile --optimize-more)
- # 6) compile "level 2" .pyo only (--no-compile --optimize-more)
- #
- # The UI for this is two option, 'compile' and 'optimize'.
+ # The options for controlling byte compilations are two independent sets:
# 'compile' is strictly boolean, and only decides whether to
# generate .pyc files. 'optimize' is three-way (0, 1, or 2), and
# decides both whether to generate .pyo files and what level of
user_options = [
('install-dir=', 'd', "directory to install to"),
- ('build-dir=','b', "build directory (where to install from)"),
+ ('build-dir=', 'b', "build directory (where to install from)"),
('force', 'f', "force installation (overwrite existing files)"),
('compile', 'c', "compile .py to .pyc [default]"),
('no-compile', None, "don't compile .py files"),
]
boolean_options = ['force', 'compile', 'skip-build']
- negative_opt = {'no-compile' : 'compile'}
+
+ negative_opt = {'no-compile': 'compile'}
def initialize_options(self):
# let the 'install_dist' command dictate our installation directory
self.set_undefined_options('install_dist',
('build_lib', 'build_dir'),
('install_lib', 'install_dir'),
- 'force', 'compile', 'optimize', 'skip_build')
+ 'force', 'compile', 'optimize',
+ 'skip_build')
if self.compile is None:
self.compile = True
return outfiles
def byte_compile(self, files):
- if sys.dont_write_bytecode:
- # XXX do we want this? because a Python runs without bytecode
- # doesn't mean that the *dists should not contain bytecode
- #--or does it?
- logger.warning('%s: byte-compiling is disabled, skipping.',
- self.get_command_name())
- return
-
from packaging.util import byte_compile # FIXME use compileall
# Get the "--root" directory supplied to the "install_dist" command,
if self.compile:
byte_compile(files, optimize=0,
force=self.force, prefix=install_root,
- dry_run=self.dry_run)
+ verbose=verbose, dry_run=self.dry_run)
if self.optimize > 0:
byte_compile(files, optimize=self.optimize,
force=self.force, prefix=install_root,
- verbose=verbose,
- dry_run=self.dry_run)
-
+ verbose=verbose, dry_run=self.dry_run)
# -- Utility methods -----------------------------------------------
if ext != PYTHON_SOURCE_EXTENSION:
continue
if self.compile:
- bytecode_files.append(imp.cache_from_source(py_file))
+ bytecode_files.append(imp.cache_from_source(
+ py_file, debug_override=True))
if self.optimize > 0:
bytecode_files.append(imp.cache_from_source(
py_file, debug_override=False))
return bytecode_files
-
# -- External interface --------------------------------------------
# (called by outsiders)
"""Syntax error in a file list template."""
-class PackagingByteCompileError(PackagingError):
- """Byte compile error."""
-
-
class PackagingPyPIError(PackagingError):
"""Any problem occuring during using the indexes."""
# misc. functions and decorators
'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
# imported from this module for backport purposes
- 'unittest', 'requires_zlib', 'skip_unless_symlink',
+ 'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
]
except ImportError:
skip_unless_symlink = unittest.skip(
'requires test.support.skip_unless_symlink')
+
+
+skip_2to3_optimize = unittest.skipIf(sys.flags.optimize,
+ "2to3 doesn't work under -O")
pycache_dir = os.path.join(pkgdest, "__pycache__")
self.assertIn("__init__.py", files)
self.assertIn("README.txt", files)
- if sys.dont_write_bytecode:
- self.assertFalse(os.path.exists(pycache_dir))
- else:
- # XXX even with -O, packaging writes pyc, not pyo; bug?
- pyc_files = os.listdir(pycache_dir)
- self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
+ pyc_files = os.listdir(pycache_dir)
+ self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
def test_empty_package_dir(self):
# See SF 1668596/1720897.
os.chdir(cwd)
sys.stdout = old_stdout
- @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
def test_byte_compile(self):
project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
os.chdir(project_dir)
found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
self.assertEqual(found, ['boiledeggs.%s.pyc' % imp.get_tag()])
- @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
def test_byte_compile_optimized(self):
project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
os.chdir(project_dir)
self.assertEqual(sorted(found), ['boiledeggs.%s.pyc' % imp.get_tag(),
'boiledeggs.%s.pyo' % imp.get_tag()])
- def test_dont_write_bytecode(self):
- # makes sure byte_compile is not used
- pkg_dir, dist = self.create_dist()
- cmd = build_py(dist)
- cmd.compile = True
- cmd.optimize = 1
-
- old_dont_write_bytecode = sys.dont_write_bytecode
+ def test_byte_compile_under_B(self):
+ # make sure byte compilation works under -B (dont_write_bytecode)
+ self.addCleanup(setattr, sys, 'dont_write_bytecode',
+ sys.dont_write_bytecode)
sys.dont_write_bytecode = True
- try:
- cmd.byte_compile([])
- finally:
- sys.dont_write_bytecode = old_dont_write_bytecode
-
- self.assertIn('byte-compiling is disabled', self.get_logs()[0])
+ self.test_byte_compile()
+ self.test_byte_compile_optimized()
def test_suite():
restore_environ = ['PYTHONPATH']
def test_finalize_options(self):
- pkg_dir, dist = self.create_dist()
+ dist = self.create_dist()[1]
cmd = install_lib(dist)
cmd.finalize_options()
cmd.finalize_options()
self.assertEqual(cmd.optimize, 2)
- @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
def test_byte_compile(self):
- pkg_dir, dist = self.create_dist()
- os.chdir(pkg_dir)
+ project_dir, dist = self.create_dist()
+ os.chdir(project_dir)
cmd = install_lib(dist)
cmd.compile = True
cmd.optimize = 1
- f = os.path.join(pkg_dir, 'foo.py')
+ f = os.path.join(project_dir, 'foo.py')
self.write_file(f, '# python file')
cmd.byte_compile([f])
- pyc_file = imp.cache_from_source('foo.py')
+ pyc_file = imp.cache_from_source('foo.py', debug_override=True)
pyo_file = imp.cache_from_source('foo.py', debug_override=False)
self.assertTrue(os.path.exists(pyc_file))
self.assertTrue(os.path.exists(pyo_file))
+ def test_byte_compile_under_B(self):
+ # make sure byte compilation works under -B (dont_write_bytecode)
+ self.addCleanup(setattr, sys, 'dont_write_bytecode',
+ sys.dont_write_bytecode)
+ sys.dont_write_bytecode = True
+ self.test_byte_compile()
+
def test_get_outputs(self):
- pkg_dir, dist = self.create_dist()
+ project_dir, dist = self.create_dist()
+ os.chdir(project_dir)
+ os.mkdir('spam')
cmd = install_lib(dist)
# setting up a dist environment
cmd.compile = True
cmd.optimize = 1
- cmd.install_dir = pkg_dir
- f = os.path.join(pkg_dir, '__init__.py')
+ cmd.install_dir = self.mkdtemp()
+ f = os.path.join(project_dir, 'spam', '__init__.py')
self.write_file(f, '# python package')
cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
- cmd.distribution.packages = [pkg_dir]
+ cmd.distribution.packages = ['spam']
- # make sure the build_lib is set the temp dir
- build_dir = os.path.split(pkg_dir)[0]
+ # make sure the build_lib is set the temp dir # XXX what? this is not
+ # needed in the same distutils test and should work without manual
+ # intervention
+ build_dir = os.path.split(project_dir)[0]
cmd.get_finalized_command('build_py').build_lib = build_dir
- # get_output should return 4 elements
- self.assertEqual(len(cmd.get_outputs()), 4)
+ # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+ # .pyo, foo.import-tag-abiflags.so / foo.pyd
+ outputs = cmd.get_outputs()
+ self.assertEqual(len(outputs), 4, outputs)
def test_get_inputs(self):
- pkg_dir, dist = self.create_dist()
+ project_dir, dist = self.create_dist()
+ os.chdir(project_dir)
+ os.mkdir('spam')
cmd = install_lib(dist)
# setting up a dist environment
cmd.compile = True
cmd.optimize = 1
- cmd.install_dir = pkg_dir
- f = os.path.join(pkg_dir, '__init__.py')
+ cmd.install_dir = self.mkdtemp()
+ f = os.path.join(project_dir, 'spam', '__init__.py')
self.write_file(f, '# python package')
cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
- cmd.distribution.packages = [pkg_dir]
-
- # get_input should return 2 elements
- self.assertEqual(len(cmd.get_inputs()), 2)
-
- def test_dont_write_bytecode(self):
- # makes sure byte_compile is not used
- pkg_dir, dist = self.create_dist()
- cmd = install_lib(dist)
- cmd.compile = True
- cmd.optimize = 1
-
- self.addCleanup(setattr, sys, 'dont_write_bytecode',
- sys.dont_write_bytecode)
- sys.dont_write_bytecode = True
- cmd.byte_compile([])
+ cmd.distribution.packages = ['spam']
- self.assertIn('byte-compiling is disabled', self.get_logs()[0])
+ # get_inputs should return 2 elements: spam/__init__.py and
+ # foo.import-tag-abiflags.so / foo.pyd
+ inputs = cmd.get_inputs()
+ self.assertEqual(len(inputs), 2, inputs)
def test_suite():
support.LoggingCatcher,
unittest.TestCase):
+ @support.skip_2to3_optimize
def test_convert_code_only(self):
# used to check if code gets converted properly.
code = "print 'test'"
from io import StringIO
from packaging.errors import (
- PackagingPlatformError, PackagingByteCompileError, PackagingFileError,
+ PackagingPlatformError, PackagingFileError,
PackagingExecError, InstallationException)
from packaging import util
from packaging.dist import Distribution
self._uname = None
os.uname = self._get_uname
- # patching POpen
- self.old_find_executable = util.find_executable
- util.find_executable = self._find_executable
- self._exes = {}
- self.old_popen = subprocess.Popen
- self.old_stdout = sys.stdout
- self.old_stderr = sys.stderr
- FakePopen.test_class = self
- subprocess.Popen = FakePopen
+ def _get_uname(self):
+ return self._uname
def tearDown(self):
# getting back the environment
os.uname = self.uname
else:
del os.uname
+ super(UtilTestCase, self).tearDown()
+
+ def mock_popen(self):
+ self.old_find_executable = util.find_executable
+ util.find_executable = self._find_executable
+ self._exes = {}
+ self.old_popen = subprocess.Popen
+ self.old_stdout = sys.stdout
+ self.old_stderr = sys.stderr
+ FakePopen.test_class = self
+ subprocess.Popen = FakePopen
+ self.addCleanup(self.unmock_popen)
+
+ def unmock_popen(self):
util.find_executable = self.old_find_executable
subprocess.Popen = self.old_popen
sys.old_stdout = self.old_stdout
sys.old_stderr = self.old_stderr
- super(UtilTestCase, self).tearDown()
-
- def _set_uname(self, uname):
- self._uname = uname
-
- def _get_uname(self):
- return self._uname
def test_convert_path(self):
# linux/mac
return None
def test_get_compiler_versions(self):
+ self.mock_popen()
# get_versions calls distutils.spawn.find_executable on
# 'gcc', 'ld' and 'dllwrap'
self.assertEqual(get_compiler_versions(), (None, None, None))
res = get_compiler_versions()
self.assertEqual(res[2], None)
- def test_dont_write_bytecode(self):
- # makes sure byte_compile raise a PackagingError
- # if sys.dont_write_bytecode is True
- old_dont_write_bytecode = sys.dont_write_bytecode
+ def test_byte_compile_under_B(self):
+ # make sure byte compilation works under -B (dont_write_bytecode)
+ self.addCleanup(setattr, sys, 'dont_write_bytecode',
+ sys.dont_write_bytecode)
sys.dont_write_bytecode = True
- try:
- self.assertRaises(PackagingByteCompileError, byte_compile, [])
- finally:
- sys.dont_write_bytecode = old_dont_write_bytecode
+ byte_compile([])
def test_newer(self):
self.assertRaises(PackagingFileError, util.newer, 'xxx', 'xxx')
self.assertRaises(ImportError, resolve_name, 'a.b.Spam')
self.assertRaises(ImportError, resolve_name, 'a.b.c.Spam')
+ @support.skip_2to3_optimize
def test_run_2to3_on_code(self):
content = "print 'test'"
converted_content = "print('test')"
file_handle.close()
self.assertEqual(new_content, converted_content)
+ @support.skip_2to3_optimize
def test_run_2to3_on_doctests(self):
# to check if text files containing doctests only get converted.
content = ">>> print 'test'\ntest\n"
@unittest.skipUnless(os.name in ('nt', 'posix'),
'runs only under posix or nt')
def test_spawn(self):
- # no patching of Popen here
- subprocess.Popen = self.old_popen
tmpdir = self.mkdtemp()
# creating something executable
self.assertEqual(args['py_modules'], dist.py_modules)
def test_generate_setup_py(self):
- # undo subprocess.Popen monkey-patching before using assert_python_*
- subprocess.Popen = self.old_popen
os.chdir(self.mkdtemp())
self.write_file('setup.cfg', textwrap.dedent("""\
[metadata]
from packaging import logger
from packaging.errors import (PackagingPlatformError, PackagingFileError,
- PackagingByteCompileError, PackagingExecError,
- InstallationException, PackagingInternalError)
+ PackagingExecError, InstallationException,
+ PackagingInternalError)
__all__ = [
# file dependencies
the source for details). The 'direct' flag is used by the script
generated in indirect mode; unless you know what you're doing, leave
it set to None.
- """
- # nothing is done if sys.dont_write_bytecode is True
- # FIXME this should not raise an error
- if sys.dont_write_bytecode:
- raise PackagingByteCompileError('byte-compiling is disabled.')
+ This function is independent from the running Python's -O or -B options;
+ it is fully controlled by the parameters passed in.
+ """
# First, if the caller didn't force us into direct or indirect mode,
# figure out which mode we should be in. We take a conservative
# approach: choose direct mode *only* if the current interpreter is
# Terminology from the py_compile module:
# cfile - byte-compiled file
# dfile - purported source filename (same as 'file' by default)
- if optimize >= 0:
- cfile = imp.cache_from_source(file,
- debug_override=not optimize)
- else:
- cfile = imp.cache_from_source(file)
+ debug_override = not optimize
+ cfile = imp.cache_from_source(file, debug_override)
dfile = file
+
if prefix:
if file[:len(prefix)] != prefix:
raise ValueError("invalid prefix: filename %r doesn't "
Library
-------
+- Byte compilation in packaging is now isolated from the calling Python -B or
+ -O options, instead of being disallowed under -B or buggy under -O.
+
- Issue #2892: preserve iterparse events in case of SyntaxError.
- Issue #13287: urllib.request and urllib.error now contains an __all__
projects / thirdparty / Python / cpython.git / commitdiff
