$ git checkout -b my-branch origin/master
$ # Add some commits to the branch, reading for testing
-2. Create ~/.buildman to tell buildman where to find tool chains. As an
-example:
+2. Create ~/.buildman to tell buildman where to find tool chains (see 'The
+.buildman file' later for details). As an example:
# Buildman settings file
to build x86 commits.
-2. Check the available toolchains
+3. Make sure you have the require Python pre-requisites
+
+Buildman uses multiprocessing, Queue, shutil, StringIO, ConfigParser and
+urllib2. These should normally be available, but if you get an error like
+this then you will need to obtain those modules:
+
+ ImportError: No module named multiprocessing
+
+
+4. Check the available toolchains
Run this check to make sure that you have a toolchain for every architecture.
be used (c88 and c99). This is a feature.
+5. Install new toolchains if needed
+
+You can download toolchains and update the [toolchain] section of the
+settings file to find them.
+
+To make this easier, buildman can automatically download and install
+toolchains from kernel.org. First list the available architectures:
+
+$ ./tools/buildman/buildman sandbox --fetch-arch list
+Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.3/
+Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.2/
+Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.5.1/
+Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.2.4/
+Available architectures: alpha am33_2.0 arm avr32 bfin cris crisv32 frv h8300
+hppa hppa64 i386 ia64 m32r m68k mips mips64 or32 powerpc powerpc64 s390x sh4
+sparc sparc64 tilegx x86_64 xtensa
+
+Then pick one and download it:
+
+$ ./tools/buildman/buildman sandbox --fetch-arch or32
+Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.3/
+Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.2/
+Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.5.1/
+Downloading: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.5.1//x86_64-gcc-4.5.1-nolibc_or32-linux.tar.xz
+Unpacking to: /home/sjg/.buildman-toolchains
+Testing
+ - looking in '/home/sjg/.buildman-toolchains/gcc-4.5.1-nolibc/or32-linux/.'
+ - looking in '/home/sjg/.buildman-toolchains/gcc-4.5.1-nolibc/or32-linux/bin'
+ - found '/home/sjg/.buildman-toolchains/gcc-4.5.1-nolibc/or32-linux/bin/or32-linux-gcc'
+Tool chain test: OK
+
+Buildman should now be set up to use your new toolchain.
+
+At the time of writing, U-Boot has these architectures:
+
+ arc, arm, avr32, blackfin, m68k, microblaze, mips, nds32, nios2, openrisc
+ powerpc, sandbox, sh, sparc, x86
+
+Of these, only arc, microblaze and nds32 are not available at kernel.org..
+
+
How to run it
=============
$ ./tools/buildman/buildman -b <branch> -n
If it can't detect the upstream branch, try checking out the branch, and
-doing something like 'git branch --set-upstream <branch> upstream/master'
-or something similar.
+doing something like 'git branch --set-upstream-to upstream/master'
+or something similar. Buildman will try to guess a suitable upstream branch
+if it can't find one (you will see a message like" Guessing upstream as ...).
As an example:
increases, and vice versa.
-Providing 'make' flags
-======================
+The .buildman file
+==================
+
+The .buildman file provides information about the available toolchains and
+also allows build flags to be passed to 'make'. It consists of several
+sections, with the section name in square brackets. Within each section are
+a set of (tag, value) pairs.
+
+'[toolchain]' section
+
+ This lists the available toolchains. The tag here doesn't matter, but
+ make sure it is unique. The value is the path to the toolchain. Buildman
+ will look in that path for a file ending in 'gcc'. It will then execute
+ it to check that it is a C compiler, passing only the --version flag to
+ it. If the return code is 0, buildman assumes that it is a valid C
+ compiler. It uses the first part of the name as the architecture and
+ strips off the last part when setting the CROSS_COMPILE environment
+ variable (parts are delimited with a hyphen).
+
+ For example powerpc-linux-gcc will be noted as a toolchain for 'powerpc'
+ and CROSS_COMPILE will be set to powerpc-linux- when using it.
+
+'[toolchain-alias]' section
+
+ This converts toolchain architecture names to U-Boot names. For example,
+ if an x86 toolchains is called i386-linux-gcc it will not normally be
+ used for architecture 'x86'. Adding 'x86: i386 x86_64' to this section
+ will tell buildman that the i386 and x86_64 toolchains can be used for
+ the x86 architecture.
+
+'[make-flags]' section
-U-Boot's build system supports a few flags (such as BUILD_TAG) which affect
-the build product. These flags can be specified in the buildman settings
-file. They can also be useful when building U-Boot against other open source
-software.
+ U-Boot's build system supports a few flags (such as BUILD_TAG) which
+ affect the build product. These flags can be specified in the buildman
+ settings file. They can also be useful when building U-Boot against other
+ open source software.
-[make-flags]
-at91-boards=ENABLE_AT91_TEST=1
-snapper9260=${at91-boards} BUILD_TAG=442
-snapper9g45=${at91-boards} BUILD_TAG=443
+ [make-flags]
+ at91-boards=ENABLE_AT91_TEST=1
+ snapper9260=${at91-boards} BUILD_TAG=442
+ snapper9g45=${at91-boards} BUILD_TAG=443
-This will use 'make ENABLE_AT91_TEST=1 BUILD_TAG=442' for snapper9260
-and 'make ENABLE_AT91_TEST=1 BUILD_TAG=443' for snapper9g45. A special
-variable ${target} is available to access the target name (snapper9260 and
-snapper9g20 in this case). Variables are resolved recursively. Note that
-variables can only contain the characters A-Z, a-z, 0-9, hyphen (-) and
-underscore (_).
+ This will use 'make ENABLE_AT91_TEST=1 BUILD_TAG=442' for snapper9260
+ and 'make ENABLE_AT91_TEST=1 BUILD_TAG=443' for snapper9g45. A special
+ variable ${target} is available to access the target name (snapper9260
+ and snapper9g20 in this case). Variables are resolved recursively. Note
+ that variables can only contain the characters A-Z, a-z, 0-9, hyphen (-)
+ and underscore (_).
-It is expected that any variables added are dealt with in U-Boot's
-config.mk file and documented in the README.
+ It is expected that any variables added are dealt with in U-Boot's
+ config.mk file and documented in the README.
+
+ Note that you can pass ad-hoc options to the build using environment
+ variables, for example:
+
+ SOME_OPTION=1234 ./tools/buildman/buildman my_board
Quick Sanity Check
enabled automatically). Use -e to see errors/warnings as well.
+Building Ranges
+===============
+
+You can build a range of commits by specifying a range instead of a branch
+when using the -b flag. For example:
+
+ upstream/master..us-buildman
+
+will build commits in us-buildman that are not in upstream/master.
+
+
Other options
=============
try:
return settings.items(section)
except ConfigParser.NoSectionError as e:
- print e
return []
except:
raise
+
+def SetItem(section, tag, value):
+ """Set an item and write it back to the settings file"""
+ global settings
+ global config_fname
+
+ settings.set(section, tag, value)
+ if config_fname is not None:
+ with open(config_fname, 'w') as fd:
+ settings.write(fd)
self.func_sizes = func_sizes
def __init__(self, toolchains, base_dir, git_dir, num_threads, num_jobs,
- gnu_make='make', checkout=True, show_unknown=True, step=1):
+ gnu_make='make', checkout=True, show_unknown=True, step=1,
+ no_subdirs=False, full_path=False, verbose_build=False):
"""Create a new Builder object
Args:
This is used for testing.
show_unknown: Show unknown boards (those not built) in summary
step: 1 to process every commit, n to process every nth commit
+ no_subdirs: Don't create subdirectories when building current
+ source for a single board
+ full_path: Return the full path in CROSS_COMPILE and don't set
+ PATH
+ verbose_build: Run build with V=1 and don't use 'make -s'
"""
self.toolchains = toolchains
self.base_dir = base_dir
self._step = step
self.in_tree = False
self._error_lines = 0
+ self.no_subdirs = no_subdirs
+ self.full_path = full_path
+ self.verbose_build = verbose_build
self.col = terminal.Color()
Args:
commit_upto: Commit number to use (0..self.count-1)
"""
+ commit_dir = None
if self.commits:
commit = self.commits[commit_upto]
subject = commit.subject.translate(trans_valid_chars)
commit_dir = ('%02d_of_%02d_g%s_%s' % (commit_upto + 1,
self.commit_count, commit.hash, subject[:20]))
- else:
+ elif not self.no_subdirs:
commit_dir = 'current'
- output_dir = os.path.join(self.base_dir, commit_dir)
- return output_dir
+ if not commit_dir:
+ return self.base_dir
+ return os.path.join(self.base_dir, commit_dir)
def GetBuildDir(self, commit_upto, target):
"""Get the name of the build directory for a commit number
create. Having left over directories is confusing when the user wants
to check the output manually.
"""
+ if not self.commits:
+ return
dir_list = []
for commit_upto in range(self.commit_count):
dir_list.append(self._GetOutputDir(commit_upto))
commit = 'current'
# Set up the environment and command line
- env = self.toolchain.MakeEnvironment()
+ env = self.toolchain.MakeEnvironment(self.builder.full_path)
Mkdir(out_dir)
args = []
cwd = work_dir
src_dir = os.getcwd()
else:
args.append('O=build')
- args.append('-s')
+ if not self.builder.verbose_build:
+ args.append('-s')
if self.builder.num_jobs is not None:
args.extend(['-j', str(self.builder.num_jobs)])
config_args = ['%s_defconfig' % brd.target]
print >>fd, 'path', result.toolchain.path
# Write out the image and function size information and an objdump
- env = result.toolchain.MakeEnvironment()
+ env = result.toolchain.MakeEnvironment(self.builder.full_path)
lines = []
for fname in ['u-boot', 'spl/u-boot-spl']:
cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname]
parser.add_option('-F', '--force-build-failures', dest='force_build_failures',
action='store_true', default=False,
help='Force build of previously-failed build')
+ parser.add_option('--fetch-arch', type='string',
+ help="Fetch a toolchain for architecture FETCH_ARCH ('list' to list)."
+ ' You can also fetch several toolchains separate by comma, or'
+ " 'all' to download all")
parser.add_option('-g', '--git', type='string',
help='Git repo containing branch to build', default='.')
parser.add_option('-G', '--config-file', type='string',
help='List available tool chains')
parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
default=False, help="Do a dry run (describe actions, but do nothing)")
+ parser.add_option('-N', '--no-subdirs', action='store_true', dest='no_subdirs',
+ default=False, help="Don't create subdirectories when building current source for a single board")
parser.add_option('-o', '--output-dir', type='string',
dest='output_dir', default='..',
help='Directory where all builds happen and buildman has its workspace (default is ../)')
parser.add_option('-Q', '--quick', action='store_true',
default=False, help='Do a rough build, with limited warning resolution')
+ parser.add_option('-p', '--full-path', action='store_true',
+ default=False, help="Use full toolchain path in CROSS_COMPILE")
parser.add_option('-s', '--summary', action='store_true',
default=False, help='Show a build summary')
parser.add_option('-S', '--show-sizes', action='store_true',
default=False, help='Show boards with unknown build result')
parser.add_option('-v', '--verbose', action='store_true',
default=False, help='Show build results while the build progresses')
+ parser.add_option('-V', '--verbose-build', action='store_true',
+ default=False, help='Run make with V=1, showing all output')
parser.add_option('-x', '--exclude', dest='exclude',
type='string', action='append',
help='Specify a list of boards to exclude, separated by comma')
print
return 0
+ if options.fetch_arch:
+ if options.fetch_arch == 'list':
+ sorted_list = toolchains.ListArchs()
+ print 'Available architectures: %s\n' % ' '.join(sorted_list)
+ return 0
+ else:
+ fetch_arch = options.fetch_arch
+ if fetch_arch == 'all':
+ fetch_arch = ','.join(toolchains.ListArchs())
+ print 'Downloading toolchains: %s\n' % fetch_arch
+ for arch in fetch_arch.split(','):
+ ret = toolchains.FetchAndInstall(arch)
+ if ret:
+ return ret
+ return 0
+
# Work out how many commits to build. We want to build everything on the
# branch. We also build the upstream commit as a control so we can see
# problems introduced by the first commit on the branch.
col = terminal.Color()
count = options.count
+ has_range = options.branch and '..' in options.branch
if count == -1:
if not options.branch:
count = 1
else:
- count = gitutil.CountCommitsInBranch(options.git_dir,
- options.branch)
+ if has_range:
+ count, msg = gitutil.CountCommitsInRange(options.git_dir,
+ options.branch)
+ else:
+ count, msg = gitutil.CountCommitsInBranch(options.git_dir,
+ options.branch)
if count is None:
- str = ("Branch '%s' not found or has no upstream" %
- options.branch)
- sys.exit(col.Color(col.RED, str))
+ sys.exit(col.Color(col.RED, msg))
+ elif count == 0:
+ sys.exit(col.Color(col.RED, "Range '%s' has no commits" %
+ options.branch))
+ if msg:
+ print col.Color(col.YELLOW, msg)
count += 1 # Build upstream commit also
if not count:
# to overwrite earlier ones by setting allow_overwrite=True
if options.branch:
if count == -1:
- range_expr = gitutil.GetRangeInBranch(options.git_dir,
- options.branch)
+ if has_range:
+ range_expr = options.branch
+ else:
+ range_expr = gitutil.GetRangeInBranch(options.git_dir,
+ options.branch)
upstream_commit = gitutil.GetUpstream(options.git_dir,
options.branch)
series = patchstream.GetMetaDataForList(upstream_commit,
if not gnu_make:
sys.exit('GNU Make not found')
- # Create a new builder with the selected options
+ # Create a new builder with the selected options.
+ output_dir = options.output_dir
if options.branch:
dirname = options.branch.replace('/', '_')
- else:
- dirname = 'current'
- output_dir = os.path.join(options.output_dir, dirname)
- if clean_dir and os.path.exists(output_dir):
+ # As a special case allow the board directory to be placed in the
+ # output directory itself rather than any subdirectory.
+ if not options.no_subdirs:
+ output_dir = os.path.join(options.output_dir, dirname)
+ if (clean_dir and output_dir != options.output_dir and
+ os.path.exists(output_dir)):
shutil.rmtree(output_dir)
builder = Builder(toolchains, output_dir, options.git_dir,
options.threads, options.jobs, gnu_make=gnu_make, checkout=True,
- show_unknown=options.show_unknown, step=options.step)
+ show_unknown=options.show_unknown, step=options.step,
+ no_subdirs=options.no_subdirs, full_path=options.full_path,
+ verbose_build=options.verbose_build)
builder.force_config_on_failure = not options.quick
if make_func:
builder.do_make = make_func
import terminal
import toolchain
+settings_data = '''
+# Buildman settings file
+
+[toolchain]
+main: /usr/sbin
+
+[toolchain-alias]
+x86: i386 x86_64
+'''
+
errors = [
'''main.c: In function 'main_loop':
main.c:260:6: warning: unused variable 'joe' [-Wunused-variable]
['Active', 'sandbox', 'sandbox', '', 'Tester', 'Sandbox board', 'board4', ''],
]
+BASE_DIR = 'base'
+
class Options:
"""Class that holds build options"""
pass
self.boards.AddBoard(board.Board(*brd))
self.boards.SelectBoards([])
+ # Add some test settings
+ bsettings.Setup(None)
+ bsettings.AddFile(settings_data)
+
# Set up the toolchains
- bsettings.Setup()
self.toolchains = toolchain.Toolchains()
self.toolchains.Add('arm-linux-gcc', test=False)
self.toolchains.Add('sparc-linux-gcc', test=False)
self.assertEqual(self.boards.SelectBoards(['sandbox sandbox',
'sandbox']),
{'all': 1, 'sandbox': 1})
+ def CheckDirs(self, build, dirname):
+ self.assertEqual('base%s' % dirname, build._GetOutputDir(1))
+ self.assertEqual('base%s/fred' % dirname,
+ build.GetBuildDir(1, 'fred'))
+ self.assertEqual('base%s/fred/done' % dirname,
+ build.GetDoneFile(1, 'fred'))
+ self.assertEqual('base%s/fred/u-boot.sizes' % dirname,
+ build.GetFuncSizesFile(1, 'fred', 'u-boot'))
+ self.assertEqual('base%s/fred/u-boot.objdump' % dirname,
+ build.GetObjdumpFile(1, 'fred', 'u-boot'))
+ self.assertEqual('base%s/fred/err' % dirname,
+ build.GetErrFile(1, 'fred'))
+
+ def testOutputDir(self):
+ build = builder.Builder(self.toolchains, BASE_DIR, None, 1, 2,
+ checkout=False, show_unknown=False)
+ build.commits = self.commits
+ build.commit_count = len(self.commits)
+ subject = self.commits[1].subject.translate(builder.trans_valid_chars)
+ dirname ='/%02d_of_%02d_g%s_%s' % (2, build.commit_count, commits[1][0],
+ subject[:20])
+ self.CheckDirs(build, dirname)
+
+ def testOutputDirCurrent(self):
+ build = builder.Builder(self.toolchains, BASE_DIR, None, 1, 2,
+ checkout=False, show_unknown=False)
+ build.commits = None
+ build.commit_count = 0
+ self.CheckDirs(build, '/current')
+
+ def testOutputDirNoSubdirs(self):
+ build = builder.Builder(self.toolchains, BASE_DIR, None, 1, 2,
+ checkout=False, show_unknown=False,
+ no_subdirs=True)
+ build.commits = None
+ build.commit_count = 0
+ self.CheckDirs(build, '')
+
+ def testToolchainAliases(self):
+ self.assertTrue(self.toolchains.Select('arm') != None)
+ with self.assertRaises(ValueError):
+ self.toolchains.Select('no-arch')
+ with self.assertRaises(ValueError):
+ self.toolchains.Select('x86')
+
+ self.toolchains = toolchain.Toolchains()
+ self.toolchains.Add('x86_64-linux-gcc', test=False)
+ self.assertTrue(self.toolchains.Select('x86') != None)
+
+ self.toolchains = toolchain.Toolchains()
+ self.toolchains.Add('i386-linux-gcc', test=False)
+ self.assertTrue(self.toolchains.Select('x86') != None)
+
+ def testToolchainDownload(self):
+ """Test that we can download toolchains"""
+ self.assertEqual('https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.3/x86_64-gcc-4.6.3-nolibc_arm-unknown-linux-gnueabi.tar.xz',
+ self.toolchains.LocateArchUrl('arm'))
+
if __name__ == "__main__":
unittest.main()
import re
import glob
+from HTMLParser import HTMLParser
import os
+import sys
+import tempfile
+import urllib2
import bsettings
import command
+# Simple class to collect links from a page
+class MyHTMLParser(HTMLParser):
+ def __init__(self, arch):
+ """Create a new parser
+
+ After the parser runs, self.links will be set to a list of the links
+ to .xz archives found in the page, and self.arch_link will be set to
+ the one for the given architecture (or None if not found).
+
+ Args:
+ arch: Architecture to search for
+ """
+ HTMLParser.__init__(self)
+ self.arch_link = None
+ self.links = []
+ self._match = '_%s-' % arch
+
+ def handle_starttag(self, tag, attrs):
+ if tag == 'a':
+ for tag, value in attrs:
+ if tag == 'href':
+ if value and value.endswith('.xz'):
+ self.links.append(value)
+ if self._match in value:
+ self.arch_link = value
+
+
class Toolchain:
"""A single toolchain
arch: Architecture of toolchain as determined from the first
component of the filename. E.g. arm-linux-gcc becomes arm
"""
-
def __init__(self, fname, test, verbose=False):
"""Create a new toolchain object.
"""
self.gcc = fname
self.path = os.path.dirname(fname)
- self.cross = os.path.basename(fname)[:-3]
+
+ # Find the CROSS_COMPILE prefix to use for U-Boot. For example,
+ # 'arm-linux-gnueabihf-gcc' turns into 'arm-linux-gnueabihf-'.
+ basename = os.path.basename(fname)
+ pos = basename.rfind('-')
+ self.cross = basename[:pos + 1] if pos != -1 else ''
+
+ # The architecture is the first part of the name
pos = self.cross.find('-')
self.arch = self.cross[:pos] if pos != -1 else 'sandbox'
- env = self.MakeEnvironment()
+ env = self.MakeEnvironment(False)
# As a basic sanity check, run the C compiler with --version
cmd = [fname, '--version']
return prio
return prio
- def MakeEnvironment(self):
+ def MakeEnvironment(self, full_path):
"""Returns an environment for using the toolchain.
- Thie takes the current environment, adds CROSS_COMPILE and
- augments PATH so that the toolchain will operate correctly.
+ Thie takes the current environment and adds CROSS_COMPILE so that
+ the tool chain will operate correctly.
+
+ Args:
+ full_path: Return the full path in CROSS_COMPILE and don't set
+ PATH
"""
env = dict(os.environ)
- env['CROSS_COMPILE'] = self.cross
- env['PATH'] += (':' + self.path)
+ if full_path:
+ env['CROSS_COMPILE'] = os.path.join(self.path, self.cross)
+ else:
+ env['CROSS_COMPILE'] = self.cross
+ env['PATH'] = self.path + ':' + env['PATH']
+
return env
self.paths = []
self._make_flags = dict(bsettings.GetItems('make-flags'))
- def GetSettings(self):
+ def GetPathList(self):
+ """Get a list of available toolchain paths
+
+ Returns:
+ List of strings, each a path to a toolchain mentioned in the
+ [toolchain] section of the settings file.
+ """
toolchains = bsettings.GetItems('toolchain')
if not toolchains:
print ("Warning: No tool chains - please add a [toolchain] section"
" to your buildman config file %s. See README for details" %
bsettings.config_fname)
+ paths = []
for name, value in toolchains:
if '*' in value:
- self.paths += glob.glob(value)
+ paths += glob.glob(value)
else:
- self.paths.append(value)
+ paths.append(value)
+ return paths
+
+ def GetSettings(self):
+ self.paths += self.GetPathList()
def Add(self, fname, test=True, verbose=False):
"""Add a toolchain to our list
if add_it:
self.toolchains[toolchain.arch] = toolchain
+ def ScanPath(self, path, verbose):
+ """Scan a path for a valid toolchain
+
+ Args:
+ path: Path to scan
+ verbose: True to print out progress information
+ Returns:
+ Filename of C compiler if found, else None
+ """
+ for subdir in ['.', 'bin', 'usr/bin']:
+ dirname = os.path.join(path, subdir)
+ if verbose: print " - looking in '%s'" % dirname
+ for fname in glob.glob(dirname + '/*gcc'):
+ if verbose: print " - found '%s'" % fname
+ return fname
+ return None
+
+
def Scan(self, verbose):
"""Scan for available toolchains and select the best for each arch.
if verbose: print 'Scanning for tool chains'
for path in self.paths:
if verbose: print " - scanning path '%s'" % path
- for subdir in ['.', 'bin', 'usr/bin']:
- dirname = os.path.join(path, subdir)
- if verbose: print " - looking in '%s'" % dirname
- for fname in glob.glob(dirname + '/*gcc'):
- if verbose: print " - found '%s'" % fname
- self.Add(fname, True, verbose)
+ fname = self.ScanPath(path, verbose)
+ if fname:
+ self.Add(fname, True, verbose)
def List(self):
"""List out the selected toolchains for each architecture"""
returns:
toolchain object, or None if none found
"""
- for name, value in bsettings.GetItems('toolchain-alias'):
- if arch == name:
- arch = value
+ for tag, value in bsettings.GetItems('toolchain-alias'):
+ if arch == tag:
+ for alias in value.split():
+ if alias in self.toolchains:
+ return self.toolchains[alias]
if not arch in self.toolchains:
raise ValueError, ("No tool chain found for arch '%s'" % arch)
else:
i += 1
return args
+
+ def LocateArchUrl(self, fetch_arch):
+ """Find a toolchain available online
+
+ Look in standard places for available toolchains. At present the
+ only standard place is at kernel.org.
+
+ Args:
+ arch: Architecture to look for, or 'list' for all
+ Returns:
+ If fetch_arch is 'list', a tuple:
+ Machine architecture (e.g. x86_64)
+ List of toolchains
+ else
+ URL containing this toolchain, if avaialble, else None
+ """
+ arch = command.OutputOneLine('uname', '-m')
+ base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
+ versions = ['4.6.3', '4.6.2', '4.5.1', '4.2.4']
+ links = []
+ for version in versions:
+ url = '%s/%s/%s/' % (base, arch, version)
+ print 'Checking: %s' % url
+ response = urllib2.urlopen(url)
+ html = response.read()
+ parser = MyHTMLParser(fetch_arch)
+ parser.feed(html)
+ if fetch_arch == 'list':
+ links += parser.links
+ elif parser.arch_link:
+ return url + parser.arch_link
+ if fetch_arch == 'list':
+ return arch, links
+ return None
+
+ def Download(self, url):
+ """Download a file to a temporary directory
+
+ Args:
+ url: URL to download
+ Returns:
+ Tuple:
+ Temporary directory name
+ Full path to the downloaded archive file in that directory,
+ or None if there was an error while downloading
+ """
+ print "Downloading: %s" % url
+ leaf = url.split('/')[-1]
+ tmpdir = tempfile.mkdtemp('.buildman')
+ response = urllib2.urlopen(url)
+ fname = os.path.join(tmpdir, leaf)
+ fd = open(fname, 'wb')
+ meta = response.info()
+ size = int(meta.getheaders("Content-Length")[0])
+ done = 0
+ block_size = 1 << 16
+ status = ''
+
+ # Read the file in chunks and show progress as we go
+ while True:
+ buffer = response.read(block_size)
+ if not buffer:
+ print chr(8) * (len(status) + 1), '\r',
+ break
+
+ done += len(buffer)
+ fd.write(buffer)
+ status = r"%10d MiB [%3d%%]" % (done / 1024 / 1024,
+ done * 100 / size)
+ status = status + chr(8) * (len(status) + 1)
+ print status,
+ sys.stdout.flush()
+ fd.close()
+ if done != size:
+ print 'Error, failed to download'
+ os.remove(fname)
+ fname = None
+ return tmpdir, fname
+
+ def Unpack(self, fname, dest):
+ """Unpack a tar file
+
+ Args:
+ fname: Filename to unpack
+ dest: Destination directory
+ Returns:
+ Directory name of the first entry in the archive, without the
+ trailing /
+ """
+ stdout = command.Output('tar', 'xvfJ', fname, '-C', dest)
+ return stdout.splitlines()[0][:-1]
+
+ def TestSettingsHasPath(self, path):
+ """Check if builmand will find this toolchain
+
+ Returns:
+ True if the path is in settings, False if not
+ """
+ paths = self.GetPathList()
+ return path in paths
+
+ def ListArchs(self):
+ """List architectures with available toolchains to download"""
+ host_arch, archives = self.LocateArchUrl('list')
+ re_arch = re.compile('[-a-z0-9.]*_([^-]*)-.*')
+ arch_set = set()
+ for archive in archives:
+ # Remove the host architecture from the start
+ arch = re_arch.match(archive[len(host_arch):])
+ if arch:
+ arch_set.add(arch.group(1))
+ return sorted(arch_set)
+
+ def FetchAndInstall(self, arch):
+ """Fetch and install a new toolchain
+
+ arch:
+ Architecture to fetch, or 'list' to list
+ """
+ # Fist get the URL for this architecture
+ url = self.LocateArchUrl(arch)
+ if not url:
+ print ("Cannot find toolchain for arch '%s' - use 'list' to list" %
+ arch)
+ return 2
+ home = os.environ['HOME']
+ dest = os.path.join(home, '.buildman-toolchains')
+ if not os.path.exists(dest):
+ os.mkdir(dest)
+
+ # Download the tar file for this toolchain and unpack it
+ tmpdir, tarfile = self.Download(url)
+ if not tarfile:
+ return 1
+ print 'Unpacking to: %s' % dest,
+ sys.stdout.flush()
+ path = self.Unpack(tarfile, dest)
+ os.remove(tarfile)
+ os.rmdir(tmpdir)
+ print
+
+ # Check that the toolchain works
+ print 'Testing'
+ dirpath = os.path.join(dest, path)
+ compiler_fname = self.ScanPath(dirpath, True)
+ if not compiler_fname:
+ print 'Could not locate C compiler - fetch failed.'
+ return 1
+ toolchain = Toolchain(compiler_fname, True, True)
+
+ # Make sure that it will be found by buildman
+ if not self.TestSettingsHasPath(dirpath):
+ print ("Adding 'download' to config file '%s'" %
+ bsettings.config_fname)
+ tools_dir = os.path.dirname(dirpath)
+ bsettings.SetItem('toolchain', 'download', '%s/*' % tools_dir)
+ return 0
patch_count = int(stdout)
return patch_count
+def NameRevision(commit_hash):
+ """Gets the revision name for a commit
+
+ Args:
+ commit_hash: Commit hash to look up
+
+ Return:
+ Name of revision, if any, else None
+ """
+ pipe = ['git', 'name-rev', commit_hash]
+ stdout = command.RunPipe([pipe], capture=True, oneline=True).stdout
+
+ # We expect a commit, a space, then a revision name
+ name = stdout.split(' ')[1].strip()
+ return name
+
+def GuessUpstream(git_dir, branch):
+ """Tries to guess the upstream for a branch
+
+ This lists out top commits on a branch and tries to find a suitable
+ upstream. It does this by looking for the first commit where
+ 'git name-rev' returns a plain branch name, with no ! or ^ modifiers.
+
+ Args:
+ git_dir: Git directory containing repo
+ branch: Name of branch
+
+ Returns:
+ Tuple:
+ Name of upstream branch (e.g. 'upstream/master') or None if none
+ Warning/error message, or None if none
+ """
+ pipe = [LogCmd(branch, git_dir=git_dir, oneline=True, count=100)]
+ result = command.RunPipe(pipe, capture=True, capture_stderr=True,
+ raise_on_error=False)
+ if result.return_code:
+ return None, "Branch '%s' not found" % branch
+ for line in result.stdout.splitlines()[1:]:
+ commit_hash = line.split(' ')[0]
+ name = NameRevision(commit_hash)
+ if '~' not in name and '^' not in name:
+ if name.startswith('remotes/'):
+ name = name[8:]
+ return name, "Guessing upstream as '%s'" % name
+ return None, "Cannot find a suitable upstream for branch '%s'" % branch
+
def GetUpstream(git_dir, branch):
"""Returns the name of the upstream for a branch
branch: Name of branch
Returns:
- Name of upstream branch (e.g. 'upstream/master') or None if none
+ Tuple:
+ Name of upstream branch (e.g. 'upstream/master') or None if none
+ Warning/error message, or None if none
"""
try:
remote = command.OutputOneLine('git', '--git-dir', git_dir, 'config',
merge = command.OutputOneLine('git', '--git-dir', git_dir, 'config',
'branch.%s.merge' % branch)
except:
- return None
+ upstream, msg = GuessUpstream(git_dir, branch)
+ return upstream, msg
if remote == '.':
return merge
elif remote and merge:
leaf = merge.split('/')[-1]
- return '%s/%s' % (remote, leaf)
+ return '%s/%s' % (remote, leaf), None
else:
raise ValueError, ("Cannot determine upstream branch for branch "
"'%s' remote='%s', merge='%s'" % (branch, remote, merge))
Expression in the form 'upstream..branch' which can be used to
access the commits. If the branch does not exist, returns None.
"""
- upstream = GetUpstream(git_dir, branch)
+ upstream, msg = GetUpstream(git_dir, branch)
if not upstream:
- return None
- return '%s%s..%s' % (upstream, '~' if include_upstream else '', branch)
+ return None, msg
+ rstr = '%s%s..%s' % (upstream, '~' if include_upstream else '', branch)
+ return rstr, msg
+
+def CountCommitsInRange(git_dir, range_expr):
+ """Returns the number of commits in the given range.
+
+ Args:
+ git_dir: Directory containing git repo
+ range_expr: Range to check
+ Return:
+ Number of patches that exist in the supplied rangem or None if none
+ were found
+ """
+ pipe = [LogCmd(range_expr, git_dir=git_dir, oneline=True)]
+ result = command.RunPipe(pipe, capture=True, capture_stderr=True,
+ raise_on_error=False)
+ if result.return_code:
+ return None, "Range '%s' not found or is invalid" % range_expr
+ patch_count = len(result.stdout.splitlines())
+ return patch_count, None
def CountCommitsInBranch(git_dir, branch, include_upstream=False):
"""Returns the number of commits in the given branch.
Number of patches that exist on top of the branch, or None if the
branch does not exist.
"""
- range_expr = GetRangeInBranch(git_dir, branch, include_upstream)
+ range_expr, msg = GetRangeInBranch(git_dir, branch, include_upstream)
if not range_expr:
- return None
- pipe = [LogCmd(range_expr, git_dir=git_dir, oneline=True),
- ['wc', '-l']]
- result = command.RunPipe(pipe, capture=True, oneline=True)
- patch_count = int(result.stdout)
- return patch_count
+ return None, msg
+ return CountCommitsInRange(git_dir, range_expr)
def CountCommits(commit_range):
"""Returns the number of commits in the given range.
projects / people / ms / u-boot.git / commitdiff
