tools/moveconfig.py

   1 #!/usr/bin/env python2
   2 #
   3 # Author: Masahiro Yamada <yamada.masahiro@socionext.com>
   4 #
   5 # SPDX-License-Identifier:      GPL-2.0+
   6 #
   7
   8 """
   9 Move config options from headers to defconfig files.
  10
  11 Since Kconfig was introduced to U-Boot, we have worked on moving
  12 config options from headers to Kconfig (defconfig).
  13
  14 This tool intends to help this tremendous work.
  15
  16
  17 Usage
  18 -----
  19
  20 This tool takes one input file.  (let's say 'recipe' file here.)
  21 The recipe describes the list of config options you want to move.
  22 Each line takes the form:
  23 <config_name> <type> <default>
  24 (the fields must be separated with whitespaces.)
  25
  26 <config_name> is the name of config option.
  27
  28 <type> is the type of the option.  It must be one of bool, tristate,
  29 string, int, and hex.
  30
  31 <default> is the default value of the option.  It must be appropriate
  32 value corresponding to the option type.  It must be either y or n for
  33 the bool type.  Tristate options can also take m (although U-Boot has
  34 not supported the module feature).
  35
  36 You can add two or more lines in the recipe file, so you can move
  37 multiple options at once.
  38
  39 Let's say, for example, you want to move CONFIG_CMD_USB and
  40 CONFIG_SYS_TEXT_BASE.
  41
  42 The type should be bool, hex, respectively.  So, the recipe file
  43 should look like this:
  44
  45   $ cat recipe
  46   CONFIG_CMD_USB bool n
  47   CONFIG_SYS_TEXT_BASE hex 0x00000000
  48
  49 Next you must edit the Kconfig to add the menu entries for the configs
  50 you are moving.
  51
  52 And then run this tool giving the file name of the recipe
  53
  54   $ tools/moveconfig.py recipe
  55
  56 The tool walks through all the defconfig files to move the config
  57 options specified by the recipe file.
  58
  59 The log is also displayed on the terminal.
  60
  61 Each line is printed in the format
  62 <defconfig_name>   :  <action>
  63
  64 <defconfig_name> is the name of the defconfig
  65 (without the suffix _defconfig).
  66
  67 <action> shows what the tool did for that defconfig.
  68 It looks like one of the followings:
  69
  70  - Move 'CONFIG_... '
  71    This config option was moved to the defconfig
  72
  73  - Default value 'CONFIG_...'.  Do nothing.
  74    The value of this option is the same as default.
  75    We do not have to add it to the defconfig.
  76
  77  - 'CONFIG_...' already exists in Kconfig.  Do nothing.
  78    This config option is already defined in Kconfig.
  79    We do not need/want to touch it.
  80
  81  - Undefined.  Do nothing.
  82    This config option was not found in the config header.
  83    Nothing to do.
  84
  85  - Failed to process.  Skip.
  86    An error occurred during processing this defconfig.  Skipped.
  87    (If -e option is passed, the tool exits immediately on error.)
  88
  89 Finally, you will be asked, Clean up headers? [y/n]:
  90
  91 If you say 'y' here, the unnecessary config defines are removed
  92 from the config headers (include/configs/*.h).
  93 It just uses the regex method, so you should not rely on it.
  94 Just in case, please do 'git diff' to see what happened.
  95
  96
  97 How does it works?
  98 ------------------
  99
 100 This tool runs configuration and builds include/autoconf.mk for every
 101 defconfig.  The config options defined in Kconfig appear in the .config
 102 file (unless they are hidden because of unmet dependency.)
 103 On the other hand, the config options defined by board headers are seen
 104 in include/autoconf.mk.  The tool looks for the specified options in both
 105 of them to decide the appropriate action for the options.  If the option
 106 is found in the .config or the value is the same as the specified default,
 107 the option does not need to be touched.  If the option is found in
 108 include/autoconf.mk, but not in the .config, and the value is different
 109 from the default, the tools adds the option to the defconfig.
 110
 111 For faster processing, this tool handles multi-threading.  It creates
 112 separate build directories where the out-of-tree build is run.  The
 113 temporary build directories are automatically created and deleted as
 114 needed.  The number of threads are chosen based on the number of the CPU
 115 cores of your system although you can change it via -j (--jobs) option.
 116
 117
 118 Toolchains
 119 ----------
 120
 121 Appropriate toolchain are necessary to generate include/autoconf.mk
 122 for all the architectures supported by U-Boot.  Most of them are available
 123 at the kernel.org site, some are not provided by kernel.org.
 124
 125 The default per-arch CROSS_COMPILE used by this tool is specified by
 126 the list below, CROSS_COMPILE.  You may wish to update the list to
 127 use your own.  Instead of modifying the list directly, you can give
 128 them via environments.
 129
 130
 131 Available options
 132 -----------------
 133
 134  -c, --color
 135    Surround each portion of the log with escape sequences to display it
 136    in color on the terminal.
 137
 138  -d, --defconfigs
 139   Specify a file containing a list of defconfigs to move
 140
 141  -n, --dry-run
 142    Peform a trial run that does not make any changes.  It is useful to
 143    see what is going to happen before one actually runs it.
 144
 145  -e, --exit-on-error
 146    Exit immediately if Make exits with a non-zero status while processing
 147    a defconfig file.
 148
 149  -H, --headers-only
 150    Only cleanup the headers; skip the defconfig processing
 151
 152  -j, --jobs
 153    Specify the number of threads to run simultaneously.  If not specified,
 154    the number of threads is the same as the number of CPU cores.
 155
 156  -v, --verbose
 157    Show any build errors as boards are built
 158
 159 To see the complete list of supported options, run
 160
 161   $ tools/moveconfig.py -h
 162
 163 """
 164
 165 import fnmatch
 166 import multiprocessing
 167 import optparse
 168 import os
 169 import re
 170 import shutil
 171 import subprocess
 172 import sys
 173 import tempfile
 174 import time
 175
 176 SHOW_GNU_MAKE = 'scripts/show-gnu-make'
 177 SLEEP_TIME=0.03
 178
 179 # Here is the list of cross-tools I use.
 180 # Most of them are available at kernel.org
 181 # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the followings:
 182 # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
 183 # blackfin: http://sourceforge.net/projects/adi-toolchain/files/
 184 # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz
 185 # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
 186 # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
 187 #
 188 # openrisc kernel.org toolchain is out of date, download latest one from
 189 # http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions
 190 CROSS_COMPILE = {
 191     'arc': 'arc-linux-',
 192     'aarch64': 'aarch64-linux-',
 193     'arm': 'arm-unknown-linux-gnueabi-',
 194     'avr32': 'avr32-linux-',
 195     'blackfin': 'bfin-elf-',
 196     'm68k': 'm68k-linux-',
 197     'microblaze': 'microblaze-linux-',
 198     'mips': 'mips-linux-',
 199     'nds32': 'nds32le-linux-',
 200     'nios2': 'nios2-linux-gnu-',
 201     'openrisc': 'or1k-elf-',
 202     'powerpc': 'powerpc-linux-',
 203     'sh': 'sh-linux-gnu-',
 204     'sparc': 'sparc-linux-',
 205     'x86': 'i386-linux-'
 206 }
 207
 208 STATE_IDLE = 0
 209 STATE_DEFCONFIG = 1
 210 STATE_AUTOCONF = 2
 211 STATE_SAVEDEFCONFIG = 3
 212
 213 ACTION_MOVE = 0
 214 ACTION_DEFAULT_VALUE = 1
 215 ACTION_ALREADY_EXIST = 2
 216 ACTION_UNDEFINED = 3
 217
 218 COLOR_BLACK        = '0;30'
 219 COLOR_RED          = '0;31'
 220 COLOR_GREEN        = '0;32'
 221 COLOR_BROWN        = '0;33'
 222 COLOR_BLUE         = '0;34'
 223 COLOR_PURPLE       = '0;35'
 224 COLOR_CYAN         = '0;36'
 225 COLOR_LIGHT_GRAY   = '0;37'
 226 COLOR_DARK_GRAY    = '1;30'
 227 COLOR_LIGHT_RED    = '1;31'
 228 COLOR_LIGHT_GREEN  = '1;32'
 229 COLOR_YELLOW       = '1;33'
 230 COLOR_LIGHT_BLUE   = '1;34'
 231 COLOR_LIGHT_PURPLE = '1;35'
 232 COLOR_LIGHT_CYAN   = '1;36'
 233 COLOR_WHITE        = '1;37'
 234
 235 ### helper functions ###
 236 def get_devnull():
 237     """Get the file object of '/dev/null' device."""
 238     try:
 239         devnull = subprocess.DEVNULL # py3k
 240     except AttributeError:
 241         devnull = open(os.devnull, 'wb')
 242     return devnull
 243
 244 def check_top_directory():
 245     """Exit if we are not at the top of source directory."""
 246     for f in ('README', 'Licenses'):
 247         if not os.path.exists(f):
 248             sys.exit('Please run at the top of source directory.')
 249
 250 def get_make_cmd():
 251     """Get the command name of GNU Make.
 252
 253     U-Boot needs GNU Make for building, but the command name is not
 254     necessarily "make". (for example, "gmake" on FreeBSD).
 255     Returns the most appropriate command name on your system.
 256     """
 257     process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
 258     ret = process.communicate()
 259     if process.returncode:
 260         sys.exit('GNU Make not found')
 261     return ret[0].rstrip()
 262
 263 def color_text(color_enabled, color, string):
 264     """Return colored string."""
 265     if color_enabled:
 266         return '\033[' + color + 'm' + string + '\033[0m'
 267     else:
 268         return string
 269
 270 def log_msg(color_enabled, color, defconfig, msg):
 271     """Return the formated line for the log."""
 272     return defconfig[:-len('_defconfig')].ljust(37) + ': ' + \
 273         color_text(color_enabled, color, msg) + '\n'
 274
 275 def update_cross_compile():
 276     """Update per-arch CROSS_COMPILE via environment variables
 277
 278     The default CROSS_COMPILE values are available
 279     in the CROSS_COMPILE list above.
 280
 281     You can override them via environment variables
 282     CROSS_COMPILE_{ARCH}.
 283
 284     For example, if you want to override toolchain prefixes
 285     for ARM and PowerPC, you can do as follows in your shell:
 286
 287     export CROSS_COMPILE_ARM=...
 288     export CROSS_COMPILE_POWERPC=...
 289     """
 290     archs = []
 291
 292     for arch in os.listdir('arch'):
 293         if os.path.exists(os.path.join('arch', arch, 'Makefile')):
 294             archs.append(arch)
 295
 296     # arm64 is a special case
 297     archs.append('aarch64')
 298
 299     for arch in archs:
 300         env = 'CROSS_COMPILE_' + arch.upper()
 301         cross_compile = os.environ.get(env)
 302         if cross_compile:
 303             CROSS_COMPILE[arch] = cross_compile
 304
 305 def cleanup_one_header(header_path, patterns, dry_run):
 306     """Clean regex-matched lines away from a file.
 307
 308     Arguments:
 309       header_path: path to the cleaned file.
 310       patterns: list of regex patterns.  Any lines matching to these
 311                 patterns are deleted.
 312       dry_run: make no changes, but still display log.
 313     """
 314     with open(header_path) as f:
 315         lines = f.readlines()
 316
 317     matched = []
 318     for i, line in enumerate(lines):
 319         for pattern in patterns:
 320             m = pattern.search(line)
 321             if m:
 322                 print '%s: %s: %s' % (header_path, i + 1, line),
 323                 matched.append(i)
 324                 break
 325
 326     if dry_run or not matched:
 327         return
 328
 329     with open(header_path, 'w') as f:
 330         for i, line in enumerate(lines):
 331             if not i in matched:
 332                 f.write(line)
 333
 334 def cleanup_headers(config_attrs, dry_run):
 335     """Delete config defines from board headers.
 336
 337     Arguments:
 338       config_attrs: A list of dictionaris, each of them includes the name,
 339                     the type, and the default value of the target config.
 340       dry_run: make no changes, but still display log.
 341     """
 342     while True:
 343         choice = raw_input('Clean up headers? [y/n]: ').lower()
 344         print choice
 345         if choice == 'y' or choice == 'n':
 346             break
 347
 348     if choice == 'n':
 349         return
 350
 351     patterns = []
 352     for config_attr in config_attrs:
 353         config = config_attr['config']
 354         patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
 355         patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
 356
 357     for dir in 'include', 'arch', 'board':
 358         for (dirpath, dirnames, filenames) in os.walk(dir):
 359             for filename in filenames:
 360                 if not fnmatch.fnmatch(filename, '*~'):
 361                     cleanup_one_header(os.path.join(dirpath, filename),
 362                                        patterns, dry_run)
 363
 364 ### classes ###
 365 class KconfigParser:
 366
 367     """A parser of .config and include/autoconf.mk."""
 368
 369     re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
 370     re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
 371
 372     def __init__(self, config_attrs, options, build_dir):
 373         """Create a new parser.
 374
 375         Arguments:
 376           config_attrs: A list of dictionaris, each of them includes the name,
 377                         the type, and the default value of the target config.
 378           options: option flags.
 379           build_dir: Build directory.
 380         """
 381         self.config_attrs = config_attrs
 382         self.options = options
 383         self.build_dir = build_dir
 384
 385     def get_cross_compile(self):
 386         """Parse .config file and return CROSS_COMPILE.
 387
 388         Returns:
 389           A string storing the compiler prefix for the architecture.
 390         """
 391         arch = ''
 392         cpu = ''
 393         dotconfig = os.path.join(self.build_dir, '.config')
 394         for line in open(dotconfig):
 395             m = self.re_arch.match(line)
 396             if m:
 397                 arch = m.group(1)
 398                 continue
 399             m = self.re_cpu.match(line)
 400             if m:
 401                 cpu = m.group(1)
 402
 403         assert arch, 'Error: arch is not defined in %s' % defconfig
 404
 405         # fix-up for aarch64
 406         if arch == 'arm' and cpu == 'armv8':
 407             arch = 'aarch64'
 408
 409         return CROSS_COMPILE.get(arch, '')
 410
 411     def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines):
 412         """Parse .config, defconfig, include/autoconf.mk for one config.
 413
 414         This function looks for the config options in the lines from
 415         defconfig, .config, and include/autoconf.mk in order to decide
 416         which action should be taken for this defconfig.
 417
 418         Arguments:
 419           config_attr: A dictionary including the name, the type,
 420                        and the default value of the target config.
 421           defconfig_lines: lines from the original defconfig file.
 422           autoconf_lines: lines from the include/autoconf.mk file.
 423
 424         Returns:
 425           A tupple of the action for this defconfig and the line
 426           matched for the config.
 427         """
 428         config = config_attr['config']
 429         not_set = '# %s is not set' % config
 430
 431         if config_attr['type'] in ('bool', 'tristate') and \
 432            config_attr['default'] == 'n':
 433             default = not_set
 434         else:
 435             default = config + '=' + config_attr['default']
 436
 437         for line in defconfig_lines:
 438             line = line.rstrip()
 439             if line.startswith(config + '=') or line == not_set:
 440                 return (ACTION_ALREADY_EXIST, line)
 441
 442         if config_attr['type'] in ('bool', 'tristate'):
 443             value = not_set
 444         else:
 445             value = '(undefined)'
 446
 447         for line in autoconf_lines:
 448             line = line.rstrip()
 449             if line.startswith(config + '='):
 450                 value = line
 451                 break
 452
 453         if value == default:
 454             action = ACTION_DEFAULT_VALUE
 455         elif value == '(undefined)':
 456             action = ACTION_UNDEFINED
 457         else:
 458             action = ACTION_MOVE
 459
 460         return (action, value)
 461
 462     def update_dotconfig(self, defconfig):
 463         """Parse files for the config options and update the .config.
 464
 465         This function parses the given defconfig, the generated .config
 466         and include/autoconf.mk searching the target options.
 467         Move the config option(s) to the .config as needed.
 468         Also, display the log to show what happened to the .config.
 469
 470         Arguments:
 471           defconfig: defconfig name.
 472         """
 473
 474         defconfig_path = os.path.join('configs', defconfig)
 475         dotconfig_path = os.path.join(self.build_dir, '.config')
 476         autoconf_path = os.path.join(self.build_dir, 'include', 'autoconf.mk')
 477         results = []
 478
 479         with open(defconfig_path) as f:
 480             defconfig_lines = f.readlines()
 481
 482         with open(autoconf_path) as f:
 483             autoconf_lines = f.readlines()
 484
 485         for config_attr in self.config_attrs:
 486             result = self.parse_one_config(config_attr, defconfig_lines,
 487                                            autoconf_lines)
 488             results.append(result)
 489
 490         log = ''
 491
 492         for (action, value) in results:
 493             if action == ACTION_MOVE:
 494                 actlog = "Move '%s'" % value
 495                 log_color = COLOR_LIGHT_GREEN
 496             elif action == ACTION_DEFAULT_VALUE:
 497                 actlog = "Default value '%s'.  Do nothing." % value
 498                 log_color = COLOR_LIGHT_BLUE
 499             elif action == ACTION_ALREADY_EXIST:
 500                 actlog = "'%s' already defined in Kconfig.  Do nothing." % value
 501                 log_color = COLOR_LIGHT_PURPLE
 502             elif action == ACTION_UNDEFINED:
 503                 actlog = "Undefined.  Do nothing."
 504                 log_color = COLOR_DARK_GRAY
 505             else:
 506                 sys.exit("Internal Error. This should not happen.")
 507
 508             log += log_msg(self.options.color, log_color, defconfig, actlog)
 509
 510         # Some threads are running in parallel.
 511         # Print log in one shot to not mix up logs from different threads.
 512         print log,
 513
 514         with open(dotconfig_path, 'a') as f:
 515             for (action, value) in results:
 516                 if action == ACTION_MOVE:
 517                     f.write(value + '\n')
 518
 519         os.remove(os.path.join(self.build_dir, 'include', 'config', 'auto.conf'))
 520         os.remove(autoconf_path)
 521
 522 class Slot:
 523
 524     """A slot to store a subprocess.
 525
 526     Each instance of this class handles one subprocess.
 527     This class is useful to control multiple threads
 528     for faster processing.
 529     """
 530
 531     def __init__(self, config_attrs, options, devnull, make_cmd):
 532         """Create a new process slot.
 533
 534         Arguments:
 535           config_attrs: A list of dictionaris, each of them includes the name,
 536                         the type, and the default value of the target config.
 537           options: option flags.
 538           devnull: A file object of '/dev/null'.
 539           make_cmd: command name of GNU Make.
 540         """
 541         self.options = options
 542         self.build_dir = tempfile.mkdtemp()
 543         self.devnull = devnull
 544         self.make_cmd = (make_cmd, 'O=' + self.build_dir)
 545         self.parser = KconfigParser(config_attrs, options, self.build_dir)
 546         self.state = STATE_IDLE
 547         self.failed_boards = []
 548
 549     def __del__(self):
 550         """Delete the working directory
 551
 552         This function makes sure the temporary directory is cleaned away
 553         even if Python suddenly dies due to error.  It should be done in here
 554         because it is guranteed the destructor is always invoked when the
 555         instance of the class gets unreferenced.
 556
 557         If the subprocess is still running, wait until it finishes.
 558         """
 559         if self.state != STATE_IDLE:
 560             while self.ps.poll() == None:
 561                 pass
 562         shutil.rmtree(self.build_dir)
 563
 564     def add(self, defconfig, num, total):
 565         """Assign a new subprocess for defconfig and add it to the slot.
 566
 567         If the slot is vacant, create a new subprocess for processing the
 568         given defconfig and add it to the slot.  Just returns False if
 569         the slot is occupied (i.e. the current subprocess is still running).
 570
 571         Arguments:
 572           defconfig: defconfig name.
 573
 574         Returns:
 575           Return True on success or False on failure
 576         """
 577         if self.state != STATE_IDLE:
 578             return False
 579         cmd = list(self.make_cmd)
 580         cmd.append(defconfig)
 581         self.ps = subprocess.Popen(cmd, stdout=self.devnull,
 582                                    stderr=subprocess.PIPE)
 583         self.defconfig = defconfig
 584         self.state = STATE_DEFCONFIG
 585         self.num = num
 586         self.total = total
 587         return True
 588
 589     def poll(self):
 590         """Check the status of the subprocess and handle it as needed.
 591
 592         Returns True if the slot is vacant (i.e. in idle state).
 593         If the configuration is successfully finished, assign a new
 594         subprocess to build include/autoconf.mk.
 595         If include/autoconf.mk is generated, invoke the parser to
 596         parse the .config and the include/autoconf.mk, and then set the
 597         slot back to the idle state.
 598
 599         Returns:
 600           Return True if the subprocess is terminated, False otherwise
 601         """
 602         if self.state == STATE_IDLE:
 603             return True
 604
 605         if self.ps.poll() == None:
 606             return False
 607
 608         if self.ps.poll() != 0:
 609             errmsg = 'Failed to process.'
 610             errout = self.ps.stderr.read()
 611             if errout.find('gcc: command not found') != -1:
 612                 errmsg = 'Compiler not found ('
 613                 errmsg += color_text(self.options.color, COLOR_YELLOW,
 614                                      self.cross_compile)
 615                 errmsg += color_text(self.options.color, COLOR_LIGHT_RED,
 616                                      ')')
 617             print >> sys.stderr, log_msg(self.options.color,
 618                                          COLOR_LIGHT_RED,
 619                                          self.defconfig,
 620                                          errmsg),
 621             if self.options.verbose:
 622                 print >> sys.stderr, color_text(self.options.color,
 623                                                 COLOR_LIGHT_CYAN, errout)
 624             if self.options.exit_on_error:
 625                 sys.exit("Exit on error.")
 626             # If --exit-on-error flag is not set, skip this board and continue.
 627             # Record the failed board.
 628             self.failed_boards.append(self.defconfig)
 629             self.state = STATE_IDLE
 630             return True
 631
 632         if self.state == STATE_AUTOCONF:
 633             self.parser.update_dotconfig(self.defconfig)
 634
 635             print ' %d defconfigs out of %d\r' % (self.num + 1, self.total),
 636             sys.stdout.flush()
 637
 638             """Save off the defconfig in a consistent way"""
 639             cmd = list(self.make_cmd)
 640             cmd.append('savedefconfig')
 641             self.ps = subprocess.Popen(cmd, stdout=self.devnull,
 642                                        stderr=subprocess.PIPE)
 643             self.state = STATE_SAVEDEFCONFIG
 644             return False
 645
 646         if self.state == STATE_SAVEDEFCONFIG:
 647             if not self.options.dry_run:
 648                 shutil.move(os.path.join(self.build_dir, 'defconfig'),
 649                             os.path.join('configs', self.defconfig))
 650             self.state = STATE_IDLE
 651             return True
 652
 653         self.cross_compile = self.parser.get_cross_compile()
 654         cmd = list(self.make_cmd)
 655         if self.cross_compile:
 656             cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
 657         cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
 658         cmd.append('include/config/auto.conf')
 659         """This will be screen-scraped, so be sure the expected text will be
 660         returned consistently on every machine by setting LANG=C"""
 661         self.ps = subprocess.Popen(cmd, stdout=self.devnull,
 662                                    env=dict(os.environ, LANG='C'),
 663                                    stderr=subprocess.PIPE)
 664         self.state = STATE_AUTOCONF
 665         return False
 666
 667     def get_failed_boards(self):
 668         """Returns a list of failed boards (defconfigs) in this slot.
 669         """
 670         return self.failed_boards
 671
 672 class Slots:
 673
 674     """Controller of the array of subprocess slots."""
 675
 676     def __init__(self, config_attrs, options):
 677         """Create a new slots controller.
 678
 679         Arguments:
 680           config_attrs: A list of dictionaris containing the name, the type,
 681                         and the default value of the target CONFIG.
 682           options: option flags.
 683         """
 684         self.options = options
 685         self.slots = []
 686         devnull = get_devnull()
 687         make_cmd = get_make_cmd()
 688         for i in range(options.jobs):
 689             self.slots.append(Slot(config_attrs, options, devnull, make_cmd))
 690
 691     def add(self, defconfig, num, total):
 692         """Add a new subprocess if a vacant slot is found.
 693
 694         Arguments:
 695           defconfig: defconfig name to be put into.
 696
 697         Returns:
 698           Return True on success or False on failure
 699         """
 700         for slot in self.slots:
 701             if slot.add(defconfig, num, total):
 702                 return True
 703         return False
 704
 705     def available(self):
 706         """Check if there is a vacant slot.
 707
 708         Returns:
 709           Return True if at lease one vacant slot is found, False otherwise.
 710         """
 711         for slot in self.slots:
 712             if slot.poll():
 713                 return True
 714         return False
 715
 716     def empty(self):
 717         """Check if all slots are vacant.
 718
 719         Returns:
 720           Return True if all the slots are vacant, False otherwise.
 721         """
 722         ret = True
 723         for slot in self.slots:
 724             if not slot.poll():
 725                 ret = False
 726         return ret
 727
 728     def show_failed_boards(self):
 729         """Display all of the failed boards (defconfigs)."""
 730         failed_boards = []
 731
 732         for slot in self.slots:
 733             failed_boards += slot.get_failed_boards()
 734
 735         if len(failed_boards) > 0:
 736             msg = [ "The following boards were not processed due to error:" ]
 737             msg += failed_boards
 738             for line in msg:
 739                 print >> sys.stderr, color_text(self.options.color,
 740                                                 COLOR_LIGHT_RED, line)
 741
 742             with open('moveconfig.failed', 'w') as f:
 743                 for board in failed_boards:
 744                     f.write(board + '\n')
 745
 746 def move_config(config_attrs, options):
 747     """Move config options to defconfig files.
 748
 749     Arguments:
 750       config_attrs: A list of dictionaris, each of them includes the name,
 751                     the type, and the default value of the target config.
 752       options: option flags
 753     """
 754     if len(config_attrs) == 0:
 755         print 'Nothing to do. exit.'
 756         sys.exit(0)
 757
 758     print 'Move the following CONFIG options (jobs: %d)' % options.jobs
 759     for config_attr in config_attrs:
 760         print '  %s (type: %s, default: %s)' % (config_attr['config'],
 761                                                 config_attr['type'],
 762                                                 config_attr['default'])
 763
 764     if options.defconfigs:
 765         defconfigs = [line.strip() for line in open(options.defconfigs)]
 766         for i, defconfig in enumerate(defconfigs):
 767             if not defconfig.endswith('_defconfig'):
 768                 defconfigs[i] = defconfig + '_defconfig'
 769             if not os.path.exists(os.path.join('configs', defconfigs[i])):
 770                 sys.exit('%s - defconfig does not exist. Stopping.' %
 771                          defconfigs[i])
 772     else:
 773         # All the defconfig files to be processed
 774         defconfigs = []
 775         for (dirpath, dirnames, filenames) in os.walk('configs'):
 776             dirpath = dirpath[len('configs') + 1:]
 777             for filename in fnmatch.filter(filenames, '*_defconfig'):
 778                 defconfigs.append(os.path.join(dirpath, filename))
 779
 780     slots = Slots(config_attrs, options)
 781
 782     # Main loop to process defconfig files:
 783     #  Add a new subprocess into a vacant slot.
 784     #  Sleep if there is no available slot.
 785     for i, defconfig in enumerate(defconfigs):
 786         while not slots.add(defconfig, i, len(defconfigs)):
 787             while not slots.available():
 788                 # No available slot: sleep for a while
 789                 time.sleep(SLEEP_TIME)
 790
 791     # wait until all the subprocesses finish
 792     while not slots.empty():
 793         time.sleep(SLEEP_TIME)
 794
 795     print ''
 796     slots.show_failed_boards()
 797
 798 def bad_recipe(filename, linenum, msg):
 799     """Print error message with the file name and the line number and exit."""
 800     sys.exit("%s: line %d: error : " % (filename, linenum) + msg)
 801
 802 def parse_recipe(filename):
 803     """Parse the recipe file and retrieve the config attributes.
 804
 805     This function parses the given recipe file and gets the name,
 806     the type, and the default value of the target config options.
 807
 808     Arguments:
 809       filename: path to file to be parsed.
 810     Returns:
 811       A list of dictionaris, each of them includes the name,
 812       the type, and the default value of the target config.
 813     """
 814     config_attrs = []
 815     linenum = 1
 816
 817     for line in open(filename):
 818         tokens = line.split()
 819         if len(tokens) != 3:
 820             bad_recipe(filename, linenum,
 821                        "%d fields in this line.  Each line must contain 3 fields"
 822                        % len(tokens))
 823
 824         (config, type, default) = tokens
 825
 826         # prefix the option name with CONFIG_ if missing
 827         if not config.startswith('CONFIG_'):
 828             config = 'CONFIG_' + config
 829
 830         # sanity check of default values
 831         if type == 'bool':
 832             if not default in ('y', 'n'):
 833                 bad_recipe(filename, linenum,
 834                            "default for bool type must be either y or n")
 835         elif type == 'tristate':
 836             if not default in ('y', 'm', 'n'):
 837                 bad_recipe(filename, linenum,
 838                            "default for tristate type must be y, m, or n")
 839         elif type == 'string':
 840             if default[0] != '"' or default[-1] != '"':
 841                 bad_recipe(filename, linenum,
 842                            "default for string type must be surrounded by double-quotations")
 843         elif type == 'int':
 844             try:
 845                 int(default)
 846             except:
 847                 bad_recipe(filename, linenum,
 848                            "type is int, but default value is not decimal")
 849         elif type == 'hex':
 850             if len(default) < 2 or default[:2] != '0x':
 851                 bad_recipe(filename, linenum,
 852                            "default for hex type must be prefixed with 0x")
 853             try:
 854                 int(default, 16)
 855             except:
 856                 bad_recipe(filename, linenum,
 857                            "type is hex, but default value is not hexadecimal")
 858         else:
 859             bad_recipe(filename, linenum,
 860                        "unsupported type '%s'. type must be one of bool, tristate, string, int, hex"
 861                        % type)
 862
 863         config_attrs.append({'config': config, 'type': type, 'default': default})
 864         linenum += 1
 865
 866     return config_attrs
 867
 868 def main():
 869     try:
 870         cpu_count = multiprocessing.cpu_count()
 871     except NotImplementedError:
 872         cpu_count = 1
 873
 874     parser = optparse.OptionParser()
 875     # Add options here
 876     parser.add_option('-c', '--color', action='store_true', default=False,
 877                       help='display the log in color')
 878     parser.add_option('-d', '--defconfigs', type='string',
 879                       help='a file containing a list of defconfigs to move')
 880     parser.add_option('-n', '--dry-run', action='store_true', default=False,
 881                       help='perform a trial run (show log with no changes)')
 882     parser.add_option('-e', '--exit-on-error', action='store_true',
 883                       default=False,
 884                       help='exit immediately on any error')
 885     parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
 886                       action='store_true', default=False,
 887                       help='only cleanup the headers')
 888     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
 889                       help='the number of jobs to run simultaneously')
 890     parser.add_option('-v', '--verbose', action='store_true', default=False,
 891                       help='show any build errors as boards are built')
 892     parser.usage += ' recipe_file\n\n' + \
 893                     'The recipe_file should describe config options you want to move.\n' + \
 894                     'Each line should contain config_name, type, default_value\n\n' + \
 895                     'Example:\n' + \
 896                     'CONFIG_FOO bool n\n' + \
 897                     'CONFIG_BAR int 100\n' + \
 898                     'CONFIG_BAZ string "hello"\n'
 899
 900     (options, args) = parser.parse_args()
 901
 902     if len(args) != 1:
 903         parser.print_usage()
 904         sys.exit(1)
 905
 906     config_attrs = parse_recipe(args[0])
 907
 908     update_cross_compile()
 909
 910     check_top_directory()
 911
 912     if not options.cleanup_headers_only:
 913         move_config(config_attrs, options)
 914
 915     cleanup_headers(config_attrs, options.dry_run)
 916
 917 if __name__ == '__main__':
 918     main()