]>
Commit | Line | Data |
---|---|---|
5a27c734 MY |
1 | #!/usr/bin/env python2 |
2 | # | |
3 | # Author: Masahiro Yamada <yamada.masahiro@socionext.com> | |
4 | # | |
5 | # SPDX-License-Identifier: GPL-2.0+ | |
6 | # | |
7 | ||
8 | """ | |
9 | Move config options from headers to defconfig files. | |
10 | ||
11 | Since Kconfig was introduced to U-Boot, we have worked on moving | |
12 | config options from headers to Kconfig (defconfig). | |
13 | ||
14 | This tool intends to help this tremendous work. | |
15 | ||
16 | ||
17 | Usage | |
18 | ----- | |
19 | ||
b6ef393a | 20 | First, you must edit the Kconfig to add the menu entries for the configs |
96464bad JH |
21 | you are moving. |
22 | ||
b6ef393a MY |
23 | And then run this tool giving CONFIG names you want to move. |
24 | For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE, | |
25 | simply type as follows: | |
5a27c734 | 26 | |
b6ef393a | 27 | $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE |
5a27c734 | 28 | |
b6ef393a | 29 | The tool walks through all the defconfig files and move the given CONFIGs. |
5a27c734 MY |
30 | |
31 | The log is also displayed on the terminal. | |
32 | ||
1d085568 | 33 | The log is printed for each defconfig as follows: |
5a27c734 | 34 | |
1d085568 MY |
35 | <defconfig_name> |
36 | <action1> | |
37 | <action2> | |
38 | <action3> | |
39 | ... | |
5a27c734 | 40 | |
1d085568 MY |
41 | <defconfig_name> is the name of the defconfig. |
42 | ||
43 | <action*> shows what the tool did for that defconfig. | |
5a27c734 MY |
44 | It looks like one of the followings: |
45 | ||
46 | - Move 'CONFIG_... ' | |
47 | This config option was moved to the defconfig | |
48 | ||
cc008299 MY |
49 | - CONFIG_... is not defined in Kconfig. Do nothing. |
50 | The entry for this CONFIG was not found in Kconfig. | |
51 | There are two common cases: | |
52 | - You forgot to create an entry for the CONFIG before running | |
53 | this tool, or made a typo in a CONFIG passed to this tool. | |
54 | - The entry was hidden due to unmet 'depends on'. | |
55 | This is correct behavior. | |
56 | ||
57 | - 'CONFIG_...' is the same as the define in Kconfig. Do nothing. | |
58 | The define in the config header matched the one in Kconfig. | |
59 | We do not need to touch it. | |
5a27c734 MY |
60 | |
61 | - Undefined. Do nothing. | |
62 | This config option was not found in the config header. | |
63 | Nothing to do. | |
64 | ||
90ed6cba MY |
65 | - Compiler is missing. Do nothing. |
66 | The compiler specified for this architecture was not found | |
67 | in your PATH environment. | |
68 | (If -e option is passed, the tool exits immediately.) | |
69 | ||
70 | - Failed to process. | |
5a27c734 MY |
71 | An error occurred during processing this defconfig. Skipped. |
72 | (If -e option is passed, the tool exits immediately on error.) | |
73 | ||
74 | Finally, you will be asked, Clean up headers? [y/n]: | |
75 | ||
76 | If you say 'y' here, the unnecessary config defines are removed | |
77 | from the config headers (include/configs/*.h). | |
78 | It just uses the regex method, so you should not rely on it. | |
79 | Just in case, please do 'git diff' to see what happened. | |
80 | ||
81 | ||
b6ef393a MY |
82 | How does it work? |
83 | ----------------- | |
5a27c734 MY |
84 | |
85 | This tool runs configuration and builds include/autoconf.mk for every | |
86 | defconfig. The config options defined in Kconfig appear in the .config | |
87 | file (unless they are hidden because of unmet dependency.) | |
88 | On the other hand, the config options defined by board headers are seen | |
89 | in include/autoconf.mk. The tool looks for the specified options in both | |
b6ef393a MY |
90 | of them to decide the appropriate action for the options. If the given |
91 | config option is found in the .config, but its value does not match the | |
92 | one from the board header, the config option in the .config is replaced | |
93 | with the define in the board header. Then, the .config is synced by | |
94 | "make savedefconfig" and the defconfig is updated with it. | |
5a27c734 MY |
95 | |
96 | For faster processing, this tool handles multi-threading. It creates | |
97 | separate build directories where the out-of-tree build is run. The | |
98 | temporary build directories are automatically created and deleted as | |
99 | needed. The number of threads are chosen based on the number of the CPU | |
100 | cores of your system although you can change it via -j (--jobs) option. | |
101 | ||
102 | ||
103 | Toolchains | |
104 | ---------- | |
105 | ||
106 | Appropriate toolchain are necessary to generate include/autoconf.mk | |
107 | for all the architectures supported by U-Boot. Most of them are available | |
108 | at the kernel.org site, some are not provided by kernel.org. | |
109 | ||
110 | The default per-arch CROSS_COMPILE used by this tool is specified by | |
111 | the list below, CROSS_COMPILE. You may wish to update the list to | |
112 | use your own. Instead of modifying the list directly, you can give | |
113 | them via environments. | |
114 | ||
115 | ||
116 | Available options | |
117 | ----------------- | |
118 | ||
119 | -c, --color | |
120 | Surround each portion of the log with escape sequences to display it | |
121 | in color on the terminal. | |
122 | ||
91040e85 JH |
123 | -d, --defconfigs |
124 | Specify a file containing a list of defconfigs to move | |
125 | ||
5a27c734 | 126 | -n, --dry-run |
b6ef393a | 127 | Perform a trial run that does not make any changes. It is useful to |
5a27c734 MY |
128 | see what is going to happen before one actually runs it. |
129 | ||
130 | -e, --exit-on-error | |
131 | Exit immediately if Make exits with a non-zero status while processing | |
132 | a defconfig file. | |
133 | ||
2144f880 JH |
134 | -H, --headers-only |
135 | Only cleanup the headers; skip the defconfig processing | |
136 | ||
5a27c734 MY |
137 | -j, --jobs |
138 | Specify the number of threads to run simultaneously. If not specified, | |
139 | the number of threads is the same as the number of CPU cores. | |
140 | ||
95bf9c7e JH |
141 | -v, --verbose |
142 | Show any build errors as boards are built | |
143 | ||
5a27c734 MY |
144 | To see the complete list of supported options, run |
145 | ||
146 | $ tools/moveconfig.py -h | |
147 | ||
148 | """ | |
149 | ||
150 | import fnmatch | |
151 | import multiprocessing | |
152 | import optparse | |
153 | import os | |
154 | import re | |
155 | import shutil | |
156 | import subprocess | |
157 | import sys | |
158 | import tempfile | |
159 | import time | |
160 | ||
161 | SHOW_GNU_MAKE = 'scripts/show-gnu-make' | |
162 | SLEEP_TIME=0.03 | |
163 | ||
164 | # Here is the list of cross-tools I use. | |
165 | # Most of them are available at kernel.org | |
166 | # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the followings: | |
167 | # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases | |
168 | # blackfin: http://sourceforge.net/projects/adi-toolchain/files/ | |
4440ecec | 169 | # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz |
5a27c734 MY |
170 | # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545 |
171 | # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu | |
e8aebc47 BM |
172 | # |
173 | # openrisc kernel.org toolchain is out of date, download latest one from | |
174 | # http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions | |
5a27c734 MY |
175 | CROSS_COMPILE = { |
176 | 'arc': 'arc-linux-', | |
177 | 'aarch64': 'aarch64-linux-', | |
178 | 'arm': 'arm-unknown-linux-gnueabi-', | |
179 | 'avr32': 'avr32-linux-', | |
180 | 'blackfin': 'bfin-elf-', | |
181 | 'm68k': 'm68k-linux-', | |
182 | 'microblaze': 'microblaze-linux-', | |
183 | 'mips': 'mips-linux-', | |
184 | 'nds32': 'nds32le-linux-', | |
185 | 'nios2': 'nios2-linux-gnu-', | |
e8aebc47 | 186 | 'openrisc': 'or1k-elf-', |
5a27c734 MY |
187 | 'powerpc': 'powerpc-linux-', |
188 | 'sh': 'sh-linux-gnu-', | |
189 | 'sparc': 'sparc-linux-', | |
190 | 'x86': 'i386-linux-' | |
191 | } | |
192 | ||
193 | STATE_IDLE = 0 | |
194 | STATE_DEFCONFIG = 1 | |
195 | STATE_AUTOCONF = 2 | |
96464bad | 196 | STATE_SAVEDEFCONFIG = 3 |
5a27c734 MY |
197 | |
198 | ACTION_MOVE = 0 | |
cc008299 MY |
199 | ACTION_NO_ENTRY = 1 |
200 | ACTION_NO_CHANGE = 2 | |
5a27c734 MY |
201 | |
202 | COLOR_BLACK = '0;30' | |
203 | COLOR_RED = '0;31' | |
204 | COLOR_GREEN = '0;32' | |
205 | COLOR_BROWN = '0;33' | |
206 | COLOR_BLUE = '0;34' | |
207 | COLOR_PURPLE = '0;35' | |
208 | COLOR_CYAN = '0;36' | |
209 | COLOR_LIGHT_GRAY = '0;37' | |
210 | COLOR_DARK_GRAY = '1;30' | |
211 | COLOR_LIGHT_RED = '1;31' | |
212 | COLOR_LIGHT_GREEN = '1;32' | |
213 | COLOR_YELLOW = '1;33' | |
214 | COLOR_LIGHT_BLUE = '1;34' | |
215 | COLOR_LIGHT_PURPLE = '1;35' | |
216 | COLOR_LIGHT_CYAN = '1;36' | |
217 | COLOR_WHITE = '1;37' | |
218 | ||
219 | ### helper functions ### | |
220 | def get_devnull(): | |
221 | """Get the file object of '/dev/null' device.""" | |
222 | try: | |
223 | devnull = subprocess.DEVNULL # py3k | |
224 | except AttributeError: | |
225 | devnull = open(os.devnull, 'wb') | |
226 | return devnull | |
227 | ||
228 | def check_top_directory(): | |
229 | """Exit if we are not at the top of source directory.""" | |
230 | for f in ('README', 'Licenses'): | |
231 | if not os.path.exists(f): | |
232 | sys.exit('Please run at the top of source directory.') | |
233 | ||
bd63e5ba MY |
234 | def check_clean_directory(): |
235 | """Exit if the source tree is not clean.""" | |
236 | for f in ('.config', 'include/config'): | |
237 | if os.path.exists(f): | |
238 | sys.exit("source tree is not clean, please run 'make mrproper'") | |
239 | ||
5a27c734 MY |
240 | def get_make_cmd(): |
241 | """Get the command name of GNU Make. | |
242 | ||
243 | U-Boot needs GNU Make for building, but the command name is not | |
244 | necessarily "make". (for example, "gmake" on FreeBSD). | |
245 | Returns the most appropriate command name on your system. | |
246 | """ | |
247 | process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) | |
248 | ret = process.communicate() | |
249 | if process.returncode: | |
250 | sys.exit('GNU Make not found') | |
251 | return ret[0].rstrip() | |
252 | ||
253 | def color_text(color_enabled, color, string): | |
254 | """Return colored string.""" | |
255 | if color_enabled: | |
1d085568 MY |
256 | # LF should not be surrounded by the escape sequence. |
257 | # Otherwise, additional whitespace or line-feed might be printed. | |
258 | return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else '' | |
259 | for s in string.split('\n') ]) | |
5a27c734 MY |
260 | else: |
261 | return string | |
262 | ||
90ed6cba | 263 | def update_cross_compile(color_enabled): |
1cc0a9f4 | 264 | """Update per-arch CROSS_COMPILE via environment variables |
5a27c734 MY |
265 | |
266 | The default CROSS_COMPILE values are available | |
267 | in the CROSS_COMPILE list above. | |
268 | ||
1cc0a9f4 | 269 | You can override them via environment variables |
5a27c734 MY |
270 | CROSS_COMPILE_{ARCH}. |
271 | ||
272 | For example, if you want to override toolchain prefixes | |
273 | for ARM and PowerPC, you can do as follows in your shell: | |
274 | ||
275 | export CROSS_COMPILE_ARM=... | |
276 | export CROSS_COMPILE_POWERPC=... | |
90ed6cba MY |
277 | |
278 | Then, this function checks if specified compilers really exist in your | |
279 | PATH environment. | |
5a27c734 MY |
280 | """ |
281 | archs = [] | |
282 | ||
283 | for arch in os.listdir('arch'): | |
284 | if os.path.exists(os.path.join('arch', arch, 'Makefile')): | |
285 | archs.append(arch) | |
286 | ||
287 | # arm64 is a special case | |
288 | archs.append('aarch64') | |
289 | ||
290 | for arch in archs: | |
291 | env = 'CROSS_COMPILE_' + arch.upper() | |
292 | cross_compile = os.environ.get(env) | |
90ed6cba MY |
293 | if not cross_compile: |
294 | cross_compile = CROSS_COMPILE.get(arch, '') | |
295 | ||
296 | for path in os.environ["PATH"].split(os.pathsep): | |
297 | gcc_path = os.path.join(path, cross_compile + 'gcc') | |
298 | if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK): | |
299 | break | |
300 | else: | |
301 | print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW, | |
302 | 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped' | |
303 | % (cross_compile, arch)) | |
304 | cross_compile = None | |
305 | ||
306 | CROSS_COMPILE[arch] = cross_compile | |
5a27c734 MY |
307 | |
308 | def cleanup_one_header(header_path, patterns, dry_run): | |
309 | """Clean regex-matched lines away from a file. | |
310 | ||
311 | Arguments: | |
312 | header_path: path to the cleaned file. | |
313 | patterns: list of regex patterns. Any lines matching to these | |
314 | patterns are deleted. | |
315 | dry_run: make no changes, but still display log. | |
316 | """ | |
317 | with open(header_path) as f: | |
318 | lines = f.readlines() | |
319 | ||
320 | matched = [] | |
321 | for i, line in enumerate(lines): | |
322 | for pattern in patterns: | |
323 | m = pattern.search(line) | |
324 | if m: | |
325 | print '%s: %s: %s' % (header_path, i + 1, line), | |
326 | matched.append(i) | |
327 | break | |
328 | ||
329 | if dry_run or not matched: | |
330 | return | |
331 | ||
332 | with open(header_path, 'w') as f: | |
333 | for i, line in enumerate(lines): | |
334 | if not i in matched: | |
335 | f.write(line) | |
336 | ||
b134bc13 | 337 | def cleanup_headers(configs, dry_run): |
5a27c734 MY |
338 | """Delete config defines from board headers. |
339 | ||
340 | Arguments: | |
b134bc13 | 341 | configs: A list of CONFIGs to remove. |
5a27c734 MY |
342 | dry_run: make no changes, but still display log. |
343 | """ | |
344 | while True: | |
345 | choice = raw_input('Clean up headers? [y/n]: ').lower() | |
346 | print choice | |
347 | if choice == 'y' or choice == 'n': | |
348 | break | |
349 | ||
350 | if choice == 'n': | |
351 | return | |
352 | ||
353 | patterns = [] | |
b134bc13 | 354 | for config in configs: |
5a27c734 MY |
355 | patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) |
356 | patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) | |
357 | ||
60727f51 JH |
358 | for dir in 'include', 'arch', 'board': |
359 | for (dirpath, dirnames, filenames) in os.walk(dir): | |
360 | for filename in filenames: | |
361 | if not fnmatch.fnmatch(filename, '*~'): | |
362 | cleanup_one_header(os.path.join(dirpath, filename), | |
363 | patterns, dry_run) | |
5a27c734 MY |
364 | |
365 | ### classes ### | |
c5e60fd4 MY |
366 | class Progress: |
367 | ||
368 | """Progress Indicator""" | |
369 | ||
370 | def __init__(self, total): | |
371 | """Create a new progress indicator. | |
372 | ||
373 | Arguments: | |
374 | total: A number of defconfig files to process. | |
375 | """ | |
376 | self.current = 0 | |
377 | self.total = total | |
378 | ||
379 | def inc(self): | |
380 | """Increment the number of processed defconfig files.""" | |
381 | ||
382 | self.current += 1 | |
383 | ||
384 | def show(self): | |
385 | """Display the progress.""" | |
386 | print ' %d defconfigs out of %d\r' % (self.current, self.total), | |
387 | sys.stdout.flush() | |
388 | ||
5a27c734 MY |
389 | class KconfigParser: |
390 | ||
391 | """A parser of .config and include/autoconf.mk.""" | |
392 | ||
393 | re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') | |
394 | re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') | |
395 | ||
522e8dcb | 396 | def __init__(self, configs, options, build_dir): |
5a27c734 MY |
397 | """Create a new parser. |
398 | ||
399 | Arguments: | |
b134bc13 | 400 | configs: A list of CONFIGs to move. |
5a27c734 MY |
401 | options: option flags. |
402 | build_dir: Build directory. | |
403 | """ | |
b134bc13 | 404 | self.configs = configs |
5a27c734 | 405 | self.options = options |
1f16992e MY |
406 | self.dotconfig = os.path.join(build_dir, '.config') |
407 | self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk') | |
408 | self.config_autoconf = os.path.join(build_dir, 'include', 'config', | |
409 | 'auto.conf') | |
5a27c734 MY |
410 | |
411 | def get_cross_compile(self): | |
412 | """Parse .config file and return CROSS_COMPILE. | |
413 | ||
414 | Returns: | |
415 | A string storing the compiler prefix for the architecture. | |
90ed6cba MY |
416 | Return a NULL string for architectures that do not require |
417 | compiler prefix (Sandbox and native build is the case). | |
418 | Return None if the specified compiler is missing in your PATH. | |
419 | Caller should distinguish '' and None. | |
5a27c734 MY |
420 | """ |
421 | arch = '' | |
422 | cpu = '' | |
1f16992e | 423 | for line in open(self.dotconfig): |
5a27c734 MY |
424 | m = self.re_arch.match(line) |
425 | if m: | |
426 | arch = m.group(1) | |
427 | continue | |
428 | m = self.re_cpu.match(line) | |
429 | if m: | |
430 | cpu = m.group(1) | |
431 | ||
90ed6cba MY |
432 | if not arch: |
433 | return None | |
5a27c734 MY |
434 | |
435 | # fix-up for aarch64 | |
436 | if arch == 'arm' and cpu == 'armv8': | |
437 | arch = 'aarch64' | |
438 | ||
90ed6cba | 439 | return CROSS_COMPILE.get(arch, None) |
5a27c734 | 440 | |
b134bc13 | 441 | def parse_one_config(self, config, dotconfig_lines, autoconf_lines): |
5a27c734 MY |
442 | """Parse .config, defconfig, include/autoconf.mk for one config. |
443 | ||
444 | This function looks for the config options in the lines from | |
445 | defconfig, .config, and include/autoconf.mk in order to decide | |
446 | which action should be taken for this defconfig. | |
447 | ||
448 | Arguments: | |
b134bc13 | 449 | config: CONFIG name to parse. |
cc008299 | 450 | dotconfig_lines: lines from the .config file. |
5a27c734 MY |
451 | autoconf_lines: lines from the include/autoconf.mk file. |
452 | ||
453 | Returns: | |
454 | A tupple of the action for this defconfig and the line | |
455 | matched for the config. | |
456 | """ | |
5a27c734 MY |
457 | not_set = '# %s is not set' % config |
458 | ||
cc008299 | 459 | for line in dotconfig_lines: |
5a27c734 MY |
460 | line = line.rstrip() |
461 | if line.startswith(config + '=') or line == not_set: | |
cc008299 MY |
462 | old_val = line |
463 | break | |
5a27c734 | 464 | else: |
cc008299 | 465 | return (ACTION_NO_ENTRY, config) |
5a27c734 MY |
466 | |
467 | for line in autoconf_lines: | |
468 | line = line.rstrip() | |
469 | if line.startswith(config + '='): | |
cc008299 | 470 | new_val = line |
5a27c734 | 471 | break |
5a27c734 | 472 | else: |
cc008299 MY |
473 | new_val = not_set |
474 | ||
475 | if old_val == new_val: | |
476 | return (ACTION_NO_CHANGE, new_val) | |
477 | ||
478 | # If this CONFIG is neither bool nor trisate | |
479 | if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set: | |
480 | # tools/scripts/define2mk.sed changes '1' to 'y'. | |
481 | # This is a problem if the CONFIG is int type. | |
482 | # Check the type in Kconfig and handle it correctly. | |
483 | if new_val[-2:] == '=y': | |
484 | new_val = new_val[:-1] + '1' | |
5a27c734 | 485 | |
cc008299 | 486 | return (ACTION_MOVE, new_val) |
5a27c734 | 487 | |
1d085568 | 488 | def update_dotconfig(self): |
6ff36d21 | 489 | """Parse files for the config options and update the .config. |
5a27c734 | 490 | |
cc008299 MY |
491 | This function parses the generated .config and include/autoconf.mk |
492 | searching the target options. | |
6ff36d21 | 493 | Move the config option(s) to the .config as needed. |
5a27c734 MY |
494 | |
495 | Arguments: | |
496 | defconfig: defconfig name. | |
522e8dcb MY |
497 | |
498 | Returns: | |
7fb0bacd MY |
499 | Return a tuple of (updated flag, log string). |
500 | The "updated flag" is True if the .config was updated, False | |
501 | otherwise. The "log string" shows what happend to the .config. | |
5a27c734 MY |
502 | """ |
503 | ||
5a27c734 | 504 | results = [] |
7fb0bacd | 505 | updated = False |
5a27c734 | 506 | |
1f16992e | 507 | with open(self.dotconfig) as f: |
cc008299 | 508 | dotconfig_lines = f.readlines() |
5a27c734 | 509 | |
1f16992e | 510 | with open(self.autoconf) as f: |
5a27c734 MY |
511 | autoconf_lines = f.readlines() |
512 | ||
b134bc13 MY |
513 | for config in self.configs: |
514 | result = self.parse_one_config(config, dotconfig_lines, | |
96464bad | 515 | autoconf_lines) |
5a27c734 MY |
516 | results.append(result) |
517 | ||
518 | log = '' | |
519 | ||
520 | for (action, value) in results: | |
521 | if action == ACTION_MOVE: | |
522 | actlog = "Move '%s'" % value | |
523 | log_color = COLOR_LIGHT_GREEN | |
cc008299 MY |
524 | elif action == ACTION_NO_ENTRY: |
525 | actlog = "%s is not defined in Kconfig. Do nothing." % value | |
5a27c734 | 526 | log_color = COLOR_LIGHT_BLUE |
cc008299 MY |
527 | elif action == ACTION_NO_CHANGE: |
528 | actlog = "'%s' is the same as the define in Kconfig. Do nothing." \ | |
529 | % value | |
5a27c734 | 530 | log_color = COLOR_LIGHT_PURPLE |
5a27c734 MY |
531 | else: |
532 | sys.exit("Internal Error. This should not happen.") | |
533 | ||
1d085568 | 534 | log += color_text(self.options.color, log_color, actlog) + '\n' |
5a27c734 | 535 | |
1f16992e | 536 | with open(self.dotconfig, 'a') as f: |
e423d17f MY |
537 | for (action, value) in results: |
538 | if action == ACTION_MOVE: | |
539 | f.write(value + '\n') | |
7fb0bacd | 540 | updated = True |
5a27c734 | 541 | |
1f16992e MY |
542 | os.remove(self.config_autoconf) |
543 | os.remove(self.autoconf) | |
5a27c734 | 544 | |
7fb0bacd | 545 | return (updated, log) |
522e8dcb | 546 | |
5a27c734 MY |
547 | class Slot: |
548 | ||
549 | """A slot to store a subprocess. | |
550 | ||
551 | Each instance of this class handles one subprocess. | |
552 | This class is useful to control multiple threads | |
553 | for faster processing. | |
554 | """ | |
555 | ||
b134bc13 | 556 | def __init__(self, configs, options, progress, devnull, make_cmd): |
5a27c734 MY |
557 | """Create a new process slot. |
558 | ||
559 | Arguments: | |
b134bc13 | 560 | configs: A list of CONFIGs to move. |
5a27c734 | 561 | options: option flags. |
c5e60fd4 | 562 | progress: A progress indicator. |
5a27c734 MY |
563 | devnull: A file object of '/dev/null'. |
564 | make_cmd: command name of GNU Make. | |
565 | """ | |
566 | self.options = options | |
c5e60fd4 | 567 | self.progress = progress |
5a27c734 MY |
568 | self.build_dir = tempfile.mkdtemp() |
569 | self.devnull = devnull | |
570 | self.make_cmd = (make_cmd, 'O=' + self.build_dir) | |
522e8dcb | 571 | self.parser = KconfigParser(configs, options, self.build_dir) |
5a27c734 MY |
572 | self.state = STATE_IDLE |
573 | self.failed_boards = [] | |
574 | ||
575 | def __del__(self): | |
576 | """Delete the working directory | |
577 | ||
578 | This function makes sure the temporary directory is cleaned away | |
579 | even if Python suddenly dies due to error. It should be done in here | |
580 | because it is guranteed the destructor is always invoked when the | |
581 | instance of the class gets unreferenced. | |
582 | ||
583 | If the subprocess is still running, wait until it finishes. | |
584 | """ | |
585 | if self.state != STATE_IDLE: | |
586 | while self.ps.poll() == None: | |
587 | pass | |
588 | shutil.rmtree(self.build_dir) | |
589 | ||
c5e60fd4 | 590 | def add(self, defconfig): |
5a27c734 MY |
591 | """Assign a new subprocess for defconfig and add it to the slot. |
592 | ||
593 | If the slot is vacant, create a new subprocess for processing the | |
594 | given defconfig and add it to the slot. Just returns False if | |
595 | the slot is occupied (i.e. the current subprocess is still running). | |
596 | ||
597 | Arguments: | |
598 | defconfig: defconfig name. | |
599 | ||
600 | Returns: | |
601 | Return True on success or False on failure | |
602 | """ | |
603 | if self.state != STATE_IDLE: | |
604 | return False | |
605 | cmd = list(self.make_cmd) | |
606 | cmd.append(defconfig) | |
25400090 JH |
607 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, |
608 | stderr=subprocess.PIPE) | |
5a27c734 MY |
609 | self.defconfig = defconfig |
610 | self.state = STATE_DEFCONFIG | |
1d085568 | 611 | self.log = '' |
5a27c734 MY |
612 | return True |
613 | ||
614 | def poll(self): | |
615 | """Check the status of the subprocess and handle it as needed. | |
616 | ||
617 | Returns True if the slot is vacant (i.e. in idle state). | |
618 | If the configuration is successfully finished, assign a new | |
619 | subprocess to build include/autoconf.mk. | |
620 | If include/autoconf.mk is generated, invoke the parser to | |
7fb0bacd MY |
621 | parse the .config and the include/autoconf.mk, moving |
622 | config options to the .config as needed. | |
623 | If the .config was updated, run "make savedefconfig" to sync | |
624 | it, update the original defconfig, and then set the slot back | |
625 | to the idle state. | |
5a27c734 MY |
626 | |
627 | Returns: | |
628 | Return True if the subprocess is terminated, False otherwise | |
629 | """ | |
630 | if self.state == STATE_IDLE: | |
631 | return True | |
632 | ||
633 | if self.ps.poll() == None: | |
634 | return False | |
635 | ||
636 | if self.ps.poll() != 0: | |
1d085568 MY |
637 | self.log += color_text(self.options.color, COLOR_LIGHT_RED, |
638 | "Failed to process.\n") | |
95bf9c7e | 639 | if self.options.verbose: |
1d085568 MY |
640 | self.log += color_text(self.options.color, COLOR_LIGHT_CYAN, |
641 | self.ps.stderr.read()) | |
4efef998 | 642 | self.finish(False) |
ff8725bb | 643 | return True |
5a27c734 MY |
644 | |
645 | if self.state == STATE_AUTOCONF: | |
7fb0bacd MY |
646 | (updated, log) = self.parser.update_dotconfig() |
647 | self.log += log | |
96464bad | 648 | |
7fb0bacd MY |
649 | if not updated: |
650 | self.finish(True) | |
651 | return True | |
c1c4d0f0 MY |
652 | self.log += color_text(self.options.color, COLOR_LIGHT_GREEN, |
653 | "Syncing by savedefconfig...\n") | |
96464bad JH |
654 | cmd = list(self.make_cmd) |
655 | cmd.append('savedefconfig') | |
656 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, | |
25400090 | 657 | stderr=subprocess.PIPE) |
96464bad JH |
658 | self.state = STATE_SAVEDEFCONFIG |
659 | return False | |
660 | ||
661 | if self.state == STATE_SAVEDEFCONFIG: | |
e423d17f MY |
662 | if not self.options.dry_run: |
663 | shutil.move(os.path.join(self.build_dir, 'defconfig'), | |
664 | os.path.join('configs', self.defconfig)) | |
4efef998 | 665 | self.finish(True) |
5a27c734 MY |
666 | return True |
667 | ||
25400090 | 668 | self.cross_compile = self.parser.get_cross_compile() |
90ed6cba | 669 | if self.cross_compile is None: |
1d085568 MY |
670 | self.log += color_text(self.options.color, COLOR_YELLOW, |
671 | "Compiler is missing. Do nothing.\n") | |
4efef998 | 672 | self.finish(False) |
90ed6cba MY |
673 | return True |
674 | ||
5a27c734 | 675 | cmd = list(self.make_cmd) |
25400090 JH |
676 | if self.cross_compile: |
677 | cmd.append('CROSS_COMPILE=%s' % self.cross_compile) | |
7740f653 | 678 | cmd.append('KCONFIG_IGNORE_DUPLICATES=1') |
5a27c734 | 679 | cmd.append('include/config/auto.conf') |
25400090 | 680 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, |
25400090 | 681 | stderr=subprocess.PIPE) |
5a27c734 MY |
682 | self.state = STATE_AUTOCONF |
683 | return False | |
684 | ||
4efef998 MY |
685 | def finish(self, success): |
686 | """Display log along with progress and go to the idle state. | |
1d085568 MY |
687 | |
688 | Arguments: | |
4efef998 MY |
689 | success: Should be True when the defconfig was processed |
690 | successfully, or False when it fails. | |
1d085568 MY |
691 | """ |
692 | # output at least 30 characters to hide the "* defconfigs out of *". | |
693 | log = self.defconfig.ljust(30) + '\n' | |
694 | ||
695 | log += '\n'.join([ ' ' + s for s in self.log.split('\n') ]) | |
696 | # Some threads are running in parallel. | |
697 | # Print log atomically to not mix up logs from different threads. | |
4efef998 MY |
698 | print >> (sys.stdout if success else sys.stderr), log |
699 | ||
700 | if not success: | |
701 | if self.options.exit_on_error: | |
702 | sys.exit("Exit on error.") | |
703 | # If --exit-on-error flag is not set, skip this board and continue. | |
704 | # Record the failed board. | |
705 | self.failed_boards.append(self.defconfig) | |
706 | ||
1d085568 MY |
707 | self.progress.inc() |
708 | self.progress.show() | |
4efef998 | 709 | self.state = STATE_IDLE |
1d085568 | 710 | |
5a27c734 MY |
711 | def get_failed_boards(self): |
712 | """Returns a list of failed boards (defconfigs) in this slot. | |
713 | """ | |
714 | return self.failed_boards | |
715 | ||
716 | class Slots: | |
717 | ||
718 | """Controller of the array of subprocess slots.""" | |
719 | ||
b134bc13 | 720 | def __init__(self, configs, options, progress): |
5a27c734 MY |
721 | """Create a new slots controller. |
722 | ||
723 | Arguments: | |
b134bc13 | 724 | configs: A list of CONFIGs to move. |
5a27c734 | 725 | options: option flags. |
c5e60fd4 | 726 | progress: A progress indicator. |
5a27c734 MY |
727 | """ |
728 | self.options = options | |
729 | self.slots = [] | |
730 | devnull = get_devnull() | |
731 | make_cmd = get_make_cmd() | |
732 | for i in range(options.jobs): | |
b134bc13 | 733 | self.slots.append(Slot(configs, options, progress, devnull, |
c5e60fd4 | 734 | make_cmd)) |
5a27c734 | 735 | |
c5e60fd4 | 736 | def add(self, defconfig): |
5a27c734 MY |
737 | """Add a new subprocess if a vacant slot is found. |
738 | ||
739 | Arguments: | |
740 | defconfig: defconfig name to be put into. | |
741 | ||
742 | Returns: | |
743 | Return True on success or False on failure | |
744 | """ | |
745 | for slot in self.slots: | |
c5e60fd4 | 746 | if slot.add(defconfig): |
5a27c734 MY |
747 | return True |
748 | return False | |
749 | ||
750 | def available(self): | |
751 | """Check if there is a vacant slot. | |
752 | ||
753 | Returns: | |
754 | Return True if at lease one vacant slot is found, False otherwise. | |
755 | """ | |
756 | for slot in self.slots: | |
757 | if slot.poll(): | |
758 | return True | |
759 | return False | |
760 | ||
761 | def empty(self): | |
762 | """Check if all slots are vacant. | |
763 | ||
764 | Returns: | |
765 | Return True if all the slots are vacant, False otherwise. | |
766 | """ | |
767 | ret = True | |
768 | for slot in self.slots: | |
769 | if not slot.poll(): | |
770 | ret = False | |
771 | return ret | |
772 | ||
773 | def show_failed_boards(self): | |
774 | """Display all of the failed boards (defconfigs).""" | |
775 | failed_boards = [] | |
776 | ||
777 | for slot in self.slots: | |
778 | failed_boards += slot.get_failed_boards() | |
779 | ||
780 | if len(failed_boards) > 0: | |
781 | msg = [ "The following boards were not processed due to error:" ] | |
782 | msg += failed_boards | |
783 | for line in msg: | |
784 | print >> sys.stderr, color_text(self.options.color, | |
785 | COLOR_LIGHT_RED, line) | |
786 | ||
2559cd89 JH |
787 | with open('moveconfig.failed', 'w') as f: |
788 | for board in failed_boards: | |
789 | f.write(board + '\n') | |
790 | ||
b134bc13 | 791 | def move_config(configs, options): |
5a27c734 MY |
792 | """Move config options to defconfig files. |
793 | ||
794 | Arguments: | |
b134bc13 | 795 | configs: A list of CONFIGs to move. |
5a27c734 MY |
796 | options: option flags |
797 | """ | |
b134bc13 | 798 | if len(configs) == 0: |
5a27c734 MY |
799 | print 'Nothing to do. exit.' |
800 | sys.exit(0) | |
801 | ||
b134bc13 | 802 | print 'Move %s (jobs: %d)' % (', '.join(configs), options.jobs) |
5a27c734 | 803 | |
91040e85 JH |
804 | if options.defconfigs: |
805 | defconfigs = [line.strip() for line in open(options.defconfigs)] | |
806 | for i, defconfig in enumerate(defconfigs): | |
807 | if not defconfig.endswith('_defconfig'): | |
808 | defconfigs[i] = defconfig + '_defconfig' | |
809 | if not os.path.exists(os.path.join('configs', defconfigs[i])): | |
810 | sys.exit('%s - defconfig does not exist. Stopping.' % | |
811 | defconfigs[i]) | |
812 | else: | |
813 | # All the defconfig files to be processed | |
814 | defconfigs = [] | |
815 | for (dirpath, dirnames, filenames) in os.walk('configs'): | |
816 | dirpath = dirpath[len('configs') + 1:] | |
817 | for filename in fnmatch.filter(filenames, '*_defconfig'): | |
818 | defconfigs.append(os.path.join(dirpath, filename)) | |
5a27c734 | 819 | |
c5e60fd4 | 820 | progress = Progress(len(defconfigs)) |
b134bc13 | 821 | slots = Slots(configs, options, progress) |
5a27c734 MY |
822 | |
823 | # Main loop to process defconfig files: | |
824 | # Add a new subprocess into a vacant slot. | |
825 | # Sleep if there is no available slot. | |
c5e60fd4 MY |
826 | for defconfig in defconfigs: |
827 | while not slots.add(defconfig): | |
5a27c734 MY |
828 | while not slots.available(): |
829 | # No available slot: sleep for a while | |
830 | time.sleep(SLEEP_TIME) | |
831 | ||
832 | # wait until all the subprocesses finish | |
833 | while not slots.empty(): | |
834 | time.sleep(SLEEP_TIME) | |
835 | ||
2e2ce6c0 | 836 | print '' |
5a27c734 MY |
837 | slots.show_failed_boards() |
838 | ||
5a27c734 MY |
839 | def main(): |
840 | try: | |
841 | cpu_count = multiprocessing.cpu_count() | |
842 | except NotImplementedError: | |
843 | cpu_count = 1 | |
844 | ||
845 | parser = optparse.OptionParser() | |
846 | # Add options here | |
847 | parser.add_option('-c', '--color', action='store_true', default=False, | |
848 | help='display the log in color') | |
91040e85 JH |
849 | parser.add_option('-d', '--defconfigs', type='string', |
850 | help='a file containing a list of defconfigs to move') | |
5a27c734 MY |
851 | parser.add_option('-n', '--dry-run', action='store_true', default=False, |
852 | help='perform a trial run (show log with no changes)') | |
853 | parser.add_option('-e', '--exit-on-error', action='store_true', | |
854 | default=False, | |
855 | help='exit immediately on any error') | |
2144f880 JH |
856 | parser.add_option('-H', '--headers-only', dest='cleanup_headers_only', |
857 | action='store_true', default=False, | |
858 | help='only cleanup the headers') | |
5a27c734 MY |
859 | parser.add_option('-j', '--jobs', type='int', default=cpu_count, |
860 | help='the number of jobs to run simultaneously') | |
95bf9c7e JH |
861 | parser.add_option('-v', '--verbose', action='store_true', default=False, |
862 | help='show any build errors as boards are built') | |
b6ef393a | 863 | parser.usage += ' CONFIG ...' |
5a27c734 | 864 | |
b6ef393a | 865 | (options, configs) = parser.parse_args() |
5a27c734 | 866 | |
b6ef393a | 867 | if len(configs) == 0: |
5a27c734 MY |
868 | parser.print_usage() |
869 | sys.exit(1) | |
870 | ||
b6ef393a MY |
871 | # prefix the option name with CONFIG_ if missing |
872 | configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config | |
873 | for config in configs ] | |
5a27c734 | 874 | |
2144f880 JH |
875 | check_top_directory() |
876 | ||
bd63e5ba MY |
877 | check_clean_directory() |
878 | ||
90ed6cba | 879 | update_cross_compile(options.color) |
4b430c98 | 880 | |
2144f880 | 881 | if not options.cleanup_headers_only: |
b134bc13 | 882 | move_config(configs, options) |
2144f880 | 883 | |
b134bc13 | 884 | cleanup_headers(configs, options.dry_run) |
5a27c734 MY |
885 | |
886 | if __name__ == '__main__': | |
887 | main() |