]>
Commit | Line | Data |
---|---|---|
5a27c734 MY |
1 | #!/usr/bin/env python2 |
2 | # | |
3 | # Author: Masahiro Yamada <yamada.masahiro@socionext.com> | |
4 | # | |
5 | # SPDX-License-Identifier: GPL-2.0+ | |
6 | # | |
7 | ||
8 | """ | |
9 | Move config options from headers to defconfig files. | |
10 | ||
11 | Since Kconfig was introduced to U-Boot, we have worked on moving | |
12 | config options from headers to Kconfig (defconfig). | |
13 | ||
14 | This tool intends to help this tremendous work. | |
15 | ||
16 | ||
17 | Usage | |
18 | ----- | |
19 | ||
b6ef393a | 20 | First, you must edit the Kconfig to add the menu entries for the configs |
96464bad JH |
21 | you are moving. |
22 | ||
b6ef393a MY |
23 | And then run this tool giving CONFIG names you want to move. |
24 | For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE, | |
25 | simply type as follows: | |
5a27c734 | 26 | |
b6ef393a | 27 | $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE |
5a27c734 | 28 | |
b6ef393a | 29 | The tool walks through all the defconfig files and move the given CONFIGs. |
5a27c734 MY |
30 | |
31 | The log is also displayed on the terminal. | |
32 | ||
1d085568 | 33 | The log is printed for each defconfig as follows: |
5a27c734 | 34 | |
1d085568 MY |
35 | <defconfig_name> |
36 | <action1> | |
37 | <action2> | |
38 | <action3> | |
39 | ... | |
5a27c734 | 40 | |
1d085568 MY |
41 | <defconfig_name> is the name of the defconfig. |
42 | ||
43 | <action*> shows what the tool did for that defconfig. | |
c21fc7e2 | 44 | It looks like one of the following: |
5a27c734 MY |
45 | |
46 | - Move 'CONFIG_... ' | |
47 | This config option was moved to the defconfig | |
48 | ||
cc008299 | 49 | - CONFIG_... is not defined in Kconfig. Do nothing. |
916224c3 MY |
50 | The entry for this CONFIG was not found in Kconfig. The option is not |
51 | defined in the config header, either. So, this case can be just skipped. | |
52 | ||
53 | - CONFIG_... is not defined in Kconfig (suspicious). Do nothing. | |
54 | This option is defined in the config header, but its entry was not found | |
55 | in Kconfig. | |
cc008299 MY |
56 | There are two common cases: |
57 | - You forgot to create an entry for the CONFIG before running | |
58 | this tool, or made a typo in a CONFIG passed to this tool. | |
59 | - The entry was hidden due to unmet 'depends on'. | |
916224c3 MY |
60 | The tool does not know if the result is reasonable, so please check it |
61 | manually. | |
cc008299 MY |
62 | |
63 | - 'CONFIG_...' is the same as the define in Kconfig. Do nothing. | |
64 | The define in the config header matched the one in Kconfig. | |
65 | We do not need to touch it. | |
5a27c734 | 66 | |
90ed6cba MY |
67 | - Compiler is missing. Do nothing. |
68 | The compiler specified for this architecture was not found | |
69 | in your PATH environment. | |
70 | (If -e option is passed, the tool exits immediately.) | |
71 | ||
72 | - Failed to process. | |
5a27c734 MY |
73 | An error occurred during processing this defconfig. Skipped. |
74 | (If -e option is passed, the tool exits immediately on error.) | |
75 | ||
76 | Finally, you will be asked, Clean up headers? [y/n]: | |
77 | ||
78 | If you say 'y' here, the unnecessary config defines are removed | |
79 | from the config headers (include/configs/*.h). | |
80 | It just uses the regex method, so you should not rely on it. | |
81 | Just in case, please do 'git diff' to see what happened. | |
82 | ||
83 | ||
b6ef393a MY |
84 | How does it work? |
85 | ----------------- | |
5a27c734 MY |
86 | |
87 | This tool runs configuration and builds include/autoconf.mk for every | |
88 | defconfig. The config options defined in Kconfig appear in the .config | |
89 | file (unless they are hidden because of unmet dependency.) | |
90 | On the other hand, the config options defined by board headers are seen | |
91 | in include/autoconf.mk. The tool looks for the specified options in both | |
b6ef393a MY |
92 | of them to decide the appropriate action for the options. If the given |
93 | config option is found in the .config, but its value does not match the | |
94 | one from the board header, the config option in the .config is replaced | |
95 | with the define in the board header. Then, the .config is synced by | |
96 | "make savedefconfig" and the defconfig is updated with it. | |
5a27c734 MY |
97 | |
98 | For faster processing, this tool handles multi-threading. It creates | |
99 | separate build directories where the out-of-tree build is run. The | |
100 | temporary build directories are automatically created and deleted as | |
101 | needed. The number of threads are chosen based on the number of the CPU | |
102 | cores of your system although you can change it via -j (--jobs) option. | |
103 | ||
104 | ||
105 | Toolchains | |
106 | ---------- | |
107 | ||
108 | Appropriate toolchain are necessary to generate include/autoconf.mk | |
109 | for all the architectures supported by U-Boot. Most of them are available | |
110 | at the kernel.org site, some are not provided by kernel.org. | |
111 | ||
112 | The default per-arch CROSS_COMPILE used by this tool is specified by | |
113 | the list below, CROSS_COMPILE. You may wish to update the list to | |
114 | use your own. Instead of modifying the list directly, you can give | |
115 | them via environments. | |
116 | ||
117 | ||
118 | Available options | |
119 | ----------------- | |
120 | ||
121 | -c, --color | |
122 | Surround each portion of the log with escape sequences to display it | |
123 | in color on the terminal. | |
124 | ||
9ede2123 SG |
125 | -C, --commit |
126 | Create a git commit with the changes when the operation is complete. A | |
127 | standard commit message is used which may need to be edited. | |
128 | ||
91040e85 | 129 | -d, --defconfigs |
0dbc9b59 MY |
130 | Specify a file containing a list of defconfigs to move. The defconfig |
131 | files can be given with shell-style wildcards. | |
91040e85 | 132 | |
5a27c734 | 133 | -n, --dry-run |
b6ef393a | 134 | Perform a trial run that does not make any changes. It is useful to |
5a27c734 MY |
135 | see what is going to happen before one actually runs it. |
136 | ||
137 | -e, --exit-on-error | |
138 | Exit immediately if Make exits with a non-zero status while processing | |
139 | a defconfig file. | |
140 | ||
8513dc04 MY |
141 | -s, --force-sync |
142 | Do "make savedefconfig" forcibly for all the defconfig files. | |
143 | If not specified, "make savedefconfig" only occurs for cases | |
144 | where at least one CONFIG was moved. | |
145 | ||
07913d1e MY |
146 | -S, --spl |
147 | Look for moved config options in spl/include/autoconf.mk instead of | |
148 | include/autoconf.mk. This is useful for moving options for SPL build | |
149 | because SPL related options (mostly prefixed with CONFIG_SPL_) are | |
150 | sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals. | |
151 | ||
2144f880 JH |
152 | -H, --headers-only |
153 | Only cleanup the headers; skip the defconfig processing | |
154 | ||
5a27c734 MY |
155 | -j, --jobs |
156 | Specify the number of threads to run simultaneously. If not specified, | |
157 | the number of threads is the same as the number of CPU cores. | |
158 | ||
6b96c1a1 JH |
159 | -r, --git-ref |
160 | Specify the git ref to clone for building the autoconf.mk. If unspecified | |
161 | use the CWD. This is useful for when changes to the Kconfig affect the | |
162 | default values and you want to capture the state of the defconfig from | |
163 | before that change was in effect. If in doubt, specify a ref pre-Kconfig | |
164 | changes (use HEAD if Kconfig changes are not committed). Worst case it will | |
165 | take a bit longer to run, but will always do the right thing. | |
166 | ||
95bf9c7e JH |
167 | -v, --verbose |
168 | Show any build errors as boards are built | |
169 | ||
6b403dfd SG |
170 | -y, --yes |
171 | Instead of prompting, automatically go ahead with all operations. This | |
172 | includes cleaning up headers and CONFIG_SYS_EXTRA_OPTIONS. | |
173 | ||
5a27c734 MY |
174 | To see the complete list of supported options, run |
175 | ||
176 | $ tools/moveconfig.py -h | |
177 | ||
178 | """ | |
179 | ||
8ba1f5de | 180 | import copy |
f2f6981a | 181 | import difflib |
c8e1b10d | 182 | import filecmp |
5a27c734 | 183 | import fnmatch |
0dbc9b59 | 184 | import glob |
5a27c734 MY |
185 | import multiprocessing |
186 | import optparse | |
187 | import os | |
188 | import re | |
189 | import shutil | |
190 | import subprocess | |
191 | import sys | |
192 | import tempfile | |
193 | import time | |
194 | ||
195 | SHOW_GNU_MAKE = 'scripts/show-gnu-make' | |
196 | SLEEP_TIME=0.03 | |
197 | ||
198 | # Here is the list of cross-tools I use. | |
199 | # Most of them are available at kernel.org | |
c21fc7e2 | 200 | # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the following: |
5a27c734 | 201 | # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases |
4440ecec | 202 | # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz |
5a27c734 MY |
203 | # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545 |
204 | # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu | |
205 | CROSS_COMPILE = { | |
206 | 'arc': 'arc-linux-', | |
207 | 'aarch64': 'aarch64-linux-', | |
208 | 'arm': 'arm-unknown-linux-gnueabi-', | |
5a27c734 MY |
209 | 'm68k': 'm68k-linux-', |
210 | 'microblaze': 'microblaze-linux-', | |
211 | 'mips': 'mips-linux-', | |
212 | 'nds32': 'nds32le-linux-', | |
213 | 'nios2': 'nios2-linux-gnu-', | |
5a27c734 MY |
214 | 'powerpc': 'powerpc-linux-', |
215 | 'sh': 'sh-linux-gnu-', | |
88e1346e MY |
216 | 'x86': 'i386-linux-', |
217 | 'xtensa': 'xtensa-linux-' | |
5a27c734 MY |
218 | } |
219 | ||
220 | STATE_IDLE = 0 | |
221 | STATE_DEFCONFIG = 1 | |
222 | STATE_AUTOCONF = 2 | |
96464bad | 223 | STATE_SAVEDEFCONFIG = 3 |
5a27c734 MY |
224 | |
225 | ACTION_MOVE = 0 | |
cc008299 | 226 | ACTION_NO_ENTRY = 1 |
916224c3 MY |
227 | ACTION_NO_ENTRY_WARN = 2 |
228 | ACTION_NO_CHANGE = 3 | |
5a27c734 MY |
229 | |
230 | COLOR_BLACK = '0;30' | |
231 | COLOR_RED = '0;31' | |
232 | COLOR_GREEN = '0;32' | |
233 | COLOR_BROWN = '0;33' | |
234 | COLOR_BLUE = '0;34' | |
235 | COLOR_PURPLE = '0;35' | |
236 | COLOR_CYAN = '0;36' | |
237 | COLOR_LIGHT_GRAY = '0;37' | |
238 | COLOR_DARK_GRAY = '1;30' | |
239 | COLOR_LIGHT_RED = '1;31' | |
240 | COLOR_LIGHT_GREEN = '1;32' | |
241 | COLOR_YELLOW = '1;33' | |
242 | COLOR_LIGHT_BLUE = '1;34' | |
243 | COLOR_LIGHT_PURPLE = '1;35' | |
244 | COLOR_LIGHT_CYAN = '1;36' | |
245 | COLOR_WHITE = '1;37' | |
246 | ||
247 | ### helper functions ### | |
248 | def get_devnull(): | |
249 | """Get the file object of '/dev/null' device.""" | |
250 | try: | |
251 | devnull = subprocess.DEVNULL # py3k | |
252 | except AttributeError: | |
253 | devnull = open(os.devnull, 'wb') | |
254 | return devnull | |
255 | ||
256 | def check_top_directory(): | |
257 | """Exit if we are not at the top of source directory.""" | |
258 | for f in ('README', 'Licenses'): | |
259 | if not os.path.exists(f): | |
260 | sys.exit('Please run at the top of source directory.') | |
261 | ||
bd63e5ba MY |
262 | def check_clean_directory(): |
263 | """Exit if the source tree is not clean.""" | |
264 | for f in ('.config', 'include/config'): | |
265 | if os.path.exists(f): | |
266 | sys.exit("source tree is not clean, please run 'make mrproper'") | |
267 | ||
5a27c734 MY |
268 | def get_make_cmd(): |
269 | """Get the command name of GNU Make. | |
270 | ||
271 | U-Boot needs GNU Make for building, but the command name is not | |
272 | necessarily "make". (for example, "gmake" on FreeBSD). | |
273 | Returns the most appropriate command name on your system. | |
274 | """ | |
275 | process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) | |
276 | ret = process.communicate() | |
277 | if process.returncode: | |
278 | sys.exit('GNU Make not found') | |
279 | return ret[0].rstrip() | |
280 | ||
25f978cb SG |
281 | def get_matched_defconfig(line): |
282 | """Get the defconfig files that match a pattern | |
283 | ||
284 | Args: | |
285 | line: Path or filename to match, e.g. 'configs/snow_defconfig' or | |
286 | 'k2*_defconfig'. If no directory is provided, 'configs/' is | |
287 | prepended | |
288 | ||
289 | Returns: | |
290 | a list of matching defconfig files | |
291 | """ | |
292 | dirname = os.path.dirname(line) | |
293 | if dirname: | |
294 | pattern = line | |
295 | else: | |
296 | pattern = os.path.join('configs', line) | |
297 | return glob.glob(pattern) + glob.glob(pattern + '_defconfig') | |
298 | ||
0dbc9b59 MY |
299 | def get_matched_defconfigs(defconfigs_file): |
300 | """Get all the defconfig files that match the patterns in a file.""" | |
301 | defconfigs = [] | |
302 | for i, line in enumerate(open(defconfigs_file)): | |
303 | line = line.strip() | |
304 | if not line: | |
305 | continue # skip blank lines silently | |
25f978cb | 306 | matched = get_matched_defconfig(line) |
0dbc9b59 MY |
307 | if not matched: |
308 | print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \ | |
309 | (defconfigs_file, i + 1, line) | |
310 | ||
311 | defconfigs += matched | |
312 | ||
313 | # use set() to drop multiple matching | |
314 | return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ] | |
315 | ||
684c306e MY |
316 | def get_all_defconfigs(): |
317 | """Get all the defconfig files under the configs/ directory.""" | |
318 | defconfigs = [] | |
319 | for (dirpath, dirnames, filenames) in os.walk('configs'): | |
320 | dirpath = dirpath[len('configs') + 1:] | |
321 | for filename in fnmatch.filter(filenames, '*_defconfig'): | |
322 | defconfigs.append(os.path.join(dirpath, filename)) | |
323 | ||
324 | return defconfigs | |
325 | ||
5a27c734 MY |
326 | def color_text(color_enabled, color, string): |
327 | """Return colored string.""" | |
328 | if color_enabled: | |
1d085568 MY |
329 | # LF should not be surrounded by the escape sequence. |
330 | # Otherwise, additional whitespace or line-feed might be printed. | |
331 | return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else '' | |
332 | for s in string.split('\n') ]) | |
5a27c734 MY |
333 | else: |
334 | return string | |
335 | ||
e9ea1221 | 336 | def show_diff(a, b, file_path, color_enabled): |
f2f6981a MY |
337 | """Show unidified diff. |
338 | ||
339 | Arguments: | |
340 | a: A list of lines (before) | |
341 | b: A list of lines (after) | |
342 | file_path: Path to the file | |
e9ea1221 | 343 | color_enabled: Display the diff in color |
f2f6981a MY |
344 | """ |
345 | ||
346 | diff = difflib.unified_diff(a, b, | |
347 | fromfile=os.path.join('a', file_path), | |
348 | tofile=os.path.join('b', file_path)) | |
349 | ||
350 | for line in diff: | |
e9ea1221 MY |
351 | if line[0] == '-' and line[1] != '-': |
352 | print color_text(color_enabled, COLOR_RED, line), | |
353 | elif line[0] == '+' and line[1] != '+': | |
354 | print color_text(color_enabled, COLOR_GREEN, line), | |
355 | else: | |
356 | print line, | |
f2f6981a | 357 | |
90ed6cba | 358 | def update_cross_compile(color_enabled): |
1cc0a9f4 | 359 | """Update per-arch CROSS_COMPILE via environment variables |
5a27c734 MY |
360 | |
361 | The default CROSS_COMPILE values are available | |
362 | in the CROSS_COMPILE list above. | |
363 | ||
1cc0a9f4 | 364 | You can override them via environment variables |
5a27c734 MY |
365 | CROSS_COMPILE_{ARCH}. |
366 | ||
367 | For example, if you want to override toolchain prefixes | |
368 | for ARM and PowerPC, you can do as follows in your shell: | |
369 | ||
370 | export CROSS_COMPILE_ARM=... | |
371 | export CROSS_COMPILE_POWERPC=... | |
90ed6cba MY |
372 | |
373 | Then, this function checks if specified compilers really exist in your | |
374 | PATH environment. | |
5a27c734 MY |
375 | """ |
376 | archs = [] | |
377 | ||
378 | for arch in os.listdir('arch'): | |
379 | if os.path.exists(os.path.join('arch', arch, 'Makefile')): | |
380 | archs.append(arch) | |
381 | ||
382 | # arm64 is a special case | |
383 | archs.append('aarch64') | |
384 | ||
385 | for arch in archs: | |
386 | env = 'CROSS_COMPILE_' + arch.upper() | |
387 | cross_compile = os.environ.get(env) | |
90ed6cba MY |
388 | if not cross_compile: |
389 | cross_compile = CROSS_COMPILE.get(arch, '') | |
390 | ||
391 | for path in os.environ["PATH"].split(os.pathsep): | |
392 | gcc_path = os.path.join(path, cross_compile + 'gcc') | |
393 | if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK): | |
394 | break | |
395 | else: | |
396 | print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW, | |
397 | 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped' | |
398 | % (cross_compile, arch)) | |
399 | cross_compile = None | |
400 | ||
401 | CROSS_COMPILE[arch] = cross_compile | |
5a27c734 | 402 | |
8ba1f5de MY |
403 | def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre, |
404 | extend_post): | |
405 | """Extend matched lines if desired patterns are found before/after already | |
406 | matched lines. | |
407 | ||
408 | Arguments: | |
409 | lines: A list of lines handled. | |
410 | matched: A list of line numbers that have been already matched. | |
411 | (will be updated by this function) | |
412 | pre_patterns: A list of regular expression that should be matched as | |
413 | preamble. | |
414 | post_patterns: A list of regular expression that should be matched as | |
415 | postamble. | |
416 | extend_pre: Add the line number of matched preamble to the matched list. | |
417 | extend_post: Add the line number of matched postamble to the matched list. | |
418 | """ | |
419 | extended_matched = [] | |
420 | ||
421 | j = matched[0] | |
422 | ||
423 | for i in matched: | |
424 | if i == 0 or i < j: | |
425 | continue | |
426 | j = i | |
427 | while j in matched: | |
428 | j += 1 | |
429 | if j >= len(lines): | |
430 | break | |
431 | ||
432 | for p in pre_patterns: | |
433 | if p.search(lines[i - 1]): | |
434 | break | |
435 | else: | |
436 | # not matched | |
437 | continue | |
438 | ||
439 | for p in post_patterns: | |
440 | if p.search(lines[j]): | |
441 | break | |
442 | else: | |
443 | # not matched | |
444 | continue | |
445 | ||
446 | if extend_pre: | |
447 | extended_matched.append(i - 1) | |
448 | if extend_post: | |
449 | extended_matched.append(j) | |
450 | ||
451 | matched += extended_matched | |
452 | matched.sort() | |
453 | ||
85edfc1f CP |
454 | def confirm(options, prompt): |
455 | if not options.yes: | |
456 | while True: | |
457 | choice = raw_input('{} [y/n]: '.format(prompt)) | |
458 | choice = choice.lower() | |
459 | print choice | |
460 | if choice == 'y' or choice == 'n': | |
461 | break | |
462 | ||
463 | if choice == 'n': | |
464 | return False | |
465 | ||
466 | return True | |
467 | ||
e9ea1221 | 468 | def cleanup_one_header(header_path, patterns, options): |
5a27c734 MY |
469 | """Clean regex-matched lines away from a file. |
470 | ||
471 | Arguments: | |
472 | header_path: path to the cleaned file. | |
473 | patterns: list of regex patterns. Any lines matching to these | |
474 | patterns are deleted. | |
e9ea1221 | 475 | options: option flags. |
5a27c734 MY |
476 | """ |
477 | with open(header_path) as f: | |
478 | lines = f.readlines() | |
479 | ||
480 | matched = [] | |
481 | for i, line in enumerate(lines): | |
a3a779f7 MY |
482 | if i - 1 in matched and lines[i - 1][-2:] == '\\\n': |
483 | matched.append(i) | |
484 | continue | |
5a27c734 | 485 | for pattern in patterns: |
8ba1f5de | 486 | if pattern.search(line): |
5a27c734 MY |
487 | matched.append(i) |
488 | break | |
489 | ||
8ba1f5de MY |
490 | if not matched: |
491 | return | |
492 | ||
493 | # remove empty #ifdef ... #endif, successive blank lines | |
494 | pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef | |
495 | pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else | |
496 | pattern_endif = re.compile(r'#\s*endif\W') # #endif | |
497 | pattern_blank = re.compile(r'^\s*$') # empty line | |
498 | ||
499 | while True: | |
500 | old_matched = copy.copy(matched) | |
501 | extend_matched_lines(lines, matched, [pattern_if], | |
502 | [pattern_endif], True, True) | |
503 | extend_matched_lines(lines, matched, [pattern_elif], | |
504 | [pattern_elif, pattern_endif], True, False) | |
505 | extend_matched_lines(lines, matched, [pattern_if, pattern_elif], | |
506 | [pattern_blank], False, True) | |
507 | extend_matched_lines(lines, matched, [pattern_blank], | |
508 | [pattern_elif, pattern_endif], True, False) | |
509 | extend_matched_lines(lines, matched, [pattern_blank], | |
510 | [pattern_blank], True, False) | |
511 | if matched == old_matched: | |
512 | break | |
513 | ||
f2f6981a MY |
514 | tolines = copy.copy(lines) |
515 | ||
516 | for i in reversed(matched): | |
517 | tolines.pop(i) | |
518 | ||
e9ea1221 | 519 | show_diff(lines, tolines, header_path, options.color) |
8ba1f5de | 520 | |
e9ea1221 | 521 | if options.dry_run: |
5a27c734 MY |
522 | return |
523 | ||
524 | with open(header_path, 'w') as f: | |
f2f6981a MY |
525 | for line in tolines: |
526 | f.write(line) | |
5a27c734 | 527 | |
e9ea1221 | 528 | def cleanup_headers(configs, options): |
5a27c734 MY |
529 | """Delete config defines from board headers. |
530 | ||
531 | Arguments: | |
b134bc13 | 532 | configs: A list of CONFIGs to remove. |
e9ea1221 | 533 | options: option flags. |
5a27c734 | 534 | """ |
85edfc1f CP |
535 | if not confirm(options, 'Clean up headers?'): |
536 | return | |
5a27c734 MY |
537 | |
538 | patterns = [] | |
b134bc13 | 539 | for config in configs: |
5a27c734 MY |
540 | patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) |
541 | patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) | |
542 | ||
60727f51 JH |
543 | for dir in 'include', 'arch', 'board': |
544 | for (dirpath, dirnames, filenames) in os.walk(dir): | |
dc6de50b MY |
545 | if dirpath == os.path.join('include', 'generated'): |
546 | continue | |
60727f51 JH |
547 | for filename in filenames: |
548 | if not fnmatch.fnmatch(filename, '*~'): | |
549 | cleanup_one_header(os.path.join(dirpath, filename), | |
e9ea1221 | 550 | patterns, options) |
5a27c734 | 551 | |
9ab0296a MY |
552 | def cleanup_one_extra_option(defconfig_path, configs, options): |
553 | """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file. | |
554 | ||
555 | Arguments: | |
556 | defconfig_path: path to the cleaned defconfig file. | |
557 | configs: A list of CONFIGs to remove. | |
558 | options: option flags. | |
559 | """ | |
560 | ||
561 | start = 'CONFIG_SYS_EXTRA_OPTIONS="' | |
562 | end = '"\n' | |
563 | ||
564 | with open(defconfig_path) as f: | |
565 | lines = f.readlines() | |
566 | ||
567 | for i, line in enumerate(lines): | |
568 | if line.startswith(start) and line.endswith(end): | |
569 | break | |
570 | else: | |
571 | # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig | |
572 | return | |
573 | ||
574 | old_tokens = line[len(start):-len(end)].split(',') | |
575 | new_tokens = [] | |
576 | ||
577 | for token in old_tokens: | |
578 | pos = token.find('=') | |
579 | if not (token[:pos] if pos >= 0 else token) in configs: | |
580 | new_tokens.append(token) | |
581 | ||
582 | if new_tokens == old_tokens: | |
583 | return | |
584 | ||
585 | tolines = copy.copy(lines) | |
586 | ||
587 | if new_tokens: | |
588 | tolines[i] = start + ','.join(new_tokens) + end | |
589 | else: | |
590 | tolines.pop(i) | |
591 | ||
592 | show_diff(lines, tolines, defconfig_path, options.color) | |
593 | ||
594 | if options.dry_run: | |
595 | return | |
596 | ||
597 | with open(defconfig_path, 'w') as f: | |
598 | for line in tolines: | |
599 | f.write(line) | |
600 | ||
601 | def cleanup_extra_options(configs, options): | |
602 | """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files. | |
603 | ||
604 | Arguments: | |
605 | configs: A list of CONFIGs to remove. | |
606 | options: option flags. | |
607 | """ | |
85edfc1f CP |
608 | if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'): |
609 | return | |
9ab0296a MY |
610 | |
611 | configs = [ config[len('CONFIG_'):] for config in configs ] | |
612 | ||
613 | defconfigs = get_all_defconfigs() | |
614 | ||
615 | for defconfig in defconfigs: | |
616 | cleanup_one_extra_option(os.path.join('configs', defconfig), configs, | |
617 | options) | |
618 | ||
ca43834d CP |
619 | def cleanup_whitelist(configs, options): |
620 | """Delete config whitelist entries | |
621 | ||
622 | Arguments: | |
623 | configs: A list of CONFIGs to remove. | |
624 | options: option flags. | |
625 | """ | |
626 | if not confirm(options, 'Clean up whitelist entries?'): | |
627 | return | |
628 | ||
629 | with open(os.path.join('scripts', 'config_whitelist.txt')) as f: | |
630 | lines = f.readlines() | |
631 | ||
632 | lines = [x for x in lines if x.strip() not in configs] | |
633 | ||
634 | with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f: | |
635 | f.write(''.join(lines)) | |
636 | ||
f90df596 CP |
637 | def find_matching(patterns, line): |
638 | for pat in patterns: | |
639 | if pat.search(line): | |
640 | return True | |
641 | return False | |
642 | ||
643 | def cleanup_readme(configs, options): | |
644 | """Delete config description in README | |
645 | ||
646 | Arguments: | |
647 | configs: A list of CONFIGs to remove. | |
648 | options: option flags. | |
649 | """ | |
650 | if not confirm(options, 'Clean up README?'): | |
651 | return | |
652 | ||
653 | patterns = [] | |
654 | for config in configs: | |
655 | patterns.append(re.compile(r'^\s+%s' % config)) | |
656 | ||
657 | with open('README') as f: | |
658 | lines = f.readlines() | |
659 | ||
660 | found = False | |
661 | newlines = [] | |
662 | for line in lines: | |
663 | if not found: | |
664 | found = find_matching(patterns, line) | |
665 | if found: | |
666 | continue | |
667 | ||
668 | if found and re.search(r'^\s+CONFIG', line): | |
669 | found = False | |
670 | ||
671 | if not found: | |
672 | newlines.append(line) | |
673 | ||
674 | with open('README', 'w') as f: | |
675 | f.write(''.join(newlines)) | |
676 | ||
ca43834d | 677 | |
5a27c734 | 678 | ### classes ### |
c5e60fd4 MY |
679 | class Progress: |
680 | ||
681 | """Progress Indicator""" | |
682 | ||
683 | def __init__(self, total): | |
684 | """Create a new progress indicator. | |
685 | ||
686 | Arguments: | |
687 | total: A number of defconfig files to process. | |
688 | """ | |
689 | self.current = 0 | |
690 | self.total = total | |
691 | ||
692 | def inc(self): | |
693 | """Increment the number of processed defconfig files.""" | |
694 | ||
695 | self.current += 1 | |
696 | ||
697 | def show(self): | |
698 | """Display the progress.""" | |
699 | print ' %d defconfigs out of %d\r' % (self.current, self.total), | |
700 | sys.stdout.flush() | |
701 | ||
5a27c734 MY |
702 | class KconfigParser: |
703 | ||
704 | """A parser of .config and include/autoconf.mk.""" | |
705 | ||
706 | re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') | |
707 | re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') | |
708 | ||
522e8dcb | 709 | def __init__(self, configs, options, build_dir): |
5a27c734 MY |
710 | """Create a new parser. |
711 | ||
712 | Arguments: | |
b134bc13 | 713 | configs: A list of CONFIGs to move. |
5a27c734 MY |
714 | options: option flags. |
715 | build_dir: Build directory. | |
716 | """ | |
b134bc13 | 717 | self.configs = configs |
5a27c734 | 718 | self.options = options |
1f16992e MY |
719 | self.dotconfig = os.path.join(build_dir, '.config') |
720 | self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk') | |
07913d1e MY |
721 | self.spl_autoconf = os.path.join(build_dir, 'spl', 'include', |
722 | 'autoconf.mk') | |
1f16992e MY |
723 | self.config_autoconf = os.path.join(build_dir, 'include', 'config', |
724 | 'auto.conf') | |
5da4f857 | 725 | self.defconfig = os.path.join(build_dir, 'defconfig') |
5a27c734 MY |
726 | |
727 | def get_cross_compile(self): | |
728 | """Parse .config file and return CROSS_COMPILE. | |
729 | ||
730 | Returns: | |
731 | A string storing the compiler prefix for the architecture. | |
90ed6cba MY |
732 | Return a NULL string for architectures that do not require |
733 | compiler prefix (Sandbox and native build is the case). | |
734 | Return None if the specified compiler is missing in your PATH. | |
735 | Caller should distinguish '' and None. | |
5a27c734 MY |
736 | """ |
737 | arch = '' | |
738 | cpu = '' | |
1f16992e | 739 | for line in open(self.dotconfig): |
5a27c734 MY |
740 | m = self.re_arch.match(line) |
741 | if m: | |
742 | arch = m.group(1) | |
743 | continue | |
744 | m = self.re_cpu.match(line) | |
745 | if m: | |
746 | cpu = m.group(1) | |
747 | ||
90ed6cba MY |
748 | if not arch: |
749 | return None | |
5a27c734 MY |
750 | |
751 | # fix-up for aarch64 | |
752 | if arch == 'arm' and cpu == 'armv8': | |
753 | arch = 'aarch64' | |
754 | ||
90ed6cba | 755 | return CROSS_COMPILE.get(arch, None) |
5a27c734 | 756 | |
b134bc13 | 757 | def parse_one_config(self, config, dotconfig_lines, autoconf_lines): |
5a27c734 MY |
758 | """Parse .config, defconfig, include/autoconf.mk for one config. |
759 | ||
760 | This function looks for the config options in the lines from | |
761 | defconfig, .config, and include/autoconf.mk in order to decide | |
762 | which action should be taken for this defconfig. | |
763 | ||
764 | Arguments: | |
b134bc13 | 765 | config: CONFIG name to parse. |
cc008299 | 766 | dotconfig_lines: lines from the .config file. |
5a27c734 MY |
767 | autoconf_lines: lines from the include/autoconf.mk file. |
768 | ||
769 | Returns: | |
770 | A tupple of the action for this defconfig and the line | |
771 | matched for the config. | |
772 | """ | |
5a27c734 MY |
773 | not_set = '# %s is not set' % config |
774 | ||
5a27c734 MY |
775 | for line in autoconf_lines: |
776 | line = line.rstrip() | |
777 | if line.startswith(config + '='): | |
cc008299 | 778 | new_val = line |
5a27c734 | 779 | break |
5a27c734 | 780 | else: |
cc008299 MY |
781 | new_val = not_set |
782 | ||
916224c3 MY |
783 | for line in dotconfig_lines: |
784 | line = line.rstrip() | |
785 | if line.startswith(config + '=') or line == not_set: | |
786 | old_val = line | |
787 | break | |
788 | else: | |
789 | if new_val == not_set: | |
790 | return (ACTION_NO_ENTRY, config) | |
791 | else: | |
792 | return (ACTION_NO_ENTRY_WARN, config) | |
793 | ||
cc008299 MY |
794 | # If this CONFIG is neither bool nor trisate |
795 | if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set: | |
796 | # tools/scripts/define2mk.sed changes '1' to 'y'. | |
797 | # This is a problem if the CONFIG is int type. | |
798 | # Check the type in Kconfig and handle it correctly. | |
799 | if new_val[-2:] == '=y': | |
800 | new_val = new_val[:-1] + '1' | |
5a27c734 | 801 | |
5030159e MY |
802 | return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE, |
803 | new_val) | |
5a27c734 | 804 | |
1d085568 | 805 | def update_dotconfig(self): |
6ff36d21 | 806 | """Parse files for the config options and update the .config. |
5a27c734 | 807 | |
cc008299 MY |
808 | This function parses the generated .config and include/autoconf.mk |
809 | searching the target options. | |
6ff36d21 | 810 | Move the config option(s) to the .config as needed. |
5a27c734 MY |
811 | |
812 | Arguments: | |
813 | defconfig: defconfig name. | |
522e8dcb MY |
814 | |
815 | Returns: | |
7fb0bacd MY |
816 | Return a tuple of (updated flag, log string). |
817 | The "updated flag" is True if the .config was updated, False | |
818 | otherwise. The "log string" shows what happend to the .config. | |
5a27c734 MY |
819 | """ |
820 | ||
5a27c734 | 821 | results = [] |
7fb0bacd | 822 | updated = False |
916224c3 | 823 | suspicious = False |
07913d1e MY |
824 | rm_files = [self.config_autoconf, self.autoconf] |
825 | ||
826 | if self.options.spl: | |
827 | if os.path.exists(self.spl_autoconf): | |
828 | autoconf_path = self.spl_autoconf | |
829 | rm_files.append(self.spl_autoconf) | |
830 | else: | |
831 | for f in rm_files: | |
832 | os.remove(f) | |
833 | return (updated, suspicious, | |
834 | color_text(self.options.color, COLOR_BROWN, | |
835 | "SPL is not enabled. Skipped.") + '\n') | |
836 | else: | |
837 | autoconf_path = self.autoconf | |
5a27c734 | 838 | |
1f16992e | 839 | with open(self.dotconfig) as f: |
cc008299 | 840 | dotconfig_lines = f.readlines() |
5a27c734 | 841 | |
07913d1e | 842 | with open(autoconf_path) as f: |
5a27c734 MY |
843 | autoconf_lines = f.readlines() |
844 | ||
b134bc13 MY |
845 | for config in self.configs: |
846 | result = self.parse_one_config(config, dotconfig_lines, | |
96464bad | 847 | autoconf_lines) |
5a27c734 MY |
848 | results.append(result) |
849 | ||
850 | log = '' | |
851 | ||
852 | for (action, value) in results: | |
853 | if action == ACTION_MOVE: | |
854 | actlog = "Move '%s'" % value | |
855 | log_color = COLOR_LIGHT_GREEN | |
cc008299 MY |
856 | elif action == ACTION_NO_ENTRY: |
857 | actlog = "%s is not defined in Kconfig. Do nothing." % value | |
5a27c734 | 858 | log_color = COLOR_LIGHT_BLUE |
916224c3 MY |
859 | elif action == ACTION_NO_ENTRY_WARN: |
860 | actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value | |
861 | log_color = COLOR_YELLOW | |
862 | suspicious = True | |
cc008299 MY |
863 | elif action == ACTION_NO_CHANGE: |
864 | actlog = "'%s' is the same as the define in Kconfig. Do nothing." \ | |
865 | % value | |
5a27c734 | 866 | log_color = COLOR_LIGHT_PURPLE |
07913d1e MY |
867 | elif action == ACTION_SPL_NOT_EXIST: |
868 | actlog = "SPL is not enabled for this defconfig. Skip." | |
869 | log_color = COLOR_PURPLE | |
5a27c734 MY |
870 | else: |
871 | sys.exit("Internal Error. This should not happen.") | |
872 | ||
1d085568 | 873 | log += color_text(self.options.color, log_color, actlog) + '\n' |
5a27c734 | 874 | |
1f16992e | 875 | with open(self.dotconfig, 'a') as f: |
e423d17f MY |
876 | for (action, value) in results: |
877 | if action == ACTION_MOVE: | |
878 | f.write(value + '\n') | |
7fb0bacd | 879 | updated = True |
5a27c734 | 880 | |
5da4f857 | 881 | self.results = results |
07913d1e MY |
882 | for f in rm_files: |
883 | os.remove(f) | |
5a27c734 | 884 | |
916224c3 | 885 | return (updated, suspicious, log) |
522e8dcb | 886 | |
5da4f857 MY |
887 | def check_defconfig(self): |
888 | """Check the defconfig after savedefconfig | |
889 | ||
890 | Returns: | |
891 | Return additional log if moved CONFIGs were removed again by | |
892 | 'make savedefconfig'. | |
893 | """ | |
894 | ||
895 | log = '' | |
896 | ||
897 | with open(self.defconfig) as f: | |
898 | defconfig_lines = f.readlines() | |
899 | ||
900 | for (action, value) in self.results: | |
901 | if action != ACTION_MOVE: | |
902 | continue | |
903 | if not value + '\n' in defconfig_lines: | |
904 | log += color_text(self.options.color, COLOR_YELLOW, | |
905 | "'%s' was removed by savedefconfig.\n" % | |
906 | value) | |
907 | ||
908 | return log | |
909 | ||
5a27c734 MY |
910 | class Slot: |
911 | ||
912 | """A slot to store a subprocess. | |
913 | ||
914 | Each instance of this class handles one subprocess. | |
915 | This class is useful to control multiple threads | |
916 | for faster processing. | |
917 | """ | |
918 | ||
6b96c1a1 | 919 | def __init__(self, configs, options, progress, devnull, make_cmd, reference_src_dir): |
5a27c734 MY |
920 | """Create a new process slot. |
921 | ||
922 | Arguments: | |
b134bc13 | 923 | configs: A list of CONFIGs to move. |
5a27c734 | 924 | options: option flags. |
c5e60fd4 | 925 | progress: A progress indicator. |
5a27c734 MY |
926 | devnull: A file object of '/dev/null'. |
927 | make_cmd: command name of GNU Make. | |
6b96c1a1 JH |
928 | reference_src_dir: Determine the true starting config state from this |
929 | source tree. | |
5a27c734 MY |
930 | """ |
931 | self.options = options | |
c5e60fd4 | 932 | self.progress = progress |
5a27c734 MY |
933 | self.build_dir = tempfile.mkdtemp() |
934 | self.devnull = devnull | |
935 | self.make_cmd = (make_cmd, 'O=' + self.build_dir) | |
6b96c1a1 | 936 | self.reference_src_dir = reference_src_dir |
522e8dcb | 937 | self.parser = KconfigParser(configs, options, self.build_dir) |
5a27c734 | 938 | self.state = STATE_IDLE |
09c6c066 MY |
939 | self.failed_boards = set() |
940 | self.suspicious_boards = set() | |
5a27c734 MY |
941 | |
942 | def __del__(self): | |
943 | """Delete the working directory | |
944 | ||
945 | This function makes sure the temporary directory is cleaned away | |
946 | even if Python suddenly dies due to error. It should be done in here | |
f2dae751 | 947 | because it is guaranteed the destructor is always invoked when the |
5a27c734 MY |
948 | instance of the class gets unreferenced. |
949 | ||
950 | If the subprocess is still running, wait until it finishes. | |
951 | """ | |
952 | if self.state != STATE_IDLE: | |
953 | while self.ps.poll() == None: | |
954 | pass | |
955 | shutil.rmtree(self.build_dir) | |
956 | ||
c5e60fd4 | 957 | def add(self, defconfig): |
5a27c734 MY |
958 | """Assign a new subprocess for defconfig and add it to the slot. |
959 | ||
960 | If the slot is vacant, create a new subprocess for processing the | |
961 | given defconfig and add it to the slot. Just returns False if | |
962 | the slot is occupied (i.e. the current subprocess is still running). | |
963 | ||
964 | Arguments: | |
965 | defconfig: defconfig name. | |
966 | ||
967 | Returns: | |
968 | Return True on success or False on failure | |
969 | """ | |
970 | if self.state != STATE_IDLE: | |
971 | return False | |
e307fa9d | 972 | |
5a27c734 | 973 | self.defconfig = defconfig |
1d085568 | 974 | self.log = '' |
f432c33f | 975 | self.current_src_dir = self.reference_src_dir |
e307fa9d | 976 | self.do_defconfig() |
5a27c734 MY |
977 | return True |
978 | ||
979 | def poll(self): | |
980 | """Check the status of the subprocess and handle it as needed. | |
981 | ||
982 | Returns True if the slot is vacant (i.e. in idle state). | |
983 | If the configuration is successfully finished, assign a new | |
984 | subprocess to build include/autoconf.mk. | |
985 | If include/autoconf.mk is generated, invoke the parser to | |
7fb0bacd MY |
986 | parse the .config and the include/autoconf.mk, moving |
987 | config options to the .config as needed. | |
988 | If the .config was updated, run "make savedefconfig" to sync | |
989 | it, update the original defconfig, and then set the slot back | |
990 | to the idle state. | |
5a27c734 MY |
991 | |
992 | Returns: | |
993 | Return True if the subprocess is terminated, False otherwise | |
994 | """ | |
995 | if self.state == STATE_IDLE: | |
996 | return True | |
997 | ||
998 | if self.ps.poll() == None: | |
999 | return False | |
1000 | ||
1001 | if self.ps.poll() != 0: | |
e307fa9d MY |
1002 | self.handle_error() |
1003 | elif self.state == STATE_DEFCONFIG: | |
f432c33f | 1004 | if self.reference_src_dir and not self.current_src_dir: |
6b96c1a1 JH |
1005 | self.do_savedefconfig() |
1006 | else: | |
1007 | self.do_autoconf() | |
e307fa9d | 1008 | elif self.state == STATE_AUTOCONF: |
f432c33f MY |
1009 | if self.current_src_dir: |
1010 | self.current_src_dir = None | |
6b96c1a1 JH |
1011 | self.do_defconfig() |
1012 | else: | |
1013 | self.do_savedefconfig() | |
e307fa9d MY |
1014 | elif self.state == STATE_SAVEDEFCONFIG: |
1015 | self.update_defconfig() | |
1016 | else: | |
1017 | sys.exit("Internal Error. This should not happen.") | |
5a27c734 | 1018 | |
e307fa9d | 1019 | return True if self.state == STATE_IDLE else False |
96464bad | 1020 | |
e307fa9d MY |
1021 | def handle_error(self): |
1022 | """Handle error cases.""" | |
8513dc04 | 1023 | |
e307fa9d MY |
1024 | self.log += color_text(self.options.color, COLOR_LIGHT_RED, |
1025 | "Failed to process.\n") | |
1026 | if self.options.verbose: | |
1027 | self.log += color_text(self.options.color, COLOR_LIGHT_CYAN, | |
1028 | self.ps.stderr.read()) | |
1029 | self.finish(False) | |
96464bad | 1030 | |
e307fa9d MY |
1031 | def do_defconfig(self): |
1032 | """Run 'make <board>_defconfig' to create the .config file.""" | |
c8e1b10d | 1033 | |
e307fa9d MY |
1034 | cmd = list(self.make_cmd) |
1035 | cmd.append(self.defconfig) | |
1036 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, | |
f432c33f MY |
1037 | stderr=subprocess.PIPE, |
1038 | cwd=self.current_src_dir) | |
e307fa9d | 1039 | self.state = STATE_DEFCONFIG |
c8e1b10d | 1040 | |
e307fa9d MY |
1041 | def do_autoconf(self): |
1042 | """Run 'make include/config/auto.conf'.""" | |
5a27c734 | 1043 | |
25400090 | 1044 | self.cross_compile = self.parser.get_cross_compile() |
90ed6cba | 1045 | if self.cross_compile is None: |
1d085568 MY |
1046 | self.log += color_text(self.options.color, COLOR_YELLOW, |
1047 | "Compiler is missing. Do nothing.\n") | |
4efef998 | 1048 | self.finish(False) |
e307fa9d | 1049 | return |
90ed6cba | 1050 | |
5a27c734 | 1051 | cmd = list(self.make_cmd) |
25400090 JH |
1052 | if self.cross_compile: |
1053 | cmd.append('CROSS_COMPILE=%s' % self.cross_compile) | |
7740f653 | 1054 | cmd.append('KCONFIG_IGNORE_DUPLICATES=1') |
5a27c734 | 1055 | cmd.append('include/config/auto.conf') |
25400090 | 1056 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, |
f432c33f MY |
1057 | stderr=subprocess.PIPE, |
1058 | cwd=self.current_src_dir) | |
5a27c734 | 1059 | self.state = STATE_AUTOCONF |
e307fa9d MY |
1060 | |
1061 | def do_savedefconfig(self): | |
1062 | """Update the .config and run 'make savedefconfig'.""" | |
1063 | ||
916224c3 MY |
1064 | (updated, suspicious, log) = self.parser.update_dotconfig() |
1065 | if suspicious: | |
1066 | self.suspicious_boards.add(self.defconfig) | |
e307fa9d MY |
1067 | self.log += log |
1068 | ||
1069 | if not self.options.force_sync and not updated: | |
1070 | self.finish(True) | |
1071 | return | |
1072 | if updated: | |
1073 | self.log += color_text(self.options.color, COLOR_LIGHT_GREEN, | |
1074 | "Syncing by savedefconfig...\n") | |
1075 | else: | |
1076 | self.log += "Syncing by savedefconfig (forced by option)...\n" | |
1077 | ||
1078 | cmd = list(self.make_cmd) | |
1079 | cmd.append('savedefconfig') | |
1080 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, | |
1081 | stderr=subprocess.PIPE) | |
1082 | self.state = STATE_SAVEDEFCONFIG | |
1083 | ||
1084 | def update_defconfig(self): | |
1085 | """Update the input defconfig and go back to the idle state.""" | |
1086 | ||
fc2661ee MY |
1087 | log = self.parser.check_defconfig() |
1088 | if log: | |
09c6c066 | 1089 | self.suspicious_boards.add(self.defconfig) |
fc2661ee | 1090 | self.log += log |
e307fa9d MY |
1091 | orig_defconfig = os.path.join('configs', self.defconfig) |
1092 | new_defconfig = os.path.join(self.build_dir, 'defconfig') | |
1093 | updated = not filecmp.cmp(orig_defconfig, new_defconfig) | |
1094 | ||
1095 | if updated: | |
06cc1d36 | 1096 | self.log += color_text(self.options.color, COLOR_LIGHT_BLUE, |
e307fa9d MY |
1097 | "defconfig was updated.\n") |
1098 | ||
1099 | if not self.options.dry_run and updated: | |
1100 | shutil.move(new_defconfig, orig_defconfig) | |
1101 | self.finish(True) | |
5a27c734 | 1102 | |
4efef998 MY |
1103 | def finish(self, success): |
1104 | """Display log along with progress and go to the idle state. | |
1d085568 MY |
1105 | |
1106 | Arguments: | |
4efef998 MY |
1107 | success: Should be True when the defconfig was processed |
1108 | successfully, or False when it fails. | |
1d085568 MY |
1109 | """ |
1110 | # output at least 30 characters to hide the "* defconfigs out of *". | |
1111 | log = self.defconfig.ljust(30) + '\n' | |
1112 | ||
1113 | log += '\n'.join([ ' ' + s for s in self.log.split('\n') ]) | |
1114 | # Some threads are running in parallel. | |
1115 | # Print log atomically to not mix up logs from different threads. | |
4efef998 MY |
1116 | print >> (sys.stdout if success else sys.stderr), log |
1117 | ||
1118 | if not success: | |
1119 | if self.options.exit_on_error: | |
1120 | sys.exit("Exit on error.") | |
1121 | # If --exit-on-error flag is not set, skip this board and continue. | |
1122 | # Record the failed board. | |
09c6c066 | 1123 | self.failed_boards.add(self.defconfig) |
4efef998 | 1124 | |
1d085568 MY |
1125 | self.progress.inc() |
1126 | self.progress.show() | |
4efef998 | 1127 | self.state = STATE_IDLE |
1d085568 | 1128 | |
5a27c734 | 1129 | def get_failed_boards(self): |
09c6c066 | 1130 | """Returns a set of failed boards (defconfigs) in this slot. |
5a27c734 MY |
1131 | """ |
1132 | return self.failed_boards | |
1133 | ||
fc2661ee | 1134 | def get_suspicious_boards(self): |
09c6c066 | 1135 | """Returns a set of boards (defconfigs) with possible misconversion. |
fc2661ee | 1136 | """ |
916224c3 | 1137 | return self.suspicious_boards - self.failed_boards |
fc2661ee | 1138 | |
5a27c734 MY |
1139 | class Slots: |
1140 | ||
1141 | """Controller of the array of subprocess slots.""" | |
1142 | ||
6b96c1a1 | 1143 | def __init__(self, configs, options, progress, reference_src_dir): |
5a27c734 MY |
1144 | """Create a new slots controller. |
1145 | ||
1146 | Arguments: | |
b134bc13 | 1147 | configs: A list of CONFIGs to move. |
5a27c734 | 1148 | options: option flags. |
c5e60fd4 | 1149 | progress: A progress indicator. |
6b96c1a1 JH |
1150 | reference_src_dir: Determine the true starting config state from this |
1151 | source tree. | |
5a27c734 MY |
1152 | """ |
1153 | self.options = options | |
1154 | self.slots = [] | |
1155 | devnull = get_devnull() | |
1156 | make_cmd = get_make_cmd() | |
1157 | for i in range(options.jobs): | |
b134bc13 | 1158 | self.slots.append(Slot(configs, options, progress, devnull, |
6b96c1a1 | 1159 | make_cmd, reference_src_dir)) |
5a27c734 | 1160 | |
c5e60fd4 | 1161 | def add(self, defconfig): |
5a27c734 MY |
1162 | """Add a new subprocess if a vacant slot is found. |
1163 | ||
1164 | Arguments: | |
1165 | defconfig: defconfig name to be put into. | |
1166 | ||
1167 | Returns: | |
1168 | Return True on success or False on failure | |
1169 | """ | |
1170 | for slot in self.slots: | |
c5e60fd4 | 1171 | if slot.add(defconfig): |
5a27c734 MY |
1172 | return True |
1173 | return False | |
1174 | ||
1175 | def available(self): | |
1176 | """Check if there is a vacant slot. | |
1177 | ||
1178 | Returns: | |
1179 | Return True if at lease one vacant slot is found, False otherwise. | |
1180 | """ | |
1181 | for slot in self.slots: | |
1182 | if slot.poll(): | |
1183 | return True | |
1184 | return False | |
1185 | ||
1186 | def empty(self): | |
1187 | """Check if all slots are vacant. | |
1188 | ||
1189 | Returns: | |
1190 | Return True if all the slots are vacant, False otherwise. | |
1191 | """ | |
1192 | ret = True | |
1193 | for slot in self.slots: | |
1194 | if not slot.poll(): | |
1195 | ret = False | |
1196 | return ret | |
1197 | ||
1198 | def show_failed_boards(self): | |
1199 | """Display all of the failed boards (defconfigs).""" | |
09c6c066 | 1200 | boards = set() |
96dccd97 | 1201 | output_file = 'moveconfig.failed' |
5a27c734 MY |
1202 | |
1203 | for slot in self.slots: | |
09c6c066 | 1204 | boards |= slot.get_failed_boards() |
96dccd97 MY |
1205 | |
1206 | if boards: | |
1207 | boards = '\n'.join(boards) + '\n' | |
1208 | msg = "The following boards were not processed due to error:\n" | |
1209 | msg += boards | |
1210 | msg += "(the list has been saved in %s)\n" % output_file | |
1211 | print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, | |
1212 | msg) | |
1213 | ||
1214 | with open(output_file, 'w') as f: | |
1215 | f.write(boards) | |
2559cd89 | 1216 | |
fc2661ee MY |
1217 | def show_suspicious_boards(self): |
1218 | """Display all boards (defconfigs) with possible misconversion.""" | |
09c6c066 | 1219 | boards = set() |
fc2661ee MY |
1220 | output_file = 'moveconfig.suspicious' |
1221 | ||
1222 | for slot in self.slots: | |
09c6c066 | 1223 | boards |= slot.get_suspicious_boards() |
fc2661ee MY |
1224 | |
1225 | if boards: | |
1226 | boards = '\n'.join(boards) + '\n' | |
1227 | msg = "The following boards might have been converted incorrectly.\n" | |
1228 | msg += "It is highly recommended to check them manually:\n" | |
1229 | msg += boards | |
1230 | msg += "(the list has been saved in %s)\n" % output_file | |
1231 | print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW, | |
1232 | msg) | |
1233 | ||
1234 | with open(output_file, 'w') as f: | |
1235 | f.write(boards) | |
1236 | ||
5cc42a51 MY |
1237 | class ReferenceSource: |
1238 | ||
1239 | """Reference source against which original configs should be parsed.""" | |
1240 | ||
1241 | def __init__(self, commit): | |
1242 | """Create a reference source directory based on a specified commit. | |
1243 | ||
1244 | Arguments: | |
1245 | commit: commit to git-clone | |
1246 | """ | |
1247 | self.src_dir = tempfile.mkdtemp() | |
1248 | print "Cloning git repo to a separate work directory..." | |
1249 | subprocess.check_output(['git', 'clone', os.getcwd(), '.'], | |
1250 | cwd=self.src_dir) | |
1251 | print "Checkout '%s' to build the original autoconf.mk." % \ | |
1252 | subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip() | |
1253 | subprocess.check_output(['git', 'checkout', commit], | |
1254 | stderr=subprocess.STDOUT, cwd=self.src_dir) | |
6b96c1a1 JH |
1255 | |
1256 | def __del__(self): | |
5cc42a51 | 1257 | """Delete the reference source directory |
6b96c1a1 JH |
1258 | |
1259 | This function makes sure the temporary directory is cleaned away | |
1260 | even if Python suddenly dies due to error. It should be done in here | |
1261 | because it is guaranteed the destructor is always invoked when the | |
1262 | instance of the class gets unreferenced. | |
1263 | """ | |
5cc42a51 MY |
1264 | shutil.rmtree(self.src_dir) |
1265 | ||
1266 | def get_dir(self): | |
1267 | """Return the absolute path to the reference source directory.""" | |
6b96c1a1 | 1268 | |
5cc42a51 | 1269 | return self.src_dir |
6b96c1a1 | 1270 | |
b134bc13 | 1271 | def move_config(configs, options): |
5a27c734 MY |
1272 | """Move config options to defconfig files. |
1273 | ||
1274 | Arguments: | |
b134bc13 | 1275 | configs: A list of CONFIGs to move. |
5a27c734 MY |
1276 | options: option flags |
1277 | """ | |
b134bc13 | 1278 | if len(configs) == 0: |
6a9f79f7 MY |
1279 | if options.force_sync: |
1280 | print 'No CONFIG is specified. You are probably syncing defconfigs.', | |
1281 | else: | |
1282 | print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.', | |
1283 | else: | |
1284 | print 'Move ' + ', '.join(configs), | |
1285 | print '(jobs: %d)\n' % options.jobs | |
5a27c734 | 1286 | |
6b96c1a1 | 1287 | if options.git_ref: |
5cc42a51 MY |
1288 | reference_src = ReferenceSource(options.git_ref) |
1289 | reference_src_dir = reference_src.get_dir() | |
1290 | else: | |
f432c33f | 1291 | reference_src_dir = None |
6b96c1a1 | 1292 | |
91040e85 | 1293 | if options.defconfigs: |
0dbc9b59 | 1294 | defconfigs = get_matched_defconfigs(options.defconfigs) |
91040e85 | 1295 | else: |
684c306e | 1296 | defconfigs = get_all_defconfigs() |
5a27c734 | 1297 | |
c5e60fd4 | 1298 | progress = Progress(len(defconfigs)) |
6b96c1a1 | 1299 | slots = Slots(configs, options, progress, reference_src_dir) |
5a27c734 MY |
1300 | |
1301 | # Main loop to process defconfig files: | |
1302 | # Add a new subprocess into a vacant slot. | |
1303 | # Sleep if there is no available slot. | |
c5e60fd4 MY |
1304 | for defconfig in defconfigs: |
1305 | while not slots.add(defconfig): | |
5a27c734 MY |
1306 | while not slots.available(): |
1307 | # No available slot: sleep for a while | |
1308 | time.sleep(SLEEP_TIME) | |
1309 | ||
1310 | # wait until all the subprocesses finish | |
1311 | while not slots.empty(): | |
1312 | time.sleep(SLEEP_TIME) | |
1313 | ||
2e2ce6c0 | 1314 | print '' |
5a27c734 | 1315 | slots.show_failed_boards() |
fc2661ee | 1316 | slots.show_suspicious_boards() |
5a27c734 | 1317 | |
5a27c734 MY |
1318 | def main(): |
1319 | try: | |
1320 | cpu_count = multiprocessing.cpu_count() | |
1321 | except NotImplementedError: | |
1322 | cpu_count = 1 | |
1323 | ||
1324 | parser = optparse.OptionParser() | |
1325 | # Add options here | |
1326 | parser.add_option('-c', '--color', action='store_true', default=False, | |
1327 | help='display the log in color') | |
9ede2123 SG |
1328 | parser.add_option('-C', '--commit', action='store_true', default=False, |
1329 | help='Create a git commit for the operation') | |
91040e85 JH |
1330 | parser.add_option('-d', '--defconfigs', type='string', |
1331 | help='a file containing a list of defconfigs to move') | |
5a27c734 MY |
1332 | parser.add_option('-n', '--dry-run', action='store_true', default=False, |
1333 | help='perform a trial run (show log with no changes)') | |
1334 | parser.add_option('-e', '--exit-on-error', action='store_true', | |
1335 | default=False, | |
1336 | help='exit immediately on any error') | |
8513dc04 MY |
1337 | parser.add_option('-s', '--force-sync', action='store_true', default=False, |
1338 | help='force sync by savedefconfig') | |
07913d1e MY |
1339 | parser.add_option('-S', '--spl', action='store_true', default=False, |
1340 | help='parse config options defined for SPL build') | |
2144f880 JH |
1341 | parser.add_option('-H', '--headers-only', dest='cleanup_headers_only', |
1342 | action='store_true', default=False, | |
1343 | help='only cleanup the headers') | |
5a27c734 MY |
1344 | parser.add_option('-j', '--jobs', type='int', default=cpu_count, |
1345 | help='the number of jobs to run simultaneously') | |
6b96c1a1 JH |
1346 | parser.add_option('-r', '--git-ref', type='string', |
1347 | help='the git ref to clone for building the autoconf.mk') | |
6b403dfd SG |
1348 | parser.add_option('-y', '--yes', action='store_true', default=False, |
1349 | help="respond 'yes' to any prompts") | |
95bf9c7e JH |
1350 | parser.add_option('-v', '--verbose', action='store_true', default=False, |
1351 | help='show any build errors as boards are built') | |
b6ef393a | 1352 | parser.usage += ' CONFIG ...' |
5a27c734 | 1353 | |
b6ef393a | 1354 | (options, configs) = parser.parse_args() |
5a27c734 | 1355 | |
6a9f79f7 | 1356 | if len(configs) == 0 and not options.force_sync: |
5a27c734 MY |
1357 | parser.print_usage() |
1358 | sys.exit(1) | |
1359 | ||
b6ef393a MY |
1360 | # prefix the option name with CONFIG_ if missing |
1361 | configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config | |
1362 | for config in configs ] | |
5a27c734 | 1363 | |
2144f880 JH |
1364 | check_top_directory() |
1365 | ||
1366 | if not options.cleanup_headers_only: | |
f7536f79 MY |
1367 | check_clean_directory() |
1368 | update_cross_compile(options.color) | |
b134bc13 | 1369 | move_config(configs, options) |
2144f880 | 1370 | |
6a9f79f7 | 1371 | if configs: |
e9ea1221 | 1372 | cleanup_headers(configs, options) |
9ab0296a | 1373 | cleanup_extra_options(configs, options) |
ca43834d | 1374 | cleanup_whitelist(configs, options) |
f90df596 | 1375 | cleanup_readme(configs, options) |
5a27c734 | 1376 | |
9ede2123 SG |
1377 | if options.commit: |
1378 | subprocess.call(['git', 'add', '-u']) | |
1379 | if configs: | |
1380 | msg = 'Convert %s %sto Kconfig' % (configs[0], | |
1381 | 'et al ' if len(configs) > 1 else '') | |
1382 | msg += ('\n\nThis converts the following to Kconfig:\n %s\n' % | |
1383 | '\n '.join(configs)) | |
1384 | else: | |
1385 | msg = 'configs: Resync with savedefconfig' | |
1386 | msg += '\n\nRsync all defconfig files using moveconfig.py' | |
1387 | subprocess.call(['git', 'commit', '-s', '-m', msg]) | |
1388 | ||
5a27c734 MY |
1389 | if __name__ == '__main__': |
1390 | main() |