]>
Commit | Line | Data |
---|---|---|
5a27c734 MY |
1 | #!/usr/bin/env python2 |
2 | # | |
3 | # Author: Masahiro Yamada <yamada.masahiro@socionext.com> | |
4 | # | |
5 | # SPDX-License-Identifier: GPL-2.0+ | |
6 | # | |
7 | ||
8 | """ | |
9 | Move config options from headers to defconfig files. | |
10 | ||
11 | Since Kconfig was introduced to U-Boot, we have worked on moving | |
12 | config options from headers to Kconfig (defconfig). | |
13 | ||
14 | This tool intends to help this tremendous work. | |
15 | ||
16 | ||
17 | Usage | |
18 | ----- | |
19 | ||
b6ef393a | 20 | First, you must edit the Kconfig to add the menu entries for the configs |
96464bad JH |
21 | you are moving. |
22 | ||
b6ef393a MY |
23 | And then run this tool giving CONFIG names you want to move. |
24 | For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE, | |
25 | simply type as follows: | |
5a27c734 | 26 | |
b6ef393a | 27 | $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE |
5a27c734 | 28 | |
b6ef393a | 29 | The tool walks through all the defconfig files and move the given CONFIGs. |
5a27c734 MY |
30 | |
31 | The log is also displayed on the terminal. | |
32 | ||
1d085568 | 33 | The log is printed for each defconfig as follows: |
5a27c734 | 34 | |
1d085568 MY |
35 | <defconfig_name> |
36 | <action1> | |
37 | <action2> | |
38 | <action3> | |
39 | ... | |
5a27c734 | 40 | |
1d085568 MY |
41 | <defconfig_name> is the name of the defconfig. |
42 | ||
43 | <action*> shows what the tool did for that defconfig. | |
c21fc7e2 | 44 | It looks like one of the following: |
5a27c734 MY |
45 | |
46 | - Move 'CONFIG_... ' | |
47 | This config option was moved to the defconfig | |
48 | ||
cc008299 | 49 | - CONFIG_... is not defined in Kconfig. Do nothing. |
916224c3 MY |
50 | The entry for this CONFIG was not found in Kconfig. The option is not |
51 | defined in the config header, either. So, this case can be just skipped. | |
52 | ||
53 | - CONFIG_... is not defined in Kconfig (suspicious). Do nothing. | |
54 | This option is defined in the config header, but its entry was not found | |
55 | in Kconfig. | |
cc008299 MY |
56 | There are two common cases: |
57 | - You forgot to create an entry for the CONFIG before running | |
58 | this tool, or made a typo in a CONFIG passed to this tool. | |
59 | - The entry was hidden due to unmet 'depends on'. | |
916224c3 MY |
60 | The tool does not know if the result is reasonable, so please check it |
61 | manually. | |
cc008299 MY |
62 | |
63 | - 'CONFIG_...' is the same as the define in Kconfig. Do nothing. | |
64 | The define in the config header matched the one in Kconfig. | |
65 | We do not need to touch it. | |
5a27c734 | 66 | |
90ed6cba MY |
67 | - Compiler is missing. Do nothing. |
68 | The compiler specified for this architecture was not found | |
69 | in your PATH environment. | |
70 | (If -e option is passed, the tool exits immediately.) | |
71 | ||
72 | - Failed to process. | |
5a27c734 MY |
73 | An error occurred during processing this defconfig. Skipped. |
74 | (If -e option is passed, the tool exits immediately on error.) | |
75 | ||
76 | Finally, you will be asked, Clean up headers? [y/n]: | |
77 | ||
78 | If you say 'y' here, the unnecessary config defines are removed | |
79 | from the config headers (include/configs/*.h). | |
80 | It just uses the regex method, so you should not rely on it. | |
81 | Just in case, please do 'git diff' to see what happened. | |
82 | ||
83 | ||
b6ef393a MY |
84 | How does it work? |
85 | ----------------- | |
5a27c734 MY |
86 | |
87 | This tool runs configuration and builds include/autoconf.mk for every | |
88 | defconfig. The config options defined in Kconfig appear in the .config | |
89 | file (unless they are hidden because of unmet dependency.) | |
90 | On the other hand, the config options defined by board headers are seen | |
91 | in include/autoconf.mk. The tool looks for the specified options in both | |
b6ef393a MY |
92 | of them to decide the appropriate action for the options. If the given |
93 | config option is found in the .config, but its value does not match the | |
94 | one from the board header, the config option in the .config is replaced | |
95 | with the define in the board header. Then, the .config is synced by | |
96 | "make savedefconfig" and the defconfig is updated with it. | |
5a27c734 MY |
97 | |
98 | For faster processing, this tool handles multi-threading. It creates | |
99 | separate build directories where the out-of-tree build is run. The | |
100 | temporary build directories are automatically created and deleted as | |
101 | needed. The number of threads are chosen based on the number of the CPU | |
102 | cores of your system although you can change it via -j (--jobs) option. | |
103 | ||
104 | ||
105 | Toolchains | |
106 | ---------- | |
107 | ||
108 | Appropriate toolchain are necessary to generate include/autoconf.mk | |
109 | for all the architectures supported by U-Boot. Most of them are available | |
110 | at the kernel.org site, some are not provided by kernel.org. | |
111 | ||
112 | The default per-arch CROSS_COMPILE used by this tool is specified by | |
113 | the list below, CROSS_COMPILE. You may wish to update the list to | |
114 | use your own. Instead of modifying the list directly, you can give | |
115 | them via environments. | |
116 | ||
117 | ||
ddf91c64 SG |
118 | Tips and trips |
119 | -------------- | |
120 | ||
121 | To sync only X86 defconfigs: | |
122 | ||
123 | ./tools/moveconfig.py -s -d <(grep -l X86 configs/*) | |
124 | ||
125 | or: | |
126 | ||
127 | grep -l X86 configs/* | ./tools/moveconfig.py -s -d - | |
128 | ||
129 | To process CONFIG_CMD_FPGAD only for a subset of configs based on path match: | |
130 | ||
131 | ls configs/{hrcon*,iocon*,strider*} | \ | |
132 | ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d - | |
133 | ||
134 | ||
99b66605 SG |
135 | Finding implied CONFIGs |
136 | ----------------------- | |
137 | ||
138 | Some CONFIG options can be implied by others and this can help to reduce | |
139 | the size of the defconfig files. For example, CONFIG_X86 implies | |
140 | CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and | |
141 | all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to | |
142 | each of the x86 defconfig files. | |
143 | ||
144 | This tool can help find such configs. To use it, first build a database: | |
145 | ||
146 | ./tools/moveconfig.py -b | |
147 | ||
148 | Then try to query it: | |
149 | ||
150 | ./tools/moveconfig.py -i CONFIG_CMD_IRQ | |
151 | CONFIG_CMD_IRQ found in 311/2384 defconfigs | |
152 | 44 : CONFIG_SYS_FSL_ERRATUM_IFC_A002769 | |
153 | 41 : CONFIG_SYS_FSL_ERRATUM_A007075 | |
154 | 31 : CONFIG_SYS_FSL_DDR_VER_44 | |
155 | 28 : CONFIG_ARCH_P1010 | |
156 | 28 : CONFIG_SYS_FSL_ERRATUM_P1010_A003549 | |
157 | 28 : CONFIG_SYS_FSL_ERRATUM_SEC_A003571 | |
158 | 28 : CONFIG_SYS_FSL_ERRATUM_IFC_A003399 | |
159 | 25 : CONFIG_SYS_FSL_ERRATUM_A008044 | |
160 | 22 : CONFIG_ARCH_P1020 | |
161 | 21 : CONFIG_SYS_FSL_DDR_VER_46 | |
162 | 20 : CONFIG_MAX_PIRQ_LINKS | |
163 | 20 : CONFIG_HPET_ADDRESS | |
164 | 20 : CONFIG_X86 | |
165 | 20 : CONFIG_PCIE_ECAM_SIZE | |
166 | 20 : CONFIG_IRQ_SLOT_COUNT | |
167 | 20 : CONFIG_I8259_PIC | |
168 | 20 : CONFIG_CPU_ADDR_BITS | |
169 | 20 : CONFIG_RAMBASE | |
170 | 20 : CONFIG_SYS_FSL_ERRATUM_A005871 | |
171 | 20 : CONFIG_PCIE_ECAM_BASE | |
172 | 20 : CONFIG_X86_TSC_TIMER | |
173 | 20 : CONFIG_I8254_TIMER | |
174 | 20 : CONFIG_CMD_GETTIME | |
175 | 19 : CONFIG_SYS_FSL_ERRATUM_A005812 | |
176 | 18 : CONFIG_X86_RUN_32BIT | |
177 | 17 : CONFIG_CMD_CHIP_CONFIG | |
178 | ... | |
179 | ||
180 | This shows a list of config options which might imply CONFIG_CMD_EEPROM along | |
181 | with how many defconfigs they cover. From this you can see that CONFIG_X86 | |
182 | implies CONFIG_CMD_EEPROM. Therefore, instead of adding CONFIG_CMD_EEPROM to | |
183 | the defconfig of every x86 board, you could add a single imply line to the | |
184 | Kconfig file: | |
185 | ||
186 | config X86 | |
187 | bool "x86 architecture" | |
188 | ... | |
189 | imply CMD_EEPROM | |
190 | ||
191 | That will cover 20 defconfigs. Many of the options listed are not suitable as | |
192 | they are not related. E.g. it would be odd for CONFIG_CMD_GETTIME to imply | |
193 | CMD_EEPROM. | |
194 | ||
195 | Using this search you can reduce the size of moveconfig patches. | |
196 | ||
cb008830 SG |
197 | You can automatically add 'imply' statements in the Kconfig with the -a |
198 | option: | |
199 | ||
200 | ./tools/moveconfig.py -s -i CONFIG_SCSI \ | |
201 | -a CONFIG_ARCH_LS1021A,CONFIG_ARCH_LS1043A | |
202 | ||
203 | This will add 'imply SCSI' to the two CONFIG options mentioned, assuming that | |
204 | the database indicates that they do actually imply CONFIG_SCSI and do not | |
205 | already have an 'imply SCSI'. | |
206 | ||
207 | The output shows where the imply is added: | |
208 | ||
209 | 18 : CONFIG_ARCH_LS1021A arch/arm/cpu/armv7/ls102xa/Kconfig:1 | |
210 | 13 : CONFIG_ARCH_LS1043A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:11 | |
211 | 12 : CONFIG_ARCH_LS1046A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:31 | |
212 | ||
213 | The first number is the number of boards which can avoid having a special | |
214 | CONFIG_SCSI option in their defconfig file if this 'imply' is added. | |
215 | The location at the right is the Kconfig file and line number where the config | |
216 | appears. For example, adding 'imply CONFIG_SCSI' to the 'config ARCH_LS1021A' | |
217 | in arch/arm/cpu/armv7/ls102xa/Kconfig at line 1 will help 18 boards to reduce | |
218 | the size of their defconfig files. | |
219 | ||
220 | If you want to add an 'imply' to every imply config in the list, you can use | |
221 | ||
222 | ./tools/moveconfig.py -s -i CONFIG_SCSI -a all | |
223 | ||
224 | To control which ones are displayed, use -I <list> where list is a list of | |
225 | options (use '-I help' to see possible options and their meaning). | |
226 | ||
227 | To skip showing you options that already have an 'imply' attached, use -A. | |
228 | ||
229 | When you have finished adding 'imply' options you can regenerate the | |
230 | defconfig files for affected boards with something like: | |
231 | ||
232 | git show --stat | ./tools/moveconfig.py -s -d - | |
233 | ||
234 | This will regenerate only those defconfigs changed in the current commit. | |
235 | If you start with (say) 100 defconfigs being changed in the commit, and add | |
236 | a few 'imply' options as above, then regenerate, hopefully you can reduce the | |
237 | number of defconfigs changed in the commit. | |
238 | ||
99b66605 | 239 | |
5a27c734 MY |
240 | Available options |
241 | ----------------- | |
242 | ||
243 | -c, --color | |
244 | Surround each portion of the log with escape sequences to display it | |
245 | in color on the terminal. | |
246 | ||
9ede2123 SG |
247 | -C, --commit |
248 | Create a git commit with the changes when the operation is complete. A | |
249 | standard commit message is used which may need to be edited. | |
250 | ||
91040e85 | 251 | -d, --defconfigs |
0dbc9b59 | 252 | Specify a file containing a list of defconfigs to move. The defconfig |
ddf91c64 | 253 | files can be given with shell-style wildcards. Use '-' to read from stdin. |
91040e85 | 254 | |
5a27c734 | 255 | -n, --dry-run |
b6ef393a | 256 | Perform a trial run that does not make any changes. It is useful to |
5a27c734 MY |
257 | see what is going to happen before one actually runs it. |
258 | ||
259 | -e, --exit-on-error | |
260 | Exit immediately if Make exits with a non-zero status while processing | |
261 | a defconfig file. | |
262 | ||
8513dc04 MY |
263 | -s, --force-sync |
264 | Do "make savedefconfig" forcibly for all the defconfig files. | |
265 | If not specified, "make savedefconfig" only occurs for cases | |
266 | where at least one CONFIG was moved. | |
267 | ||
07913d1e MY |
268 | -S, --spl |
269 | Look for moved config options in spl/include/autoconf.mk instead of | |
270 | include/autoconf.mk. This is useful for moving options for SPL build | |
271 | because SPL related options (mostly prefixed with CONFIG_SPL_) are | |
272 | sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals. | |
273 | ||
2144f880 JH |
274 | -H, --headers-only |
275 | Only cleanup the headers; skip the defconfig processing | |
276 | ||
5a27c734 MY |
277 | -j, --jobs |
278 | Specify the number of threads to run simultaneously. If not specified, | |
279 | the number of threads is the same as the number of CPU cores. | |
280 | ||
6b96c1a1 JH |
281 | -r, --git-ref |
282 | Specify the git ref to clone for building the autoconf.mk. If unspecified | |
283 | use the CWD. This is useful for when changes to the Kconfig affect the | |
284 | default values and you want to capture the state of the defconfig from | |
285 | before that change was in effect. If in doubt, specify a ref pre-Kconfig | |
286 | changes (use HEAD if Kconfig changes are not committed). Worst case it will | |
287 | take a bit longer to run, but will always do the right thing. | |
288 | ||
95bf9c7e JH |
289 | -v, --verbose |
290 | Show any build errors as boards are built | |
291 | ||
6b403dfd SG |
292 | -y, --yes |
293 | Instead of prompting, automatically go ahead with all operations. This | |
ddf91c64 SG |
294 | includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist |
295 | and the README. | |
6b403dfd | 296 | |
5a27c734 MY |
297 | To see the complete list of supported options, run |
298 | ||
299 | $ tools/moveconfig.py -h | |
300 | ||
301 | """ | |
302 | ||
99b66605 | 303 | import collections |
8ba1f5de | 304 | import copy |
f2f6981a | 305 | import difflib |
c8e1b10d | 306 | import filecmp |
5a27c734 | 307 | import fnmatch |
0dbc9b59 | 308 | import glob |
5a27c734 MY |
309 | import multiprocessing |
310 | import optparse | |
311 | import os | |
d73fcb12 | 312 | import Queue |
5a27c734 MY |
313 | import re |
314 | import shutil | |
315 | import subprocess | |
316 | import sys | |
317 | import tempfile | |
d73fcb12 | 318 | import threading |
5a27c734 MY |
319 | import time |
320 | ||
cb008830 SG |
321 | sys.path.append(os.path.join(os.path.dirname(__file__), 'buildman')) |
322 | import kconfiglib | |
323 | ||
5a27c734 MY |
324 | SHOW_GNU_MAKE = 'scripts/show-gnu-make' |
325 | SLEEP_TIME=0.03 | |
326 | ||
327 | # Here is the list of cross-tools I use. | |
328 | # Most of them are available at kernel.org | |
c21fc7e2 | 329 | # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the following: |
5a27c734 | 330 | # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases |
4440ecec | 331 | # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz |
5a27c734 MY |
332 | # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545 |
333 | # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu | |
334 | CROSS_COMPILE = { | |
335 | 'arc': 'arc-linux-', | |
336 | 'aarch64': 'aarch64-linux-', | |
337 | 'arm': 'arm-unknown-linux-gnueabi-', | |
5a27c734 MY |
338 | 'm68k': 'm68k-linux-', |
339 | 'microblaze': 'microblaze-linux-', | |
340 | 'mips': 'mips-linux-', | |
341 | 'nds32': 'nds32le-linux-', | |
342 | 'nios2': 'nios2-linux-gnu-', | |
5a27c734 MY |
343 | 'powerpc': 'powerpc-linux-', |
344 | 'sh': 'sh-linux-gnu-', | |
88e1346e MY |
345 | 'x86': 'i386-linux-', |
346 | 'xtensa': 'xtensa-linux-' | |
5a27c734 MY |
347 | } |
348 | ||
349 | STATE_IDLE = 0 | |
350 | STATE_DEFCONFIG = 1 | |
351 | STATE_AUTOCONF = 2 | |
96464bad | 352 | STATE_SAVEDEFCONFIG = 3 |
5a27c734 MY |
353 | |
354 | ACTION_MOVE = 0 | |
cc008299 | 355 | ACTION_NO_ENTRY = 1 |
916224c3 MY |
356 | ACTION_NO_ENTRY_WARN = 2 |
357 | ACTION_NO_CHANGE = 3 | |
5a27c734 MY |
358 | |
359 | COLOR_BLACK = '0;30' | |
360 | COLOR_RED = '0;31' | |
361 | COLOR_GREEN = '0;32' | |
362 | COLOR_BROWN = '0;33' | |
363 | COLOR_BLUE = '0;34' | |
364 | COLOR_PURPLE = '0;35' | |
365 | COLOR_CYAN = '0;36' | |
366 | COLOR_LIGHT_GRAY = '0;37' | |
367 | COLOR_DARK_GRAY = '1;30' | |
368 | COLOR_LIGHT_RED = '1;31' | |
369 | COLOR_LIGHT_GREEN = '1;32' | |
370 | COLOR_YELLOW = '1;33' | |
371 | COLOR_LIGHT_BLUE = '1;34' | |
372 | COLOR_LIGHT_PURPLE = '1;35' | |
373 | COLOR_LIGHT_CYAN = '1;36' | |
374 | COLOR_WHITE = '1;37' | |
375 | ||
f3b8e647 | 376 | AUTO_CONF_PATH = 'include/config/auto.conf' |
d73fcb12 | 377 | CONFIG_DATABASE = 'moveconfig.db' |
f3b8e647 | 378 | |
cb008830 | 379 | CONFIG_LEN = len('CONFIG_') |
f3b8e647 | 380 | |
5a27c734 MY |
381 | ### helper functions ### |
382 | def get_devnull(): | |
383 | """Get the file object of '/dev/null' device.""" | |
384 | try: | |
385 | devnull = subprocess.DEVNULL # py3k | |
386 | except AttributeError: | |
387 | devnull = open(os.devnull, 'wb') | |
388 | return devnull | |
389 | ||
390 | def check_top_directory(): | |
391 | """Exit if we are not at the top of source directory.""" | |
392 | for f in ('README', 'Licenses'): | |
393 | if not os.path.exists(f): | |
394 | sys.exit('Please run at the top of source directory.') | |
395 | ||
bd63e5ba MY |
396 | def check_clean_directory(): |
397 | """Exit if the source tree is not clean.""" | |
398 | for f in ('.config', 'include/config'): | |
399 | if os.path.exists(f): | |
400 | sys.exit("source tree is not clean, please run 'make mrproper'") | |
401 | ||
5a27c734 MY |
402 | def get_make_cmd(): |
403 | """Get the command name of GNU Make. | |
404 | ||
405 | U-Boot needs GNU Make for building, but the command name is not | |
406 | necessarily "make". (for example, "gmake" on FreeBSD). | |
407 | Returns the most appropriate command name on your system. | |
408 | """ | |
409 | process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) | |
410 | ret = process.communicate() | |
411 | if process.returncode: | |
412 | sys.exit('GNU Make not found') | |
413 | return ret[0].rstrip() | |
414 | ||
25f978cb SG |
415 | def get_matched_defconfig(line): |
416 | """Get the defconfig files that match a pattern | |
417 | ||
418 | Args: | |
419 | line: Path or filename to match, e.g. 'configs/snow_defconfig' or | |
420 | 'k2*_defconfig'. If no directory is provided, 'configs/' is | |
421 | prepended | |
422 | ||
423 | Returns: | |
424 | a list of matching defconfig files | |
425 | """ | |
426 | dirname = os.path.dirname(line) | |
427 | if dirname: | |
428 | pattern = line | |
429 | else: | |
430 | pattern = os.path.join('configs', line) | |
431 | return glob.glob(pattern) + glob.glob(pattern + '_defconfig') | |
432 | ||
0dbc9b59 | 433 | def get_matched_defconfigs(defconfigs_file): |
ee4e61bd SG |
434 | """Get all the defconfig files that match the patterns in a file. |
435 | ||
436 | Args: | |
437 | defconfigs_file: File containing a list of defconfigs to process, or | |
438 | '-' to read the list from stdin | |
439 | ||
440 | Returns: | |
441 | A list of paths to defconfig files, with no duplicates | |
442 | """ | |
0dbc9b59 | 443 | defconfigs = [] |
ee4e61bd SG |
444 | if defconfigs_file == '-': |
445 | fd = sys.stdin | |
446 | defconfigs_file = 'stdin' | |
447 | else: | |
448 | fd = open(defconfigs_file) | |
449 | for i, line in enumerate(fd): | |
0dbc9b59 MY |
450 | line = line.strip() |
451 | if not line: | |
452 | continue # skip blank lines silently | |
2ddd85dc SG |
453 | if ' ' in line: |
454 | line = line.split(' ')[0] # handle 'git log' input | |
25f978cb | 455 | matched = get_matched_defconfig(line) |
0dbc9b59 MY |
456 | if not matched: |
457 | print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \ | |
458 | (defconfigs_file, i + 1, line) | |
459 | ||
460 | defconfigs += matched | |
461 | ||
462 | # use set() to drop multiple matching | |
463 | return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ] | |
464 | ||
684c306e MY |
465 | def get_all_defconfigs(): |
466 | """Get all the defconfig files under the configs/ directory.""" | |
467 | defconfigs = [] | |
468 | for (dirpath, dirnames, filenames) in os.walk('configs'): | |
469 | dirpath = dirpath[len('configs') + 1:] | |
470 | for filename in fnmatch.filter(filenames, '*_defconfig'): | |
471 | defconfigs.append(os.path.join(dirpath, filename)) | |
472 | ||
473 | return defconfigs | |
474 | ||
5a27c734 MY |
475 | def color_text(color_enabled, color, string): |
476 | """Return colored string.""" | |
477 | if color_enabled: | |
1d085568 MY |
478 | # LF should not be surrounded by the escape sequence. |
479 | # Otherwise, additional whitespace or line-feed might be printed. | |
480 | return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else '' | |
481 | for s in string.split('\n') ]) | |
5a27c734 MY |
482 | else: |
483 | return string | |
484 | ||
e9ea1221 | 485 | def show_diff(a, b, file_path, color_enabled): |
f2f6981a MY |
486 | """Show unidified diff. |
487 | ||
488 | Arguments: | |
489 | a: A list of lines (before) | |
490 | b: A list of lines (after) | |
491 | file_path: Path to the file | |
e9ea1221 | 492 | color_enabled: Display the diff in color |
f2f6981a MY |
493 | """ |
494 | ||
495 | diff = difflib.unified_diff(a, b, | |
496 | fromfile=os.path.join('a', file_path), | |
497 | tofile=os.path.join('b', file_path)) | |
498 | ||
499 | for line in diff: | |
e9ea1221 MY |
500 | if line[0] == '-' and line[1] != '-': |
501 | print color_text(color_enabled, COLOR_RED, line), | |
502 | elif line[0] == '+' and line[1] != '+': | |
503 | print color_text(color_enabled, COLOR_GREEN, line), | |
504 | else: | |
505 | print line, | |
f2f6981a | 506 | |
90ed6cba | 507 | def update_cross_compile(color_enabled): |
1cc0a9f4 | 508 | """Update per-arch CROSS_COMPILE via environment variables |
5a27c734 MY |
509 | |
510 | The default CROSS_COMPILE values are available | |
511 | in the CROSS_COMPILE list above. | |
512 | ||
1cc0a9f4 | 513 | You can override them via environment variables |
5a27c734 MY |
514 | CROSS_COMPILE_{ARCH}. |
515 | ||
516 | For example, if you want to override toolchain prefixes | |
517 | for ARM and PowerPC, you can do as follows in your shell: | |
518 | ||
519 | export CROSS_COMPILE_ARM=... | |
520 | export CROSS_COMPILE_POWERPC=... | |
90ed6cba MY |
521 | |
522 | Then, this function checks if specified compilers really exist in your | |
523 | PATH environment. | |
5a27c734 MY |
524 | """ |
525 | archs = [] | |
526 | ||
527 | for arch in os.listdir('arch'): | |
528 | if os.path.exists(os.path.join('arch', arch, 'Makefile')): | |
529 | archs.append(arch) | |
530 | ||
531 | # arm64 is a special case | |
532 | archs.append('aarch64') | |
533 | ||
534 | for arch in archs: | |
535 | env = 'CROSS_COMPILE_' + arch.upper() | |
536 | cross_compile = os.environ.get(env) | |
90ed6cba MY |
537 | if not cross_compile: |
538 | cross_compile = CROSS_COMPILE.get(arch, '') | |
539 | ||
540 | for path in os.environ["PATH"].split(os.pathsep): | |
541 | gcc_path = os.path.join(path, cross_compile + 'gcc') | |
542 | if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK): | |
543 | break | |
544 | else: | |
545 | print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW, | |
546 | 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped' | |
547 | % (cross_compile, arch)) | |
548 | cross_compile = None | |
549 | ||
550 | CROSS_COMPILE[arch] = cross_compile | |
5a27c734 | 551 | |
8ba1f5de MY |
552 | def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre, |
553 | extend_post): | |
554 | """Extend matched lines if desired patterns are found before/after already | |
555 | matched lines. | |
556 | ||
557 | Arguments: | |
558 | lines: A list of lines handled. | |
559 | matched: A list of line numbers that have been already matched. | |
560 | (will be updated by this function) | |
561 | pre_patterns: A list of regular expression that should be matched as | |
562 | preamble. | |
563 | post_patterns: A list of regular expression that should be matched as | |
564 | postamble. | |
565 | extend_pre: Add the line number of matched preamble to the matched list. | |
566 | extend_post: Add the line number of matched postamble to the matched list. | |
567 | """ | |
568 | extended_matched = [] | |
569 | ||
570 | j = matched[0] | |
571 | ||
572 | for i in matched: | |
573 | if i == 0 or i < j: | |
574 | continue | |
575 | j = i | |
576 | while j in matched: | |
577 | j += 1 | |
578 | if j >= len(lines): | |
579 | break | |
580 | ||
581 | for p in pre_patterns: | |
582 | if p.search(lines[i - 1]): | |
583 | break | |
584 | else: | |
585 | # not matched | |
586 | continue | |
587 | ||
588 | for p in post_patterns: | |
589 | if p.search(lines[j]): | |
590 | break | |
591 | else: | |
592 | # not matched | |
593 | continue | |
594 | ||
595 | if extend_pre: | |
596 | extended_matched.append(i - 1) | |
597 | if extend_post: | |
598 | extended_matched.append(j) | |
599 | ||
600 | matched += extended_matched | |
601 | matched.sort() | |
602 | ||
85edfc1f CP |
603 | def confirm(options, prompt): |
604 | if not options.yes: | |
605 | while True: | |
606 | choice = raw_input('{} [y/n]: '.format(prompt)) | |
607 | choice = choice.lower() | |
608 | print choice | |
609 | if choice == 'y' or choice == 'n': | |
610 | break | |
611 | ||
612 | if choice == 'n': | |
613 | return False | |
614 | ||
615 | return True | |
616 | ||
e9ea1221 | 617 | def cleanup_one_header(header_path, patterns, options): |
5a27c734 MY |
618 | """Clean regex-matched lines away from a file. |
619 | ||
620 | Arguments: | |
621 | header_path: path to the cleaned file. | |
622 | patterns: list of regex patterns. Any lines matching to these | |
623 | patterns are deleted. | |
e9ea1221 | 624 | options: option flags. |
5a27c734 MY |
625 | """ |
626 | with open(header_path) as f: | |
627 | lines = f.readlines() | |
628 | ||
629 | matched = [] | |
630 | for i, line in enumerate(lines): | |
a3a779f7 MY |
631 | if i - 1 in matched and lines[i - 1][-2:] == '\\\n': |
632 | matched.append(i) | |
633 | continue | |
5a27c734 | 634 | for pattern in patterns: |
8ba1f5de | 635 | if pattern.search(line): |
5a27c734 MY |
636 | matched.append(i) |
637 | break | |
638 | ||
8ba1f5de MY |
639 | if not matched: |
640 | return | |
641 | ||
642 | # remove empty #ifdef ... #endif, successive blank lines | |
643 | pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef | |
644 | pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else | |
645 | pattern_endif = re.compile(r'#\s*endif\W') # #endif | |
646 | pattern_blank = re.compile(r'^\s*$') # empty line | |
647 | ||
648 | while True: | |
649 | old_matched = copy.copy(matched) | |
650 | extend_matched_lines(lines, matched, [pattern_if], | |
651 | [pattern_endif], True, True) | |
652 | extend_matched_lines(lines, matched, [pattern_elif], | |
653 | [pattern_elif, pattern_endif], True, False) | |
654 | extend_matched_lines(lines, matched, [pattern_if, pattern_elif], | |
655 | [pattern_blank], False, True) | |
656 | extend_matched_lines(lines, matched, [pattern_blank], | |
657 | [pattern_elif, pattern_endif], True, False) | |
658 | extend_matched_lines(lines, matched, [pattern_blank], | |
659 | [pattern_blank], True, False) | |
660 | if matched == old_matched: | |
661 | break | |
662 | ||
f2f6981a MY |
663 | tolines = copy.copy(lines) |
664 | ||
665 | for i in reversed(matched): | |
666 | tolines.pop(i) | |
667 | ||
e9ea1221 | 668 | show_diff(lines, tolines, header_path, options.color) |
8ba1f5de | 669 | |
e9ea1221 | 670 | if options.dry_run: |
5a27c734 MY |
671 | return |
672 | ||
673 | with open(header_path, 'w') as f: | |
f2f6981a MY |
674 | for line in tolines: |
675 | f.write(line) | |
5a27c734 | 676 | |
e9ea1221 | 677 | def cleanup_headers(configs, options): |
5a27c734 MY |
678 | """Delete config defines from board headers. |
679 | ||
680 | Arguments: | |
b134bc13 | 681 | configs: A list of CONFIGs to remove. |
e9ea1221 | 682 | options: option flags. |
5a27c734 | 683 | """ |
85edfc1f CP |
684 | if not confirm(options, 'Clean up headers?'): |
685 | return | |
5a27c734 MY |
686 | |
687 | patterns = [] | |
b134bc13 | 688 | for config in configs: |
5a27c734 MY |
689 | patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) |
690 | patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) | |
691 | ||
60727f51 JH |
692 | for dir in 'include', 'arch', 'board': |
693 | for (dirpath, dirnames, filenames) in os.walk(dir): | |
dc6de50b MY |
694 | if dirpath == os.path.join('include', 'generated'): |
695 | continue | |
60727f51 JH |
696 | for filename in filenames: |
697 | if not fnmatch.fnmatch(filename, '*~'): | |
698 | cleanup_one_header(os.path.join(dirpath, filename), | |
e9ea1221 | 699 | patterns, options) |
5a27c734 | 700 | |
9ab0296a MY |
701 | def cleanup_one_extra_option(defconfig_path, configs, options): |
702 | """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file. | |
703 | ||
704 | Arguments: | |
705 | defconfig_path: path to the cleaned defconfig file. | |
706 | configs: A list of CONFIGs to remove. | |
707 | options: option flags. | |
708 | """ | |
709 | ||
710 | start = 'CONFIG_SYS_EXTRA_OPTIONS="' | |
711 | end = '"\n' | |
712 | ||
713 | with open(defconfig_path) as f: | |
714 | lines = f.readlines() | |
715 | ||
716 | for i, line in enumerate(lines): | |
717 | if line.startswith(start) and line.endswith(end): | |
718 | break | |
719 | else: | |
720 | # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig | |
721 | return | |
722 | ||
723 | old_tokens = line[len(start):-len(end)].split(',') | |
724 | new_tokens = [] | |
725 | ||
726 | for token in old_tokens: | |
727 | pos = token.find('=') | |
728 | if not (token[:pos] if pos >= 0 else token) in configs: | |
729 | new_tokens.append(token) | |
730 | ||
731 | if new_tokens == old_tokens: | |
732 | return | |
733 | ||
734 | tolines = copy.copy(lines) | |
735 | ||
736 | if new_tokens: | |
737 | tolines[i] = start + ','.join(new_tokens) + end | |
738 | else: | |
739 | tolines.pop(i) | |
740 | ||
741 | show_diff(lines, tolines, defconfig_path, options.color) | |
742 | ||
743 | if options.dry_run: | |
744 | return | |
745 | ||
746 | with open(defconfig_path, 'w') as f: | |
747 | for line in tolines: | |
748 | f.write(line) | |
749 | ||
750 | def cleanup_extra_options(configs, options): | |
751 | """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files. | |
752 | ||
753 | Arguments: | |
754 | configs: A list of CONFIGs to remove. | |
755 | options: option flags. | |
756 | """ | |
85edfc1f CP |
757 | if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'): |
758 | return | |
9ab0296a MY |
759 | |
760 | configs = [ config[len('CONFIG_'):] for config in configs ] | |
761 | ||
762 | defconfigs = get_all_defconfigs() | |
763 | ||
764 | for defconfig in defconfigs: | |
765 | cleanup_one_extra_option(os.path.join('configs', defconfig), configs, | |
766 | options) | |
767 | ||
ca43834d CP |
768 | def cleanup_whitelist(configs, options): |
769 | """Delete config whitelist entries | |
770 | ||
771 | Arguments: | |
772 | configs: A list of CONFIGs to remove. | |
773 | options: option flags. | |
774 | """ | |
775 | if not confirm(options, 'Clean up whitelist entries?'): | |
776 | return | |
777 | ||
778 | with open(os.path.join('scripts', 'config_whitelist.txt')) as f: | |
779 | lines = f.readlines() | |
780 | ||
781 | lines = [x for x in lines if x.strip() not in configs] | |
782 | ||
783 | with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f: | |
784 | f.write(''.join(lines)) | |
785 | ||
f90df596 CP |
786 | def find_matching(patterns, line): |
787 | for pat in patterns: | |
788 | if pat.search(line): | |
789 | return True | |
790 | return False | |
791 | ||
792 | def cleanup_readme(configs, options): | |
793 | """Delete config description in README | |
794 | ||
795 | Arguments: | |
796 | configs: A list of CONFIGs to remove. | |
797 | options: option flags. | |
798 | """ | |
799 | if not confirm(options, 'Clean up README?'): | |
800 | return | |
801 | ||
802 | patterns = [] | |
803 | for config in configs: | |
804 | patterns.append(re.compile(r'^\s+%s' % config)) | |
805 | ||
806 | with open('README') as f: | |
807 | lines = f.readlines() | |
808 | ||
809 | found = False | |
810 | newlines = [] | |
811 | for line in lines: | |
812 | if not found: | |
813 | found = find_matching(patterns, line) | |
814 | if found: | |
815 | continue | |
816 | ||
817 | if found and re.search(r'^\s+CONFIG', line): | |
818 | found = False | |
819 | ||
820 | if not found: | |
821 | newlines.append(line) | |
822 | ||
823 | with open('README', 'w') as f: | |
824 | f.write(''.join(newlines)) | |
825 | ||
ca43834d | 826 | |
5a27c734 | 827 | ### classes ### |
c5e60fd4 MY |
828 | class Progress: |
829 | ||
830 | """Progress Indicator""" | |
831 | ||
832 | def __init__(self, total): | |
833 | """Create a new progress indicator. | |
834 | ||
835 | Arguments: | |
836 | total: A number of defconfig files to process. | |
837 | """ | |
838 | self.current = 0 | |
839 | self.total = total | |
840 | ||
841 | def inc(self): | |
842 | """Increment the number of processed defconfig files.""" | |
843 | ||
844 | self.current += 1 | |
845 | ||
846 | def show(self): | |
847 | """Display the progress.""" | |
848 | print ' %d defconfigs out of %d\r' % (self.current, self.total), | |
849 | sys.stdout.flush() | |
850 | ||
cb008830 SG |
851 | |
852 | class KconfigScanner: | |
853 | """Kconfig scanner.""" | |
854 | ||
855 | def __init__(self): | |
856 | """Scan all the Kconfig files and create a Config object.""" | |
857 | # Define environment variables referenced from Kconfig | |
858 | os.environ['srctree'] = os.getcwd() | |
859 | os.environ['UBOOTVERSION'] = 'dummy' | |
860 | os.environ['KCONFIG_OBJDIR'] = '' | |
861 | self.conf = kconfiglib.Config() | |
862 | ||
863 | ||
5a27c734 MY |
864 | class KconfigParser: |
865 | ||
866 | """A parser of .config and include/autoconf.mk.""" | |
867 | ||
868 | re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') | |
869 | re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') | |
870 | ||
522e8dcb | 871 | def __init__(self, configs, options, build_dir): |
5a27c734 MY |
872 | """Create a new parser. |
873 | ||
874 | Arguments: | |
b134bc13 | 875 | configs: A list of CONFIGs to move. |
5a27c734 MY |
876 | options: option flags. |
877 | build_dir: Build directory. | |
878 | """ | |
b134bc13 | 879 | self.configs = configs |
5a27c734 | 880 | self.options = options |
1f16992e MY |
881 | self.dotconfig = os.path.join(build_dir, '.config') |
882 | self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk') | |
07913d1e MY |
883 | self.spl_autoconf = os.path.join(build_dir, 'spl', 'include', |
884 | 'autoconf.mk') | |
f3b8e647 | 885 | self.config_autoconf = os.path.join(build_dir, AUTO_CONF_PATH) |
5da4f857 | 886 | self.defconfig = os.path.join(build_dir, 'defconfig') |
5a27c734 MY |
887 | |
888 | def get_cross_compile(self): | |
889 | """Parse .config file and return CROSS_COMPILE. | |
890 | ||
891 | Returns: | |
892 | A string storing the compiler prefix for the architecture. | |
90ed6cba MY |
893 | Return a NULL string for architectures that do not require |
894 | compiler prefix (Sandbox and native build is the case). | |
895 | Return None if the specified compiler is missing in your PATH. | |
896 | Caller should distinguish '' and None. | |
5a27c734 MY |
897 | """ |
898 | arch = '' | |
899 | cpu = '' | |
1f16992e | 900 | for line in open(self.dotconfig): |
5a27c734 MY |
901 | m = self.re_arch.match(line) |
902 | if m: | |
903 | arch = m.group(1) | |
904 | continue | |
905 | m = self.re_cpu.match(line) | |
906 | if m: | |
907 | cpu = m.group(1) | |
908 | ||
90ed6cba MY |
909 | if not arch: |
910 | return None | |
5a27c734 MY |
911 | |
912 | # fix-up for aarch64 | |
913 | if arch == 'arm' and cpu == 'armv8': | |
914 | arch = 'aarch64' | |
915 | ||
90ed6cba | 916 | return CROSS_COMPILE.get(arch, None) |
5a27c734 | 917 | |
b134bc13 | 918 | def parse_one_config(self, config, dotconfig_lines, autoconf_lines): |
5a27c734 MY |
919 | """Parse .config, defconfig, include/autoconf.mk for one config. |
920 | ||
921 | This function looks for the config options in the lines from | |
922 | defconfig, .config, and include/autoconf.mk in order to decide | |
923 | which action should be taken for this defconfig. | |
924 | ||
925 | Arguments: | |
b134bc13 | 926 | config: CONFIG name to parse. |
cc008299 | 927 | dotconfig_lines: lines from the .config file. |
5a27c734 MY |
928 | autoconf_lines: lines from the include/autoconf.mk file. |
929 | ||
930 | Returns: | |
931 | A tupple of the action for this defconfig and the line | |
932 | matched for the config. | |
933 | """ | |
5a27c734 MY |
934 | not_set = '# %s is not set' % config |
935 | ||
5a27c734 MY |
936 | for line in autoconf_lines: |
937 | line = line.rstrip() | |
938 | if line.startswith(config + '='): | |
cc008299 | 939 | new_val = line |
5a27c734 | 940 | break |
5a27c734 | 941 | else: |
cc008299 MY |
942 | new_val = not_set |
943 | ||
916224c3 MY |
944 | for line in dotconfig_lines: |
945 | line = line.rstrip() | |
946 | if line.startswith(config + '=') or line == not_set: | |
947 | old_val = line | |
948 | break | |
949 | else: | |
950 | if new_val == not_set: | |
951 | return (ACTION_NO_ENTRY, config) | |
952 | else: | |
953 | return (ACTION_NO_ENTRY_WARN, config) | |
954 | ||
cc008299 MY |
955 | # If this CONFIG is neither bool nor trisate |
956 | if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set: | |
957 | # tools/scripts/define2mk.sed changes '1' to 'y'. | |
958 | # This is a problem if the CONFIG is int type. | |
959 | # Check the type in Kconfig and handle it correctly. | |
960 | if new_val[-2:] == '=y': | |
961 | new_val = new_val[:-1] + '1' | |
5a27c734 | 962 | |
5030159e MY |
963 | return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE, |
964 | new_val) | |
5a27c734 | 965 | |
1d085568 | 966 | def update_dotconfig(self): |
6ff36d21 | 967 | """Parse files for the config options and update the .config. |
5a27c734 | 968 | |
cc008299 MY |
969 | This function parses the generated .config and include/autoconf.mk |
970 | searching the target options. | |
6ff36d21 | 971 | Move the config option(s) to the .config as needed. |
5a27c734 MY |
972 | |
973 | Arguments: | |
974 | defconfig: defconfig name. | |
522e8dcb MY |
975 | |
976 | Returns: | |
7fb0bacd MY |
977 | Return a tuple of (updated flag, log string). |
978 | The "updated flag" is True if the .config was updated, False | |
979 | otherwise. The "log string" shows what happend to the .config. | |
5a27c734 MY |
980 | """ |
981 | ||
5a27c734 | 982 | results = [] |
7fb0bacd | 983 | updated = False |
916224c3 | 984 | suspicious = False |
07913d1e MY |
985 | rm_files = [self.config_autoconf, self.autoconf] |
986 | ||
987 | if self.options.spl: | |
988 | if os.path.exists(self.spl_autoconf): | |
989 | autoconf_path = self.spl_autoconf | |
990 | rm_files.append(self.spl_autoconf) | |
991 | else: | |
992 | for f in rm_files: | |
993 | os.remove(f) | |
994 | return (updated, suspicious, | |
995 | color_text(self.options.color, COLOR_BROWN, | |
996 | "SPL is not enabled. Skipped.") + '\n') | |
997 | else: | |
998 | autoconf_path = self.autoconf | |
5a27c734 | 999 | |
1f16992e | 1000 | with open(self.dotconfig) as f: |
cc008299 | 1001 | dotconfig_lines = f.readlines() |
5a27c734 | 1002 | |
07913d1e | 1003 | with open(autoconf_path) as f: |
5a27c734 MY |
1004 | autoconf_lines = f.readlines() |
1005 | ||
b134bc13 MY |
1006 | for config in self.configs: |
1007 | result = self.parse_one_config(config, dotconfig_lines, | |
96464bad | 1008 | autoconf_lines) |
5a27c734 MY |
1009 | results.append(result) |
1010 | ||
1011 | log = '' | |
1012 | ||
1013 | for (action, value) in results: | |
1014 | if action == ACTION_MOVE: | |
1015 | actlog = "Move '%s'" % value | |
1016 | log_color = COLOR_LIGHT_GREEN | |
cc008299 MY |
1017 | elif action == ACTION_NO_ENTRY: |
1018 | actlog = "%s is not defined in Kconfig. Do nothing." % value | |
5a27c734 | 1019 | log_color = COLOR_LIGHT_BLUE |
916224c3 MY |
1020 | elif action == ACTION_NO_ENTRY_WARN: |
1021 | actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value | |
1022 | log_color = COLOR_YELLOW | |
1023 | suspicious = True | |
cc008299 MY |
1024 | elif action == ACTION_NO_CHANGE: |
1025 | actlog = "'%s' is the same as the define in Kconfig. Do nothing." \ | |
1026 | % value | |
5a27c734 | 1027 | log_color = COLOR_LIGHT_PURPLE |
07913d1e MY |
1028 | elif action == ACTION_SPL_NOT_EXIST: |
1029 | actlog = "SPL is not enabled for this defconfig. Skip." | |
1030 | log_color = COLOR_PURPLE | |
5a27c734 MY |
1031 | else: |
1032 | sys.exit("Internal Error. This should not happen.") | |
1033 | ||
1d085568 | 1034 | log += color_text(self.options.color, log_color, actlog) + '\n' |
5a27c734 | 1035 | |
1f16992e | 1036 | with open(self.dotconfig, 'a') as f: |
e423d17f MY |
1037 | for (action, value) in results: |
1038 | if action == ACTION_MOVE: | |
1039 | f.write(value + '\n') | |
7fb0bacd | 1040 | updated = True |
5a27c734 | 1041 | |
5da4f857 | 1042 | self.results = results |
07913d1e MY |
1043 | for f in rm_files: |
1044 | os.remove(f) | |
5a27c734 | 1045 | |
916224c3 | 1046 | return (updated, suspicious, log) |
522e8dcb | 1047 | |
5da4f857 MY |
1048 | def check_defconfig(self): |
1049 | """Check the defconfig after savedefconfig | |
1050 | ||
1051 | Returns: | |
1052 | Return additional log if moved CONFIGs were removed again by | |
1053 | 'make savedefconfig'. | |
1054 | """ | |
1055 | ||
1056 | log = '' | |
1057 | ||
1058 | with open(self.defconfig) as f: | |
1059 | defconfig_lines = f.readlines() | |
1060 | ||
1061 | for (action, value) in self.results: | |
1062 | if action != ACTION_MOVE: | |
1063 | continue | |
1064 | if not value + '\n' in defconfig_lines: | |
1065 | log += color_text(self.options.color, COLOR_YELLOW, | |
1066 | "'%s' was removed by savedefconfig.\n" % | |
1067 | value) | |
1068 | ||
1069 | return log | |
1070 | ||
d73fcb12 SG |
1071 | |
1072 | class DatabaseThread(threading.Thread): | |
1073 | """This thread processes results from Slot threads. | |
1074 | ||
1075 | It collects the data in the master config directary. There is only one | |
1076 | result thread, and this helps to serialise the build output. | |
1077 | """ | |
1078 | def __init__(self, config_db, db_queue): | |
1079 | """Set up a new result thread | |
1080 | ||
1081 | Args: | |
1082 | builder: Builder which will be sent each result | |
1083 | """ | |
1084 | threading.Thread.__init__(self) | |
1085 | self.config_db = config_db | |
1086 | self.db_queue= db_queue | |
1087 | ||
1088 | def run(self): | |
1089 | """Called to start up the result thread. | |
1090 | ||
1091 | We collect the next result job and pass it on to the build. | |
1092 | """ | |
1093 | while True: | |
1094 | defconfig, configs = self.db_queue.get() | |
1095 | self.config_db[defconfig] = configs | |
1096 | self.db_queue.task_done() | |
1097 | ||
1098 | ||
5a27c734 MY |
1099 | class Slot: |
1100 | ||
1101 | """A slot to store a subprocess. | |
1102 | ||
1103 | Each instance of this class handles one subprocess. | |
1104 | This class is useful to control multiple threads | |
1105 | for faster processing. | |
1106 | """ | |
1107 | ||
d73fcb12 SG |
1108 | def __init__(self, configs, options, progress, devnull, make_cmd, |
1109 | reference_src_dir, db_queue): | |
5a27c734 MY |
1110 | """Create a new process slot. |
1111 | ||
1112 | Arguments: | |
b134bc13 | 1113 | configs: A list of CONFIGs to move. |
5a27c734 | 1114 | options: option flags. |
c5e60fd4 | 1115 | progress: A progress indicator. |
5a27c734 MY |
1116 | devnull: A file object of '/dev/null'. |
1117 | make_cmd: command name of GNU Make. | |
6b96c1a1 JH |
1118 | reference_src_dir: Determine the true starting config state from this |
1119 | source tree. | |
d73fcb12 | 1120 | db_queue: output queue to write config info for the database |
5a27c734 MY |
1121 | """ |
1122 | self.options = options | |
c5e60fd4 | 1123 | self.progress = progress |
5a27c734 MY |
1124 | self.build_dir = tempfile.mkdtemp() |
1125 | self.devnull = devnull | |
1126 | self.make_cmd = (make_cmd, 'O=' + self.build_dir) | |
6b96c1a1 | 1127 | self.reference_src_dir = reference_src_dir |
d73fcb12 | 1128 | self.db_queue = db_queue |
522e8dcb | 1129 | self.parser = KconfigParser(configs, options, self.build_dir) |
5a27c734 | 1130 | self.state = STATE_IDLE |
09c6c066 MY |
1131 | self.failed_boards = set() |
1132 | self.suspicious_boards = set() | |
5a27c734 MY |
1133 | |
1134 | def __del__(self): | |
1135 | """Delete the working directory | |
1136 | ||
1137 | This function makes sure the temporary directory is cleaned away | |
1138 | even if Python suddenly dies due to error. It should be done in here | |
f2dae751 | 1139 | because it is guaranteed the destructor is always invoked when the |
5a27c734 MY |
1140 | instance of the class gets unreferenced. |
1141 | ||
1142 | If the subprocess is still running, wait until it finishes. | |
1143 | """ | |
1144 | if self.state != STATE_IDLE: | |
1145 | while self.ps.poll() == None: | |
1146 | pass | |
1147 | shutil.rmtree(self.build_dir) | |
1148 | ||
c5e60fd4 | 1149 | def add(self, defconfig): |
5a27c734 MY |
1150 | """Assign a new subprocess for defconfig and add it to the slot. |
1151 | ||
1152 | If the slot is vacant, create a new subprocess for processing the | |
1153 | given defconfig and add it to the slot. Just returns False if | |
1154 | the slot is occupied (i.e. the current subprocess is still running). | |
1155 | ||
1156 | Arguments: | |
1157 | defconfig: defconfig name. | |
1158 | ||
1159 | Returns: | |
1160 | Return True on success or False on failure | |
1161 | """ | |
1162 | if self.state != STATE_IDLE: | |
1163 | return False | |
e307fa9d | 1164 | |
5a27c734 | 1165 | self.defconfig = defconfig |
1d085568 | 1166 | self.log = '' |
f432c33f | 1167 | self.current_src_dir = self.reference_src_dir |
e307fa9d | 1168 | self.do_defconfig() |
5a27c734 MY |
1169 | return True |
1170 | ||
1171 | def poll(self): | |
1172 | """Check the status of the subprocess and handle it as needed. | |
1173 | ||
1174 | Returns True if the slot is vacant (i.e. in idle state). | |
1175 | If the configuration is successfully finished, assign a new | |
1176 | subprocess to build include/autoconf.mk. | |
1177 | If include/autoconf.mk is generated, invoke the parser to | |
7fb0bacd MY |
1178 | parse the .config and the include/autoconf.mk, moving |
1179 | config options to the .config as needed. | |
1180 | If the .config was updated, run "make savedefconfig" to sync | |
1181 | it, update the original defconfig, and then set the slot back | |
1182 | to the idle state. | |
5a27c734 MY |
1183 | |
1184 | Returns: | |
1185 | Return True if the subprocess is terminated, False otherwise | |
1186 | """ | |
1187 | if self.state == STATE_IDLE: | |
1188 | return True | |
1189 | ||
1190 | if self.ps.poll() == None: | |
1191 | return False | |
1192 | ||
1193 | if self.ps.poll() != 0: | |
e307fa9d MY |
1194 | self.handle_error() |
1195 | elif self.state == STATE_DEFCONFIG: | |
f432c33f | 1196 | if self.reference_src_dir and not self.current_src_dir: |
6b96c1a1 JH |
1197 | self.do_savedefconfig() |
1198 | else: | |
1199 | self.do_autoconf() | |
e307fa9d | 1200 | elif self.state == STATE_AUTOCONF: |
f432c33f MY |
1201 | if self.current_src_dir: |
1202 | self.current_src_dir = None | |
6b96c1a1 | 1203 | self.do_defconfig() |
d73fcb12 SG |
1204 | elif self.options.build_db: |
1205 | self.do_build_db() | |
6b96c1a1 JH |
1206 | else: |
1207 | self.do_savedefconfig() | |
e307fa9d MY |
1208 | elif self.state == STATE_SAVEDEFCONFIG: |
1209 | self.update_defconfig() | |
1210 | else: | |
1211 | sys.exit("Internal Error. This should not happen.") | |
5a27c734 | 1212 | |
e307fa9d | 1213 | return True if self.state == STATE_IDLE else False |
96464bad | 1214 | |
e307fa9d MY |
1215 | def handle_error(self): |
1216 | """Handle error cases.""" | |
8513dc04 | 1217 | |
e307fa9d MY |
1218 | self.log += color_text(self.options.color, COLOR_LIGHT_RED, |
1219 | "Failed to process.\n") | |
1220 | if self.options.verbose: | |
1221 | self.log += color_text(self.options.color, COLOR_LIGHT_CYAN, | |
1222 | self.ps.stderr.read()) | |
1223 | self.finish(False) | |
96464bad | 1224 | |
e307fa9d MY |
1225 | def do_defconfig(self): |
1226 | """Run 'make <board>_defconfig' to create the .config file.""" | |
c8e1b10d | 1227 | |
e307fa9d MY |
1228 | cmd = list(self.make_cmd) |
1229 | cmd.append(self.defconfig) | |
1230 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, | |
f432c33f MY |
1231 | stderr=subprocess.PIPE, |
1232 | cwd=self.current_src_dir) | |
e307fa9d | 1233 | self.state = STATE_DEFCONFIG |
c8e1b10d | 1234 | |
e307fa9d | 1235 | def do_autoconf(self): |
f3b8e647 | 1236 | """Run 'make AUTO_CONF_PATH'.""" |
5a27c734 | 1237 | |
25400090 | 1238 | self.cross_compile = self.parser.get_cross_compile() |
90ed6cba | 1239 | if self.cross_compile is None: |
1d085568 MY |
1240 | self.log += color_text(self.options.color, COLOR_YELLOW, |
1241 | "Compiler is missing. Do nothing.\n") | |
4efef998 | 1242 | self.finish(False) |
e307fa9d | 1243 | return |
90ed6cba | 1244 | |
5a27c734 | 1245 | cmd = list(self.make_cmd) |
25400090 JH |
1246 | if self.cross_compile: |
1247 | cmd.append('CROSS_COMPILE=%s' % self.cross_compile) | |
7740f653 | 1248 | cmd.append('KCONFIG_IGNORE_DUPLICATES=1') |
f3b8e647 | 1249 | cmd.append(AUTO_CONF_PATH) |
25400090 | 1250 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, |
f432c33f MY |
1251 | stderr=subprocess.PIPE, |
1252 | cwd=self.current_src_dir) | |
5a27c734 | 1253 | self.state = STATE_AUTOCONF |
e307fa9d | 1254 | |
d73fcb12 SG |
1255 | def do_build_db(self): |
1256 | """Add the board to the database""" | |
1257 | configs = {} | |
1258 | with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd: | |
1259 | for line in fd.readlines(): | |
1260 | if line.startswith('CONFIG'): | |
1261 | config, value = line.split('=', 1) | |
1262 | configs[config] = value.rstrip() | |
1263 | self.db_queue.put([self.defconfig, configs]) | |
1264 | self.finish(True) | |
1265 | ||
e307fa9d MY |
1266 | def do_savedefconfig(self): |
1267 | """Update the .config and run 'make savedefconfig'.""" | |
1268 | ||
916224c3 MY |
1269 | (updated, suspicious, log) = self.parser.update_dotconfig() |
1270 | if suspicious: | |
1271 | self.suspicious_boards.add(self.defconfig) | |
e307fa9d MY |
1272 | self.log += log |
1273 | ||
1274 | if not self.options.force_sync and not updated: | |
1275 | self.finish(True) | |
1276 | return | |
1277 | if updated: | |
1278 | self.log += color_text(self.options.color, COLOR_LIGHT_GREEN, | |
1279 | "Syncing by savedefconfig...\n") | |
1280 | else: | |
1281 | self.log += "Syncing by savedefconfig (forced by option)...\n" | |
1282 | ||
1283 | cmd = list(self.make_cmd) | |
1284 | cmd.append('savedefconfig') | |
1285 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, | |
1286 | stderr=subprocess.PIPE) | |
1287 | self.state = STATE_SAVEDEFCONFIG | |
1288 | ||
1289 | def update_defconfig(self): | |
1290 | """Update the input defconfig and go back to the idle state.""" | |
1291 | ||
fc2661ee MY |
1292 | log = self.parser.check_defconfig() |
1293 | if log: | |
09c6c066 | 1294 | self.suspicious_boards.add(self.defconfig) |
fc2661ee | 1295 | self.log += log |
e307fa9d MY |
1296 | orig_defconfig = os.path.join('configs', self.defconfig) |
1297 | new_defconfig = os.path.join(self.build_dir, 'defconfig') | |
1298 | updated = not filecmp.cmp(orig_defconfig, new_defconfig) | |
1299 | ||
1300 | if updated: | |
06cc1d36 | 1301 | self.log += color_text(self.options.color, COLOR_LIGHT_BLUE, |
e307fa9d MY |
1302 | "defconfig was updated.\n") |
1303 | ||
1304 | if not self.options.dry_run and updated: | |
1305 | shutil.move(new_defconfig, orig_defconfig) | |
1306 | self.finish(True) | |
5a27c734 | 1307 | |
4efef998 MY |
1308 | def finish(self, success): |
1309 | """Display log along with progress and go to the idle state. | |
1d085568 MY |
1310 | |
1311 | Arguments: | |
4efef998 MY |
1312 | success: Should be True when the defconfig was processed |
1313 | successfully, or False when it fails. | |
1d085568 MY |
1314 | """ |
1315 | # output at least 30 characters to hide the "* defconfigs out of *". | |
1316 | log = self.defconfig.ljust(30) + '\n' | |
1317 | ||
1318 | log += '\n'.join([ ' ' + s for s in self.log.split('\n') ]) | |
1319 | # Some threads are running in parallel. | |
1320 | # Print log atomically to not mix up logs from different threads. | |
4efef998 MY |
1321 | print >> (sys.stdout if success else sys.stderr), log |
1322 | ||
1323 | if not success: | |
1324 | if self.options.exit_on_error: | |
1325 | sys.exit("Exit on error.") | |
1326 | # If --exit-on-error flag is not set, skip this board and continue. | |
1327 | # Record the failed board. | |
09c6c066 | 1328 | self.failed_boards.add(self.defconfig) |
4efef998 | 1329 | |
1d085568 MY |
1330 | self.progress.inc() |
1331 | self.progress.show() | |
4efef998 | 1332 | self.state = STATE_IDLE |
1d085568 | 1333 | |
5a27c734 | 1334 | def get_failed_boards(self): |
09c6c066 | 1335 | """Returns a set of failed boards (defconfigs) in this slot. |
5a27c734 MY |
1336 | """ |
1337 | return self.failed_boards | |
1338 | ||
fc2661ee | 1339 | def get_suspicious_boards(self): |
09c6c066 | 1340 | """Returns a set of boards (defconfigs) with possible misconversion. |
fc2661ee | 1341 | """ |
916224c3 | 1342 | return self.suspicious_boards - self.failed_boards |
fc2661ee | 1343 | |
5a27c734 MY |
1344 | class Slots: |
1345 | ||
1346 | """Controller of the array of subprocess slots.""" | |
1347 | ||
d73fcb12 | 1348 | def __init__(self, configs, options, progress, reference_src_dir, db_queue): |
5a27c734 MY |
1349 | """Create a new slots controller. |
1350 | ||
1351 | Arguments: | |
b134bc13 | 1352 | configs: A list of CONFIGs to move. |
5a27c734 | 1353 | options: option flags. |
c5e60fd4 | 1354 | progress: A progress indicator. |
6b96c1a1 JH |
1355 | reference_src_dir: Determine the true starting config state from this |
1356 | source tree. | |
d73fcb12 | 1357 | db_queue: output queue to write config info for the database |
5a27c734 MY |
1358 | """ |
1359 | self.options = options | |
1360 | self.slots = [] | |
1361 | devnull = get_devnull() | |
1362 | make_cmd = get_make_cmd() | |
1363 | for i in range(options.jobs): | |
b134bc13 | 1364 | self.slots.append(Slot(configs, options, progress, devnull, |
d73fcb12 | 1365 | make_cmd, reference_src_dir, db_queue)) |
5a27c734 | 1366 | |
c5e60fd4 | 1367 | def add(self, defconfig): |
5a27c734 MY |
1368 | """Add a new subprocess if a vacant slot is found. |
1369 | ||
1370 | Arguments: | |
1371 | defconfig: defconfig name to be put into. | |
1372 | ||
1373 | Returns: | |
1374 | Return True on success or False on failure | |
1375 | """ | |
1376 | for slot in self.slots: | |
c5e60fd4 | 1377 | if slot.add(defconfig): |
5a27c734 MY |
1378 | return True |
1379 | return False | |
1380 | ||
1381 | def available(self): | |
1382 | """Check if there is a vacant slot. | |
1383 | ||
1384 | Returns: | |
1385 | Return True if at lease one vacant slot is found, False otherwise. | |
1386 | """ | |
1387 | for slot in self.slots: | |
1388 | if slot.poll(): | |
1389 | return True | |
1390 | return False | |
1391 | ||
1392 | def empty(self): | |
1393 | """Check if all slots are vacant. | |
1394 | ||
1395 | Returns: | |
1396 | Return True if all the slots are vacant, False otherwise. | |
1397 | """ | |
1398 | ret = True | |
1399 | for slot in self.slots: | |
1400 | if not slot.poll(): | |
1401 | ret = False | |
1402 | return ret | |
1403 | ||
1404 | def show_failed_boards(self): | |
1405 | """Display all of the failed boards (defconfigs).""" | |
09c6c066 | 1406 | boards = set() |
96dccd97 | 1407 | output_file = 'moveconfig.failed' |
5a27c734 MY |
1408 | |
1409 | for slot in self.slots: | |
09c6c066 | 1410 | boards |= slot.get_failed_boards() |
96dccd97 MY |
1411 | |
1412 | if boards: | |
1413 | boards = '\n'.join(boards) + '\n' | |
1414 | msg = "The following boards were not processed due to error:\n" | |
1415 | msg += boards | |
1416 | msg += "(the list has been saved in %s)\n" % output_file | |
1417 | print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, | |
1418 | msg) | |
1419 | ||
1420 | with open(output_file, 'w') as f: | |
1421 | f.write(boards) | |
2559cd89 | 1422 | |
fc2661ee MY |
1423 | def show_suspicious_boards(self): |
1424 | """Display all boards (defconfigs) with possible misconversion.""" | |
09c6c066 | 1425 | boards = set() |
fc2661ee MY |
1426 | output_file = 'moveconfig.suspicious' |
1427 | ||
1428 | for slot in self.slots: | |
09c6c066 | 1429 | boards |= slot.get_suspicious_boards() |
fc2661ee MY |
1430 | |
1431 | if boards: | |
1432 | boards = '\n'.join(boards) + '\n' | |
1433 | msg = "The following boards might have been converted incorrectly.\n" | |
1434 | msg += "It is highly recommended to check them manually:\n" | |
1435 | msg += boards | |
1436 | msg += "(the list has been saved in %s)\n" % output_file | |
1437 | print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW, | |
1438 | msg) | |
1439 | ||
1440 | with open(output_file, 'w') as f: | |
1441 | f.write(boards) | |
1442 | ||
5cc42a51 MY |
1443 | class ReferenceSource: |
1444 | ||
1445 | """Reference source against which original configs should be parsed.""" | |
1446 | ||
1447 | def __init__(self, commit): | |
1448 | """Create a reference source directory based on a specified commit. | |
1449 | ||
1450 | Arguments: | |
1451 | commit: commit to git-clone | |
1452 | """ | |
1453 | self.src_dir = tempfile.mkdtemp() | |
1454 | print "Cloning git repo to a separate work directory..." | |
1455 | subprocess.check_output(['git', 'clone', os.getcwd(), '.'], | |
1456 | cwd=self.src_dir) | |
1457 | print "Checkout '%s' to build the original autoconf.mk." % \ | |
1458 | subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip() | |
1459 | subprocess.check_output(['git', 'checkout', commit], | |
1460 | stderr=subprocess.STDOUT, cwd=self.src_dir) | |
6b96c1a1 JH |
1461 | |
1462 | def __del__(self): | |
5cc42a51 | 1463 | """Delete the reference source directory |
6b96c1a1 JH |
1464 | |
1465 | This function makes sure the temporary directory is cleaned away | |
1466 | even if Python suddenly dies due to error. It should be done in here | |
1467 | because it is guaranteed the destructor is always invoked when the | |
1468 | instance of the class gets unreferenced. | |
1469 | """ | |
5cc42a51 MY |
1470 | shutil.rmtree(self.src_dir) |
1471 | ||
1472 | def get_dir(self): | |
1473 | """Return the absolute path to the reference source directory.""" | |
6b96c1a1 | 1474 | |
5cc42a51 | 1475 | return self.src_dir |
6b96c1a1 | 1476 | |
d73fcb12 | 1477 | def move_config(configs, options, db_queue): |
5a27c734 MY |
1478 | """Move config options to defconfig files. |
1479 | ||
1480 | Arguments: | |
b134bc13 | 1481 | configs: A list of CONFIGs to move. |
5a27c734 MY |
1482 | options: option flags |
1483 | """ | |
b134bc13 | 1484 | if len(configs) == 0: |
6a9f79f7 MY |
1485 | if options.force_sync: |
1486 | print 'No CONFIG is specified. You are probably syncing defconfigs.', | |
d73fcb12 SG |
1487 | elif options.build_db: |
1488 | print 'Building %s database' % CONFIG_DATABASE | |
6a9f79f7 MY |
1489 | else: |
1490 | print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.', | |
1491 | else: | |
1492 | print 'Move ' + ', '.join(configs), | |
1493 | print '(jobs: %d)\n' % options.jobs | |
5a27c734 | 1494 | |
6b96c1a1 | 1495 | if options.git_ref: |
5cc42a51 MY |
1496 | reference_src = ReferenceSource(options.git_ref) |
1497 | reference_src_dir = reference_src.get_dir() | |
1498 | else: | |
f432c33f | 1499 | reference_src_dir = None |
6b96c1a1 | 1500 | |
91040e85 | 1501 | if options.defconfigs: |
0dbc9b59 | 1502 | defconfigs = get_matched_defconfigs(options.defconfigs) |
91040e85 | 1503 | else: |
684c306e | 1504 | defconfigs = get_all_defconfigs() |
5a27c734 | 1505 | |
c5e60fd4 | 1506 | progress = Progress(len(defconfigs)) |
d73fcb12 | 1507 | slots = Slots(configs, options, progress, reference_src_dir, db_queue) |
5a27c734 MY |
1508 | |
1509 | # Main loop to process defconfig files: | |
1510 | # Add a new subprocess into a vacant slot. | |
1511 | # Sleep if there is no available slot. | |
c5e60fd4 MY |
1512 | for defconfig in defconfigs: |
1513 | while not slots.add(defconfig): | |
5a27c734 MY |
1514 | while not slots.available(): |
1515 | # No available slot: sleep for a while | |
1516 | time.sleep(SLEEP_TIME) | |
1517 | ||
1518 | # wait until all the subprocesses finish | |
1519 | while not slots.empty(): | |
1520 | time.sleep(SLEEP_TIME) | |
1521 | ||
2e2ce6c0 | 1522 | print '' |
5a27c734 | 1523 | slots.show_failed_boards() |
fc2661ee | 1524 | slots.show_suspicious_boards() |
5a27c734 | 1525 | |
cb008830 SG |
1526 | def find_kconfig_rules(kconf, config, imply_config): |
1527 | """Check whether a config has a 'select' or 'imply' keyword | |
1528 | ||
1529 | Args: | |
1530 | kconf: Kconfig.Config object | |
1531 | config: Name of config to check (without CONFIG_ prefix) | |
1532 | imply_config: Implying config (without CONFIG_ prefix) which may or | |
1533 | may not have an 'imply' for 'config') | |
1534 | ||
1535 | Returns: | |
1536 | Symbol object for 'config' if found, else None | |
1537 | """ | |
1538 | sym = kconf.get_symbol(imply_config) | |
1539 | if sym: | |
1540 | for sel in sym.get_selected_symbols(): | |
1541 | if sel.get_name() == config: | |
1542 | return sym | |
1543 | return None | |
1544 | ||
1545 | def check_imply_rule(kconf, config, imply_config): | |
1546 | """Check if we can add an 'imply' option | |
1547 | ||
1548 | This finds imply_config in the Kconfig and looks to see if it is possible | |
1549 | to add an 'imply' for 'config' to that part of the Kconfig. | |
1550 | ||
1551 | Args: | |
1552 | kconf: Kconfig.Config object | |
1553 | config: Name of config to check (without CONFIG_ prefix) | |
1554 | imply_config: Implying config (without CONFIG_ prefix) which may or | |
1555 | may not have an 'imply' for 'config') | |
1556 | ||
1557 | Returns: | |
1558 | tuple: | |
1559 | filename of Kconfig file containing imply_config, or None if none | |
1560 | line number within the Kconfig file, or 0 if none | |
1561 | message indicating the result | |
1562 | """ | |
1563 | sym = kconf.get_symbol(imply_config) | |
1564 | if not sym: | |
1565 | return 'cannot find sym' | |
1566 | locs = sym.get_def_locations() | |
1567 | if len(locs) != 1: | |
1568 | return '%d locations' % len(locs) | |
1569 | fname, linenum = locs[0] | |
1570 | cwd = os.getcwd() | |
1571 | if cwd and fname.startswith(cwd): | |
1572 | fname = fname[len(cwd) + 1:] | |
1573 | file_line = ' at %s:%d' % (fname, linenum) | |
1574 | with open(fname) as fd: | |
1575 | data = fd.read().splitlines() | |
1576 | if data[linenum - 1] != 'config %s' % imply_config: | |
1577 | return None, 0, 'bad sym format %s%s' % (data[linenum], file_line) | |
1578 | return fname, linenum, 'adding%s' % file_line | |
1579 | ||
1580 | def add_imply_rule(config, fname, linenum): | |
1581 | """Add a new 'imply' option to a Kconfig | |
1582 | ||
1583 | Args: | |
1584 | config: config option to add an imply for (without CONFIG_ prefix) | |
1585 | fname: Kconfig filename to update | |
1586 | linenum: Line number to place the 'imply' before | |
1587 | ||
1588 | Returns: | |
1589 | Message indicating the result | |
1590 | """ | |
1591 | file_line = ' at %s:%d' % (fname, linenum) | |
1592 | data = open(fname).read().splitlines() | |
1593 | linenum -= 1 | |
1594 | ||
1595 | for offset, line in enumerate(data[linenum:]): | |
1596 | if line.strip().startswith('help') or not line: | |
1597 | data.insert(linenum + offset, '\timply %s' % config) | |
1598 | with open(fname, 'w') as fd: | |
1599 | fd.write('\n'.join(data) + '\n') | |
1600 | return 'added%s' % file_line | |
1601 | ||
1602 | return 'could not insert%s' | |
1603 | ||
1604 | (IMPLY_MIN_2, IMPLY_TARGET, IMPLY_CMD, IMPLY_NON_ARCH_BOARD) = ( | |
1605 | 1, 2, 4, 8) | |
9b2a2e87 SG |
1606 | |
1607 | IMPLY_FLAGS = { | |
1608 | 'min2': [IMPLY_MIN_2, 'Show options which imply >2 boards (normally >5)'], | |
1609 | 'target': [IMPLY_TARGET, 'Allow CONFIG_TARGET_... options to imply'], | |
1610 | 'cmd': [IMPLY_CMD, 'Allow CONFIG_CMD_... to imply'], | |
cb008830 SG |
1611 | 'non-arch-board': [ |
1612 | IMPLY_NON_ARCH_BOARD, | |
1613 | 'Allow Kconfig options outside arch/ and /board/ to imply'], | |
9b2a2e87 SG |
1614 | }; |
1615 | ||
cb008830 SG |
1616 | def do_imply_config(config_list, add_imply, imply_flags, skip_added, |
1617 | check_kconfig=True, find_superset=False): | |
99b66605 SG |
1618 | """Find CONFIG options which imply those in the list |
1619 | ||
1620 | Some CONFIG options can be implied by others and this can help to reduce | |
1621 | the size of the defconfig files. For example, CONFIG_X86 implies | |
1622 | CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and | |
1623 | all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to | |
1624 | each of the x86 defconfig files. | |
1625 | ||
1626 | This function uses the moveconfig database to find such options. It | |
1627 | displays a list of things that could possibly imply those in the list. | |
1628 | The algorithm ignores any that start with CONFIG_TARGET since these | |
1629 | typically refer to only a few defconfigs (often one). It also does not | |
1630 | display a config with less than 5 defconfigs. | |
1631 | ||
1632 | The algorithm works using sets. For each target config in config_list: | |
1633 | - Get the set 'defconfigs' which use that target config | |
1634 | - For each config (from a list of all configs): | |
1635 | - Get the set 'imply_defconfig' of defconfigs which use that config | |
1636 | - | |
1637 | - If imply_defconfigs contains anything not in defconfigs then | |
1638 | this config does not imply the target config | |
1639 | ||
1640 | Params: | |
1641 | config_list: List of CONFIG options to check (each a string) | |
cb008830 | 1642 | add_imply: Automatically add an 'imply' for each config. |
9b2a2e87 SG |
1643 | imply_flags: Flags which control which implying configs are allowed |
1644 | (IMPLY_...) | |
cb008830 SG |
1645 | skip_added: Don't show options which already have an imply added. |
1646 | check_kconfig: Check if implied symbols already have an 'imply' or | |
1647 | 'select' for the target config, and show this information if so. | |
99b66605 SG |
1648 | find_superset: True to look for configs which are a superset of those |
1649 | already found. So for example if CONFIG_EXYNOS5 implies an option, | |
1650 | but CONFIG_EXYNOS covers a larger set of defconfigs and also | |
1651 | implies that option, this will drop the former in favour of the | |
1652 | latter. In practice this option has not proved very used. | |
1653 | ||
1654 | Note the terminoloy: | |
1655 | config - a CONFIG_XXX options (a string, e.g. 'CONFIG_CMD_EEPROM') | |
1656 | defconfig - a defconfig file (a string, e.g. 'configs/snow_defconfig') | |
1657 | """ | |
cb008830 SG |
1658 | kconf = KconfigScanner().conf if check_kconfig else None |
1659 | if add_imply and add_imply != 'all': | |
1660 | add_imply = add_imply.split() | |
1661 | ||
99b66605 SG |
1662 | # key is defconfig name, value is dict of (CONFIG_xxx, value) |
1663 | config_db = {} | |
1664 | ||
1665 | # Holds a dict containing the set of defconfigs that contain each config | |
1666 | # key is config, value is set of defconfigs using that config | |
1667 | defconfig_db = collections.defaultdict(set) | |
1668 | ||
1669 | # Set of all config options we have seen | |
1670 | all_configs = set() | |
1671 | ||
1672 | # Set of all defconfigs we have seen | |
1673 | all_defconfigs = set() | |
1674 | ||
1675 | # Read in the database | |
1676 | configs = {} | |
1677 | with open(CONFIG_DATABASE) as fd: | |
1678 | for line in fd.readlines(): | |
1679 | line = line.rstrip() | |
1680 | if not line: # Separator between defconfigs | |
1681 | config_db[defconfig] = configs | |
1682 | all_defconfigs.add(defconfig) | |
1683 | configs = {} | |
1684 | elif line[0] == ' ': # CONFIG line | |
1685 | config, value = line.strip().split('=', 1) | |
1686 | configs[config] = value | |
1687 | defconfig_db[config].add(defconfig) | |
1688 | all_configs.add(config) | |
1689 | else: # New defconfig | |
1690 | defconfig = line | |
1691 | ||
1692 | # Work through each target config option in tern, independently | |
1693 | for config in config_list: | |
1694 | defconfigs = defconfig_db.get(config) | |
1695 | if not defconfigs: | |
1696 | print '%s not found in any defconfig' % config | |
1697 | continue | |
1698 | ||
1699 | # Get the set of defconfigs without this one (since a config cannot | |
1700 | # imply itself) | |
1701 | non_defconfigs = all_defconfigs - defconfigs | |
1702 | num_defconfigs = len(defconfigs) | |
1703 | print '%s found in %d/%d defconfigs' % (config, num_defconfigs, | |
1704 | len(all_configs)) | |
1705 | ||
1706 | # This will hold the results: key=config, value=defconfigs containing it | |
1707 | imply_configs = {} | |
1708 | rest_configs = all_configs - set([config]) | |
1709 | ||
1710 | # Look at every possible config, except the target one | |
1711 | for imply_config in rest_configs: | |
9b2a2e87 | 1712 | if 'ERRATUM' in imply_config: |
99b66605 | 1713 | continue |
9b2a2e87 SG |
1714 | if not (imply_flags & IMPLY_CMD): |
1715 | if 'CONFIG_CMD' in imply_config: | |
1716 | continue | |
1717 | if not (imply_flags & IMPLY_TARGET): | |
1718 | if 'CONFIG_TARGET' in imply_config: | |
1719 | continue | |
99b66605 SG |
1720 | |
1721 | # Find set of defconfigs that have this config | |
1722 | imply_defconfig = defconfig_db[imply_config] | |
1723 | ||
1724 | # Get the intersection of this with defconfigs containing the | |
1725 | # target config | |
1726 | common_defconfigs = imply_defconfig & defconfigs | |
1727 | ||
1728 | # Get the set of defconfigs containing this config which DO NOT | |
1729 | # also contain the taret config. If this set is non-empty it means | |
1730 | # that this config affects other defconfigs as well as (possibly) | |
1731 | # the ones affected by the target config. This means it implies | |
1732 | # things we don't want to imply. | |
1733 | not_common_defconfigs = imply_defconfig & non_defconfigs | |
1734 | if not_common_defconfigs: | |
1735 | continue | |
1736 | ||
1737 | # If there are common defconfigs, imply_config may be useful | |
1738 | if common_defconfigs: | |
1739 | skip = False | |
1740 | if find_superset: | |
1741 | for prev in imply_configs.keys(): | |
1742 | prev_count = len(imply_configs[prev]) | |
1743 | count = len(common_defconfigs) | |
1744 | if (prev_count > count and | |
1745 | (imply_configs[prev] & common_defconfigs == | |
1746 | common_defconfigs)): | |
1747 | # skip imply_config because prev is a superset | |
1748 | skip = True | |
1749 | break | |
1750 | elif count > prev_count: | |
1751 | # delete prev because imply_config is a superset | |
1752 | del imply_configs[prev] | |
1753 | if not skip: | |
1754 | imply_configs[imply_config] = common_defconfigs | |
1755 | ||
1756 | # Now we have a dict imply_configs of configs which imply each config | |
1757 | # The value of each dict item is the set of defconfigs containing that | |
1758 | # config. Rank them so that we print the configs that imply the largest | |
1759 | # number of defconfigs first. | |
cb008830 | 1760 | ranked_iconfigs = sorted(imply_configs, |
99b66605 | 1761 | key=lambda k: len(imply_configs[k]), reverse=True) |
cb008830 SG |
1762 | kconfig_info = '' |
1763 | cwd = os.getcwd() | |
1764 | add_list = collections.defaultdict(list) | |
1765 | for iconfig in ranked_iconfigs: | |
1766 | num_common = len(imply_configs[iconfig]) | |
99b66605 SG |
1767 | |
1768 | # Don't bother if there are less than 5 defconfigs affected. | |
9b2a2e87 | 1769 | if num_common < (2 if imply_flags & IMPLY_MIN_2 else 5): |
99b66605 | 1770 | continue |
cb008830 | 1771 | missing = defconfigs - imply_configs[iconfig] |
99b66605 SG |
1772 | missing_str = ', '.join(missing) if missing else 'all' |
1773 | missing_str = '' | |
cb008830 SG |
1774 | show = True |
1775 | if kconf: | |
1776 | sym = find_kconfig_rules(kconf, config[CONFIG_LEN:], | |
1777 | iconfig[CONFIG_LEN:]) | |
1778 | kconfig_info = '' | |
1779 | if sym: | |
1780 | locs = sym.get_def_locations() | |
1781 | if len(locs) == 1: | |
1782 | fname, linenum = locs[0] | |
1783 | if cwd and fname.startswith(cwd): | |
1784 | fname = fname[len(cwd) + 1:] | |
1785 | kconfig_info = '%s:%d' % (fname, linenum) | |
1786 | if skip_added: | |
1787 | show = False | |
1788 | else: | |
1789 | sym = kconf.get_symbol(iconfig[CONFIG_LEN:]) | |
1790 | fname = '' | |
1791 | if sym: | |
1792 | locs = sym.get_def_locations() | |
1793 | if len(locs) == 1: | |
1794 | fname, linenum = locs[0] | |
1795 | if cwd and fname.startswith(cwd): | |
1796 | fname = fname[len(cwd) + 1:] | |
1797 | in_arch_board = not sym or (fname.startswith('arch') or | |
1798 | fname.startswith('board')) | |
1799 | if (not in_arch_board and | |
1800 | not (imply_flags & IMPLY_NON_ARCH_BOARD)): | |
1801 | continue | |
1802 | ||
1803 | if add_imply and (add_imply == 'all' or | |
1804 | iconfig in add_imply): | |
1805 | fname, linenum, kconfig_info = (check_imply_rule(kconf, | |
1806 | config[CONFIG_LEN:], iconfig[CONFIG_LEN:])) | |
1807 | if fname: | |
1808 | add_list[fname].append(linenum) | |
1809 | ||
1810 | if show and kconfig_info != 'skip': | |
1811 | print '%5d : %-30s%-25s %s' % (num_common, iconfig.ljust(30), | |
1812 | kconfig_info, missing_str) | |
1813 | ||
1814 | # Having collected a list of things to add, now we add them. We process | |
1815 | # each file from the largest line number to the smallest so that | |
1816 | # earlier additions do not affect our line numbers. E.g. if we added an | |
1817 | # imply at line 20 it would change the position of each line after | |
1818 | # that. | |
1819 | for fname, linenums in add_list.iteritems(): | |
1820 | for linenum in sorted(linenums, reverse=True): | |
1821 | add_imply_rule(config[CONFIG_LEN:], fname, linenum) | |
99b66605 SG |
1822 | |
1823 | ||
5a27c734 MY |
1824 | def main(): |
1825 | try: | |
1826 | cpu_count = multiprocessing.cpu_count() | |
1827 | except NotImplementedError: | |
1828 | cpu_count = 1 | |
1829 | ||
1830 | parser = optparse.OptionParser() | |
1831 | # Add options here | |
cb008830 SG |
1832 | parser.add_option('-a', '--add-imply', type='string', default='', |
1833 | help='comma-separated list of CONFIG options to add ' | |
1834 | "an 'imply' statement to for the CONFIG in -i") | |
1835 | parser.add_option('-A', '--skip-added', action='store_true', default=False, | |
1836 | help="don't show options which are already marked as " | |
1837 | 'implying others') | |
d73fcb12 SG |
1838 | parser.add_option('-b', '--build-db', action='store_true', default=False, |
1839 | help='build a CONFIG database') | |
5a27c734 MY |
1840 | parser.add_option('-c', '--color', action='store_true', default=False, |
1841 | help='display the log in color') | |
9ede2123 SG |
1842 | parser.add_option('-C', '--commit', action='store_true', default=False, |
1843 | help='Create a git commit for the operation') | |
91040e85 | 1844 | parser.add_option('-d', '--defconfigs', type='string', |
ee4e61bd SG |
1845 | help='a file containing a list of defconfigs to move, ' |
1846 | "one per line (for example 'snow_defconfig') " | |
1847 | "or '-' to read from stdin") | |
99b66605 SG |
1848 | parser.add_option('-i', '--imply', action='store_true', default=False, |
1849 | help='find options which imply others') | |
9b2a2e87 SG |
1850 | parser.add_option('-I', '--imply-flags', type='string', default='', |
1851 | help="control the -i option ('help' for help") | |
5a27c734 MY |
1852 | parser.add_option('-n', '--dry-run', action='store_true', default=False, |
1853 | help='perform a trial run (show log with no changes)') | |
1854 | parser.add_option('-e', '--exit-on-error', action='store_true', | |
1855 | default=False, | |
1856 | help='exit immediately on any error') | |
8513dc04 MY |
1857 | parser.add_option('-s', '--force-sync', action='store_true', default=False, |
1858 | help='force sync by savedefconfig') | |
07913d1e MY |
1859 | parser.add_option('-S', '--spl', action='store_true', default=False, |
1860 | help='parse config options defined for SPL build') | |
2144f880 JH |
1861 | parser.add_option('-H', '--headers-only', dest='cleanup_headers_only', |
1862 | action='store_true', default=False, | |
1863 | help='only cleanup the headers') | |
5a27c734 MY |
1864 | parser.add_option('-j', '--jobs', type='int', default=cpu_count, |
1865 | help='the number of jobs to run simultaneously') | |
6b96c1a1 JH |
1866 | parser.add_option('-r', '--git-ref', type='string', |
1867 | help='the git ref to clone for building the autoconf.mk') | |
6b403dfd SG |
1868 | parser.add_option('-y', '--yes', action='store_true', default=False, |
1869 | help="respond 'yes' to any prompts") | |
95bf9c7e JH |
1870 | parser.add_option('-v', '--verbose', action='store_true', default=False, |
1871 | help='show any build errors as boards are built') | |
b6ef393a | 1872 | parser.usage += ' CONFIG ...' |
5a27c734 | 1873 | |
b6ef393a | 1874 | (options, configs) = parser.parse_args() |
5a27c734 | 1875 | |
99b66605 SG |
1876 | if len(configs) == 0 and not any((options.force_sync, options.build_db, |
1877 | options.imply)): | |
5a27c734 MY |
1878 | parser.print_usage() |
1879 | sys.exit(1) | |
1880 | ||
b6ef393a MY |
1881 | # prefix the option name with CONFIG_ if missing |
1882 | configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config | |
1883 | for config in configs ] | |
5a27c734 | 1884 | |
2144f880 JH |
1885 | check_top_directory() |
1886 | ||
99b66605 | 1887 | if options.imply: |
9b2a2e87 SG |
1888 | imply_flags = 0 |
1889 | for flag in options.imply_flags.split(): | |
1890 | if flag == 'help' or flag not in IMPLY_FLAGS: | |
1891 | print "Imply flags: (separate with ',')" | |
1892 | for name, info in IMPLY_FLAGS.iteritems(): | |
1893 | print ' %-15s: %s' % (name, info[1]) | |
1894 | parser.print_usage() | |
1895 | sys.exit(1) | |
1896 | imply_flags |= IMPLY_FLAGS[flag][0] | |
1897 | ||
cb008830 SG |
1898 | do_imply_config(configs, options.add_imply, imply_flags, |
1899 | options.skip_added) | |
99b66605 SG |
1900 | return |
1901 | ||
d73fcb12 SG |
1902 | config_db = {} |
1903 | db_queue = Queue.Queue() | |
1904 | t = DatabaseThread(config_db, db_queue) | |
1905 | t.setDaemon(True) | |
1906 | t.start() | |
1907 | ||
2144f880 | 1908 | if not options.cleanup_headers_only: |
f7536f79 MY |
1909 | check_clean_directory() |
1910 | update_cross_compile(options.color) | |
d73fcb12 SG |
1911 | move_config(configs, options, db_queue) |
1912 | db_queue.join() | |
2144f880 | 1913 | |
6a9f79f7 | 1914 | if configs: |
e9ea1221 | 1915 | cleanup_headers(configs, options) |
9ab0296a | 1916 | cleanup_extra_options(configs, options) |
ca43834d | 1917 | cleanup_whitelist(configs, options) |
f90df596 | 1918 | cleanup_readme(configs, options) |
5a27c734 | 1919 | |
9ede2123 SG |
1920 | if options.commit: |
1921 | subprocess.call(['git', 'add', '-u']) | |
1922 | if configs: | |
1923 | msg = 'Convert %s %sto Kconfig' % (configs[0], | |
1924 | 'et al ' if len(configs) > 1 else '') | |
1925 | msg += ('\n\nThis converts the following to Kconfig:\n %s\n' % | |
1926 | '\n '.join(configs)) | |
1927 | else: | |
1928 | msg = 'configs: Resync with savedefconfig' | |
1929 | msg += '\n\nRsync all defconfig files using moveconfig.py' | |
1930 | subprocess.call(['git', 'commit', '-s', '-m', msg]) | |
1931 | ||
d73fcb12 SG |
1932 | if options.build_db: |
1933 | with open(CONFIG_DATABASE, 'w') as fd: | |
1934 | for defconfig, configs in config_db.iteritems(): | |
1935 | print >>fd, '%s' % defconfig | |
1936 | for config in sorted(configs.keys()): | |
1937 | print >>fd, ' %s=%s' % (config, configs[config]) | |
1938 | print >>fd | |
1939 | ||
5a27c734 MY |
1940 | if __name__ == '__main__': |
1941 | main() |