lto: Overhaul approach to location streaming [PR65536]
PR65536 is about location-related issues that arise when the LTO front end
reads one or several large object files and runs out of location_t space to
encode all the locations. The main reason for the potential problem is that
the libcpp linemap is designed to be used for incrementally reading source
files (and their included header files) in the natural order. When it is
used in another way, such as by LTO, which adds locations in the order in
which it happened to read different entities, then the assumptions that
justify its design are no longer applicable, and it is not hard to run out
of location_t values (e.g., because of a large number of file changes, or
lines being added out of order.)
The PR remains open because it is not theoretically resolved, but it has
been resolved for all practical purposes by the following two improvements:
1) The lto_location_cache class now tries hard to optimize the number of
maps that it creates, especially by sorting the locations before
adding them. This approach only goes so far, because it can only work
at the LTO section level, so each function is handled independently
of the others, but it helps a lot with reducing the number of maps
required for a single function.
2) We moved to 64-bit location_t, which means everything is a lot more
forgiving of wasting location_t space.
Point 2) made this not much of an issue in practice, although sufficiently
large files (especially with very long lines) could still trigger a problem
in theory. While locations are now by and large working fine in LTO, there
was an interesting discussion on PR65536, starting around:
about how the location streaming might be optimized by streaming the linemap
structure itself, rather than a (file name, line number, column number)
triplet for each location. In the end, this approach was not taken because
solution 1) above was adequate and less disruptive. But the arguments
presented in favor of streaming the linemap directly are still interesting.
That discussion did not touch on another topic, namely the need to make
`#pragma GCC diagnostic' usable in LTO. Right now, it does not work, because
the diagnostic pragmas are not streamed out. But even if they were streamed
out, the existing approach to locations in the LTO front end is not
compatible with enforcing the pragmas. Diagnostic pragmas require a global
ordering on location_t values, so that the machinery in option-classifier.cc
can determine which pragmas are in force at each location. This requires
remembering the order in which each source line was encountered, and that is
exactly what is encoded in the linemap. This cannot be reconstructed just
from source file names and line numbers; consider, for example, that the
same file might be used in different translation units or multiple times in
the same translation unit with different diagnostic pragmas in place each
time.
These considerations tip the scales in favor of adding linemap streaming as
Manuel proposed on the PR. This patch prepares to support `#pragma GCC
diagnostic' in the LTO front end by changing the approach to location
streaming along these lines. The new approach has these general
characteristics:
o There is a new LTO section (LTO_section_linemap) that contains the
information needed to reconstruct the linemap. There is an entry for
each line_map_ordinary object that was used by at least one
streamed-out location. When the LTO front end reads one of these map
entries, it adds a new map with the corresponding properties to its
own linemap using the new line_map_add_raw_map interface in libcpp.
o When a location needs to be output, we stream out two integers: one to
identify which linemap contains the location, and one containing the
offset from the start of that map to the location.
o When the language front ends stream out their data, they produce a
single linemap section (labeled as linemap.0) that applies to the
whole translation unit. When WPA prepares partitioned files for
LTRANS, it may copy function bodies into the LTRANS files without
reading them, so it needs also to copy the linemap sections they refer
to. Since the same linemap section will often be needed by more than
one partition, this is done by putting all of the linemap sections
into one additional object file, which is provided as input to each
LTRANS process via the new option -fltrans-linemap-file.
o The naming of the LTRANS linemap sections is stable so as not to
inhibit incremental LTO.
o Once the reader has processed the linemap sections, there is no
further overhead associated with inputting a location, unless it is an
adhoc location for the purpose of associating a discriminator or a
tree with the location. For the adhoc location case, the existing
lto_location_cache setup is still useful to avoid creating unneeded
adhoc locations, so this has been left in place as before. The
restriction that there be only one lto_location_cache at a time is no
longer strictly necessary; but it is still useful so that the IPA
passes can access the currently active one simply, so I have not
changed this for now.
With this new setup, PR65536 can be closed. The subsequent patches in this
series will enable support for diagnostic pragmas.
gcc/lto/ChangeLog:
PR lto/65536
* lang.opt: Add -fltrans-linemap-file.
* lto-common.cc (lto_read_in_decl_state): Read the linemap ID from
function sections.
(linemap_section_id): New function.
(create_subid_section_table): Note how many linemap sections were
found in each input section.
(loc_map_decl_data): New GC root.
(lto_file_read): Make ORDER into a static variable so it counts
continuously across all files and sub-files provided on the command
line.
(lto_file_finalize): Handle reading the new LTRANS linemap file.
(read_cgraph_and_symbols): Likewise. Also stop freeing
ALL_FILE_DECL_DATA at the end; the ordered list of files is now
useful later on for lto_copy_linemaps().
* lto.cc (stream_out_linemaps): New function.
(lto_wpa_write_files): Stream out the linemaps for LTRANS to use.
gcc/testsuite/ChangeLog:
PR lto/65536
* gcc.misc-tests/outputs.exp: Adjust LTO -save-temps tests to expect
the new linemap file.
gcc/ChangeLog:
PR lto/65536
* doc/lto.texi: Document the new LTO_section_linemap and the new
option -fltrans-linemap-file.
* lto-opts.cc (lto_write_options): Handle the new option.
* opts.cc (gen_command_line_string): Likewise.
* lto-section-in.cc (lto_section_name[]): Add new name for
LTO_section_linemap.
* lto-streamer-in.cc (get_location_from_idx): New function.
(lto_location_cache::cmp_loc): Remove.
(bp_unpack_delta): New function.
(create_loc_map): New function.
(get_loc_map): New function.
(lto_location_cache::override_loc_map): New function.
(lto_location_cache::apply_location_cache): Pervasive changes to
implement new location streaming format.
(lto_location_cache::input_location_and_block): Likewise.
(lto_location_cache::input_location): Rename argument LOC -> DEST
for clarity.
(lto_read_body_or_constructor): Handle LINEMAP_ID in the decl state.
(lto_data_in_create): Add NEED_LOCATION_CACHE argument. Rename local
variable DATA_IN to D to avoid clash with the type name.
* lto-streamer-out.cc (clear_line_info): Update for change to class
output_block.
(compute_map_hash): New function.
(class location_output): New class.
(location_output::record_location): New function.
(bp_pack_delta): New function.
(location_output::produce_linemap_section): New function.
(lto_output_location_1): Pervasive changes to implement new location
streaming format.
(copy_function_or_variable): Set the LINEMAP_ID in the decl state.
(copy_linemap_section): New function.
(copy_linemap_sections): New function.
(lto_copy_linemaps): New function.
(lto_register_linemap_for_output): New function.
(lto_output_decl_state_refs): Output the LINEMAP_ID for function
decls.
(lto_out_decl_state_written_size): Adapt for new LINEMAP_ID output.
(produce_asm_for_decls): Output the linemap section when needed.
* lto-streamer.cc (lto_get_section_name): Handle linemap sections,
which need an order suffix.
* lto-streamer.h (enum lto_section_type): Fix typo in the
comment. Add LTO_section_linemap.
(struct lto_loc_map): New struct.
(class lto_location_cache): Pervasive changes to implement new
location streaming format.
(struct lto_in_decl_state): Add LINEMAP_ID member.
(struct lto_out_decl_state): Likewise.
(struct lto_file_decl_data): Add LOC_MAP_DECL_DATA, LOC_MAPS,
and NUM_LINEMAP_SECTIONS members.
(lto_linemap_output_id): New function.
(struct output_block): Adjust members for new streaming format.
(data_in::data_in): New function.
(lto_data_in_create): Adjust prototype for new NEED_LOCATION_CACHE
argument.
(lto_register_linemap_for_output): Declare.
(lto_copy_linemaps): Declare.
* lto-wrapper.cc (run_gcc): Pass new argument -fltrans-linemap-file.
* timevar.def (TV_IPA_LTO_LINEMAP_IN): New timevar.
(TV_IPA_LTO_LINEMAP_OUT): New timevar.
(TV_IPA_LTO_LINEMAP_COPY): New timevar.
projects / thirdparty / gcc.git / commit
