Implement Morello TLS relaxations

The majority of the code change here is around TLS data stubs.  The new
TLS ABI requires that when relaxing a General Dynamic or Initial Exec
access to a variable to a Local Exec access, the linker emits data stubs
in a read-only section.

We do this with the below approach:
  - check_relocs notices that we need TLS data stubs by recognising that
    some relocation will need to be relaxed to a local-exec relocation.
  - check_relocs then records a hash table entry mapping the symbol that
    we are relocating against to a position in some data stub section.
    It also ensures that this data stub section has been created, and
    increments our data stub section size.
  - This section is placed in the resulting binary using the standard
    subsection and wildcard matching implemented by the generic linker.
  - In elfNN_aarch64_size_dynamic_sections we allocate the actual buffer
    for our data stub section.
  - When it comes to actually relaxing the TLS sequence,
    relocate_section directly populates the data stub using the address
    and size of the TLS object that has already been calculated, it then
    uses final_link_relocate to handle adjusting the text so that it
    points to this data stub.

Notes on implementation:

Mechanism by which we create and populate a TLS data section:
  There are currently three different ways by which the AArch64 backend
  creates and populates a special section.  These are
    - The method by which the .got (and related sections) are populated.
    - The method by which the interworking stubs are populated.
    - The method by which erratum 843419 stub sections are populated.

  We have gone with an approach that mostly follows that used to
  populate the .got.  Here we give an outline of the approaches and
  provide the reasoning by which the approach used by the .got was
  chosen.

  Handling the .got section:
  - Create a section on an existing BFD.
  - Mark that section as SEC_LINKER_CREATED.
  - Record the existing BFD as the `dynobj`.
  - bfd_elf_final_link still calls elf_link_input_bfd on the object.
  - elf_link_input_bfd avoids emitting the section (because of
    SEC_LINKER_CREATED).
  - bfd_elf_final_link then emits the special sections on `dynobj` after
    all the non-special sections on all objects have been relocated.
  - Allows updating the .got input section in relocate_section &
    final_link_relocate knowing that its contents will be output once all
    relocations on standard input sections have been processed.

   Handling interworking stub sections.
   - Create a special stub file.
   - Create sections on that stub file for each input section we need a
     stub for.
   - Manually populate the sections in build_stubs (which is called
     through `ldemul_finish` *before* `ldwrite` and hence before any
     other files are relocated).

   Handling erratum 843419 stub sections.
   - Create a special stub file.
   - Create sections on that stub file for each input section we need a
     stub for.
   - Ensure that the stub file is marked with class ELFCLASSNONE.
   - Ensure that the list of input sections for the relevant output
     section statement has the veneered input section *directly before*
     the stub section which has the veneer.
   - When relocating and outputting sections, having ELFCLASSNONE means
     that we output sections on the stub_file only when we see the
     corresponding input statement.  Without that class marker
     bfd_elf_final_link calls elf_link_input_bfd which writes out the
     data for all input sections on the relevant BFD.
   - Since we have ensured the input statement for our stub section is
     directly after the input statement for the section we are emitting
     veneers for, we know that the veneered section will be relocated
     and output before we output our stub section.
   - Hence we can copy relocated data from the veneered section into our
     stub section and know that our stub section will be output after
     this modification has been made.

  In deciding what to do with the read-only TLS data stubs we noticed
  the following problems with each approach:
  - The ABI requires that the read-only TLS data stubs are emitted into
    a read-only section.  This will necessarily be a different output
    section to .text where the requirement for these stubs is found.
    The temporal order in which output sections are written to the
    output file is tied to the order in which the in-memory linker
    statements are kept, and that is tied to the linker script provided
    by the user.  Hence we can not rely on ordering and ELFCLASSNONE to
    ensure that our data stub section is emitted after the relevant TLS
    sequences have been relaxed.  (We need to know our data stub section
    is written to the output after we have populated it as otherwise the
    data would not propagate to the resulting binary).
  - I think it is easier and simpler to find the data needed for the TLS
    data stubs in relocate_section just as we relax the relevant TLS
    sequences.  Hence I don't want to use the approach used for
    interworking stubs of populating the entire section beforehand.
  - Adding a section to `dynobj` would mean that we're adding a section
    to a user input BFD, which is not quite as clear as having a
    separate BFD for our special stub section.  It also means we treat
    this particular section as a "dynamic" section.  This is a little
    confusing nomenclature-wise.

  Based on the above trade-offs we chose the .got approach (accepting
  the negative that this will be stored on a user BFD).  N.b. using the
  .got approach and requiring the section get allocated in
  `size_dynamic_sections` is not problematic for static executables
  despite the nomenclature.  This function always gets called.

  One difference between how we handle the data stubs and how the .got
  is handled is that we do not count the number of data stubs required
  in size_dynamic_sections, but rather total it as we see relocations
  needing these stubs in check_relocs.
  We do this largely to avoid requiring another data member on all
  symbols to indicate information about whether this symbol needs a data
  stub and where that data stub is.  The number of TLS symbols are
  expected to be much smaller than the number of symbols with an entry
  in the GOT and hence a separate hash table just containing entries for
  those symbols which need such information is likely to often be
  smaller.

  N.b. it is interesting to mention that for all relocations which need
  a data stub we would make an input section on `dynobj` in
  `check_relocs` if that relaxation were not performed.  This is since
  if we did not realise they could be relaxed these relocations would
  have needed a .got entry.

  N.b. we must use make_section_anyway to create our TLS data stubs
  section in order to avoid any problems with our linker defined section
  having the same name as a section already defined by the user.

We do not use local stub symbols:
  The TLS ABI describes data stubs using specially named symbols.  These
  are not part of the ABI.  We could have associated the position of a
  data stub with a particular symbol by generating a symbol internally
  using some name mangling scheme that matches that in the TLS ABI
  examples and points to the data stub for a particular symbol.  We take
  the current approach on the belief that it is "neater" to avoid
  relying on such a name-mangling scheme and the associated sprintf
  calls.

final_link_relocate handling the adjusted relocation for data stubs:
  final_link_relocate does not actually use the `h` or `sym` arguments
  to decide anything for the two relocations we need to handle once we
  have relaxed an IE or GD TLS access sequence to a LE one.

  The relocations we need are BFD_RELOC_MORELLO_ADR_HI20_PCREL and
  BFD_RELOC_AARCH64_ADD_LO12.  For both (and in fact for most
  relocations) we only use `h` and `sym` for catching and reporting
  errors.

  This means we don't actually have to update the `h` and/or `sym`
  variables before calling elfNN_aarch64_final_link_relocate.

Allocate TLS data stubs on dynobj
  This uses the same approach that the linker uses to ensure that the
  .got sections are emitted after all relocations have been processed.

  elf_link_input_bfd avoids sections with SEC_LINKER_CREATED, and
  bfd_elf_final_link emits all SEC_LINKER_CREATED sections on the dynobj
  *after* standard sections have been relocated.

  This means that we can populate the contents of the TLS data stub
  section while performing relocations on all our other sections (i.e.
  in the same place as we perform the relaxations on the TLS sequences
  that we recognise need these data stubs).

Assert that copy_indirect_symbol is not a problem
  copy_indirect_symbol takes information from one symbol and puts it
  onto another.  The point is to ensure that any symbol which simply
  refers to another has all its cached information on that symbol to
  which it refers rather than itself.  If we could ever call this
  function on a symbol which we have found needs an associated data stub
  created, then we could have to handle adjusting the hash table
  associating a symbol with a data stub.  We do not believe this is
  needed, and add an assert instead.

  The proof that this is not a problem is a little tricky.  However it
  *shouldn't* be a problem given what it's handling.  This is handling
  moving cached information from an indirected symbol to the symbol it
  represents.  That is needed when the information was originally put on
  the indirected symbol, and that happens when the indirection was
  originally the other way around.  The two ways that this reversal of
  indirection can happen is through resolving dynamic weak symbols and
  versioned symbols.  Both of these are not something we can see with
  SYMBOL_REFERENCES_LOCAL TLS symbols (see below).

  We only need to worry about copy_indirect_symbol transferring
  information *from* a symbol which we have generated a TLS relaxation
  against to LE.

  In order to satisfy the criteria that we have generated a TLS
  relaxation hash entry against a symbol, we must have already have run
  check_relocs.  This means that of the ways in which
  copy_indirect_symbol can be called we have eliminated all but
  _bfd_elf_fix_symbol_flags and bfd_elf_record_link_assignment.

  bfd_elf_record_link_assignment handles symbol assignments in a linker
  script.  Such assignments can not be made on TLS symbols (we end up
  generating a non-TLS symbol).

  _bfd_elf_fix_symbol_flags only calls copy_indirect_symbol on symbols
  which have is_weakalias set.  These are symbols "from a dynamic
  object", and we only ever call the hook when the real definition is in
  a non-shared object.  Hence we would not have performed this
  relaxation on the symbol (because it is not SYMBOL_REFERENCES_LOCAL).

  Hence I don't believe this is something that we can trigger and we add
  an assertion here rather than add code to handle the case.

diff --git a/bfd/elfnn-aarch64.c b/bfd/elfnn-aarch64.c
index 7e951a5da73c3f427bbcf29093dda521e86d5e75..ac357c4fa82c910892fe1c46a91f173883c19df8 100644 (file)
--- a/bfd/elfnn-aarch64.c
+++ b/bfd/elfnn-aarch64.c
@@ -341,7 +341,8 @@
    relocation for it when we can get an untagged zero capability by just
    loading some zeros.  */
 #define c64_needs_relocation(info, h) \
-    (!((h)->root.type == bfd_link_hash_undefweak \
+    (!((h) \
+       && (h)->root.type == bfd_link_hash_undefweak \
        && (UNDEFWEAK_NO_DYNAMIC_RELOC ((info), (h)) \
           || !elf_hash_table ((info))->dynamic_sections_created)))
 
@@ -3032,6 +3033,19 @@ struct elf_aarch64_stub_hash_entry
   bfd_vma adrp_offset;
 };
 
+struct elf_c64_tls_data_stub_hash_entry
+{
+  /* Symbol table entry, if any, for which this stub is made.  */
+  struct elf_aarch64_link_hash_entry *h;
+  /* Local symbol table index and BFD associated with it, if required, for
+     which this stub is made.  These are only used when `h` is NULL.  */
+  unsigned long r_symndx;
+  bfd *input_bfd;
+  /* Offset within htab->sc64_tls_stubs of the beginning of this stub.  */
+  bfd_vma tls_stub_offset;
+  bfd_boolean populated;
+};
+
 /* Used to build a map of a section.  This is required for mixed-endian
    code/data.  */
 
@@ -3327,8 +3341,11 @@ struct elf_aarch64_link_hash_table
 
   /* Used for capability relocations.  */
   asection *srelcaps;
+  asection *sc64_tls_stubs;
   int c64_rel;
   bfd_boolean c64_output;
+  htab_t c64_tls_data_stub_hash_table;
+  void * tls_data_stub_memory;
 };
 
 /* Create an entry in an AArch64 ELF linker hash table.  */
@@ -3402,6 +3419,119 @@ stub_hash_newfunc (struct bfd_hash_entry *entry,
   return entry;
 }
 
+static hashval_t
+c64_tls_data_stub_hash (const void *ptr)
+{
+  struct elf_c64_tls_data_stub_hash_entry *entry
+    = (struct elf_c64_tls_data_stub_hash_entry *) ptr;
+  if (entry->h)
+    return htab_hash_pointer (entry->h);
+  struct elf_aarch64_local_symbol *l;
+  l = elf_aarch64_locals (entry->input_bfd);
+  return htab_hash_pointer (l + entry->r_symndx);
+}
+
+static int
+c64_tls_data_stub_eq (const void *ptr1, const void *ptr2)
+{
+  struct elf_c64_tls_data_stub_hash_entry *entry1
+     = (struct elf_c64_tls_data_stub_hash_entry *) ptr1;
+  struct elf_c64_tls_data_stub_hash_entry *entry2
+    = (struct elf_c64_tls_data_stub_hash_entry *) ptr2;
+
+  if (entry1->h && !entry2->h)
+    return 0;
+  if (!entry1->h && entry2->h)
+    return 0;
+  if (entry1->h && entry2->h)
+    return entry1->h == entry2->h;
+
+  if (entry1->input_bfd != entry2->input_bfd)
+    return 0;
+  return entry1->r_symndx == entry2->r_symndx;
+}
+
+static bfd_boolean
+c64_record_tls_stub (struct elf_aarch64_link_hash_table *htab,
+                    bfd *input_bfd,
+                    struct elf_link_hash_entry *h,
+                    unsigned long r_symndx)
+{
+  if (htab->root.dynobj == NULL)
+    htab->root.dynobj = input_bfd;
+
+  if (!htab->sc64_tls_stubs)
+    {
+      asection *stub_sec;
+      flagword flags;
+
+      flags = (SEC_ALLOC | SEC_LOAD | SEC_READONLY
+              | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_KEEP
+              | SEC_LINKER_CREATED);
+      /* Using the .rodata section is ABI -- N.b. I do have an outstanding
+        clarification on this on the ABI PR.
+        https://github.com/ARM-software/abi-aa/pull/80  */
+      stub_sec = bfd_make_section_anyway_with_flags
+                   (htab->root.dynobj, ".rodata", flags);
+      if (stub_sec == NULL)
+       return FALSE;
+
+      /* Section contains stubs of 64 bit values, hence requires 8 byte
+        alignment.  */
+      bfd_set_section_alignment (stub_sec, 3);
+      htab->sc64_tls_stubs = stub_sec;
+      BFD_ASSERT (htab->sc64_tls_stubs->size == 0);
+    }
+
+  if (h)
+    BFD_ASSERT (h->type == STT_TLS);
+
+  void **slot;
+  struct elf_c64_tls_data_stub_hash_entry e, *new_entry;
+  e.h = (struct elf_aarch64_link_hash_entry *)h;
+  e.r_symndx = r_symndx;
+  e.input_bfd = input_bfd;
+
+  slot = htab_find_slot (htab->c64_tls_data_stub_hash_table, &e, INSERT);
+  if (!slot)
+    return FALSE;
+
+  if (*slot)
+    return TRUE;
+  new_entry = (struct elf_c64_tls_data_stub_hash_entry *)
+    objalloc_alloc ((struct objalloc *) htab->tls_data_stub_memory,
+                   sizeof (struct elf_c64_tls_data_stub_hash_entry));
+  if (new_entry)
+    {
+      new_entry->h = (struct elf_aarch64_link_hash_entry *)h;
+      new_entry->r_symndx = r_symndx;
+      new_entry->input_bfd = input_bfd;
+      new_entry->tls_stub_offset = htab->sc64_tls_stubs->size;
+      new_entry->populated = FALSE;
+      /* Size of a Morello data stub is 2 * 8 byte integers.  */
+      htab->sc64_tls_stubs->size += 16;
+      *slot = new_entry;
+      return TRUE;
+    }
+  return FALSE;
+}
+
+static struct elf_c64_tls_data_stub_hash_entry *
+c64_tls_stub_find (struct elf_link_hash_entry *h,
+                  bfd *input_bfd,
+                  unsigned long r_symndx,
+                  struct elf_aarch64_link_hash_table *htab)
+{
+  void *ret;
+  struct elf_c64_tls_data_stub_hash_entry e;
+  e.h = (struct elf_aarch64_link_hash_entry *)h;
+  e.r_symndx = r_symndx;
+  e.input_bfd = input_bfd;
+
+  ret = htab_find (htab->c64_tls_data_stub_hash_table, &e);
+  return (struct elf_c64_tls_data_stub_hash_entry *)ret;
+}
+
 /* Compute a hash of a local hash entry.  We use elf_link_hash_entry
   for local symbol so that we can handle local STT_GNU_IFUNC symbols
   as global symbol.  We reuse indx and dynstr_index for local symbol
@@ -3491,6 +3621,13 @@ elfNN_aarch64_copy_indirect_symbol (struct bfd_link_info *info,
        }
     }
 
+  /* This function is called to take the indirect symbol information from eind
+     and put it onto edir.  We never create a TLS data stub for a symbol which
+     needs such a transformation, hence there is no need to worry about
+     removing the hash entry for the indirected symbol and ensuring there is
+     now one for the final one.  */
+  BFD_ASSERT (!c64_tls_stub_find (ind, NULL, 0,
+                                 elf_aarch64_hash_table (info)));
   _bfd_elf_link_hash_copy_indirect (info, dir, ind);
 }
 
@@ -3532,6 +3669,11 @@ elfNN_aarch64_link_hash_table_free (bfd *obfd)
   if (ret->loc_hash_memory)
     objalloc_free ((struct objalloc *) ret->loc_hash_memory);
 
+  if (ret->c64_tls_data_stub_hash_table)
+    htab_delete (ret->c64_tls_data_stub_hash_table);
+  if (ret->tls_data_stub_memory)
+    objalloc_free ((struct objalloc *) ret->tls_data_stub_memory);
+
   bfd_hash_table_free (&ret->stub_hash_table);
   _bfd_elf_link_hash_table_free (obfd);
 }
@@ -3571,6 +3713,16 @@ elfNN_aarch64_link_hash_table_create (bfd *abfd)
       return NULL;
     }
 
+  ret->c64_tls_data_stub_hash_table
+    = htab_try_create (256, c64_tls_data_stub_hash, c64_tls_data_stub_eq,
+                      NULL);
+  ret->tls_data_stub_memory = objalloc_create ();
+  if (!ret->c64_tls_data_stub_hash_table || !ret->tls_data_stub_memory)
+    {
+      elfNN_aarch64_link_hash_table_free (abfd);
+      return NULL;
+    }
+
   ret->loc_hash_table = htab_try_create (1024,
                                         elfNN_aarch64_local_htab_hash,
                                         elfNN_aarch64_local_htab_eq,
@@ -5992,18 +6144,13 @@ static bfd_reloc_code_real_type
 aarch64_tls_transition_without_check (bfd_reloc_code_real_type r_type,
                                      struct bfd_link_info *info,
                                      struct elf_link_hash_entry *h,
-                                     bfd_boolean morello_reloc)
+                                     bfd_boolean *requires_c64_tls_stub)
 {
   bfd_boolean local_exec = (bfd_link_executable (info)
                            && TLS_SYMBOL_REFERENCES_LOCAL (info, h));
 
   switch (r_type)
     {
-    case BFD_RELOC_MORELLO_TLSDESC_ADR_PAGE20:
-      return (local_exec
-             ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G1
-             : r_type);
-
     case BFD_RELOC_AARCH64_TLSDESC_ADR_PAGE21:
     case BFD_RELOC_AARCH64_TLSGD_ADR_PAGE21:
       return (local_exec
@@ -6035,16 +6182,24 @@ aarch64_tls_transition_without_check (bfd_reloc_code_real_type r_type,
              ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G2
              : BFD_RELOC_AARCH64_TLSIE_MOVW_GOTTPREL_G1);
 
-    case BFD_RELOC_MORELLO_TLSDESC_LD128_LO12:
-      return (local_exec
-             ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G0_NC : r_type);
-
     case BFD_RELOC_AARCH64_TLSDESC_LDNN_LO12_NC:
     case BFD_RELOC_AARCH64_TLSGD_ADD_LO12_NC:
       return (local_exec
              ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G0_NC
              : BFD_RELOC_AARCH64_TLSIE_LDNN_GOTTPREL_LO12_NC);
 
+    case BFD_RELOC_MORELLO_TLSIE_ADR_GOTTPREL_PAGE20:
+      if (!local_exec)
+       return r_type;
+      *requires_c64_tls_stub = TRUE;
+      return BFD_RELOC_MORELLO_ADR_HI20_PCREL;
+
+    case BFD_RELOC_MORELLO_TLSIE_ADD_LO12:
+      if (!local_exec)
+       return r_type;
+      *requires_c64_tls_stub = TRUE;
+      return BFD_RELOC_AARCH64_ADD_LO12;
+
     case BFD_RELOC_AARCH64_TLSIE_ADR_GOTTPREL_PAGE21:
       return local_exec ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G1 : r_type;
 
@@ -6059,19 +6214,32 @@ aarch64_tls_transition_without_check (bfd_reloc_code_real_type r_type,
              ? BFD_RELOC_AARCH64_TLSLE_ADD_TPREL_HI12
              : BFD_RELOC_AARCH64_TLSIE_LD_GOTTPREL_PREL19);
 
+    case BFD_RELOC_MORELLO_TLSDESC_ADR_PAGE20:
+      if (local_exec)
+       {
+         *requires_c64_tls_stub = TRUE;
+         return BFD_RELOC_MORELLO_ADR_HI20_PCREL;
+       }
+      return BFD_RELOC_MORELLO_TLSIE_ADR_GOTTPREL_PAGE20;
+
+    case BFD_RELOC_MORELLO_TLSDESC_LD128_LO12:
+      if (local_exec)
+       {
+         *requires_c64_tls_stub = TRUE;
+         return BFD_RELOC_AARCH64_ADD_LO12;
+       }
+      return BFD_RELOC_MORELLO_TLSIE_ADD_LO12;
+
     case BFD_RELOC_MORELLO_TLSDESC_CALL:
-      return (local_exec
-             ? BFD_RELOC_AARCH64_NONE : r_type);
+      /* Instructions with this relocation will be fully resolved during the
+        transition into an add and scbnds pair.  */
+      return BFD_RELOC_AARCH64_NONE;
 
     case BFD_RELOC_AARCH64_TLSDESC_ADD_LO12:
-      if (morello_reloc && !local_exec)
-       return r_type;
-      /* Fall through.  */
     case BFD_RELOC_AARCH64_TLSDESC_ADD:
     case BFD_RELOC_AARCH64_TLSDESC_CALL:
       /* Instructions with these relocations will be fully resolved during the
-         transition into either a NOP in the A64 case or movk and add in
-        C64.  */
+        transition into either a NOP in the A64 case or an ldp in C64.  */
       return BFD_RELOC_AARCH64_NONE;
 
     case BFD_RELOC_AARCH64_TLSLD_ADD_LO12_NC:
@@ -6200,8 +6368,12 @@ aarch64_tls_transition (bfd *input_bfd,
                        struct bfd_link_info *info,
                        const Elf_Internal_Rela *rel,
                        struct elf_link_hash_entry *h,
-                       unsigned long r_symndx)
+                       unsigned long r_symndx,
+                       bfd_boolean *requires_c64_tls_stub)
 {
+  /* Initialisation done here.  The set to TRUE is done in
+     aarch64_tls_transition_without_check if necessary.  */
+  *requires_c64_tls_stub = FALSE;
   bfd_reloc_code_real_type bfd_r_type
     = elfNN_aarch64_bfd_reloc_from_type (input_bfd,
                                         ELFNN_R_TYPE (rel->r_info));
@@ -6209,12 +6381,8 @@ aarch64_tls_transition (bfd *input_bfd,
   if (! aarch64_can_relax_tls (input_bfd, info, rel, h, r_symndx))
     return bfd_r_type;
 
-  bfd_boolean morello_reloc = (bfd_r_type == BFD_RELOC_AARCH64_TLSDESC_ADD_LO12
-                              && (ELFNN_R_TYPE (rel[1].r_info)
-                                  == MORELLO_R (TLSDESC_CALL)));
-
   return aarch64_tls_transition_without_check (bfd_r_type, info, h,
-                                              morello_reloc);
+                                              requires_c64_tls_stub);
 }
 
 /* Return the base VMA address which should be subtracted from real addresses
@@ -7841,7 +8009,7 @@ elfNN_aarch64_final_link_relocate (reloc_howto_type *howto,
 
          outrel.r_addend = signed_addend;
 
-         if (h && !c64_needs_relocation (info, h))
+         if (!c64_needs_relocation (info, h))
            {
              /* If we know this symbol does not need a C64 dynamic relocation
                 then it must be because this is an undefined weak symbol which
@@ -7952,6 +8120,12 @@ elfNN_aarch64_final_link_relocate (reloc_howto_type *howto,
 # define movz_hw_R0    (0x52c00000)
 #endif
 
+/* C64 only instructions.  */
+#define add_C0_C0      (0x02000000)
+#define ldp_X0_X1_C0    (0xa9400400)
+#define add_C0_C2_X0    (0xc2a06040)
+#define scbnds_C0_C0_X1 (0xc2c10000)
+
 /* Structure to hold payload for _bfd_aarch64_erratum_843419_clear_stub,
    it is used to identify the stub information to reset.  */
 
@@ -8027,53 +8201,18 @@ static bfd_reloc_status_type
 elfNN_aarch64_tls_relax (bfd *input_bfd, struct bfd_link_info *info,
                         asection *input_section,
                         bfd_byte *contents, Elf_Internal_Rela *rel,
-                        struct elf_link_hash_entry *h, unsigned long r_symndx)
+                        struct elf_link_hash_entry *h)
 {
   bfd_boolean local_exec = (bfd_link_executable (info)
                            && TLS_SYMBOL_REFERENCES_LOCAL (info, h));
   unsigned int r_type = ELFNN_R_TYPE (rel->r_info);
   unsigned long insn;
-  bfd_vma sym_size = 0;
   struct elf_aarch64_link_hash_table *globals = elf_aarch64_hash_table (info);
 
   BFD_ASSERT (globals && input_bfd && contents && rel);
 
-  if (local_exec)
-    {
-      if (h != NULL)
-       sym_size = h->size;
-      else
-       {
-         Elf_Internal_Sym *sym;
-
-         sym = bfd_sym_from_r_symndx (&globals->root.sym_cache, input_bfd,
-                                      r_symndx);
-         BFD_ASSERT (sym != NULL);
-         sym_size = sym->st_size;
-       }
-    }
-
   switch (elfNN_aarch64_bfd_reloc_from_type (input_bfd, r_type))
     {
-    case BFD_RELOC_MORELLO_TLSDESC_ADR_PAGE20:
-      if (local_exec)
-       {
-         /* GD->LE relaxation:
-            nop                     =>   movz x1, objsize_hi16
-            adrp x0, :tlsdesc:var   =>   movz x0, :tprel_g1:var  */
-         bfd_putl32 (BUILD_MOVZ(1, sym_size), contents + rel->r_offset - 4);
-         bfd_putl32 (movz_R0, contents + rel->r_offset);
-
-         /* We have relaxed the adrp into a mov, we may have to clear any
-            pending erratum fixes.  */
-         clear_erratum_843419_entry (globals, rel->r_offset, input_section);
-         return bfd_reloc_continue;
-       }
-      else
-       {
-         /* GD->IE relaxation: Not implemented.  */
-         return bfd_reloc_continue;
-       }
     case BFD_RELOC_AARCH64_TLSDESC_ADR_PAGE21:
     case BFD_RELOC_AARCH64_TLSGD_ADR_PAGE21:
       if (local_exec)
@@ -8239,22 +8378,21 @@ elfNN_aarch64_tls_relax (bfd *input_bfd, struct bfd_link_info *info,
       return bfd_reloc_continue;
 #endif
 
+    case BFD_RELOC_MORELLO_TLSIE_ADR_GOTTPREL_PAGE20:
+      /* IE->LE relaxation:
+        adrp c0, :gottprel:var   =>   adrp c0, __var_data
+        Instruction does not change (just the relocation on it).  */
+      return bfd_reloc_continue;
+
+    case BFD_RELOC_MORELLO_TLSIE_ADD_LO12:
+      /* IE->LE relaxation:
+        add c0, c0, :gottprel_lo12:var   =>   add c0, c0, :lo12:__var_data
+        Instruction does not change (just the relocation on it).  */
+      return bfd_reloc_continue;
+
     case BFD_RELOC_AARCH64_TLSIE_LD_GOTTPREL_PREL19:
       return bfd_reloc_continue;
 
-    case BFD_RELOC_MORELLO_TLSDESC_LD128_LO12:
-      if (local_exec)
-       {
-         /* GD->LE relaxation:
-            ldr xd, [x0, #:tlsdesc_lo12:var] => movk x0, :tprel_g0_nc:var  */
-         bfd_putl32 (movk_R0, contents + rel->r_offset);
-         return bfd_reloc_continue;
-       }
-      else
-       {
-         /* GD->IE relaxation: not implemented.  */
-         return bfd_reloc_continue;
-       }
     case BFD_RELOC_AARCH64_TLSDESC_LDNN_LO12_NC:
       if (local_exec)
        {
@@ -8319,25 +8457,39 @@ elfNN_aarch64_tls_relax (bfd *input_bfd, struct bfd_link_info *info,
          return bfd_reloc_continue;
        }
 
-    case BFD_RELOC_MORELLO_TLSDESC_CALL:
+    case BFD_RELOC_MORELLO_TLSDESC_ADR_PAGE20:
+      /* GD->IE relaxation:
+        adrp c0, :tlsdesc:var   =>   adrp c0, :gottprel:var
+        GD->LE relaxation:
+        adrp c0, :tlsdesc:var   =>   adrp c0, __var_data
+        (No change in instructions, just need a different relocation).  */
+      return bfd_reloc_continue;
+
+    case BFD_RELOC_MORELLO_TLSDESC_LD128_LO12:
       /* GD->LE relaxation:
-        blr cd                           =>   add c0, c2, x0  */
-      if (local_exec)
-       {
-         bfd_putl32 (0xc2a06040, contents + rel->r_offset);
-         return bfd_reloc_ok;
-       }
-      else
-       goto set_nop;
+        ldr c1, [c0, #:tlsdesc_lo12:var] => add c0, c0, :lo12:__var_data
+        GD->IE relaxation:
+        ldr c1, [c0, #:tlsdesc_lo12:var] => add c0, c0, :gottprel_lo12:__var_data
+        (same relaxation in terms of instruction change, only different in
+        terms of relocation that is used).  */
+      bfd_putl32 (add_C0_C0, contents + rel->r_offset);
+      return bfd_reloc_continue;
+
+    case BFD_RELOC_MORELLO_TLSDESC_CALL:
+      /* GD->LE relaxation  AND  GD->IE relaxation:
+        nop                              =>   add c0, c2, x0
+        blr cd                           =>   scbnds c0, c0, x1  */
+      bfd_putl32 (add_C0_C2_X0, contents + rel->r_offset - 4);
+      bfd_putl32 (scbnds_C0_C0_X1, contents + rel->r_offset);
+      return bfd_reloc_ok;
 
     case BFD_RELOC_AARCH64_TLSDESC_ADD_LO12:
-      /* GD->LE relaxation:
-        ldr cd, [c0, #:tlsdesc_lo12:var] => movk x1, objsize_lo16  */
-      if (local_exec
-         && ELFNN_R_TYPE (rel[1].r_info) == MORELLO_R (TLSDESC_CALL))
+      /* GD->LE relaxation  AND  GD->IE relaxation:
+        add c0, c0, :tlsdesc_lo12:var    =>  ldp x0, x1, [c0]  */
+      if (ELFNN_R_TYPE (rel[1].r_info) == MORELLO_R (TLSDESC_CALL))
        {
-         bfd_putl32 (BUILD_MOVK(1, sym_size), contents + rel->r_offset);
-         return bfd_reloc_continue;
+         bfd_putl32 (ldp_X0_X1_C0, contents + rel->r_offset);
+         return bfd_reloc_ok;
        }
 
       /* Fall through.  */
@@ -8347,7 +8499,6 @@ elfNN_aarch64_tls_relax (bfd *input_bfd, struct bfd_link_info *info,
         add x0, x0, #:tlsdesc_lo12:var   =>   nop
         blr xd                           =>   nop
        */
-set_nop:
       bfd_putl32 (INSN_NOP, contents + rel->r_offset);
       return bfd_reloc_ok;
 
@@ -8489,6 +8640,35 @@ set_nop:
 
 /* Relocate an AArch64 ELF section.  */
 
+static bfd_vma
+c64_populate_tls_data_stub (struct elf_aarch64_link_hash_table *globals,
+                           bfd *input_bfd,
+                           struct elf_link_hash_entry *h,
+                           unsigned long r_symndx,
+                           bfd_vma offset, bfd_vma size,
+                           bfd *output_bfd)
+{
+  struct elf_c64_tls_data_stub_hash_entry *found
+    = c64_tls_stub_find (h, input_bfd, r_symndx, globals);
+  BFD_ASSERT (found);
+
+  if (!found->populated)
+    {
+      bfd_put_NN (output_bfd, offset,
+                 globals->sc64_tls_stubs->contents +
+                 found->tls_stub_offset);
+      bfd_put_NN (output_bfd, size,
+                 globals->sc64_tls_stubs->contents +
+                 found->tls_stub_offset + 8);
+      found->populated = TRUE;
+    }
+
+  return globals->sc64_tls_stubs->output_section->vma
+    + globals->sc64_tls_stubs->output_offset
+    + found->tls_stub_offset;
+}
+
+
 static bfd_boolean
 elfNN_aarch64_relocate_section (bfd *output_bfd,
                                struct bfd_link_info *info,
@@ -8643,16 +8823,26 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
         We call elfNN_aarch64_final_link_relocate unless we're completely
         done, i.e., the relaxation produced the final output we want.  */
 
+      bfd_boolean requires_c64_tls_stub;
       relaxed_bfd_r_type = aarch64_tls_transition (input_bfd, info, rel,
-                                                  h, r_symndx);
+                                                  h, r_symndx,
+                                                  &requires_c64_tls_stub);
       if (relaxed_bfd_r_type != bfd_r_type)
        {
          bfd_r_type = relaxed_bfd_r_type;
          howto = elfNN_aarch64_howto_from_bfd_reloc (bfd_r_type);
          BFD_ASSERT (howto != NULL);
          r_type = howto->type;
+         if (requires_c64_tls_stub)
+           {
+             relocation
+               = c64_populate_tls_data_stub (globals, input_bfd, h, r_symndx,
+                                             relocation - tpoff_base (info),
+                                             h ? h->size : sym->st_size,
+                                             output_bfd);
+           }
          r = elfNN_aarch64_tls_relax (input_bfd, info, input_section,
-                                      contents, rel, h, r_symndx);
+                                      contents, rel, h);
          unresolved_reloc = 0;
        }
       else
@@ -8717,7 +8907,7 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
 
                  loc = globals->root.srelgot->contents;
                  loc += globals->root.srelgot->reloc_count++
-                   * RELOC_SIZE (htab);
+                   * RELOC_SIZE (globals);
                  bfd_elfNN_swap_reloca_out (output_bfd, &rela, loc);
 
                  bfd_reloc_code_real_type real_type =
@@ -8779,6 +8969,9 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
 
        case BFD_RELOC_MORELLO_TLSIE_ADR_GOTTPREL_PAGE20:
        case BFD_RELOC_MORELLO_TLSIE_ADD_LO12:
+         c64_rtype = TRUE;
+         /* Fall through.  */
+
        case BFD_RELOC_AARCH64_TLSIE_ADR_GOTTPREL_PAGE21:
        case BFD_RELOC_AARCH64_TLSIE_LDNN_GOTTPREL_LO12_NC:
        case BFD_RELOC_AARCH64_TLSIE_LD_GOTTPREL_PREL19:
@@ -8800,6 +8993,7 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
                (h == NULL
                 || ELF_ST_VISIBILITY (h->other) == STV_DEFAULT
                 || h->root.type != bfd_link_hash_undefweak);
+             BFD_ASSERT (!c64_rtype || c64_needs_relocation (info, h));
              need_relocs = need_relocs || c64_rtype;
 
              BFD_ASSERT (globals->root.srelgot != NULL);
@@ -8822,7 +9016,7 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
 
                  loc = globals->root.srelgot->contents;
                  loc += globals->root.srelgot->reloc_count++
-                   * RELOC_SIZE (htab);
+                   * RELOC_SIZE (globals);
 
                  bfd_elfNN_swap_reloca_out (output_bfd, &rela, loc);
 
@@ -8864,6 +9058,7 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
              need_relocs = (h == NULL
                             || ELF_ST_VISIBILITY (h->other) == STV_DEFAULT
                             || h->root.type != bfd_link_hash_undefweak);
+             BFD_ASSERT (!c64_rtype || c64_needs_relocation (info, h));
              need_relocs = need_relocs || c64_rtype;
 
              BFD_ASSERT (globals->root.srelgot != NULL);
@@ -9562,7 +9757,16 @@ elfNN_aarch64_check_relocs (bfd *abfd, struct bfd_link_info *info,
        }
 
       /* Could be done earlier, if h were already available.  */
-      bfd_r_type = aarch64_tls_transition (abfd, info, rel, h, r_symndx);
+      bfd_boolean requires_c64_tls_stub;
+      bfd_r_type = aarch64_tls_transition (abfd, info, rel, h, r_symndx,
+                                          &requires_c64_tls_stub);
+      if (requires_c64_tls_stub
+         && !c64_record_tls_stub (htab, abfd, h, r_symndx))
+       {
+         _bfd_error_handler (_("%pB: failed to record TLS stub"), abfd);
+         bfd_set_error (bfd_error_no_memory);
+         return FALSE;
+       }
 
       if (h != NULL)
        {
@@ -9950,7 +10154,7 @@ elfNN_aarch64_check_relocs (bfd *abfd, struct bfd_link_info *info,
          break;
 
        case BFD_RELOC_MORELLO_CAPINIT:
-         if (h && !c64_needs_relocation (info, h))
+         if (!c64_needs_relocation (info, h))
            /* If this symbol does not need a relocation, then there's no
               reason to increase the srelcaps size for a relocation.  */
            break;
@@ -10565,14 +10769,17 @@ elfNN_aarch64_allocate_dynrelocs (struct elf_link_hash_entry *h, void *inf)
              htab->root.sgot->size += GOT_ENTRY_SIZE (htab);
            }
 
+         /* We avoid TLS relocations on undefweak symbols since it is not
+            well defined.  Hence we should not be seeing got entries
+            on any symbol which would not need a relocation in a C64 binary.
+            */
+         BFD_ASSERT (c64_needs_relocation (info, h));
          indx = h && h->dynindx != -1 ? h->dynindx : 0;
          if ((ELF_ST_VISIBILITY (h->other) == STV_DEFAULT
               || h->root.type != bfd_link_hash_undefweak)
              && (!bfd_link_executable (info)
                  || indx != 0
                  || WILL_CALL_FINISH_DYNAMIC_SYMBOL (dyn, 0, h)
-                 /* On Morello support only TLSDESC_GD to TLSLE relaxation;
-                    for everything else we must emit a dynamic relocation.  */
                  || htab->c64_rel))
            {
              if (got_type & GOT_TLSDESC_GD)
@@ -10980,7 +11187,8 @@ elfNN_aarch64_size_dynamic_sections (bfd *output_bfd,
          || s == htab->root.iplt
          || s == htab->root.igotplt
          || s == htab->root.sdynbss
-         || s == htab->root.sdynrelro)
+         || s == htab->root.sdynrelro
+         || s == htab->sc64_tls_stubs)
        {
          /* Strip this section if we don't need it; see the
             comment below.  */