iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes

The iotlb_sync_map iommu ops allows drivers to perform necessary cache
flushes when new mappings are established. For the Intel iommu driver,
this callback specifically serves two purposes:

- To flush caches when a second-stage page table is attached to a device
  whose iommu is operating in caching mode (CAP_REG.CM==1).
- To explicitly flush internal write buffers to ensure updates to memory-
  resident remapping structures are visible to hardware (CAP_REG.RWBF==1).

However, in scenarios where neither caching mode nor the RWBF flag is
active, the cache_tag_flush_range_np() helper, which is called in the
iotlb_sync_map path, effectively becomes a no-op.

Despite being a no-op, cache_tag_flush_range_np() involves iterating
through all cache tags of the iommu's attached to the domain, protected
by a spinlock. This unnecessary execution path introduces overhead,
leading to a measurable I/O performance regression. On systems with NVMes
under the same bridge, performance was observed to drop from approximately
~6150 MiB/s down to ~4985 MiB/s.

Introduce a flag in the dmar_domain structure. This flag will only be set
when iotlb_sync_map is required (i.e., when CM or RWBF is set). The
cache_tag_flush_range_np() is called only for domains where this flag is
set. This flag, once set, is immutable, given that there won't be mixed
configurations in real-world scenarios where some IOMMUs in a system
operate in caching mode while others do not. Theoretically, the
immutability of this flag does not impact functionality.

Reported-by: Ioanna Alifieraki <ioanna-maria.alifieraki@canonical.com>
Closes: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2115738
Link: https://lore.kernel.org/r/20250701171154.52435-1-ioanna-maria.alifieraki@canonical.com
Fixes: 129dab6e1286 ("iommu/vt-d: Use cache_tag_flush_range_np() in iotlb_sync_map")
Cc: stable@vger.kernel.org
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Link: https://lore.kernel.org/r/20250703031545.3378602-1-baolu.lu@linux.intel.com
Link: https://lore.kernel.org/r/20250714045028.958850-3-baolu.lu@linux.intel.com
Signed-off-by: Will Deacon <will@kernel.org>

diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 7aa3932251b2fd6d632f5040cac4ba21d474a3f5..f60201ee4be0fd66568611529c39ea99d7aca15e 100644 (file)
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -1796,6 +1796,18 @@ static int domain_setup_first_level(struct intel_iommu *iommu,
                                          (pgd_t *)pgd, flags, old);
 }
 
+static bool domain_need_iotlb_sync_map(struct dmar_domain *domain,
+                                      struct intel_iommu *iommu)
+{
+       if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
+               return true;
+
+       if (rwbf_quirk || cap_rwbf(iommu->cap))
+               return true;
+
+       return false;
+}
+
 static int dmar_domain_attach_device(struct dmar_domain *domain,
                                     struct device *dev)
 {
@@ -1833,6 +1845,8 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
        if (ret)
                goto out_block_translation;
 
+       domain->iotlb_sync_map |= domain_need_iotlb_sync_map(domain, iommu);
+
        return 0;
 
 out_block_translation:
@@ -3945,7 +3959,10 @@ static bool risky_device(struct pci_dev *pdev)
 static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
                                      unsigned long iova, size_t size)
 {
-       cache_tag_flush_range_np(to_dmar_domain(domain), iova, iova + size - 1);
+       struct dmar_domain *dmar_domain = to_dmar_domain(domain);
+
+       if (dmar_domain->iotlb_sync_map)
+               cache_tag_flush_range_np(dmar_domain, iova, iova + size - 1);
 
        return 0;
 }

diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
index 3ddbcc603de23b704fd5c2ac5e5764e802f2a882..7ab2c34a5ecc40d0d26c784aebe2ecb6ea082836 100644 (file)
--- a/drivers/iommu/intel/iommu.h
+++ b/drivers/iommu/intel/iommu.h
@@ -614,6 +614,9 @@ struct dmar_domain {
        u8 has_mappings:1;              /* Has mappings configured through
                                         * iommu_map() interface.
                                         */
+       u8 iotlb_sync_map:1;            /* Need to flush IOTLB cache or write
+                                        * buffer when creating mappings.
+                                        */
 
        spinlock_t lock;                /* Protect device tracking lists */
        struct list_head devices;       /* all devices' list */