drm/i915: Workaround incoherence between fences and LLC across multiple CPUs

commit 25ff1195f8a0b3724541ae7bbe331b4296de9c06 upstream.

In order to fully serialize access to the fenced region and the update
to the fence register we need to take extreme measures on SNB+, and
manually flush writes to memory prior to writing the fence register in
conjunction with the memory barriers placed around the register write.

Fixes i-g-t/gem_fence_thrash

v2: Bring a bigger gun
v3: Switch the bigger gun for heavier bullets (Arjan van de Ven)
v4: Remove changes for working generations.
v5: Reduce to a per-cpu wbinvd() call prior to updating the fences.
v6: Rewrite comments to ellide forgotten history.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=62191
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Jon Bloomfield <jon.bloomfield@intel.com>
Tested-by: Jon Bloomfield <jon.bloomfield@intel.com> (v2)
Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
[bwh: Backported to 3.2: insert the cache flush in i915_gem_object_get_fence()]
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index b0186b8567e549b975bbafbf2c9f5d9e17a3d2ae..2865b44dfab2b05fef8bcabee459aa4cc30f5a92 100644 (file)
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2520,6 +2520,11 @@ i915_find_fence_reg(struct drm_device *dev,
        return avail;
 }
 
+static void i915_gem_write_fence__ipi(void *data)
+{
+       wbinvd();
+}
+
 /**
  * i915_gem_object_get_fence - set up a fence reg for an object
  * @obj: object to map through a fence reg
@@ -2640,6 +2645,17 @@ update:
        switch (INTEL_INFO(dev)->gen) {
        case 7:
        case 6:
+               /* In order to fully serialize access to the fenced region and
+                * the update to the fence register we need to take extreme
+                * measures on SNB+. In theory, the write to the fence register
+                * flushes all memory transactions before, and coupled with the
+                * mb() placed around the register write we serialise all memory
+                * operations with respect to the changes in the tiler. Yet, on
+                * SNB+ we need to take a step further and emit an explicit wbinvd()
+                * on each processor in order to manually flush all memory
+                * transactions before updating the fence register.
+                */
+               on_each_cpu(i915_gem_write_fence__ipi, NULL, 1);
                ret = sandybridge_write_fence_reg(obj, pipelined);
                break;
        case 5: