2 * Copyright © 2015-2016 Intel Corporation
4 * Permission is hereby granted, free of charge, to any person obtaining a
5 * copy of this software and associated documentation files (the "Software"),
6 * to deal in the Software without restriction, including without limitation
7 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
8 * and/or sell copies of the Software, and to permit persons to whom the
9 * Software is furnished to do so, subject to the following conditions:
11 * The above copyright notice and this permission notice (including the next
12 * paragraph) shall be included in all copies or substantial portions of the
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
20 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
24 * Robert Bragg <robert@sixbynine.org>
29 * DOC: i915 Perf Overview
31 * Gen graphics supports a large number of performance counters that can help
32 * driver and application developers understand and optimize their use of the
35 * This i915 perf interface enables userspace to configure and open a file
36 * descriptor representing a stream of GPU metrics which can then be read() as
37 * a stream of sample records.
39 * The interface is particularly suited to exposing buffered metrics that are
40 * captured by DMA from the GPU, unsynchronized with and unrelated to the CPU.
42 * Streams representing a single context are accessible to applications with a
43 * corresponding drm file descriptor, such that OpenGL can use the interface
44 * without special privileges. Access to system-wide metrics requires root
45 * privileges by default, unless changed via the dev.i915.perf_event_paranoid
51 * DOC: i915 Perf History and Comparison with Core Perf
53 * The interface was initially inspired by the core Perf infrastructure but
54 * some notable differences are:
56 * i915 perf file descriptors represent a "stream" instead of an "event"; where
57 * a perf event primarily corresponds to a single 64bit value, while a stream
58 * might sample sets of tightly-coupled counters, depending on the
59 * configuration. For example the Gen OA unit isn't designed to support
60 * orthogonal configurations of individual counters; it's configured for a set
61 * of related counters. Samples for an i915 perf stream capturing OA metrics
62 * will include a set of counter values packed in a compact HW specific format.
63 * The OA unit supports a number of different packing formats which can be
64 * selected by the user opening the stream. Perf has support for grouping
65 * events, but each event in the group is configured, validated and
66 * authenticated individually with separate system calls.
68 * i915 perf stream configurations are provided as an array of u64 (key,value)
69 * pairs, instead of a fixed struct with multiple miscellaneous config members,
70 * interleaved with event-type specific members.
72 * i915 perf doesn't support exposing metrics via an mmap'd circular buffer.
73 * The supported metrics are being written to memory by the GPU unsynchronized
74 * with the CPU, using HW specific packing formats for counter sets. Sometimes
75 * the constraints on HW configuration require reports to be filtered before it
76 * would be acceptable to expose them to unprivileged applications - to hide
77 * the metrics of other processes/contexts. For these use cases a read() based
78 * interface is a good fit, and provides an opportunity to filter data as it
79 * gets copied from the GPU mapped buffers to userspace buffers.
82 * Issues hit with first prototype based on Core Perf
83 * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
85 * The first prototype of this driver was based on the core perf
86 * infrastructure, and while we did make that mostly work, with some changes to
87 * perf, we found we were breaking or working around too many assumptions baked
88 * into perf's currently cpu centric design.
90 * In the end we didn't see a clear benefit to making perf's implementation and
91 * interface more complex by changing design assumptions while we knew we still
92 * wouldn't be able to use any existing perf based userspace tools.
94 * Also considering the Gen specific nature of the Observability hardware and
95 * how userspace will sometimes need to combine i915 perf OA metrics with
96 * side-band OA data captured via MI_REPORT_PERF_COUNT commands; we're
97 * expecting the interface to be used by a platform specific userspace such as
98 * OpenGL or tools. This is to say; we aren't inherently missing out on having
99 * a standard vendor/architecture agnostic interface by not using perf.
102 * For posterity, in case we might re-visit trying to adapt core perf to be
103 * better suited to exposing i915 metrics these were the main pain points we
106 * - The perf based OA PMU driver broke some significant design assumptions:
108 * Existing perf pmus are used for profiling work on a cpu and we were
109 * introducing the idea of _IS_DEVICE pmus with different security
110 * implications, the need to fake cpu-related data (such as user/kernel
111 * registers) to fit with perf's current design, and adding _DEVICE records
112 * as a way to forward device-specific status records.
114 * The OA unit writes reports of counters into a circular buffer, without
115 * involvement from the CPU, making our PMU driver the first of a kind.
117 * Given the way we were periodically forward data from the GPU-mapped, OA
118 * buffer to perf's buffer, those bursts of sample writes looked to perf like
119 * we were sampling too fast and so we had to subvert its throttling checks.
121 * Perf supports groups of counters and allows those to be read via
122 * transactions internally but transactions currently seem designed to be
123 * explicitly initiated from the cpu (say in response to a userspace read())
124 * and while we could pull a report out of the OA buffer we can't
125 * trigger a report from the cpu on demand.
127 * Related to being report based; the OA counters are configured in HW as a
128 * set while perf generally expects counter configurations to be orthogonal.
129 * Although counters can be associated with a group leader as they are
130 * opened, there's no clear precedent for being able to provide group-wide
131 * configuration attributes (for example we want to let userspace choose the
132 * OA unit report format used to capture all counters in a set, or specify a
133 * GPU context to filter metrics on). We avoided using perf's grouping
134 * feature and forwarded OA reports to userspace via perf's 'raw' sample
135 * field. This suited our userspace well considering how coupled the counters
136 * are when dealing with normalizing. It would be inconvenient to split
137 * counters up into separate events, only to require userspace to recombine
138 * them. For Mesa it's also convenient to be forwarded raw, periodic reports
139 * for combining with the side-band raw reports it captures using
140 * MI_REPORT_PERF_COUNT commands.
142 * - As a side note on perf's grouping feature; there was also some concern
143 * that using PERF_FORMAT_GROUP as a way to pack together counter values
144 * would quite drastically inflate our sample sizes, which would likely
145 * lower the effective sampling resolutions we could use when the available
146 * memory bandwidth is limited.
148 * With the OA unit's report formats, counters are packed together as 32
149 * or 40bit values, with the largest report size being 256 bytes.
151 * PERF_FORMAT_GROUP values are 64bit, but there doesn't appear to be a
152 * documented ordering to the values, implying PERF_FORMAT_ID must also be
153 * used to add a 64bit ID before each value; giving 16 bytes per counter.
155 * Related to counter orthogonality; we can't time share the OA unit, while
156 * event scheduling is a central design idea within perf for allowing
157 * userspace to open + enable more events than can be configured in HW at any
158 * one time. The OA unit is not designed to allow re-configuration while in
159 * use. We can't reconfigure the OA unit without losing internal OA unit
160 * state which we can't access explicitly to save and restore. Reconfiguring
161 * the OA unit is also relatively slow, involving ~100 register writes. From
162 * userspace Mesa also depends on a stable OA configuration when emitting
163 * MI_REPORT_PERF_COUNT commands and importantly the OA unit can't be
164 * disabled while there are outstanding MI_RPC commands lest we hang the
167 * The contents of sample records aren't extensible by device drivers (i.e.
168 * the sample_type bits). As an example; Sourab Gupta had been looking to
169 * attach GPU timestamps to our OA samples. We were shoehorning OA reports
170 * into sample records by using the 'raw' field, but it's tricky to pack more
171 * than one thing into this field because events/core.c currently only lets a
172 * pmu give a single raw data pointer plus len which will be copied into the
173 * ring buffer. To include more than the OA report we'd have to copy the
174 * report into an intermediate larger buffer. I'd been considering allowing a
175 * vector of data+len values to be specified for copying the raw data, but
176 * it felt like a kludge to being using the raw field for this purpose.
178 * - It felt like our perf based PMU was making some technical compromises
179 * just for the sake of using perf:
181 * perf_event_open() requires events to either relate to a pid or a specific
182 * cpu core, while our device pmu related to neither. Events opened with a
183 * pid will be automatically enabled/disabled according to the scheduling of
184 * that process - so not appropriate for us. When an event is related to a
185 * cpu id, perf ensures pmu methods will be invoked via an inter process
186 * interrupt on that core. To avoid invasive changes our userspace opened OA
187 * perf events for a specific cpu. This was workable but it meant the
188 * majority of the OA driver ran in atomic context, including all OA report
189 * forwarding, which wasn't really necessary in our case and seems to make
190 * our locking requirements somewhat complex as we handled the interaction
191 * with the rest of the i915 driver.
194 #include <linux/anon_inodes.h>
195 #include <linux/nospec.h>
196 #include <linux/sizes.h>
197 #include <linux/uuid.h>
199 #include "gem/i915_gem_context.h"
200 #include "gem/i915_gem_internal.h"
201 #include "gt/intel_engine_pm.h"
202 #include "gt/intel_engine_regs.h"
203 #include "gt/intel_engine_user.h"
204 #include "gt/intel_execlists_submission.h"
205 #include "gt/intel_gpu_commands.h"
206 #include "gt/intel_gt.h"
207 #include "gt/intel_gt_clock_utils.h"
208 #include "gt/intel_gt_mcr.h"
209 #include "gt/intel_gt_regs.h"
210 #include "gt/intel_lrc.h"
211 #include "gt/intel_lrc_reg.h"
212 #include "gt/intel_rc6.h"
213 #include "gt/intel_ring.h"
214 #include "gt/uc/intel_guc_slpc.h"
216 #include "i915_drv.h"
217 #include "i915_file_private.h"
218 #include "i915_perf.h"
219 #include "i915_perf_oa_regs.h"
220 #include "i915_reg.h"
222 /* HW requires this to be a power of two, between 128k and 16M, though driver
223 * is currently generally designed assuming the largest 16M size is used such
224 * that the overflow cases are unlikely in normal operation.
226 #define OA_BUFFER_SIZE SZ_16M
228 #define OA_TAKEN(tail, head) ((tail - head) & (OA_BUFFER_SIZE - 1))
231 * DOC: OA Tail Pointer Race
233 * There's a HW race condition between OA unit tail pointer register updates and
234 * writes to memory whereby the tail pointer can sometimes get ahead of what's
235 * been written out to the OA buffer so far (in terms of what's visible to the
238 * Although this can be observed explicitly while copying reports to userspace
239 * by checking for a zeroed report-id field in tail reports, we want to account
240 * for this earlier, as part of the oa_buffer_check_unlocked to avoid lots of
241 * redundant read() attempts.
243 * We workaround this issue in oa_buffer_check_unlocked() by reading the reports
244 * in the OA buffer, starting from the tail reported by the HW until we find a
245 * report with its first 2 dwords not 0 meaning its previous report is
246 * completely in memory and ready to be read. Those dwords are also set to 0
247 * once read and the whole buffer is cleared upon OA buffer initialization. The
248 * first dword is the reason for this report while the second is the timestamp,
249 * making the chances of having those 2 fields at 0 fairly unlikely. A more
250 * detailed explanation is available in oa_buffer_check_unlocked().
252 * Most of the implementation details for this workaround are in
253 * oa_buffer_check_unlocked() and _append_oa_reports()
255 * Note for posterity: previously the driver used to define an effective tail
256 * pointer that lagged the real pointer by a 'tail margin' measured in bytes
257 * derived from %OA_TAIL_MARGIN_NSEC and the configured sampling frequency.
258 * This was flawed considering that the OA unit may also automatically generate
259 * non-periodic reports (such as on context switch) or the OA unit may be
260 * enabled without any periodic sampling.
262 #define OA_TAIL_MARGIN_NSEC 100000ULL
263 #define INVALID_TAIL_PTR 0xffffffff
265 /* The default frequency for checking whether the OA unit has written new
266 * reports to the circular OA buffer...
268 #define DEFAULT_POLL_FREQUENCY_HZ 200
269 #define DEFAULT_POLL_PERIOD_NS (NSEC_PER_SEC / DEFAULT_POLL_FREQUENCY_HZ)
271 /* for sysctl proc_dointvec_minmax of dev.i915.perf_stream_paranoid */
272 static u32 i915_perf_stream_paranoid
= true;
274 /* The maximum exponent the hardware accepts is 63 (essentially it selects one
275 * of the 64bit timestamp bits to trigger reports from) but there's currently
276 * no known use case for sampling as infrequently as once per 47 thousand years.
278 * Since the timestamps included in OA reports are only 32bits it seems
279 * reasonable to limit the OA exponent where it's still possible to account for
280 * overflow in OA report timestamps.
282 #define OA_EXPONENT_MAX 31
284 #define INVALID_CTX_ID 0xffffffff
286 /* On Gen8+ automatically triggered OA reports include a 'reason' field... */
287 #define OAREPORT_REASON_MASK 0x3f
288 #define OAREPORT_REASON_MASK_EXTENDED 0x7f
289 #define OAREPORT_REASON_SHIFT 19
290 #define OAREPORT_REASON_TIMER (1<<0)
291 #define OAREPORT_REASON_CTX_SWITCH (1<<3)
292 #define OAREPORT_REASON_CLK_RATIO (1<<5)
294 #define HAS_MI_SET_PREDICATE(i915) (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50))
296 /* For sysctl proc_dointvec_minmax of i915_oa_max_sample_rate
298 * The highest sampling frequency we can theoretically program the OA unit
299 * with is always half the timestamp frequency: E.g. 6.25Mhz for Haswell.
301 * Initialized just before we register the sysctl parameter.
303 static int oa_sample_rate_hard_limit
;
305 /* Theoretically we can program the OA unit to sample every 160ns but don't
306 * allow that by default unless root...
308 * The default threshold of 100000Hz is based on perf's similar
309 * kernel.perf_event_max_sample_rate sysctl parameter.
311 static u32 i915_oa_max_sample_rate
= 100000;
313 /* XXX: beware if future OA HW adds new report formats that the current
314 * code assumes all reports have a power-of-two size and ~(size - 1) can
315 * be used as a mask to align the OA tail pointer.
317 static const struct i915_oa_format oa_formats
[I915_OA_FORMAT_MAX
] = {
318 [I915_OA_FORMAT_A13
] = { 0, 64 },
319 [I915_OA_FORMAT_A29
] = { 1, 128 },
320 [I915_OA_FORMAT_A13_B8_C8
] = { 2, 128 },
321 /* A29_B8_C8 Disallowed as 192 bytes doesn't factor into buffer size */
322 [I915_OA_FORMAT_B4_C8
] = { 4, 64 },
323 [I915_OA_FORMAT_A45_B8_C8
] = { 5, 256 },
324 [I915_OA_FORMAT_B4_C8_A16
] = { 6, 128 },
325 [I915_OA_FORMAT_C4_B8
] = { 7, 64 },
326 [I915_OA_FORMAT_A12
] = { 0, 64 },
327 [I915_OA_FORMAT_A12_B8_C8
] = { 2, 128 },
328 [I915_OA_FORMAT_A32u40_A4u32_B8_C8
] = { 5, 256 },
329 [I915_OAR_FORMAT_A32u40_A4u32_B8_C8
] = { 5, 256 },
330 [I915_OA_FORMAT_A24u40_A14u32_B8_C8
] = { 5, 256 },
331 [I915_OAM_FORMAT_MPEC8u64_B8_C8
] = { 1, 192, TYPE_OAM
, HDR_64_BIT
},
332 [I915_OAM_FORMAT_MPEC8u32_B8_C8
] = { 2, 128, TYPE_OAM
, HDR_64_BIT
},
335 static const u32 mtl_oa_base
[] = {
336 [PERF_GROUP_OAM_SAMEDIA_0
] = 0x393000,
339 #define SAMPLE_OA_REPORT (1<<0)
342 * struct perf_open_properties - for validated properties given to open a stream
343 * @sample_flags: `DRM_I915_PERF_PROP_SAMPLE_*` properties are tracked as flags
344 * @single_context: Whether a single or all gpu contexts should be monitored
345 * @hold_preemption: Whether the preemption is disabled for the filtered
347 * @ctx_handle: A gem ctx handle for use with @single_context
348 * @metrics_set: An ID for an OA unit metric set advertised via sysfs
349 * @oa_format: An OA unit HW report format
350 * @oa_periodic: Whether to enable periodic OA unit sampling
351 * @oa_period_exponent: The OA unit sampling period is derived from this
352 * @engine: The engine (typically rcs0) being monitored by the OA unit
353 * @has_sseu: Whether @sseu was specified by userspace
354 * @sseu: internal SSEU configuration computed either from the userspace
355 * specified configuration in the opening parameters or a default value
356 * (see get_default_sseu_config())
357 * @poll_oa_period: The period in nanoseconds at which the CPU will check for OA
360 * As read_properties_unlocked() enumerates and validates the properties given
361 * to open a stream of metrics the configuration is built up in the structure
362 * which starts out zero initialized.
364 struct perf_open_properties
{
367 u64 single_context
:1;
368 u64 hold_preemption
:1;
371 /* OA sampling state */
375 int oa_period_exponent
;
377 struct intel_engine_cs
*engine
;
380 struct intel_sseu sseu
;
385 struct i915_oa_config_bo
{
386 struct llist_node node
;
388 struct i915_oa_config
*oa_config
;
389 struct i915_vma
*vma
;
392 static struct ctl_table_header
*sysctl_header
;
394 static enum hrtimer_restart
oa_poll_check_timer_cb(struct hrtimer
*hrtimer
);
396 void i915_oa_config_release(struct kref
*ref
)
398 struct i915_oa_config
*oa_config
=
399 container_of(ref
, typeof(*oa_config
), ref
);
401 kfree(oa_config
->flex_regs
);
402 kfree(oa_config
->b_counter_regs
);
403 kfree(oa_config
->mux_regs
);
405 kfree_rcu(oa_config
, rcu
);
408 struct i915_oa_config
*
409 i915_perf_get_oa_config(struct i915_perf
*perf
, int metrics_set
)
411 struct i915_oa_config
*oa_config
;
414 oa_config
= idr_find(&perf
->metrics_idr
, metrics_set
);
416 oa_config
= i915_oa_config_get(oa_config
);
422 static void free_oa_config_bo(struct i915_oa_config_bo
*oa_bo
)
424 i915_oa_config_put(oa_bo
->oa_config
);
425 i915_vma_put(oa_bo
->vma
);
430 struct i915_perf_regs
*__oa_regs(struct i915_perf_stream
*stream
)
432 return &stream
->engine
->oa_group
->regs
;
435 static u32
gen12_oa_hw_tail_read(struct i915_perf_stream
*stream
)
437 struct intel_uncore
*uncore
= stream
->uncore
;
439 return intel_uncore_read(uncore
, __oa_regs(stream
)->oa_tail_ptr
) &
440 GEN12_OAG_OATAILPTR_MASK
;
443 static u32
gen8_oa_hw_tail_read(struct i915_perf_stream
*stream
)
445 struct intel_uncore
*uncore
= stream
->uncore
;
447 return intel_uncore_read(uncore
, GEN8_OATAILPTR
) & GEN8_OATAILPTR_MASK
;
450 static u32
gen7_oa_hw_tail_read(struct i915_perf_stream
*stream
)
452 struct intel_uncore
*uncore
= stream
->uncore
;
453 u32 oastatus1
= intel_uncore_read(uncore
, GEN7_OASTATUS1
);
455 return oastatus1
& GEN7_OASTATUS1_TAIL_MASK
;
458 #define oa_report_header_64bit(__s) \
459 ((__s)->oa_buffer.format->header == HDR_64_BIT)
461 static u64
oa_report_id(struct i915_perf_stream
*stream
, void *report
)
463 return oa_report_header_64bit(stream
) ? *(u64
*)report
: *(u32
*)report
;
466 static u64
oa_report_reason(struct i915_perf_stream
*stream
, void *report
)
468 return (oa_report_id(stream
, report
) >> OAREPORT_REASON_SHIFT
) &
469 (GRAPHICS_VER(stream
->perf
->i915
) == 12 ?
470 OAREPORT_REASON_MASK_EXTENDED
:
471 OAREPORT_REASON_MASK
);
474 static void oa_report_id_clear(struct i915_perf_stream
*stream
, u32
*report
)
476 if (oa_report_header_64bit(stream
))
482 static bool oa_report_ctx_invalid(struct i915_perf_stream
*stream
, void *report
)
484 return !(oa_report_id(stream
, report
) &
485 stream
->perf
->gen8_valid_ctx_bit
) &&
486 GRAPHICS_VER(stream
->perf
->i915
) <= 11;
489 static u64
oa_timestamp(struct i915_perf_stream
*stream
, void *report
)
491 return oa_report_header_64bit(stream
) ?
492 *((u64
*)report
+ 1) :
493 *((u32
*)report
+ 1);
496 static void oa_timestamp_clear(struct i915_perf_stream
*stream
, u32
*report
)
498 if (oa_report_header_64bit(stream
))
499 *(u64
*)&report
[2] = 0;
504 static u32
oa_context_id(struct i915_perf_stream
*stream
, u32
*report
)
506 u32 ctx_id
= oa_report_header_64bit(stream
) ? report
[4] : report
[2];
508 return ctx_id
& stream
->specific_ctx_id_mask
;
511 static void oa_context_id_squash(struct i915_perf_stream
*stream
, u32
*report
)
513 if (oa_report_header_64bit(stream
))
514 report
[4] = INVALID_CTX_ID
;
516 report
[2] = INVALID_CTX_ID
;
520 * oa_buffer_check_unlocked - check for data and update tail ptr state
521 * @stream: i915 stream instance
523 * This is either called via fops (for blocking reads in user ctx) or the poll
524 * check hrtimer (atomic ctx) to check the OA buffer tail pointer and check
525 * if there is data available for userspace to read.
527 * This function is central to providing a workaround for the OA unit tail
528 * pointer having a race with respect to what data is visible to the CPU.
529 * It is responsible for reading tail pointers from the hardware and giving
530 * the pointers time to 'age' before they are made available for reading.
531 * (See description of OA_TAIL_MARGIN_NSEC above for further details.)
533 * Besides returning true when there is data available to read() this function
534 * also updates the tail in the oa_buffer object.
536 * Note: It's safe to read OA config state here unlocked, assuming that this is
537 * only called while the stream is enabled, while the global OA configuration
540 * Returns: %true if the OA buffer contains data, else %false
542 static bool oa_buffer_check_unlocked(struct i915_perf_stream
*stream
)
544 u32 gtt_offset
= i915_ggtt_offset(stream
->oa_buffer
.vma
);
545 int report_size
= stream
->oa_buffer
.format
->size
;
546 u32 head
, tail
, read_tail
;
550 u32 partial_report_size
;
552 /* We have to consider the (unlikely) possibility that read() errors
553 * could result in an OA buffer reset which might reset the head and
556 spin_lock_irqsave(&stream
->oa_buffer
.ptr_lock
, flags
);
558 hw_tail
= stream
->perf
->ops
.oa_hw_tail_read(stream
);
560 /* The tail pointer increases in 64 byte increments, not in report_size
561 * steps. Also the report size may not be a power of 2. Compute
562 * potentially partially landed report in the OA buffer
564 partial_report_size
= OA_TAKEN(hw_tail
, stream
->oa_buffer
.tail
);
565 partial_report_size
%= report_size
;
567 /* Subtract partial amount off the tail */
568 hw_tail
= OA_TAKEN(hw_tail
, partial_report_size
);
570 /* NB: The head we observe here might effectively be a little
571 * out of date. If a read() is in progress, the head could be
572 * anywhere between this head and stream->oa_buffer.tail.
574 head
= stream
->oa_buffer
.head
- gtt_offset
;
575 read_tail
= stream
->oa_buffer
.tail
- gtt_offset
;
579 /* Walk the stream backward until we find a report with report
580 * id and timestmap not at 0. Since the circular buffer pointers
581 * progress by increments of 64 bytes and that reports can be up
582 * to 256 bytes long, we can't tell whether a report has fully
583 * landed in memory before the report id and timestamp of the
584 * following report have effectively landed.
586 * This is assuming that the writes of the OA unit land in
587 * memory in the order they were written to.
588 * If not : (╯°□°)╯︵ ┻━┻
590 while (OA_TAKEN(tail
, read_tail
) >= report_size
) {
591 void *report
= stream
->oa_buffer
.vaddr
+ tail
;
593 if (oa_report_id(stream
, report
) ||
594 oa_timestamp(stream
, report
))
597 tail
= (tail
- report_size
) & (OA_BUFFER_SIZE
- 1);
600 if (OA_TAKEN(hw_tail
, tail
) > report_size
&&
601 __ratelimit(&stream
->perf
->tail_pointer_race
))
602 drm_notice(&stream
->uncore
->i915
->drm
,
603 "unlanded report(s) head=0x%x tail=0x%x hw_tail=0x%x\n",
604 head
, tail
, hw_tail
);
606 stream
->oa_buffer
.tail
= gtt_offset
+ tail
;
608 pollin
= OA_TAKEN(stream
->oa_buffer
.tail
,
609 stream
->oa_buffer
.head
) >= report_size
;
611 spin_unlock_irqrestore(&stream
->oa_buffer
.ptr_lock
, flags
);
617 * append_oa_status - Appends a status record to a userspace read() buffer.
618 * @stream: An i915-perf stream opened for OA metrics
619 * @buf: destination buffer given by userspace
620 * @count: the number of bytes userspace wants to read
621 * @offset: (inout): the current position for writing into @buf
622 * @type: The kind of status to report to userspace
624 * Writes a status record (such as `DRM_I915_PERF_RECORD_OA_REPORT_LOST`)
625 * into the userspace read() buffer.
627 * The @buf @offset will only be updated on success.
629 * Returns: 0 on success, negative error code on failure.
631 static int append_oa_status(struct i915_perf_stream
*stream
,
635 enum drm_i915_perf_record_type type
)
637 struct drm_i915_perf_record_header header
= { type
, 0, sizeof(header
) };
639 if ((count
- *offset
) < header
.size
)
642 if (copy_to_user(buf
+ *offset
, &header
, sizeof(header
)))
645 (*offset
) += header
.size
;
651 * append_oa_sample - Copies single OA report into userspace read() buffer.
652 * @stream: An i915-perf stream opened for OA metrics
653 * @buf: destination buffer given by userspace
654 * @count: the number of bytes userspace wants to read
655 * @offset: (inout): the current position for writing into @buf
656 * @report: A single OA report to (optionally) include as part of the sample
658 * The contents of a sample are configured through `DRM_I915_PERF_PROP_SAMPLE_*`
659 * properties when opening a stream, tracked as `stream->sample_flags`. This
660 * function copies the requested components of a single sample to the given
663 * The @buf @offset will only be updated on success.
665 * Returns: 0 on success, negative error code on failure.
667 static int append_oa_sample(struct i915_perf_stream
*stream
,
673 int report_size
= stream
->oa_buffer
.format
->size
;
674 struct drm_i915_perf_record_header header
;
675 int report_size_partial
;
678 header
.type
= DRM_I915_PERF_RECORD_SAMPLE
;
680 header
.size
= stream
->sample_size
;
682 if ((count
- *offset
) < header
.size
)
686 if (copy_to_user(buf
, &header
, sizeof(header
)))
688 buf
+= sizeof(header
);
690 oa_buf_end
= stream
->oa_buffer
.vaddr
+ OA_BUFFER_SIZE
;
691 report_size_partial
= oa_buf_end
- report
;
693 if (report_size_partial
< report_size
) {
694 if (copy_to_user(buf
, report
, report_size_partial
))
696 buf
+= report_size_partial
;
698 if (copy_to_user(buf
, stream
->oa_buffer
.vaddr
,
699 report_size
- report_size_partial
))
701 } else if (copy_to_user(buf
, report
, report_size
)) {
705 (*offset
) += header
.size
;
711 * gen8_append_oa_reports - Copies all buffered OA reports into
712 * userspace read() buffer.
713 * @stream: An i915-perf stream opened for OA metrics
714 * @buf: destination buffer given by userspace
715 * @count: the number of bytes userspace wants to read
716 * @offset: (inout): the current position for writing into @buf
718 * Notably any error condition resulting in a short read (-%ENOSPC or
719 * -%EFAULT) will be returned even though one or more records may
720 * have been successfully copied. In this case it's up to the caller
721 * to decide if the error should be squashed before returning to
724 * Note: reports are consumed from the head, and appended to the
725 * tail, so the tail chases the head?... If you think that's mad
726 * and back-to-front you're not alone, but this follows the
727 * Gen PRM naming convention.
729 * Returns: 0 on success, negative error code on failure.
731 static int gen8_append_oa_reports(struct i915_perf_stream
*stream
,
736 struct intel_uncore
*uncore
= stream
->uncore
;
737 int report_size
= stream
->oa_buffer
.format
->size
;
738 u8
*oa_buf_base
= stream
->oa_buffer
.vaddr
;
739 u32 gtt_offset
= i915_ggtt_offset(stream
->oa_buffer
.vma
);
740 u32 mask
= (OA_BUFFER_SIZE
- 1);
741 size_t start_offset
= *offset
;
746 if (drm_WARN_ON(&uncore
->i915
->drm
, !stream
->enabled
))
749 spin_lock_irqsave(&stream
->oa_buffer
.ptr_lock
, flags
);
751 head
= stream
->oa_buffer
.head
;
752 tail
= stream
->oa_buffer
.tail
;
754 spin_unlock_irqrestore(&stream
->oa_buffer
.ptr_lock
, flags
);
757 * NB: oa_buffer.head/tail include the gtt_offset which we don't want
758 * while indexing relative to oa_buf_base.
764 * An out of bounds or misaligned head or tail pointer implies a driver
765 * bug since we validate + align the tail pointers we read from the
766 * hardware and we are in full control of the head pointer which should
767 * only be incremented by multiples of the report size.
769 if (drm_WARN_ONCE(&uncore
->i915
->drm
,
770 head
> OA_BUFFER_SIZE
||
771 tail
> OA_BUFFER_SIZE
,
772 "Inconsistent OA buffer pointers: head = %u, tail = %u\n",
778 OA_TAKEN(tail
, head
);
779 head
= (head
+ report_size
) & mask
) {
780 u8
*report
= oa_buf_base
+ head
;
781 u32
*report32
= (void *)report
;
786 * The reason field includes flags identifying what
787 * triggered this specific report (mostly timer
788 * triggered or e.g. due to a context switch).
790 * In MMIO triggered reports, some platforms do not set the
791 * reason bit in this field and it is valid to have a reason
794 reason
= oa_report_reason(stream
, report
);
795 ctx_id
= oa_context_id(stream
, report32
);
798 * Squash whatever is in the CTX_ID field if it's marked as
799 * invalid to be sure we avoid false-positive, single-context
802 * Note: that we don't clear the valid_ctx_bit so userspace can
803 * understand that the ID has been squashed by the kernel.
805 if (oa_report_ctx_invalid(stream
, report
)) {
806 ctx_id
= INVALID_CTX_ID
;
807 oa_context_id_squash(stream
, report32
);
811 * NB: For Gen 8 the OA unit no longer supports clock gating
812 * off for a specific context and the kernel can't securely
813 * stop the counters from updating as system-wide / global
816 * Automatic reports now include a context ID so reports can be
817 * filtered on the cpu but it's not worth trying to
818 * automatically subtract/hide counter progress for other
819 * contexts while filtering since we can't stop userspace
820 * issuing MI_REPORT_PERF_COUNT commands which would still
821 * provide a side-band view of the real values.
823 * To allow userspace (such as Mesa/GL_INTEL_performance_query)
824 * to normalize counters for a single filtered context then it
825 * needs be forwarded bookend context-switch reports so that it
826 * can track switches in between MI_REPORT_PERF_COUNT commands
827 * and can itself subtract/ignore the progress of counters
828 * associated with other contexts. Note that the hardware
829 * automatically triggers reports when switching to a new
830 * context which are tagged with the ID of the newly active
831 * context. To avoid the complexity (and likely fragility) of
832 * reading ahead while parsing reports to try and minimize
833 * forwarding redundant context switch reports (i.e. between
834 * other, unrelated contexts) we simply elect to forward them
837 * We don't rely solely on the reason field to identify context
838 * switches since it's not-uncommon for periodic samples to
839 * identify a switch before any 'context switch' report.
842 stream
->specific_ctx_id
== ctx_id
||
843 stream
->oa_buffer
.last_ctx_id
== stream
->specific_ctx_id
||
844 reason
& OAREPORT_REASON_CTX_SWITCH
) {
847 * While filtering for a single context we avoid
848 * leaking the IDs of other contexts.
851 stream
->specific_ctx_id
!= ctx_id
) {
852 oa_context_id_squash(stream
, report32
);
855 ret
= append_oa_sample(stream
, buf
, count
, offset
,
860 stream
->oa_buffer
.last_ctx_id
= ctx_id
;
863 if (is_power_of_2(report_size
)) {
865 * Clear out the report id and timestamp as a means
866 * to detect unlanded reports.
868 oa_report_id_clear(stream
, report32
);
869 oa_timestamp_clear(stream
, report32
);
871 u8
*oa_buf_end
= stream
->oa_buffer
.vaddr
+
873 u32 part
= oa_buf_end
- (u8
*)report32
;
875 /* Zero out the entire report */
876 if (report_size
<= part
) {
877 memset(report32
, 0, report_size
);
879 memset(report32
, 0, part
);
880 memset(oa_buf_base
, 0, report_size
- part
);
885 if (start_offset
!= *offset
) {
886 i915_reg_t oaheadptr
;
888 oaheadptr
= GRAPHICS_VER(stream
->perf
->i915
) == 12 ?
889 __oa_regs(stream
)->oa_head_ptr
:
892 spin_lock_irqsave(&stream
->oa_buffer
.ptr_lock
, flags
);
895 * We removed the gtt_offset for the copy loop above, indexing
896 * relative to oa_buf_base so put back here...
899 intel_uncore_write(uncore
, oaheadptr
,
900 head
& GEN12_OAG_OAHEADPTR_MASK
);
901 stream
->oa_buffer
.head
= head
;
903 spin_unlock_irqrestore(&stream
->oa_buffer
.ptr_lock
, flags
);
910 * gen8_oa_read - copy status records then buffered OA reports
911 * @stream: An i915-perf stream opened for OA metrics
912 * @buf: destination buffer given by userspace
913 * @count: the number of bytes userspace wants to read
914 * @offset: (inout): the current position for writing into @buf
916 * Checks OA unit status registers and if necessary appends corresponding
917 * status records for userspace (such as for a buffer full condition) and then
918 * initiate appending any buffered OA reports.
920 * Updates @offset according to the number of bytes successfully copied into
921 * the userspace buffer.
923 * NB: some data may be successfully copied to the userspace buffer
924 * even if an error is returned, and this is reflected in the
927 * Returns: zero on success or a negative error code
929 static int gen8_oa_read(struct i915_perf_stream
*stream
,
934 struct intel_uncore
*uncore
= stream
->uncore
;
936 i915_reg_t oastatus_reg
;
939 if (drm_WARN_ON(&uncore
->i915
->drm
, !stream
->oa_buffer
.vaddr
))
942 oastatus_reg
= GRAPHICS_VER(stream
->perf
->i915
) == 12 ?
943 __oa_regs(stream
)->oa_status
:
946 oastatus
= intel_uncore_read(uncore
, oastatus_reg
);
949 * We treat OABUFFER_OVERFLOW as a significant error:
951 * Although theoretically we could handle this more gracefully
952 * sometimes, some Gens don't correctly suppress certain
953 * automatically triggered reports in this condition and so we
954 * have to assume that old reports are now being trampled
957 * Considering how we don't currently give userspace control
958 * over the OA buffer size and always configure a large 16MB
959 * buffer, then a buffer overflow does anyway likely indicate
960 * that something has gone quite badly wrong.
962 if (oastatus
& GEN8_OASTATUS_OABUFFER_OVERFLOW
) {
963 ret
= append_oa_status(stream
, buf
, count
, offset
,
964 DRM_I915_PERF_RECORD_OA_BUFFER_LOST
);
968 drm_dbg(&stream
->perf
->i915
->drm
,
969 "OA buffer overflow (exponent = %d): force restart\n",
970 stream
->period_exponent
);
972 stream
->perf
->ops
.oa_disable(stream
);
973 stream
->perf
->ops
.oa_enable(stream
);
976 * Note: .oa_enable() is expected to re-init the oabuffer and
977 * reset GEN8_OASTATUS for us
979 oastatus
= intel_uncore_read(uncore
, oastatus_reg
);
982 if (oastatus
& GEN8_OASTATUS_REPORT_LOST
) {
983 ret
= append_oa_status(stream
, buf
, count
, offset
,
984 DRM_I915_PERF_RECORD_OA_REPORT_LOST
);
988 intel_uncore_rmw(uncore
, oastatus_reg
,
989 GEN8_OASTATUS_COUNTER_OVERFLOW
|
990 GEN8_OASTATUS_REPORT_LOST
,
991 IS_GRAPHICS_VER(uncore
->i915
, 8, 11) ?
992 (GEN8_OASTATUS_HEAD_POINTER_WRAP
|
993 GEN8_OASTATUS_TAIL_POINTER_WRAP
) : 0);
996 return gen8_append_oa_reports(stream
, buf
, count
, offset
);
1000 * gen7_append_oa_reports - Copies all buffered OA reports into
1001 * userspace read() buffer.
1002 * @stream: An i915-perf stream opened for OA metrics
1003 * @buf: destination buffer given by userspace
1004 * @count: the number of bytes userspace wants to read
1005 * @offset: (inout): the current position for writing into @buf
1007 * Notably any error condition resulting in a short read (-%ENOSPC or
1008 * -%EFAULT) will be returned even though one or more records may
1009 * have been successfully copied. In this case it's up to the caller
1010 * to decide if the error should be squashed before returning to
1013 * Note: reports are consumed from the head, and appended to the
1014 * tail, so the tail chases the head?... If you think that's mad
1015 * and back-to-front you're not alone, but this follows the
1016 * Gen PRM naming convention.
1018 * Returns: 0 on success, negative error code on failure.
1020 static int gen7_append_oa_reports(struct i915_perf_stream
*stream
,
1025 struct intel_uncore
*uncore
= stream
->uncore
;
1026 int report_size
= stream
->oa_buffer
.format
->size
;
1027 u8
*oa_buf_base
= stream
->oa_buffer
.vaddr
;
1028 u32 gtt_offset
= i915_ggtt_offset(stream
->oa_buffer
.vma
);
1029 u32 mask
= (OA_BUFFER_SIZE
- 1);
1030 size_t start_offset
= *offset
;
1031 unsigned long flags
;
1035 if (drm_WARN_ON(&uncore
->i915
->drm
, !stream
->enabled
))
1038 spin_lock_irqsave(&stream
->oa_buffer
.ptr_lock
, flags
);
1040 head
= stream
->oa_buffer
.head
;
1041 tail
= stream
->oa_buffer
.tail
;
1043 spin_unlock_irqrestore(&stream
->oa_buffer
.ptr_lock
, flags
);
1045 /* NB: oa_buffer.head/tail include the gtt_offset which we don't want
1046 * while indexing relative to oa_buf_base.
1051 /* An out of bounds or misaligned head or tail pointer implies a driver
1052 * bug since we validate + align the tail pointers we read from the
1053 * hardware and we are in full control of the head pointer which should
1054 * only be incremented by multiples of the report size (notably also
1055 * all a power of two).
1057 if (drm_WARN_ONCE(&uncore
->i915
->drm
,
1058 head
> OA_BUFFER_SIZE
|| head
% report_size
||
1059 tail
> OA_BUFFER_SIZE
|| tail
% report_size
,
1060 "Inconsistent OA buffer pointers: head = %u, tail = %u\n",
1066 OA_TAKEN(tail
, head
);
1067 head
= (head
+ report_size
) & mask
) {
1068 u8
*report
= oa_buf_base
+ head
;
1069 u32
*report32
= (void *)report
;
1071 /* All the report sizes factor neatly into the buffer
1072 * size so we never expect to see a report split
1073 * between the beginning and end of the buffer.
1075 * Given the initial alignment check a misalignment
1076 * here would imply a driver bug that would result
1079 if (drm_WARN_ON(&uncore
->i915
->drm
,
1080 (OA_BUFFER_SIZE
- head
) < report_size
)) {
1081 drm_err(&uncore
->i915
->drm
,
1082 "Spurious OA head ptr: non-integral report offset\n");
1086 /* The report-ID field for periodic samples includes
1087 * some undocumented flags related to what triggered
1088 * the report and is never expected to be zero so we
1089 * can check that the report isn't invalid before
1090 * copying it to userspace...
1092 if (report32
[0] == 0) {
1093 if (__ratelimit(&stream
->perf
->spurious_report_rs
))
1094 drm_notice(&uncore
->i915
->drm
,
1095 "Skipping spurious, invalid OA report\n");
1099 ret
= append_oa_sample(stream
, buf
, count
, offset
, report
);
1103 /* Clear out the first 2 dwords as a mean to detect unlanded
1110 if (start_offset
!= *offset
) {
1111 spin_lock_irqsave(&stream
->oa_buffer
.ptr_lock
, flags
);
1113 /* We removed the gtt_offset for the copy loop above, indexing
1114 * relative to oa_buf_base so put back here...
1118 intel_uncore_write(uncore
, GEN7_OASTATUS2
,
1119 (head
& GEN7_OASTATUS2_HEAD_MASK
) |
1120 GEN7_OASTATUS2_MEM_SELECT_GGTT
);
1121 stream
->oa_buffer
.head
= head
;
1123 spin_unlock_irqrestore(&stream
->oa_buffer
.ptr_lock
, flags
);
1130 * gen7_oa_read - copy status records then buffered OA reports
1131 * @stream: An i915-perf stream opened for OA metrics
1132 * @buf: destination buffer given by userspace
1133 * @count: the number of bytes userspace wants to read
1134 * @offset: (inout): the current position for writing into @buf
1136 * Checks Gen 7 specific OA unit status registers and if necessary appends
1137 * corresponding status records for userspace (such as for a buffer full
1138 * condition) and then initiate appending any buffered OA reports.
1140 * Updates @offset according to the number of bytes successfully copied into
1141 * the userspace buffer.
1143 * Returns: zero on success or a negative error code
1145 static int gen7_oa_read(struct i915_perf_stream
*stream
,
1150 struct intel_uncore
*uncore
= stream
->uncore
;
1154 if (drm_WARN_ON(&uncore
->i915
->drm
, !stream
->oa_buffer
.vaddr
))
1157 oastatus1
= intel_uncore_read(uncore
, GEN7_OASTATUS1
);
1159 /* XXX: On Haswell we don't have a safe way to clear oastatus1
1160 * bits while the OA unit is enabled (while the tail pointer
1161 * may be updated asynchronously) so we ignore status bits
1162 * that have already been reported to userspace.
1164 oastatus1
&= ~stream
->perf
->gen7_latched_oastatus1
;
1166 /* We treat OABUFFER_OVERFLOW as a significant error:
1168 * - The status can be interpreted to mean that the buffer is
1169 * currently full (with a higher precedence than OA_TAKEN()
1170 * which will start to report a near-empty buffer after an
1171 * overflow) but it's awkward that we can't clear the status
1172 * on Haswell, so without a reset we won't be able to catch
1175 * - Since it also implies the HW has started overwriting old
1176 * reports it may also affect our sanity checks for invalid
1177 * reports when copying to userspace that assume new reports
1178 * are being written to cleared memory.
1180 * - In the future we may want to introduce a flight recorder
1181 * mode where the driver will automatically maintain a safe
1182 * guard band between head/tail, avoiding this overflow
1183 * condition, but we avoid the added driver complexity for
1186 if (unlikely(oastatus1
& GEN7_OASTATUS1_OABUFFER_OVERFLOW
)) {
1187 ret
= append_oa_status(stream
, buf
, count
, offset
,
1188 DRM_I915_PERF_RECORD_OA_BUFFER_LOST
);
1192 drm_dbg(&stream
->perf
->i915
->drm
,
1193 "OA buffer overflow (exponent = %d): force restart\n",
1194 stream
->period_exponent
);
1196 stream
->perf
->ops
.oa_disable(stream
);
1197 stream
->perf
->ops
.oa_enable(stream
);
1199 oastatus1
= intel_uncore_read(uncore
, GEN7_OASTATUS1
);
1202 if (unlikely(oastatus1
& GEN7_OASTATUS1_REPORT_LOST
)) {
1203 ret
= append_oa_status(stream
, buf
, count
, offset
,
1204 DRM_I915_PERF_RECORD_OA_REPORT_LOST
);
1207 stream
->perf
->gen7_latched_oastatus1
|=
1208 GEN7_OASTATUS1_REPORT_LOST
;
1211 return gen7_append_oa_reports(stream
, buf
, count
, offset
);
1215 * i915_oa_wait_unlocked - handles blocking IO until OA data available
1216 * @stream: An i915-perf stream opened for OA metrics
1218 * Called when userspace tries to read() from a blocking stream FD opened
1219 * for OA metrics. It waits until the hrtimer callback finds a non-empty
1220 * OA buffer and wakes us.
1222 * Note: it's acceptable to have this return with some false positives
1223 * since any subsequent read handling will return -EAGAIN if there isn't
1224 * really data ready for userspace yet.
1226 * Returns: zero on success or a negative error code
1228 static int i915_oa_wait_unlocked(struct i915_perf_stream
*stream
)
1230 /* We would wait indefinitely if periodic sampling is not enabled */
1231 if (!stream
->periodic
)
1234 return wait_event_interruptible(stream
->poll_wq
,
1235 oa_buffer_check_unlocked(stream
));
1239 * i915_oa_poll_wait - call poll_wait() for an OA stream poll()
1240 * @stream: An i915-perf stream opened for OA metrics
1241 * @file: An i915 perf stream file
1242 * @wait: poll() state table
1244 * For handling userspace polling on an i915 perf stream opened for OA metrics,
1245 * this starts a poll_wait with the wait queue that our hrtimer callback wakes
1246 * when it sees data ready to read in the circular OA buffer.
1248 static void i915_oa_poll_wait(struct i915_perf_stream
*stream
,
1252 poll_wait(file
, &stream
->poll_wq
, wait
);
1256 * i915_oa_read - just calls through to &i915_oa_ops->read
1257 * @stream: An i915-perf stream opened for OA metrics
1258 * @buf: destination buffer given by userspace
1259 * @count: the number of bytes userspace wants to read
1260 * @offset: (inout): the current position for writing into @buf
1262 * Updates @offset according to the number of bytes successfully copied into
1263 * the userspace buffer.
1265 * Returns: zero on success or a negative error code
1267 static int i915_oa_read(struct i915_perf_stream
*stream
,
1272 return stream
->perf
->ops
.read(stream
, buf
, count
, offset
);
1275 static struct intel_context
*oa_pin_context(struct i915_perf_stream
*stream
)
1277 struct i915_gem_engines_iter it
;
1278 struct i915_gem_context
*ctx
= stream
->ctx
;
1279 struct intel_context
*ce
;
1280 struct i915_gem_ww_ctx ww
;
1283 for_each_gem_engine(ce
, i915_gem_context_lock_engines(ctx
), it
) {
1284 if (ce
->engine
!= stream
->engine
) /* first match! */
1290 i915_gem_context_unlock_engines(ctx
);
1293 return ERR_PTR(err
);
1295 i915_gem_ww_ctx_init(&ww
, true);
1298 * As the ID is the gtt offset of the context's vma we
1299 * pin the vma to ensure the ID remains fixed.
1301 err
= intel_context_pin_ww(ce
, &ww
);
1302 if (err
== -EDEADLK
) {
1303 err
= i915_gem_ww_ctx_backoff(&ww
);
1307 i915_gem_ww_ctx_fini(&ww
);
1310 return ERR_PTR(err
);
1312 stream
->pinned_ctx
= ce
;
1313 return stream
->pinned_ctx
;
1317 __store_reg_to_mem(struct i915_request
*rq
, i915_reg_t reg
, u32 ggtt_offset
)
1321 cmd
= MI_STORE_REGISTER_MEM
| MI_SRM_LRM_GLOBAL_GTT
;
1322 if (GRAPHICS_VER(rq
->i915
) >= 8)
1325 cs
= intel_ring_begin(rq
, 4);
1330 *cs
++ = i915_mmio_reg_offset(reg
);
1331 *cs
++ = ggtt_offset
;
1334 intel_ring_advance(rq
, cs
);
1340 __read_reg(struct intel_context
*ce
, i915_reg_t reg
, u32 ggtt_offset
)
1342 struct i915_request
*rq
;
1345 rq
= i915_request_create(ce
);
1349 i915_request_get(rq
);
1351 err
= __store_reg_to_mem(rq
, reg
, ggtt_offset
);
1353 i915_request_add(rq
);
1354 if (!err
&& i915_request_wait(rq
, 0, HZ
/ 2) < 0)
1357 i915_request_put(rq
);
1363 gen12_guc_sw_ctx_id(struct intel_context
*ce
, u32
*ctx_id
)
1365 struct i915_vma
*scratch
;
1369 scratch
= __vm_create_scratch_for_read_pinned(&ce
->engine
->gt
->ggtt
->vm
, 4);
1370 if (IS_ERR(scratch
))
1371 return PTR_ERR(scratch
);
1373 err
= i915_vma_sync(scratch
);
1377 err
= __read_reg(ce
, RING_EXECLIST_STATUS_HI(ce
->engine
->mmio_base
),
1378 i915_ggtt_offset(scratch
));
1382 val
= i915_gem_object_pin_map_unlocked(scratch
->obj
, I915_MAP_WB
);
1389 i915_gem_object_unpin_map(scratch
->obj
);
1392 i915_vma_unpin_and_release(&scratch
, 0);
1397 * For execlist mode of submission, pick an unused context id
1398 * 0 - (NUM_CONTEXT_TAG -1) are used by other contexts
1399 * XXX_MAX_CONTEXT_HW_ID is used by idle context
1401 * For GuC mode of submission read context id from the upper dword of the
1402 * EXECLIST_STATUS register. Note that we read this value only once and expect
1403 * that the value stays fixed for the entire OA use case. There are cases where
1404 * GuC KMD implementation may deregister a context to reuse it's context id, but
1405 * we prevent that from happening to the OA context by pinning it.
1407 static int gen12_get_render_context_id(struct i915_perf_stream
*stream
)
1412 if (intel_engine_uses_guc(stream
->engine
)) {
1413 ret
= gen12_guc_sw_ctx_id(stream
->pinned_ctx
, &ctx_id
);
1417 mask
= ((1U << GEN12_GUC_SW_CTX_ID_WIDTH
) - 1) <<
1418 (GEN12_GUC_SW_CTX_ID_SHIFT
- 32);
1419 } else if (GRAPHICS_VER_FULL(stream
->engine
->i915
) >= IP_VER(12, 50)) {
1420 ctx_id
= (XEHP_MAX_CONTEXT_HW_ID
- 1) <<
1421 (XEHP_SW_CTX_ID_SHIFT
- 32);
1423 mask
= ((1U << XEHP_SW_CTX_ID_WIDTH
) - 1) <<
1424 (XEHP_SW_CTX_ID_SHIFT
- 32);
1426 ctx_id
= (GEN12_MAX_CONTEXT_HW_ID
- 1) <<
1427 (GEN11_SW_CTX_ID_SHIFT
- 32);
1429 mask
= ((1U << GEN11_SW_CTX_ID_WIDTH
) - 1) <<
1430 (GEN11_SW_CTX_ID_SHIFT
- 32);
1432 stream
->specific_ctx_id
= ctx_id
& mask
;
1433 stream
->specific_ctx_id_mask
= mask
;
1438 static bool oa_find_reg_in_lri(u32
*state
, u32 reg
, u32
*offset
, u32 end
)
1441 u32 len
= min(MI_LRI_LEN(state
[idx
]) + idx
, end
);
1445 for (; idx
< len
; idx
+= 2) {
1446 if (state
[idx
] == reg
) {
1456 static u32
oa_context_image_offset(struct intel_context
*ce
, u32 reg
)
1458 u32 offset
, len
= (ce
->engine
->context_size
- PAGE_SIZE
) / 4;
1459 u32
*state
= ce
->lrc_reg_state
;
1461 if (drm_WARN_ON(&ce
->engine
->i915
->drm
, !state
))
1464 for (offset
= 0; offset
< len
; ) {
1465 if (IS_MI_LRI_CMD(state
[offset
])) {
1467 * We expect reg-value pairs in MI_LRI command, so
1468 * MI_LRI_LEN() should be even, if not, issue a warning.
1470 drm_WARN_ON(&ce
->engine
->i915
->drm
,
1471 MI_LRI_LEN(state
[offset
]) & 0x1);
1473 if (oa_find_reg_in_lri(state
, reg
, &offset
, len
))
1480 return offset
< len
? offset
: U32_MAX
;
1483 static int set_oa_ctx_ctrl_offset(struct intel_context
*ce
)
1485 i915_reg_t reg
= GEN12_OACTXCONTROL(ce
->engine
->mmio_base
);
1486 struct i915_perf
*perf
= &ce
->engine
->i915
->perf
;
1487 u32 offset
= perf
->ctx_oactxctrl_offset
;
1489 /* Do this only once. Failure is stored as offset of U32_MAX */
1493 offset
= oa_context_image_offset(ce
, i915_mmio_reg_offset(reg
));
1494 perf
->ctx_oactxctrl_offset
= offset
;
1496 drm_dbg(&ce
->engine
->i915
->drm
,
1497 "%s oa ctx control at 0x%08x dword offset\n",
1498 ce
->engine
->name
, offset
);
1501 return offset
&& offset
!= U32_MAX
? 0 : -ENODEV
;
1504 static bool engine_supports_mi_query(struct intel_engine_cs
*engine
)
1506 return engine
->class == RENDER_CLASS
;
1510 * oa_get_render_ctx_id - determine and hold ctx hw id
1511 * @stream: An i915-perf stream opened for OA metrics
1513 * Determine the render context hw id, and ensure it remains fixed for the
1514 * lifetime of the stream. This ensures that we don't have to worry about
1515 * updating the context ID in OACONTROL on the fly.
1517 * Returns: zero on success or a negative error code
1519 static int oa_get_render_ctx_id(struct i915_perf_stream
*stream
)
1521 struct intel_context
*ce
;
1524 ce
= oa_pin_context(stream
);
1528 if (engine_supports_mi_query(stream
->engine
) &&
1529 HAS_LOGICAL_RING_CONTEXTS(stream
->perf
->i915
)) {
1531 * We are enabling perf query here. If we don't find the context
1532 * offset here, just return an error.
1534 ret
= set_oa_ctx_ctrl_offset(ce
);
1536 intel_context_unpin(ce
);
1537 drm_err(&stream
->perf
->i915
->drm
,
1538 "Enabling perf query failed for %s\n",
1539 stream
->engine
->name
);
1544 switch (GRAPHICS_VER(ce
->engine
->i915
)) {
1547 * On Haswell we don't do any post processing of the reports
1548 * and don't need to use the mask.
1550 stream
->specific_ctx_id
= i915_ggtt_offset(ce
->state
);
1551 stream
->specific_ctx_id_mask
= 0;
1557 if (intel_engine_uses_guc(ce
->engine
)) {
1559 * When using GuC, the context descriptor we write in
1560 * i915 is read by GuC and rewritten before it's
1561 * actually written into the hardware. The LRCA is
1562 * what is put into the context id field of the
1563 * context descriptor by GuC. Because it's aligned to
1564 * a page, the lower 12bits are always at 0 and
1565 * dropped by GuC. They won't be part of the context
1566 * ID in the OA reports, so squash those lower bits.
1568 stream
->specific_ctx_id
= ce
->lrc
.lrca
>> 12;
1571 * GuC uses the top bit to signal proxy submission, so
1574 stream
->specific_ctx_id_mask
=
1575 (1U << (GEN8_CTX_ID_WIDTH
- 1)) - 1;
1577 stream
->specific_ctx_id_mask
=
1578 (1U << GEN8_CTX_ID_WIDTH
) - 1;
1579 stream
->specific_ctx_id
= stream
->specific_ctx_id_mask
;
1585 ret
= gen12_get_render_context_id(stream
);
1589 MISSING_CASE(GRAPHICS_VER(ce
->engine
->i915
));
1592 ce
->tag
= stream
->specific_ctx_id
;
1594 drm_dbg(&stream
->perf
->i915
->drm
,
1595 "filtering on ctx_id=0x%x ctx_id_mask=0x%x\n",
1596 stream
->specific_ctx_id
,
1597 stream
->specific_ctx_id_mask
);
1603 * oa_put_render_ctx_id - counterpart to oa_get_render_ctx_id releases hold
1604 * @stream: An i915-perf stream opened for OA metrics
1606 * In case anything needed doing to ensure the context HW ID would remain valid
1607 * for the lifetime of the stream, then that can be undone here.
1609 static void oa_put_render_ctx_id(struct i915_perf_stream
*stream
)
1611 struct intel_context
*ce
;
1613 ce
= fetch_and_zero(&stream
->pinned_ctx
);
1615 ce
->tag
= 0; /* recomputed on next submission after parking */
1616 intel_context_unpin(ce
);
1619 stream
->specific_ctx_id
= INVALID_CTX_ID
;
1620 stream
->specific_ctx_id_mask
= 0;
1624 free_oa_buffer(struct i915_perf_stream
*stream
)
1626 i915_vma_unpin_and_release(&stream
->oa_buffer
.vma
,
1627 I915_VMA_RELEASE_MAP
);
1629 stream
->oa_buffer
.vaddr
= NULL
;
1633 free_oa_configs(struct i915_perf_stream
*stream
)
1635 struct i915_oa_config_bo
*oa_bo
, *tmp
;
1637 i915_oa_config_put(stream
->oa_config
);
1638 llist_for_each_entry_safe(oa_bo
, tmp
, stream
->oa_config_bos
.first
, node
)
1639 free_oa_config_bo(oa_bo
);
1643 free_noa_wait(struct i915_perf_stream
*stream
)
1645 i915_vma_unpin_and_release(&stream
->noa_wait
, 0);
1648 static bool engine_supports_oa(const struct intel_engine_cs
*engine
)
1650 return engine
->oa_group
;
1653 static bool engine_supports_oa_format(struct intel_engine_cs
*engine
, int type
)
1655 return engine
->oa_group
&& engine
->oa_group
->type
== type
;
1658 static void i915_oa_stream_destroy(struct i915_perf_stream
*stream
)
1660 struct i915_perf
*perf
= stream
->perf
;
1661 struct intel_gt
*gt
= stream
->engine
->gt
;
1662 struct i915_perf_group
*g
= stream
->engine
->oa_group
;
1664 if (WARN_ON(stream
!= g
->exclusive_stream
))
1668 * Unset exclusive_stream first, it will be checked while disabling
1669 * the metric set on gen8+.
1671 * See i915_oa_init_reg_state() and lrc_configure_all_contexts()
1673 WRITE_ONCE(g
->exclusive_stream
, NULL
);
1674 perf
->ops
.disable_metric_set(stream
);
1676 free_oa_buffer(stream
);
1679 * Wa_16011777198:dg2: Unset the override of GUCRC mode to enable rc6.
1681 if (stream
->override_gucrc
)
1682 drm_WARN_ON(>
->i915
->drm
,
1683 intel_guc_slpc_unset_gucrc_mode(>
->uc
.guc
.slpc
));
1685 intel_uncore_forcewake_put(stream
->uncore
, FORCEWAKE_ALL
);
1686 intel_engine_pm_put(stream
->engine
);
1689 oa_put_render_ctx_id(stream
);
1691 free_oa_configs(stream
);
1692 free_noa_wait(stream
);
1694 if (perf
->spurious_report_rs
.missed
) {
1695 drm_notice(>
->i915
->drm
,
1696 "%d spurious OA report notices suppressed due to ratelimiting\n",
1697 perf
->spurious_report_rs
.missed
);
1701 static void gen7_init_oa_buffer(struct i915_perf_stream
*stream
)
1703 struct intel_uncore
*uncore
= stream
->uncore
;
1704 u32 gtt_offset
= i915_ggtt_offset(stream
->oa_buffer
.vma
);
1705 unsigned long flags
;
1707 spin_lock_irqsave(&stream
->oa_buffer
.ptr_lock
, flags
);
1709 /* Pre-DevBDW: OABUFFER must be set with counters off,
1710 * before OASTATUS1, but after OASTATUS2
1712 intel_uncore_write(uncore
, GEN7_OASTATUS2
, /* head */
1713 gtt_offset
| GEN7_OASTATUS2_MEM_SELECT_GGTT
);
1714 stream
->oa_buffer
.head
= gtt_offset
;
1716 intel_uncore_write(uncore
, GEN7_OABUFFER
, gtt_offset
);
1718 intel_uncore_write(uncore
, GEN7_OASTATUS1
, /* tail */
1719 gtt_offset
| OABUFFER_SIZE_16M
);
1721 /* Mark that we need updated tail pointers to read from... */
1722 stream
->oa_buffer
.tail
= gtt_offset
;
1724 spin_unlock_irqrestore(&stream
->oa_buffer
.ptr_lock
, flags
);
1726 /* On Haswell we have to track which OASTATUS1 flags we've
1727 * already seen since they can't be cleared while periodic
1728 * sampling is enabled.
1730 stream
->perf
->gen7_latched_oastatus1
= 0;
1732 /* NB: although the OA buffer will initially be allocated
1733 * zeroed via shmfs (and so this memset is redundant when
1734 * first allocating), we may re-init the OA buffer, either
1735 * when re-enabling a stream or in error/reset paths.
1737 * The reason we clear the buffer for each re-init is for the
1738 * sanity check in gen7_append_oa_reports() that looks at the
1739 * report-id field to make sure it's non-zero which relies on
1740 * the assumption that new reports are being written to zeroed
1743 memset(stream
->oa_buffer
.vaddr
, 0, OA_BUFFER_SIZE
);
1746 static void gen8_init_oa_buffer(struct i915_perf_stream
*stream
)
1748 struct intel_uncore
*uncore
= stream
->uncore
;
1749 u32 gtt_offset
= i915_ggtt_offset(stream
->oa_buffer
.vma
);
1750 unsigned long flags
;
1752 spin_lock_irqsave(&stream
->oa_buffer
.ptr_lock
, flags
);
1754 intel_uncore_write(uncore
, GEN8_OASTATUS
, 0);
1755 intel_uncore_write(uncore
, GEN8_OAHEADPTR
, gtt_offset
);
1756 stream
->oa_buffer
.head
= gtt_offset
;
1758 intel_uncore_write(uncore
, GEN8_OABUFFER_UDW
, 0);
1763 * "This MMIO must be set before the OATAILPTR
1764 * register and after the OAHEADPTR register. This is
1765 * to enable proper functionality of the overflow
1768 intel_uncore_write(uncore
, GEN8_OABUFFER
, gtt_offset
|
1769 OABUFFER_SIZE_16M
| GEN8_OABUFFER_MEM_SELECT_GGTT
);
1770 intel_uncore_write(uncore
, GEN8_OATAILPTR
, gtt_offset
& GEN8_OATAILPTR_MASK
);
1772 /* Mark that we need updated tail pointers to read from... */
1773 stream
->oa_buffer
.tail
= gtt_offset
;
1776 * Reset state used to recognise context switches, affecting which
1777 * reports we will forward to userspace while filtering for a single
1780 stream
->oa_buffer
.last_ctx_id
= INVALID_CTX_ID
;
1782 spin_unlock_irqrestore(&stream
->oa_buffer
.ptr_lock
, flags
);
1785 * NB: although the OA buffer will initially be allocated
1786 * zeroed via shmfs (and so this memset is redundant when
1787 * first allocating), we may re-init the OA buffer, either
1788 * when re-enabling a stream or in error/reset paths.
1790 * The reason we clear the buffer for each re-init is for the
1791 * sanity check in gen8_append_oa_reports() that looks at the
1792 * reason field to make sure it's non-zero which relies on
1793 * the assumption that new reports are being written to zeroed
1796 memset(stream
->oa_buffer
.vaddr
, 0, OA_BUFFER_SIZE
);
1799 static void gen12_init_oa_buffer(struct i915_perf_stream
*stream
)
1801 struct intel_uncore
*uncore
= stream
->uncore
;
1802 u32 gtt_offset
= i915_ggtt_offset(stream
->oa_buffer
.vma
);
1803 unsigned long flags
;
1805 spin_lock_irqsave(&stream
->oa_buffer
.ptr_lock
, flags
);
1807 intel_uncore_write(uncore
, __oa_regs(stream
)->oa_status
, 0);
1808 intel_uncore_write(uncore
, __oa_regs(stream
)->oa_head_ptr
,
1809 gtt_offset
& GEN12_OAG_OAHEADPTR_MASK
);
1810 stream
->oa_buffer
.head
= gtt_offset
;
1815 * "This MMIO must be set before the OATAILPTR
1816 * register and after the OAHEADPTR register. This is
1817 * to enable proper functionality of the overflow
1820 intel_uncore_write(uncore
, __oa_regs(stream
)->oa_buffer
, gtt_offset
|
1821 OABUFFER_SIZE_16M
| GEN8_OABUFFER_MEM_SELECT_GGTT
);
1822 intel_uncore_write(uncore
, __oa_regs(stream
)->oa_tail_ptr
,
1823 gtt_offset
& GEN12_OAG_OATAILPTR_MASK
);
1825 /* Mark that we need updated tail pointers to read from... */
1826 stream
->oa_buffer
.tail
= gtt_offset
;
1829 * Reset state used to recognise context switches, affecting which
1830 * reports we will forward to userspace while filtering for a single
1833 stream
->oa_buffer
.last_ctx_id
= INVALID_CTX_ID
;
1835 spin_unlock_irqrestore(&stream
->oa_buffer
.ptr_lock
, flags
);
1838 * NB: although the OA buffer will initially be allocated
1839 * zeroed via shmfs (and so this memset is redundant when
1840 * first allocating), we may re-init the OA buffer, either
1841 * when re-enabling a stream or in error/reset paths.
1843 * The reason we clear the buffer for each re-init is for the
1844 * sanity check in gen8_append_oa_reports() that looks at the
1845 * reason field to make sure it's non-zero which relies on
1846 * the assumption that new reports are being written to zeroed
1849 memset(stream
->oa_buffer
.vaddr
, 0,
1850 stream
->oa_buffer
.vma
->size
);
1853 static int alloc_oa_buffer(struct i915_perf_stream
*stream
)
1855 struct drm_i915_private
*i915
= stream
->perf
->i915
;
1856 struct intel_gt
*gt
= stream
->engine
->gt
;
1857 struct drm_i915_gem_object
*bo
;
1858 struct i915_vma
*vma
;
1861 if (drm_WARN_ON(&i915
->drm
, stream
->oa_buffer
.vma
))
1864 BUILD_BUG_ON_NOT_POWER_OF_2(OA_BUFFER_SIZE
);
1865 BUILD_BUG_ON(OA_BUFFER_SIZE
< SZ_128K
|| OA_BUFFER_SIZE
> SZ_16M
);
1867 bo
= i915_gem_object_create_shmem(stream
->perf
->i915
, OA_BUFFER_SIZE
);
1869 drm_err(&i915
->drm
, "Failed to allocate OA buffer\n");
1873 i915_gem_object_set_cache_coherency(bo
, I915_CACHE_LLC
);
1875 /* PreHSW required 512K alignment, HSW requires 16M */
1876 vma
= i915_vma_instance(bo
, >
->ggtt
->vm
, NULL
);
1883 * PreHSW required 512K alignment.
1884 * HSW and onwards, align to requested size of OA buffer.
1886 ret
= i915_vma_pin(vma
, 0, SZ_16M
, PIN_GLOBAL
| PIN_HIGH
);
1888 drm_err(>
->i915
->drm
, "Failed to pin OA buffer %d\n", ret
);
1892 stream
->oa_buffer
.vma
= vma
;
1894 stream
->oa_buffer
.vaddr
=
1895 i915_gem_object_pin_map_unlocked(bo
, I915_MAP_WB
);
1896 if (IS_ERR(stream
->oa_buffer
.vaddr
)) {
1897 ret
= PTR_ERR(stream
->oa_buffer
.vaddr
);
1904 __i915_vma_unpin(vma
);
1907 i915_gem_object_put(bo
);
1909 stream
->oa_buffer
.vaddr
= NULL
;
1910 stream
->oa_buffer
.vma
= NULL
;
1915 static u32
*save_restore_register(struct i915_perf_stream
*stream
, u32
*cs
,
1916 bool save
, i915_reg_t reg
, u32 offset
,
1922 cmd
= save
? MI_STORE_REGISTER_MEM
: MI_LOAD_REGISTER_MEM
;
1923 cmd
|= MI_SRM_LRM_GLOBAL_GTT
;
1924 if (GRAPHICS_VER(stream
->perf
->i915
) >= 8)
1927 for (d
= 0; d
< dword_count
; d
++) {
1929 *cs
++ = i915_mmio_reg_offset(reg
) + 4 * d
;
1930 *cs
++ = i915_ggtt_offset(stream
->noa_wait
) + offset
+ 4 * d
;
1937 static int alloc_noa_wait(struct i915_perf_stream
*stream
)
1939 struct drm_i915_private
*i915
= stream
->perf
->i915
;
1940 struct intel_gt
*gt
= stream
->engine
->gt
;
1941 struct drm_i915_gem_object
*bo
;
1942 struct i915_vma
*vma
;
1943 const u64 delay_ticks
= 0xffffffffffffffff -
1944 intel_gt_ns_to_clock_interval(to_gt(stream
->perf
->i915
),
1945 atomic64_read(&stream
->perf
->noa_programming_delay
));
1946 const u32 base
= stream
->engine
->mmio_base
;
1947 #define CS_GPR(x) GEN8_RING_CS_GPR(base, x)
1948 u32
*batch
, *ts0
, *cs
, *jump
;
1949 struct i915_gem_ww_ctx ww
;
1959 i915_reg_t mi_predicate_result
= HAS_MI_SET_PREDICATE(i915
) ?
1960 MI_PREDICATE_RESULT_2_ENGINE(base
) :
1961 MI_PREDICATE_RESULT_1(RENDER_RING_BASE
);
1964 * gt->scratch was being used to save/restore the GPR registers, but on
1965 * MTL the scratch uses stolen lmem. An MI_SRM to this memory region
1966 * causes an engine hang. Instead allocate an additional page here to
1967 * save/restore GPR registers
1969 bo
= i915_gem_object_create_internal(i915
, 8192);
1972 "Failed to allocate NOA wait batchbuffer\n");
1976 i915_gem_ww_ctx_init(&ww
, true);
1978 ret
= i915_gem_object_lock(bo
, &ww
);
1983 * We pin in GGTT because we jump into this buffer now because
1984 * multiple OA config BOs will have a jump to this address and it
1985 * needs to be fixed during the lifetime of the i915/perf stream.
1987 vma
= i915_vma_instance(bo
, >
->ggtt
->vm
, NULL
);
1993 ret
= i915_vma_pin_ww(vma
, &ww
, 0, 0, PIN_GLOBAL
| PIN_HIGH
);
1997 batch
= cs
= i915_gem_object_pin_map(bo
, I915_MAP_WB
);
1998 if (IS_ERR(batch
)) {
1999 ret
= PTR_ERR(batch
);
2003 stream
->noa_wait
= vma
;
2005 #define GPR_SAVE_OFFSET 4096
2006 #define PREDICATE_SAVE_OFFSET 4160
2008 /* Save registers. */
2009 for (i
= 0; i
< N_CS_GPR
; i
++)
2010 cs
= save_restore_register(
2011 stream
, cs
, true /* save */, CS_GPR(i
),
2012 GPR_SAVE_OFFSET
+ 8 * i
, 2);
2013 cs
= save_restore_register(
2014 stream
, cs
, true /* save */, mi_predicate_result
,
2015 PREDICATE_SAVE_OFFSET
, 1);
2017 /* First timestamp snapshot location. */
2021 * Initial snapshot of the timestamp register to implement the wait.
2022 * We work with 32b values, so clear out the top 32b bits of the
2023 * register because the ALU works 64bits.
2025 *cs
++ = MI_LOAD_REGISTER_IMM(1);
2026 *cs
++ = i915_mmio_reg_offset(CS_GPR(START_TS
)) + 4;
2028 *cs
++ = MI_LOAD_REGISTER_REG
| (3 - 2);
2029 *cs
++ = i915_mmio_reg_offset(RING_TIMESTAMP(base
));
2030 *cs
++ = i915_mmio_reg_offset(CS_GPR(START_TS
));
2033 * This is the location we're going to jump back into until the
2034 * required amount of time has passed.
2039 * Take another snapshot of the timestamp register. Take care to clear
2040 * up the top 32bits of CS_GPR(1) as we're using it for other
2043 *cs
++ = MI_LOAD_REGISTER_IMM(1);
2044 *cs
++ = i915_mmio_reg_offset(CS_GPR(NOW_TS
)) + 4;
2046 *cs
++ = MI_LOAD_REGISTER_REG
| (3 - 2);
2047 *cs
++ = i915_mmio_reg_offset(RING_TIMESTAMP(base
));
2048 *cs
++ = i915_mmio_reg_offset(CS_GPR(NOW_TS
));
2051 * Do a diff between the 2 timestamps and store the result back into
2055 *cs
++ = MI_MATH_LOAD(MI_MATH_REG_SRCA
, MI_MATH_REG(NOW_TS
));
2056 *cs
++ = MI_MATH_LOAD(MI_MATH_REG_SRCB
, MI_MATH_REG(START_TS
));
2057 *cs
++ = MI_MATH_SUB
;
2058 *cs
++ = MI_MATH_STORE(MI_MATH_REG(DELTA_TS
), MI_MATH_REG_ACCU
);
2059 *cs
++ = MI_MATH_STORE(MI_MATH_REG(JUMP_PREDICATE
), MI_MATH_REG_CF
);
2062 * Transfer the carry flag (set to 1 if ts1 < ts0, meaning the
2063 * timestamp have rolled over the 32bits) into the predicate register
2064 * to be used for the predicated jump.
2066 *cs
++ = MI_LOAD_REGISTER_REG
| (3 - 2);
2067 *cs
++ = i915_mmio_reg_offset(CS_GPR(JUMP_PREDICATE
));
2068 *cs
++ = i915_mmio_reg_offset(mi_predicate_result
);
2070 if (HAS_MI_SET_PREDICATE(i915
))
2071 *cs
++ = MI_SET_PREDICATE
| 1;
2073 /* Restart from the beginning if we had timestamps roll over. */
2074 *cs
++ = (GRAPHICS_VER(i915
) < 8 ?
2075 MI_BATCH_BUFFER_START
:
2076 MI_BATCH_BUFFER_START_GEN8
) |
2078 *cs
++ = i915_ggtt_offset(vma
) + (ts0
- batch
) * 4;
2081 if (HAS_MI_SET_PREDICATE(i915
))
2082 *cs
++ = MI_SET_PREDICATE
;
2085 * Now add the diff between to previous timestamps and add it to :
2086 * (((1 * << 64) - 1) - delay_ns)
2088 * When the Carry Flag contains 1 this means the elapsed time is
2089 * longer than the expected delay, and we can exit the wait loop.
2091 *cs
++ = MI_LOAD_REGISTER_IMM(2);
2092 *cs
++ = i915_mmio_reg_offset(CS_GPR(DELTA_TARGET
));
2093 *cs
++ = lower_32_bits(delay_ticks
);
2094 *cs
++ = i915_mmio_reg_offset(CS_GPR(DELTA_TARGET
)) + 4;
2095 *cs
++ = upper_32_bits(delay_ticks
);
2098 *cs
++ = MI_MATH_LOAD(MI_MATH_REG_SRCA
, MI_MATH_REG(DELTA_TS
));
2099 *cs
++ = MI_MATH_LOAD(MI_MATH_REG_SRCB
, MI_MATH_REG(DELTA_TARGET
));
2100 *cs
++ = MI_MATH_ADD
;
2101 *cs
++ = MI_MATH_STOREINV(MI_MATH_REG(JUMP_PREDICATE
), MI_MATH_REG_CF
);
2103 *cs
++ = MI_ARB_CHECK
;
2106 * Transfer the result into the predicate register to be used for the
2109 *cs
++ = MI_LOAD_REGISTER_REG
| (3 - 2);
2110 *cs
++ = i915_mmio_reg_offset(CS_GPR(JUMP_PREDICATE
));
2111 *cs
++ = i915_mmio_reg_offset(mi_predicate_result
);
2113 if (HAS_MI_SET_PREDICATE(i915
))
2114 *cs
++ = MI_SET_PREDICATE
| 1;
2116 /* Predicate the jump. */
2117 *cs
++ = (GRAPHICS_VER(i915
) < 8 ?
2118 MI_BATCH_BUFFER_START
:
2119 MI_BATCH_BUFFER_START_GEN8
) |
2121 *cs
++ = i915_ggtt_offset(vma
) + (jump
- batch
) * 4;
2124 if (HAS_MI_SET_PREDICATE(i915
))
2125 *cs
++ = MI_SET_PREDICATE
;
2127 /* Restore registers. */
2128 for (i
= 0; i
< N_CS_GPR
; i
++)
2129 cs
= save_restore_register(
2130 stream
, cs
, false /* restore */, CS_GPR(i
),
2131 GPR_SAVE_OFFSET
+ 8 * i
, 2);
2132 cs
= save_restore_register(
2133 stream
, cs
, false /* restore */, mi_predicate_result
,
2134 PREDICATE_SAVE_OFFSET
, 1);
2136 /* And return to the ring. */
2137 *cs
++ = MI_BATCH_BUFFER_END
;
2139 GEM_BUG_ON(cs
- batch
> PAGE_SIZE
/ sizeof(*batch
));
2141 i915_gem_object_flush_map(bo
);
2142 __i915_gem_object_release_map(bo
);
2147 i915_vma_unpin_and_release(&vma
, 0);
2149 if (ret
== -EDEADLK
) {
2150 ret
= i915_gem_ww_ctx_backoff(&ww
);
2154 i915_gem_ww_ctx_fini(&ww
);
2156 i915_gem_object_put(bo
);
2160 static u32
*write_cs_mi_lri(u32
*cs
,
2161 const struct i915_oa_reg
*reg_data
,
2166 for (i
= 0; i
< n_regs
; i
++) {
2167 if ((i
% MI_LOAD_REGISTER_IMM_MAX_REGS
) == 0) {
2168 u32 n_lri
= min_t(u32
,
2170 MI_LOAD_REGISTER_IMM_MAX_REGS
);
2172 *cs
++ = MI_LOAD_REGISTER_IMM(n_lri
);
2174 *cs
++ = i915_mmio_reg_offset(reg_data
[i
].addr
);
2175 *cs
++ = reg_data
[i
].value
;
2181 static int num_lri_dwords(int num_regs
)
2186 count
+= DIV_ROUND_UP(num_regs
, MI_LOAD_REGISTER_IMM_MAX_REGS
);
2187 count
+= num_regs
* 2;
2193 static struct i915_oa_config_bo
*
2194 alloc_oa_config_buffer(struct i915_perf_stream
*stream
,
2195 struct i915_oa_config
*oa_config
)
2197 struct drm_i915_gem_object
*obj
;
2198 struct i915_oa_config_bo
*oa_bo
;
2199 struct i915_gem_ww_ctx ww
;
2200 size_t config_length
= 0;
2204 oa_bo
= kzalloc(sizeof(*oa_bo
), GFP_KERNEL
);
2206 return ERR_PTR(-ENOMEM
);
2208 config_length
+= num_lri_dwords(oa_config
->mux_regs_len
);
2209 config_length
+= num_lri_dwords(oa_config
->b_counter_regs_len
);
2210 config_length
+= num_lri_dwords(oa_config
->flex_regs_len
);
2211 config_length
+= 3; /* MI_BATCH_BUFFER_START */
2212 config_length
= ALIGN(sizeof(u32
) * config_length
, I915_GTT_PAGE_SIZE
);
2214 obj
= i915_gem_object_create_shmem(stream
->perf
->i915
, config_length
);
2220 i915_gem_ww_ctx_init(&ww
, true);
2222 err
= i915_gem_object_lock(obj
, &ww
);
2226 cs
= i915_gem_object_pin_map(obj
, I915_MAP_WB
);
2232 cs
= write_cs_mi_lri(cs
,
2233 oa_config
->mux_regs
,
2234 oa_config
->mux_regs_len
);
2235 cs
= write_cs_mi_lri(cs
,
2236 oa_config
->b_counter_regs
,
2237 oa_config
->b_counter_regs_len
);
2238 cs
= write_cs_mi_lri(cs
,
2239 oa_config
->flex_regs
,
2240 oa_config
->flex_regs_len
);
2242 /* Jump into the active wait. */
2243 *cs
++ = (GRAPHICS_VER(stream
->perf
->i915
) < 8 ?
2244 MI_BATCH_BUFFER_START
:
2245 MI_BATCH_BUFFER_START_GEN8
);
2246 *cs
++ = i915_ggtt_offset(stream
->noa_wait
);
2249 i915_gem_object_flush_map(obj
);
2250 __i915_gem_object_release_map(obj
);
2252 oa_bo
->vma
= i915_vma_instance(obj
,
2253 &stream
->engine
->gt
->ggtt
->vm
,
2255 if (IS_ERR(oa_bo
->vma
)) {
2256 err
= PTR_ERR(oa_bo
->vma
);
2260 oa_bo
->oa_config
= i915_oa_config_get(oa_config
);
2261 llist_add(&oa_bo
->node
, &stream
->oa_config_bos
);
2264 if (err
== -EDEADLK
) {
2265 err
= i915_gem_ww_ctx_backoff(&ww
);
2269 i915_gem_ww_ctx_fini(&ww
);
2272 i915_gem_object_put(obj
);
2276 return ERR_PTR(err
);
2281 static struct i915_vma
*
2282 get_oa_vma(struct i915_perf_stream
*stream
, struct i915_oa_config
*oa_config
)
2284 struct i915_oa_config_bo
*oa_bo
;
2287 * Look for the buffer in the already allocated BOs attached
2290 llist_for_each_entry(oa_bo
, stream
->oa_config_bos
.first
, node
) {
2291 if (oa_bo
->oa_config
== oa_config
&&
2292 memcmp(oa_bo
->oa_config
->uuid
,
2294 sizeof(oa_config
->uuid
)) == 0)
2298 oa_bo
= alloc_oa_config_buffer(stream
, oa_config
);
2300 return ERR_CAST(oa_bo
);
2303 return i915_vma_get(oa_bo
->vma
);
2307 emit_oa_config(struct i915_perf_stream
*stream
,
2308 struct i915_oa_config
*oa_config
,
2309 struct intel_context
*ce
,
2310 struct i915_active
*active
)
2312 struct i915_request
*rq
;
2313 struct i915_vma
*vma
;
2314 struct i915_gem_ww_ctx ww
;
2317 vma
= get_oa_vma(stream
, oa_config
);
2319 return PTR_ERR(vma
);
2321 i915_gem_ww_ctx_init(&ww
, true);
2323 err
= i915_gem_object_lock(vma
->obj
, &ww
);
2327 err
= i915_vma_pin_ww(vma
, &ww
, 0, 0, PIN_GLOBAL
| PIN_HIGH
);
2331 intel_engine_pm_get(ce
->engine
);
2332 rq
= i915_request_create(ce
);
2333 intel_engine_pm_put(ce
->engine
);
2339 if (!IS_ERR_OR_NULL(active
)) {
2340 /* After all individual context modifications */
2341 err
= i915_request_await_active(rq
, active
,
2342 I915_ACTIVE_AWAIT_ACTIVE
);
2344 goto err_add_request
;
2346 err
= i915_active_add_request(active
, rq
);
2348 goto err_add_request
;
2351 err
= i915_vma_move_to_active(vma
, rq
, 0);
2353 goto err_add_request
;
2355 err
= rq
->engine
->emit_bb_start(rq
,
2356 i915_vma_offset(vma
), 0,
2357 I915_DISPATCH_SECURE
);
2359 goto err_add_request
;
2362 i915_request_add(rq
);
2364 i915_vma_unpin(vma
);
2366 if (err
== -EDEADLK
) {
2367 err
= i915_gem_ww_ctx_backoff(&ww
);
2372 i915_gem_ww_ctx_fini(&ww
);
2377 static struct intel_context
*oa_context(struct i915_perf_stream
*stream
)
2379 return stream
->pinned_ctx
?: stream
->engine
->kernel_context
;
2383 hsw_enable_metric_set(struct i915_perf_stream
*stream
,
2384 struct i915_active
*active
)
2386 struct intel_uncore
*uncore
= stream
->uncore
;
2391 * OA unit is using “crclk” for its functionality. When trunk
2392 * level clock gating takes place, OA clock would be gated,
2393 * unable to count the events from non-render clock domain.
2394 * Render clock gating must be disabled when OA is enabled to
2395 * count the events from non-render domain. Unit level clock
2396 * gating for RCS should also be disabled.
2398 intel_uncore_rmw(uncore
, GEN7_MISCCPCTL
,
2399 GEN7_DOP_CLOCK_GATE_ENABLE
, 0);
2400 intel_uncore_rmw(uncore
, GEN6_UCGCTL1
,
2401 0, GEN6_CSUNIT_CLOCK_GATE_DISABLE
);
2403 return emit_oa_config(stream
,
2404 stream
->oa_config
, oa_context(stream
),
2408 static void hsw_disable_metric_set(struct i915_perf_stream
*stream
)
2410 struct intel_uncore
*uncore
= stream
->uncore
;
2412 intel_uncore_rmw(uncore
, GEN6_UCGCTL1
,
2413 GEN6_CSUNIT_CLOCK_GATE_DISABLE
, 0);
2414 intel_uncore_rmw(uncore
, GEN7_MISCCPCTL
,
2415 0, GEN7_DOP_CLOCK_GATE_ENABLE
);
2417 intel_uncore_rmw(uncore
, GDT_CHICKEN_BITS
, GT_NOA_ENABLE
, 0);
2420 static u32
oa_config_flex_reg(const struct i915_oa_config
*oa_config
,
2423 u32 mmio
= i915_mmio_reg_offset(reg
);
2427 * This arbitrary default will select the 'EU FPU0 Pipeline
2428 * Active' event. In the future it's anticipated that there
2429 * will be an explicit 'No Event' we can select, but not yet...
2434 for (i
= 0; i
< oa_config
->flex_regs_len
; i
++) {
2435 if (i915_mmio_reg_offset(oa_config
->flex_regs
[i
].addr
) == mmio
)
2436 return oa_config
->flex_regs
[i
].value
;
2442 * NB: It must always remain pointer safe to run this even if the OA unit
2443 * has been disabled.
2445 * It's fine to put out-of-date values into these per-context registers
2446 * in the case that the OA unit has been disabled.
2449 gen8_update_reg_state_unlocked(const struct intel_context
*ce
,
2450 const struct i915_perf_stream
*stream
)
2452 u32 ctx_oactxctrl
= stream
->perf
->ctx_oactxctrl_offset
;
2453 u32 ctx_flexeu0
= stream
->perf
->ctx_flexeu0_offset
;
2454 /* The MMIO offsets for Flex EU registers aren't contiguous */
2455 static const i915_reg_t flex_regs
[] = {
2464 u32
*reg_state
= ce
->lrc_reg_state
;
2467 reg_state
[ctx_oactxctrl
+ 1] =
2468 (stream
->period_exponent
<< GEN8_OA_TIMER_PERIOD_SHIFT
) |
2469 (stream
->periodic
? GEN8_OA_TIMER_ENABLE
: 0) |
2470 GEN8_OA_COUNTER_RESUME
;
2472 for (i
= 0; i
< ARRAY_SIZE(flex_regs
); i
++)
2473 reg_state
[ctx_flexeu0
+ i
* 2 + 1] =
2474 oa_config_flex_reg(stream
->oa_config
, flex_regs
[i
]);
2484 gen8_store_flex(struct i915_request
*rq
,
2485 struct intel_context
*ce
,
2486 const struct flex
*flex
, unsigned int count
)
2491 cs
= intel_ring_begin(rq
, 4 * count
);
2495 offset
= i915_ggtt_offset(ce
->state
) + LRC_STATE_OFFSET
;
2497 *cs
++ = MI_STORE_DWORD_IMM_GEN4
| MI_USE_GGTT
;
2498 *cs
++ = offset
+ flex
->offset
* sizeof(u32
);
2500 *cs
++ = flex
->value
;
2501 } while (flex
++, --count
);
2503 intel_ring_advance(rq
, cs
);
2509 gen8_load_flex(struct i915_request
*rq
,
2510 struct intel_context
*ce
,
2511 const struct flex
*flex
, unsigned int count
)
2515 GEM_BUG_ON(!count
|| count
> 63);
2517 cs
= intel_ring_begin(rq
, 2 * count
+ 2);
2521 *cs
++ = MI_LOAD_REGISTER_IMM(count
);
2523 *cs
++ = i915_mmio_reg_offset(flex
->reg
);
2524 *cs
++ = flex
->value
;
2525 } while (flex
++, --count
);
2528 intel_ring_advance(rq
, cs
);
2533 static int gen8_modify_context(struct intel_context
*ce
,
2534 const struct flex
*flex
, unsigned int count
)
2536 struct i915_request
*rq
;
2539 rq
= intel_engine_create_kernel_request(ce
->engine
);
2543 /* Serialise with the remote context */
2544 err
= intel_context_prepare_remote_request(ce
, rq
);
2546 err
= gen8_store_flex(rq
, ce
, flex
, count
);
2548 i915_request_add(rq
);
2553 gen8_modify_self(struct intel_context
*ce
,
2554 const struct flex
*flex
, unsigned int count
,
2555 struct i915_active
*active
)
2557 struct i915_request
*rq
;
2560 intel_engine_pm_get(ce
->engine
);
2561 rq
= i915_request_create(ce
);
2562 intel_engine_pm_put(ce
->engine
);
2566 if (!IS_ERR_OR_NULL(active
)) {
2567 err
= i915_active_add_request(active
, rq
);
2569 goto err_add_request
;
2572 err
= gen8_load_flex(rq
, ce
, flex
, count
);
2574 goto err_add_request
;
2577 i915_request_add(rq
);
2581 static int gen8_configure_context(struct i915_perf_stream
*stream
,
2582 struct i915_gem_context
*ctx
,
2583 struct flex
*flex
, unsigned int count
)
2585 struct i915_gem_engines_iter it
;
2586 struct intel_context
*ce
;
2589 for_each_gem_engine(ce
, i915_gem_context_lock_engines(ctx
), it
) {
2590 GEM_BUG_ON(ce
== ce
->engine
->kernel_context
);
2592 if (ce
->engine
->class != RENDER_CLASS
)
2595 /* Otherwise OA settings will be set upon first use */
2596 if (!intel_context_pin_if_active(ce
))
2599 flex
->value
= intel_sseu_make_rpcs(ce
->engine
->gt
, &ce
->sseu
);
2600 err
= gen8_modify_context(ce
, flex
, count
);
2602 intel_context_unpin(ce
);
2606 i915_gem_context_unlock_engines(ctx
);
2611 static int gen12_configure_oar_context(struct i915_perf_stream
*stream
,
2612 struct i915_active
*active
)
2615 struct intel_context
*ce
= stream
->pinned_ctx
;
2616 u32 format
= stream
->oa_buffer
.format
->format
;
2617 u32 offset
= stream
->perf
->ctx_oactxctrl_offset
;
2618 struct flex regs_context
[] = {
2622 active
? GEN8_OA_COUNTER_RESUME
: 0,
2625 /* Offsets in regs_lri are not used since this configuration is only
2626 * applied using LRI. Initialize the correct offsets for posterity.
2628 #define GEN12_OAR_OACONTROL_OFFSET 0x5B0
2629 struct flex regs_lri
[] = {
2631 GEN12_OAR_OACONTROL
,
2632 GEN12_OAR_OACONTROL_OFFSET
+ 1,
2633 (format
<< GEN12_OAR_OACONTROL_COUNTER_FORMAT_SHIFT
) |
2634 (active
? GEN12_OAR_OACONTROL_COUNTER_ENABLE
: 0)
2637 RING_CONTEXT_CONTROL(ce
->engine
->mmio_base
),
2638 CTX_CONTEXT_CONTROL
,
2639 _MASKED_FIELD(GEN12_CTX_CTRL_OAR_CONTEXT_ENABLE
,
2641 GEN12_CTX_CTRL_OAR_CONTEXT_ENABLE
:
2646 /* Modify the context image of pinned context with regs_context */
2647 err
= intel_context_lock_pinned(ce
);
2651 err
= gen8_modify_context(ce
, regs_context
,
2652 ARRAY_SIZE(regs_context
));
2653 intel_context_unlock_pinned(ce
);
2657 /* Apply regs_lri using LRI with pinned context */
2658 return gen8_modify_self(ce
, regs_lri
, ARRAY_SIZE(regs_lri
), active
);
2662 * Manages updating the per-context aspects of the OA stream
2663 * configuration across all contexts.
2665 * The awkward consideration here is that OACTXCONTROL controls the
2666 * exponent for periodic sampling which is primarily used for system
2667 * wide profiling where we'd like a consistent sampling period even in
2668 * the face of context switches.
2670 * Our approach of updating the register state context (as opposed to
2671 * say using a workaround batch buffer) ensures that the hardware
2672 * won't automatically reload an out-of-date timer exponent even
2673 * transiently before a WA BB could be parsed.
2675 * This function needs to:
2676 * - Ensure the currently running context's per-context OA state is
2678 * - Ensure that all existing contexts will have the correct per-context
2679 * OA state if they are scheduled for use.
2680 * - Ensure any new contexts will be initialized with the correct
2681 * per-context OA state.
2683 * Note: it's only the RCS/Render context that has any OA state.
2684 * Note: the first flex register passed must always be R_PWR_CLK_STATE
2687 oa_configure_all_contexts(struct i915_perf_stream
*stream
,
2690 struct i915_active
*active
)
2692 struct drm_i915_private
*i915
= stream
->perf
->i915
;
2693 struct intel_engine_cs
*engine
;
2694 struct intel_gt
*gt
= stream
->engine
->gt
;
2695 struct i915_gem_context
*ctx
, *cn
;
2698 lockdep_assert_held(>
->perf
.lock
);
2701 * The OA register config is setup through the context image. This image
2702 * might be written to by the GPU on context switch (in particular on
2703 * lite-restore). This means we can't safely update a context's image,
2704 * if this context is scheduled/submitted to run on the GPU.
2706 * We could emit the OA register config through the batch buffer but
2707 * this might leave small interval of time where the OA unit is
2708 * configured at an invalid sampling period.
2710 * Note that since we emit all requests from a single ring, there
2711 * is still an implicit global barrier here that may cause a high
2712 * priority context to wait for an otherwise independent low priority
2713 * context. Contexts idle at the time of reconfiguration are not
2714 * trapped behind the barrier.
2716 spin_lock(&i915
->gem
.contexts
.lock
);
2717 list_for_each_entry_safe(ctx
, cn
, &i915
->gem
.contexts
.list
, link
) {
2718 if (!kref_get_unless_zero(&ctx
->ref
))
2721 spin_unlock(&i915
->gem
.contexts
.lock
);
2723 err
= gen8_configure_context(stream
, ctx
, regs
, num_regs
);
2725 i915_gem_context_put(ctx
);
2729 spin_lock(&i915
->gem
.contexts
.lock
);
2730 list_safe_reset_next(ctx
, cn
, link
);
2731 i915_gem_context_put(ctx
);
2733 spin_unlock(&i915
->gem
.contexts
.lock
);
2736 * After updating all other contexts, we need to modify ourselves.
2737 * If we don't modify the kernel_context, we do not get events while
2740 for_each_uabi_engine(engine
, i915
) {
2741 struct intel_context
*ce
= engine
->kernel_context
;
2743 if (engine
->class != RENDER_CLASS
)
2746 regs
[0].value
= intel_sseu_make_rpcs(engine
->gt
, &ce
->sseu
);
2748 err
= gen8_modify_self(ce
, regs
, num_regs
, active
);
2757 gen12_configure_all_contexts(struct i915_perf_stream
*stream
,
2758 const struct i915_oa_config
*oa_config
,
2759 struct i915_active
*active
)
2761 struct flex regs
[] = {
2763 GEN8_R_PWR_CLK_STATE(RENDER_RING_BASE
),
2764 CTX_R_PWR_CLK_STATE
,
2768 if (stream
->engine
->class != RENDER_CLASS
)
2771 return oa_configure_all_contexts(stream
,
2772 regs
, ARRAY_SIZE(regs
),
2777 lrc_configure_all_contexts(struct i915_perf_stream
*stream
,
2778 const struct i915_oa_config
*oa_config
,
2779 struct i915_active
*active
)
2781 u32 ctx_oactxctrl
= stream
->perf
->ctx_oactxctrl_offset
;
2782 /* The MMIO offsets for Flex EU registers aren't contiguous */
2783 const u32 ctx_flexeu0
= stream
->perf
->ctx_flexeu0_offset
;
2784 #define ctx_flexeuN(N) (ctx_flexeu0 + 2 * (N) + 1)
2785 struct flex regs
[] = {
2787 GEN8_R_PWR_CLK_STATE(RENDER_RING_BASE
),
2788 CTX_R_PWR_CLK_STATE
,
2794 { EU_PERF_CNTL0
, ctx_flexeuN(0) },
2795 { EU_PERF_CNTL1
, ctx_flexeuN(1) },
2796 { EU_PERF_CNTL2
, ctx_flexeuN(2) },
2797 { EU_PERF_CNTL3
, ctx_flexeuN(3) },
2798 { EU_PERF_CNTL4
, ctx_flexeuN(4) },
2799 { EU_PERF_CNTL5
, ctx_flexeuN(5) },
2800 { EU_PERF_CNTL6
, ctx_flexeuN(6) },
2806 (stream
->period_exponent
<< GEN8_OA_TIMER_PERIOD_SHIFT
) |
2807 (stream
->periodic
? GEN8_OA_TIMER_ENABLE
: 0) |
2808 GEN8_OA_COUNTER_RESUME
;
2810 for (i
= 2; i
< ARRAY_SIZE(regs
); i
++)
2811 regs
[i
].value
= oa_config_flex_reg(oa_config
, regs
[i
].reg
);
2813 return oa_configure_all_contexts(stream
,
2814 regs
, ARRAY_SIZE(regs
),
2819 gen8_enable_metric_set(struct i915_perf_stream
*stream
,
2820 struct i915_active
*active
)
2822 struct intel_uncore
*uncore
= stream
->uncore
;
2823 struct i915_oa_config
*oa_config
= stream
->oa_config
;
2827 * We disable slice/unslice clock ratio change reports on SKL since
2828 * they are too noisy. The HW generates a lot of redundant reports
2829 * where the ratio hasn't really changed causing a lot of redundant
2830 * work to processes and increasing the chances we'll hit buffer
2833 * Although we don't currently use the 'disable overrun' OABUFFER
2834 * feature it's worth noting that clock ratio reports have to be
2835 * disabled before considering to use that feature since the HW doesn't
2836 * correctly block these reports.
2838 * Currently none of the high-level metrics we have depend on knowing
2839 * this ratio to normalize.
2841 * Note: This register is not power context saved and restored, but
2842 * that's OK considering that we disable RC6 while the OA unit is
2845 * The _INCLUDE_CLK_RATIO bit allows the slice/unslice frequency to
2846 * be read back from automatically triggered reports, as part of the
2849 if (IS_GRAPHICS_VER(stream
->perf
->i915
, 9, 11)) {
2850 intel_uncore_write(uncore
, GEN8_OA_DEBUG
,
2851 _MASKED_BIT_ENABLE(GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS
|
2852 GEN9_OA_DEBUG_INCLUDE_CLK_RATIO
));
2856 * Update all contexts prior writing the mux configurations as we need
2857 * to make sure all slices/subslices are ON before writing to NOA
2860 ret
= lrc_configure_all_contexts(stream
, oa_config
, active
);
2864 return emit_oa_config(stream
,
2865 stream
->oa_config
, oa_context(stream
),
2869 static u32
oag_report_ctx_switches(const struct i915_perf_stream
*stream
)
2871 return _MASKED_FIELD(GEN12_OAG_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS
,
2872 (stream
->sample_flags
& SAMPLE_OA_REPORT
) ?
2873 0 : GEN12_OAG_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS
);
2877 gen12_enable_metric_set(struct i915_perf_stream
*stream
,
2878 struct i915_active
*active
)
2880 struct drm_i915_private
*i915
= stream
->perf
->i915
;
2881 struct intel_uncore
*uncore
= stream
->uncore
;
2882 struct i915_oa_config
*oa_config
= stream
->oa_config
;
2883 bool periodic
= stream
->periodic
;
2884 u32 period_exponent
= stream
->period_exponent
;
2889 * Wa_1508761755:xehpsdv, dg2
2890 * EU NOA signals behave incorrectly if EU clock gating is enabled.
2891 * Disable thread stall DOP gating and EU DOP gating.
2893 if (IS_XEHPSDV(i915
) || IS_DG2(i915
)) {
2894 intel_gt_mcr_multicast_write(uncore
->gt
, GEN8_ROW_CHICKEN
,
2895 _MASKED_BIT_ENABLE(STALL_DOP_GATING_DISABLE
));
2896 intel_uncore_write(uncore
, GEN7_ROW_CHICKEN2
,
2897 _MASKED_BIT_ENABLE(GEN12_DISABLE_DOP_GATING
));
2900 intel_uncore_write(uncore
, __oa_regs(stream
)->oa_debug
,
2901 /* Disable clk ratio reports, like previous Gens. */
2902 _MASKED_BIT_ENABLE(GEN12_OAG_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS
|
2903 GEN12_OAG_OA_DEBUG_INCLUDE_CLK_RATIO
) |
2905 * If the user didn't require OA reports, instruct
2906 * the hardware not to emit ctx switch reports.
2908 oag_report_ctx_switches(stream
));
2910 intel_uncore_write(uncore
, __oa_regs(stream
)->oa_ctx_ctrl
, periodic
?
2911 (GEN12_OAG_OAGLBCTXCTRL_COUNTER_RESUME
|
2912 GEN12_OAG_OAGLBCTXCTRL_TIMER_ENABLE
|
2913 (period_exponent
<< GEN12_OAG_OAGLBCTXCTRL_TIMER_PERIOD_SHIFT
))
2917 * Initialize Super Queue Internal Cnt Register
2918 * Set PMON Enable in order to collect valid metrics.
2919 * Enable byets per clock reporting in OA for XEHPSDV onward.
2921 sqcnt1
= GEN12_SQCNT1_PMON_ENABLE
|
2922 (HAS_OA_BPC_REPORTING(i915
) ? GEN12_SQCNT1_OABPC
: 0);
2924 intel_uncore_rmw(uncore
, GEN12_SQCNT1
, 0, sqcnt1
);
2927 * Update all contexts prior writing the mux configurations as we need
2928 * to make sure all slices/subslices are ON before writing to NOA
2931 ret
= gen12_configure_all_contexts(stream
, oa_config
, active
);
2936 * For Gen12, performance counters are context
2937 * saved/restored. Only enable it for the context that
2941 ret
= gen12_configure_oar_context(stream
, active
);
2946 return emit_oa_config(stream
,
2947 stream
->oa_config
, oa_context(stream
),
2951 static void gen8_disable_metric_set(struct i915_perf_stream
*stream
)
2953 struct intel_uncore
*uncore
= stream
->uncore
;
2955 /* Reset all contexts' slices/subslices configurations. */
2956 lrc_configure_all_contexts(stream
, NULL
, NULL
);
2958 intel_uncore_rmw(uncore
, GDT_CHICKEN_BITS
, GT_NOA_ENABLE
, 0);
2961 static void gen11_disable_metric_set(struct i915_perf_stream
*stream
)
2963 struct intel_uncore
*uncore
= stream
->uncore
;
2965 /* Reset all contexts' slices/subslices configurations. */
2966 lrc_configure_all_contexts(stream
, NULL
, NULL
);
2968 /* Make sure we disable noa to save power. */
2969 intel_uncore_rmw(uncore
, RPM_CONFIG1
, GEN10_GT_NOA_ENABLE
, 0);
2972 static void gen12_disable_metric_set(struct i915_perf_stream
*stream
)
2974 struct intel_uncore
*uncore
= stream
->uncore
;
2975 struct drm_i915_private
*i915
= stream
->perf
->i915
;
2979 * Wa_1508761755:xehpsdv, dg2
2980 * Enable thread stall DOP gating and EU DOP gating.
2982 if (IS_XEHPSDV(i915
) || IS_DG2(i915
)) {
2983 intel_gt_mcr_multicast_write(uncore
->gt
, GEN8_ROW_CHICKEN
,
2984 _MASKED_BIT_DISABLE(STALL_DOP_GATING_DISABLE
));
2985 intel_uncore_write(uncore
, GEN7_ROW_CHICKEN2
,
2986 _MASKED_BIT_DISABLE(GEN12_DISABLE_DOP_GATING
));
2989 /* Reset all contexts' slices/subslices configurations. */
2990 gen12_configure_all_contexts(stream
, NULL
, NULL
);
2992 /* disable the context save/restore or OAR counters */
2994 gen12_configure_oar_context(stream
, NULL
);
2996 /* Make sure we disable noa to save power. */
2997 intel_uncore_rmw(uncore
, RPM_CONFIG1
, GEN10_GT_NOA_ENABLE
, 0);
2999 sqcnt1
= GEN12_SQCNT1_PMON_ENABLE
|
3000 (HAS_OA_BPC_REPORTING(i915
) ? GEN12_SQCNT1_OABPC
: 0);
3002 /* Reset PMON Enable to save power. */
3003 intel_uncore_rmw(uncore
, GEN12_SQCNT1
, sqcnt1
, 0);
3006 static void gen7_oa_enable(struct i915_perf_stream
*stream
)
3008 struct intel_uncore
*uncore
= stream
->uncore
;
3009 struct i915_gem_context
*ctx
= stream
->ctx
;
3010 u32 ctx_id
= stream
->specific_ctx_id
;
3011 bool periodic
= stream
->periodic
;
3012 u32 period_exponent
= stream
->period_exponent
;
3013 u32 report_format
= stream
->oa_buffer
.format
->format
;
3016 * Reset buf pointers so we don't forward reports from before now.
3018 * Think carefully if considering trying to avoid this, since it
3019 * also ensures status flags and the buffer itself are cleared
3020 * in error paths, and we have checks for invalid reports based
3021 * on the assumption that certain fields are written to zeroed
3022 * memory which this helps maintains.
3024 gen7_init_oa_buffer(stream
);
3026 intel_uncore_write(uncore
, GEN7_OACONTROL
,
3027 (ctx_id
& GEN7_OACONTROL_CTX_MASK
) |
3029 GEN7_OACONTROL_TIMER_PERIOD_SHIFT
) |
3030 (periodic
? GEN7_OACONTROL_TIMER_ENABLE
: 0) |
3031 (report_format
<< GEN7_OACONTROL_FORMAT_SHIFT
) |
3032 (ctx
? GEN7_OACONTROL_PER_CTX_ENABLE
: 0) |
3033 GEN7_OACONTROL_ENABLE
);
3036 static void gen8_oa_enable(struct i915_perf_stream
*stream
)
3038 struct intel_uncore
*uncore
= stream
->uncore
;
3039 u32 report_format
= stream
->oa_buffer
.format
->format
;
3042 * Reset buf pointers so we don't forward reports from before now.
3044 * Think carefully if considering trying to avoid this, since it
3045 * also ensures status flags and the buffer itself are cleared
3046 * in error paths, and we have checks for invalid reports based
3047 * on the assumption that certain fields are written to zeroed
3048 * memory which this helps maintains.
3050 gen8_init_oa_buffer(stream
);
3053 * Note: we don't rely on the hardware to perform single context
3054 * filtering and instead filter on the cpu based on the context-id
3057 intel_uncore_write(uncore
, GEN8_OACONTROL
,
3058 (report_format
<< GEN8_OA_REPORT_FORMAT_SHIFT
) |
3059 GEN8_OA_COUNTER_ENABLE
);
3062 static void gen12_oa_enable(struct i915_perf_stream
*stream
)
3064 const struct i915_perf_regs
*regs
;
3068 * If we don't want OA reports from the OA buffer, then we don't even
3069 * need to program the OAG unit.
3071 if (!(stream
->sample_flags
& SAMPLE_OA_REPORT
))
3074 gen12_init_oa_buffer(stream
);
3076 regs
= __oa_regs(stream
);
3077 val
= (stream
->oa_buffer
.format
->format
<< regs
->oa_ctrl_counter_format_shift
) |
3078 GEN12_OAG_OACONTROL_OA_COUNTER_ENABLE
;
3080 intel_uncore_write(stream
->uncore
, regs
->oa_ctrl
, val
);
3084 * i915_oa_stream_enable - handle `I915_PERF_IOCTL_ENABLE` for OA stream
3085 * @stream: An i915 perf stream opened for OA metrics
3087 * [Re]enables hardware periodic sampling according to the period configured
3088 * when opening the stream. This also starts a hrtimer that will periodically
3089 * check for data in the circular OA buffer for notifying userspace (e.g.
3090 * during a read() or poll()).
3092 static void i915_oa_stream_enable(struct i915_perf_stream
*stream
)
3094 stream
->pollin
= false;
3096 stream
->perf
->ops
.oa_enable(stream
);
3098 if (stream
->sample_flags
& SAMPLE_OA_REPORT
)
3099 hrtimer_start(&stream
->poll_check_timer
,
3100 ns_to_ktime(stream
->poll_oa_period
),
3101 HRTIMER_MODE_REL_PINNED
);
3104 static void gen7_oa_disable(struct i915_perf_stream
*stream
)
3106 struct intel_uncore
*uncore
= stream
->uncore
;
3108 intel_uncore_write(uncore
, GEN7_OACONTROL
, 0);
3109 if (intel_wait_for_register(uncore
,
3110 GEN7_OACONTROL
, GEN7_OACONTROL_ENABLE
, 0,
3112 drm_err(&stream
->perf
->i915
->drm
,
3113 "wait for OA to be disabled timed out\n");
3116 static void gen8_oa_disable(struct i915_perf_stream
*stream
)
3118 struct intel_uncore
*uncore
= stream
->uncore
;
3120 intel_uncore_write(uncore
, GEN8_OACONTROL
, 0);
3121 if (intel_wait_for_register(uncore
,
3122 GEN8_OACONTROL
, GEN8_OA_COUNTER_ENABLE
, 0,
3124 drm_err(&stream
->perf
->i915
->drm
,
3125 "wait for OA to be disabled timed out\n");
3128 static void gen12_oa_disable(struct i915_perf_stream
*stream
)
3130 struct intel_uncore
*uncore
= stream
->uncore
;
3132 intel_uncore_write(uncore
, __oa_regs(stream
)->oa_ctrl
, 0);
3133 if (intel_wait_for_register(uncore
,
3134 __oa_regs(stream
)->oa_ctrl
,
3135 GEN12_OAG_OACONTROL_OA_COUNTER_ENABLE
, 0,
3137 drm_err(&stream
->perf
->i915
->drm
,
3138 "wait for OA to be disabled timed out\n");
3140 intel_uncore_write(uncore
, GEN12_OA_TLB_INV_CR
, 1);
3141 if (intel_wait_for_register(uncore
,
3142 GEN12_OA_TLB_INV_CR
,
3145 drm_err(&stream
->perf
->i915
->drm
,
3146 "wait for OA tlb invalidate timed out\n");
3150 * i915_oa_stream_disable - handle `I915_PERF_IOCTL_DISABLE` for OA stream
3151 * @stream: An i915 perf stream opened for OA metrics
3153 * Stops the OA unit from periodically writing counter reports into the
3154 * circular OA buffer. This also stops the hrtimer that periodically checks for
3155 * data in the circular OA buffer, for notifying userspace.
3157 static void i915_oa_stream_disable(struct i915_perf_stream
*stream
)
3159 stream
->perf
->ops
.oa_disable(stream
);
3161 if (stream
->sample_flags
& SAMPLE_OA_REPORT
)
3162 hrtimer_cancel(&stream
->poll_check_timer
);
3165 static const struct i915_perf_stream_ops i915_oa_stream_ops
= {
3166 .destroy
= i915_oa_stream_destroy
,
3167 .enable
= i915_oa_stream_enable
,
3168 .disable
= i915_oa_stream_disable
,
3169 .wait_unlocked
= i915_oa_wait_unlocked
,
3170 .poll_wait
= i915_oa_poll_wait
,
3171 .read
= i915_oa_read
,
3174 static int i915_perf_stream_enable_sync(struct i915_perf_stream
*stream
)
3176 struct i915_active
*active
;
3179 active
= i915_active_create();
3183 err
= stream
->perf
->ops
.enable_metric_set(stream
, active
);
3185 __i915_active_wait(active
, TASK_UNINTERRUPTIBLE
);
3187 i915_active_put(active
);
3192 get_default_sseu_config(struct intel_sseu
*out_sseu
,
3193 struct intel_engine_cs
*engine
)
3195 const struct sseu_dev_info
*devinfo_sseu
= &engine
->gt
->info
.sseu
;
3197 *out_sseu
= intel_sseu_from_device_info(devinfo_sseu
);
3199 if (GRAPHICS_VER(engine
->i915
) == 11) {
3201 * We only need subslice count so it doesn't matter which ones
3202 * we select - just turn off low bits in the amount of half of
3203 * all available subslices per slice.
3205 out_sseu
->subslice_mask
=
3206 ~(~0 << (hweight8(out_sseu
->subslice_mask
) / 2));
3207 out_sseu
->slice_mask
= 0x1;
3212 get_sseu_config(struct intel_sseu
*out_sseu
,
3213 struct intel_engine_cs
*engine
,
3214 const struct drm_i915_gem_context_param_sseu
*drm_sseu
)
3216 if (drm_sseu
->engine
.engine_class
!= engine
->uabi_class
||
3217 drm_sseu
->engine
.engine_instance
!= engine
->uabi_instance
)
3220 return i915_gem_user_to_context_sseu(engine
->gt
, drm_sseu
, out_sseu
);
3224 * OA timestamp frequency = CS timestamp frequency in most platforms. On some
3225 * platforms OA unit ignores the CTC_SHIFT and the 2 timestamps differ. In such
3226 * cases, return the adjusted CS timestamp frequency to the user.
3228 u32
i915_perf_oa_timestamp_frequency(struct drm_i915_private
*i915
)
3231 * Wa_18013179988:dg2
3232 * Wa_14015846243:mtl
3234 if (IS_DG2(i915
) || IS_METEORLAKE(i915
)) {
3235 intel_wakeref_t wakeref
;
3238 with_intel_runtime_pm(to_gt(i915
)->uncore
->rpm
, wakeref
)
3239 reg
= intel_uncore_read(to_gt(i915
)->uncore
, RPM_CONFIG0
);
3241 shift
= REG_FIELD_GET(GEN10_RPM_CONFIG0_CTC_SHIFT_PARAMETER_MASK
,
3244 return to_gt(i915
)->clock_frequency
<< (3 - shift
);
3247 return to_gt(i915
)->clock_frequency
;
3251 * i915_oa_stream_init - validate combined props for OA stream and init
3252 * @stream: An i915 perf stream
3253 * @param: The open parameters passed to `DRM_I915_PERF_OPEN`
3254 * @props: The property state that configures stream (individually validated)
3256 * While read_properties_unlocked() validates properties in isolation it
3257 * doesn't ensure that the combination necessarily makes sense.
3259 * At this point it has been determined that userspace wants a stream of
3260 * OA metrics, but still we need to further validate the combined
3261 * properties are OK.
3263 * If the configuration makes sense then we can allocate memory for
3264 * a circular OA buffer and apply the requested metric set configuration.
3266 * Returns: zero on success or a negative error code.
3268 static int i915_oa_stream_init(struct i915_perf_stream
*stream
,
3269 struct drm_i915_perf_open_param
*param
,
3270 struct perf_open_properties
*props
)
3272 struct drm_i915_private
*i915
= stream
->perf
->i915
;
3273 struct i915_perf
*perf
= stream
->perf
;
3274 struct i915_perf_group
*g
;
3275 struct intel_gt
*gt
;
3278 if (!props
->engine
) {
3279 drm_dbg(&stream
->perf
->i915
->drm
,
3280 "OA engine not specified\n");
3283 gt
= props
->engine
->gt
;
3284 g
= props
->engine
->oa_group
;
3287 * If the sysfs metrics/ directory wasn't registered for some
3288 * reason then don't let userspace try their luck with config
3291 if (!perf
->metrics_kobj
) {
3292 drm_dbg(&stream
->perf
->i915
->drm
,
3293 "OA metrics weren't advertised via sysfs\n");
3297 if (!(props
->sample_flags
& SAMPLE_OA_REPORT
) &&
3298 (GRAPHICS_VER(perf
->i915
) < 12 || !stream
->ctx
)) {
3299 drm_dbg(&stream
->perf
->i915
->drm
,
3300 "Only OA report sampling supported\n");
3304 if (!perf
->ops
.enable_metric_set
) {
3305 drm_dbg(&stream
->perf
->i915
->drm
,
3306 "OA unit not supported\n");
3311 * To avoid the complexity of having to accurately filter
3312 * counter reports and marshal to the appropriate client
3313 * we currently only allow exclusive access
3315 if (g
->exclusive_stream
) {
3316 drm_dbg(&stream
->perf
->i915
->drm
,
3317 "OA unit already in use\n");
3321 if (!props
->oa_format
) {
3322 drm_dbg(&stream
->perf
->i915
->drm
,
3323 "OA report format not specified\n");
3327 stream
->engine
= props
->engine
;
3328 stream
->uncore
= stream
->engine
->gt
->uncore
;
3330 stream
->sample_size
= sizeof(struct drm_i915_perf_record_header
);
3332 stream
->oa_buffer
.format
= &perf
->oa_formats
[props
->oa_format
];
3333 if (drm_WARN_ON(&i915
->drm
, stream
->oa_buffer
.format
->size
== 0))
3336 stream
->sample_flags
= props
->sample_flags
;
3337 stream
->sample_size
+= stream
->oa_buffer
.format
->size
;
3339 stream
->hold_preemption
= props
->hold_preemption
;
3341 stream
->periodic
= props
->oa_periodic
;
3342 if (stream
->periodic
)
3343 stream
->period_exponent
= props
->oa_period_exponent
;
3346 ret
= oa_get_render_ctx_id(stream
);
3348 drm_dbg(&stream
->perf
->i915
->drm
,
3349 "Invalid context id to filter with\n");
3354 ret
= alloc_noa_wait(stream
);
3356 drm_dbg(&stream
->perf
->i915
->drm
,
3357 "Unable to allocate NOA wait batch buffer\n");
3358 goto err_noa_wait_alloc
;
3361 stream
->oa_config
= i915_perf_get_oa_config(perf
, props
->metrics_set
);
3362 if (!stream
->oa_config
) {
3363 drm_dbg(&stream
->perf
->i915
->drm
,
3364 "Invalid OA config id=%i\n", props
->metrics_set
);
3369 /* PRM - observability performance counters:
3371 * OACONTROL, performance counter enable, note:
3373 * "When this bit is set, in order to have coherent counts,
3374 * RC6 power state and trunk clock gating must be disabled.
3375 * This can be achieved by programming MMIO registers as
3376 * 0xA094=0 and 0xA090[31]=1"
3378 * In our case we are expecting that taking pm + FORCEWAKE
3379 * references will effectively disable RC6.
3381 intel_engine_pm_get(stream
->engine
);
3382 intel_uncore_forcewake_get(stream
->uncore
, FORCEWAKE_ALL
);
3385 * Wa_16011777198:dg2: GuC resets render as part of the Wa. This causes
3386 * OA to lose the configuration state. Prevent this by overriding GUCRC
3389 if (intel_uc_uses_guc_rc(>
->uc
) &&
3390 (IS_DG2_GRAPHICS_STEP(gt
->i915
, G10
, STEP_A0
, STEP_C0
) ||
3391 IS_DG2_GRAPHICS_STEP(gt
->i915
, G11
, STEP_A0
, STEP_B0
))) {
3392 ret
= intel_guc_slpc_override_gucrc_mode(>
->uc
.guc
.slpc
,
3393 SLPC_GUCRC_MODE_GUCRC_NO_RC6
);
3395 drm_dbg(&stream
->perf
->i915
->drm
,
3396 "Unable to override gucrc mode\n");
3400 stream
->override_gucrc
= true;
3403 ret
= alloc_oa_buffer(stream
);
3405 goto err_oa_buf_alloc
;
3407 stream
->ops
= &i915_oa_stream_ops
;
3409 stream
->engine
->gt
->perf
.sseu
= props
->sseu
;
3410 WRITE_ONCE(g
->exclusive_stream
, stream
);
3412 ret
= i915_perf_stream_enable_sync(stream
);
3414 drm_dbg(&stream
->perf
->i915
->drm
,
3415 "Unable to enable metric set\n");
3419 drm_dbg(&stream
->perf
->i915
->drm
,
3420 "opening stream oa config uuid=%s\n",
3421 stream
->oa_config
->uuid
);
3423 hrtimer_init(&stream
->poll_check_timer
,
3424 CLOCK_MONOTONIC
, HRTIMER_MODE_REL
);
3425 stream
->poll_check_timer
.function
= oa_poll_check_timer_cb
;
3426 init_waitqueue_head(&stream
->poll_wq
);
3427 spin_lock_init(&stream
->oa_buffer
.ptr_lock
);
3428 mutex_init(&stream
->lock
);
3433 WRITE_ONCE(g
->exclusive_stream
, NULL
);
3434 perf
->ops
.disable_metric_set(stream
);
3436 free_oa_buffer(stream
);
3439 if (stream
->override_gucrc
)
3440 intel_guc_slpc_unset_gucrc_mode(>
->uc
.guc
.slpc
);
3443 intel_uncore_forcewake_put(stream
->uncore
, FORCEWAKE_ALL
);
3444 intel_engine_pm_put(stream
->engine
);
3446 free_oa_configs(stream
);
3449 free_noa_wait(stream
);
3453 oa_put_render_ctx_id(stream
);
3458 void i915_oa_init_reg_state(const struct intel_context
*ce
,
3459 const struct intel_engine_cs
*engine
)
3461 struct i915_perf_stream
*stream
;
3463 if (engine
->class != RENDER_CLASS
)
3466 /* perf.exclusive_stream serialised by lrc_configure_all_contexts() */
3467 stream
= READ_ONCE(engine
->oa_group
->exclusive_stream
);
3468 if (stream
&& GRAPHICS_VER(stream
->perf
->i915
) < 12)
3469 gen8_update_reg_state_unlocked(ce
, stream
);
3473 * i915_perf_read - handles read() FOP for i915 perf stream FDs
3474 * @file: An i915 perf stream file
3475 * @buf: destination buffer given by userspace
3476 * @count: the number of bytes userspace wants to read
3477 * @ppos: (inout) file seek position (unused)
3479 * The entry point for handling a read() on a stream file descriptor from
3480 * userspace. Most of the work is left to the i915_perf_read_locked() and
3481 * &i915_perf_stream_ops->read but to save having stream implementations (of
3482 * which we might have multiple later) we handle blocking read here.
3484 * We can also consistently treat trying to read from a disabled stream
3485 * as an IO error so implementations can assume the stream is enabled
3488 * Returns: The number of bytes copied or a negative error code on failure.
3490 static ssize_t
i915_perf_read(struct file
*file
,
3495 struct i915_perf_stream
*stream
= file
->private_data
;
3499 /* To ensure it's handled consistently we simply treat all reads of a
3500 * disabled stream as an error. In particular it might otherwise lead
3501 * to a deadlock for blocking file descriptors...
3503 if (!stream
->enabled
|| !(stream
->sample_flags
& SAMPLE_OA_REPORT
))
3506 if (!(file
->f_flags
& O_NONBLOCK
)) {
3507 /* There's the small chance of false positives from
3508 * stream->ops->wait_unlocked.
3510 * E.g. with single context filtering since we only wait until
3511 * oabuffer has >= 1 report we don't immediately know whether
3512 * any reports really belong to the current context
3515 ret
= stream
->ops
->wait_unlocked(stream
);
3519 mutex_lock(&stream
->lock
);
3520 ret
= stream
->ops
->read(stream
, buf
, count
, &offset
);
3521 mutex_unlock(&stream
->lock
);
3522 } while (!offset
&& !ret
);
3524 mutex_lock(&stream
->lock
);
3525 ret
= stream
->ops
->read(stream
, buf
, count
, &offset
);
3526 mutex_unlock(&stream
->lock
);
3529 /* We allow the poll checking to sometimes report false positive EPOLLIN
3530 * events where we might actually report EAGAIN on read() if there's
3531 * not really any data available. In this situation though we don't
3532 * want to enter a busy loop between poll() reporting a EPOLLIN event
3533 * and read() returning -EAGAIN. Clearing the oa.pollin state here
3534 * effectively ensures we back off until the next hrtimer callback
3535 * before reporting another EPOLLIN event.
3536 * The exception to this is if ops->read() returned -ENOSPC which means
3537 * that more OA data is available than could fit in the user provided
3538 * buffer. In this case we want the next poll() call to not block.
3541 stream
->pollin
= false;
3543 /* Possible values for ret are 0, -EFAULT, -ENOSPC, -EIO, ... */
3544 return offset
?: (ret
?: -EAGAIN
);
3547 static enum hrtimer_restart
oa_poll_check_timer_cb(struct hrtimer
*hrtimer
)
3549 struct i915_perf_stream
*stream
=
3550 container_of(hrtimer
, typeof(*stream
), poll_check_timer
);
3552 if (oa_buffer_check_unlocked(stream
)) {
3553 stream
->pollin
= true;
3554 wake_up(&stream
->poll_wq
);
3557 hrtimer_forward_now(hrtimer
,
3558 ns_to_ktime(stream
->poll_oa_period
));
3560 return HRTIMER_RESTART
;
3564 * i915_perf_poll_locked - poll_wait() with a suitable wait queue for stream
3565 * @stream: An i915 perf stream
3566 * @file: An i915 perf stream file
3567 * @wait: poll() state table
3569 * For handling userspace polling on an i915 perf stream, this calls through to
3570 * &i915_perf_stream_ops->poll_wait to call poll_wait() with a wait queue that
3571 * will be woken for new stream data.
3573 * Returns: any poll events that are ready without sleeping
3575 static __poll_t
i915_perf_poll_locked(struct i915_perf_stream
*stream
,
3579 __poll_t events
= 0;
3581 stream
->ops
->poll_wait(stream
, file
, wait
);
3583 /* Note: we don't explicitly check whether there's something to read
3584 * here since this path may be very hot depending on what else
3585 * userspace is polling, or on the timeout in use. We rely solely on
3586 * the hrtimer/oa_poll_check_timer_cb to notify us when there are
3596 * i915_perf_poll - call poll_wait() with a suitable wait queue for stream
3597 * @file: An i915 perf stream file
3598 * @wait: poll() state table
3600 * For handling userspace polling on an i915 perf stream, this ensures
3601 * poll_wait() gets called with a wait queue that will be woken for new stream
3604 * Note: Implementation deferred to i915_perf_poll_locked()
3606 * Returns: any poll events that are ready without sleeping
3608 static __poll_t
i915_perf_poll(struct file
*file
, poll_table
*wait
)
3610 struct i915_perf_stream
*stream
= file
->private_data
;
3613 mutex_lock(&stream
->lock
);
3614 ret
= i915_perf_poll_locked(stream
, file
, wait
);
3615 mutex_unlock(&stream
->lock
);
3621 * i915_perf_enable_locked - handle `I915_PERF_IOCTL_ENABLE` ioctl
3622 * @stream: A disabled i915 perf stream
3624 * [Re]enables the associated capture of data for this stream.
3626 * If a stream was previously enabled then there's currently no intention
3627 * to provide userspace any guarantee about the preservation of previously
3630 static void i915_perf_enable_locked(struct i915_perf_stream
*stream
)
3632 if (stream
->enabled
)
3635 /* Allow stream->ops->enable() to refer to this */
3636 stream
->enabled
= true;
3638 if (stream
->ops
->enable
)
3639 stream
->ops
->enable(stream
);
3641 if (stream
->hold_preemption
)
3642 intel_context_set_nopreempt(stream
->pinned_ctx
);
3646 * i915_perf_disable_locked - handle `I915_PERF_IOCTL_DISABLE` ioctl
3647 * @stream: An enabled i915 perf stream
3649 * Disables the associated capture of data for this stream.
3651 * The intention is that disabling an re-enabling a stream will ideally be
3652 * cheaper than destroying and re-opening a stream with the same configuration,
3653 * though there are no formal guarantees about what state or buffered data
3654 * must be retained between disabling and re-enabling a stream.
3656 * Note: while a stream is disabled it's considered an error for userspace
3657 * to attempt to read from the stream (-EIO).
3659 static void i915_perf_disable_locked(struct i915_perf_stream
*stream
)
3661 if (!stream
->enabled
)
3664 /* Allow stream->ops->disable() to refer to this */
3665 stream
->enabled
= false;
3667 if (stream
->hold_preemption
)
3668 intel_context_clear_nopreempt(stream
->pinned_ctx
);
3670 if (stream
->ops
->disable
)
3671 stream
->ops
->disable(stream
);
3674 static long i915_perf_config_locked(struct i915_perf_stream
*stream
,
3675 unsigned long metrics_set
)
3677 struct i915_oa_config
*config
;
3678 long ret
= stream
->oa_config
->id
;
3680 config
= i915_perf_get_oa_config(stream
->perf
, metrics_set
);
3684 if (config
!= stream
->oa_config
) {
3688 * If OA is bound to a specific context, emit the
3689 * reconfiguration inline from that context. The update
3690 * will then be ordered with respect to submission on that
3693 * When set globally, we use a low priority kernel context,
3694 * so it will effectively take effect when idle.
3696 err
= emit_oa_config(stream
, config
, oa_context(stream
), NULL
);
3698 config
= xchg(&stream
->oa_config
, config
);
3703 i915_oa_config_put(config
);
3709 * i915_perf_ioctl_locked - support ioctl() usage with i915 perf stream FDs
3710 * @stream: An i915 perf stream
3711 * @cmd: the ioctl request
3712 * @arg: the ioctl data
3714 * Returns: zero on success or a negative error code. Returns -EINVAL for
3715 * an unknown ioctl request.
3717 static long i915_perf_ioctl_locked(struct i915_perf_stream
*stream
,
3722 case I915_PERF_IOCTL_ENABLE
:
3723 i915_perf_enable_locked(stream
);
3725 case I915_PERF_IOCTL_DISABLE
:
3726 i915_perf_disable_locked(stream
);
3728 case I915_PERF_IOCTL_CONFIG
:
3729 return i915_perf_config_locked(stream
, arg
);
3736 * i915_perf_ioctl - support ioctl() usage with i915 perf stream FDs
3737 * @file: An i915 perf stream file
3738 * @cmd: the ioctl request
3739 * @arg: the ioctl data
3741 * Implementation deferred to i915_perf_ioctl_locked().
3743 * Returns: zero on success or a negative error code. Returns -EINVAL for
3744 * an unknown ioctl request.
3746 static long i915_perf_ioctl(struct file
*file
,
3750 struct i915_perf_stream
*stream
= file
->private_data
;
3753 mutex_lock(&stream
->lock
);
3754 ret
= i915_perf_ioctl_locked(stream
, cmd
, arg
);
3755 mutex_unlock(&stream
->lock
);
3761 * i915_perf_destroy_locked - destroy an i915 perf stream
3762 * @stream: An i915 perf stream
3764 * Frees all resources associated with the given i915 perf @stream, disabling
3765 * any associated data capture in the process.
3767 * Note: The >->perf.lock mutex has been taken to serialize
3768 * with any non-file-operation driver hooks.
3770 static void i915_perf_destroy_locked(struct i915_perf_stream
*stream
)
3772 if (stream
->enabled
)
3773 i915_perf_disable_locked(stream
);
3775 if (stream
->ops
->destroy
)
3776 stream
->ops
->destroy(stream
);
3779 i915_gem_context_put(stream
->ctx
);
3785 * i915_perf_release - handles userspace close() of a stream file
3786 * @inode: anonymous inode associated with file
3787 * @file: An i915 perf stream file
3789 * Cleans up any resources associated with an open i915 perf stream file.
3791 * NB: close() can't really fail from the userspace point of view.
3793 * Returns: zero on success or a negative error code.
3795 static int i915_perf_release(struct inode
*inode
, struct file
*file
)
3797 struct i915_perf_stream
*stream
= file
->private_data
;
3798 struct i915_perf
*perf
= stream
->perf
;
3799 struct intel_gt
*gt
= stream
->engine
->gt
;
3802 * Within this call, we know that the fd is being closed and we have no
3803 * other user of stream->lock. Use the perf lock to destroy the stream
3806 mutex_lock(>
->perf
.lock
);
3807 i915_perf_destroy_locked(stream
);
3808 mutex_unlock(>
->perf
.lock
);
3810 /* Release the reference the perf stream kept on the driver. */
3811 drm_dev_put(&perf
->i915
->drm
);
3817 static const struct file_operations fops
= {
3818 .owner
= THIS_MODULE
,
3819 .llseek
= no_llseek
,
3820 .release
= i915_perf_release
,
3821 .poll
= i915_perf_poll
,
3822 .read
= i915_perf_read
,
3823 .unlocked_ioctl
= i915_perf_ioctl
,
3824 /* Our ioctl have no arguments, so it's safe to use the same function
3825 * to handle 32bits compatibility.
3827 .compat_ioctl
= i915_perf_ioctl
,
3832 * i915_perf_open_ioctl_locked - DRM ioctl() for userspace to open a stream FD
3833 * @perf: i915 perf instance
3834 * @param: The open parameters passed to 'DRM_I915_PERF_OPEN`
3835 * @props: individually validated u64 property value pairs
3838 * See i915_perf_ioctl_open() for interface details.
3840 * Implements further stream config validation and stream initialization on
3841 * behalf of i915_perf_open_ioctl() with the >->perf.lock mutex
3842 * taken to serialize with any non-file-operation driver hooks.
3844 * Note: at this point the @props have only been validated in isolation and
3845 * it's still necessary to validate that the combination of properties makes
3848 * In the case where userspace is interested in OA unit metrics then further
3849 * config validation and stream initialization details will be handled by
3850 * i915_oa_stream_init(). The code here should only validate config state that
3851 * will be relevant to all stream types / backends.
3853 * Returns: zero on success or a negative error code.
3856 i915_perf_open_ioctl_locked(struct i915_perf
*perf
,
3857 struct drm_i915_perf_open_param
*param
,
3858 struct perf_open_properties
*props
,
3859 struct drm_file
*file
)
3861 struct i915_gem_context
*specific_ctx
= NULL
;
3862 struct i915_perf_stream
*stream
= NULL
;
3863 unsigned long f_flags
= 0;
3864 bool privileged_op
= true;
3868 if (props
->single_context
) {
3869 u32 ctx_handle
= props
->ctx_handle
;
3870 struct drm_i915_file_private
*file_priv
= file
->driver_priv
;
3872 specific_ctx
= i915_gem_context_lookup(file_priv
, ctx_handle
);
3873 if (IS_ERR(specific_ctx
)) {
3874 drm_dbg(&perf
->i915
->drm
,
3875 "Failed to look up context with ID %u for opening perf stream\n",
3877 ret
= PTR_ERR(specific_ctx
);
3883 * On Haswell the OA unit supports clock gating off for a specific
3884 * context and in this mode there's no visibility of metrics for the
3885 * rest of the system, which we consider acceptable for a
3886 * non-privileged client.
3888 * For Gen8->11 the OA unit no longer supports clock gating off for a
3889 * specific context and the kernel can't securely stop the counters
3890 * from updating as system-wide / global values. Even though we can
3891 * filter reports based on the included context ID we can't block
3892 * clients from seeing the raw / global counter values via
3893 * MI_REPORT_PERF_COUNT commands and so consider it a privileged op to
3894 * enable the OA unit by default.
3896 * For Gen12+ we gain a new OAR unit that only monitors the RCS on a
3897 * per context basis. So we can relax requirements there if the user
3898 * doesn't request global stream access (i.e. query based sampling
3899 * using MI_RECORD_PERF_COUNT.
3901 if (IS_HASWELL(perf
->i915
) && specific_ctx
)
3902 privileged_op
= false;
3903 else if (GRAPHICS_VER(perf
->i915
) == 12 && specific_ctx
&&
3904 (props
->sample_flags
& SAMPLE_OA_REPORT
) == 0)
3905 privileged_op
= false;
3907 if (props
->hold_preemption
) {
3908 if (!props
->single_context
) {
3909 drm_dbg(&perf
->i915
->drm
,
3910 "preemption disable with no context\n");
3914 privileged_op
= true;
3918 * Asking for SSEU configuration is a priviliged operation.
3920 if (props
->has_sseu
)
3921 privileged_op
= true;
3923 get_default_sseu_config(&props
->sseu
, props
->engine
);
3925 /* Similar to perf's kernel.perf_paranoid_cpu sysctl option
3926 * we check a dev.i915.perf_stream_paranoid sysctl option
3927 * to determine if it's ok to access system wide OA counters
3928 * without CAP_PERFMON or CAP_SYS_ADMIN privileges.
3930 if (privileged_op
&&
3931 i915_perf_stream_paranoid
&& !perfmon_capable()) {
3932 drm_dbg(&perf
->i915
->drm
,
3933 "Insufficient privileges to open i915 perf stream\n");
3938 stream
= kzalloc(sizeof(*stream
), GFP_KERNEL
);
3944 stream
->perf
= perf
;
3945 stream
->ctx
= specific_ctx
;
3946 stream
->poll_oa_period
= props
->poll_oa_period
;
3948 ret
= i915_oa_stream_init(stream
, param
, props
);
3952 /* we avoid simply assigning stream->sample_flags = props->sample_flags
3953 * to have _stream_init check the combination of sample flags more
3954 * thoroughly, but still this is the expected result at this point.
3956 if (WARN_ON(stream
->sample_flags
!= props
->sample_flags
)) {
3961 if (param
->flags
& I915_PERF_FLAG_FD_CLOEXEC
)
3962 f_flags
|= O_CLOEXEC
;
3963 if (param
->flags
& I915_PERF_FLAG_FD_NONBLOCK
)
3964 f_flags
|= O_NONBLOCK
;
3966 stream_fd
= anon_inode_getfd("[i915_perf]", &fops
, stream
, f_flags
);
3967 if (stream_fd
< 0) {
3972 if (!(param
->flags
& I915_PERF_FLAG_DISABLED
))
3973 i915_perf_enable_locked(stream
);
3975 /* Take a reference on the driver that will be kept with stream_fd
3976 * until its release.
3978 drm_dev_get(&perf
->i915
->drm
);
3983 if (stream
->ops
->destroy
)
3984 stream
->ops
->destroy(stream
);
3989 i915_gem_context_put(specific_ctx
);
3994 static u64
oa_exponent_to_ns(struct i915_perf
*perf
, int exponent
)
3996 u64 nom
= (2ULL << exponent
) * NSEC_PER_SEC
;
3997 u32 den
= i915_perf_oa_timestamp_frequency(perf
->i915
);
3999 return div_u64(nom
+ den
- 1, den
);
4002 static __always_inline
bool
4003 oa_format_valid(struct i915_perf
*perf
, enum drm_i915_oa_format format
)
4005 return test_bit(format
, perf
->format_mask
);
4008 static __always_inline
void
4009 oa_format_add(struct i915_perf
*perf
, enum drm_i915_oa_format format
)
4011 __set_bit(format
, perf
->format_mask
);
4015 * read_properties_unlocked - validate + copy userspace stream open properties
4016 * @perf: i915 perf instance
4017 * @uprops: The array of u64 key value pairs given by userspace
4018 * @n_props: The number of key value pairs expected in @uprops
4019 * @props: The stream configuration built up while validating properties
4021 * Note this function only validates properties in isolation it doesn't
4022 * validate that the combination of properties makes sense or that all
4023 * properties necessary for a particular kind of stream have been set.
4025 * Note that there currently aren't any ordering requirements for properties so
4026 * we shouldn't validate or assume anything about ordering here. This doesn't
4027 * rule out defining new properties with ordering requirements in the future.
4029 static int read_properties_unlocked(struct i915_perf
*perf
,
4032 struct perf_open_properties
*props
)
4034 struct drm_i915_gem_context_param_sseu user_sseu
;
4035 const struct i915_oa_format
*f
;
4036 u64 __user
*uprop
= uprops
;
4037 bool config_instance
= false;
4038 bool config_class
= false;
4039 bool config_sseu
= false;
4044 memset(props
, 0, sizeof(struct perf_open_properties
));
4045 props
->poll_oa_period
= DEFAULT_POLL_PERIOD_NS
;
4047 /* Considering that ID = 0 is reserved and assuming that we don't
4048 * (currently) expect any configurations to ever specify duplicate
4049 * values for a particular property ID then the last _PROP_MAX value is
4050 * one greater than the maximum number of properties we expect to get
4053 if (!n_props
|| n_props
>= DRM_I915_PERF_PROP_MAX
) {
4054 drm_dbg(&perf
->i915
->drm
,
4055 "Invalid number of i915 perf properties given\n");
4059 /* Defaults when class:instance is not passed */
4060 class = I915_ENGINE_CLASS_RENDER
;
4063 for (i
= 0; i
< n_props
; i
++) {
4064 u64 oa_period
, oa_freq_hz
;
4067 ret
= get_user(id
, uprop
);
4071 ret
= get_user(value
, uprop
+ 1);
4075 if (id
== 0 || id
>= DRM_I915_PERF_PROP_MAX
) {
4076 drm_dbg(&perf
->i915
->drm
,
4077 "Unknown i915 perf property ID\n");
4081 switch ((enum drm_i915_perf_property_id
)id
) {
4082 case DRM_I915_PERF_PROP_CTX_HANDLE
:
4083 props
->single_context
= 1;
4084 props
->ctx_handle
= value
;
4086 case DRM_I915_PERF_PROP_SAMPLE_OA
:
4088 props
->sample_flags
|= SAMPLE_OA_REPORT
;
4090 case DRM_I915_PERF_PROP_OA_METRICS_SET
:
4092 drm_dbg(&perf
->i915
->drm
,
4093 "Unknown OA metric set ID\n");
4096 props
->metrics_set
= value
;
4098 case DRM_I915_PERF_PROP_OA_FORMAT
:
4099 if (value
== 0 || value
>= I915_OA_FORMAT_MAX
) {
4100 drm_dbg(&perf
->i915
->drm
,
4101 "Out-of-range OA report format %llu\n",
4105 if (!oa_format_valid(perf
, value
)) {
4106 drm_dbg(&perf
->i915
->drm
,
4107 "Unsupported OA report format %llu\n",
4111 props
->oa_format
= value
;
4113 case DRM_I915_PERF_PROP_OA_EXPONENT
:
4114 if (value
> OA_EXPONENT_MAX
) {
4115 drm_dbg(&perf
->i915
->drm
,
4116 "OA timer exponent too high (> %u)\n",
4121 /* Theoretically we can program the OA unit to sample
4122 * e.g. every 160ns for HSW, 167ns for BDW/SKL or 104ns
4123 * for BXT. We don't allow such high sampling
4124 * frequencies by default unless root.
4127 BUILD_BUG_ON(sizeof(oa_period
) != 8);
4128 oa_period
= oa_exponent_to_ns(perf
, value
);
4130 /* This check is primarily to ensure that oa_period <=
4131 * UINT32_MAX (before passing to do_div which only
4132 * accepts a u32 denominator), but we can also skip
4133 * checking anything < 1Hz which implicitly can't be
4134 * limited via an integer oa_max_sample_rate.
4136 if (oa_period
<= NSEC_PER_SEC
) {
4137 u64 tmp
= NSEC_PER_SEC
;
4138 do_div(tmp
, oa_period
);
4143 if (oa_freq_hz
> i915_oa_max_sample_rate
&& !perfmon_capable()) {
4144 drm_dbg(&perf
->i915
->drm
,
4145 "OA exponent would exceed the max sampling frequency (sysctl dev.i915.oa_max_sample_rate) %uHz without CAP_PERFMON or CAP_SYS_ADMIN privileges\n",
4146 i915_oa_max_sample_rate
);
4150 props
->oa_periodic
= true;
4151 props
->oa_period_exponent
= value
;
4153 case DRM_I915_PERF_PROP_HOLD_PREEMPTION
:
4154 props
->hold_preemption
= !!value
;
4156 case DRM_I915_PERF_PROP_GLOBAL_SSEU
: {
4157 if (GRAPHICS_VER_FULL(perf
->i915
) >= IP_VER(12, 50)) {
4158 drm_dbg(&perf
->i915
->drm
,
4159 "SSEU config not supported on gfx %x\n",
4160 GRAPHICS_VER_FULL(perf
->i915
));
4164 if (copy_from_user(&user_sseu
,
4165 u64_to_user_ptr(value
),
4166 sizeof(user_sseu
))) {
4167 drm_dbg(&perf
->i915
->drm
,
4168 "Unable to copy global sseu parameter\n");
4174 case DRM_I915_PERF_PROP_POLL_OA_PERIOD
:
4175 if (value
< 100000 /* 100us */) {
4176 drm_dbg(&perf
->i915
->drm
,
4177 "OA availability timer too small (%lluns < 100us)\n",
4181 props
->poll_oa_period
= value
;
4183 case DRM_I915_PERF_PROP_OA_ENGINE_CLASS
:
4185 config_class
= true;
4187 case DRM_I915_PERF_PROP_OA_ENGINE_INSTANCE
:
4188 instance
= (u8
)value
;
4189 config_instance
= true;
4199 if ((config_class
&& !config_instance
) ||
4200 (config_instance
&& !config_class
)) {
4201 drm_dbg(&perf
->i915
->drm
,
4202 "OA engine-class and engine-instance parameters must be passed together\n");
4206 props
->engine
= intel_engine_lookup_user(perf
->i915
, class, instance
);
4207 if (!props
->engine
) {
4208 drm_dbg(&perf
->i915
->drm
,
4209 "OA engine class and instance invalid %d:%d\n",
4214 if (!engine_supports_oa(props
->engine
)) {
4215 drm_dbg(&perf
->i915
->drm
,
4216 "Engine not supported by OA %d:%d\n",
4222 * Wa_14017512683: mtl[a0..c0): Use of OAM must be preceded with Media
4223 * C6 disable in BIOS. Fail if Media C6 is enabled on steppings where OAM
4224 * does not work as expected.
4226 if (IS_MTL_MEDIA_STEP(props
->engine
->i915
, STEP_A0
, STEP_C0
) &&
4227 props
->engine
->oa_group
->type
== TYPE_OAM
&&
4228 intel_check_bios_c6_setup(&props
->engine
->gt
->rc6
)) {
4229 drm_dbg(&perf
->i915
->drm
,
4230 "OAM requires media C6 to be disabled in BIOS\n");
4234 i
= array_index_nospec(props
->oa_format
, I915_OA_FORMAT_MAX
);
4235 f
= &perf
->oa_formats
[i
];
4236 if (!engine_supports_oa_format(props
->engine
, f
->type
)) {
4237 drm_dbg(&perf
->i915
->drm
,
4238 "Invalid OA format %d for class %d\n",
4239 f
->type
, props
->engine
->class);
4244 ret
= get_sseu_config(&props
->sseu
, props
->engine
, &user_sseu
);
4246 drm_dbg(&perf
->i915
->drm
,
4247 "Invalid SSEU configuration\n");
4250 props
->has_sseu
= true;
4257 * i915_perf_open_ioctl - DRM ioctl() for userspace to open a stream FD
4259 * @data: ioctl data copied from userspace (unvalidated)
4262 * Validates the stream open parameters given by userspace including flags
4263 * and an array of u64 key, value pair properties.
4265 * Very little is assumed up front about the nature of the stream being
4266 * opened (for instance we don't assume it's for periodic OA unit metrics). An
4267 * i915-perf stream is expected to be a suitable interface for other forms of
4268 * buffered data written by the GPU besides periodic OA metrics.
4270 * Note we copy the properties from userspace outside of the i915 perf
4271 * mutex to avoid an awkward lockdep with mmap_lock.
4273 * Most of the implementation details are handled by
4274 * i915_perf_open_ioctl_locked() after taking the >->perf.lock
4275 * mutex for serializing with any non-file-operation driver hooks.
4277 * Return: A newly opened i915 Perf stream file descriptor or negative
4278 * error code on failure.
4280 int i915_perf_open_ioctl(struct drm_device
*dev
, void *data
,
4281 struct drm_file
*file
)
4283 struct i915_perf
*perf
= &to_i915(dev
)->perf
;
4284 struct drm_i915_perf_open_param
*param
= data
;
4285 struct intel_gt
*gt
;
4286 struct perf_open_properties props
;
4287 u32 known_open_flags
;
4291 drm_dbg(&perf
->i915
->drm
,
4292 "i915 perf interface not available for this system\n");
4296 known_open_flags
= I915_PERF_FLAG_FD_CLOEXEC
|
4297 I915_PERF_FLAG_FD_NONBLOCK
|
4298 I915_PERF_FLAG_DISABLED
;
4299 if (param
->flags
& ~known_open_flags
) {
4300 drm_dbg(&perf
->i915
->drm
,
4301 "Unknown drm_i915_perf_open_param flag\n");
4305 ret
= read_properties_unlocked(perf
,
4306 u64_to_user_ptr(param
->properties_ptr
),
4307 param
->num_properties
,
4312 gt
= props
.engine
->gt
;
4314 mutex_lock(>
->perf
.lock
);
4315 ret
= i915_perf_open_ioctl_locked(perf
, param
, &props
, file
);
4316 mutex_unlock(>
->perf
.lock
);
4322 * i915_perf_register - exposes i915-perf to userspace
4323 * @i915: i915 device instance
4325 * In particular OA metric sets are advertised under a sysfs metrics/
4326 * directory allowing userspace to enumerate valid IDs that can be
4327 * used to open an i915-perf stream.
4329 void i915_perf_register(struct drm_i915_private
*i915
)
4331 struct i915_perf
*perf
= &i915
->perf
;
4332 struct intel_gt
*gt
= to_gt(i915
);
4337 /* To be sure we're synchronized with an attempted
4338 * i915_perf_open_ioctl(); considering that we register after
4339 * being exposed to userspace.
4341 mutex_lock(>
->perf
.lock
);
4343 perf
->metrics_kobj
=
4344 kobject_create_and_add("metrics",
4345 &i915
->drm
.primary
->kdev
->kobj
);
4347 mutex_unlock(>
->perf
.lock
);
4351 * i915_perf_unregister - hide i915-perf from userspace
4352 * @i915: i915 device instance
4354 * i915-perf state cleanup is split up into an 'unregister' and
4355 * 'deinit' phase where the interface is first hidden from
4356 * userspace by i915_perf_unregister() before cleaning up
4357 * remaining state in i915_perf_fini().
4359 void i915_perf_unregister(struct drm_i915_private
*i915
)
4361 struct i915_perf
*perf
= &i915
->perf
;
4363 if (!perf
->metrics_kobj
)
4366 kobject_put(perf
->metrics_kobj
);
4367 perf
->metrics_kobj
= NULL
;
4370 static bool gen8_is_valid_flex_addr(struct i915_perf
*perf
, u32 addr
)
4372 static const i915_reg_t flex_eu_regs
[] = {
4383 for (i
= 0; i
< ARRAY_SIZE(flex_eu_regs
); i
++) {
4384 if (i915_mmio_reg_offset(flex_eu_regs
[i
]) == addr
)
4390 static bool reg_in_range_table(u32 addr
, const struct i915_range
*table
)
4392 while (table
->start
|| table
->end
) {
4393 if (addr
>= table
->start
&& addr
<= table
->end
)
4402 #define REG_EQUAL(addr, mmio) \
4403 ((addr) == i915_mmio_reg_offset(mmio))
4405 static const struct i915_range gen7_oa_b_counters
[] = {
4406 { .start
= 0x2710, .end
= 0x272c }, /* OASTARTTRIG[1-8] */
4407 { .start
= 0x2740, .end
= 0x275c }, /* OAREPORTTRIG[1-8] */
4408 { .start
= 0x2770, .end
= 0x27ac }, /* OACEC[0-7][0-1] */
4412 static const struct i915_range gen12_oa_b_counters
[] = {
4413 { .start
= 0x2b2c, .end
= 0x2b2c }, /* GEN12_OAG_OA_PESS */
4414 { .start
= 0xd900, .end
= 0xd91c }, /* GEN12_OAG_OASTARTTRIG[1-8] */
4415 { .start
= 0xd920, .end
= 0xd93c }, /* GEN12_OAG_OAREPORTTRIG1[1-8] */
4416 { .start
= 0xd940, .end
= 0xd97c }, /* GEN12_OAG_CEC[0-7][0-1] */
4417 { .start
= 0xdc00, .end
= 0xdc3c }, /* GEN12_OAG_SCEC[0-7][0-1] */
4418 { .start
= 0xdc40, .end
= 0xdc40 }, /* GEN12_OAG_SPCTR_CNF */
4419 { .start
= 0xdc44, .end
= 0xdc44 }, /* GEN12_OAA_DBG_REG */
4423 static const struct i915_range mtl_oam_b_counters
[] = {
4424 { .start
= 0x393000, .end
= 0x39301c }, /* GEN12_OAM_STARTTRIG1[1-8] */
4425 { .start
= 0x393020, .end
= 0x39303c }, /* GEN12_OAM_REPORTTRIG1[1-8] */
4426 { .start
= 0x393040, .end
= 0x39307c }, /* GEN12_OAM_CEC[0-7][0-1] */
4427 { .start
= 0x393200, .end
= 0x39323C }, /* MPES[0-7] */
4431 static const struct i915_range xehp_oa_b_counters
[] = {
4432 { .start
= 0xdc48, .end
= 0xdc48 }, /* OAA_ENABLE_REG */
4433 { .start
= 0xdd00, .end
= 0xdd48 }, /* OAG_LCE0_0 - OAA_LENABLE_REG */
4437 static const struct i915_range gen7_oa_mux_regs
[] = {
4438 { .start
= 0x91b8, .end
= 0x91cc }, /* OA_PERFCNT[1-2], OA_PERFMATRIX */
4439 { .start
= 0x9800, .end
= 0x9888 }, /* MICRO_BP0_0 - NOA_WRITE */
4440 { .start
= 0xe180, .end
= 0xe180 }, /* HALF_SLICE_CHICKEN2 */
4444 static const struct i915_range hsw_oa_mux_regs
[] = {
4445 { .start
= 0x09e80, .end
= 0x09ea4 }, /* HSW_MBVID2_NOA[0-9] */
4446 { .start
= 0x09ec0, .end
= 0x09ec0 }, /* HSW_MBVID2_MISR0 */
4447 { .start
= 0x25100, .end
= 0x2ff90 },
4451 static const struct i915_range chv_oa_mux_regs
[] = {
4452 { .start
= 0x182300, .end
= 0x1823a4 },
4456 static const struct i915_range gen8_oa_mux_regs
[] = {
4457 { .start
= 0x0d00, .end
= 0x0d2c }, /* RPM_CONFIG[0-1], NOA_CONFIG[0-8] */
4458 { .start
= 0x20cc, .end
= 0x20cc }, /* WAIT_FOR_RC6_EXIT */
4462 static const struct i915_range gen11_oa_mux_regs
[] = {
4463 { .start
= 0x91c8, .end
= 0x91dc }, /* OA_PERFCNT[3-4] */
4467 static const struct i915_range gen12_oa_mux_regs
[] = {
4468 { .start
= 0x0d00, .end
= 0x0d04 }, /* RPM_CONFIG[0-1] */
4469 { .start
= 0x0d0c, .end
= 0x0d2c }, /* NOA_CONFIG[0-8] */
4470 { .start
= 0x9840, .end
= 0x9840 }, /* GDT_CHICKEN_BITS */
4471 { .start
= 0x9884, .end
= 0x9888 }, /* NOA_WRITE */
4472 { .start
= 0x20cc, .end
= 0x20cc }, /* WAIT_FOR_RC6_EXIT */
4478 * 0x20cc is repurposed on MTL, so use a separate array for MTL.
4480 static const struct i915_range mtl_oa_mux_regs
[] = {
4481 { .start
= 0x0d00, .end
= 0x0d04 }, /* RPM_CONFIG[0-1] */
4482 { .start
= 0x0d0c, .end
= 0x0d2c }, /* NOA_CONFIG[0-8] */
4483 { .start
= 0x9840, .end
= 0x9840 }, /* GDT_CHICKEN_BITS */
4484 { .start
= 0x9884, .end
= 0x9888 }, /* NOA_WRITE */
4485 { .start
= 0x38d100, .end
= 0x38d114}, /* VISACTL */
4489 static bool gen7_is_valid_b_counter_addr(struct i915_perf
*perf
, u32 addr
)
4491 return reg_in_range_table(addr
, gen7_oa_b_counters
);
4494 static bool gen8_is_valid_mux_addr(struct i915_perf
*perf
, u32 addr
)
4496 return reg_in_range_table(addr
, gen7_oa_mux_regs
) ||
4497 reg_in_range_table(addr
, gen8_oa_mux_regs
);
4500 static bool gen11_is_valid_mux_addr(struct i915_perf
*perf
, u32 addr
)
4502 return reg_in_range_table(addr
, gen7_oa_mux_regs
) ||
4503 reg_in_range_table(addr
, gen8_oa_mux_regs
) ||
4504 reg_in_range_table(addr
, gen11_oa_mux_regs
);
4507 static bool hsw_is_valid_mux_addr(struct i915_perf
*perf
, u32 addr
)
4509 return reg_in_range_table(addr
, gen7_oa_mux_regs
) ||
4510 reg_in_range_table(addr
, hsw_oa_mux_regs
);
4513 static bool chv_is_valid_mux_addr(struct i915_perf
*perf
, u32 addr
)
4515 return reg_in_range_table(addr
, gen7_oa_mux_regs
) ||
4516 reg_in_range_table(addr
, chv_oa_mux_regs
);
4519 static bool gen12_is_valid_b_counter_addr(struct i915_perf
*perf
, u32 addr
)
4521 return reg_in_range_table(addr
, gen12_oa_b_counters
);
4524 static bool mtl_is_valid_oam_b_counter_addr(struct i915_perf
*perf
, u32 addr
)
4526 if (HAS_OAM(perf
->i915
) &&
4527 GRAPHICS_VER_FULL(perf
->i915
) >= IP_VER(12, 70))
4528 return reg_in_range_table(addr
, mtl_oam_b_counters
);
4533 static bool xehp_is_valid_b_counter_addr(struct i915_perf
*perf
, u32 addr
)
4535 return reg_in_range_table(addr
, xehp_oa_b_counters
) ||
4536 reg_in_range_table(addr
, gen12_oa_b_counters
) ||
4537 mtl_is_valid_oam_b_counter_addr(perf
, addr
);
4540 static bool gen12_is_valid_mux_addr(struct i915_perf
*perf
, u32 addr
)
4542 if (IS_METEORLAKE(perf
->i915
))
4543 return reg_in_range_table(addr
, mtl_oa_mux_regs
);
4545 return reg_in_range_table(addr
, gen12_oa_mux_regs
);
4548 static u32
mask_reg_value(u32 reg
, u32 val
)
4550 /* HALF_SLICE_CHICKEN2 is programmed with a the
4551 * WaDisableSTUnitPowerOptimization workaround. Make sure the value
4552 * programmed by userspace doesn't change this.
4554 if (REG_EQUAL(reg
, HALF_SLICE_CHICKEN2
))
4555 val
= val
& ~_MASKED_BIT_ENABLE(GEN8_ST_PO_DISABLE
);
4557 /* WAIT_FOR_RC6_EXIT has only one bit fullfilling the function
4558 * indicated by its name and a bunch of selection fields used by OA
4561 if (REG_EQUAL(reg
, WAIT_FOR_RC6_EXIT
))
4562 val
= val
& ~_MASKED_BIT_ENABLE(HSW_WAIT_FOR_RC6_EXIT_ENABLE
);
4567 static struct i915_oa_reg
*alloc_oa_regs(struct i915_perf
*perf
,
4568 bool (*is_valid
)(struct i915_perf
*perf
, u32 addr
),
4572 struct i915_oa_reg
*oa_regs
;
4579 /* No is_valid function means we're not allowing any register to be programmed. */
4580 GEM_BUG_ON(!is_valid
);
4582 return ERR_PTR(-EINVAL
);
4584 oa_regs
= kmalloc_array(n_regs
, sizeof(*oa_regs
), GFP_KERNEL
);
4586 return ERR_PTR(-ENOMEM
);
4588 for (i
= 0; i
< n_regs
; i
++) {
4591 err
= get_user(addr
, regs
);
4595 if (!is_valid(perf
, addr
)) {
4596 drm_dbg(&perf
->i915
->drm
,
4597 "Invalid oa_reg address: %X\n", addr
);
4602 err
= get_user(value
, regs
+ 1);
4606 oa_regs
[i
].addr
= _MMIO(addr
);
4607 oa_regs
[i
].value
= mask_reg_value(addr
, value
);
4616 return ERR_PTR(err
);
4619 static ssize_t
show_dynamic_id(struct kobject
*kobj
,
4620 struct kobj_attribute
*attr
,
4623 struct i915_oa_config
*oa_config
=
4624 container_of(attr
, typeof(*oa_config
), sysfs_metric_id
);
4626 return sprintf(buf
, "%d\n", oa_config
->id
);
4629 static int create_dynamic_oa_sysfs_entry(struct i915_perf
*perf
,
4630 struct i915_oa_config
*oa_config
)
4632 sysfs_attr_init(&oa_config
->sysfs_metric_id
.attr
);
4633 oa_config
->sysfs_metric_id
.attr
.name
= "id";
4634 oa_config
->sysfs_metric_id
.attr
.mode
= S_IRUGO
;
4635 oa_config
->sysfs_metric_id
.show
= show_dynamic_id
;
4636 oa_config
->sysfs_metric_id
.store
= NULL
;
4638 oa_config
->attrs
[0] = &oa_config
->sysfs_metric_id
.attr
;
4639 oa_config
->attrs
[1] = NULL
;
4641 oa_config
->sysfs_metric
.name
= oa_config
->uuid
;
4642 oa_config
->sysfs_metric
.attrs
= oa_config
->attrs
;
4644 return sysfs_create_group(perf
->metrics_kobj
,
4645 &oa_config
->sysfs_metric
);
4649 * i915_perf_add_config_ioctl - DRM ioctl() for userspace to add a new OA config
4651 * @data: ioctl data (pointer to struct drm_i915_perf_oa_config) copied from
4652 * userspace (unvalidated)
4655 * Validates the submitted OA register to be saved into a new OA config that
4656 * can then be used for programming the OA unit and its NOA network.
4658 * Returns: A new allocated config number to be used with the perf open ioctl
4659 * or a negative error code on failure.
4661 int i915_perf_add_config_ioctl(struct drm_device
*dev
, void *data
,
4662 struct drm_file
*file
)
4664 struct i915_perf
*perf
= &to_i915(dev
)->perf
;
4665 struct drm_i915_perf_oa_config
*args
= data
;
4666 struct i915_oa_config
*oa_config
, *tmp
;
4667 struct i915_oa_reg
*regs
;
4671 drm_dbg(&perf
->i915
->drm
,
4672 "i915 perf interface not available for this system\n");
4676 if (!perf
->metrics_kobj
) {
4677 drm_dbg(&perf
->i915
->drm
,
4678 "OA metrics weren't advertised via sysfs\n");
4682 if (i915_perf_stream_paranoid
&& !perfmon_capable()) {
4683 drm_dbg(&perf
->i915
->drm
,
4684 "Insufficient privileges to add i915 OA config\n");
4688 if ((!args
->mux_regs_ptr
|| !args
->n_mux_regs
) &&
4689 (!args
->boolean_regs_ptr
|| !args
->n_boolean_regs
) &&
4690 (!args
->flex_regs_ptr
|| !args
->n_flex_regs
)) {
4691 drm_dbg(&perf
->i915
->drm
,
4692 "No OA registers given\n");
4696 oa_config
= kzalloc(sizeof(*oa_config
), GFP_KERNEL
);
4698 drm_dbg(&perf
->i915
->drm
,
4699 "Failed to allocate memory for the OA config\n");
4703 oa_config
->perf
= perf
;
4704 kref_init(&oa_config
->ref
);
4706 if (!uuid_is_valid(args
->uuid
)) {
4707 drm_dbg(&perf
->i915
->drm
,
4708 "Invalid uuid format for OA config\n");
4713 /* Last character in oa_config->uuid will be 0 because oa_config is
4716 memcpy(oa_config
->uuid
, args
->uuid
, sizeof(args
->uuid
));
4718 oa_config
->mux_regs_len
= args
->n_mux_regs
;
4719 regs
= alloc_oa_regs(perf
,
4720 perf
->ops
.is_valid_mux_reg
,
4721 u64_to_user_ptr(args
->mux_regs_ptr
),
4725 drm_dbg(&perf
->i915
->drm
,
4726 "Failed to create OA config for mux_regs\n");
4727 err
= PTR_ERR(regs
);
4730 oa_config
->mux_regs
= regs
;
4732 oa_config
->b_counter_regs_len
= args
->n_boolean_regs
;
4733 regs
= alloc_oa_regs(perf
,
4734 perf
->ops
.is_valid_b_counter_reg
,
4735 u64_to_user_ptr(args
->boolean_regs_ptr
),
4736 args
->n_boolean_regs
);
4739 drm_dbg(&perf
->i915
->drm
,
4740 "Failed to create OA config for b_counter_regs\n");
4741 err
= PTR_ERR(regs
);
4744 oa_config
->b_counter_regs
= regs
;
4746 if (GRAPHICS_VER(perf
->i915
) < 8) {
4747 if (args
->n_flex_regs
!= 0) {
4752 oa_config
->flex_regs_len
= args
->n_flex_regs
;
4753 regs
= alloc_oa_regs(perf
,
4754 perf
->ops
.is_valid_flex_reg
,
4755 u64_to_user_ptr(args
->flex_regs_ptr
),
4759 drm_dbg(&perf
->i915
->drm
,
4760 "Failed to create OA config for flex_regs\n");
4761 err
= PTR_ERR(regs
);
4764 oa_config
->flex_regs
= regs
;
4767 err
= mutex_lock_interruptible(&perf
->metrics_lock
);
4771 /* We shouldn't have too many configs, so this iteration shouldn't be
4774 idr_for_each_entry(&perf
->metrics_idr
, tmp
, id
) {
4775 if (!strcmp(tmp
->uuid
, oa_config
->uuid
)) {
4776 drm_dbg(&perf
->i915
->drm
,
4777 "OA config already exists with this uuid\n");
4783 err
= create_dynamic_oa_sysfs_entry(perf
, oa_config
);
4785 drm_dbg(&perf
->i915
->drm
,
4786 "Failed to create sysfs entry for OA config\n");
4790 /* Config id 0 is invalid, id 1 for kernel stored test config. */
4791 oa_config
->id
= idr_alloc(&perf
->metrics_idr
,
4794 if (oa_config
->id
< 0) {
4795 drm_dbg(&perf
->i915
->drm
,
4796 "Failed to create sysfs entry for OA config\n");
4797 err
= oa_config
->id
;
4802 drm_dbg(&perf
->i915
->drm
,
4803 "Added config %s id=%i\n", oa_config
->uuid
, oa_config
->id
);
4804 mutex_unlock(&perf
->metrics_lock
);
4809 mutex_unlock(&perf
->metrics_lock
);
4811 i915_oa_config_put(oa_config
);
4812 drm_dbg(&perf
->i915
->drm
,
4813 "Failed to add new OA config\n");
4818 * i915_perf_remove_config_ioctl - DRM ioctl() for userspace to remove an OA config
4820 * @data: ioctl data (pointer to u64 integer) copied from userspace
4823 * Configs can be removed while being used, the will stop appearing in sysfs
4824 * and their content will be freed when the stream using the config is closed.
4826 * Returns: 0 on success or a negative error code on failure.
4828 int i915_perf_remove_config_ioctl(struct drm_device
*dev
, void *data
,
4829 struct drm_file
*file
)
4831 struct i915_perf
*perf
= &to_i915(dev
)->perf
;
4833 struct i915_oa_config
*oa_config
;
4837 drm_dbg(&perf
->i915
->drm
,
4838 "i915 perf interface not available for this system\n");
4842 if (i915_perf_stream_paranoid
&& !perfmon_capable()) {
4843 drm_dbg(&perf
->i915
->drm
,
4844 "Insufficient privileges to remove i915 OA config\n");
4848 ret
= mutex_lock_interruptible(&perf
->metrics_lock
);
4852 oa_config
= idr_find(&perf
->metrics_idr
, *arg
);
4854 drm_dbg(&perf
->i915
->drm
,
4855 "Failed to remove unknown OA config\n");
4860 GEM_BUG_ON(*arg
!= oa_config
->id
);
4862 sysfs_remove_group(perf
->metrics_kobj
, &oa_config
->sysfs_metric
);
4864 idr_remove(&perf
->metrics_idr
, *arg
);
4866 mutex_unlock(&perf
->metrics_lock
);
4868 drm_dbg(&perf
->i915
->drm
,
4869 "Removed config %s id=%i\n", oa_config
->uuid
, oa_config
->id
);
4871 i915_oa_config_put(oa_config
);
4876 mutex_unlock(&perf
->metrics_lock
);
4880 static struct ctl_table oa_table
[] = {
4882 .procname
= "perf_stream_paranoid",
4883 .data
= &i915_perf_stream_paranoid
,
4884 .maxlen
= sizeof(i915_perf_stream_paranoid
),
4886 .proc_handler
= proc_dointvec_minmax
,
4887 .extra1
= SYSCTL_ZERO
,
4888 .extra2
= SYSCTL_ONE
,
4891 .procname
= "oa_max_sample_rate",
4892 .data
= &i915_oa_max_sample_rate
,
4893 .maxlen
= sizeof(i915_oa_max_sample_rate
),
4895 .proc_handler
= proc_dointvec_minmax
,
4896 .extra1
= SYSCTL_ZERO
,
4897 .extra2
= &oa_sample_rate_hard_limit
,
4902 static u32
num_perf_groups_per_gt(struct intel_gt
*gt
)
4907 static u32
__oam_engine_group(struct intel_engine_cs
*engine
)
4909 if (GRAPHICS_VER_FULL(engine
->i915
) >= IP_VER(12, 70)) {
4911 * There's 1 SAMEDIA gt and 1 OAM per SAMEDIA gt. All media slices
4912 * within the gt use the same OAM. All MTL SKUs list 1 SA MEDIA.
4914 drm_WARN_ON(&engine
->i915
->drm
,
4915 engine
->gt
->type
!= GT_MEDIA
);
4917 return PERF_GROUP_OAM_SAMEDIA_0
;
4920 return PERF_GROUP_INVALID
;
4923 static u32
__oa_engine_group(struct intel_engine_cs
*engine
)
4925 switch (engine
->class) {
4927 return PERF_GROUP_OAG
;
4929 case VIDEO_DECODE_CLASS
:
4930 case VIDEO_ENHANCEMENT_CLASS
:
4931 return __oam_engine_group(engine
);
4934 return PERF_GROUP_INVALID
;
4938 static struct i915_perf_regs
__oam_regs(u32 base
)
4940 return (struct i915_perf_regs
) {
4942 GEN12_OAM_HEAD_POINTER(base
),
4943 GEN12_OAM_TAIL_POINTER(base
),
4944 GEN12_OAM_BUFFER(base
),
4945 GEN12_OAM_CONTEXT_CONTROL(base
),
4946 GEN12_OAM_CONTROL(base
),
4947 GEN12_OAM_DEBUG(base
),
4948 GEN12_OAM_STATUS(base
),
4949 GEN12_OAM_CONTROL_COUNTER_FORMAT_SHIFT
,
4953 static struct i915_perf_regs
__oag_regs(void)
4955 return (struct i915_perf_regs
) {
4957 GEN12_OAG_OAHEADPTR
,
4958 GEN12_OAG_OATAILPTR
,
4960 GEN12_OAG_OAGLBCTXCTRL
,
4961 GEN12_OAG_OACONTROL
,
4964 GEN12_OAG_OACONTROL_OA_COUNTER_FORMAT_SHIFT
,
4968 static void oa_init_groups(struct intel_gt
*gt
)
4970 int i
, num_groups
= gt
->perf
.num_perf_groups
;
4972 for (i
= 0; i
< num_groups
; i
++) {
4973 struct i915_perf_group
*g
= >
->perf
.group
[i
];
4975 /* Fused off engines can result in a group with num_engines == 0 */
4976 if (g
->num_engines
== 0)
4979 if (i
== PERF_GROUP_OAG
&& gt
->type
!= GT_MEDIA
) {
4980 g
->regs
= __oag_regs();
4982 } else if (GRAPHICS_VER_FULL(gt
->i915
) >= IP_VER(12, 70)) {
4983 g
->regs
= __oam_regs(mtl_oa_base
[i
]);
4989 static int oa_init_gt(struct intel_gt
*gt
)
4991 u32 num_groups
= num_perf_groups_per_gt(gt
);
4992 struct intel_engine_cs
*engine
;
4993 struct i915_perf_group
*g
;
4994 intel_engine_mask_t tmp
;
4996 g
= kcalloc(num_groups
, sizeof(*g
), GFP_KERNEL
);
5000 for_each_engine_masked(engine
, gt
, ALL_ENGINES
, tmp
) {
5001 u32 index
= __oa_engine_group(engine
);
5003 engine
->oa_group
= NULL
;
5004 if (index
< num_groups
) {
5005 g
[index
].num_engines
++;
5006 engine
->oa_group
= &g
[index
];
5010 gt
->perf
.num_perf_groups
= num_groups
;
5018 static int oa_init_engine_groups(struct i915_perf
*perf
)
5020 struct intel_gt
*gt
;
5023 for_each_gt(gt
, perf
->i915
, i
) {
5024 ret
= oa_init_gt(gt
);
5032 static void oa_init_supported_formats(struct i915_perf
*perf
)
5034 struct drm_i915_private
*i915
= perf
->i915
;
5035 enum intel_platform platform
= INTEL_INFO(i915
)->platform
;
5039 oa_format_add(perf
, I915_OA_FORMAT_A13
);
5040 oa_format_add(perf
, I915_OA_FORMAT_A13
);
5041 oa_format_add(perf
, I915_OA_FORMAT_A29
);
5042 oa_format_add(perf
, I915_OA_FORMAT_A13_B8_C8
);
5043 oa_format_add(perf
, I915_OA_FORMAT_B4_C8
);
5044 oa_format_add(perf
, I915_OA_FORMAT_A45_B8_C8
);
5045 oa_format_add(perf
, I915_OA_FORMAT_B4_C8_A16
);
5046 oa_format_add(perf
, I915_OA_FORMAT_C4_B8
);
5049 case INTEL_BROADWELL
:
5050 case INTEL_CHERRYVIEW
:
5053 case INTEL_KABYLAKE
:
5054 case INTEL_GEMINILAKE
:
5055 case INTEL_COFFEELAKE
:
5056 case INTEL_COMETLAKE
:
5058 case INTEL_ELKHARTLAKE
:
5059 case INTEL_JASPERLAKE
:
5060 case INTEL_TIGERLAKE
:
5061 case INTEL_ROCKETLAKE
:
5063 case INTEL_ALDERLAKE_S
:
5064 case INTEL_ALDERLAKE_P
:
5065 oa_format_add(perf
, I915_OA_FORMAT_A12
);
5066 oa_format_add(perf
, I915_OA_FORMAT_A12_B8_C8
);
5067 oa_format_add(perf
, I915_OA_FORMAT_A32u40_A4u32_B8_C8
);
5068 oa_format_add(perf
, I915_OA_FORMAT_C4_B8
);
5072 oa_format_add(perf
, I915_OAR_FORMAT_A32u40_A4u32_B8_C8
);
5073 oa_format_add(perf
, I915_OA_FORMAT_A24u40_A14u32_B8_C8
);
5076 case INTEL_METEORLAKE
:
5077 oa_format_add(perf
, I915_OAR_FORMAT_A32u40_A4u32_B8_C8
);
5078 oa_format_add(perf
, I915_OA_FORMAT_A24u40_A14u32_B8_C8
);
5079 oa_format_add(perf
, I915_OAM_FORMAT_MPEC8u64_B8_C8
);
5080 oa_format_add(perf
, I915_OAM_FORMAT_MPEC8u32_B8_C8
);
5084 MISSING_CASE(platform
);
5088 static void i915_perf_init_info(struct drm_i915_private
*i915
)
5090 struct i915_perf
*perf
= &i915
->perf
;
5092 switch (GRAPHICS_VER(i915
)) {
5094 perf
->ctx_oactxctrl_offset
= 0x120;
5095 perf
->ctx_flexeu0_offset
= 0x2ce;
5096 perf
->gen8_valid_ctx_bit
= BIT(25);
5099 perf
->ctx_oactxctrl_offset
= 0x128;
5100 perf
->ctx_flexeu0_offset
= 0x3de;
5101 perf
->gen8_valid_ctx_bit
= BIT(16);
5104 perf
->ctx_oactxctrl_offset
= 0x124;
5105 perf
->ctx_flexeu0_offset
= 0x78e;
5106 perf
->gen8_valid_ctx_bit
= BIT(16);
5110 * Calculate offset at runtime in oa_pin_context for gen12 and
5111 * cache the value in perf->ctx_oactxctrl_offset.
5115 MISSING_CASE(GRAPHICS_VER(i915
));
5120 * i915_perf_init - initialize i915-perf state on module bind
5121 * @i915: i915 device instance
5123 * Initializes i915-perf state without exposing anything to userspace.
5125 * Note: i915-perf initialization is split into an 'init' and 'register'
5126 * phase with the i915_perf_register() exposing state to userspace.
5128 int i915_perf_init(struct drm_i915_private
*i915
)
5130 struct i915_perf
*perf
= &i915
->perf
;
5132 perf
->oa_formats
= oa_formats
;
5133 if (IS_HASWELL(i915
)) {
5134 perf
->ops
.is_valid_b_counter_reg
= gen7_is_valid_b_counter_addr
;
5135 perf
->ops
.is_valid_mux_reg
= hsw_is_valid_mux_addr
;
5136 perf
->ops
.is_valid_flex_reg
= NULL
;
5137 perf
->ops
.enable_metric_set
= hsw_enable_metric_set
;
5138 perf
->ops
.disable_metric_set
= hsw_disable_metric_set
;
5139 perf
->ops
.oa_enable
= gen7_oa_enable
;
5140 perf
->ops
.oa_disable
= gen7_oa_disable
;
5141 perf
->ops
.read
= gen7_oa_read
;
5142 perf
->ops
.oa_hw_tail_read
= gen7_oa_hw_tail_read
;
5143 } else if (HAS_LOGICAL_RING_CONTEXTS(i915
)) {
5144 /* Note: that although we could theoretically also support the
5145 * legacy ringbuffer mode on BDW (and earlier iterations of
5146 * this driver, before upstreaming did this) it didn't seem
5147 * worth the complexity to maintain now that BDW+ enable
5148 * execlist mode by default.
5150 perf
->ops
.read
= gen8_oa_read
;
5151 i915_perf_init_info(i915
);
5153 if (IS_GRAPHICS_VER(i915
, 8, 9)) {
5154 perf
->ops
.is_valid_b_counter_reg
=
5155 gen7_is_valid_b_counter_addr
;
5156 perf
->ops
.is_valid_mux_reg
=
5157 gen8_is_valid_mux_addr
;
5158 perf
->ops
.is_valid_flex_reg
=
5159 gen8_is_valid_flex_addr
;
5161 if (IS_CHERRYVIEW(i915
)) {
5162 perf
->ops
.is_valid_mux_reg
=
5163 chv_is_valid_mux_addr
;
5166 perf
->ops
.oa_enable
= gen8_oa_enable
;
5167 perf
->ops
.oa_disable
= gen8_oa_disable
;
5168 perf
->ops
.enable_metric_set
= gen8_enable_metric_set
;
5169 perf
->ops
.disable_metric_set
= gen8_disable_metric_set
;
5170 perf
->ops
.oa_hw_tail_read
= gen8_oa_hw_tail_read
;
5171 } else if (GRAPHICS_VER(i915
) == 11) {
5172 perf
->ops
.is_valid_b_counter_reg
=
5173 gen7_is_valid_b_counter_addr
;
5174 perf
->ops
.is_valid_mux_reg
=
5175 gen11_is_valid_mux_addr
;
5176 perf
->ops
.is_valid_flex_reg
=
5177 gen8_is_valid_flex_addr
;
5179 perf
->ops
.oa_enable
= gen8_oa_enable
;
5180 perf
->ops
.oa_disable
= gen8_oa_disable
;
5181 perf
->ops
.enable_metric_set
= gen8_enable_metric_set
;
5182 perf
->ops
.disable_metric_set
= gen11_disable_metric_set
;
5183 perf
->ops
.oa_hw_tail_read
= gen8_oa_hw_tail_read
;
5184 } else if (GRAPHICS_VER(i915
) == 12) {
5185 perf
->ops
.is_valid_b_counter_reg
=
5186 HAS_OA_SLICE_CONTRIB_LIMITS(i915
) ?
5187 xehp_is_valid_b_counter_addr
:
5188 gen12_is_valid_b_counter_addr
;
5189 perf
->ops
.is_valid_mux_reg
=
5190 gen12_is_valid_mux_addr
;
5191 perf
->ops
.is_valid_flex_reg
=
5192 gen8_is_valid_flex_addr
;
5194 perf
->ops
.oa_enable
= gen12_oa_enable
;
5195 perf
->ops
.oa_disable
= gen12_oa_disable
;
5196 perf
->ops
.enable_metric_set
= gen12_enable_metric_set
;
5197 perf
->ops
.disable_metric_set
= gen12_disable_metric_set
;
5198 perf
->ops
.oa_hw_tail_read
= gen12_oa_hw_tail_read
;
5202 if (perf
->ops
.enable_metric_set
) {
5203 struct intel_gt
*gt
;
5206 for_each_gt(gt
, i915
, i
)
5207 mutex_init(>
->perf
.lock
);
5209 /* Choose a representative limit */
5210 oa_sample_rate_hard_limit
= to_gt(i915
)->clock_frequency
/ 2;
5212 mutex_init(&perf
->metrics_lock
);
5213 idr_init_base(&perf
->metrics_idr
, 1);
5215 /* We set up some ratelimit state to potentially throttle any
5216 * _NOTES about spurious, invalid OA reports which we don't
5217 * forward to userspace.
5219 * We print a _NOTE about any throttling when closing the
5220 * stream instead of waiting until driver _fini which no one
5223 * Using the same limiting factors as printk_ratelimit()
5225 ratelimit_state_init(&perf
->spurious_report_rs
, 5 * HZ
, 10);
5226 /* Since we use a DRM_NOTE for spurious reports it would be
5227 * inconsistent to let __ratelimit() automatically print a
5228 * warning for throttling.
5230 ratelimit_set_flags(&perf
->spurious_report_rs
,
5231 RATELIMIT_MSG_ON_RELEASE
);
5233 ratelimit_state_init(&perf
->tail_pointer_race
,
5235 ratelimit_set_flags(&perf
->tail_pointer_race
,
5236 RATELIMIT_MSG_ON_RELEASE
);
5238 atomic64_set(&perf
->noa_programming_delay
,
5239 500 * 1000 /* 500us */);
5243 ret
= oa_init_engine_groups(perf
);
5246 "OA initialization failed %d\n", ret
);
5250 oa_init_supported_formats(perf
);
5256 static int destroy_config(int id
, void *p
, void *data
)
5258 i915_oa_config_put(p
);
5262 int i915_perf_sysctl_register(void)
5264 sysctl_header
= register_sysctl("dev/i915", oa_table
);
5268 void i915_perf_sysctl_unregister(void)
5270 unregister_sysctl_table(sysctl_header
);
5274 * i915_perf_fini - Counter part to i915_perf_init()
5275 * @i915: i915 device instance
5277 void i915_perf_fini(struct drm_i915_private
*i915
)
5279 struct i915_perf
*perf
= &i915
->perf
;
5280 struct intel_gt
*gt
;
5286 for_each_gt(gt
, perf
->i915
, i
)
5287 kfree(gt
->perf
.group
);
5289 idr_for_each(&perf
->metrics_idr
, destroy_config
, perf
);
5290 idr_destroy(&perf
->metrics_idr
);
5292 memset(&perf
->ops
, 0, sizeof(perf
->ops
));
5297 * i915_perf_ioctl_version - Version of the i915-perf subsystem
5298 * @i915: The i915 device
5300 * This version number is used by userspace to detect available features.
5302 int i915_perf_ioctl_version(struct drm_i915_private
*i915
)
5305 * 1: Initial version
5306 * I915_PERF_IOCTL_ENABLE
5307 * I915_PERF_IOCTL_DISABLE
5309 * 2: Added runtime modification of OA config.
5310 * I915_PERF_IOCTL_CONFIG
5312 * 3: Add DRM_I915_PERF_PROP_HOLD_PREEMPTION parameter to hold
5313 * preemption on a particular context so that performance data is
5314 * accessible from a delta of MI_RPC reports without looking at the
5317 * 4: Add DRM_I915_PERF_PROP_ALLOWED_SSEU to limit what contexts can
5318 * be run for the duration of the performance recording based on
5319 * their SSEU configuration.
5321 * 5: Add DRM_I915_PERF_PROP_POLL_OA_PERIOD parameter that controls the
5322 * interval for the hrtimer used to check for OA data.
5324 * 6: Add DRM_I915_PERF_PROP_OA_ENGINE_CLASS and
5325 * DRM_I915_PERF_PROP_OA_ENGINE_INSTANCE
5327 * 7: Add support for video decode and enhancement classes.
5331 * Wa_14017512683: mtl[a0..c0): Use of OAM must be preceded with Media
5332 * C6 disable in BIOS. If Media C6 is enabled in BIOS, return version 6
5333 * to indicate that OA media is not supported.
5335 if (IS_MTL_MEDIA_STEP(i915
, STEP_A0
, STEP_C0
)) {
5336 struct intel_gt
*gt
;
5339 for_each_gt(gt
, i915
, i
) {
5340 if (gt
->type
== GT_MEDIA
&&
5341 intel_check_bios_c6_setup(>
->rc6
))
5349 #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
5350 #include "selftests/i915_perf.c"