media: verisilicon: Fix TRY_FMT on encoder OUTPUT

[ Upstream commit b3b4c9d3cb3bf8725a3ded26f7042b1a37f25333 ]

Commit f100ce3bbd6a ("media: verisilicon: Fix crash when probing
encoder") removed vpu_fmt from hantro_try_fmt(), since it was
initialized from vpu_dst_fmt, which may not be initialized, when TRY_FMT
is called. It was replaced by fmt, which is found using the pixelformat.

For the encoder, this changed the fmt to contain the raw format instead
of the coded format. The format constraints as of fmt->frmsize are only
valid for the coded format and are 0 for the raw formats. Therefore, the
size of a encoder OUTPUT device is constrained to 0 and the
v4l2-compliance tests for G_FMT, TRY_FMT, and SET_FMT fail.

Bring back vpu_fmt to use the coded format on an encoder OUTPUT device,
but initialize it using the currently set pixelformat on dst_fmt, which
is the coded format on an encoder.

Fixes: f100ce3bbd6a ("media: verisilicon: Fix crash when probing encoder")
Signed-off-by: Michael Tretter <m.tretter@pengutronix.de>
Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Signed-off-by: Sasha Levin <sashal@kernel.org>

diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
index 61cfaaf4e927b35a81e55ccae6e2d4b8b2f13dd5..e56d58fe28022f2e29ba24623c69abc30c07a707 100644 (file)
--- a/drivers/media/platform/verisilicon/hantro_v4l2.c
+++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
@@ -276,6 +276,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
                          enum v4l2_buf_type type)
 {
        const struct hantro_fmt *fmt;
+       const struct hantro_fmt *vpu_fmt;
        bool capture = V4L2_TYPE_IS_CAPTURE(type);
        bool coded;
 
@@ -295,19 +296,23 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
 
        if (coded) {
                pix_mp->num_planes = 1;
-       } else if (!ctx->is_encoder) {
+               vpu_fmt = fmt;
+       } else if (ctx->is_encoder) {
+               vpu_fmt = hantro_find_format(ctx, ctx->dst_fmt.pixelformat);
+       } else {
                /*
                 * Width/height on the CAPTURE end of a decoder are ignored and
                 * replaced by the OUTPUT ones.
                 */
                pix_mp->width = ctx->src_fmt.width;
                pix_mp->height = ctx->src_fmt.height;
+               vpu_fmt = fmt;
        }
 
        pix_mp->field = V4L2_FIELD_NONE;
 
        v4l2_apply_frmsize_constraints(&pix_mp->width, &pix_mp->height,
-                                      &fmt->frmsize);
+                                      &vpu_fmt->frmsize);
 
        if (!coded) {
                /* Fill remaining fields */