migration/ram: Implement save_postcopy_prepare()

Implement save_postcopy_prepare(), preparing for the enablement
of both multifd and postcopy.

Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Message-ID: <20250411114534.3370816-5-ppandit@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration: Add save_postcopy_prepare() savevm handler

Add a savevm handler for a module to opt-in sending extra sections right
before postcopy starts, and before VM is stopped.

RAM will start to use this new savevm handler in the next patch to do flush
and sync for multifd pages.

Note that we choose to do it before VM stopped because the current only
potential user is not sensitive to VM status, so doing it before VM is
stopped is preferred to enlarge any postcopy downtime.

It is still a bit unfortunate that we need to introduce such a new savevm
handler just for the only use case, however it's so far the cleanest.

Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Message-ID: <20250411114534.3370816-4-ppandit@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration: refactor channel discovery mechanism

The various logical migration channels don't have a
standardized way of advertising themselves and their
connections may be seen out of order by the migration
destination. When a new connection arrives, the incoming
migration currently make use of heuristics to determine
which channel it belongs to.

The next few patches will need to change how the multifd
and postcopy capabilities interact and that affects the
channel discovery heuristic.

Refactor the channel discovery heuristic to make it less
opaque and simplify the subsequent patches.

Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Message-ID: <20250411114534.3370816-3-ppandit@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration/multifd: move macros to multifd header

Move MULTIFD_ macros to the header file so that
they are accessible from other source files.

Reviewed-by: Fabiano Rosas <farosas@suse.de>
Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
Reviewed-by: Peter Xu <peterx@redhat.com>
Message-ID: <20250411114534.3370816-2-ppandit@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration: Fix latent bug in migrate_params_test_apply()

migrate_params_test_apply() neglects to apply tls_authz.  Currently
harmless, because migrate_params_check() doesn't care.  Fix it anyway.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Message-ID: <20250407072833.2118928-1-armbru@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration: Add qtest for migration over RDMA

This qtest requires there is a RDMA(RoCE) link in the host.
In order to make the test work smoothly, introduce a
scripts/rdma-migration-helper.sh to detect existing RoCE link before
running the test.

Test will be skipped if there is no available RoCE link.
 # Start of rdma tests
 # Running /x86_64/migration/precopy/rdma/plain
 ok 1 /x86_64/migration/precopy/rdma/plain # SKIP No rdma link available
 # To enable the test:
 # Run 'scripts/rdma-migration-helper.sh setup' with root to setup a new rdma/rxe link and rerun the test
 # Optional: run 'scripts/rdma-migration-helper.sh clean' to revert the 'setup'

 # End of rdma tests

Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
Cc: Stefan Hajnoczi <stefanha@gmail.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
Message-ID: <20250311024221.363421-1-lizhijian@fujitsu.com>
[add 'head -1' to script, reformat test message]
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration: Unfold control_save_page()

control_save_page() is for RDMA only, unfold it to make the code more
clear.
In addition:
 - Similar to other branches style in ram_save_target_page(), involve RDMA
   only if the condition 'migrate_rdma()' is true.
 - Further simplify the code by removing the RAM_SAVE_CONTROL_NOT_SUPP.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
Message-ID: <20250305062825.772629-6-lizhijian@fujitsu.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration/rdma: Remove redundant migration_in_postcopy checks

Since we have disabled RDMA + postcopy, it's safe to remove
the migration_in_postcopy() that follows the migrate_rdma().

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
Message-ID: <20250305062825.772629-5-lizhijian@fujitsu.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration: disable RDMA + postcopy-ram

It's believed that RDMA + postcopy-ram has been broken for a while.
Rather than spending time re-enabling it, let's simply disable it as a
trade-off.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
Message-ID: <20250305062825.772629-4-lizhijian@fujitsu.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

migration: check RDMA and capabilities are compatible on both sides

Depending on the order of starting RDMA and setting capability,
they can be categorized into the following scenarios:
Source:
 S1: [set capabilities] -> [Start RDMA outgoing]
Destination:
 D1: [set capabilities] -> [Start RDMA incoming]
 D2: [Start RDMA incoming] -> [set capabilities]

Previously, compatibility between RDMA and capabilities was verified only
in scenario D1, potentially causing migration failures in other situations.

For scenarios S1 and D1, we can seamlessly incorporate
migration_transport_compatible() to address compatibility between
channels and capabilities vs transport.

For scenario D2, ensure compatibility within migrate_caps_check().

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
Message-ID: <20250305062825.772629-3-lizhijian@fujitsu.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

Merge tag 'block-pull-request' of https://gitlab.com/stefanha/qemu into staging

Pull request

Kevin's fix for the divide-by-zero in my recent discard commit, triggered when
a host block device does not support discard.

# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCgAdFiEEhpWov9P5fNqsNXdanKSrs4Grc8gFAmgSRfsACgkQnKSrs4Gr
# c8hGBwf8CBoDZzCJAE1sw2GSKnnd3J2qGf4Kg6CcMYOSZ8TLssDKQj6HG2gfWaJZ
# it9g9zq7TsodWCyV/qXrzOy5aa7WX8Tsf10O/87baFqGOp82KMPX8jQK1csRnTTF
# QyDocZhIvO+QJXnmnFjtvY7qfaxkzaT/8U+mWgaQM2zG83BNGg3uNyRPyz+RAfYl
# tVM3xNf2ETbN3D8SIOcpr80/tiWP8dZ8xTLyTfBYPbIP59QX2+Iu8BtLFt9npwT6
# kABnFkqnE/pA6FJz0ZIVenduOBs7IUSQFNvmxAjYIwxowQKsk4WFfjJEKHIHzwwO
# a64i43DcH8XgjCcueJ11DnmoB5RfAg==
# =yODA
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 30 Apr 2025 11:47:07 EDT
# gpg:                using RSA key 8695A8BFD3F97CDAAC35775A9CA4ABB381AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>" [ultimate]
# gpg:                 aka "Stefan Hajnoczi <stefanha@gmail.com>" [ultimate]
# Primary key fingerprint: 8695 A8BF D3F9 7CDA AC35  775A 9CA4 ABB3 81AB 73C8

* tag 'block-pull-request' of https://gitlab.com/stefanha/qemu:
  file-posix: Fix crash on discard_granularity == 0

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

Merge tag 'pull-tcg-20250429' of https://gitlab.com/rth7680/qemu into staging

Convert TCG backend code generators to TCGOutOp structures,
decomposing the monolithic tcg_out_op functions.

# -----BEGIN PGP SIGNATURE-----
#
# iQFRBAABCgA7FiEEekgeeIaLTbaoWgXAZN846K9+IV8FAmgRDgcdHHJpY2hhcmQu
# aGVuZGVyc29uQGxpbmFyby5vcmcACgkQZN846K9+IV+0SQf9Ef8IE5QWjXEws706
# q+1deeDV2xa1sKAHG5fnskk1bXM31Cy4koNdDQbHTtDWedNEzWKyR7FxjlRm+zSW
# 6CposaEEsCGxdKUbvraflGbWuf2NUZpJoreNGo9BYZkfNWE9yJ0HlypjpxclziRA
# G0Ro4XMevi+yVA3cd8lEmft9cW+woFrVWu5I4tucMwY/8gzWRiHV4Z5YCeCEjD3C
# 3YYukhRTaA+7Lyd1G3rcqh8uSsAGGv2NLO26upK1mnVtZOoS/CgKWfBq5enVEuDd
# X6T544ipz8Z3eXFgzTzK4nl7TpmO+XEUbja3op7psrLHU84puX1/47HLk4TkHlE+
# 6/95eA==
# =vzOx
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 29 Apr 2025 13:36:07 EDT
# gpg:                using RSA key 7A481E78868B4DB6A85A05C064DF38E8AF7E215F
# gpg:                issuer "richard.henderson@linaro.org"
# gpg: Good signature from "Richard Henderson <richard.henderson@linaro.org>" [full]
# Primary key fingerprint: 7A48 1E78 868B 4DB6 A85A  05C0 64DF 38E8 AF7E 215F

* tag 'pull-tcg-20250429' of https://gitlab.com/rth7680/qemu: (161 commits)
  tcg/sparc64: Implement CTPOP
  tcg/sparc64: Unexport use_vis3_instructions
  tcg: Remove tcg_out_op
  tcg: Convert qemu_st{2} to TCGOutOpLdSt{2}
  tcg: Convert qemu_ld{2} to TCGOutOpLoad{2}
  tcg: Merge INDEX_op_{ld,st}_{i32,i64,i128}
  tcg: Remove INDEX_op_qemu_st8_*
  tcg: Stash MemOp size in TCGOP_FLAGS
  tcg: Merge INDEX_op_st*_{i32,i64}
  tcg: Convert st to TCGOutOpStore
  tcg: Merge INDEX_op_ld*_{i32,i64}
  tcg: Convert ld to TCGOutOpLoad
  tcg: Formalize tcg_out_goto_ptr
  tcg: Formalize tcg_out_br
  tcg: Formalize tcg_out_mb
  tcg: Remove add2/sub2 opcodes
  tcg/tci: Implement add/sub carry opcodes
  tcg/sparc64: Implement add/sub carry opcodes
  tcg/sparc64: Hoist tcg_cond_to_bcond lookup out of tcg_out_movcc
  tcg/s390x: Use ADD LOGICAL WITH SIGNED IMMEDIATE
  ...

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

file-posix: Fix crash on discard_granularity == 0

Block devices that don't support discard have a discard_granularity of
0. Currently, this results in a division by zero when we try to make
sure that it's a multiple of request_alignment. Only try to update
bs->bl.pdiscard_alignment when we got a non-zero discard_granularity
from sysfs.

Fixes: f605796aae4 ('file-posix: probe discard alignment on Linux block devices')
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-ID: <20250429155654.102735-1-kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

tcg/sparc64: Implement CTPOP

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/sparc64: Unexport use_vis3_instructions

This variable is no longer used outside tcg-target.c.inc.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Remove tcg_out_op

All integer opcodes are now converted to TCGOutOp.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert qemu_st{2} to TCGOutOpLdSt{2}

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert qemu_ld{2} to TCGOutOpLoad{2}

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_{ld,st}_{i32,i64,i128}

Merge into INDEX_op_{ld,st,ld2,st2}, where "2" indicates that two
inputs or outputs are required. This simplifies the processing of
i64/i128 depending on host word size.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Remove INDEX_op_qemu_st8_*

The i386 backend can now check TCGOP_FLAGS to select
the correct set of constraints.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Stash MemOp size in TCGOP_FLAGS

This will enable removing INDEX_op_qemu_st8_*_i32,
by exposing the operand size to constraint selection.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_st*_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert st to TCGOutOpStore

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_ld*_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert ld to TCGOutOpLoad

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Formalize tcg_out_goto_ptr

Split these functions out from tcg_out_op.
Define outop_goto_ptr generically.
Call tcg_out_goto_ptr from tcg_reg_alloc_op.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Formalize tcg_out_br

Split these functions out from tcg_out_op.
Call it directly from tcg_gen_code.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Formalize tcg_out_mb

Most tcg backends already have a function for this;
the rest can split one out from tcg_out_op.
Call it directly from tcg_gen_code.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Remove add2/sub2 opcodes

All uses have been replaced by add/sub carry opcodes.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Implement add/sub carry opcodes

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/sparc64: Implement add/sub carry opcodes

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/sparc64: Hoist tcg_cond_to_bcond lookup out of tcg_out_movcc

Pass the sparc COND_* value not the tcg TCG_COND_* value.
This makes the usage within add2/sub2 clearer.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/s390x: Use ADD LOGICAL WITH SIGNED IMMEDIATE

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/s390x: Implement add/sub carry opcodes

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/s390x: Add TCG_CT_CONST_N32

We were using S32 | U32 for add2/sub2.  But the ALGFI and SLGFI
insns that implement this both have uint32_t immediates.
This makes the composite range balanced and
enables use of -0xffffffff ... -0x80000001.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/s390x: Honor carry_live in tcg_out_movi

Do not clobber flags if they're live.  Required in order
to perform register allocation on add/sub carry opcodes.
LA and AGHI are the same size, so use LA unconditionally.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/ppc: Implement add/sub carry opcodes

Tested-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/arm: Implement add/sub carry opcodes

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/aarch64: Implement add/sub carry opcodes

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/tricore: Use tcg_gen_addcio_i32 for gen_addc_CC

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/sparc: Use tcg_gen_addcio_tl for gen_op_addcc_int

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/sh4: Use tcg_gen_addcio_i32 for addc

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/s390x: Use tcg_gen_addcio_i64 for op_addc64

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/ppc: Use tcg_gen_addcio_tl for ADD and SUBF

Tested-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/openrisc: Use tcg_gen_addcio_* for ADDC

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/microblaze: Use tcg_gen_addcio_i32

Use this in gen_addc and gen_rsubc, both of which need
add with carry-in and carry-out.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/hppa: Use tcg_gen_addcio_i64

Use this in do_add, do_sub, and do_ds, all of which need
add with carry-in and carry-out.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

target/arm: Use tcg_gen_addcio_* for ADCS

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Add tcg_gen_addcio_{i32,i64,tl}

Create a function for performing an add with carry-in
and producing carry out.  The carry-out result is boolean.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/i386: Special case addci r, 0, 0

Using addci with two zeros as input in order to capture the value
of the carry-in bit is common.  Special case this with sbb+neg so
that we do not have to load 0 into a register first.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/i386: Implement add/sub carry opcodes

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/i386: Honor carry_live in tcg_out_movi

Do not clobber flags if they're live.  Required in order
to perform register allocation on add/sub carry opcodes.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Use sub carry opcodes to expand sub2

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Use add carry opcodes to expand add2

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/optimize: With two const operands, prefer 0 in arg1

For most binary operands, two const operands fold.
However, the add/sub carry opcodes have a third input.
Prefer "reg, zero, const" since many risc hosts have a
zero register that can fit a "reg, reg, const" insn format.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/optimize: Handle add/sub with carry opcodes

Propagate known carry when possible, and simplify the opcodes
to not require carry-in when known.  The result will be cleaned
up further by the subsequent liveness analysis pass.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Add TCGOutOp structures for add/sub carry opcodes

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Add add/sub with carry opcodes and infrastructure

Liveness needs to track carry-live state in order to
determine if the (hidden) output of the opcode is used.
Code generation needs to track carry-live state in order
to avoid clobbering cpu flags when loading constants.

So far, output routines and backends are unchanged.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Sink def, nb_iargs, nb_oargs loads in liveness_pass_1

Sink the sets of the def, nb_iargs, nb_oargs variables to
the default and do_not_remove labels.  They're not really
needed beforehand, and it avoids preceding code from having
to keep them up-to-date.  Note that def had *not* been kept
up-to-date; thankfully only def->flags had been used and
those bits were constant between opcode changes.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Move i into each for loop in liveness_pass_1

Use per-loop variables instead of one 'i' for the function.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/riscv: Drop support for add2/sub2

We now produce exactly the same code via generic expansion.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/mips: Drop support for add2/sub2

We now produce exactly the same code via generic expansion.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Do not default add2/sub2_i32 for 32-bit hosts

Require TCG_TARGET_HAS_{add2,sub2}_i32 be defined,
one way or another.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Expand fallback sub2 with 32-bit operations

No need to expand to i64 to perform the subtract.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Expand fallback add2 with 32-bit operations

No need to expand to i64 to perform the add.
This is smaller on a loongarch64 host, e.g.

bstrpick_d  r28, r27, 31, 0
bstrpick_d  r29, r24, 31, 0
add_d       r28, r28, r29
addi_w      r29, r28, 0
srai_d      r28, r28, 32
  ---
add_w       r28, r27, r24
sltu        r29, r28, r24

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_extract2_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert extract2 to TCGOutOpExtract2

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_deposit_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/aarch64: Improve deposit

Use ANDI for deposit 0 into a register.
Use UBFIZ, aka UBFM, for deposit register into 0.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert deposit to TCGOutOpDeposit

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert extrh_i64_i32 to TCGOutOpUnary

At the same time, make extrh_i64_i32 mandatory.  This closes a hole
in which move arguments could be cast between TCGv_i32 and TCGv_i64.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert extrl_i64_i32 to TCGOutOpUnary

Drop the cast from TCGv_i64 to TCGv_i32 in tcg_gen_extrl_i64_i32
an emit extrl_i64_i32 unconditionally.  Move that special case
to tcg_gen_code when we find out if the output is live or dead.
In this way even hosts that canonicalize truncations can make
use of a store directly from the 64-bit host register.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert extu_i32_i64 to TCGOutOpUnary

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert ext_i32_i64 to TCGOutOpUnary

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_sextract_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert sextract to TCGOutOpExtract

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_extract_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert extract to TCGOutOpExtract

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Rename INDEX_op_bswap64_i64 to INDEX_op_bswap64

Even though bswap64 can only be used with TCG_TYPE_I64,
rename the opcode to maintain uniformity.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert bswap64 to TCGOutOpUnary

Use TCGOutOpUnary instead of TCGOutOpBswap because the
flags are not used with this opcode; they are merely
present for uniformity with the smaller bswaps.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_bswap32_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert bswap32 to TCGOutOpBswap

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_bswap16_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert bswap16 to TCGOutOpBswap

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert setcond2_i32 to TCGOutOpSetcond2

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert brcond2_i32 to TCGOutOpBrcond2

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/ppc: Expand arguments to tcg_out_cmp2

Tested-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/arm: Expand arguments to tcg_out_cmp2

Pass explicit arguments instead of arrays.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/ppc: Drop fallback constant loading in tcg_out_cmp

Use U and C constraints for brcond2 and setcond2, so that tcg_out_cmp2
automatically passes in-range constants to tcg_out_cmp.

Tested-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_movcond_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert movcond to TCGOutOpMovcond

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_brcond_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert brcond to TCGOutOpBrcond

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_{neg}setcond_{i32,i64}`

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Convert setcond, negsetcond to TCGOutOpSetcond

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Remove TCG_TARGET_HAS_negsetcond_{i32,i64}

All targets now provide negsetcond, so remove the conditional.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Support negsetcond

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/mips: Support negsetcond

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/loongarch64: Support negsetcond

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Merge INDEX_op_mulu2_{i32,i64}

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>