btree_gist: Fix memory allocation formula

btree_gist: Fix memory allocation formula

This change has been suggested by the two authors listed in this commit,
both of them providing an incomplete solution (David's formula relied on
a "bytea *", while Bertrand's did not use palloc_array()).  The solution
provided in this commit uses GBT_VARKEY instead of the inconsistent
bytea for the allocation size, with a palloc_array().

The change related to Vsrt is one I am flipping to a more consistent
style, in passing.

Author: David Geier <geidav.pg@gmail.com>
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Discussion: https://postgr.es/m/ad0748d4-3080-436e-b0bc-ac8f86a3466a@gmail.com
Discussion: https://postgr.es/m/aTrG3Fi4APtfiCvQ@ip-10-97-1-34.eu-west-3.compute.internal