RAR5 reader: fix ASan errors, fix OSSFuzz samples, add a unit test

RAR5 reader: fix ASan errors, fix OSSFuzz samples, add a unit test

This commit fixes errors reported by ASan, as well as fixes runtime
behavior of RAR5 reader on OSSFuzz sample files:

    #12999, #13029, #13144, #13478, #13490

Root cause for these changes is that merge_block() function was
sometimes called in a recursive way. But this function shouldn't be used
this way, because calling it recursively overwrites the global state
that is used by the function. So, the commit ensures the function will
not be called recursively.

There is also one fix that changes some tabs to spaces, because whole
file originally used space indentation.