lua: don't accept a table as a return value from match

lua: don't accept a table as a return value from match

Remove the half finished support for accepting a table returned from a
Lua rule's match function. This is not documented, not tested, and not
really implemented.

Also, use lua_tointeger to get the return value from the match function
as an integer instead of a float.

Ticket: #6941