RAR5 reader: fix potential memory leak

If a file declares more than 8192 filters at once, without consuming
them, then the filter allocation function leaks memory.

The backing array used for storing filter pointers can hold up to 8192
pointers. After that it won't accept any new entries. The problem was
that the caller code didn't check if the backing array has accepted the
pointer; it silently assumed the pointer was registered, disposing the
only variable that was holding the pointer to allocated memory.

The fix is to fail the creation of a new filter structure if the backing
array is full. This will result in failure to unpack, but having more
than 8192 filters in one file at the same time seems unlikely.

Fixes issue #2891

diff --git a/libarchive/archive_read_support_format_rar5.c b/libarchive/archive_read_support_format_rar5.c
index 17e501e02e9f1733c0c2f7abf4c6026db0f898a7..cc7a584c4161917a85fd39b15b27e3db4ff1d7a2 100644 (file)
--- a/libarchive/archive_read_support_format_rar5.c
+++ b/libarchive/archive_read_support_format_rar5.c
@@ -429,8 +429,7 @@ static int cdeque_front(struct cdeque* d, void** value) {
                return CDE_OUT_OF_BOUNDS;
 }
 
-/* Pushes a new element into the end of this circular deque object. If current
- * size will exceed capacity, the oldest element will be overwritten. */
+/* Pushes a new element into the end of this circular deque object. */
 static int cdeque_push_back(struct cdeque* d, void* item) {
        if(d == NULL)
                return CDE_PARAM;
@@ -554,7 +553,11 @@ static struct filter_info* add_new_filter(struct rar5* rar) {
                return NULL;
        }
 
-       cdeque_push_back(&rar->cstate.filters, cdeque_filter(f));
+       if (CDE_OK != cdeque_push_back(&rar->cstate.filters, cdeque_filter(f))) {
+               free(f);
+               return NULL;
+       }
+
        return f;
 }