factor: avoid out-of-bounds prime table access

With an ASAN build we can trigger the issue like:

  $ src/factor 22486472939460651857
  src/factor.c:888:36: runtime error:
  index 675 out of bounds for type 'int_least16_t [675]'

* src/factor.c (factor_using_division): Guard the final unrolled block
before reading a subsequent prime.
* tests/factor/factor.pl: Add a test case.
* NEWS: Mention the bug fix.
Problem caught by Coverity analysis.

diff --git a/NEWS b/NEWS
index b1ec92562caa00f8f241b41a67f613cb1d1c80a7..0e3d9bd5f9196ee8f1f17d5736668287b706ddfa 100644 (file)
--- a/NEWS
+++ b/NEWS
@@ -16,6 +16,9 @@ GNU coreutils NEWS                                    -*- outline -*-
   if N is negative.  Previously it behaved as if N were zero.
   [bug introduced in coreutils-9.4]
 
+  'factor' avoids a buffer over-read (CWE-126) for certain values.
+  [bug introduced in coreutils-9.8]
+
   'head' and 'tail' now quote names in file headers when needed.
   [This bug was present in "the beginning".]
 

diff --git a/src/factor.c b/src/factor.c
index e9c37320728b0494e879f3c4d9b2189582d3226c..615cf6ada7c94ebb38d44ceb3668db185d281c46 100644 (file)
--- a/src/factor.c
+++ b/src/factor.c
@@ -885,6 +885,8 @@ factor_using_division (struct factors *factors, mp_limb_t t1, mp_limb_t t0,
       t0 = divblock (factors, t0, pd, i, 6);
       t0 = divblock (factors, t0, pd, i, 7);
 
+      if (PRIMES_PTAB_ENTRIES <= i + 8)
+        break;
       int_least32_t p = primes_ptab[i + 8];
       if (p * p > t0)
         break;

diff --git a/tests/factor/factor.pl b/tests/factor/factor.pl
index 2b8e540869603011f8468ae03ee54e8cb7d4a5f1..524c57baa8114b724996d350ebc5ebf3a0da972e 100755 (executable)
--- a/tests/factor/factor.pl
+++ b/tests/factor/factor.pl
@@ -73,6 +73,8 @@ my @Tests =
      ['bug-2012-c', '6635692801', {OUT => '57601 115201'}],
      ['bug-2012-d', '17709149503', {OUT => '94099 188197'}],
      ['bug-2012-e', '17754345703', {OUT => '94219 188437'}],
+     ['bug-prime-table-end', '22486472939460651857',
+      {OUT => '4993 4503599627370449'}],
      # Infinite loop bugs in v8.20 to 8.26 inclusive
      ['bug-2016-a', '158909489063877810457',
       {OUT => '3401347 3861211 12099721'}],