]> git.ipfire.org Git - thirdparty/vectorscan.git/log
thirdparty/vectorscan.git
2 years agoUse std::vector instead of boost::container::small_vector under MSan 149/head
Azat Khuzhin [Wed, 10 May 2023 13:38:27 +0000 (15:38 +0200)] 
Use std::vector instead of boost::container::small_vector under MSan

There are some issues with dtors in boost::container::small_vector
and/or vector, which is reported by MSan as an error.

The suppression __attribute__((no_sanitize_memory)) works until
clang-15, but since clang-16 it does not. It looks like before clang-16
this no_sanitize_memory works for all child functions, while since
clang-16 only for this function. I've tried to add few others, but a) it
looks icky b) I haven't managed to finish this process.

Also I've measured the performance and it hadn't been changed. Though
boost::small_vector should be faster then std::vector, but apparently my
particular case hadn't affected too much.

And one more thing, MSan reports this only with -O0, with -O3 - it is
not reproduced.

<details>

<summary>MSan report:</summary>

_Note: it was slightly trimmed_

```
==11364==WARNING: MemorySanitizer: use-of-uninitialized-value
2023.05.10 15:40:53.000233 [ 11620 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 1012.32 MiB, peak 1012.32 MiB, free memory in arenas 0.00 B, will set to 1015.82 MiB (RSS), difference: 3.50 MiB
    0 0x55558d13289f in boost::container::vector_alloc_holder<boost::container::small_vector_allocator<std::__1::pair<unsigned char, unsigned char>, std::__1::allocator<void>, void>, unsigned long, boost::move_detail::integral_constant<unsigned int, 1u>>::deallocate(std::__1::pair<unsigned char, unsigned char>* const&, unsigned long) .cmake-llvm16-msan/./contrib/boost/boost/container/vector.hpp:455:7
    1 0x55558d139e8e in boost::container::vector_alloc_holder<boost::container::small_vector_allocator<std::__1::pair<unsigned char, unsigned char>, std::__1::allocator<void>, void>, unsigned long, boost::move_detail::integral_constant<unsigned int, 1u>>::~vector_alloc_holder() .cmake-llvm16-msan/./contrib/boost/boost/container/vector.hpp:420:16
    2 0x55558d139e0b in boost::container::vector<std::__1::pair<unsigned char, unsigned char>, boost::container::small_vector_allocator<std::__1::pair<unsigned char, unsigned char>, std::__1::allocator<void>, void>, void>::~vector() .cmake-llvm16-msan/./contrib/boost/boost/container/vector.hpp:1141:4
    3 0x55558d12a4fa in boost::container::small_vector_base<std::__1::pair<unsigned char, unsigned char>, std::__1::allocator<std::__1::pair<unsigned char, unsigned char>>, void>::~small_vector_base() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:445:80
    4 0x55558d12a4fa in boost::container::small_vector<std::__1::pair<unsigned char, unsigned char>, 1ul, std::__1::allocator<std::__1::pair<unsigned char, unsigned char>>, void>::~small_vector() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:564:7
    5 0x55558d13a21b in std::__1::__tuple_leaf<0ul, boost::container::small_vector<std::__1::pair<unsigned char, unsigned char>, 1ul, std::__1::allocator<std::__1::pair<unsigned char, unsigned char>>, void>, false>::~__tuple_leaf() .cmake-llvm16-msan/./contrib/llvm-project/libcxx/include/tuple:265:7
    6 0x55558d13a13a in std::__1::__tuple_impl<>::~__tuple_impl() .cmake-llvm16-msan/./contrib/llvm-project/libcxx/include/tuple:451:37
    7 0x55558d13a05b in std::__1::tuple<>::~tuple() .cmake-llvm16-msan/./contrib/llvm-project/libcxx/include/tuple:538:28
    8 0x55558d139f7b in ue2::flat_detail::flat_base<>::~flat_base() .cmake-llvm16-msan/./contrib/vectorscan/src/util/flat_containers.h:89:7
    9 0x55558d1299da in ue2::flat_set<>::~flat_set() .cmake-llvm16-msan/./contrib/vectorscan/src/util/flat_containers.h:152:7
    10 0x55558d4e4dda in ue2::(anonymous namespace)::DAccelScheme::~DAccelScheme() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:301:8
    11 0x55558d4ff6cf in void boost::container::allocator_traits<>::priv_destroy<ue2::(anonymous namespace)::DAccelScheme>(boost::move_detail::integral_constant<bool, false>, boost::container::small_vector_allocator<ue2::(anonymous namespace)::DAccelScheme, boost::container::new_allocator<void>, void>&, ue2::(anonymous namespace)::DAccelScheme*) .cmake-llvm16-msan/./contrib/boost/boost/container/allocator_traits.hpp:403:11
    12 0x55558d4fefde in void boost::container::allocator_traits<>::destroy<ue2::(anonymous namespace)::DAccelScheme>(boost::container::small_vector_allocator<ue2::(anonymous namespace)::DAccelScheme, boost::container::new_allocator<void>, void>&, ue2::(anonymous namespace)::DAccelScheme*) .cmake-llvm16-msan/./contrib/boost/boost/container/allocator_traits.hpp:331:7
    13 0x55558d4fc364 in boost::container::dtl::disable_if_trivially_destructible<>::type boost::container::destroy_alloc_n<>(boost::container::small_vector_allocator<ue2::(anonymous namespace)::DAccelScheme, boost::container::new_allocator<void>, void>&, ue2::(anonymous namespace)::DAccelScheme*, unsigned long) .cmake-llvm16-msan/./contrib/boost/boost/container/detail/copy_move_algo.hpp:988:7
    14 0x55558d517962 in boost::container::vector<>::~vector() .cmake-llvm16-msan/./contrib/boost/boost/container/vector.hpp:1138:7
    15 0x55558d4f724d in boost::container::small_vector_base<>::~small_vector_base() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:445:80
    16 0x55558d4f724d in boost::container::small_vector<>::~small_vector() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:564:7
    17 0x55558d4f2ff3 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:444:1
    18 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    19 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    20 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    21 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    22 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    23 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    24 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    25 0x55558d4e4af5 in ue2::findBestDoubleAccelScheme() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:556:5
    26 0x55558d4e2659 in ue2::findBestAccelScheme() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:569:27
    27 0x55558d3aa8ff in ue2::look_for_offset_accel(ue2::raw_dfa const&, unsigned short, unsigned int) .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/accel_dfa_build_strat.cpp:197:22
    28 0x55558d3a9727 in ue2::accel_dfa_build_strat::find_escape_strings(unsigned short) const .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/accel_dfa_build_strat.cpp:414:13
    29 0x55558d3b2119 in ue2::accel_dfa_build_strat::getAccelInfo(ue2::Grey const&)::$_0::operator()(unsigned long) const .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/accel_dfa_build_strat.cpp:606:26
    30 0x55558d3aefd4 in ue2::accel_dfa_build_strat::getAccelInfo(ue2::Grey const&) .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/accel_dfa_build_strat.cpp:627:13
    31 0x55558d2fc61f in ue2::mcclellanCompile8() .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/mcclellancompile.cpp:935:22
    32 0x55558d2e89ec in ue2::mcclellanCompile_i() .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/mcclellancompile.cpp:1510:15
    33 0x55558d2ff502 in ue2::mcclellanCompile() .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/mcclellancompile.cpp:1527:12
    34 0x55558fb13b52 in ue2::getDfa() .cmake-llvm16-msan/./contrib/vectorscan/src/rose/rose_build_bytecode.cpp:646:15
    35 0x55558fb7e8c8 in ue2::makeLeftNfa() .cmake-llvm16-msan/./contrib/vectorscan/src/rose/rose_build_bytecode.cpp:854:22
    36 0x55558fb6bd36 in ue2::buildLeftfix() .cmake-llvm16-msan/./contrib/vectorscan/src/rose/rose_build_bytecode.cpp:1123:15
    37 0x55558fb21020 in ue2::buildLeftfixes() .cmake-llvm16-msan/./contrib/vectorscan/src/rose/rose_build_bytecode.cpp:1579:9
    38 0x55558fad972c in ue2::buildNfas() .cmake-llvm16-msan/./contrib/vectorscan/src/rose/rose_build_bytecode.cpp:2063:10
    39 0x55558fac9843 in ue2::RoseBuildImpl::buildFinalEngine(unsigned int) .cmake-llvm16-msan/./contrib/vectorscan/src/rose/rose_build_bytecode.cpp:3660:10
    40 0x55558f2b2d86 in ue2::RoseBuildImpl::buildRose(unsigned int) .cmake-llvm16-msan/./contrib/vectorscan/src/rose/rose_build_compile.cpp:1796:12

  Uninitialized value was stored to memory at
    0 0x55558d132898 in boost::container::vector_alloc_holder<boost::container::small_vector_allocator<std::__1::pair<unsigned char, unsigned char>, std::__1::allocator<void>, void>, unsigned long, boost::move_detail::integral_constant<unsigned int, 1u>>::deallocate(std::__1::pair<unsigned char, unsigned char>* const&, unsigned long) .cmake-llvm16-msan/./contrib/boost/boost/container/vector.hpp:455:56
    1 0x55558d139e8e in boost::container::vector_alloc_holder<>::~vector_alloc_holder() .cmake-llvm16-msan/./contrib/boost/boost/container/vector.hpp:420:16
    2 0x55558d139e0b in boost::container::vector<>::~vector() .cmake-llvm16-msan/./contrib/boost/boost/container/vector.hpp:1141:4
    3 0x55558d12a4fa in boost::container::small_vector_base<>::~small_vector_base() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:445:80
    4 0x55558d12a4fa in boost::container::small_vector<std::__1::pair<unsigned char, unsigned char>, 1ul, std::__1::allocator<std::__1::pair<unsigned char, unsigned char>>, void>::~small_vector() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:564:7
    5 0x55558d13a21b in std::__1::__tuple_leaf<>::~__tuple_leaf() .cmake-llvm16-msan/./contrib/llvm-project/libcxx/include/tuple:265:7
    6 0x55558d13a13a in std::__1::__tuple_impl<>::~__tuple_impl .cmake-llvm16-msan/./contrib/llvm-project/libcxx/include/tuple:451:37
    7 0x55558d13a05b in std::__1::tuple<>::~tuple() .cmake-llvm16-msan/./contrib/llvm-project/libcxx/include/tuple:538:28
    8 0x55558d139f7b in ue2::flat_detail::flat_base<>::~flat_base() .cmake-llvm16-msan/./contrib/vectorscan/src/util/flat_containers.h:89:7
    9 0x55558d1299da in ue2::flat_set<>::~flat_set() .cmake-llvm16-msan/./contrib/vectorscan/src/util/flat_containers.h:1
52:7
    10 0x55558d4e4dda in ue2::(anonymous namespace)::DAccelScheme::~DAccelScheme() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:301:8
    11 0x55558d4ff6cf in void boost::container::allocator_traits<>::priv_destroy<ue2::(anonymous namespace)::DAccelScheme>() .cmake-llvm16-msan/./contrib/boost/boost/container/allocator_traits.hpp:403:11
    12 0x55558d4fefde in void boost::container::allocator_traits<>::destroy<ue2::(anonymous namespace)::DAccelScheme>(boost::container::small_vector_allocator<>&, ue2::(anonymous namespace)::DAccelScheme*) .cmake-llvm16-msan/./contrib/boost/boost/container/allocator_traits.hpp:331:7
    13 0x55558d4fc364 in boost::container::dtl::disable_if_trivially_destructible<>::type boost::container::destroy_alloc_n<boost::container::small_vector_allocator<ue2::(anonymous namespace)::DAccelScheme, boost::container::new_allocator<void>, void>, ue2::(anonymous namespace)::DAccelScheme*, unsigned long>(boost::container::small_vector_allocator<ue2::(anonymous namespace)::DAccelScheme, boost::container::new_allocator<void>, void>&, ue2::(anonymous namespace)::DAccelScheme*, unsigned long) .cmake-llvm16-msan/./contrib/boost/boost/container/detail/copy_move_algo.hpp:988:7
    14 0x55558d517962 in boost::container::vector<ue2::(anonymous namespace)::DAccelScheme, boost::container::small_vector_allocator<ue2::(anonymous namespace)::DAccelScheme, boost::container::new_allocator<void>, void>, void>::~vector() .cmake-llvm16-msan/./contrib/boost/boost/container/vector.hpp:1138:7
    15 0x55558d4f724d in boost::container::small_vector_base<>::~small_vector_base() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:445:80
    16 0x55558d4f724d in boost::container::small_vector<>::~small_vector() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:564:7
    17 0x55558d4f2ff3 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:444:1
    18 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    19 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    20 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    21 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9

  Member fields were destroyed
    0 0x5555652e08dd in __sanitizer_dtor_callback_fields /src/llvm/worktrees/llvm-16/compiler-rt/lib/msan/msan_interceptors.cpp:961:5
    1 0x55558d4f71a6 in boost::container::small_vector<>::~small_vector() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:528:8
    2 0x55558d4f71a6 in boost::container::small_vector<>::~small_vector() .cmake-llvm16-msan/./contrib/boost/boost/container/small_vector.hpp:564:7
    3 0x55558d4f2ff3 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:444:1
    4 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    5 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    6 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    7 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    8 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    9 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    10 0x55558d4f2f41 in ue2::findDoubleBest() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:442:9
    11 0x55558d4e4af5 in ue2::findBestDoubleAccelScheme() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:556:5
    12 0x55558d4e2659 in ue2::findBestAccelScheme() .cmake-llvm16-msan/./contrib/vectorscan/src/nfagraph/ng_limex_accel.cpp:569:27
    13 0x55558d3aa8ff in ue2::look_for_offset_accel(ue2::raw_dfa const&, unsigned short, unsigned int) .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/accel_dfa_build_strat.cpp:197:22
    14 0x55558d3a9727 in ue2::accel_dfa_build_strat::find_escape_strings(unsigned short) const .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/accel_dfa_build_strat.cpp:414:13
    15 0x55558d3b2119 in ue2::accel_dfa_build_strat::getAccelInfo(ue2::Grey const&)::$_0::operator()(unsigned long) const .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/accel_dfa_build_strat.cpp:606:26
    16 0x55558d3aefd4 in ue2::accel_dfa_build_strat::getAccelInfo(ue2::Grey const&) .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/accel_dfa_build_strat.cpp:627:13
    17 0x55558d2fc61f in ue2::mcclellanCompile8(ue2::(anonymous namespace)::dfa_info&, ue2::CompileContext const&, std::__1::set<unsigned short, std::__1::less<unsigned short>, std::__1::allocator<unsigned short>>*) .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/mcclellancompile.cpp:935:22
    18 0x55558d2e89ec in ue2::mcclellanCompile_i(ue2::raw_dfa&, ue2::accel_dfa_build_strat&, ue2::CompileContext const&, bool, std::__1::set<unsigned short, std::__1::less<unsigned short>, std::__1::allocator<unsigned short>>*) .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/mcclellancompile.cpp:1510:15
    19 0x55558d2ff502 in ue2::mcclellanCompile(ue2::raw_dfa&, ue2::CompileContext const&, ue2::ReportManager const&, bool, bool, std::__1::set<unsigned short, std::__1::less<unsigned short>, std::__1::allocator<unsigned short>>*) .cmake-llvm16-msan/./contrib/vectorscan/src/nfa/mcclellancompile.cpp:1527:12
    20 0x55558fb13b52 in ue2::getDfa(ue2::raw_dfa&, bool, ue2::CompileContext const&, ue2::ReportManager const&) .cmake-llvm16-msan/./contrib/vectorscan/src/rose/rose_build_bytecode.cpp:646:15
```

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2 years agoMerge pull request #144 from rschu1ze/rs/fix-libcxx16
Konstantinos Margaritis [Wed, 29 Mar 2023 09:55:48 +0000 (12:55 +0300)] 
Merge pull request #144 from rschu1ze/rs/fix-libcxx16

Fix compilation with libcxx 16

2 years agoFix compilation with libcxx 16 144/head
Robert Schulze [Tue, 28 Mar 2023 21:34:35 +0000 (21:34 +0000)] 
Fix compilation with libcxx 16

After upgrading our (ClickHouse's) libcxx from 15 to 16, the compiler
started to complain about usage of an incomplete type "RoseInstruction"
in this (header) function:

  void RoseProgram::replace(Iter it, std::unique_ptr<RoseInstruction> ri) {
    ...

The reason is that libcxx 16 is the first version which implements C++23
constexpr std::unique_ptr (P2273R3, see (*)). RoseProgram::replace()
happens to be be const-evaluatable and the compiler tries to run
std::unique_ptr's ctor + dtor. This fails because at this point
RoseInstruction isn't defined yet.

There are two ways of fixing this:
1. Include rose_build_instruction.h (which contains RoseInstruction)
   into rose_build_program.h. Disadvantage: The new include will
   propagate transitively into all callers.
2. Move the function implementation into the source file which sees
   RoseInstruction's definition already. Disadvantage: Template
   instantiation is no longer automatic, instead there must be either a)
   explicit template instantiation (e.g. in rose_build_program.cpp) or
   b) all callers which instantiate the function must live in the same
   source file and do the instantiations by themselves. Fortunately, the
   latter is the case here, but potential future code outside
   rose_build_program.cpp will require ugly explicit instantiation.

(*) https://en.cppreference.com/w/cpp/23

2 years agoMerge pull request #142 from VectorCamp/feature/bump-version 143/head
Konstantinos Margaritis [Thu, 23 Mar 2023 08:31:06 +0000 (10:31 +0200)] 
Merge pull request #142 from VectorCamp/feature/bump-version

Bump version

2 years agoBump version 142/head
Konstantinos Margaritis [Thu, 23 Mar 2023 08:29:20 +0000 (08:29 +0000)] 
Bump version

2 years agoMerge pull request #141 from VectorCamp/bugfix/hs-flag-utf8-signed-char-on-arm
Konstantinos Margaritis [Thu, 23 Mar 2023 08:14:59 +0000 (10:14 +0200)] 
Merge pull request #141 from VectorCamp/bugfix/hs-flag-utf8-signed-char-on-arm

Set Ragel.rl char type to unsigned, #135

2 years agoclang 14 makes some test failed because val is uninitialized 141/head
Konstantinos Margaritis [Wed, 22 Mar 2023 19:39:03 +0000 (21:39 +0200)] 
clang 14 makes some test failed because val is uninitialized

2 years agofix ExpressionParser.cpp path
Konstantinos Margaritis [Wed, 22 Mar 2023 09:29:28 +0000 (11:29 +0200)] 
fix ExpressionParser.cpp path

2 years agoadd -funsigned-char to RAGEL_C_FLAGS, move util build after that
Konstantinos Margaritis [Wed, 22 Mar 2023 09:29:02 +0000 (11:29 +0200)] 
add -funsigned-char to RAGEL_C_FLAGS, move util build after that

2 years agoForce -funsigned-char to RAGEL_C_FLAGS
Konstantinos Margaritis [Wed, 22 Mar 2023 08:49:19 +0000 (08:49 +0000)] 
Force -funsigned-char to RAGEL_C_FLAGS

2 years agofix typo
Konstantinos Margaritis [Tue, 21 Mar 2023 18:11:17 +0000 (18:11 +0000)] 
fix typo

2 years agoSet Ragel.rl char type to unsigned, #135
Konstantinos Margaritis [Tue, 21 Mar 2023 18:07:06 +0000 (18:07 +0000)] 
Set Ragel.rl char type to unsigned, #135

2 years agoMerge pull request #136 from VectorCamp/feature/prefix-assume-aligned
Konstantinos Margaritis [Tue, 1 Nov 2022 15:41:39 +0000 (17:41 +0200)] 
Merge pull request #136 from VectorCamp/feature/prefix-assume-aligned

prefix assume_aligned to avoid clash with std::assume_aligned in c++20

2 years agoprefix assume_aligned to avoid clash with std::assume_aligned in c++20 136/head
Konstantinos Margaritis [Tue, 1 Nov 2022 10:29:22 +0000 (10:29 +0000)] 
prefix assume_aligned to avoid clash with std::assume_aligned in c++20

2 years agoMerge pull request #125 from abondarev84/master 131/head
Konstantinos Margaritis [Mon, 19 Sep 2022 09:44:14 +0000 (12:44 +0300)] 
Merge pull request #125 from abondarev84/master

cmake change for correct placement of autodetected tune & arch flags of GCC and SVE enablement on AARCH64

2 years agoclang SVE build fix 125/head
Alex Bondarev [Sun, 18 Sep 2022 16:42:45 +0000 (19:42 +0300)] 
clang SVE build fix

2 years agoSVE enabled on user input. updated README
Alex Bondarev [Sun, 18 Sep 2022 09:04:05 +0000 (12:04 +0300)] 
SVE enabled on user input. updated README

tune and arch flags will be applied from autodetect only if they have been created by the process, otherwise the old logical flow remains wrt the flags

2 years agoMerge pull request #129 from VectorCamp/bugfix/fix-clang-on-power
Konstantinos Margaritis [Fri, 16 Sep 2022 16:04:06 +0000 (19:04 +0300)] 
Merge pull request #129 from VectorCamp/bugfix/fix-clang-on-power

Fix compile errors on clang and Power

2 years agomove variable 129/head
Konstantinos Margaritis [Fri, 16 Sep 2022 11:05:31 +0000 (14:05 +0300)] 
move variable

2 years agoclang 13+ gives wrong -Wunused-but-set-variable error on nfa/mcclellancompile.cpp...
Konstantinos Margaritis [Fri, 16 Sep 2022 11:04:59 +0000 (14:04 +0300)] 
clang 13+ gives wrong -Wunused-but-set-variable error on nfa/mcclellancompile.cpp about total_daddy variable, disabling

2 years agoremove leftover debug print
Konstantinos Margaritis [Fri, 16 Sep 2022 11:03:17 +0000 (14:03 +0300)] 
remove leftover debug print

2 years agoclang 14 does not allow bitwise OR for bools
Konstantinos Margaritis [Fri, 16 Sep 2022 11:02:53 +0000 (14:02 +0300)] 
clang 14 does not allow bitwise OR for bools

2 years agoclang 14 complains about this, needs investigation
Konstantinos Margaritis [Fri, 16 Sep 2022 11:02:26 +0000 (14:02 +0300)] 
clang 14 complains about this, needs investigation

2 years ago[VSX] movemask needs to be explicitly aligned on clang for vec_ste
Konstantinos Margaritis [Fri, 16 Sep 2022 09:50:33 +0000 (12:50 +0300)] 
[VSX] movemask needs to be explicitly aligned on clang for vec_ste

2 years ago[VSX] clang complains about the order of __vector 128/head
Konstantinos Margaritis [Fri, 16 Sep 2022 09:41:08 +0000 (12:41 +0300)] 
[VSX] clang complains about the order of __vector

2 years agoadditional mcpu flag cleanup
Alex Bondarev [Thu, 15 Sep 2022 21:03:08 +0000 (00:03 +0300)] 
additional mcpu flag cleanup

2 years agoremoved cpu reference flags and fixed tune flag
Alex Bondarev [Thu, 15 Sep 2022 15:38:01 +0000 (18:38 +0300)] 
removed cpu reference flags and fixed tune flag

2 years agoupdated README to reflect CMake changes
Alex Bondarev [Tue, 13 Sep 2022 15:29:06 +0000 (18:29 +0300)] 
updated README to reflect CMake changes

2 years agofix to correctly place the autodetected flags and to activate SVE options
Alex Bondarev [Tue, 13 Sep 2022 15:21:10 +0000 (18:21 +0300)] 
fix to correctly place the autodetected flags and to activate SVE options

2 years agobump version vectorscan/5.4.8
Konstantinos Margaritis [Tue, 13 Sep 2022 12:57:08 +0000 (12:57 +0000)] 
bump version

2 years agoMerge pull request #124 from VectorCamp/develop
Konstantinos Margaritis [Tue, 13 Sep 2022 12:52:20 +0000 (15:52 +0300)] 
Merge pull request #124 from VectorCamp/develop

Merge develop to master

2 years agoMerge pull request #123 from VectorCamp/feature/neon-shift-optimizations 124/head
Konstantinos Margaritis [Tue, 13 Sep 2022 06:13:05 +0000 (09:13 +0300)] 
Merge pull request #123 from VectorCamp/feature/neon-shift-optimizations

[NEON] simplify/optimize shift/align primitives

2 years ago[NEON] simplify/optimize shift/align primitives 123/head
Konstantinos Margaritis [Mon, 12 Sep 2022 13:09:51 +0000 (13:09 +0000)] 
[NEON] simplify/optimize shift/align primitives

2 years agoMerge pull request #121 from liquidaty/mingw64-develop
Konstantinos Margaritis [Fri, 9 Sep 2022 10:42:49 +0000 (13:42 +0300)] 
Merge pull request #121 from liquidaty/mingw64-develop

fix to enable successful build with mingw64

2 years agofix to enable successful build with mingw64 121/head
liquidaty [Thu, 8 Sep 2022 16:59:37 +0000 (09:59 -0700)] 
fix to enable successful build with mingw64

2 years agoMerge pull request #119 from VectorCamp/feature/vsx-optimizations
Konstantinos Margaritis [Thu, 8 Sep 2022 10:41:13 +0000 (13:41 +0300)] 
Merge pull request #119 from VectorCamp/feature/vsx-optimizations

VSX optimizations

2 years agouse correct intrinsic for lshiftbyte_m128 119/head
Konstantinos Margaritis [Wed, 7 Sep 2022 13:00:10 +0000 (16:00 +0300)] 
use correct intrinsic for lshiftbyte_m128

2 years agoprovide non-immediate versions of lshiftbyte/rshiftbyte on x86
Konstantinos Margaritis [Wed, 7 Sep 2022 12:07:20 +0000 (15:07 +0300)] 
provide non-immediate versions of lshiftbyte/rshiftbyte on x86

2 years agoreadd simd_onebit_masks for x86, needs more work
Konstantinos Margaritis [Wed, 7 Sep 2022 10:42:25 +0000 (13:42 +0300)] 
readd simd_onebit_masks for x86, needs more work

2 years ago[NEON] optimize mask1bit128, get rid of simd_onebit_masks
Konstantinos Margaritis [Wed, 7 Sep 2022 10:20:01 +0000 (10:20 +0000)] 
[NEON] optimize mask1bit128, get rid of simd_onebit_masks

2 years agoremove simd_onebit_masks from arm/x86 headers, as they moved to common
Konstantinos Margaritis [Wed, 7 Sep 2022 09:41:32 +0000 (12:41 +0300)] 
remove simd_onebit_masks from arm/x86 headers, as they moved to common

2 years ago[VSX] optimize alignr method
Konstantinos Margaritis [Wed, 7 Sep 2022 09:35:28 +0000 (12:35 +0300)] 
[VSX] optimize alignr method

2 years ago[VSX] optimize shift operators
Konstantinos Margaritis [Wed, 7 Sep 2022 09:16:14 +0000 (12:16 +0300)] 
[VSX] optimize shift operators

2 years ago[VSX] optimize shifting methods, replace template Unroller
Konstantinos Margaritis [Wed, 7 Sep 2022 08:52:08 +0000 (11:52 +0300)] 
[VSX] optimize shifting methods, replace template Unroller

2 years agooptimize comparemask implementation, clean up code, use union types instead of casts
Konstantinos Margaritis [Tue, 6 Sep 2022 23:02:11 +0000 (02:02 +0300)] 
optimize comparemask implementation, clean up code, use union types instead of casts

2 years ago[VSX] add algorithm for alignr w/o use of immediates
Konstantinos Margaritis [Tue, 6 Sep 2022 21:01:54 +0000 (00:01 +0300)] 
[VSX] add algorithm for alignr w/o use of immediates

2 years ago[VSX] correct lshiftbyte_m128/rshiftbyte_m128, variable_byte_shift
Konstantinos Margaritis [Tue, 6 Sep 2022 20:39:44 +0000 (23:39 +0300)] 
[VSX] correct lshiftbyte_m128/rshiftbyte_m128, variable_byte_shift

2 years ago[VSX] huge optimization of movemask128
Konstantinos Margaritis [Tue, 6 Sep 2022 17:08:44 +0000 (20:08 +0300)] 
[VSX] huge optimization of movemask128

2 years ago[VSX] optimize and correct lshift_m128/rshift_m128
Konstantinos Margaritis [Tue, 6 Sep 2022 15:46:39 +0000 (18:46 +0300)] 
[VSX] optimize and correct lshift_m128/rshift_m128

2 years ago[VSX] optimized mask1bit128(), moved simd_onebit_masks to common
Konstantinos Margaritis [Tue, 6 Sep 2022 15:10:55 +0000 (18:10 +0300)] 
[VSX] optimized mask1bit128(), moved simd_onebit_masks to common

2 years agoadd popcount32x4, popcount64x4 helper functions
Konstantinos Margaritis [Fri, 2 Sep 2022 12:12:56 +0000 (15:12 +0300)] 
add popcount32x4, popcount64x4 helper functions

2 years agoMerge pull request #118 from VectorCamp/bugfix/hyperscan-backport-202208
Konstantinos Margaritis [Sat, 3 Sep 2022 06:32:43 +0000 (09:32 +0300)] 
Merge pull request #118 from VectorCamp/bugfix/hyperscan-backport-202208

Bugfix/hyperscan backport 202208

2 years agoAdd missing <memory> header 118/head
Konstantinos Margaritis [Tue, 30 Aug 2022 17:40:23 +0000 (20:40 +0300)] 
Add missing <memory> header

2 years agofix build with glibc-2.34
Liu Zixian [Mon, 27 Jun 2022 08:07:16 +0000 (16:07 +0800)] 
fix build with glibc-2.34

SIGTSKSZ is no long a constant after glibc 2.34
https://sourceware.org/pipermail/libc-alpha/2021-August/129718.html

2 years agoliteral API: add empty string check.
Hong, Yang A [Thu, 4 Mar 2021 16:13:46 +0000 (16:13 +0000)] 
literal API: add empty string check.

fixes github issue #302, #304

2 years agochimera: fix SKIP flag issue
Hong, Yang A [Thu, 28 Jul 2022 04:59:34 +0000 (04:59 +0000)] 
chimera: fix SKIP flag issue

fix github issue #360

2 years agoCorpus editor: fix random char value of UTF-8.
Chang, Harry [Thu, 12 May 2022 08:20:29 +0000 (08:20 +0000)] 
Corpus editor: fix random char value of UTF-8.

2 years agoCorpus generator: fix random char value of UTF-8.
Chang, Harry [Thu, 12 May 2022 02:15:07 +0000 (02:15 +0000)] 
Corpus generator: fix random char value of UTF-8.

fixes github issue #184

2 years agobugfix: fix overflow risk of strlen function
Hong, Yang A [Thu, 28 Apr 2022 10:11:32 +0000 (10:11 +0000)] 
bugfix: fix overflow risk of strlen function

2 years agoFix segfaults on allocation failure (#4)
hongyang7 [Thu, 16 Dec 2021 11:02:17 +0000 (19:02 +0800)] 
Fix segfaults on allocation failure (#4)

Throw std::bad_alloc instead of returning nullptr from
ue2::AlignedAllocator. Allocators for STL containers are expected never
to return with an invalid pointer, and instead must throw on failure.
Violating this expectation can lead to invalid pointer dereferences.

Co-authored-by: johanngan <johanngan.us@gmail.com>
fixes github issue #317 (PR #320)

2 years agoLogical Combination: bypass combination flag in hs_expression_info.
Chang, Harry [Wed, 10 Mar 2021 07:20:01 +0000 (07:20 +0000)] 
Logical Combination: bypass combination flag in hs_expression_info.

Fixes github issue #291

2 years agoupdate year for bugfix #302-#305
Hong, Yang A [Thu, 11 Mar 2021 15:20:55 +0000 (15:20 +0000)] 
update year for bugfix #302-#305

2 years agomcclellan: improve wide-state checking in Sherman optimization
Hong, Yang A [Thu, 4 Mar 2021 17:00:34 +0000 (17:00 +0000)] 
mcclellan: improve wide-state checking in Sherman optimization

fixes github issue #305

2 years agoliteral API: add instruction support
Hong, Yang A [Thu, 4 Mar 2021 16:50:14 +0000 (16:50 +0000)] 
literal API: add instruction support

fixes github issue #303

3 years agoMerge pull request #116 from pareenaverma/develop
Konstantinos Margaritis [Wed, 20 Jul 2022 20:08:11 +0000 (23:08 +0300)] 
Merge pull request #116 from pareenaverma/develop

Fixed the PCRE download location

3 years agoMerge pull request #113 from danlark1/develop
Konstantinos Margaritis [Wed, 20 Jul 2022 13:41:33 +0000 (16:41 +0300)] 
Merge pull request #113 from danlark1/develop

Optimize vectorscan for aarch64 by using shrn instruction

3 years agoFixed the PCRE download location 116/head
Ubuntu [Wed, 20 Jul 2022 13:26:52 +0000 (13:26 +0000)] 
Fixed the PCRE download location

3 years agoFix avx512 movemask call 113/head
Danila Kutenin [Wed, 20 Jul 2022 08:03:50 +0000 (09:03 +0100)] 
Fix avx512 movemask call

3 years agoFix ppc64el debug
Danila Kutenin [Sun, 26 Jun 2022 23:05:17 +0000 (23:05 +0000)] 
Fix ppc64el debug

3 years agoMinor fix
Danila Kutenin [Sun, 26 Jun 2022 23:02:02 +0000 (23:02 +0000)] 
Minor fix

3 years agoFix formatting of a couple files
Danila Kutenin [Sun, 26 Jun 2022 22:59:58 +0000 (22:59 +0000)] 
Fix formatting of a couple files

3 years agoOptimize vectorscan for aarch64 by using shrn instruction
Danila Kutenin [Sun, 26 Jun 2022 22:50:05 +0000 (22:50 +0000)] 
Optimize vectorscan for aarch64 by using shrn instruction

This optimization is based on the thread
https://twitter.com/Danlark1/status/1539344279268691970 and uses
shift right and narrow by 4 instruction https://developer.arm.com/documentation/ddi0596/2020-12/SIMD-FP-Instructions/SHRN--SHRN2--Shift-Right-Narrow--immediate--

To achieve that, I needed to redesign a little movemask into comparemask
and have an additional step towards mask iteration. Our benchmarks
showed 10-15% improvement on average for long matches.

3 years agoMerge pull request #108 from jth/cmake-python
Konstantinos Margaritis [Fri, 20 May 2022 06:05:11 +0000 (09:05 +0300)] 
Merge pull request #108 from jth/cmake-python

CMake: Use non-deprecated method for finding python

3 years agoBump scripts to python3 108/head
Jan Henning [Thu, 19 May 2022 14:25:08 +0000 (16:25 +0200)] 
Bump scripts to python3

3 years agoUse non-deprecated method of finding python
Jan Henning [Thu, 19 May 2022 08:20:17 +0000 (10:20 +0200)] 
Use non-deprecated method of finding python

3 years agoUpdate CMakeLists.txt vectorscan/5.4.7
Konstantinos Margaritis [Thu, 5 May 2022 09:14:53 +0000 (12:14 +0300)] 
Update CMakeLists.txt

3 years agoMerge pull request #103 from VectorCamp/develop
Konstantinos Margaritis [Thu, 5 May 2022 07:34:56 +0000 (10:34 +0300)] 
Merge pull request #103 from VectorCamp/develop

Develop

3 years agoUpdate Jenkinsfile
Konstantinos Margaritis [Wed, 4 May 2022 18:57:38 +0000 (21:57 +0300)] 
Update Jenkinsfile

3 years agoUpdate Jenkinsfile
Konstantinos Margaritis [Wed, 4 May 2022 13:41:10 +0000 (16:41 +0300)] 
Update Jenkinsfile

3 years agoUpdate Jenkinsfile
Konstantinos Margaritis [Wed, 4 May 2022 13:30:22 +0000 (16:30 +0300)] 
Update Jenkinsfile

3 years agoMerge pull request #105 from VectorCamp/bugfix/jenkins
Konstantinos Margaritis [Wed, 4 May 2022 13:27:22 +0000 (16:27 +0300)] 
Merge pull request #105 from VectorCamp/bugfix/jenkins

fix large pipeline error

3 years agofix large pipeline error 105/head
Konstantinos Margaritis [Wed, 4 May 2022 13:26:02 +0000 (16:26 +0300)] 
fix large pipeline error

3 years agoDelete JenkinsFile
Konstantinos Margaritis [Wed, 4 May 2022 13:14:19 +0000 (16:14 +0300)] 
Delete JenkinsFile

3 years agoMerge pull request #104 from VectorCamp/bugfix/jenkinsfile
Konstantinos Margaritis [Wed, 4 May 2022 13:04:00 +0000 (16:04 +0300)] 
Merge pull request #104 from VectorCamp/bugfix/jenkinsfile

add Jenkinsfile back to master branch

3 years agoadd Jenkinsfile back to master branch 104/head
Konstantinos Margaritis [Tue, 19 Apr 2022 08:36:25 +0000 (11:36 +0300)] 
add Jenkinsfile back to master branch

3 years agoadd Jenkinsfile back to master branch
Konstantinos Margaritis [Tue, 19 Apr 2022 08:36:25 +0000 (11:36 +0300)] 
add Jenkinsfile back to master branch

3 years agoMerge pull request #102 from danlark1/patch-2 103/head
Konstantinos Margaritis [Mon, 18 Apr 2022 17:56:26 +0000 (20:56 +0300)] 
Merge pull request #102 from danlark1/patch-2

Optimized and correct version of movemask128 for ARM

3 years agoOptimized and correct version of movemask128 for ARM 102/head
Daniel Kutenin [Mon, 18 Apr 2022 12:37:53 +0000 (13:37 +0100)] 
Optimized and correct version of movemask128 for ARM

Closes #99

https://gcc.godbolt.org/z/cTjKqzcvn

Previous version was not correct because movemask thought of having bytes 0xFF. We can fully match the semantics + do it faster with USRA instructions.

Re-submission to a develop branch

3 years agoMerge pull request #94 from a16bitsysop/fat_runtime
Konstantinos Margaritis [Mon, 18 Apr 2022 08:08:29 +0000 (11:08 +0300)] 
Merge pull request #94 from a16bitsysop/fat_runtime

change FAT_RUNTIME to a normal option so it can be set to off

3 years agoMerge pull request #93 from danlark1/master
Konstantinos Margaritis [Mon, 18 Apr 2022 08:07:18 +0000 (11:07 +0300)] 
Merge pull request #93 from danlark1/master

Fix all ASAN issues in vectorscan

3 years agomove to original position 94/head
Duncan Bellamy [Tue, 22 Feb 2022 19:21:18 +0000 (19:21 +0000)] 
move to original position

3 years agochange FAT_RUNTIME to a normal option so it can be set to off
Duncan Bellamy [Sun, 20 Feb 2022 13:14:11 +0000 (13:14 +0000)] 
change FAT_RUNTIME to a normal option so it can be set to off

fixes #89

3 years agoFix a couple of tests 93/head
Danila Kutenin [Fri, 18 Feb 2022 19:31:03 +0000 (19:31 +0000)] 
Fix a couple of tests

3 years agoAdd sanitize options
Danila Kutenin [Fri, 18 Feb 2022 18:35:26 +0000 (18:35 +0000)] 
Add sanitize options

3 years agoFix all ASAN issues in vectorscan 92/head
Danila Kutenin [Fri, 18 Feb 2022 17:14:51 +0000 (17:14 +0000)] 
Fix all ASAN issues in vectorscan

3 years agoMerge pull request #90 from BigRedEye/vectorscan-master
Konstantinos Margaritis [Tue, 8 Feb 2022 07:20:34 +0000 (09:20 +0200)] 
Merge pull request #90 from BigRedEye/vectorscan-master

Fix word boundary assertions under C++20

3 years agofix: Mark operator bool explicit 90/head
BigRedEye [Sat, 8 Jan 2022 01:12:16 +0000 (04:12 +0300)] 
fix: Mark operator bool explicit

3 years agoMerge pull request #86 from VectorCamp/develop vectorscan/5.4.6
Konstantinos Margaritis [Fri, 21 Jan 2022 10:25:40 +0000 (12:25 +0200)] 
Merge pull request #86 from VectorCamp/develop

New release 5.4.6

3 years agoMerge pull request #87 from VectorCamp/feature/move-debian-package-to-separate-branch 86/head
Konstantinos Margaritis [Fri, 21 Jan 2022 10:24:03 +0000 (12:24 +0200)] 
Merge pull request #87 from VectorCamp/feature/move-debian-package-to-separate-branch

keep debian folder in a separate branch

3 years agokeep debian folder in a separate branch 87/head
Konstantinos Margaritis [Fri, 21 Jan 2022 10:07:25 +0000 (12:07 +0200)] 
keep debian folder in a separate branch