When adding macOS CI to Axiom, set operation tests kept failing intermittently — but only in macOS debug CI. Linux CI (debug and release) passed consistently. Local runs always passed. The root cause turned out to be a bug in Velox — a dependency managed as a Git submodule. This post describes the process of debugging a CI-only failure when the bug lives in a different repository.

The problem​

Axiom is a set of open-source composable libraries for building SQL and dataframe processing front ends on top of Velox. It integrates Velox as a Git submodule.

After adding macOS CI builds (#1168), we couldn't get the test suite to pass reliably. Various set operation tests (SqlTest.set_*) failed intermittently in macOS debug CI — different tests on different runs, all involving EXCEPT ALL or INTERSECT ALL. Linux debug and release CI passed consistently. Running the same tests locally — same build type, same data, same configuration — always passed.

Why CI-only failures are hard to debug​

The obvious approach — reproduce locally, attach a debugger, step through the code — doesn't work. You are limited to CI runs, and each run takes about 10 minutes end-to-end: push the change, wait for CI to pick it up, build (~8 minutes even with ccache), run tests, check results. That's 3-4 iterations per hour at best. Every CI run has to be designed to extract maximum signal.

Step 1: Make CI runs count​

The first change was to restructure CI to focus on the problem:

• Disable unrelated builds. Linux debug, Linux release, and macOS release were all passing. Disabling them reduced the CI cycle time.
• Run only SqlTest.set_* tests 5 times. The tests were flaky — sometimes passing, sometimes failing. Running just the set operation tests (not the full suite) 5 times increased the chances of triggering the failure together with the debug logging while keeping the run fast. Without this, a CI run could pass by luck, wasting 10 minutes with no useful signal.
• Run the full test suite after. The 5x set-test loop ran first. If it passed, the full suite ran to check for regressions.

Step 2: Iterate on Velox changes using Axiom CI​

The failing tests all involved EXCEPT ALL and INTERSECT ALL queries. Axiom's SQL parser and optimizer translate these into Velox execution plans using counting joins — a Velox-specific join type that, unlike regular semi/anti joins that check whether a key exists, tracks how many times each key appears on the build side. Since the implementation lives in Velox, the bug had to be there too.

But the tests were in Axiom. Axiom uses Velox as a Git submodule pointing to a specific commit on facebookincubator/velox. To add debug logging to Velox and test it in Axiom CI, we needed Axiom to build from a modified Velox.

The key insight is that you don't have to land changes to Velox first. You can point Axiom's submodule at a fork branch and let CI validate end-to-end.

The workflow:

1. Fork Velox and push changes to a branch. Create a branch on your fork (e.g., mbasmanova/velox) with the debug logging or fix.
2. Point Axiom's submodule at the fork. Two changes are needed:
   • Update .gitmodules to use the fork URL:

     [submodule "velox"]
         path = velox
         url = https://github.com/mbasmanova/velox.git
     
   • Update the submodule to the desired commit:

     git -C velox checkout <commit>
     git add velox .gitmodules
     
3. Push and let CI build. Axiom CI checks out submodules recursively, so it picks up the modified Velox automatically.

One subtlety: the Velox PR and the Axiom submodule both pushed to branches on the same fork (mbasmanova/velox). Initially we used the same branch name for both, which caused force-pushes from one repo to overwrite the other. The fix was to use separate branches — fix-counting-join-merge for the Velox PR and fix-counting-join-merge-axiom for the Axiom submodule.

Step 3: Form a hypothesis, then add targeted logging​

With limited CI iterations, random logging is wasteful. Each round of logging must be designed to prove or disprove one or more hypotheses.

The hypothesis was:

When multiple build drivers process overlapping keys, each driver builds its own hash table with per-key counts. When these tables are merged, duplicate keys are dropped without summing their counts.

To test this, we added logging at three points in the counting join lifecycle, logging full input/output data per driver including the driver ID:

1. Build input — log each batch of rows received by each build driver.
2. Probe input — log each batch of probe rows.
3. Probe output — log the rows emitted by each probe driver.

By comparing build inputs across drivers (to see which keys each driver processed) with probe outputs (to see the final results), we could determine whether the merged hash table had correct per-key counts.

Before pushing to CI, we ran the tests locally. The bug didn't reproduce locally, but the logging code paths still executed. This validated that the log output was readable, the format was correct, and the information needed to confirm or reject the hypothesis would be present.

One CI run with this logging confirmed the hypothesis: multiple build drivers processed overlapping keys, but the probe output showed counts as if only one driver's keys were present — the merge had dropped duplicate key counts.

Step 4: The fix​

The actual bug was straightforward once identified. Velox's HashTable has three code paths for inserting rows during hash table merge:

• arrayPushRow for array-based hash mode (small key domains)
• buildFullProbe with normalized-key comparison (medium key domains)
• buildFullProbe with full key comparison (complex types)

All three paths handled duplicate keys correctly for regular joins (linking rows via a "next" pointer), but for counting joins (which use a count instead of a next pointer), duplicates were silently dropped.

The fix adds an addCount method to sum counts when a duplicate key is found during merge. Since Axiom's build does not compile Velox tests, we built Velox standalone to develop and run the test.

Designing a test that exercises all three code paths was non-trivial. The hash table mode is chosen automatically based on key types, number of distinct values, and value ranges. Our first test attempts only hit array mode because the key domain was small. We had to study the mode selection logic in decideHashMode() to find key configurations that force each mode. This analysis was time-consuming enough that we documented the hash modes (#16953) to spare other developers the same exercise. The key configurations we found:

• Small integers {1, 2} → array mode
• Two integer columns with 1500 distinct values each (combined cardinality exceeds the 2M array threshold) → normalized-key mode
• ARRAY(INTEGER) keys (complex types don't support value IDs) → hash mode

We verified each variant independently by reverting the fix for one code path at a time and confirming the test failed.

Step 5: Landing the fix​

With the fix validated in Axiom CI, we created two PRs:

• Velox PR (#16949): The fix, test, documentation updates, and a new hashtable.hashMode runtime stat.
• Axiom PR (#1175): Update the Velox submodule and stop skipping set operation tests in macOS debug CI (they were excluded via GTEST_FILTER="-SqlTest.set_*" as a workaround).

After the Velox PR merges, the Axiom submodule will be updated to point back to facebookincubator/velox.

Why only macOS debug?​

Honestly, we don't fully know.

The bug requires splits with overlapping keys to land on different build drivers. The test uses 4 drivers and 3 data splits (one per input vector). Velox assigns splits to drivers on demand — whichever driver is ready first grabs the next split. If all splits happen to be processed by the same driver, there is no merge and no bug.

Note that this is not a race condition. Given the same split-to-driver assignment, the result is deterministic and reproducibly wrong. The non-determinism is purely in which driver grabs which split, which depends on thread scheduling.

What we can't explain is why macOS debug CI triggers this while the same macOS debug build locally does not. Same OS, same build type, same compiler, same test data, same number of drivers. Yet the thread scheduling differs enough to change split distribution. Differences in CPU, load, or runtime environment on the CI runner may play a role.

If you have insights into what could cause such a difference in thread scheduling between local and CI macOS environments, please comment on axiom#1170.

Takeaways​

• Don't dismiss "flaky" tests. A test that sometimes passes and sometimes fails is not necessarily a bad test. In this case, the test was correct — the production code was buggy. The flakiness was the only signal that something was wrong. We initially excluded the set tests via GTEST_FILTER to land the macOS CI work, but came back to investigate right away. Leaving them disabled would have hidden a real bug in Velox affecting all users of EXCEPT ALL and INTERSECT ALL.
• Design each CI run for maximum signal. When you get 3-4 iterations per hour, you can't afford exploratory logging. Form a hypothesis first, design logging to confirm or reject it, validate the logging locally, then push.
• Run flaky tests multiple times. Not to distinguish flaky from broken, but to ensure you reliably trigger the failure together with your debug logging in a single CI run.
• Use downstream CI to iterate on dependency fixes. When a bug lives in a dependency, you can debug and validate fixes without landing anything. Point the submodule at a fork branch and let downstream CI run end-to-end. See Step 2 above for details.

Velox evaluates SQL expressions as trees of functions. A query like if(array_gte(a, b), multiply(x, y), 0) compiles into a tree where each node processes an entire vector of rows at a time. When a query runs slowly, the first question usually is: which function is consuming the most CPU? Is it the expensive array comparison, or the cheap arithmetic called millions of times? This problem is even more prominent in use cases like training data loading, when very long and deeply nested expression trees are common, and jobs may run for many hours, or days; in such cases, the CPU usage of even seemingly short-lived functions may add up to substantial overhead. Without a detailed per-function CPU usage breakdown, you may be left guessing — or worse, optimizing the wrong thing.

Velox supports CPU usage tracking through the expression.track_cpu_usage query config. When enabled, every function invocation is wrapped in a CpuWallTimer — an RAII guard that makes the clock_gettime() system call at entry and exit, recording the CPU and wall time spent in each function. This gives precise, per-function cost attribution across the entire expression tree.

The Overhead Problem​

That precision comes at a price. Each CpuWallTimer invocation performs a heap allocation, two clock_gettime() syscalls, and a heap deallocation — roughly ~5μs of fixed overhead, depending on the platform. For a function like array_gte(), which spends milliseconds per vector iterating over array elements, 5μs is a negligible noise. But for a function like multiply that incurs a single instruction, the timer takes longer than the work itself:

Benchmark for multiply(a, b) on double columns. Times are per 10K iterations.

This overhead forced production systems to disable per-function CPU tracking entirely, losing visibility into more granular CPU consumption. Selective per-function tracking (expression.track_cpu_usage_for_functions) was added to Velox in the past to let operators opt in specific functions, but even for tracked functions the overhead remained — you just chose which functions to pay the cost for.

Why Uniform Sampling Falls Short​

The obvious next step is to sample: measure every Nth vector instead of every one, and extrapolate. A uniform rate applied to all functions would be simple — one counter, one config key. But a single rate cannot serve both cheap and expensive functions.

Consider array_gte, where the timer overhead is less than 0.1% of execution time. Sampling in this case throws away measurement precision for no benefit — the function can afford full tracking. Now consider multiply on small vectors, where the timer is 2500% of the function cost. Even a 1/1000 rate may not be aggressive enough. A rate that works for one function is either inaccurate or insufficient for another.

Adaptive Per-Function Sampling​

When a query starts running, adaptive sampling does not yet know which functions are cheap and which are expensive. So it watches.

The first vector through each function is a warmup — caches and branch predictors settle, and the system measures the cost of the CpuWallTimer itself by creating and destroying a dummy one. Over the next 5 vectors, each function runs with a lightweight stopwatch instead, quietly accumulating how long it actually takes without any timer overhead.

By the sixth vector, the system has what it needs: the ratio of timer overhead to function cost. If the ratio is within a configurable threshold (default 1%), the function switches to full tracking: every future vector gets a CpuWallTimer, with zero information loss. If the timer dominates, the function switches to sampling at a rate that bounds overhead to the threshold:

Expensive functions get exact tracking. Cheap functions sample aggressively. No per-function configuration needed — just a single "max acceptable overhead" knob.

When stats are read, sampled functions have their CPU and wall time scaled up by the ratio of total vectors to timed vectors. To handle short queries where the sampling counter might never fire, the first post-calibration vector is always timed, guaranteeing at least one measurement to extrapolate from.

Results​

So does it work? We benchmarked adaptive sampling against both no tracking and full tracking, across cheap and expensive functions, at different vector sizes, and with different adaptive rates (1% and 0.5%). Results are presented below:

(Percentages are relative to "No Tracking" — higher is better)

For multiply on 100-row vectors, overhead drops from 64% to under 3%. For array_gte, overhead was already negligible and remains so — the function is fully tracked with near zero sampling.

The natural concern with sampling is accuracy. We ran a correctness benchmark comparing the extrapolated CPU time from adaptive sampling against the true value from full tracking over 1,000 vectors:

Estimates land within 1–9% of the true value. For expensive functions that end up in always track mode, accuracy is exact — every vector is timed.

Configuration​

Two config properties control adaptive sampling:

Adaptive sampling interacts cleanly with existing tracking modes. expression.track_cpu_usage (global tracking for all functions) takes highest priority and always wins. expression.track_cpu_usage_for_functions (selective per-function tracking) overrides adaptive for named functions. Adaptive sampling applies to everything else.

Takeaway​

Adaptive per-function CPU time tracking eliminates the tradeoff between observability and performance. Every function gets the best tracking mode for its cost profile — cheap functions run at near-native speed with minimal overhead, expensive functions get exact tracking at zero additional cost. The system is self-tuning and requires no per-function configuration.

We optimized two Unicode string helpers — cappedLengthUnicode and cappedByteLengthUnicode — by replacing byte-by-byte utf8proc_char_length calls with a SIMD-based scanning loop. The new implementation processes register-width blocks at a time: pure-ASCII blocks skip in one step, while mixed blocks use bitmask arithmetic to count character starts. Both helpers now share a single parameterized template, eliminating code duplication.

On a comprehensive benchmark matrix covering string lengths from 4 to 1024 bytes and ASCII ratios from 0% to 100%, we measured 2–15× speedups across most configurations, with no regressions on Unicode-heavy inputs. The optimization benefits all callers of these helpers, including the Iceberg truncate transform and various string functions.

Why SIMD, why here​

Velox is a high-performance execution engine, and SIMD is one of the primary ways we deliver that performance across compute-heavy paths. This is not a one-off micro-optimization; it is a principle we apply consistently when the data and algorithms make SIMD a natural fit. String processing is one of those places: it is ubiquitous in query execution and often dominated by byte-wise work that maps well to vectorized scanning.

In this post, we describe how we applied this principle to two Unicode helpers that sit on many string processing paths, and how we balanced speed, correctness, and maintainability.

The old implementation calls utf8proc_char_length on every UTF character. That is robust, but it is inefficient for long ASCII segments where every character has only 1 byte.

What these helpers do​

• cappedLengthUnicode returns the number of UTF-8 characters in a string, stopping early once maxChars is reached.
• cappedByteLengthUnicode returns the byte position of the Nth character, also capped by maxChars. This is the key primitive for truncating strings by character count without re-parsing the entire string.

Consider a UTF-8 string that mixes multi-byte characters and ASCII:

input = "你好abc世界"  // 15 bytes, 7 characters

With maxChars = 3:

• cappedLengthUnicode returns 3.
• cappedByteLengthUnicode returns 7 (the byte position after the third character, "好").

Both need to handle mixed ASCII and multi-byte UTF-8 inputs, invalid bytes, and early-exit behavior. These helpers are called on the Unicode code path — the path used when the input is not known to be pure ASCII at the call site. They are hot in several places, most notably the Iceberg truncate transform, which must find the byte position of the Nth character to truncate partition keys. When we profiled the Iceberg write benchmark, cappedByteLengthUnicode stood out as a clear hot spot.

Byte-by-byte UTF-8 scanning becomes a bottleneck on large, ASCII-heavy strings, but we cannot assume pure ASCII because inputs may mix ASCII and multi-byte characters.

The SIMD design​

We introduced a shared helper, detail::cappedUnicodeImpl, and parameterized it by whether it should return characters or bytes. The key idea is to process the string in SIMD-width blocks using xsimd:

1. ASCII fast path. Load a register-width block and check whether any byte has its high bit set. If not, the whole block is ASCII. We can skip the entire block and advance the counters in one step.
2. Mixed UTF-8 blocks. For a block containing non-ASCII bytes, we count character starts using (byte & 0xC0) != 0x80, convert the predicate into a bitmask, and use popcount to get the number of UTF-8 characters in the block.
3. Finding the Nth character. For cappedByteLengthUnicode, if the maxChars target falls inside a mixed block, we use ctz on the bitmask to locate the Nth character start and add a small inline helper (getUtf8CharLength) to compute the byte length of that UTF-8 character.
4. Remainder handling. For the tail, we keep two loops:
   • A non-ASCII remainder loop that avoids ASCII fast-path mispredicts.
   • A branching ASCII loop that is cheaper than the branchless version for short strings.

The journey: what we tried and learned​

The final implementation emerged through several rounds of iteration and experimentation. Here are the key insights from that process.

Optimize at the core, not the call site​

The initial version added the SIMD fast path directly in the Iceberg truncate function. But the real bottleneck was in cappedByteLengthUnicode itself. Moving the optimization into the shared helper meant every call site benefits — not just Iceberg.

Avoid switching between code paths​

The first SIMD version only accelerated the leading ASCII prefix: it scanned SIMD blocks until it hit the first non-ASCII byte, then fell back to scalar for the rest of the string. This is useless if the first character is non-ASCII but the rest are ASCII. The obvious fix — re-entering the SIMD path after each non-ASCII block — caused a ~20% regression on mixed and Unicode-heavy inputs due to the overhead of switching between SIMD and scalar modes. The solution was a single SIMD loop that handles both ASCII and mixed blocks, avoiding frequent code-path switches and extra branching.

Unify the two helpers​

cappedLengthUnicode and cappedByteLengthUnicode have nearly identical logic — the only difference is what they return. Factoring them into a shared detail::cappedUnicodeImpl<returnByteLength> template eliminated the code duplication and ensures both helpers stay in sync.

Short strings need special care​

After the SIMD path was generalized, the expanded benchmark matrix revealed regressions on very short strings (4 bytes). The fix was simple: skip the SIMD loop entirely when the string is shorter than one register width. Short strings go straight to the scalar path, which avoids the overhead of loading and testing a partial SIMD register.

A branchless getUtf8CharLength​

The scalar remainder loop originally called utf8proc_char_length, which involves several branches. A branchless alternative using std::countl_zero compiles to a handful of instructions:

auto n = std::countl_zero((~static_cast<uint32_t>(c) << 24) | 1);
return (n == 0 || n > 4) ? 1 : n;

This improved performance on the short-string and remainder paths.

Confidence through comprehensive testing​

Since cappedLengthUnicode is widely used across Velox (unlike cappedByteLengthUnicode which is limited to Iceberg and Spark), the change needed high confidence. The PR added tests for multi-byte UTF-8 sequences that straddle SIMD boundaries, continuation bytes in the scalar tail, and strings that are exactly one register width.

Benchmark results​

We added StringCoreBenchmark.cpp to measure both helpers across a matrix of string lengths (4, 16, 64, 256, 1024 bytes) and ASCII ratios (0%, 1%, 25%, 50%, 75%, 99%, 100%). Each benchmark processes 10,000 strings with maxChars = 128.

ASCII ratio indicates the percentage of characters in each string that are single-byte ASCII; the remaining characters are multi-byte UTF-8. The following table highlights representative speedups from the full matrix (measured on an Apple M3):

The small regressions on 4-byte strings are within noise for such tiny inputs. For strings ≥ 16 bytes — which dominate real workloads — speedups range from 2× to over 15×, with the largest gains on mixed-content strings where the SIMD loop handles both ASCII and non-ASCII blocks without falling back to scalar.

We also validated the results on an x86_64 server (Intel Xeon Skylake with AVX-512) and observed similar improvement patterns. A full set of raw numbers is available in the perf data gist and summarized in the PR.

Takeaways​

• Measure first, optimize second. The optimization started from a real hot spot found in the Iceberg write benchmark — not from speculation.
• Optimize the core, not the symptom. Rather than patching the call site, we pushed the improvement into the shared helper so that all callers benefit.
• SIMD is a first-class tool in Velox for performance-critical paths. String processing — ubiquitous in query execution and dominated by byte-wise work — is a natural fit for vectorized scanning.
• Avoid switching between code paths. Re-entering the SIMD path after every non-ASCII block introduced extra branching and switching overhead; a unified loop that handles both ASCII and mixed blocks was consistently faster.
• Branchless is not always faster. A single branchless remainder loop was slower than two tailored loops on short strings.
• Avoid duplication. The two helpers share a single parameterized implementation, keeping behavior consistent and reducing maintenance burden.
• Data-driven iteration. We tried multiple strategies and compared them against a realistic benchmark matrix. We kept the version that improved the common case without regressing Unicode-heavy workloads.
• Comprehensive benchmarks across different string lengths and ASCII mixing ratios are essential for validating correctness-sensitive optimizations.

Acknowledgements​

Special thanks to Masha Basmanova and Yuhta for their guidance and detailed code reviews that shaped the SIMD design and pushed for comprehensive benchmarking.

In Presto, split takes a literal string delimiter and regexp_split is a separate function for regex-based splitting. Spark's split, however, always treats the delimiter as a regular expression.

Both LIKE and Spark's split can silently produce wrong results and waste CPU when used with column values instead of constants. Understanding why this happens helps write faster, more correct queries — and helps engine developers make better design choices.

LIKE is not contains​

A very common query pattern is to check whether one string contains another:

SELECT * FROM t WHERE name LIKE '%' || search_term || '%'

This looks intuitive: wrap search_term in % wildcards and you get a "contains" check. But LIKE is not the same as substring matching. LIKE treats _ as a single-character wildcard and % as a multi-character wildcard. If search_term comes from a column and contains these characters, the results are silently wrong:

SELECT url,
       url LIKE '%' || search_term || '%' AS like_result,
       strpos(url, search_term) > 0 AS contains_result
FROM (VALUES
    ('https://site.com/home'),
    ('https://site.com/user_profile'),
    ('https://site.com/username')
) AS t(url)
CROSS JOIN (VALUES ('user_')) AS s(search_term);

              url              | like_result | contains_result
-------------------------------+-------------+----------------
https://site.com/home          |    false    |     false
https://site.com/user_profile  |    true     |     true
https://site.com/username      |    true     |     false

LIKE '%user_%' matches 'https://site.com/username' because _ is a wildcard that matches any single character — in this case, n. But strpos(url, 'user_') > 0 treats _ as a literal underscore and correctly reports that 'https://site.com/username' does not contain the substring 'user_'.

When the pattern is a constant, this distinction is visible and intentional. But when users write x LIKE '%' || y || '%' where y is a column, the values of y may contain _ or % characters — and they will be silently interpreted as wildcards, producing wrong results.

Spark's split treats delimiters as regex​

In Presto, the split function takes a literal string delimiter, while regexp_split is a separate function for regex-based splitting. This distinction makes the intent clear.

Spark's split function, however, always treats the delimiter as a regular expression. Users rarely realize this, and a common pattern is to split a string using a value from another column:

select split(dir_path, location_path)[1] as partition_name from t

Here, a table stores Hive partition metadata: dir_path is the full partition path (e.g., /data/warehouse/db.name/table/ds=2024-01-01) and location_path is the table path (e.g., /data/warehouse/db.name/table). The user wants to strip the table path prefix to get the partition name.

This works for simple paths. But location_path is interpreted as a regular expression, not a literal string. If it contains . — as in db.name — the . matches any character, not a literal dot. Characters like (, ), [, +, *, ?, and $ would also cause wrong results or errors.

A correct alternative that also executes faster uses simple string operations:

IF(starts_with(dir_path, location_path),
   substr(dir_path, length(location_path) + 2)) as partition_name

This is a bit more verbose than split(dir_path, location_path)[1], but it is correct for all inputs and avoids regex compilation entirely.

Performance trap​

Beyond correctness, there is a performance problem. Both LIKE and Spark's split use RE2 as the regex engine. RE2 is fast and safe, but compiling a regular expression can take up to 200x more CPU time than evaluating it.

When the pattern or delimiter is a constant, the regex is compiled once and reused for every row. The cost is negligible. But when the pattern comes from a column, a new regex may need to be compiled for every distinct value. A table with thousands of distinct location_path values means thousands of regex compilations — each one expensive and none of them necessary.

Velox limits the number of compiled regular expressions per function instance per thread of execution via the expression.max_compiled_regexes configuration property (default: 100). When this limit is reached, the query fails with an error.

Tempting but wrong fix​

When users hit this limit, the natural reaction is to ask the engine developers to raise or eliminate the cap. A recent pull request proposed replacing the fixed-size cache with an evicting cache: when the limit is reached, the oldest compiled regex is evicted to make room for the new one.

This sounds reasonable, and the motivation is understandable — users migrating from Spark don't want to rewrite working queries. But it makes things worse:

• It hides the correctness bug. The query no longer fails, so users never discover that their LIKE pattern or split delimiter is being interpreted as a regex and producing wrong results for inputs with special characters.
• It makes the performance problem worse. With thousands of distinct patterns, the cache churns constantly — evicting one compiled regex only to compile another. The query runs, but dramatically slower than necessary, and the user has no indication why. In shared multi-tenant clusters, a single slow query like this can consume excessive CPU and affect other users' workloads.

The error is a feature, not a bug. It is an early warning that catches misuse before it leads to silently wrong results in production and prevents a single query from wasting shared cluster resources.

Right fix​

For users: replace LIKE with literal string operations when checking for substrings. Use strpos(x, y) > 0 or contains(x, y) instead of x LIKE '%' || y || '%'. For Spark's split with literal delimiters, use substr or other string functions that don't involve regex.

For engine developers: optimize the functions to avoid regex when it isn't needed. Velox's LIKE implementation already does this. As described in James Xu's earlier blog post, the engine analyzes each pattern and uses fast paths — prefix match, suffix match, substring search — whenever the pattern contains only regular characters and _ wildcards. For simple patterns, this gives up to 750x speedup over regex. Regex is compiled only for patterns that truly require it, and these optimized patterns are not counted toward the compiled regex limit.

The same approach should be applied to Spark's split function. The engine can check whether the delimiter contains any regex metacharacters. If it doesn't, a simple string search can be used instead of compiling a regex. This would make queries like split(dir_path, location_path) both fast and correct — without users needing to change anything and without removing the safety net for cases that genuinely require regex.

Takeaways​

• LIKE is not contains. The _ and % wildcards can silently corrupt results when the pattern comes from a column.
• Spark's split treats delimiters as regex. Characters like . in column values are interpreted as regex metacharacters, not literal characters. Presto avoids this by separating split (literal) and regexp_split (regex).
• When a query hits the compiled regex limit, the right response is to fix the query, not to raise the limit.
• Engine developers should optimize functions to avoid regex when the input is a plain string, rather than making it easier to misuse regex at scale.

Strings are ubiquitously used in large-scale analytic query processing. From storing identifiers, names, labels, or structured data (like json/xml), to simply descriptive text, like a product description or the contents of this very blog post, there is hardly a SQL query that does not require the manipulation of string data.

This post describes in more detail how Velox handles columns of strings, the low-level C++ APIs involved and some recent changes made to them, and presents best practices for string usage throughout Velox's codebase.

StringView​

To efficiently enable string operations, Velox provides a specialized physical C++ type, called velox::StringView. The main purpose of this layout, also called a German String Layout, is to optimize common string operations over small strings, or operations that can be performed by only accessing a string's prefix.

StringViews are 16-byte objects that provide a non-owning string-like API that refers to data stored elsewhere in a columnar Velox Vector buffer. The lifetime of a velox::StringView is always tied to the lifetime of the underlying Velox Vector buffer, in the sense that it only provides a pointer (view) into the external buffer. It provides a similar abstraction to std::string_view, with the following additional optimizations:

• Small String Optimization (SSO): strings up to 12 characters are always stored inline, within the object itself. With the assumption that strings are in many cases small, it removes the need for dereferencing the larger data buffer, hopefully reducing cache misses and memory bus traffic.

• Prefix optimization: a 4 character prefix is always stored inline within the StringView object to enable failed comparison to be short circuited, also skipping accesses to the external buffer.

To enable better interoperability with other engines, a few years ago we collaborated with the Arrow community in what resulted to be the BinaryView Arrow format for string data. Ever since, we have seen increased adoption of this format in a series of other systems and libraries.

The Old Unsafe API​

For convenience, in Velox we used to allow developers to implicitly convert velox::StringView into std::string, since many APIs are built based on std::string. For example, one could naively do:

// Given this signature.
void myFunction(const std::string& input) {
  // ...
}

// All would silently result in a string copy
// (valueAt() returns a velox::StringView):

myFunction(stringVector->valueAt(0));

std::string myStr = stringVector->valueAt(0);

std::unordered_map<std::string, size_t> myMap;
myMap[stringVector->valueAt(0)] = 1;

While these usages seem harmless, they would all result in a full string copy, and potentially a memory allocation, depending on the string size. We have found this anti-pattern to be commonly used; in most cases, the developer did not intend for a copy to be performed, and this behavior silently added unnecessary overhead.

What Changed​

To prevent such inadvertent string copies, in #15946 we removed implicit conversion from velox::StringView to std::string. You can still do so if needed, but you now have to explicitly state it, for clarity:

// Compilation failure from now on:
myFunction(stringVector->valueAt(0));

// Ok:
myFunction(std::string(stringVector->valueAt(0)));

Instead, we are now making conversion from velox::StringView to std::string_view available implicitly. std::string_view's are non-owning, so constructing them based on a pointer and size is essentially free.

A series of PR were landed before this change to clean up any dependencies on the old behavior throughout Velox's codebase, making them explicitly defined when needed.

folly::StringPiece​

folly::StringPiece's are now superseded by std::string_view. As part of this work, we have also cleaned up and removed every usage of folly::StringPiece in Velox; do not use folly::StringPiece in future code in Velox. Use std::string_view or velox::StringView instead.

If you need to use a library that takes a folly::StringPiece, for example, folly::parseJson(), you can pass a std::string_view instead, since the std::string_view => folly::StringPiece conversion can be done implicitly.

The RValue Gotcha​

A slightly unintuitive gotcha of velox::StringView is the handling of rvalues and temporary values. Since velox::StringView may contain inlined strings, if you take a pointer to the string contents, the string happened to be small and inlined, and the velox::StringView object is destroyed, you would end up with a dangling pointer. For example:

// Unsafe: would result in a std::string_view pointing to
// the temporary object that gets destroyed right away.
std::string_view sv = stringVector->valueAt(0);
LOG(INFO) << "Happily would have crashed: " << sv;

To prevent these unsafe usages, the rvalue-based version of these conversions, both explicit and implicit, and for both std::string and std::string_view, are disabled. This means that the snippets above will give you a compilation error.

The safe behavior would be to create a non-temporary object in the stack:

velox::StringView sv1 = stringVector->valueAt(0);
std::string_view sv2 = sv1; // all good, as long as sv2 is not used after
                             // sv1 is destroyed.

Best Practices​

To summarize, from now on, the conversions:

• velox::StringView => std::string: won't be done implicitly anymore. This now needs to be explicitly stated, for clarity.
• velox::StringView => std::string_view: can be done implicitly.
• velox::StringView => folly::StringPiece: do not use. If needed for legacy external libraries, pass a std::string_view instead.
• rvalue velox::StringView&& => any other type: compilation error to minimize risk of dangling pointer.

Please reach out on Slack if you have any questions.

Velox Task Barriers provide a synchronization mechanism that not only enables efficient task reuse, important for workloads such as AI training data loading, but also delivers the strict sequencing and checkpointing semantics required for streaming workloads.

By injecting a barrier split, users guarantee that no subsequent data is processed until the entire DAG is flushed and the synchronization signal is unblocked. This capability serves two critical patterns:

1. Task Reuse: Eliminates the overhead of repeated task initialization and teardown by safely reconfiguring warm tasks for new queries. This is a recurring pattern in AI training data loading workloads.

2. Streaming Processing: Enables continuous data handling with consistent checkpoints, allowing stateful operators to maintain context across batches without service interruption.

See the Task Barrier Developer Guide for implementation details.

The Problem​

Creating a new Velox task for every data request introduces overhead:

• Memory pool initialization and configuration
• Pipeline and operator construction
• Connector and data source setup
• Thread pool registration

For high-throughput workloads that process thousands of requests, this overhead may accumulate and impact overall system performance.

The Solution: Task Barrier​

Task Barrier introduces a lightweight synchronization mechanism that:

1. Drains all in-flight data - Ensures all buffered data flows through the pipeline to completion.

2. Resets stateful operators - Clears internal state in operators like aggregations, joins, and sorts.

3. Preserves task infrastructure - Keeps memory pools, thread registrations, and connector sessions intact.

How It Works​

The barrier is triggered by a special BarrierSplit injected into the task's source operators via requestBarrier(). When received:

1. Source operators propagate the barrier signal downstream.
2. Each operator calls finishDriverBarrier() to flush and reset its state.
3. Once all drivers complete barrier processing, the task is ready for a new split.

Workflow Example (Pseudo-code)​

The following example demonstrates the usage pattern: the user adds splits, requests a barrier, and then must continue to drain results until the barrier is reached. This ensures all in-flight data leaves the pipeline, allowing the barrier to complete.

// 1. Initialize the task infrastructure once
auto task = Task::create(config, memoryPool);

// 2. Loop to process multiple batches using the same task
while (scheduler.hasMoreRequests()) {

  // A. Get new data splits (e.g., from a file or query)
  auto splits = scheduler.getNextSplits();

  // B. Add splits to the running task
  task->addSplits(sourcePlanNodeId, splits);

  // C. Request a barrier to mark the end of this batch.
  // This injects a BarrierSplit that flows behind the data.
  // Note: Any splits added to the task AFTER this call will be blocked
  // and will not be processed until this barrier is fully cleared.
  auto barrierFuture = task->requestBarrier();

  // D. Drain the pipeline until the barrier is passed.
  // Critical: We must keep pulling results so the task can flush
  // its buffers and reach the barrier.
  while (!barrierFuture.isReady()) {
    auto result = task->next(); // Get next result batch
    if (result != nullptr) {
      userOutput.consume(result);
    }
  }

  // E. Barrier reached.
  // The task has been reset (operators cleared, buffers flushed).
  // It is now ready for the next iteration immediately.
}

Operator Support​

Task Barrier requires operators to implement finishDriverBarrier() to properly reset their state:

• TableScan: Clears current split and reader state.
• HashBuild/HashProbe: Resets hash tables and probe state.
• Aggregation: Flushes partial aggregates and clears accumulators.
• OrderBy: Outputs sorted data and resets sort buffer.
• Exchange: Drains exchange queues.

Performance Impact​

By eliminating repeated task creation overhead, Task Barrier provides significant performance improvements for workloads with:

• Short-lived Queries: Specifically those that recur frequently with the exact same plan DAG, avoiding the need for repeated reconstruction.
• AI/ML Pipelines: High-frequency data loading requests where task initialization would otherwise dominate execution time.
• Iterative Processing: Queries that can benefit from "warm" caches and pre-allocated memory pools.

Learn More​

For detailed implementation information, API reference, and operator-specific behavior, see the Task Barrier documentation.

References​

• Data Ingestion for Machine Learning Training at Meta
• Flink Stateful Streaming Processing
• Velox Task Model
• Velox Task Barrier

Velox is a fully vectorized execution engine[1]. Its internal columnar memory layout enhances cache locality, exposes more inter-instruction parallelism to CPUs, and enables the use of SIMD instructions, significantly accelerating large-scale query processing.

However, some operators in Velox utilize a hybrid layout, where datasets can be temporarily converted to a row-oriented format. The OrderBy operator is one example, where our implementation first materializes the input vectors into rows, containing both sort keys and payload columns, sorts them, and converts the rows back to vectors.

In this article, we explain the rationale behind this design decision and provide experimental evidence for its implementation. We show a prototype of a hybrid sorting strategy that materializes only the sort-key columns, reducing the overhead of materializing payload columns. Contrary to expectations, the end-to-end performance did not improve—in fact, it was even up to 3× slower. We present the two variants and discuss why one is counter-intuitively faster than the other.

Row-based vs. Non-Materialized​

Row-based Sort​

The OrderBy operator in Velox’s current implementation uses a utility called SortBuffer to perform the sorting, which consists of three stages:

1. Input Stage: Serializes input Columnar Vectors into a row format, stored in a RowContainer.
2. Sort Stage: Sorts based on keys within the RowContainer.
3. Output Stage: Extract output vectors column by column from the RowContainer in sorted order.

While row-based sorting is more efficient than column-based sorting[2,3], what if we only materialize the sort key columns? We could then use the resulting sort indices to gather the payload data into the output vectors directly. This would save the cost of converting the payload columns to rows and back again. More importantly, it would allow us to spill the original vectors directly to disk rather than first converting rows back into vectors for spilling.

Non-Materialized Sort​

We have implemented a non-materializing sort strategy designed to improve sorting performance. The approach materializes only the sort key columns and their original vector indices, which are then used to gather the corresponding rows from the original input vectors into the output vector after the sort completes. It changes the SortBuffer to NonMaterizedSortBuffer, which consists of three stages:

1. Input Stage: Holds the input vector (its shared pointer) in a list, serializes key columns and additional index columns (VectorIndex and RowIndex) into rows, stored in a RowContainer.
2. Sort Stage: Sorts based on keys within the RowContainer.
3. Output Stage: Extracts the VectorIndex and RowIndex columns, uses them together to gather the corresponding rows from the original input vectors into the output vector.

In theory, this should have significantly reduced the overhead of materializing payload columns, especially for wide tables, since only sorting keys are materialized. However, the benchmark results were the exact opposite of our expectations. Despite successfully eliminating expensive serialization overhead and reducing the total instruction the end-to-end performance was 3x times slower.

Benchmark Result​

To validate the effectiveness of our new strategy, we designed a benchmark with a varying number of payload columns:

• Inputs: 1000 Input Vectors, 4096 rows per vector.
• Number of payload columns: 64, 128, 256.
• L2 cache: 80 MiB, L3 cache: 108 MiB.

The benchmark results confirm that Row-based Sort is the superior strategy, delivering a 1.9x to 3.9x overall speedup compared to Columnar Sort. While Row-based Sort incurs a significantly higher upfront cost during the Input phase (peaking at 104s), it maintains a highly stable and efficient Output phase (maximum 32s). In contrast, Columnar Sort suffers from severe performance degradation in the Output phase as the payload increases, with execution times surging from 42s to 283s, resulting in a much slower total execution time despite its negligible input overhead.

To identify the root cause of the performance divergence, we utilized perf stat to analyze micro-architectural efficiency and perf mem to profile memory access patterns during the critical Output phase.

The results reveal a stark contrast in CPU utilization. Although the Row-based approach executes 17% more instructions (due to serialization overhead), it maintains a high IPC of 2.4, indicating a fully utilized pipeline. In contrast, the Columnar approach suffers from a low IPC of 0.82, meaning the CPU is stalled for the majority of cycles. This is directly driven by the 35x difference in LLC Load Misses, which forces the Columnar implementation to fetch data from main memory repeatedly. The memory profile further confirms this bottleneck: Columnar mode is severely latency-bound, spending 38.1% of its execution time waiting for DRAM (RAM Hit) and experiencing significant congestion in the Line Fill Buffer (18.9% LFB Hit), while Row-based mode effectively utilizes the cache hierarchy.

The Memory Access Pattern​

Why does the non-materializing sort, specifically its gather method, cause so many cache misses? The answer lies in its memory access pattern. Since Velox is a columnar engine, the output is constructed column by column. For each column in an output vector, the gather process does the following:

1. It iterates through all rows of the current output vector.
2. For each row, locate the corresponding input vector via the sorted vector index.
3. Locates the source row in the corresponding input vector.
4. Copies the data from that single source cell to the target cell.

The sorted indices, by nature, offer low predictability. This forces the gather operation for a single output column to jump unpredictably across as many as different input vectors, fetching just one value from each. This random access pattern has two devastating consequences for performance.

First, at the micro-level, every single data read becomes a "long-distance" memory jump. The CPU's hardware prefetcher is rendered completely ineffective by this chaotic access pattern, resulting in almost every lookup yielding a cache miss.

Second, at the macro-level, the problem compounds with each column processed. The sheer volume of data touched potentially exceeds the size of the L3 cache. This ensures that by the time we start processing the next payload column, the necessary vectors have already been evicted from the cache. Consequently, the gather process must re-fetch the same vector metadata and data from main memory over and over again for each of the 256 payload columns. This results in 256 passes of cache-thrashing, random memory access, leading to a catastrophic number of cache misses and explaining the severe performance degradation.

In contrast, Velox’s current row-based approach serializes all input vectors into rows, with each allocation producing a contiguous buffer that holds a subset of those rows. Despite the serialization, the row layout preserves strong locality when materializing output vectors: once rows are in the cache, they can be used to extract multiple output columns. This leads to much better cache-line utilization and fewer cache misses than a columnar layout, where each fetched line often yields only a single value per column. Moreover, the largely sequential scans over contiguous buffers let the hardware prefetcher operate effectively, boosting throughput even in the presence of serialization overhead.

Conclusion​

This study reinforces the core principle of performance engineering: Hardware Sympathy. Without understanding the characteristics of the memory hierarchy and optimizing for it, simply reducing the instruction count usually does not guarantee better performance.

Reference​

• [1] https://velox-lib.io/blog/velox-primer-part-1/
• [2] https://duckdb.org/pdf/ICDE2023-kuiper-muehleisen-sorting.pdf
• [3] https://duckdb.org/2021/08/27/external-sorting

Efficiently merging sorted data partitions at scale is crucial for a variety of training data preparation workloads, especially for Generative Recommenders (GRs) a new paradigm introduced in the paper Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. A key requirement is to merge training data across partitions—for example, merging hourly partitions into daily ones—while ensuring that all rows sharing the same primary key are stored consecutively. Training data is typically partitioned and bucketed by primary key, with rows sharing the same key stored consecutively, so merging across partitions essentially becomes a multi-way merge problem.

Normally, Apache Spark can be used for this sort-merge requirement — for example, via CLUSTER BY. However, training datasets for a single job can often reach the PB scale, which in turn generates shuffle data at PB scale. Although we typically apply bucketing and ordering by key when preparing training data in production, Spark can eliminate the shuffle when merging training data from multiple hourly partitions. However, each Spark task can only read the files planned from various partitions within a split sequentially, placing them into the sorter and spilling as needed. Only after all files have been read does Spark perform a sort-merge of the spilled files. This process produces a large number of small spill files, which further degrades efficiency.

Moreover, Spark’s spill is row-based with a low compression ratio, resulting in approximately 4 times amplification compared to the original columnar training data in the data lake. These factors significantly degrade task stability and performance. Velox has a LocalMerge operator that can be introduced into Apache Spark via Gluten or PySpark on Velox.

Note: To keep the focus on merging, the remainder of this article also assumes that each partition’s training data is already sorted by primary key—a common setup in training data pipelines.

LocalMerge Operator​

The LocalMerge operator consolidates its sources’ outputs into a single, sorted stream of rows. It runs single-threaded, while its upstream sources may run multi-threaded within the same task, producing multiple sorted inputs concurrently. For example, when merging 24 hourly partitions into a single daily partition (as shown in the figure below), the merge plan fragment is split into two pipelines:

• Pipeline 0: contains two operators, TableScan and CallbackSink. 24 drivers are instantiated to scan the 24 hourly partitions.
• Pipeline 1: contains only a single operator, LocalMerge, with one driver responsible for performing the sort merge.

A CallbackSink operator is installed at the end of each driver in Pipeline 0. It pushes the TableScan operator’s output vectors into the queues backing the merge streams. Inside LocalMerge, a TreeOfLosers performs a k-way merge over the 24 merge streams supplied by the Pipeline 0 drivers, producing a single, globally sorted output stream.

Multi-Round Spill Merge​

Although LocalMerge minimizes comparisons during merging, preserves row-ordering guarantees, and cleanly isolates the single-threaded merge from the multi-threaded scan phase for predictable performance, it can cause substantial memory pressure—particularly in training-data pipelines. In these workloads, extremely wide tables are common, and even after column pruning, thousands of columns may remain.

Moreover, training data is typically stored in PAX-style formats such as Parquet, ORC, or DRWF. Using Parquet as an example, the reader often needs to keep at least one page per column in memory. As a result, simply opening a Parquet file with thousands of columns can consume significant memory even before any merging occurs. Wide schemas further amplify per-column metadata, dictionary pages, and decompression buffers, inflating the overall footprint. In addition, the k-way merge must hold input vectors from multiple sources concurrently, which drives peak memory usage even higher.

To cap memory usage and avoid OOM when merging a large number of partitions, we extend LocalMerge to process fewer local sources at a time, leverage existing spill facilities to persist intermediate results, and introduce lazy-start activation for merge inputs. Using the case of merging 24 hourly partitions into a single daily partition, the process is organized into two phases:

Phase 1

1. Break the scan-and-merge into multiple rounds (e.g., 3 rounds).
2. In each round, lazily start a limited number of drivers (e.g., drivers 0–7, eight at a time).
3. The started drivers scan data and push it into the queues backing their respective merge streams.
4. Perform an in-memory k-way merge and spill the results, producing a spill-file group (one or more spill files per group).
5. After all inputs from drivers 0–7 are consumed and spilled, the drivers will be closed, and close the file streams opened by their TableScan operators, and release associated memory.
6. Repeat the above steps for the remaining rounds (drivers 8–15, then drivers 16–23), ensuring peak memory stays within budget.

Phase 2

1. Create a concatenated file stream for each spill-file group produced in Phase 1.
2. Schedule one async callback for each concatenated stream to prefetch and push data into a merge stream.
3. Merge the outputs of the three merge streams using a k-way merge (e.g., a loser-tree), and begin streaming the final, globally sorted results to downstream operators.
4. The output batch rows is limited adaptively by estimating row size from the merge streams which use the averaged row size from the first batch.

How To Use​

Set local_merge_spill_enabled to true to enable spilling for the LocalMerge operator (it is false by default). Then, set local_merge_max_num_merge_sources to control the number of merge sources per round according to your memory management strategy.

Note: An executor must be configured for spilling, as it would schedule an asynchronous callback for each concatenated stream to prefetch data and push it into the merge stream.

Future Work​

The number of merge sources is adjusted dynamically based on available memory, rather than being determined by the local_merge_max_num_merge_sources parameter. The process starts with a small number of sources, such as 2, and incrementally increases this number for subsequent rounds (e.g., to 4) as long as sufficient memory is available. The number of sources stops increasing once it reaches a memory-constrained limit.

Acknowledgements​

Thanks to Xiaoxuan Meng and Pedro Pederia for their guidance, review, and brainstorming. I also appreciate the excellent collaboration and work from my colleagues, Xiang Yao, Gang Wang, and Weixin Xu.

The State of the Velox Build System​

Velox’s codebase was started in Meta’s internal monorepo, which still serves as a source of truth. Changes from pull requests in the Github repository are not merged directly via the web UI. Instead, the changes are imported into the internal review and CI tool Phabricator, as a ‘diff’. There, it has to pass an additional set of CI checks before being merged into the monorepo and in turn, exported to Github as a commit on the ‘main’ branch.

Internally, Meta uses the Buck2 build system which, like its predecessor Buck, promotes small, granular build targets to improve caching and distributed compilation. Externally, however, the open-source community relies on CMake as a build system. When the CMake build system was first added, its structure mirrored the granular nature of the internal build targets.

The result was hundreds of targets: over one hundred library targets, built as static libraries, and more than two hundred executables for tests and examples. While Buck(2) is optimized for managing such granular targets, the same can not be said for CMake. Each subdirectory maintained its own CMakeLists.txt, and header includes were managed through global include_directories(), with no direct link between targets and their associated header files. This approach encouraged tight coupling across module boundaries. Over the years, dependencies accumulated organically, resulting in a highly interconnected, and in parts cyclic, dependency graph.

Static Linking vs. Shared Linking​

The combination of 300+ targets and static linking resulted in a massive build tree, dozens of GiB when building in release mode and several times larger when built with debug information. This grew to the point where we had to provision larger CI runners just to accommodate the build tree in debug mode!

Individual test executables could reach several GiB, making it impractical to transfer the executables between runners to parallelize the execution of the tests across different CI runners. Significantly delaying CI feedback for developers when pushing changes to a PR.

This size is the result of each static library containing a full copy of all of its dependencies' object files. With over 100+ strongly connected libraries, we end up with countless redundant copies of the same objects scattered throughout the build tree, blowing up the total size.

CMake usually requires the library dependency graph to be acyclic (a DAG). However it allows circular dependencies between static libraries, because it’s possible to adjust the linker invocation to work around possible issues with missing symbols.

For example, say library A depends on foo() from library B, and B in turn depends on bar() defined in A. If we link them in order A B, the linker will find foo() in B for use in A but will fail to find bar() for use in B. This happens because the linker processes the libraries from left to right and symbols that are not (yet) required are effectively ignored.

The trick is now simply to repeat the entire circular group in the linker arguments A B A B. Previously ignored symbols are now in the list of required symbols and can be resolved when the linker processes the repeated libraries.

However, there is no workaround for shared libraries. So the moment we attempted to build Velox as a set of shared libraries, CMake failed to configure. The textbook solution would involve refactoring dependencies between targets, explicitly marking dependencies - as PUBLIC( meaning forwarded to both direct and transitive dependents) or PRIVATE, ( required only for this target at build time), and manually breaking cycles. But with hundreds of targets and years of organic growth, this became an unmanageable task.

Solution Attempt 1: Automate​

Manually addressing this was unmanageable so, our first instinct as Software Engineers was, of course, to automate the process. I wrote a python package that parses all CMakelists.txt files in the repository using Antlr4, tracks which source files and headers belong to which target and reconstructs the library dependency graph. Based on the graph and whether headers are included from a source or a header file, the tool grouped the linked targets as either PRIVATE or PUBLIC.

While this worked locally and made it easier to resolve the circular dependencies, it proved far too brittle and disruptive to implement at scale. Velox's high development velocity meant many parts of the code base changed daily, making it impossible to land modifications across over 250+ CMake files in a timely manner while also allowing for thorough review and testing. Maintaining this carefully constructed DAG of dependencies would also have required an unsustainable amount of specialized reviewer attention, time that could be spent much more productively elsewhere.

Solution: Monolithic Library via CMake Wrappers​

Given the practical constraints we needed a solution that would unblock shared builds without disrupting the ongoing development. To achieve this we used a common pattern of larger CMake code bases, project specific wrapper functions. A special thanks to Marcus Hanwell for introducing me to the pattern he successfully used in VTK.

We created wrapper functions for a selection of the CMake commands that create and manage build targets: add_library became velox_add_library, target_link_libraries turned into velox_link_libraries etc. These functions just forward the arguments to the wrapped core command, unless the option to build Velox as a monolithic library is set. In the latter case instead of ~100 only a single library target is created.

Each call to velox_add_library just adds the relevant source files to the main velox target. To ensure drop-in compatibility, for every original library target, the wrapper creates an ALIAS target referring to velox. This way, all existing code, downstream consumers, old and new test targets, can continue to link to their accustomed velox_xyz targets, which effectively point to the same monolithic library. There is no need to adjust CMake files throughout the codebase, keeping the migration quick and entirely transparent to developers and users.

Some special-case libraries, such as those involving the CUDA driver, still needed to be built separately for runtime or integration reasons. The wrapper logic allows exemptions for these, letting them be built as standalone libraries even in monolithic mode.

Here's a simplified version of the relevant logic:

function(velox_add_library TARGET)
  if(VELOX_MONO_LIBRARY)
      if(TARGET velox)
        # Target already exists, append sources to it.
        target_sources(velox PRIVATE ${ARGN})
      else()
        add_library(velox ${ARGN})
      endif()
      # create alias for compatibility
      add_library(${TARGET} ALIAS velox)
  else()
    # Just call add_library as normal
    add_library(${TARGET} ${ARGN})
  endif()
endfunction()

This new, monolithic library obviously doesn’t suffer from cyclic dependencies, as it only has external dependencies. So after implementing the wrapper functions, building Velox as a shared library only required some minor adjustments!

We have now switched the default configuration to build the monolithic library and transitioned to build the shared library in our main PR CI and macOS builds.

Result​

The results of our efforts can be clearly seen in our Build Metrics Report with the size of the resulting binaries being the biggest win as seen in Table 1. The workflow that collects these metrics uses our regular CI image with GCC 12.2 and ld 2.38.

The new executables built against the shared libvelox.so are a mere ~4% of their previous size! This unlocks improvements across the board both for the local developer experience, packaging/deployments for downstream users as well as CI.

Table 1: Total size of test, fuzzer and benchmark executables

Less impressive but still notable is the reduction of the debug build time by almost two hours¹. The improvement is entirely explained by the much faster link time for the shared build of ~30 minutes compared to the 2 hours and 20 minutes it takes to link the static library.

Table 2: Total time spent on compilation and linking across all cores

In addition to unblocking shared builds, the wrapper function pattern offers further benefits for future feature additions or build-system wide adjustments with minimal disruption. We already use them to automate installation of header files along the Velox libraries, with no changes to the targets themselves.

The wrapper functions will also help with defining and enforcing the public Velox API by allowing us to manage header visibility, component libraries like GPU and Cloud extensions and versioning in a consistent and centralized manner.

The C++ standard used in a project determines what built-in functionality developers can use to ease development and extended capabilities.

Since its inception in August of 2021 Velox used the C++17 standard.

Benefits​

Switching to C++20 contributes to having a modern ecosystem that is attractive in use, to develop in, and to maintain. Going forward Velox is looking to enhance the codebase by making use of newer compiler functionalities, such as sanitization checks, by also moving to support newer compiler versions for both GCC and Clang.

Changes​

New minimum compiler versions​

This change also changes the minimum required compiler versions for GCC and CLang. The following minimum versions are now required:

• GCC11 and later
• CLang 15 and later

C++20 major new features relevant to Velox​

• coroutines language feature
• modules language feature
• Calendar and Timezone library <chrono>
• constraints and concepts language feature
• 3-way operator language feature
• Ranges library feature

Please refer to the C++20 standard for a complete list of new features.

The minimum targeted compiler versions support

• coroutines
• Calendar and Timezone library <chrono>

Newer versions of compilers implement more and more and C++20 features and library support. Supporting Velox on newer compiler versions is a continuous effort.

Lessons​

There was some interesting behavior by the compilers. Changing the C++20 standard caused some compile errors in the existing code. One of these errors was caused by a compiler issue itself.

In GCC 12.2.1, which is used in the CI, the std::string + operator implementation ran into a known issue causing a warning. Because all warnings are errors we had to explictly add an exemption for this particlar compiler version.

In general, however, the found compile errors were reasonably easy to fix. Most of the changes were compatible with C++17 as well which means the code is bit more clean. However, one change caused slight trouble because it emitted warnings in C++17 causing build failures due to turning all warnigns into errors. This was the change to require this in the lambda capture where applicable. On the other hand, not addressing the changes to the lamda capture caused errors in C++20.

Overall, the switch to C++20 took some time but was not overly complicated. No changes to the CI pipelines used in the project were needed. It was limited to CMake and code changes.

This post describes the design principles and software components for extending Velox with hardware acceleration libraries like NVIDIA's cuDF. Velox provides a flexible execution model for hardware accelerators, and cuDF's data structures and algorithms align well with core components in Velox.

How Velox and cuDF Fit Together​

Extensibility is a key feature in Velox. The cuDF library integrates with Velox using the DriverAdapter interface to rewrite query plans for GPU execution. The rewriting process allows for operator replacement, operator fusion and operator addition, giving cuDF a lot of freedom when preparing a query plan with GPU operators. Once Velox converts a query plan into executable pipelines, the cuDF DriverAdapter rewrites the plan to use GPU operators:

Generally the cuDF DriverAdapter replaces operators one-to-one. An exception happens when a GPU operator is not yet implemented in cuDF, in which case a CudfConversion operator must be added to transfer from GPU to CPU, fallback to CPU execution, and then transfer back to GPU. For end-to-end GPU execution where cuDF replaces all of the Velox CPU operators, cuDF relies on Velox's pipeline-based execution model to separate stages of execution, partition the work across drivers, and schedule concurrent work on the GPU.

Velox and cuDF benefit from a shared data format, using columnar layout and Arrow-compatible buffers. The CudfVector type in the experimental cuDF backend inherits from Velox's RowVector type, and manages a std::unique_ptr to a cudf::table (link) which owns the GPU-resident data. CudfVector also maintains an rmm::cuda_stream_view (link) to ensure that asynchronously launched GPU operations are stream-ordered. Even when the data resides in GPU memory, CudfVector allows the operator interfaces to follow Velox's standard execution model of producing and consuming RowVector objects.

The first set of GPU operators in the experimental cuDF backend include TableScan, HashJoin, HashAggregation, FilterProject, OrderBy and more. cuDF's C++ API covers many features beyond this initial set, including broad support for list and struct types, high-performance string operations, and configurable null handling. These features are critical for running workloads in Velox with correct semantics for Presto and Spark users. Future work will integrate cuDF support for more Velox operators.

GPU execution: fewer drivers and larger batches​

Compared to the typical settings for CPU execution, GPU execution with cuDF benefits from a lower driver count and larger batch size. Whereas Velox CPU performs best with ~1K rows per batch and driver count equal to the number of physical CPU cores, we have found Velox's cuDF backend to perform best with ~1 GiB batch size and 2-8 drivers for a single GPU. cuDF GPU operators launch one or more device-wide kernels and a single driver may not fully utilize GPU compute. Additional drivers improve throughput by queuing up more GPU work and avoiding stalling during host-to-device copies. Adding more drivers increases memory pressure linearly, and query processing throughput saturates once GPU compute is fully utilized. Please check out our talk, “Accelerating Velox with RAPIDS cuDF” from VeloxCon 2025 to learn more.

Call to Action​

The experimental cuDF backend in Velox is under active development. It is implemented entirely in C++ and does not require CUDA programming knowledge. There are dozens more APIs in cuDF that can be integrated as Velox operators, and many design areas to explore such as spilling, remote storage connectors, and expression handling.

If you're interested in bringing GPU acceleration to the Velox ecosystem, please join us in the #velox-oss-community channel on the Velox Slack workspace. Your contributions will help push the limits of performance in Presto, Spark and other tools powered by Velox.

Velox depends on several libraries. Some of these dependencies include open-source libraries from Meta, including Folly and Facebook Thrift. These libraries are in active development and also depend on each other, so they all have to be updated to the same version at the same time.

Updating these dependencies typically involves modifying the Velox code to align with any public API or semantic changes in these dependencies. However, a recent upgrade of Folly and Facebook Thrift to version v2025.04.28.00 caused a SEGFAULT only in one unit test in Velox named velox_functions_remote_client_test.

Investigation​

We immediately put on our gdb gloves and looked at the stack traces. This issue was also reproducible in a debug build. The SEGFAULT occurred in Facebook Thrift's ThriftServer Class during it's initialization but the offending call was invoking a destructor of a certain handler. However, the corresponding source code was pointing to an invocation of a different function. And this code was present inside a Facebook Thrift header called AsyncProcessor.h.

This handler (RemoteServer) was implemented in Velox as a Thrift definition. Velox compiled this thrift file using Facebook Thrift, and the generated code was using the ServerInterface class in Facebook Thrift. This ServerInterface class was further extended from both the AsyncProcessorFactory and ServiceHandlerBase interfaces in Facebook Thrift.

One of the culprits resulting in SEGFAULTs in the past was the conflict due to the usage of Apache Thrift and Facebook Thrift. However, this was not the root cause this time because we were able to reproduce this issue by just building the test without the Apache Thrift dependency installed. We were entering a new territory to investigate, and we were not sure where to start.

We then compiled an example called EchoService in Facebook Thrift that was very similar to the RemoteServer, and it worked. Then we copied and compiled the Velox RemoteServer in Facebook Thrift and that worked too! So the culprit was likely in the compilation flags, which likely differed between Facebook Thrift and Velox. We enabled the verbose logging for both builds and were able to spot one difference. We saw the GCC coroutines flag being used in the Facebook Thrift build.

We were also curious about the invocation of the destructor instead of the actual function. We put our gdb gloves back on and dumped the entire vtable for the RemoteServer class and its base classes. The vtables were different when it was built in Velox vs. Facebook Thrift. Specifically, the list of functions inherited from ServiceHandlerBase was different.

The vtable for the RemoteServer handler in the Velox build had the following entries:

folly::SemiFuture<folly::Unit> semifuture_onStartServing()
folly::SemiFuture<folly::Unit> semifuture_onStopRequested()
Thunk ServiceHandlerBase::~ServiceHandlerBase

The vtable for the RemoteServer handler in the Facebook Thrift build had the following entries:

folly::coro::Task<void> co_onStartServing()
folly::coro::Task<void> co_onStopRequested()
folly::SemiFuture<folly::Unit> semifuture_onStartServing()
folly::SemiFuture<folly::Unit> semifuture_onStopRequested()
Thunk ServiceHandlerBase::~ServiceHandlerBase

Tying up both pieces of evidence, we could conclude that Velox generated a different vtable structure compared to what Facebook Thrift (and thus ThriftServer) was built with. Looking around further, we noticed that the ServiceHandlerBase was conditionally adding functions based on the coroutines compile flag that influences the FOLLY_HAS_COROUTINES macro from the portability header.

class ServiceHandlerBase {
  ....
 public:
#if FOLLY_HAS_COROUTINES
  virtual folly::coro::Task<void> co_onStartServing() { co_return; }
  virtual folly::coro::Task<void> co_onStopRequested() {
    throw MethodNotImplemented();
  }
#endif

  virtual folly::SemiFuture<folly::Unit> semifuture_onStartServing() {
#if FOLLY_HAS_COROUTINES
    return co_onStartServing().semi();
#else
    return folly::makeSemiFuture();
#endif
  }

As a result, the ThriftServer would access an incorrect function (~ServiceHandlerBase destructor at offset 3 in the first vtable above) instead of the expected initialization function (semifuture_onStartServing at offset 3 in the second vtable above), thus resulting in a SEGFAULT. We recompiled the Facebook Thrift dependency for Velox with the coroutines compile flag disabled, and the test passed.

SELECT l_partkey, count(*) FROM lineitem GROUP BY l_partkey;

We discussed how a query starts, how tasks are set up, and the interactions between plans, operators, and drivers. We have also presented how the first stage of the query is executed, from table scan to partitioned output - or the producer side of the shuffle.  

In this article, we will discuss the second query stage, or the consumer side of the shuffle.

Shuffle Consumer​

As presented in the previous post, on the first query stage each worker reads the table, then produces a series of information packets (SerializedPages) intended for different workers of stage 2. In our example, the lineitem table has no particular physical partitioning or clustering key. This means that any row of the table can have any value of l_partkey in any of the files forming the table. So in order to group data based on l_partkey, we first need to make sure that rows containing the same values for l_partkey are processed by the same worker – the data shuffle at the end of stage 1.

The overall query structure is as follows:

The query coordinator distributes table scan splits to stage 1 workers in no particular order. The workers process these and, as a side effect, fill destination buffers that will be consumed by stage 2 workers. Assuming there are 100 stage 2 workers, every stage 1 Driver has its own PartitionedOutput which has 100 destinations. When the buffered serializations grow large enough, these are handed off to the worker's OutputBufferManager. 

Now let’s focus on the stage 2 query fragment. Each stage 2 worker has the following plan fragment: PartitionedOutput over Aggregation over LocalExchange over Exchange.  

Each stage 2 Task corresponds to one destination in the OutputBufferManager of each stage 1 worker. The first stage 2 Tasks fetches destination zero from all the stage 1 Tasks. The second stage 2 fetches destination 1 from the first stage Tasks and so on.  Everybody talks to everybody else. The shuffle proceeds without any central coordination.

But let's zoom in to what actually happens at the start of stage 2.

The plan has a LocalExchange node after the Exchange. This becomes two pipelines: Exchange and LocalPartition on one side, and LocalExchange, HashAggregation, and PartitionedOutput on the other side. 

The Velox Task is intended to be multithreaded, with typically 5 to 30 Drivers. There can be hundreds of Tasks per stage, thus amounting to thousands of threads per stage. So, each of the 100 second stage workers is consuming 1/100 of the total output of the first stage. But it is doing this in a multithreaded manner, with many threads consuming from the ExchangeSource. We will explain this later.

In order to execute a multithreaded group by, we can either have a thread-safe hash table or we can partition the data in n disjoint streams, and then proceed aggregating each stream on its own thread. On a CPU, we almost always prefer to have threads working on their own memory, so data will be locally partitioned based on l_partkey using a local exchange. CPUs have complex cache coherence protocols to give observers a consistent ordered view of memory, so reconciliation after many cores have written the same cache line is both mandatory and expensive. Strict memory-to-thread affinity is what makes multicore scalability work. 

LocalPartition and LocalExchange​

To create efficient and independent memory access patterns, the second stage reshuffles the data using a local exchange. In concept, this is like the remote exchange between tasks, but is scoped inside the Task. The producer side (LocalPartition) calculates a hash on the partitioning column l_partkey, and divides this into one destination per Driver in the consumer pipeline. The consumer pipeline has a source operator LocalExchange that reads from the queue filled by LocalPartition. See LocalPartition.h for details. Also see the code in Task.cpp for setting up the queues between local exchange producers and consumers.

While remote shuffles work with serialized data, local exchanges pass in-memory vectors between threads. This is also the first time we encounter the notion of using columnar encodings to accelerate vectorized execution. Velox became known by its extensive use of such techniques, which we call compressed execution. In this instance, we use dictionaries to slice vectors across multiple destinations – we will discuss it next.

Dictionary Magic​

Query execution often requires changes to the cardinality (number of rows in a result) of the data. This is essentially what filters and joins do – they either reduce the cardinality of the data, e.g., filters or selective joins, or increase the cardinality of the data, e.g. joins with multiple key matches.

Repartitioning in LocalPartition assigns a destination to each row of the input based on the partitioning key. It then makes a vector for each destination with just the rows for that destination. Suppose rows 2, 8 and 100 of the current input hash to 1. Then the vector for destination 1 would only have rows 2, 8 and 100 from the original input. We could make a vector of three rows by copying the data. Instead, we save the copy by wrapping each column of the original input in a DictionaryVector with length 3 and indices 2, 8 and 100. This is much more efficient than copying, especially for wide and nested data.

Later on, the LocalExchange consumer thread running destination 1 will see a DictionaryVector of 3 rows. When this is accessed by the HashAggregation Operator, the aggregation sees a dictionary and will then take the indirection and will access for row 0 the value at 2 in the base (inner) vector, for row 1 the value at 8, and so forth. The consumer for destination 0 does the exact same thing but will access, for example, rows 4, 9 and 50.

The base of the dictionaries is the same on all the consumer threads. Each consumer thread just looks at a different subset. The cores read the same cache lines, but because the base is not written to (read-only), there is no cache-coherence overhead. 

To summarize, a DictionaryVector<T> is a wrapper around any vector of T. The DictionaryVector specifies indices, which give indices into the base vector. Dictionary encoding is typically used when there are few distinct values in a column. Take the strings “experiment” and “baseline”. If a column only has these values, it is far more efficient to represent it as a vector with “experiment” at 0 and “baseline” at 1, and a DictionaryVector that has, say, 1000 elements, where these are either the index 0 or 1.  

Besides this, DictionaryVectors can also be used to denote a subset or a reordering of elements of the base. Because all places that accept vectors also accept DictionaryVectors, the DictionaryVector becomes the universal way of representing selection. This is a central precept of Velox and other modern vectorized engines. We will often come across this concept.

Aggregation and Pipeline Barrier​

We have now arrived at the second pipeline of stage 2. It begins with LocalExchange, which then feeds into HashAggregation. The LocalExchange picks a fraction of the Task's input specific to its local destination, about 1/number-of-drivers of the task's input.

We will talk about hash tables, their specific layout and other tricks in a later post. For now, we look at HashAggregation as a black box. This specific aggregation is a final aggregation, which is a full pipeline barrier that only produces output after all input is received.

How does the aggregation know it has received all its input? Let's trace the progress of the completion signal through the shuffles. A leaf worker knows that it is at end if the Task has received the “no more splits” message in the last task update from the coordinator.

So, if one DataSource inside a tableScan is at end and there are no more splits, this particular TableScan is not blocked and it is at end. This will have the Driver call PartitionedOutput::noMoreInput() on the tableScan's PartitionedOutput. This will cause any buffered data for any destination to get flushed and moved over to OutputBufferManager with a note that no more is coming. OutputBufferManager knows how many Drivers there are in the TableScan pipeline. After it has received this many “no more data coming” messages, it can tell all destinations that this Task will generate no more data for them. 

Now, when stage 2 tasks query stage 1 producers, they will know that they are at end when all the producers have signalled that there is no more data coming. The response to the get data for destination request has a flag identifying the last batch. The ExchangeSource on the stage 2 Task set the no-more-data flag. This is then queried by all the Drivers and each of the Exchange Operators sees this. This then calls noMoreInput in the LocalPartition. This queues up a “no more data” signal in the local exchange queues. If the LocalExchange at the start of the second pipeline of stage 2 sees a “no more data” from each of its sources, then it is at end and noMoreInput is called on the HashAggregation. 

This is how the end of data propagates. Up until now, HashAggregation has produced no output, since the counts are not known until all input is received. Now, HashAggregation starts producing batches of output, which contain the l_partkey value and the number of times this has been seen. This reaches the last PartitionedOutput, which in this case has only one destination, the final worker that produces the result set. This will be at end when all the 100 sources have reported their own end of data.

Recap​

We have finally walked through the distributed execution of a simple query. We presented how data is partitioned between workers in the cluster, and then a second time over inside each worker.

Velox and Presto are designed to aggressively parallelize execution, which means creating distinct, non-overlapping sets of data to process on each thread. The more threads, the more throughput. Also remember that for CPU threads to be effective, they must process tasks that are large enough (often over 100 microseconds worth of cpu time), and not communicate too much with other threads or write to memory other threads are writing to. This is accomplished with local exchanges.

Another important thing to remember from this article is how columnar encodings (DictionaryVectors, in particular) can be used as a zero-copy way of representing selection/reorder/duplication. We will see this pattern again with filters, joins, and other relational operations.

Next we will be looking at joins, filters, and hash aggregations. Stay tuned!

SELECT l_partkey, count(*) FROM lineitem GROUP BY l_partkey;

We use the TPC-H schema to illustrate the example, and Prestissimo as the compute engine orchestrating distributed query execution. Prestissimo is responsible for the query engine frontend (parsing, resolving metadata, planning, optimizing) and distributed execution (allocating resources and shipping query fragments), and Velox is responsible for the execution of plan fragments within a single worker node. Throughout this article, we will present which functions are performed by Velox and which by the distributed engine - Prestissimo, in this example.

Query Setup​

Prestissimo first receives the query through a coordinator node, which is responsible for parsing and planning the query. For our sample query, a distributed query plan with three query fragments will be created:

1. The first fragment reads the l_partkey column from the lineitem table and divides its output according to a hash of l_partkey.
2. The second fragment reads the output from the first fragment and updates a hash table from l_partkey containing the number of times the particular value of l_partkey has been seen (the count(*) aggregate function implementation).
3. The final fragment then reads the content of the hash tables, once the second fragment has received all the rows from the first fragment.

The shuffle between the two first fragments partitions the data according to l_partkey. Suppose there are 100 instances of the second fragment. If the hash of l_partkey modulo 100 is 0, the row goes to the first task in the second stage; if it is 1, the row goes to the second task, and so forth. In this way, each second stage Task gets a distinct subset of the rows. The shuffle between the second and third stage is a gather, meaning that there is one Task in the third stage that will read the output of all 100 tasks in the second stage.

A Stage is the set of Tasks that share the same plan fragment. A Task is the main integration point between Prestissimo and Velox; it’s the Velox execution instance that physically processes all or part of the data that passes through the stage.

To set up the distributed execution, Prestissimo first selects the workers from the pool of Prestissimo server processes it manages. Assuming that stage 1 is 10 workers wide, it selects 10 server processes and sends the first stage plan to them. It then selects 100 workers for stage 2 and sends the second stage plan to these. The last stage that gathers the result has only one worker, so Prestissimo sends the final plan to only one worker. The set of workers for each stage may overlap, so a single worker process may host multiple stages of one query.

Let's now look more closely at what each worker does at query setup time.

Task Setup​

In Prestissimo, the message that sets up a Task in a worker is called Task Update. A Task Update has the following information: the plan, configuration settings, and an optional list of splits. Splits are further qualified by what plan node they are intended for, and whether more splits for the recipient plan node and split group will be coming.

Since split generation involves enumerating files from storage (so they may take a while), Presto allows splits to be sent to workers asynchronously, such that the generation of splits can run in parallel with the execution of the first splits. Therefore, the first task update specifies the plan and the config. Subsequent ones only add more splits.

Besides the plan, the coordinator provides configs as maps from string key to string value, both top level and connector level. The connector configs have settings for each connector; connectors are used by table scan and table writer to deal with storage and file formats. These configs and other information, like thread pools, top level memory pool etc. are handed to the Task in a QueryCtx object. See velox/core/QueryCtx.h in the Velox repo for details.

From Plans to Drivers and Operators​

Once the Velox Task is created, a TaskManager hands it splits to work on. This is done with Task::addSplit(), and can be done after the Task has started executing. See velox/exec/Task.h for details.

Let's zoom into what happens at Task creation: A PlanNode tree specifying what the Task does is given to the Task as part of a PlanFragment. The most important step done at Task creation is splitting the plan tree into pipelines. Each pipeline then gets a DriverFactory, which is the factory class that makes Drivers for the pipeline. The Drivers, in their turn, contain the Operators that do the work of running the query. The DriverFactories are made in LocalPlanner.cpp. See LocalPlanner::plan for details.

Following the execution model known as Volcano, the plan is represented by an operator tree where each node consumes the output of its child operators, and returns output to the parent operator. The root node is typically a PartitionedOutputNode or a TableWriteNode. The leaf nodes are either TableScanNode, ExchangeNode or ValuesNode (used for query literals). The full set of Velox PlanNode can be found at velox/core/PlanNode.h.

The PlanNodes mostly correspond to Operators. PlanNodes are not executable as such; they are only a structure describing how to make Drivers and Operators, which do the actual execution. If the tree of nodes has a single branch, then the plan is a single pipeline. If it has nodes with more than one child (input), then the second input of the node becomes a separate pipeline.

Task::start() creates the DriverFactories, which then create the Drivers. To start execution, the Drivers are queued on a thread pool executor. The main function that runs Operators is Driver::runInternal(). See this function for the details of how Operators and the Driver interface: Operator::isBlocked() determines if the Driver can advance. If it cannot, it goes off thread until a future is realized, which then puts it back on the executor.

getOutput() retrieves data from an Operator and addInputs() feeds data into another Operator. The order of execution is to advance the last Operator which can produce output and then feed this to the next Operator. if an Operator cannot produce output, then getOutput() is called on the Operator before it until one is found that can produce data. If no operator is blocked and no Operator can produce output, then the plan is at end. The noMoreInput() method is called on each operator. This can unblock production of results, for example, an OrderBy can only produce its output after it knows that it has all the input.

The Minimal Pipeline: Table Scan and Repartitioning​

Table Scan. With the table scan stage of our sample query, we have one pipeline with two operators: TableScan and PartitionedOutput. Assume this pipeline has five Drivers, and that all these five Drivers go on the thread pool executor. The PartitionedOutput cannot do anything because it has no input. The TableScan::getOutput() is then called. See velox/exec/TableScan.cpp for details. The first action TableScan takes is to look for a Split with task::getSplitOrFuture(). If there is no split available, this returns a future. The Driver will then park itself off thread and install a callback on the future that will reschedule the Driver when a split is available.

It could also be the case that there is no split and the Task has been notified that no more splits will be coming. In this case TableScan would be at the end. Finally, if a Split is available, TableScan interprets it. Given a TableHandle specification provided as part of the plan (list of columns and filters), the Connector (as specified in the Split) makes a DataSource. The DataSource handles the details of IO and file and table formats.

The DataSource is then given the split. After this, DataSource::next() can be called repeatedly to get vectors (batches) of output from the file/section of file specified by the Split. If the DataSource is at the end, TableScan looks for the next split. See Connector.h for the Connector and DataSource interfaces.

Repartitioning. Now we have traced execution up to the TableScan returning its first batch of output. The Driver feeds this to PartitionedOutput::addInput(). See PartitionedOutput.cpp for details. PartitionedOutput first calculates a hash on the partitioning key, in this case, l_partkey, producing a destination number for each row in the batch RowVectorPtr input_.

Each destination has a partly filled serialization buffer (VectorStreamGroup) for each destination worker. If there are 100 Tasks in the second stage, each PartitionedOutput has 100 destinations, each with one VectorStreamGroup. The main function of a VectorStreamGroup is append(), which takes a RowVectorPtr and a set of row numbers in it. It serializes each value identified by the row numbers and adds it to the partly formed serialization. When enough rows are accumulated in the VectorStreamGroup, it produces a SerializedPage. See flush() in PartitionedOutput.cpp.

The SerializedPage is a self contained serialized packet of information that can be transmitted over the wire to the next stage. Each such page only contains rows intended for the same recipient. These pages are then queued up in the worker process' OutputBufferManager. Note the code with BlockingReason in flush(). The buffer manager maintains separate queues of all consumers of all Tasks. If a queue is full, adding output may block. This returns a future that is realized when there is space to add more data to the queue. This depends on when the Task's consumer Task fetches the data.

Shuffles in Prestissimo are implemented by PartitionedOutput at the producer and Exchange at the consumer end. The OutputBufferManager keeps ready serialized data for consumers to pick up. The binding of these to the Presto wire protocols is in TaskManager.cpp for the producer side, and in PrestoExchangeSource.cpp for the consumer side.

Recap​

We presented how a plan becomes executable and how data moves between and inside Operators. We discussed that a Driver can block (go off thread) to wait for a Split to become available, or to wait for its output to be consumed. We have now scratched the surface of running a leaf stage of a distributed query. There is much more to Operators and vectors, though. In the next installment of Velox Primer, we will look at what the second stage of our minimal sample query does.

Distributed Query Execution​

Velox is a library that provides the functions that go into a query fragment in a distributed compute engine. Distributed compute engines, like Presto and Spark run so-called exchange parallel plans. Exchange is also known as a data shuffle, and allows data to flow from one stage to the next. Query fragments are the things connected by shuffles, and comprise the processing that is executed within a single worker node. A shuffle takes input from a set of fragments and routes rows of input to particular consumers based on a characteristic of the data, i.e., a partitioning key. The consumers of the shuffle read from the shuffle and get rows of data from whatever producer, such that the partitioning key of these rows matches the consumer.

Take the following query as an example:

SELECT key, count(*) FROM T GROUP BY key

Suppose it has n leaf fragments that scan different parts of T (the green circles at stage 1). At the end of the leaf fragment, assume there is a shuffle that shuffles the rows based on key. There are m consumer fragments (the yellow circles at stage 2), where each gets a non-overlapping selection of rows based on column key. Each consumer then constructs a hash table keyed on key, where they store a count of how many times the specific value of key has been seen.

Now, if there are 100B different values of key, the hash table would not conveniently fit on a single machine. For efficiency, there is a point in dividing this into 100 hash tables of 1B entries each. This is the point of exchange parallel scale-out. Think of shuffle as a way for consumers to each consume their own distinct slice of a large stream produced by multiple workers.

Distributed query engines like Presto and Spark both fit the above description. The difference is in, among other things, how they do the shuffle, but we will get back to that later.

Tasks and Splits​

Within a worker node, the Velox representation of a query fragment is called a Task (velox::exec::Task). A task is informed by mainly two things: 

• Plan - velox::core::PlanNode: specifies what the Task does.
• Splits - velox::exec::Split: specifies the data the Task operates on.

The Splits correspond to the plan that the Task is executing. For the first stage Tasks (table scanning), the Splits specify pieces of files to scan. For the second stage Tasks (group by), their Splits identify the table scan Tasks from which the group by reads its input. There are file splits (velox::connector::ConnectorSplit) and remote splits (velox::exec::RemoteConnectorSplit). The first identifies data to read, the second identifies a running Task.

The distributed engine makes PlanNodes and Splits. Velox takes these and makes Tasks. Tasks send back statistics, errors and other status information to the distributed engine. 

Pipelines, Drivers and Operators​

Inside a Task, there are Pipelines. Each pipeline is a linear sequence of operators (velox::exec::Operator), and operators are the objects that implement relational logic. In the case of the group by example, the first task has one pipeline, with a TableScan (velox::exec::TableScan) and a PartitionedOutput (velox::exec::PartitionedOutput). The second Task too has one pipeline, with an Exchange, a LocalExchange, HashAggregation, and a PartitionedOutput.

Each pipeline has one or more Drivers. A Driver is a container for one linear sequence of Operators, and typically runs on its own thread. The pipeline is the collection of Drivers with the same sequence of Operators. The individual Operator instances belong to each Driver. The Drivers belong to the Task. These are interlinked with smart pointers such that they are kept alive for as long as needed.

An example of a Task with two pipelines is a hash join, with separate pipelines for the build and for the probe side. This makes sense because the build must be complete before the probe can proceed. We will talk more about this later.

The Operators on each Driver communicate with each other through the Driver. The Driver picks output from one Operator, keeps tabs on stats and passes it to the next Operator. The data passed between Operators consists of vectors. In particular, an Operator produces/consumes a RowVector, which is a vector with a child vector for every column of the relation - it is the equivalent of RecordBatch in Arrow. All vectors are subclasses of velox::BaseVector.

Operator Sources, Sinks and State​

The first Operator in a Driver is called a source. The source operator takes Splits to figure the file/remote Task that provides its data, and produces a sequence of RowVectors. The last operator in the pipeline is called a sink. The sink operator does not produce an output RowVector, but rather puts the data somewhere for a consumer to retrieve. This is typically a PartitionedOutput. The consumer of the PartitionedOutput is an Exchange in a different task, where the Exchange is in the source position. Operators that are neither sources or sinks are things like FilterProject, HashProbe, HashAggregation and so on, more on these later.

Operators also contain a state. They can be blocked, accepting input, have output to produce or not, or may be notified that no more input is coming. Operators do not call the API functions of other Operators directly, but instead the Driver decides which Operator to advance next. This has the benefit that no part of the Driver's state is captured in nested function calls. The Driver has a flat stack and therefore can go on and off thread at any time, without having to unwind and restore nested function frames. 

Recap​

We have seen how a distributed query consists of fragments joined by shuffles. Each fragment has a Task, which has pipelines, Drivers and Operators. PlanNodes represent what the task is supposed to do. These tell the Task how to set up Drivers and Operators. Splits tell the Task where to find input, e.g. from files or from the other Tasks. A Driver corresponds to a thread of execution. It may be running or blocked, e.g. waiting for data to become available or for its consumer to consume already produced data. Operators pass data between themselves as vectors. 

In the next article we will discuss the different stages in the life of a query.

The query trace tool helps analyze and debug query performance and correctness issues. It helps prevent interference from external noise in a production environment (such as storage, network, etc.) by allowing replay of a part of the query plan and dataset in an isolated environment, such as a local machine. This is much more efficient for query performance analysis and issue debugging, as it eliminates the need to replay the whole query in a production environment.

How Tracing Works​

The tracing process consists of two distinct phases: the tracing phase and the replaying phase. The tracing phase is executed within a production environment, while the replaying phase is conducted in a local development environment.

Tracing Phase

1. Trace replay required metadata, including the query plan fragment, query configuration, and connector properties, is recorded during the query task initiation.
2. Throughout query processing, each traced operator logs the input vectors or splits storing them in a designated storage location.
3. The metadata and splits are serialized in JSON format, and the operator data inputs are serialized using a presto serializer.

Replaying Phase

1. Read and deserialize the recorded query plan, extract the traced plan node, and assemble a plan fragment with customized source and sink nodes.
2. The source node reads the input from the serialized operator inputs on storage and the sink operator prints or logs out the execution stats.
3. Build a task with the assembled plan fragment in step 1. Apply the recorded query configuration and connector properties to replay the task with the same input and configuration setup as in production.

NOTE: The presto serialization might lose input vector encoding, such as lazy vector and nested dictionary encoding, which affects the operator’s execution. Hence, it might not always be the same as in production.

Tracing Framework​

Trace Writers​

There are three types of writers: TaskTraceMetadataWriter, OperatorTraceInputWriter, and OperatorTraceSplitWriter. They are used in the prod or shadow environment to record the real execution data.

• The TaskTraceMetadataWriter records the query metadata during task creation, serializes it, and saves it into a file in JSON format.
• The OperatorTraceInputWriter records the input vectors from the target operator, it uses a Presto serializer to serialize each vector batch and flush immediately to ensure that replay is possible even if a crash occurs during execution.
• The OperatorTraceSplitWriter captures the input splits from the target TableScan operator. It serializes each split and immediately flushes it to ensure that replay is possible even if a crash occurs during execution.

Storage Location​

It is recommended to store traced data in a remote storage system to ensure its preservation and accessibility even if the computation clusters are reconfigured or encounter issues. This also helps prevent nodes in the cluster from failing due to local disk exhaustion.

Users should start by creating a root directory. Writers will then create subdirectories within this root directory to organize the traced data. A well-designed directory structure will keep the data organized and accessible for replay and analysis.

Metadata Location

The TaskTraceMetadataWriter is set up during the task creation so it creates a trace directory named $rootDir/$queryId/$taskId.

Input Data and Split Location

The node ID consolidates the tracing for the same tracing plan node. The pipeline ID isolates the tracing data between operators created from the same plan node (e.g., HashProbe and HashBuild from the HashJoinNode). The driver ID isolates the tracing data of peer operators in the same pipeline from different drivers.

Correspondingly, to ensure the organized and isolated tracing data storage, the OperatorTraceInputWriter and OpeartorTraceSplitWriter are set up during the operator initialization and create a data or split tracing directory in

$rootDir/$queryId$taskId/$nodeId/$pipelineId/$driverId

Memory Management​

Add a new leaf system pool named tracePool for tracing memory usage, and expose it like memory::MemoryManager::getInstance()->tracePool().

Trace Readers​

Three types of readers correspond to the query trace writers: TaskTraceMetadataReader, OperatorTraceInputReader, and OperatorTraceSplitReader. The replayers typically use them in the local environment, which will be described in detail in the Query Trace Replayer section.

• The TaskTraceMetadataReader can load the query metadata JSON file and extract the query configurations, connector properties, and a plan fragment. The replayer uses these to build a replay task.
• The OperatorTraceInputReader reads and deserializes the input vectors in a tracing data file. It is created and used by a QueryTraceScan operator which will be described in detail in the Query Trace Scan section.
• The OperatorTraceSplitReader reads and deserializes the input splits in tracing split info files, and produces a list of exec::Split for the query replay.

Trace Scan​

As outlined in the How Tracing Works section, replaying a non-leaf operator requires a specialized source operator. This operator is responsible for reading data records during the tracing phase and integrating with Velox’s LocalPlanner with a customized plan node and operator translator.

TraceScanNode

We introduce a customized ‘TraceScanNode’ to replay a non-leaf operator. This node acts as the source node and creates a specialized scan operator, known as OperatorTraceScan with one per driver during the replay. The TraceScanNode contains the trace directory for the designated trace node, the pipeline ID associated with it, and a driver ID list passed during the replaying by users so that the OperatorTraceScan can locate the right trace input data or split directory.

OperatorTraceScan

As described in the Storage Location section, a plan node may be split into multiple pipelines, each pipeline can be divided into multiple operators. Each operator corresponds to a driver, which is a thread of execution. There may be multiple tracing data files for a single plan node, one file per driver.

Query Trace Replayer​

The query trace replayer is typically used in the local environment and works as follows:

1. Load traced query configurations, connector properties, and a plan fragment.
2. Extract the target plan node from the plan fragment using the specified plan node ID.
3. Use the target plan node in step 2 to create a replay plan node. Create a replay plan.
4. If the target plan node is a TableScanNode, add the replay plan node to the replay plan as the source node. Get all the traced splits using OperatorInputSplitReader. Use the splits as inputs for task replaying.
5. For a non-leaf operator, add a QueryTraceScanNode as the source node to the replay plan and then add the replay plan node.
6. Add a sink node, apply the query configurations (disable tracing), and connector properties, and execute the replay plan.

Detail usage please see the tracing doc in https://facebookincubator.github.io/velox/develop/debugging/tracing.html

Future Work​

• Add support for more operators
• Customize the replay task execution instead of AssertQueryBuilder
• Supports IcebergHiveConnector
• Add trace replay for an entire pipeline

TL;DR​

In late 2023, the Meta OSS (Open Source Software) Team requested all Meta teams to move the CI deployments from CircleCI to Github Actions. Voltron Data and Meta in collaboration migrated all the deployed Velox CI jobs. For the year 2024, Velox CI spend was on track to overshoot the allocated resources by a considerable amount of money. As part of this migration effort, the CI workloads were consolidated and optimized by Q2 2024, bringing down the projected 2024 CI spend by 51%.

Velox’s Continuous Integration Workload​

Continuous Integration (CI) is crucial for Velox’s success as an open source project as it helps protect from bugs and errors, reduces likelihood of conflicts and leads to increased community trust in the project. This is to ensure the Velox builds works well on a myriad of system architectures, operating systems and compilers - along with the ones used internally at Meta. The OSS build version of Velox also supports additional features that aren't used internally in Meta (for example, support for Cloud blob stores, etc.).

When a pull request is submitted to Velox, the following jobs are executed:

1. Linting and Formatting workflows:
   1. Header checks
   2. License checks
   3. Basic Linters
2. Ensure Velox builds on various platforms
   1. MacOS (Intel, M1)
   2. Linux (Ubuntu/Centos)
3. Ensure Velox builds under its various configurations
   1. Debug / Release builds
   2. Build default Velox build
   3. Build Velox with support for Parquet, Arrow and External Adapters (S3/HDFS/GCS etc.)
   4. PyVelox builds
4. Run prerequisite tests
   1. Unit Tests
   2. Benchmarking Tests
      1. Conbench is used to store and compare results, and also alert users on regressions
   3. Various Fuzzer Tests (Expression / Aggregation/ Exchange / Join etc)
   4. Signature Check and Biased Fuzzer Tests ( Expression / Aggregation)
   5. Fuzzer Tests using Presto as source of truth
5. Docker Image build jobs
   1. If an underlying dependency is changed, a new Docker CI image is built for
      1. Ubuntu Linux
      2. Centos
      3. Presto Linux image
6. Documentation build and publish Job
   1. If underlying documentation is changed, Velox documentation pages are rebuilt and published
   2. Netlify is used for publishing Velox web pages

Velox CI Optimization​

Previous implementation of CI in CircleCI grew organically and was unoptimized, resulting in long build times, and also significantly costlier. This opportunity to migrate to Github Actions helped to take a holistic view of CI deployments and actively optimized to reduce build times and CI spend. Note however, that there has been continued investment in reducing test times to further improve Velox reliability, stability and developer experience. Some of the optimizations completed are:

1. Persisting build artifacts across builds: During every build, the object files and binaries produced are cached. In addition to this, artifacts such as scalar function signatures and aggregate function signatures are produced. These signatures are used to compare with the baseline version, by comparing against the changes in the current PR to determine if the current changes are backwards incompatible or bias the newly added changes. Using a stash to persist these artifacts helps save one build cycle.

2. Optimizing our Instances: Building Velox is Memory and CPU intensive job. Some beefy instances (16-core machines) are used to build Velox. After the build, the build artifacts are copied to smaller instances (4 core machines) to run fuzzer tests and other jobs. Since these fuzzers often run for an hour and are less intensive than the build process, it resulted in significant CI savings while increasing the test coverage.

Instrumenting Velox CI Builds​

Velox CI builds were instrumented in Conbench so that it can capture various metrics about the builds:

1. Build times at translation unit / library/ project level.
2. Binary sizes produced at TLU/ .a,.so / executable level.
3. Memory pressure
4. Measure across time how our changes affect binary sizes

A nightly job is run to capture these build metrics and it is uploaded to Conbench. Velox build metrics report is available here: Velox Build Metrics Report

Acknowledgements​

A large part of the credit goes to Jacob Wujciak and the team at Voltron Data. We would also like to thank other collaborators in the Open Source Community and at Meta, including but not limited to:

Meta: Sridhar Anumandla, Pedro Eugenio Rocha Pedreira, Deepak Majeti, Meta OSS Team, and others

Voltron Data: Jacob Wujciak, Austin Dickey, Marcus Hanwell, Sri Nadukudy, and others

Queries that use TRY or TRY_CAST may experience poor performance and high CPU usage due to excessive exception throwing. We optimized CAST to indicate failure without throwing and introduced a mechanism for scalar functions to do the same. Microbenchmark measuring worst case performance of CAST improved 100x. Samples of production queries show 30x cpu time improvement.

TRY and TRY(CAST)​

TRY construct can be applied to any expression to suppress errors and turn them into NULL results. TRY_CAST is a version of CAST that suppresses errors and returns NULL instead.

For example, parse_datetime('2024-05-', 'YYYY-MM-DD') fails:

	 Invalid format: "2024-05-" is too short

, but TRY(parse_datetime('2024-05-', 'YYYY-MM-DD')) succeeds and returns NULL.

Similarly, CAST('foo' AS INTEGER) fails:

	Cannot cast 'foo' to INT

, but TRY_CAST('foo' AS INTEGER) succeeds and returns NULL.

TRY_CAST vs. TRY(CAST)​

TRY can wrap any expression, so one can wrap CAST as well:

  TRY(CAST('foo' AS INTEGER))

Wrapping CAST in TRY is similar to TRY_CAST, but not equivalent. TRY_CAST suppresses only cast errors, while TRY suppresses any error in the expression tree.

For example, CAST(1/0 AS VARCHAR) fails:

  Division by zero

, TRY_CAST(1/0 AS VARCHAR) also fails:

  Division by zero

, but TRY(CAST(1/0 AS VARCHAR)) succeeds and returns NULL.

In this case, the error is generated by division operation (1/0). TRY_CAST cannot suppress that error, but TRY can. More generally, TRY(CAST(...)) suppresses all errors in all expressions that are evaluated to produce an input for CAST as well as errors in CAST itself, but TRY_CAST suppresses errors in CAST only.

What happens when many rows fail?​

In most cases only a fraction of rows generates an error. However, there are queries where a large percentage of rows fail. In these cases, a lot of CPU time goes into handling exceptions.

For example, one Prestissimo query used 3 weeks of CPU time, 93% of which was spent processing try(date_parse(...)) expressions where most rows failed. Here is a profile for that query that shows that all the time went into stack unwinding:

This query processes 14B rows, ~70% of which fail in date_parse(...) function due to the date string being empty.

    presto> select try(date_parse('', '%Y-%m-%d'));
     _col0
    -------
     NULL
    (1 row)


    – TRY suppressed Invalid format: "" error and produced a NULL.

Velox tracks the number of suppressed exceptions per operator / plan node and reports these as numSilentThrow runtime stat. For this query, Velox reported 21B throws for a single FilterProject node that processed 14B rows. Before the optimizations, each failing row used to throw twice. An earlier blog post from Laith Sakka explains why. After the optimizations this query’s CPU time dropped to 17h: 30x difference from the original cpu time. Compared to Presto Java, this query uses 4x less cpu time (originally it used 6x more).

We observed similar issues with queries that use other functions that parse strings as well as casts from strings.

Solution​

To avoid the performance penalty of throwing exceptions we need to report errors differently. Google’s Abseil library uses absl::Status to return errors from void functions and absl::StatusOr to return value or error from non-void functions. Arrow library has similar Status and Result. Our own Folly has folly::Expected. Inspired by these examples we introduced velox::Status and velox::Expected.

velox::Status holds a generic error code and an error message.

velox::Expected<T> is a typedef for folly::Expected<T, velox::Status>.

For example, a non-throwing modulo operation can be implemented like this:

  Expected<int> mod(int a, int b) {
    if (b == 0) {
      return folly::makeUnexpected(Status::UserError(“Division by zero”));
    }

    return a % b;
  }

Non-throwing Simple Functions​

We extended the Simple Function API to allow authoring non-throwing scalar functions. The function author can now define a ‘call’ method that returns Status. Such a function can indicate an error by returning a non-OK status.

  Status call(result&, arg1, arg2,..)

These functions are still allowed to throw and exceptions will be handled properly, but not throwing improves performance of expressions that use TRY.

Modulo SQL function would look like this:

    template <typename TExec>
    struct NoThrowModFunction {
      VELOX_DEFINE_FUNCTION_TYPES(TExec);

      Status call(int64_t& result, const int64_t& a, const int64_t& b) {
        if (b == 0) {
          return Status::UserError("Division by zero");
        }

        result = a % b;
        return Status::OK();
      }
    };

We changed date_parse, parse_datetime, and from_iso8601_date Presto functions to use the new API and report errors without throwing.

Non-throwing Vector functions​

Vector functions can implement non-throwing behavior by leveraging the new EvalCtx::setStatus(row, status) API. However, nowadays we expect virtually all functions to be written using Simple Function API.

Non-throwing CAST​

CAST is complex. A single name refers to multiple dozen individual operations. The full matrix of supported conversions is available in the Velox documentation. Not all casts throw. For example, cast from an integer to a string does not throw. However, casts from strings may fail in multiple ways. A common failure scenario is cast from an empty string. Laith Sakka optimized this use case earlier.

> select cast('' as integer);
Cannot cast '' to INT

However, we are also seeing failures in casting non-empty strings and NaN floating point values to integers.

> select cast(nan() as bigint);
Unable to cast NaN to bigint

> select cast('123x' as integer);
Cannot cast '123x' to INT

CAST from string to integer and floating point value is implemented using folly::to template. Luckily there is a non-throwing version: folly::tryTo. We changed our CAST implementation to use folly::tryTo to avoid throwing. Not throwing helped improve performance of TRY_CAST by 20x.

Still, the profile showed that there is room for further improvement.

Do not produce or store error messages under TRY​

After switching to non-throwing implementation, the profile showed that half the cpu time went into folly::makeConversionError. folly::tryTo returns result or ConversionCode enum. CAST uses folly::makeConversionError to convert ConversionCode into a user-friendly error message. This involves allocating and populating a string for the error message, copying it into the std::range_error object, then copying it again into Status. This error message is very helpful if it is being propagated all the way to the user, but it is not needed if the error is suppressed via TRY or TRY_CAST.

To solve this problem we introduced a thread-local flag, threadSkipErrorDetails, that indicates whether Status needs to include a detailed error message or not. By default, this flag is ‘false’, but TRY and TRY_CAST set it to ‘true’. CAST logic checks this flag to decide whether to call folly::makeConversionError or not. This change gives a 3x performance boost to TRY_CAST and 2x to TRY.

    if (threadSkipErrorDetails()) {
      return folly::makeUnexpected(Status::UserError());
    }

    return folly::makeUnexpected(Status::UserError(
        "{}", folly::makeConversionError(result.error(), "").what()));

After this optimization, we observed that TRY(CAST(...)) is up to 5x slower than TRY_CAST when many rows fail.

The profile revealed that 30% of cpu time went to EvalCtx::ensureErrorsVectorSize. For every row that fails, we call EvalCtx::ensureErrorsVectorSize to resize the error vector to accommodate that row. When many rows fail we end up resizing a lot: resize(1), resize(2), resize(3),...resize(n). We fixed this by pre-allocating the error vector in the TRY expression.

Another 30% of cpu time went into managing reference counts for std::shared_ptr<std::exception_ptr> stored in the errors vector. We do not need error details for TRY, hence, no need to store these values. We fixed this by making error values in error vector optional and updating EvalCtx::setStatus to skip writing these under TRY.

After all these optimizations, the microbenchmark that measures performance of casting invalid strings into integers showed 100x improvement. The benchmark evaluates 4 expressions:

• TRY_CAST(‘’ AS INTEGER)
• TRY(CAST(‘’ AS INTEGER))
• TRY_CAST(‘$’ AS INTEGER)
• TRY(CAST(‘$’ AS INTEGER))

When we started, the benchmark results were:

===============================================================
[...]hmarks/ExpressionBenchmarkBuilder.cpp     relative  time/iter   iters/s
================================================================
cast##try_cast_invalid_empty_input                          2.40ms    417.47
cast##tryexpr_cast_invalid_empty_input                    402.63ms      2.48
cast##try_cast_invalid_nan                                392.14ms      2.55
cast##tryexpr_cast_invalid_nan                            827.09ms      1.21

At the end the numbers improved 100x:

cast##try_cast_invalid_empty_input                          2.16ms    463.62
cast##tryexpr_cast_invalid_empty_input                      4.29ms    232.95
cast##try_cast_invalid_nan                                  5.47ms    182.83
cast##tryexpr_cast_invalid_nan                              7.76ms    128.81

Note: The performance of TRY_CAST(‘’ AS INTEGER) hasn’t changed because this particular use case has been optimized by Laith Sakka earlier.

Next steps​

We can identify queries with a high percentage of numSilentThrow rows and change throwing functions to not throw.

For simple functions this involves changing the ‘call’ method to return Status and replacing ‘throw’ statements with return Status::UserError(...). You get extra points for producing error messages conditionally based on thread-local flag threadSkipErrorDetails().

template <typename TExec>
struct NoThrowModFunction {
  VELOX_DEFINE_FUNCTION_TYPES(TExec);

  Status call(int64_t& result, const int64_t& a, const int64_t& b) {
    if (b == 0) {
      If (threadSkipErrorDetails()) {
          return Status::UserError();
      }
      return Status::UserError("Division by zero");
    }

    result = a % b;
    return Status::OK();
  }
};

We are changing CAST(varchar AS date) to not throw.

We provided a non-throwing ‘call’ API for simple functions that never return a NULL for a non-NULL input. This covers the majority of Presto functions. For completeness, we would want to provide non-throwing ‘call’ APIs for all other use cases:

• bool call() for returning NULL sometimes
• callAscii for processing all-ASCII inputs
• callNullable for processing possibly NULL inputs
• callNullFree for processing complex inputs with all NULLs removed.

Acknowledgements​

Thank you Laith Sakka for doing the initial work to investigate and optimize TRY_CAST for empty strings and sharing your findings in a blog post.

Thank you Orri Erling for adding numSilentThrow runtime stat to report number of suppressed exceptions.

Thank you Pedro Eugenio Rocha Pedreira for introducing the velox::Status class.

Thank you Bikramjeet Vig, Jimmy Lu, Orri Erling, Pedro Eugenio Rocha Pedreira and Xiaoxuan Meng for brainstorming and helping with code reviews.

LIKE is a very useful SQL operator. It is used to do string pattern matching. The following examples for LIKE usage are from the Presto doc:

SELECT * FROM (VALUES ('abc'), ('bcd'), ('cde')) AS t (name)
WHERE name LIKE '%b%'
--returns 'abc' and  'bcd'

SELECT * FROM (VALUES ('abc'), ('bcd'), ('cde')) AS t (name)
WHERE name LIKE '_b%'
--returns 'abc'

SELECT * FROM (VALUES ('a_c'), ('_cd'), ('cde')) AS t (name)
WHERE name LIKE '%#_%' ESCAPE '#'
--returns 'a_c' and  '_cd'

These examples show the basic usage of LIKE:

• Use % to match zero or more characters.
• Use _ to match exactly one character.
• If we need to match % and _ literally, we can specify an escape char to escape them.

When we use Velox as the backend to evaluate Presto's query, LIKE operation is translated into Velox's function call, e.g. name LIKE '%b%' is translated to like(name, '%b%'). Internally Velox converts the pattern string into a regular expression and then uses regular expression library RE2 to do the pattern matching. RE2 is a very good regular expression library. It is fast and safe, which gives Velox LIKE function a good performance. But some popularly used simple patterns can be optimized using direct simple C++ string functions instead of regex. e.g. Pattern hello% matches inputs that start with hello, which can be implemented by direct memory comparison of prefix ('hello' in this case) bytes of input:

// Match the first 'length' characters of string 'input' and prefix pattern.
bool matchPrefixPattern(
    StringView input,
    const std::string& pattern,
    size_t length) {
  return input.size() >= length &&
      std::memcmp(input.data(), pattern.data(), length) == 0;
}

It is much faster than using RE2. Benchmark shows it gives us a 750x speedup. We can do similar optimizations for some other patterns:

• %hello: matches inputs that end with hello. It can be optimized by direct memory comparison of suffix bytes of the inputs.
• %hello%: matches inputs that contain hello. It can be optimized by using std::string_view::find to check whether inputs contain hello.

These simple patterns are straightforward to optimize. There are some more relaxed patterns that are not so straightforward:

• hello_velox%: matches inputs that start with 'hello', followed by any character, then followed by 'velox'.
• %hello_velox: matches inputs that end with 'hello', followed by any character, then followed by 'velox'.
• %hello_velox%: matches inputs that contain both 'hello' and 'velox', and there is a single character separating them.

Although these patterns look similar to previous ones, but they are not so straightforward to optimize, _ here matches any single character, we can not simply use memory comparison to do the matching. And if user's input is not pure ASCII, _ might match more than one byte which makes the implementation even more complex. Also note that the above patterns are just for illustrative purpose. Actual patterns can be more complex. e.g. h_e_l_l_o, so trivial algorithm will not work.

Optimizing Relaxed Patterns​

We optimized these patterns as follows. First, we split the patterns into a list of sub patterns, e.g. hello_velox% is split into sub-patterns: hello, _, velox, %, because there is a % at the end, we determine it as a kRelaxedPrefix pattern, which means we need to do some prefix matching, but it is not a trivial prefix matching, we need to match three sub-patterns:

• kLiteralString: hello
• kSingleCharWildcard: _
• kLiteralString: velox

For kLiteralString we simply do a memory comparison:

if (subPattern.kind == SubPatternKind::kLiteralString &&
    std::memcmp(
        input.data() + start + subPattern.start,
        patternMetadata.fixedPattern().data() + subPattern.start,
        subPattern.length) != 0) {
  return false;
}

Note that since it is a memory comparison, it handles both pure ASCII inputs and inputs that contain Unicode characters.

Matching _ is more complex considering that there are variable length multi-bytes character in unicode inputs. Fortunately there are existing libraries which provides unicode related operations: utf8proc. It provides functions that tells us whether a byte in input is the start of a character or not, how many bytes current character consists of etc. So to match a sequence of _ our algorithm is:

if (subPattern.kind == SubPatternKind::kSingleCharWildcard) {
  // Match every single char wildcard.
  for (auto i = 0; i < subPattern.length; i++) {
    if (cursor >= input.size()) {
      return false;
    }

    auto numBytes = unicodeCharLength(input.data() + cursor);
    cursor += numBytes;
  }
}

Here:

• cursor is the index in the input we are trying to match.
• unicodeCharLength is a function which wraps utf8proc function to determine how many bytes current character consists of.

So the logic is basically repeatedly calculate size of current character and skip it.

It seems not that complex, but we should note that this logic is not effective for pure ASCII input. Every character is one byte in pure ASCII input. So to match a sequence of _, we don't need to calculate the size of each character and compare in a for-loop. In fact, we don't need to explicitly match _ for pure ASCII input as well. We can use the following logic instead:

for (const auto& subPattern : patternMetadata.subPatterns()) {
    if (subPattern.kind == SubPatternKind::kLiteralString &&
        std::memcmp(
            input.data() + start + subPattern.start,
            patternMetadata.fixedPattern().data() + subPattern.start,
            subPattern.length) != 0) {
      return false;
    }
}

It only matches the kLiteralString pattern at the right position of the inputs, _ is automatically matched(actually skipped). No need to match it explicitly. With this optimization we get 40x speedup for kRelaxedPrefix patterns, 100x speedup for kRelaxedSuffix patterns.

Thank you Maria Basmanova for spending a lot of time reviewing the code.

Reduce_agg is the only lambda aggregate Presto function. It allows users to define arbitrary aggregation logic using 2 lambda functions.

reduce_agg(inputValue T, initialState S, inputFunction(S, T, S), combineFunction(S, S, S)) → S

Reduces all non-NULL input values into a single value. inputFunction will be invoked for
each non-NULL input value. If all inputs are NULL, the result is NULL. In addition to taking
the input value, inputFunction takes the current state, initially initialState, and returns the
new state. combineFunction will be invoked to combine two states into a new state. The final
state is returned. Throws an error if initialState is NULL or inputFunction or combineFunction
returns a NULL.

Once can think of reduce_agg as using inputFunction to implement partial aggregation and combineFunction to implement final aggregation. Partial aggregation processes a list of input values and produces an intermediate state:

auto s = initialState;
for (auto x : input) {
   s = inputFunction(s, x);
}

return s;

Final aggregation processes a list of intermediate states and computes the final state.

auto s = intermediates[0];
for (auto i = 1; i < intermediates.size(); ++i)
   s = combineFunction(s, intermediates[i]);
}

return s;

For example, one can implement SUM aggregation using reduce_agg as follows:

reduce_agg(c, 0, (s, x) -> s + x, (s, s2) -> s + s2)

Implementation of AVG aggregation is a bit trickier. For AVG, state is a tuple of sum and count. Hence, reduce_agg can be used to compute (sum, count) pair, but it cannot compute the actual average. One needs to apply a scalar function on top of reduce_agg to get the average.

SELECT id, sum_and_count.sum / sum_and_count.count FROM (
  SELECT id, reduce_agg(value, CAST(row(0, 0) AS row(sum double, count bigint)),
    (s, x) -> CAST(row(s.sum + x, s.count + 1) AS row(sum double, count bigint)),
    (s, s2) -> CAST(row(s.sum + s2.sum, s.count + s2.count) AS row(sum double, count bigint))) AS sum_and_count
  FROM t
  GROUP BY id
);

The examples of using reduce_agg to compute SUM and AVG are for illustrative purposes. One should not use reduce_agg if a specialized aggregation function is available.

One use case for reduce_agg we see in production is to compute a product of input values.

reduce_agg(c, 1.0, (s, x) -> s * x, (s, s2) -> s * s2)

Another example is to compute a list of top N distinct values from all input arrays.

reduce_agg(x, array[],
            (a, b) -> slice(reverse(array_sort(array_distinct(concat(a, b)))), 1, 1000),
            (a, b) -> slice(reverse(array_sort(array_distinct(concat(a, b)))), 1, 1000))

Note that this is equivalent to the following query:

SELECT array_agg(v) FROM (
    SELECT DISTINCT v
    FROM t, UNNEST(x) AS u(v)
    ORDER BY v DESC
    LIMIT 1000
)

Implementation​

Efficient implementation of reduce_agg lambda function is not straightforward. Let’s consider the logic for partial aggregation.

auto s = initialState;
for (auto x : input) {
   s = inputFunction(s, x);
}

This is a data-dependent loop, i.e. the next loop iteration depends on the results of the previous iteration. inputFunction needs to be invoked on each input value x separately. Since inputFunction is a user-defined lambda, invoking inputFunction means evaluating an expression. And since expression evaluation in Velox is optimized for processing large batches of values at a time, evaluating expressions on one value at a time is very inefficient. To optimize the implementation of reduce_agg we need to reduce the number of times we evaluate user-defined lambdas and increase the number of values we process each time.

One approach is to

1. convert all input values into states by evaluating inputFunction(initialState, x);
2. split states into pairs and evaluate combineFunction on all pairs;
3. repeat step (2) until we have only one state left.

Let’s say we have 1024 values to process. Step 1 evaluates inputFunction expression on 1024 values at once. Step 2 evaluates combineFunction on 512 pairs, then on 256 pairs, then on 128 pairs, 64, 32, 16, 8, 4, 2, finally producing a single state. Step 2 evaluates combineFunction 9 times. In total, this implementation evaluates user-defined expressions 10 times on multiple values each time. This is a lot more efficient than the original implementation that evaluates user-defined expressions 1024 times.

In general, given N inputs, the original implementation evaluates expressions N times while the new one log2(N) times.

Note that in case when N is not a power of two, splitting states into pairs may leave an extra state. For example, splitting 11 states produces 5 pairs + one extra state. In this case, we set aside the extra state, evaluate combineFunction on 5 pairs, then bring extra state back to a total of 6 states and continue.

A benchmark, velox/functions/prestosql/aggregates/benchmarks/ReduceAgg.cpp, shows that initial implementation of reduce_agg is 60x slower than SUM, while the optimized implementation is only 3x slower. A specialized aggregation function will always be more efficient than generic reduce_agg, hence, reduce_agg should be used only when specialized aggregation function is not available.

Finally, a side effect of the optimized implementation is that it doesn't support applying reduce_agg to sorted inputs. I.e. one cannot use reduce_agg to compute an equivalent of

	SELECT a, array_agg(b ORDER BY b) FROM t GROUP BY 1

The array_agg computation depends on order of inputs. A comparable implementation using reduce_agg would look like this:

	SELECT a,
        reduce_agg(b, array[],
                    (s, x) -> concat(s, array[x]),
                    (s, s2) -> concat(s, s2)
                    ORDER BY b)
    FROM t GROUP BY 1

To respect ORDER BY b, the reduce_agg would have to apply inputFunction to each input value one at a time using a data-dependent loop from above. As we saw, this is very expensive. The optimization we apply does not preserve the order of inputs, hence, cannot support the query above. Note that Presto doesn't support applying reduce_agg to sorted inputs either.

Thank you Orri Erling for brainstorming and Xiaoxuan Meng and Pedro Eugênio Rocha Pedreira for reviewing the code.