GH-1234: Release serialization buffers when compression fails - #1290
Merged
jbonofre merged 2 commits intoOct 5, 2026
Merged
Conversation
pj-workspace
requested review from
jbonofre,
laurentgo,
lidavidm and
wgtmac
as code owners
September 10, 2026 03:38
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
Comment on lines
+38
to
+55
| @ParameterizedTest | ||
| @ValueSource(longs = {0, 16}) | ||
| void compressionAllocationFailureReleasesBuffers(long availableBytes) { | ||
| try (BufferAllocator allocator = new RootAllocator(); | ||
| IntVector vector = new IntVector("values", allocator); | ||
| VectorSchemaRoot root = VectorSchemaRoot.of(vector)) { | ||
| vector.allocateNew(1); | ||
| vector.set(0, 42); | ||
| root.setRowCount(1); | ||
| long allocatedBefore = allocator.getAllocatedMemory(); | ||
| int referencesBefore = vector.getDataBuffer().getReferenceManager().getRefCount(); | ||
| allocator.setLimit(allocatedBefore + availableBytes); | ||
| VectorUnloader unloader = new VectorUnloader(root, true, new CopyCodec(), true); | ||
|
|
||
| assertThrows(OutOfMemoryException.class, unloader::getRecordBatch); | ||
| assertEquals(allocatedBefore, allocator.getAllocatedMemory()); | ||
| assertEquals(referencesBefore, vector.getDataBuffer().getReferenceManager().getRefCount()); | ||
| assertEquals(42, vector.get(0)); |
Comment on lines
+145
to
+148
| @Override | ||
| public CompressionUtil.CodecType getCodecType() { | ||
| return CompressionUtil.CodecType.LZ4_FRAME; | ||
| } |
jbonofre
approved these changes
Oct 5, 2026
jbonofre
left a comment
jbonofre
left a comment
Member
There was a problem hiding this comment.
Thanks for the fix and good test coverage @pj-workspace!
The separation between VectorUnloader and AbstractCompressionCodec is clean and strictly adheres to the CompressionCodec contract:
- using
try (uncompressedBuffer)inAbstractCompressionCodec.compress()ensures the input buffer reference on all exit paths (empty check, compression failure, packaging fallback) without double-freeing - the rollback cleanup via
AutoCloseable.close(e, buffers)inVectorUnloader.getRecordBatch()reliably frees collected buffers and preserves the root exception - the regression test suite is great.
(nit and non blocker: we could apply similar try-with-resource semantics to AbstractCompressionCodec.decompress() and align StructVectorUnloader in the C module)
Generated-by: OpenAI Codex
Generated-by: OpenAI Codex
jbonofre
force-pushed
the
fix/vector-unloader-exception-cleanup
branch
from
October 5, 2026 11:11
dc61ee7 to
a5bf863
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's Changed
When record batch serialization fails after retaining or compressing buffers,
VectorUnloadercan leave those buffers alive even after the source vectors are closed. Compression allocation failures can also leave an extra reference to the input buffer inAbstractCompressionCodec.Release collected batch buffers if traversal or batch construction throws, while preserving the original exception. Use try-with-resources in
AbstractCompressionCodec.compressso the codec releases the input it owns on both success and failure. Cleanup remains in the component that owns each buffer: the unloader does not release an input already handed to a custom codec.Regression coverage checks allocator exhaustion with zero allocation headroom, runtime exceptions and errors injected on the second compression call, an empty input buffer, and a malformed later vector. The compression-only test stub is explicitly named
SimulatedLz4Codec; it exercises allocation and ownership, not LZ4 round trips. The tests assert memory usage and reference counts return to their previous values and that the source data remains readable. Additional tests exercise allocation failure with the real LZ4 and ZSTD codecs.The memory-leak report and initial outer-cleanup proposal are from #1234; this change also handles the codec's input ownership.
Validation
git diff --checkpassed.mvn -B -ntp -pl compression -am test -Dtest=TestVectorUnloadLoad,TestVectorUnloaderFailure,TestCompressionCodec,TestArrowReaderWriterWithCompression,TestCompressionCodecServiceProvider -Dsurefire.failIfNoSpecifiedTests=falseCloses #1234.
AI assistance: OpenAI Codex was used for implementation and local validation.