Skip to content

Fix excessive memory usage in local storage range downloads - #2192

Open
mneha05 wants to merge 1 commit into
apache:trunkfrom
mneha05:fix/local-range-streaming
Open

mneha05 wants to merge 1 commit into
apache:trunkfrom
mneha05:fix/local-range-streaming

Conversation

@mneha05

@mneha05 mneha05 commented Oct 6, 2026

Copy link
Copy Markdown

Fix excessive memory usage in local storage range downloads

Description

Fixes #2191.

The local driver's range-stream method reads the entire source file to determine its size, then yields the whole requested range as one chunk. Even a tiny range therefore allocates memory proportional to the source size, and chunk_size is ignored. The file-download method then buffers the complete range before writing.

Use os.fstat on the open descriptor for the size, seek directly to the requested offset, and read at most the requested chunk size and remaining range length. Write each chunk to the destination as it arrives. Preserve non-inclusive end offsets and existing range validation. Use the standard default chunk size when omitted or zero, reject negative chunk sizes, and stop on early EOF.

Add five regression test methods covering bounded/default chunks, exact range reads, incremental writes, negative chunk sizes, and early EOF. All five fail against the original implementation. Add a storage changelog entry.

A local tracemalloc experiment requested 1 KiB from a 64 MiB sparse file with chunk_size=128:

Implementation Peak traced Python allocation Output
Original 67,113,884 bytes One 1,024-byte chunk
Fixed 6,242 bytes Eight 128-byte chunks

These are allocations for this single experiment, not total process memory or a general benchmark.

Status

Done, ready for review. Local checks pass; upstream CI has not yet run.

Validation on Python 3.12.14 / Linux:

  • pytest -q libcloud/test/storage libcloud/test/test_utils.py --ignore=libcloud/test/storage/test_list_objects_filtering_performance.py: 982 passed, 12 skipped, with six additional passing subtests. Existing deprecation warnings remain.
  • All applicable prek hooks for the three changed files pass, including Black, isort, flake8, pyupgrade, and codespell.
  • git diff --check passes.

Checklist

AI assistance

Generated-by: OpenAI Codex (GPT-6)

Codex assisted with investigation, implementation, regression tests, local validation, and this description. This assistance is disclosed following the ASF Generative Tooling Guidance.

Small range downloads currently allocate memory proportional to the source
file size and ignore the requested chunk size. File downloads also retain
the complete range before writing, which makes large local objects costly
to access under memory limits.

Fixes apache#2191

Generated-by: OpenAI Codex (GPT-6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local storage range downloads buffer the source and ignore chunk_size

1 participant