fix: generate a valid duckdb gateway config for DuckLake destinations (#5914) - #6118
Open
AnnasMazhar wants to merge 2 commits into
Open
AnnasMazhar wants to merge 2 commits into
AnnasMazhar wants to merge 2 commits into
Conversation
A `ducklake` entry in config.yaml fell through the generic credentials path and produced an invalid duckdb gateway config, which surfaced as a ConfigError from connection.py with a message identical to an unrelated failure — so the real diagnostic was lost. DuckLake connections are a duckdb gateway with the lake attached as a catalog, so build that block explicitly. Unsupported catalog drivers are cut to a clear ClickException rather than propagating the generic error; postgres/mysql/MotherDuck catalogs are not yet mapped and are tracked on the issue. Tests: 3 new (duckdb catalog, sqlite catalog, unsupported driver) plus the existing dlt pipeline tests. Verified to fail on unpatched source and pass on this commit. Signed-off-by: Syed Annas <annas.mazhar10@gmail.com>
Signed-off-by: Syed Annas <annas.mazhar10@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
ducklakedestination inconfig.yamlproduced an invalid gateway config. The failure surfaced fromconnection.pyasConfigError: Unknown connection type 'ducklake'.— a message identical to the oneraised for a genuinely unknown type, so the real diagnostic was lost.
DuckLake is a DuckDB gateway with the lake attached as a catalog, so the config needs that block built
explicitly rather than falling through the generic credentials path.
Fix
sqlmesh/integrations/dlt.pyducklakedestination: the catalog and the ATTACH SQL.catalog.database,storage_url,ducklake_nameandmetadata_schemawere interpolated raw, so a path containing aleading quote or a newline produced invalid YAML and the generated project would not load.
ClickExceptioninstead of letting the generic errorpropagate.
client_config: t.Anywith aTYPE_CHECKINGforward reference, so the module still importswhen the optional
ducklakedependency is absent.db_type(
weaviate,pandas,qdrant, typos) still raisesConfigError: Unknown connection type '<type>'.at that boundary. That behaviour is unchanged by thispull request. This issue reports the
ducklakedestination specifically("When using dlt with a
ducklakedestination ..."), so the narrower fix is deliberate. I am happy towiden it to a generic guard at that boundary if you would prefer that here.
Tests
tests/cli/test_cli.py8 tests added, 1 strengthened:
test_dlt_ducklake_pipeline— parses the emitted YAML and asserts on structure, not substringstest_dlt_ducklake_explicit_metadata_schema,..._override_data_path,..._custom_nametest_dlt_ducklake_pathological_paths_round_trip— leading quote, newline,: #, leading spacetest_dlt_ducklake_yaml_inline_helper,..._block_coexists_with_second_catalogtest_dlt_ducklake_unsupported_catalog— asserts the specific driver in the message and spies thebranch, so it cannot pass for the wrong reason
Ordinary paths emit byte-identical YAML to the previous commit; the
elsebranch (non-ducklake,non-filesystem) is byte-identical to
main. The new tests fail when the source fix is stashed and passwith it.
Notes
dlt[ducklake]pipeline run and thepostgres/mysql/MotherDuck catalogmapping, which is a deliberate scope cut.
format_configis untouched apart from the comment above, so the type-ordering change in fix(dlt): prevent credential type field overwriting db_type in format_config #5857 canland either way without a conflict.