Limitations

  • Lineage is static. SQL built at runtime, Jinja that dbt has not compiled, Python models, stored procedures and UDF internals are not followed; each gap is listed under Warnings in REPORT.md.

  • Without compiled SQL, macros that query the warehouse while compiling (run_query, introspective dbt_utils.get_column_values) render as empty results, so models that depend on them may differ from dbt’s compiled SQL.

  • SELECT * over a relation whose columns Konstruct cannot learn (no warehouse, no catalog) still gets lineage for the columns the query goes on to use, and is reported as a warning; columns passed straight through by the star to the output are only those other queries were seen to read, so the relation it writes is taken to have columns not listed too.

  • An unqualified column in a scope that reads one relation whose columns Konstruct cannot learn, or knows only some of, is taken to be that relation’s, as SQL looks a name up in the innermost scope first. In a subquery, where the name could be an outer query’s column, the build warns, naming the column, whenever a relation of the outer query has or may have that name. Qualify a subquery’s references to the outer query (o.id, not id) for them to be traced to the outer relation.

  • A query’s readings of one relation are numbered by what it says of each. Of six or more readings said exactly the same of, at most 120 numberings are compared, and those still tied are numbered in the order the query names them, so writing them in another order can read as a change. Adding a column that tells two readings apart, such as MAX(a.amount) in a symmetric self-join, can renumber them, and so reads as a change to the join too. Conditions are compared as written but for the order of their operands, so one written another way that means the same, such as NOT a >= b for a < b or a BETWEEN for two comparisons, reads as a change. A CTE the query defines but never reads still has its conditions listed, and its relations count as read, so adding or removing one reads as a change.

  • Dagster ops compose statically only: a graph body built in a loop is counted once (with a warning), ops created by factories are not seen, and sensors that choose a job at run time (RunRequest(job_name=...) without job=) are not linked. Op inputs and outputs are not tracked as data; lineage comes from the SQL an op runs.

  • Airflow parsing covers the common ways of writing DAGs. Task ids or SQL built in loops or from runtime values, dynamic task mapping over computed lists, and DAG factories are skipped with a warning. Selections by --selector, state:, result: and source_status: are not resolved.

  • Identifiers are matched case-insensitively. Two relations whose names differ only by case are treated as one.

  • query ranks by words, not meaning. It finds customer_lifetime_value from “lifetime value” but not from “how much a customer is worth”; the agent is expected to follow up with search or describe.

  • Map keys are read at run time, so attrs['colour'] on a map traces to the attrs column, not to a key. Positional UNNEST ... AS t(a, b) over rows, without field names, traces to the list column.

  • verify runs against a DuckDB file, a Postgres database (in a transaction that is rolled back), or a sample of Snowflake, BigQuery, Databricks or Trino copied into DuckDB, where models whose SQL DuckDB cannot run are skipped, and so are models that unnest or index into a nested column (an array or record), which a sample keeps as JSON; Iceberg catalogues are not supported. It perturbs columns one at a time, so influences that need two inputs to change together are not observed. A Trino sample keeps timestamps to the microsecond, and a timestamp with a time zone as an instant, without the zone’s name.

  • The ontology names entities from their key columns and relations by convention (customer_id, stg_customers.id, stg_stripe__payments.id); a term of kind entity names one that convention gets wrong. A relationship from an entity to itself, such as a customer’s referrer, is not recorded, and a relationship’s evidence lists every test and join between columns of the same names.

  • With several repositories, SELECT * over a relation another repository builds is not expanded, since each repository’s SQL is analysed on its own; name the columns, or let the warehouse supply them, to get their lineage.

  • PySpark jobs are read for the SQL they pass to spark.sql on an object named like a session (spark, session); sqlContext.sql(...) and SparkSession.builder.getOrCreate().sql(...) are not read, nor are DataFrame writes (saveAsTable, insertInto, write). SQL held in a class attribute or an annotated constant counts as built at run time. MERGE INTO gives the table it writes but not its column lineage.

  • Looker explores and joins, references to another view’s fields, Liquid and @{constants} in SQL, native derived tables (explore_source) and persistent derived tables that read other views are not followed, and a derived table’s fields are tied to the view, not to its columns.

  • A dbt-trino profile is read for the none, ldap and jwt methods; for other methods, set the Trino connection in konstruct.toml.

  • Power BI is read from TMDL only, not from .pbix files or a published dataset. A table whose partition is a native query, or that merges several tables, reads those tables as a whole, not column by column. M steps that rename or add columns, relationships, calculation groups and row-level security are not followed, and a partition whose M builds its table name from parameters has no warehouse table.

  • Airbyte keeps no lineage in the warehouse, so without its connection’s configuration (Octavia, an API export or Terraform) Airbyte tables get a sync task but no source system, and there is no column lineage from a raw table to the final one. A stream with mappers gets table lineage only. Fivetran lineage needs the Platform Connector to land in the same warehouse as the tables it describes, or its Terraform schema configuration.

  • Loader configuration is read for the names it spells out: a dlt pipeline’s data built at run time, a Meltano select, Sling stream or DMS selection with wildcards, a Lakeflow schema object, a Terraform value from a variable or another module, and SQL a loader runs give no tables, and columns are linked only by name, so a column a loader renames on the way is not.

  • Hevo, Datastream, Openflow, Matillion, Rivery, Polytomic, PeerDB, Segment, RudderStack, Snowplow, Census and Hightouch tables are recognised in the warehouse with a sync task, but get no source system or column lineage. In the warehouse, DMS is recognised only by its control tables, and Lakeflow Connect, Portable, Supermetrics, Windsor.ai and Azure Data Factory only when a dbt source names them as its loader:; DMS and Lakeflow Connect tables are also read from their configuration files.

  • history renders old versions of a dbt model with today’s macros and the current manifest, so a change made only in a macro is not listed, and a ref to a since-renamed model is noted as a formula it could not work out. The SQL inside a DAG task (not a SQL file it runs) has no history.

  • diff reads dbt YAML for tests, descriptions, sources and exposures, but not for other configuration (materialisation, contracts, meta).

  • graph.json is read only up to 2 GiB; a larger file is refused with an error that says so. A 2,000-model project’s is about 30 MB.

Not Yet (v1)

  • Column-level lineage for dbt Python models and through UDFs.

  • Incremental warehouse metadata (it is re-read on every build; it is fast for local warehouses but not for large remote ones).