Inputs and How Lineage Works

Input	What Konstruct reads	Status
dbt	target/manifest.json and catalog.json: models, seeds, snapshots, sources, tests (attached to columns), primary_key constraints, exposures, descriptions, column types. Compiled SQL for lineage; ref() and source() are resolved from the manifest when only raw SQL is available	v0
SQL	Column-level lineage with sqlglot for dbt models, warehouse views, Airflow SQL and standalone .sql files (CREATE ... AS, INSERT ... SELECT, MERGE, UPDATE, DELETE, COPY). Dialect from the dbt adapter or configuration	v0
Airflow	DAG files parsed statically (never imported): with DAG, assigned DAGs, @dag, TaskFlow, TaskGroups, >>, <<, chain, cross_downstream. Per task: dbt commands (BashOperator, @task.bash, Cosmos, dbt operators) resolved to models with dbt’s selection syntax; SQL inline or from files; target and source tables; inlet and outlet assets	v0
DuckDB	Tables, views, columns, types, comments, primary keys and view definitions, read only	v0
Postgres	Tables, views, materialised views, partitioned tables, columns, types, comments, primary keys and view definitions, read only	v0
Iceberg	Any PyIceberg catalog, REST included: namespaces, tables, columns (nested struct fields as parent.child), field ids, identifier fields, partitioning, current snapshot. Metadata only; no data files are read	v0
Dagster	Definition files parsed statically (never imported): @asset, @multi_asset and @dbt_assets (with its dbt selection), keys and prefixes, dependencies from parameters, ins= and deps=, groups, asset jobs with their selections, and schedules. An asset’s data is the dbt model, warehouse table or SQL target of the same name, and SQL an asset runs gets column lineage. Op-based jobs: @op, @graph and @job (nested graphs, .alias(), fan-in, .map()/.collect(), graph.to_job()), one task per op invocation named by its Dagster handle, with column lineage from each op’s SQL and dbt runs from dbt.cli([...]). @schedule and sensors (@sensor, @asset_sensor, @multi_asset_sensor, @run_status_sensor, @run_failure_sensor): an asset sensor orders its job after the asset, a success sensor after the monitored job	v0
PySpark	Python jobs parsed statically (never imported): files that import PySpark, Databricks notebooks exported as source, and files that call spark.sql. Each SQL string passed to .sql(...) on a Spark session (an object whose name ends in spark or session, or only spark in a file that uses Snowpark) is analysed in the Spark dialect ([spark] dialect), so CREATE TABLE ... AS, INSERT INTO and INSERT OVERWRITE give the tables they write column lineage; spark_catalog. is the session’s own catalogue and is left out of names. A string is read when it is a literal, an f-string without placeholders, a concatenation of literals or a module-level name bound once; SQL built at run time, or formatted with keyword arguments, is a warning	v0
Looker	LookML files (*.lkml), parsed with the lkml parser and never run, with views resolved across files: a refinement (view: +orders) adds to its view, a view that extends others takes their table and fields, and an abstract view (extension: required) is a base only. Each view is an exposure: a view over a table (sql_table_name, or the view’s own name) reads that table, and each field reads the columns its SQL names as ${TABLE}.column (in sql, sql_start, sql_end, sql_latitude, sql_longitude), directly or through the view’s other fields (${field}, ${view.field}, a dimension group’s ${created_date}) and the fields a measure filters on, so impact on a column reaches the Looker fields built on it and no others. A view over a derived table reads the tables its SQL reads. A table name built with Liquid or parameters, a native derived table, and a field reading a column the table does not have are warnings	v0
Power BI	Semantic models saved as TMDL (*.tmdl, as a Power BI project keeps them under <name>.SemanticModel/definition/), read as text, indented with tabs or spaces; no M or DAX is evaluated. Each table is an exposure named after its model and table, with its description. It reads the warehouse table its partition’s M source navigates to: Source{[Schema="dbo",Item="orders"]} after Sql.Database("server", "database"), or {[Name=...,Kind="Database"]}, {[Name=...,Kind="Schema"]} and {[Name=...,Kind="Table"]} navigation (Snowflake, Databricks, BigQuery), directly or through a shared expression in expressions.tmdl; the tables a native query reads (Value.NativeQuery or [Query="..."]), in the connector’s SQL dialect (T-SQL for SQL Server); for a Direct Lake partition, its schemaName and entityName; or, for incremental refresh, the policy’s sourceExpression. Each column reads the warehouse column its sourceColumn names. Measures, calculated columns and calculated tables read the columns their DAX names, resolved across the model: Table[Column] on any table, [Measure] through the columns that measure reads, and a calculated column through its own DAX (names bound with VAR are not tables), so a measures table built on other tables is an exposure of their columns, and impact on a column reaches the Power BI fields built on it. Automatic date tables are left out. A partition with no warehouse table, a native query that reads none, and a column the table does not have are warnings	v0
Loaders	The tables ingestion tools land, from the marks each leaves in the warehouse: metadata columns, bookkeeping tables or a schema of its own (see Loaders). Each schema a tool lands gets one sync task (task.<loader>.<schema>) that feeds its tables, and each table records its loader in loaded_by. Landed tables are roots: they count towards no lineage coverage, their formulas stop there, and, fed by their sync task, they are not orphans in REPORT.md	v0
Loader configuration	The files that configure loaders, read without running them: dlt pipelines (parsed with ast) and exported dlt schemas, meltano.yml, Singer catalogs and state files, Airbyte connections (Octavia, API exports and Terraform), Fivetran’s Terraform resources, Debezium and other Kafka Connect connectors, Estuary catalogs, Sling replications, Lakeflow Connect bundles and AWS DMS table mappings (see Loader configuration). Each gives the loader’s task, the source system’s tables it reads, outside the warehouse and named after the system (postgres.app.public.orders), and the tables it lands, with column lineage where the file names the columns. [ingestion] paths chooses the files	v0
Fivetran lineage	When a warehouse holds the Fivetran Platform Connector’s lineage tables (source_schema, source_table, source_column, destination_*, table_lineage and column_lineage, in fivetran_log or fivetran_metadata), each connection’s source tables and columns are added, outside the warehouse and named in full after the connection (salesforce_prod.Salesforce.Account), with column lineage into the tables it lands in that warehouse, so trace reaches the source system while short names such as account.name keep meaning the landed table. The warehouse’s own destination is the only one the log describes, or else the one that lands the lineage schema; the other destinations of the account are left out, and so are tables and schemas deleted from the source. Both the current connection table and the older connector are read, and an unreadable lineage schema, or one whose destination is not clear, is a warning. [fivetran] lineage = false turns the queries off	v0
dlt lineage	From each dataset’s _dlt_version table, the newest version of each schema dlt loaded into it, read with one query: each table’s resource, as a source table outside the warehouse (dlt.github.issues) feeding the dataset’s dlt sync task, with the table as evidence; nested tables and dlt’s own are left out. An unreadable table is a warning, and [dlt] lineage = false turns the query off	v0
Business definitions	definitions.yml files: each term’s name, kind (entity, measure or term), synonyms, definition, owner, status (draft, agreed, approved, retired), approval date, grain, bound columns, required filters, unit, currency, related terms and exceptions. Terms become term nodes with defines edges to their columns, and the build warns when a bound column or filter column no longer exists (see Business Definitions)	v0
Vocabularies	SKOS concepts and OWL classes in Turtle, N-Triples or RDF/XML files named in [vocabularies] paths, read as business terms with their labels, definitions and broader concepts, through konstruct[rdf] (see Business Definitions)	v0
Apache Ossie	*.ossie.yaml semantic models (0.1.1 and the 0.2.0 draft): each metric becomes a business term of kind measure bound to the source columns it reads, primary keys mark entity keys and relationships link entities; anything that does not resolve is a build warning (see Apache Ossie Models)	v0
Snowflake	Tables, views, columns, types, comments and view definitions from each database’s information_schema, through konstruct[snowflake]	v0
BigQuery	Datasets, tables, views and view queries through the client API, with nested RECORD fields and their descriptions, through konstruct[bigquery]	v0
Databricks	Unity Catalog information_schema for each catalogue, including struct columns, through konstruct[databricks]	v0
Trino	Tables, views, columns, types (with row fields as parent.child), view definitions and column comments from each catalogue’s information_schema, and table comments from system.metadata.table_comments when the user may read it, through konstruct[trino]; Starburst too. SQL is read in the Trino dialect	v0

Loaders

Konstruct recognises a loader’s tables only by the full set of marks the loader documents, so an application’s own op, deleted or received_at column, or a tracks table of its own, is not taken for a loader’s. A mark that several tools share names the family (singer, segment_spec); a companion table that only one of them leaves in the same schema refines it, and so does a loader the project names for the table in a dbt source’s loader:. A dbt source whose loader: names a tool, with no mark in the warehouse or no warehouse at all, is that tool’s landed table all the same, under the name in lower case with words joined by _ (Fivetran is fivetran).

Loader	loaded_by	What marks its tables in the warehouse
Fivetran	fivetran	_fivetran_synced
Airbyte	airbyte	_airbyte_raw_id and _airbyte_extracted_at (Destinations V2), or _airbyte_ab_id and _airbyte_emitted_at (V1). A raw table in airbyte_internal (<namespace>_raw__stream_<stream> or raw_<namespace>__<stream>) feeds the task of its namespace
Singer (Meltano, PipelineWise and other Singer targets)	singer	_sdc_received_at, _sdc_batched_at, _sdc_sequence and _sdc_table_version, or _sdc_extracted_at, _sdc_batched_at and _sdc_deleted_at
Stitch	stitch	The Singer columns, in a schema that also holds _sdc_rejected (or sdc_rejected)
Meltano	meltano	The Singer columns, on a table the project says Meltano lands (a dbt source’s loader:)
dlt	dlt	_dlt_load_id and _dlt_id; nested tables _dlt_id, _dlt_parent_id and _dlt_list_idx; its _dlt_loads, _dlt_pipeline_state and _dlt_version tables, as bookkeeping
Hevo	hevo	__hevo__ingested_at (or __hevo_ingested_at)
Estuary Flow	estuary	flow_published_at with flow_document or _meta/op
Debezium change events	debezium	__op or __deleted, with a field of the event’s source block that its ExtractNewRecordState transformation adds, such as __source_ts_ms, __source_table or __lsn
Kafka Connect	kafka_connect	RECORD_METADATA and RECORD_CONTENT, as the Snowflake connector for Kafka writes a table without schematisation
AWS DMS	dms	Its control tables (awsdms_apply_exceptions, awsdms_status, awsdms_suspended_tables, awsdms_history), as bookkeeping; the columns it adds to replicated tables are named by its rules, so they mark nothing
Google Datastream	datastream	The datastream_metadata struct, with uuid and source_timestamp
Snowflake Openflow	openflow	_SNOWFLAKE_INSERTED_AT, _SNOWFLAKE_UPDATED_AT and _SNOWFLAKE_DELETED (the Snowflake connectors for PostgreSQL and MySQL too); its journal tables (<table>_JOURNAL_<timestamp>_<number>) as bookkeeping
Databricks Lakeflow Connect	lakeflow	Nothing of its own; its tables come from its pipelines’ configuration (see Loader configuration)
Matillion Data Loader	matillion	MTLN_CDC_LAST_CHANGE_TYPE, MTLN_CDC_LAST_COMMIT_TIMESTAMP and MTLN_CDC_SEQUENCE_NUMBER (change log), or MTLN_CDC_DELETED (soft deletes)
Rivery (Boomi Data Integration)	rivery	_rivery_last_update, _rivery_run_id and _rivery_river_id
Polytomic	polytomic	_polytomic_created_at, _polytomic_updated_at and _polytomic_deleted_at
PeerDB	peerdb	_peerdb_synced_at or _peerdb_is_deleted
Sling	sling	_sling_loaded_at, _sling_synced_at with _sling_synced_op, or _sling_deleted_at
Segment-spec event tables (Segment, RudderStack)	segment_spec	received_at, sent_at, uuid_ts and anonymous_id, with event and event_text on each event’s table (tracks included) or as identifies, pages, screens, groups or aliases; users and accounts beside them
RudderStack	rudderstack	The Segment-spec tables, in a schema that also holds rudder_discards
Segment	segment	The Segment-spec tables, on a table the project says Segment lands
Snowplow	snowplow	The atomic events table: app_id, platform, etl_tstamp, collector_tstamp, event_id, v_collector, v_etl, domain_userid and network_userid
Census	census	Its sync_log (log_id, sync_id, sync_run_id, _census_logged_at), with source_objects, destination_objects and destinations beside it, as reverse ETL logs
Hightouch	hightouch	sync_runs, sync_changelog and sync_snapshot in hightouch_audit, as reverse ETL logs

A loader’s bookkeeping tables and reverse ETL logs have loader_role set (metadata, or raw for Airbyte’s raw tables), are fed by its task, and are not counted as landed tables. Any schema where a loader landed data, except Census and Hightouch, which only log what they sent, is evidence for an ontology pack.

Portable, Supermetrics, Windsor.ai, Azure Data Factory and Databricks Lakeflow Connect leave no documented mark on the tables they write (their names and columns are the user’s choice, or the source’s own), so in the warehouse their tables are recognised only when a dbt source names the loader; Lakeflow Connect’s are also read from its pipelines’ configuration.

Loader Configuration

Discovery reads a file only when its name and a look at its text show one of these formats, so other Python, YAML and JSON files cost one read and are left alone; files over 2 MB are not read. Nothing is imported, run or connected to. A file Konstruct cannot read, such as one nested too deeply or with a value of the wrong shape, is a warning, and the build goes on without it. Every YAML file Konstruct reads, these and the dbt projects, profiles, definitions, Ossie files and ontology packs, cannot be read when its aliases would add more than a million values once expanded, or when it holds a value YAML cannot give, such as a date that does not exist, an integer of more than 4,300 digits or an escaped lone surrogate; a dbt project, profile, definitions, Ossie or pack file cannot be read either when it is neither UTF-8 nor UTF-16. A table name filled in when the loader runs ($VAR, ${VAR}, a Terraform expression) is a warning too, and that table’s lineage is left out rather than read as a name.

File	Loader	What Konstruct reads
dlt pipeline (*.py that imports dlt and creates a pipeline)	dlt	Each dlt.pipeline(...) assigned to a name, its pipeline_name and dataset_name (or <pipeline>_dataset), and what each run loads: @dlt.resource functions (name, table_name, columns), @dlt.source functions and the resources they return, .with_resources(...), sql_database and sql_table with literal tables (the system and database from a literal connection URL: postgres.app.public.orders), and rest_api_source with a literal configuration (the system is the API’s host). table_name= on run names the landed table. Data built any other way is a warning
dlt schema (*.schema.yaml with engine_version and tables)	dlt	Each table and the resource it comes from; nested tables and dlt’s own tables are left out, and the tables are found by name, since the dataset is not in the file
meltano.yml	meltano	Each extractor paired with the loader that follows it in a job’s tasks, with a schedule’s loader, or with the only loader; plugins with what they inherit_from; the streams its select names without wildcards (public-orders.*, public-customers.id, ! excluding a column), each landed as its table (public-orders is orders); the loader’s default_target_schema and database, from the file or else from the defaults in the lock files meltano lock writes (plugins/loaders/*.lock, plugins/extractors/*.lock), with $MELTANO_EXTRACT__LOAD_SCHEMA as the extractor’s load_schema, or its namespace (tap_postgres for tap-postgres); without a schema, the table is found by its name; the source database from the extractor’s database
Singer catalog (*.json with tap_stream_id)	singer	Each selected stream, its source schema (schema-name or database-name), table and selected columns, the tap from a tap-<name> in the file’s path
Singer state (*state*.json whose bookmarks map each stream to an object, and that is Meltano’s singer_state, has currently_syncing, is in a tap-<name> directory or named tap-<name>.<...>, or holds replication_key, replication_key_value or lsn bookmarks)	singer	The streams it holds bookmarks for
Airbyte Octavia project (definition_type: connection, with the sources/ and destinations/ it points to)	airbyte	Each selected stream with its namespace and fields (the selected fields, or every property of its schema), the source connector (airbyte/source-postgres is postgres) and its database, and the landed schema from namespace_definition (the stream’s namespace, the destination’s schema or dataset_id, or namespace_format with ${SOURCE_NAMESPACE}) and table from prefix and the stream’s name. A stream without a namespace reads the source’s schema when its schemas lists one; otherwise it lands, as Airbyte lands it, in the destination’s default, as does a blank namespace_format. A namespace definition, format or prefix that is not a literal (${VAR}, a Terraform expression) is a warning
Airbyte connection exported from its API (*.json with syncCatalog, or sourceId, destinationId and configurations)	airbyte	The same, from the configuration API’s syncCatalog or the public API’s configurations.streams, one connection, a list, or a data list; without the source’s type, the system is named after the connection
Terraform (*.tf with fivetran_ or airbyte_ resources), a directory at a time	fivetran, airbyte	airbyte_connection with the airbyte_source_<type> and airbyte_destination_<type> resources it refers to and its configurations.streams (selected_fields); fivetran_connector_schema_config (its schemas map, or the older schema blocks) with the fivetran_connector or fivetran_connection it configures: each enabled table and column, a hashed column as transformed, landed in <prefix>_<schema> for a database connection or in its destination_schema name. Expressions Terraform evaluates (variables, references, functions other than jsonencode) are not evaluated
Kafka Connect connector (*.json with connector.class, alone, with name and config, or in a list; or a .properties file)	debezium, kafka_connect	A Debezium connector (PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, Db2, MongoDB): each table its table.include.list (or collection.include.list) names, its database, the columns column.include.list names, and the topic it writes (<topic.prefix>.<schema>.<table>, with the database for SQL Server), after RegexRouter and ByLogicalTableRouter transformations, with Java’s $1 and ${name} replacements. The expressions run in time linear in the topic, with \w, \d, \s and (?i) matching ASCII as Java’s do by default; one with a backreference, lookaround, a possessive repeat or a Unicode class, which that matcher cannot run, one that compiles to more than 64 KiB, a connector with more than 32 transformations, a file with more than 64 distinct expressions, and a topic longer than Kafka’s 249 characters, before or after routing, are warnings, and that connector’s topics are left out. A sink: each topic in topics and the table it writes, by the connector’s rules: the Debezium JDBC sink (collection.name.format: ${topic} with the topic’s dots as underscores, or unchanged with the passthrough naming strategy, and ${source.schema} and ${source.table}, which only a <prefix>.<schema>.<table> topic fills, from its last two names; any other placeholder is a warning; the database from connection.url), the Confluent JDBC sink (table.name.format), Snowflake (snowflake.topic2table.map, or the topic in capitals, in snowflake.database.name and snowflake.schema.name), BigQuery (topic2TableMap, sanitizeTopics, defaultDataset) and Iceberg (iceberg.tables). Topics are nodes of their own (kafka.app.public.orders), so a connector and the sink that reads its topics make one chain. Include lists with patterns, topics.regex and dynamic routing are warnings
Estuary Flow catalog (flow.yaml, *.flow.yaml)	estuary	Each capture’s bindings (namespace and stream) into collections, and each materialization’s bindings from collections into tables (table, with the binding’s or the endpoint’s schema or dataset and database); the system from the connector image (ghcr.io/estuary/source-postgres is postgres). Collections are nodes of their own (estuary.acme/public/orders)
Sling replication (YAML with source, target and streams, no top-level keys but those and defaults, env and hooks, and an object)	sling	Each stream (schema.table, or a file URL) not disabled, landed in the object its stream or defaults names, with its runtime variables filled in ({stream_schema}, {stream_table}, {stream_name}, {stream_file_name}, and {target_schema} left for the table to be found by name), and the columns its select names. The source system is named after the source connection. Wildcard streams, streams with their own sql, and objects with variables known only at run time are warnings
Databricks Asset Bundle with Lakeflow Connect pipelines (YAML with ingestion_definition)	lakeflow	Each pipeline’s table objects (source_catalog, source_schema, source_table, destination_catalog, destination_schema, destination_table, and table_configuration.include_columns), the system named after its connection_name, or its gateway’s for a database connector. A schema object, which ingests tables it does not list, is a warning
AWS DMS table mappings (*.json with rule-type and object-locator)	dms	Each table a selection rule includes by name and no rule excludes, landed under the names its transformation rules give (rename, add-prefix, add-suffix, remove-prefix, remove-suffix, case conversion), and the columns its rules rename. The task and system are named after the file. A selection by % pattern is a warning

A file’s tables refine what the warehouse shows: Singer columns on a table meltano.yml lands make it Meltano’s. When the warehouse already has the loader’s sync task for a landed table, the file’s source tables feed that task, which records the file in path and the pipeline in pipeline; otherwise the task is the pipeline’s own (task.dlt.shop). Each edge from a file carries it as evidence (pipelines/shop.py:4). A file listed in [ingestion] paths adds a table the warehouse does not have, or every table when there is no warehouse, as an inferred table. A file that discovery found reaches only the tables the warehouse, dbt or SQL already show, since an application’s own file can look like a loader’s; list a file in [ingestion] paths to add the tables it lands. A column the file names is linked to the source system’s column of the same name when the landed table has it, or when Konstruct knows no columns of the landed table.

Several Repositories

repositories in konstruct.toml lists other repositories to build into the same graph, such as a marketing dbt project that reads tables a finance repository builds:

repositories = ["../marketing", "../platform/airflow"]

Each repository is built with its own configuration (its own konstruct.toml, if it has one), under the options of the build that lists it: --no-dbt, --no-warehouse and --dialect apply to every repository. Its analyses are cached under konstruct-out/repositories/<name>/ of the repository that lists it; the name is the folder’s, with its parent’s added when two folders share one (b-analytics). Konstruct writes nothing to the other repositories, though dbt writes its usual target/ there when it runs. The repositories a listed repository lists in turn are not followed.

A relation two repositories share is one node, and a model one repository builds that another reads as a dbt source is linked column by column, so impact and trace cross from one repository to the next. A business term in one repository’s definitions.yml can name a column another repository builds, so the finance and marketing repositories share one definition of revenue; a term defined in more than one repository is a build warning. Each repository’s SQL is analysed before the graphs are merged, so a SELECT * over a relation another repository builds is not expanded (a build warning says so); columns the SQL names are linked. Two different things with the same id, such as two dbt packages both called shop with a model stg in different schemas, stay apart: the later repository’s keeps its id with @<name> added (model.shop.stg@marketing). Nodes from another repository carry its name in repository and their paths relative to this one, the build watches their files for --update and konstruct status, konstruct status lists what it finds in each repository, and their warnings are prefixed with the repository’s name.

How Lineage Works

• Konstruct qualifies each statement against the columns it knows (from the dbt catalog, the warehouse, or the lineage of upstream models, processed in dependency order), so SELECT * expands. A relation whose columns only queries that read it, or its YAML, name is not taken to have only those. sqlglot then follows every output column through CTEs, subqueries, aliases, unions and window functions to physical columns.

• A path made only of column references and renames is direct; anything else (arithmetic, functions, aggregates, CASE, window functions) is transformed.

• Columns that change which rows a model returns, without being selected (join keys, filters, GROUP BY, HAVING, QUALIFY, DISTINCT and DISTINCT ON, the columns a UNION without ALL, an INTERSECT or an EXCEPT compares, and the ORDER BY of a LIMIT), are recorded as feeds edges with role: reference, traced through CTEs. Impact reports the relations that use them but does not then mark every downstream column, which would bury the answer.

• Nested data is followed: o.customer.address.city, customer['country'], items[1].sku, unnest(items) and FROM t, UNNEST(t.items) AS i with i.sku trace to the nested column (orders.items.sku), in outputs and in conditions, through CTEs; reading a field or unnesting a list counts as a direct pass-through. Struct columns, and the structs inside lists and map values, are listed field by field for every engine, named the way SQL reads them.

• Without compiled SQL, dbt models are rendered by Konstruct in a sandbox: loops, variables, var() with the values vars in dbt_project.yml gives the project’s models (the global ones, with the block under the project’s name over them; a package’s macros called from a model see the same), ref, source, config, the project’s and packages’ macros (dbt_utils.*), adapter.dispatch and dbt’s cross-database macros (dbt.date_trunc, dbt.cast, dbt.concat, dbt.hash, dbt.date_spine and others). An unknown macro becomes a function call that keeps its arguments, so the columns passed to it stay in the lineage.

• var() reads only the project’s dbt_project.yml: a package’s own dbt_project.yml vars and values passed with dbt --vars are not seen, and a value that is itself a template ("{{ env_var('START_DATE') }}") is unknown, so the call’s default is used. Build the project with dbt, so its compiled SQL is read, where those values change the lineage.

• dbt models, warehouse tables with the same relation name, Airflow SQL targets and Iceberg tables merge into one node, so lineage crosses tools: a column computed in dbt, copied into a reporting table by an Airflow task, and used by a dashboard is one chain.