Dataset Mappings
The wiring that connects a logical dataset to a specific physical table in a deployment — and detects schema drift when the two diverge.
orders dataset points to connection Y, table public.orders." It's the binding that resolves a logical dataset reference to a physical table at query time.When to use it: When adding a new deployment, adding a new dataset, or when a physical table has been renamed or moved.
Where to find it: Lumen → Physical Layer → Datasets → [select dataset] → Mappings, or Lumen → Physical Layer → Deployments → [select deployment] → Dataset Mappings.
Who can use it: Luminary only.
A dataset mapping is the join between a dataset and a physical table, scoped to a deployment. Every dataset must have a mapping for every deployment that queries it — otherwise queries will error.
Mapping schema
Each mapping has:
| Field | Description |
|---|---|
deployment_id | Which deployment this mapping belongs to |
dataset_id | Which logical dataset this mapping resolves |
connection_id | Which warehouse connection to query through |
physical_table | The fully-qualified table name (e.g. public.orders, analytics.fact_orders) |
database_override | Optional: overrides the connection's default database |
schema_override | Optional: overrides the connection's default schema |
compatibility_pct | 0–100 score computed by comparing dataset columns vs. physical table columns |
missing_columns | Array of column names in the dataset but not found in the physical table |
extra_columns | Array of column names in the physical table not in the dataset definition |
last_validated_at | When this mapping was last introspected |
Each (deployment_id, dataset_id) pair is unique — one physical table per dataset per deployment.
Creating a mapping
Open the dataset
Go to Lumen → Physical Layer → Datasets and click the dataset you want to map.
Add a mapping
In the Mappings section, click Add Mapping.
Select deployment and connection
Choose the target deployment (e.g. Production) and the warehouse connection to use.
Specify the physical table
Type the fully-qualified table name. Use the format appropriate for your warehouse:
| Warehouse | Format example |
|---|---|
| Postgres / Redshift | public.orders or schema.table |
| Snowflake | DATABASE.SCHEMA.TABLE or SCHEMA.TABLE |
| BigQuery | dataset.table (project ID comes from the connection) |
| DuckDB | schema.table or just table |
Save and validate
Click Save Mapping. Celiq immediately introspects the physical table, compares its columns against the dataset definition, and computes a compatibility score.
Schema drift detection
When a mapping is created or validated, Celiq introspects the physical table and compares its columns to the dataset's column definitions:
compatibility_pct = (dataset columns found in physical table) / (total dataset columns) × 100| Score | Indicator | Meaning |
|---|---|---|
| 100% | Green | All dataset columns present in physical table |
| 80–99% | Orange | Some columns missing — queries using them will error |
| < 80% | Red | Significant schema drift — mapping likely stale |
The UI shows a compatibility bar on each mapping card, with a red border on cards below 100%.
Missing vs. extra columns
- Missing columns (
missing_columns): In the dataset definition but not in the physical table. Queries that SELECT these columns will fail. - Extra columns (
extra_columns): In the physical table but not in the dataset definition. These are ignored — Celiq won't select them until you add them to the dataset.
Extra columns don't affect the compatibility score. Only missing columns (columns the semantic model expects but the warehouse doesn't have) reduce compatibility.
Validating a mapping
Validation re-introspects the physical table and recalculates the compatibility score:
Validate a single mapping: Open the dataset detail page, find the mapping card, and click Re-validate. Validate all mappings for a dataset: On the dataset detail page, click Validate All Mappings. Celiq introspects all physical tables in parallel and returns updated scores.Run validation:
- After schema changes in the warehouse (added or removed columns)
- After pointing a mapping at a new physical table
- As part of a periodic schema-health check
Using database and schema overrides
By default, Celiq uses the database and schema configured in the connection settings. Use overrides when a single connection serves multiple databases or schemas:
database_override: Celiq uses this database name instead of the connection's defaultschema_override: Celiq uses this schema name instead of the connection's default
Example: A Snowflake connection defaults to database ANALYTICS. For your dev deployment, you want to use ANALYTICS_DEV instead. Set database_override = ANALYTICS_DEV on each mapping in the dev deployment.
What happens at query time
When Discover runs a query through a node that references dataset orders:
- Celiq looks up the active deployment (workspace default, or the Mosaic's pinned deployment)
- Finds the mapping for
(active_deployment, orders_dataset_id) - Retrieves
connection_idandphysical_tablefrom the mapping - Generates SQL against
{physical_table}using the specified connection - If no mapping exists, falls back to
primary_connection_id+ the original table
Deleting a mapping
Deleting a mapping is immediate. Any queries that run through nodes using that dataset in the affected deployment will fall back to primary_connection_id, or fail if no fallback exists.
Auditing mapping health
The Deployments list view shows a schema drift indicator for each deployment: the number of datasets with compatibility below 100%. Use this as a health dashboard for your Physical Layer.
For detailed analysis, open a specific deployment and scan the mapping cards for orange/red compatibility bars.
Related pages
- Physical Layer Overview — how mappings fit the full picture
- Datasets — logical table definitions and column schemas
- Deployments — target environments
- Connections — warehouse connection settings