Docs/Physical Layer/Datasets

Datasets

Logical table definitions that decouple your semantic model from physical warehouse tables.

LuminaryWeaverUpdated June 2026 · 6 min read

✦

In this section

Part of Celiq's semantic layer platform. Connect your warehouse, model your data once, query it everywhere.

💡

Tip

What it is: A dataset is a named, versioned schema (list of columns and types) that represents a table in your warehouse. It's the logical counterpart to a physical table — the definition your semantic model nodes reference.
When to use it: When onboarding a new table into your semantic model, or when you need the same table referenced across multiple deployment environments.
Where to find it: Lumen → Physical Layer → Datasets.
Who can use it: Luminary (create/edit/delete), Weaver (view).

A dataset is the abstract definition of a table: its name and its list of columns with their types. It doesn't belong to any one warehouse or environment — that binding is done separately through dataset mappings.

Why datasets exist

Without datasets, every node in your semantic model would reference a physical table directly. That means:

Moving a table (e.g. staging.orders → public.orders) requires editing every node that references it
Running the same semantic model against a dev warehouse requires forking the entire model

Datasets solve this by introducing an indirection layer. Nodes reference datasets; datasets are mapped to physical tables per deployment. Change a mapping — not the model.

Dataset schema

Each dataset has:

Field	Type	Description
`id`	UUID	Unique identifier
`name`	text	Human-readable name, unique per workspace (e.g. `orders`, `customers`)
`description`	text	Optional documentation shown in the UI
`primary_connection_id`	UUID	The connection used when the dataset was created; used as fallback if no deployment mapping exists

Dataset columns

Each dataset also has a set of column definitions:

Field	Type	Description
`name`	text	Column name as it appears in the warehouse (e.g. `user_id`, `created_at`)
`data_type`	text	SQL type (e.g. `INTEGER`, `TIMESTAMP`, `VARCHAR`, `FLOAT`)
`is_nullable`	boolean	Whether NULL values are allowed
`ordinal_position`	integer	Column order in the original warehouse table

Column definitions are introspected automatically from the warehouse when you create a dataset with a physical_table specified. You can also edit column definitions manually.

Creating a dataset

You can create a dataset two ways:

Option A — auto-introspect from a table (recommended):

Open Physical Layer

Go to Lumen → Physical Layer → Datasets and click New Dataset.

Name the dataset

Give it a logical name (e.g. orders). This is what CML nodes will reference, so pick a stable name independent of the physical table name.

Choose a connection and table

Select a warehouse connection, then type or browse to the physical table (e.g. public.orders). Celiq will introspect the table and populate column definitions automatically.

Save

Click Create Dataset. Celiq creates the dataset and its column definitions, and sets the selected connection as primary_connection_id.

Option B — create without a table (define columns manually):

Create the dataset with only a name and description. Add column definitions manually. This is useful when designing a semantic model ahead of warehouse table creation.

Editing a dataset

You can update a dataset's name and description at any time. Renaming a dataset does not automatically update node references in CML — update the dataset: field in your nodes manually after renaming.

You cannot edit column definitions through the dataset UI directly; column definitions update when you re-validate the dataset's mappings (which re-introspects the physical table and recalculates compatibility).

Validating a dataset

Validation checks all of a dataset's mappings against their physical tables:

Open the dataset

Go to Lumen → Physical Layer → Datasets and click the dataset name.

Run validation

Click Validate All Mappings. Celiq introspects each mapped physical table and computes a fresh compatibility score for each mapping.

Review results

Each mapping shows its updated compatibility percentage. Mappings below 100% show which columns are missing in the physical table.

Deleting a dataset

Deleting a dataset also deletes all of its mappings (cascade). It does not delete any nodes in CML that reference it — those nodes will fail validation until you update them.

Before deleting, check Used by nodes on the dataset detail page to see which nodes would be affected.

The `dataset:` field in CML

Nodes reference datasets by name using the dataset: field:

cml

node: orders
  dataset: orders        # the logical dataset name (not a table)
  dimensions:
    - name: created_at
      type: time
      sql: created_at
  measures:
    - name: total_revenue
      type: sum
      sql: revenue

If dataset: is omitted, Celiq falls back to matching by node name — but explicitly setting dataset: is strongly recommended to make the binding clear.

Compatibility scores

When a dataset is mapped to a physical table, Celiq computes a compatibility score:

compatibility_pct = (columns in dataset found in physical table) / (total columns in dataset) × 100

Score	Meaning
100%	All dataset columns exist in the physical table
80–99%	Some columns are missing — queries using those columns will error
< 80%	Significant schema drift — the mapping is likely stale

Columns that exist in the physical table but are not in the dataset (extra_columns) don't reduce the score — they're just ignored.

⚠️

Warning

A dataset at less than 100% compatibility does not prevent queries from running — only queries that reference the missing columns will error. Fix schema drift by either updating the dataset column definitions or updating the physical table in the warehouse.

Physical Layer Overview — how datasets, deployments, and mappings fit together
Dataset Mappings — wire datasets to physical tables per deployment
Deployments — manage target environments
CML Reference — the dataset: field in node definitions