FabCon 2026 — Recap Part 2

Deep Dives, Best Practices, and What's Coming in Microsoft Fabric

My notes from Fabcon sessions covering Delta Lake internals, SQL in Fabric, Agentic AI with Fabric IQ, and the new OneLake Spark Catalog.

Published after FabCon 2026 · Sessions: Delta Lake • SQL Fun in Fabric • Fabric IQ & Foundry IQ • OneLake Spark Catalog

FabCon packed in a lot of technical depth in its sessions. These are my session notes, organized by topic. There's a lot of practical guidance here that I plan to take back and apply directly to our Fabric environment — especially around Delta Lake optimization and the OneLake Spark Catalog migration.

Session 1

Delta Lake Internals & Best Practices

This session covered a lot of ground on how Delta Lake actually works under the hood, and more importantly, what you should be doing (and not doing) to get the best performance.

What Delta Lake gives you

Delta Lake adds transactional features on top of Parquet — merge, update, delete, and insert — tracked via a JSON-based transaction log. The transaction log is the heart of Delta, providing:

Full ACID compliance
Decoupled compute and storage
Time travel and snapshot generation
Data skipping with min/max metadata on the first 32 columns (like an index)
Change tracking and logical deletes with recovery

V-order: write-time optimization

V-order is a write-time organization technique that makes reads up to 10x faster. It's particularly valuable for Power BI Direct Lake, and for silver and gold layers in a medallion architecture.

File size: it's complicated

One of the most nuanced parts of the session. The right file size strategy depends on your query pattern:

Power BI prefers fewer, larger files — one 20 GB file with 160 row groups of 128 MB each outperforms 160 individual 128 MB files.
Selective lookups benefit from more smaller files, because each file has its own min/max values for skipping irrelevant data.
The worst scenario: 5,000 small files with 4 MB row groups perform about 50x worse than 160 files with 128 MB row groups. The fix is to run Optimize.

Key recommendation

Always use Adaptive File Target Size: set spark.microsoft.delta.targetFileSize.adaptive.enabled = True. This automatically evaluates your table size and applies the best file size settings for the data.

Optimize, Fast Optimize, and Auto Compaction

Fast Optimize reduces write amplification and only compacts bins when estimated as optimal — it's about 5x faster than standard Optimize. Use Fast Optimize.
Auto Compaction (spark.databricks.delta.autoCompact.enabled=True) evaluates automatically when compaction is needed. Set minNumFiles = 50 as a starting point. Always use Auto Compaction.
Optimize Write is a pre-write compaction that's faster for streaming, small batch, and partitioned tables — but don't use it when adaptive file size is enabled.

Deletion vectors

Parquet files are immutable. Historically, a delete meant rewriting an entire file. Deletion vectors solve this by writing a small sidecar file that says "skip this record when reading" — the original Parquet file is untouched. Enable them with:

spark.databricks.delta.properties.defaults.enableDeletionVectors = True

Liquid clustering

Liquid clustering improves file skipping by physically ordering data on disk by a chosen key. A few important rules:

The clustering unit (zcube) is 100 GB — so data will be rewritten.
Don't use it for tables under 100 GB. It's most valuable at 1 TB and above.
Similarly, don't partition tables under 1 TB.

Delta Lake 4.1 highlights

Optimized liquid clustering: now incremental, with auto-reclustering — 2.6x cheaper and faster, and it works with Fast Optimize.
Type widening: automatically promotes column types (e.g., int to bigint) — enabled by default.
Multi-table transactions: coming soon.

Golden configuration

Enable deletion vectors + adaptive target file size + auto compaction. Use Spark resource profiles to ensure data is saved according to profile requirements.

SQL Fun in Fabric

This session (presented by Shabnam Watson — check out shabnamwatson.com and the O'Reilly book Analytics Engineering with MS Fabric and Power BI) covered the landscape of SQL options in Fabric and the practical differences between them.

Read-only vs. read-write

Lakehouse: read-only from a SQL analytics endpoint perspective.
Data Warehouse and Fabric SQL: read/write.

Fabric SQL database

Fabric SQL is the relational database option in Fabric. Key characteristics:

Performance is controlled by capacity size.
Uses MDF & LDF files, with a SQL analytics endpoint that mirrors those files as Parquet in OneLake in near real-time.
Restores are only possible within Fabric.
Has SQL Audit Logs.
Enforces foreign key constraints (unlike the SQL Warehouse).
Has a long GUID in its connection string name; the analytics endpoint does not.

SQL Warehouse

The Warehouse is the distributed, OLAP-oriented engine — equivalent to Synapse Analytics. Key notes:

Two execution engine pools: one for SELECT, one for DML.
Parquet column storage means no clustered indexes — only nonclustered indexes are available.
Foreign key constraints are not enforced.
Built-in backups with restore points that are pointers to Delta history. Manual creation is available, but restore is in-place only.
Use COPY INTO <Table> for bulk inserts — Bulk Insert is deprecated and slower.
Use OPENROWSET to query across workspaces.
Use bigint for ID columns.

Session 3

Fabric IQ & Foundry IQ: Making AI Agents Productive

This session introduced Microsoft's vision for AI Agents in the Fabric ecosystem — the goal being to make agents "as trusted and productive as the best employees."

Three layers of IQ

Fabric IQ: Context on how your business operates — business entities, systems of record, and actions that bind data to Fabric IQ through an Ontology (i.e., a knowledge graph that models your environment and how your business works).
Foundry IQ: The tools for building and deploying enterprise AI apps securely — point-and-click data transformation that automatically chunks data, with network isolation including custom VNet and Private Links.
Work IQ: Context on people — collaboration and workflows.

Ontology = Knowledge Graph

The "Ontology" in Fabric IQ is essentially a knowledge graph. You search the ontology to create a knowledge source for your AI agents. One token ≈ 4 characters of text.

Agentic AI with Fabric IQ

The broader vision here is a shift from human operators to human governors — moving from 100% human-driven operations toward a model where roughly 45–55% of the work is AI-powered. Fabric IQ's ontology provides the business meaning that makes this possible.

Status Preview

Fabric IQ & Foundry integration is currently in preview.

OneLake Spark Catalog & Migrating to Schema-Enabled Lakehouses

This was probably the most operationally impactful session of the day for anyone running Fabric at scale. It covered how Spark discovers data and the significant improvements coming with the new OneLake Spark Catalog.

The old vs. new Spark catalog

The legacy Spark catalog maps table names to file locations and works within a single workspace. Its problems: chatty, slow, no row-level or column-level security, no VNet support.

The new OneLake Spark Catalog is a fundamentally different architecture:

Metadata lives with the data in the Delta log — no separate metastore.
Immediate table detection after ingestion.
Multi-workspace queries.
Compatible with schema-enabled lakehouses.
Can resolve anything the tenant can access — including Databricks catalogs and external tables via shortcuts.
Supports OneLake row-level security and column-level security.

Naming conventions matter

Local context: SELECT * FROM product — only for the linked lakehouse.
2-part name: market.product — ambiguous (schema or another lakehouse?).
3-part name: cross data-source.
4-part name: fabric-wide cross-workspace query.

For non-schema lakehouses, use dbo as the schema name.

Should you switch to schema-enabled lakehouses?

Switch now if...

All Fabric resources are governed by a single group, shortcut usage is well-known, and notebooks and other code are well-documented.

Delay the switch if...

Governance is fragmented, or most tables are interlinked between multiple storage items.

What breaks when you switch

The switch is instant and irreversible. Tables move under the dbo schema — no data files are moved, but the following will break:

Notebook code referencing tables by old names.
RLS/CLS policies (they need schema references updated).
Semantic models.
Shortcuts pointing to the lakehouse.

Migration checklist

Copy catalog elements to OneLake (Microsoft has a script for this).
Enable the OneLake catalog.
Test notebook runs.
Refactor notebooks to reference tables with the dbo schema.
Run PySpark scripts to schema-enable the lakehouses.
Update OneLake security policies, semantic models, and Direct Lake references using the provided scripts.
Update all shortcuts pointing to updated lakehouses. Note: there is currently no lineage information to help find shortcuts automatically.
The SQL analytics endpoint won't change.

What's coming

Microsoft is working on a migration assistant, a Copilot agent with Fabric skills, and a fully automated switch process.

Scripts & instructions

aka.ms/LakehouseSwitch →Notebook scripts and property switch API instructions for enabling schema-enabled lakehouses.

These are personal session notes from FabCon 2026 — Part 2. Accuracy reflects my understanding at the time; always verify details against official Microsoft documentation before making architectural decisions.

Lady Runa - SQL Stitcher

Monday, April 20, 2026