FabCon 2026 — Recap Part 2
Deep Dives, Best Practices, and What's Coming in Microsoft Fabric
My notes from Fabcon sessions covering Delta Lake internals, SQL in Fabric, Agentic AI with Fabric IQ, and the new OneLake Spark Catalog.
FabCon packed in a lot of technical depth in its sessions. These are my session notes, organized by topic. There's a lot of practical guidance here that I plan to take back and apply directly to our Fabric environment — especially around Delta Lake optimization and the OneLake Spark Catalog migration.
Delta Lake Internals & Best Practices
This session covered a lot of ground on how Delta Lake actually works under the hood, and more importantly, what you should be doing (and not doing) to get the best performance.
What Delta Lake gives you
Delta Lake adds transactional features on top of Parquet — merge, update, delete, and insert — tracked via a JSON-based transaction log. The transaction log is the heart of Delta, providing:
- Full ACID compliance
- Decoupled compute and storage
- Time travel and snapshot generation
- Data skipping with min/max metadata on the first 32 columns (like an index)
- Change tracking and logical deletes with recovery
V-order: write-time optimization
V-order is a write-time organization technique that makes reads up to 10x faster. It's particularly valuable for Power BI Direct Lake, and for silver and gold layers in a medallion architecture.
File size: it's complicated
One of the most nuanced parts of the session. The right file size strategy depends on your query pattern:
- Power BI prefers fewer, larger files — one 20 GB file with 160 row groups of 128 MB each outperforms 160 individual 128 MB files.
- Selective lookups benefit from more smaller files, because each file has its own min/max values for skipping irrelevant data.
- The worst scenario: 5,000 small files with 4 MB row groups perform about 50x worse than 160 files with 128 MB row groups. The fix is to run Optimize.
spark.microsoft.delta.targetFileSize.adaptive.enabled = True. This automatically evaluates your table size and applies the best file size settings for the data.Optimize, Fast Optimize, and Auto Compaction
- Fast Optimize reduces write amplification and only compacts bins when estimated as optimal — it's about 5x faster than standard Optimize. Use Fast Optimize.
- Auto Compaction (
spark.databricks.delta.autoCompact.enabled=True) evaluates automatically when compaction is needed. SetminNumFiles = 50as a starting point. Always use Auto Compaction. - Optimize Write is a pre-write compaction that's faster for streaming, small batch, and partitioned tables — but don't use it when adaptive file size is enabled.
Deletion vectors
Parquet files are immutable. Historically, a delete meant rewriting an entire file. Deletion vectors solve this by writing a small sidecar file that says "skip this record when reading" — the original Parquet file is untouched. Enable them with:
spark.databricks.delta.properties.defaults.enableDeletionVectors = True
Liquid clustering
Liquid clustering improves file skipping by physically ordering data on disk by a chosen key. A few important rules:
- The clustering unit (zcube) is 100 GB — so data will be rewritten.
- Don't use it for tables under 100 GB. It's most valuable at 1 TB and above.
- Similarly, don't partition tables under 1 TB.
Delta Lake 4.1 highlights
- Optimized liquid clustering: now incremental, with auto-reclustering — 2.6x cheaper and faster, and it works with Fast Optimize.
- Type widening: automatically promotes column types (e.g., int to bigint) — enabled by default.
- Multi-table transactions: coming soon.
SQL Fun in Fabric
This session (presented by Shabnam Watson — check out shabnamwatson.com and the O'Reilly book Analytics Engineering with MS Fabric and Power BI) covered the landscape of SQL options in Fabric and the practical differences between them.
Read-only vs. read-write
- Lakehouse: read-only from a SQL analytics endpoint perspective.
- Data Warehouse and Fabric SQL: read/write.
Fabric SQL database
Fabric SQL is the relational database option in Fabric. Key characteristics:
- Performance is controlled by capacity size.
- Uses MDF & LDF files, with a SQL analytics endpoint that mirrors those files as Parquet in OneLake in near real-time.
- Restores are only possible within Fabric.
- Has SQL Audit Logs.
- Enforces foreign key constraints (unlike the SQL Warehouse).
- Has a long GUID in its connection string name; the analytics endpoint does not.
SQL Warehouse
The Warehouse is the distributed, OLAP-oriented engine — equivalent to Synapse Analytics. Key notes:
- Two execution engine pools: one for SELECT, one for DML.
- Parquet column storage means no clustered indexes — only nonclustered indexes are available.
- Foreign key constraints are not enforced.
- Built-in backups with restore points that are pointers to Delta history. Manual creation is available, but restore is in-place only.
- Use
COPY INTO <Table>for bulk inserts — Bulk Insert is deprecated and slower. - Use
OPENROWSETto query across workspaces. - Use
bigintfor ID columns.
Fabric IQ & Foundry IQ: Making AI Agents Productive
This session introduced Microsoft's vision for AI Agents in the Fabric ecosystem — the goal being to make agents "as trusted and productive as the best employees."
Three layers of IQ
- Fabric IQ: Context on how your business operates — business entities, systems of record, and actions that bind data to Fabric IQ through an Ontology (i.e., a knowledge graph that models your environment and how your business works).
- Foundry IQ: The tools for building and deploying enterprise AI apps securely — point-and-click data transformation that automatically chunks data, with network isolation including custom VNet and Private Links.
- Work IQ: Context on people — collaboration and workflows.
Agentic AI with Fabric IQ
The broader vision here is a shift from human operators to human governors — moving from 100% human-driven operations toward a model where roughly 45–55% of the work is AI-powered. Fabric IQ's ontology provides the business meaning that makes this possible.
OneLake Spark Catalog & Migrating to Schema-Enabled Lakehouses
This was probably the most operationally impactful session of the day for anyone running Fabric at scale. It covered how Spark discovers data and the significant improvements coming with the new OneLake Spark Catalog.
The old vs. new Spark catalog
The legacy Spark catalog maps table names to file locations and works within a single workspace. Its problems: chatty, slow, no row-level or column-level security, no VNet support.
The new OneLake Spark Catalog is a fundamentally different architecture:
- Metadata lives with the data in the Delta log — no separate metastore.
- Immediate table detection after ingestion.
- Multi-workspace queries.
- Compatible with schema-enabled lakehouses.
- Can resolve anything the tenant can access — including Databricks catalogs and external tables via shortcuts.
- Supports OneLake row-level security and column-level security.
Naming conventions matter
- Local context:
SELECT * FROM product— only for the linked lakehouse. - 2-part name:
market.product— ambiguous (schema or another lakehouse?). - 3-part name: cross data-source.
- 4-part name: fabric-wide cross-workspace query.
For non-schema lakehouses, use dbo as the schema name.
Should you switch to schema-enabled lakehouses?
What breaks when you switch
The switch is instant and irreversible. Tables move under the dbo schema — no data files are moved, but the following will break:
- Notebook code referencing tables by old names.
- RLS/CLS policies (they need schema references updated).
- Semantic models.
- Shortcuts pointing to the lakehouse.
Migration checklist
- Copy catalog elements to OneLake (Microsoft has a script for this).
- Enable the OneLake catalog.
- Test notebook runs.
- Refactor notebooks to reference tables with the
dboschema. - Run PySpark scripts to schema-enable the lakehouses.
- Update OneLake security policies, semantic models, and Direct Lake references using the provided scripts.
- Update all shortcuts pointing to updated lakehouses. Note: there is currently no lineage information to help find shortcuts automatically.
- The SQL analytics endpoint won't change.
What's coming
Microsoft is working on a migration assistant, a Copilot agent with Fabric skills, and a fully automated switch process.
These are personal session notes from FabCon 2026 — Part 2. Accuracy reflects my understanding at the time; always verify details against official Microsoft documentation before making architectural decisions.
