REFERENCE ARCHITECTURE · OPEN LAKEHOUSE
An Open Lakehouse Designed Around Access, Trust and Portability.
Unify object-backed tables, distributed SQL access, analytics engineering and metadata visibility in one composable pattern. Choose Trino Open Source or Starburst for query access, MinIO AIStor for object and native Iceberg table services, dbt for transformation workflows, and DataHub for metadata discovery and lineage where integrations are validated.
For chief data architects · data platform leads · analytics engineering teams. Stack: MinIO AIStor · Trino OSS or Starburst · dbt Core / dbt OSS · DataHub Core
THE CHALLENGE
Open Data Is Only Useful When It Is Trusted and Accessible.
Data spread across warehouses, operational databases and file stores is expensive to duplicate and hard to govern. Proprietary formats tie analytics to one engine. Moving to open table formats on object storage solves part of the problem, but it also introduces new decisions: which catalog, which SQL engine and support route, how models are tested and promoted, where lineage is captured and where access control is actually enforced.
- Format and engine lock-in — Analytical data is tied to a single engine, which limits choice and makes migrations costly.
- Duplicated copies — The same datasets are copied into several platforms, each with its own definitions and quality.
- Unclear ownership and lineage — Teams cannot see who owns a table, how it was built or what depends on it.
- Governance gaps — Catalogs, engines and storage each have their own permission models, and nobody is sure which one is authoritative.
ARCHITECTURE OVERVIEW
Composable Layers With Clear Responsibilities
Data lands once in customer-controlled object storage as Apache Iceberg tables. One distributed SQL engine, chosen as Trino OSS or Starburst, reads and writes those tables through a catalog. dbt builds and tests models through that engine, and DataHub collects the metadata. Authorisation stays with the engine, the catalog and the storage.
Figure 1. Open Iceberg Lakehouse & Governed Federation reference architecture (conceptual). Colour key: see How to read our diagrams.
Logical Data Path
- Land approved dataLand approved source datasets from operational platforms, files or event streams into customer-controlled object and Iceberg table storage.
- Store Iceberg tablesUse MinIO AIStor Tables for native Iceberg catalog and S3/object workloads as supported by the selected AIStor release and table interface.
- Configure one SQL engineConfigure Trino OSS OR Starburst catalogs to access permitted tables through supported Iceberg REST/catalog mechanisms, with separately tested auth and feature behavior.
- Model and test with dbtUse dbt Core/dbt OSS where compatible adapters and runtime versions support the chosen engine; define quality tests, model ownership, promotion and lineage capture.
- Collect metadataCollect metadata into DataHub Core with supported ingestion connectors and ownership policies; data authorization stays enforced by query engines, storage or adjacent control points.
- Plan resilience per layerPlan resource isolation, cluster capacity, snapshot/backup/restore, DR, catalog high availability and object-store durability as independent concerns.
Conceptual pattern. Connectors, licensed capabilities, versions, security policies and topology are validated in a project-specific architecture.
Last reviewed: — · Version scope: —
COMPONENT ROLES
Who Does What in the Stack
| Component | Classification | Role | Qualification |
|---|---|---|---|
| MinIO AIStor (object storage + AIStor Tables) | Strategic partner | S3-compatible object storage and native Iceberg table/catalog services | Table interface, catalog API and features depend on the selected AIStor release and licence |
| Trino OSS | Independent OSS support (alternative to Starburst) | Distributed SQL over Iceberg tables and approved federated sources | Operated by Bassirah under independent contract; no implied upstream SLA |
| Starburst Enterprise | Strategic partner (alternative to Trino OSS) | Trino-based distributed SQL with licensed enterprise capabilities | Subject to product licensing and entitlements |
| dbt Core / dbt OSS | Independent OSS support | Transformation models, tests, documentation and promotion | Requires a compatible adapter and runtime version for the chosen engine |
| DataHub Core | Independent OSS support | Metadata discovery, ownership, glossary and lineage | Does not enforce data authorisation; ingestion connectors validated per source |
| Source systems & ingestion | Customer-owned or separately selected | Operational platforms, files and event streams landing approved data | Landing contracts, schema evolution and quality rules agreed per source |
Links: MinIO AIStor · Trino OSS · Starburst · dbt · DataHub
DESIGN TRADEOFFS
Architecture Decision Points
The same lakehouse pattern can be delivered open-source-first or partner-led. Choose per layer, based on support expectations, licensed capabilities and operating model.
OSS-First Versus Partner-Led
| Decision | Trino OSS / OSS-first path | Partner-led enterprise path |
|---|---|---|
| Distributed SQL | Trino OSS, operated by Bassirah under independent contract | Starburst Enterprise, subject to product licensing/entitlements |
| Operational support | Bassirah incident response without implied upstream SLA | Vendor-aligned escalation where explicitly contracted |
| Storage/table services | AIStor S3/Iceberg; validate required release and API behavior | Same AIStor patterns with supported Starburst interoperability |
| Metadata and models | DataHub, dbt with tested connectors and adapters | Same tools where compatible, possibly plus commercial controls |
Further Design Decisions
| Decision | Options | Choose based on | Validate with |
|---|---|---|---|
| Iceberg catalog | AIStor Tables (Iceberg REST) · Another supported REST, Hive or JDBC catalog | Release support, HA needs, multi-engine access | Catalog failover test; engine compatibility with the chosen catalog |
| Data landing | Batch file landing · Streaming / CDC into Iceberg · Engine-managed INSERT/MERGE | Freshness needs, source capabilities, write concurrency | Commit conflicts, small-file growth, freshness measurement |
| Federation scope | Lakehouse tables only · Plus selected federated sources | Data duplication cost versus query performance and source load | Connector semantics, pushdown behaviour, source impact |
| Table maintenance | Scheduled compaction, snapshot expiry and orphan-file cleanup | Write patterns, retention and time-travel needs | Query latency trend, storage growth, restore from snapshot |
| Authorisation point | Engine access control · Catalog-level controls · Storage policies (combined as designed) | Where policies can be enforced consistently | Negative access tests per role; audit log review |
| Coexistence | One engine (recommended) · Phased Trino OSS → Starburst migration | Commercial and support roadmap | Deliberate migration plan with clear end state |
- Trino OSS and Starburst are alternatives within one federated-query decision, unless a phased coexistence strategy is deliberately required.
- Federated SQL access can reduce data duplication for selected queries; ingestion, modelling, metadata and governance are still needed.
- DataHub is the metadata and governance control plane, not an authorisation enforcement point.
Compare Trino OSS and Starburst → · Starburst partner profile →
Choosing between Trino OSS and Starburst?
We will compare support, licensed capabilities and operating responsibilities for your workloads.
SECURITY & RESILIENCE
Enforce Access Where Data Is Served. Recover Each Layer Independently.
Identity & access
Integrate the SQL engine, catalog and storage with your identity provider. Enforce table, column and row policies in the engine and catalog, and bucket policies in storage, as supported by the chosen products.
Policy consistency
Document which control point enforces each rule. DataHub records ownership and classification to inform policy, but enforcement happens in the engine, catalog or storage.
Encryption & keys
Encrypt traffic between engine, catalog and storage, and data at rest in object storage, with keys managed in your KMS where supported.
Residency & egress
Customer-controlled storage and compute designed to align with your data residency and regulatory requirements. Review all outbound connections and telemetry.
Backup, snapshots & DR
Treat object-store durability, catalog backup and HA, snapshot retention and engine configuration as separate recovery concerns with their own RPO/RTO.
Monitoring & capacity
Monitor query queues, cluster resources, catalog latency, commit failures, file counts and storage growth, with resource groups to isolate workloads.
PROOF & OUTCOMES
Validate Compatibility Before You Commit
Open components only work together when versions, connectors and behaviours have been tested for your workloads. We prove the combination before production.
Validation Checklist
- AIStor release, Iceberg catalog API and table features confirmed for the chosen engine
- Trino OSS or Starburst catalog configuration, authentication and feature behaviour tested
- dbt adapter and runtime versions validated, with tests, ownership and promotion defined
- DataHub ingestion connectors validated for the engine, dbt and catalog; lineage coverage checked
- Negative access tests per role at each enforcement point
- Snapshot restore, catalog recovery and object-store replication drills
- Table maintenance jobs scheduled and their effect measured
- Resource isolation and concurrency tested with representative workloads
Measurement Plan
| Metric | How it is measured | Target |
|---|---|---|
| Time to insight | Time from source data landing to a tested, documented model available to analysts | Agreed per domain |
| Query performance | p95 / p99 latency for representative queries at agreed concurrency | Agreed in PoC acceptance criteria |
| Metadata coverage | Share of tables with owner, description, classification and lineage in DataHub | Agreed with data governance |
| Access policy evidence | Pass rate of negative access tests and completeness of access audit logs | 100% pass for defined policies |
| Recovery | RPO/RTO achieved for catalog, tables and engine configuration in a drill | Agreed per layer |
| Storage economics | Storage and compute cost at target retention compared with current baseline | Assessed through workload and TCO modelling |
What You Receive
- Target lakehouse architecture and version compatibility matrix
- Engine route recommendation (Trino OSS or Starburst) with decision record
- Access-control model mapping each policy to its enforcement point
- dbt project structure, test standards and promotion workflow
- DataHub ingestion and ownership setup
- Recovery and table-maintenance runbooks
Open Iceberg Lakehouse FAQs
Should we use Trino OSS or Starburst?
Both provide distributed SQL based on Trino. Trino OSS is operated by Bassirah under an independent engineering contract, without implied upstream SLAs. Starburst Enterprise adds licensed capabilities and vendor-aligned escalation where contracted. They are alternatives within one decision; we help you choose based on support needs, features and operating model.
What are AIStor Tables?
AIStor Tables is a MinIO AIStor capability that provides native Apache Iceberg table and catalog services alongside S3-compatible object storage. Supported interfaces and features depend on the AIStor release, so we validate them against your chosen query engine before design sign-off.
Does DataHub control who can access our data?
No. DataHub Core provides metadata discovery, ownership, glossary and lineage. Data authorisation is enforced by the query engine, the catalog, storage policies or adjacent control points. DataHub information helps define and review those policies.
Does federation remove the need for ETL?
No. Federated SQL access can reduce data duplication for selected queries, but most lakehouses still need ingestion, modelling, quality testing, metadata and governance. We decide which data to land as Iceberg tables and which to query in place.
Which dbt version do you support?
Bassirah supports dbt Core and dbt OSS releases listed in our current support matrix, using adapters validated for the chosen engine. Adapter compatibility is checked for your engine and versions before go-live.
Is this lakehouse portable between engines?
Apache Iceberg is an open table format that multiple engines can read and write, which improves portability. Actual portability depends on catalog compatibility, the table features in use and each engine’s Iceberg support, so we test it rather than assume it.
Related Pages
Build an Open Lakehouse You Can Trust.
Share your sources, workloads, governance model and support expectations. We will define the target architecture, validate compatibility and plan a proof of concept.