Streaming lakehouse development
Realfuture Enterprise Streaming Lakehouse Platform connects ingestion, metadata, development and operations. Teams can build continuous pipelines around databases, streams and lakehouse tables while managing processing and maintenance tasks.
The parts of a lakehouse pipeline
This is a reference architecture. Each stage needs a validated component combination.
| Stage | Typical work | Platform capabilities |
|---|---|---|
| Ingestion | Capture changes from databases or messaging systems | CDC pipelines, SQL connectors and job publishing |
| Lakehouse storage | Write into components such as Paimon | Corresponding connectors, catalogs and configuration resources |
| Processing | Filter, join or aggregate lakehouse data | Flink SQL, visual ETL and custom applications |
| Validation | Inspect schemas and processing results | Catalogs, Flink SQL queries and Native SQL for supported databases |
| Downstream serving | Continuously supply systems such as Doris or StarRocks | Supported sink connectors and job operations |
| Table maintenance | Maintain files, snapshots or branches | Adapted Paimon Actions and application resources |
The corresponding components provide storage and table formats; Flink executes processing; Realfuture organizes development, configuration, publishing and operations. Data moves through sources, runtimes and targets.
Catalogs connect development and data
Browse and reuse lakehouse tables through catalogs, inspecting columns, types and properties. Reference these objects in SQL or ETL, check logic through queries and validation, then publish to a runtime.
Lakehouse catalogs, the platform database, object storage and state storage have separate responsibilities. Check their addresses, identities, connectivity and versions independently. See catalogs and metadata.
Paimon development and maintenance
The platform includes Paimon catalog and connector resources and adapted Flink Actions for:
- Compaction and related file maintenance.
- Snapshot and partition maintenance.
- Branch/tag management and supported rollback operations.
Actions run through corresponding application resources and Flink environments. Available actions, parameters and Hive, HDFS or object storage dependencies vary by resource version. Not every action has a dedicated visual page.
Maintenance can affect readers, writers and retention. Check the target table, permissions, parameters and concurrent jobs. Validate retention and recovery plans before deletion or rollback.
Validate development and production separately
During development, check references, SQL logic and query output. Before production, also test initial ingestion and incremental processing, keys and types, required schema changes, checkpoint recovery, storage access and maintenance policies.
Independent source and sink support does not validate every combination with a Flink/CDC version. Review the component checklist and define outcomes in the evaluation guide.
Explore data integration or development and publishing.