Streaming lakehouse
Realfuture is an AI-driven streaming lakehouse platform. It connects ingestion, catalogs, AI-assisted development, queries and operations across compute, storage and analytics systems.
The parts of a lakehouse pipeline
The following stages describe how the platform connects a streaming lakehouse workflow. Choose the storage and serving systems for your workload; a pipeline does not need to pass through every system listed here.
| Stage | Typical work | Platform capabilities |
|---|---|---|
| Ingestion | Capture database changes or consume message streams | Flink CDC pipelines, Flink SQL connectors, routing, transformations and job publishing |
| Lakehouse storage | Write Paimon lake tables or Fluss streaming tables | Corresponding connectors, catalogs and configuration resources |
| Processing | Filter, join or aggregate lakehouse data | Flink SQL, visual ETL and custom applications, with AI assistance during development |
| Validation | Inspect schemas and processing results | Catalogs, Flink SQL queries and Native SQL for supported databases |
| Downstream serving | Continuously supply systems such as Doris or StarRocks | Supported sink connectors and job operations |
| Table maintenance | Maintain files, snapshots or branches | Adapted Paimon Actions and application resources |
For supported source-to-target paths, Flink CDC can replicate an entire database and propagate schema changes into downstream tables. Configure schema evolution alongside routing and transformation rules, then validate both table structure and data in the target.
Paimon provides lake tables and table maintenance mechanisms. Fluss provides streaming storage. Doris and StarRocks provide analytics engines and database tables. Flink executes stream processing. Realfuture connects their metadata, development, AI assistance and operating workflows; business records move through the configured sources, runtimes and targets.
Catalogs connect development and data
Browse and reuse lakehouse tables through catalogs, inspecting columns, types and properties. Reference these objects in SQL or ETL, check logic through queries and validation, then publish to a runtime.
Attach relevant table schemas when asking AI to generate SQL, CDC configuration or ETL definitions. During operations, use AI diagnosis with the job’s runtime evidence to investigate ingestion or processing problems. Storage-specific administration remains with the corresponding system and supported platform integrations.
Lakehouse catalogs, the platform database, object storage and state storage have separate responsibilities. Check their addresses, identities, connectivity and versions independently. See catalogs and metadata.
Lakehouse systems and maintenance
The platform includes Paimon catalog and connector resources and adapted Flink Actions for:
- Compaction and related file maintenance.
- Snapshot and partition maintenance.
- Branch/tag management and supported rollback operations.
Actions run through corresponding application resources and Flink environments. Available actions, parameters and Hive, HDFS or object storage dependencies vary by resource version. Not every action has a dedicated visual page.
Fluss catalog configuration includes connection and table properties. Doris and StarRocks catalogs support metadata discovery, supported table operations and Native SQL queries. These integrations do not imply that Paimon maintenance actions apply to the other systems or that Realfuture provisions every storage cluster.
Maintenance can affect readers, writers and retention. Check the target table, permissions, parameters and concurrent jobs. Validate retention and recovery plans before deletion or rollback.
Validate development and production separately
During development, check references, AI-generated changes, SQL logic and query output. Before production, also test initial ingestion and incremental processing, keys and types, required schema changes, checkpoint recovery, storage access and maintenance policies.
Independent source and sink support does not validate every combination with a Flink/CDC version. Review the component checklist and define outcomes in the evaluation guide.
Explore data integration or development and publishing.