Development and operations
Develop, validate, publish and operate streaming lakehouse workloads in one workspace. Realfuture connects shared catalogs, AI assistance, job drafts, query results and reusable resources to the runtime where each job will run.
Choose a workflow
| Workflow | Suitable work | Development and publishing |
|---|---|---|
| Flink SQL | Filtering, transformations, joins, aggregations and continuous processing | Catalog references, completion, validation, previews and runtime configuration |
| Flink CDC Pipeline | Table-level or whole-database synchronization to lakehouse and other targets | YAML configuration for sources, sinks, routing, transformations and schema evolution |
| Visual ETL | Common processing configured from table schemas | Source tables, field mappings, filters, joins and generated SQL |
| JAR | Existing Flink applications or custom logic | Application resources, entry class, arguments and dependencies |
| Git | Building and publishing from source | Repository references, build logs, artifacts and commit differences |
Visual ETL currently supports one sink table; multiple sources require corresponding join conditions. Use SQL or applications for more complex processing. Existing Flink JARs can be deployed as application resources after checking their dependencies and runtime compatibility.
Real-time data integration
Choose Flink SQL for table-based processing and custom SQL logic, or Flink CDC Pipeline for configuration-based replication of selected tables or entire databases.
- Connect sources. Select compatible source and sink resources. Configure database or table selection, connection properties and the initial snapshot or incremental startup behavior supported by the source.
- Define rules in Realfuture. Use the CDC editor’s configuration hints and validation to set table routing, row filters, transformations and field mappings. AI can help draft the YAML and explain validation errors.
- Write to the target. Publish the pipeline to a configured runtime and continuously synchronize data into lakehouse or analytics tables. Inspect job status, checkpoints and target data as part of operations.
Schema evolution is part of the CDC pipeline configuration. Use the schema change policy and supported sink options to control how source DDL changes are handled downstream. Test the specific changes you need, such as adding, renaming or changing a column, against the chosen source, Flink CDC release and sink. Flink SQL jobs and CDC pipelines have different schema-handling behavior.
Prepare and validate
Browse objects through catalogs and check fields, types and keys. Prepare connectors, formats, UDFs and application resources. After source schema changes, check platform metadata, SQL and target tables before updating jobs.
SQL completion and validation use catalog, column and function context. Resolve syntax, type and reference errors, then validate logic with data. CDC defines the pipeline through configuration; visual ETL exposes generated SQL; Git workflows provide build status, logs and artifacts.
Queries and previews
| Path | Execution location | Useful for |
|---|---|---|
| Flink SQL queries and previews | Configured Flink query environment | Data reads, processing results and output schemas |
| Native SQL | Catalog database, connected through JDBC | Native queries and source or target data inspection |
Native SQL supports JDBC, Doris, StarRocks, Oracle and Kingbase catalog types, with matching database and driver versions. It uses database SQL semantics rather than arbitrary cross-system federation. Configure a dedicated read-only database account for production queries.
Queries access actual data systems. Choose appropriate accounts, data ranges and result sizes, and cancel continuous queries when finished. Validation and previews do not replace full reconciliation or business acceptance.
Resources and publishing
Resources include applications, catalogs, SQL connectors, pipeline connectors, formats and UDFs, with property parsing, grouping and dependency configuration.
- Save the draft and check SQL/YAML, artifacts, arguments and dependencies.
- Select a target and check Flink version, storage, identity and runtime parameters.
- Publish the draft and inspect build and submission details.
- Open job operations to confirm status, logs and output.
One Realfuture installation can target multiple YARN and Kubernetes clusters. Choose the environment for each workload, with its own cluster identity, namespace or queue, storage and runtime settings. The platform can manage deployments across clouds and data centers when the required connectivity and permissions are in place.
Draft history rollback restores development definitions. Running stateful jobs also require publishing, restart and checkpoint/savepoint compatibility decisions. Git builds and history do not imply a complete approval or automatic environment promotion workflow.
Operations and diagnosis
Investigate status, runtime instances, release versions, metrics, logs and checkpoints together. Startup reconciliation and lost-connection checks help restore observation of external jobs. Source, runtime and target failures still require separate diagnosis.
Identify whether a failure happened during build, submission, initialization, processing or output before selecting an action. Confirm available state, compatible versions and expected data behavior before migration or recovery.
Batch jobs can be scheduled. Validate trigger times, timezone and failure handling against the configured schedule, and include monitoring and alert dependencies in deployment planning.
AI assistance and team access
AI assistance supports SQL and CDC generation, ETL definitions, validation-error explanations and job diagnosis. Attach relevant table schemas to give generation the context of your data. Review a candidate before applying it to the editor, then validate and test it before publishing.
For diagnosis, AI can request scoped, read-only evidence such as job configuration, metrics, exceptions, checkpoints and limited log excerpts. It provides an investigation for the operator; it does not autonomously restart or modify production jobs.
Configure an enabled model provider and model before use. Depending on the task, the provider receives instructions, editor excerpts, referenced schemas and selected diagnostic evidence. The platform applies workspace scope, redaction and size limits. Review provider endpoints and data handling under your deployment policy; model access and AI capabilities are part of the agreed edition and service scope.
Workspaces, permissions and operation records organize team access. Configure database account permissions separately from platform permissions; exact roles and publishing controls depend on the installed release.
Explore streaming lakehouse development or book a demo.