Development, validation and publishing
Realfuture organizes data objects, job drafts, query results, resources and runtime configuration in a development workspace. Teams can reuse table definitions and application resources while choosing the appropriate workflow.
Choose a workflow
| Workflow | Suitable work | Development and publishing |
|---|---|---|
| Flink SQL | Filtering, transformations, joins, aggregations and continuous processing | Catalog references, completion, validation, previews and runtime configuration |
| CDC Pipeline | Continuous synchronization from supported databases to lakehouse and other targets | YAML source, sink, routing and transformation configuration |
| Visual ETL | Common processing configured from table schemas | Source tables, field mappings, filters, joins and generated SQL |
| JAR | Existing Flink applications or custom logic | Application resources, entry class, arguments and dependencies |
| Git | Building and publishing from source | Repository references, build logs, artifacts and commit differences |
Visual ETL currently supports one sink table; multiple sources require corresponding join conditions. Use SQL or applications for more complex processing. Existing JARs can remain in use without being rewritten to adopt the platform.
Prepare and validate
Browse objects through catalogs and check fields, types and keys. Prepare connectors, formats, UDFs and application resources. After source schema changes, check platform metadata, SQL and target tables before updating jobs.
SQL completion and validation use catalog, column and function context. Resolve syntax, type and reference errors, then validate logic with data. CDC defines the pipeline through configuration; visual ETL exposes generated SQL; Git workflows provide build status, logs and artifacts.
Queries and previews
| Path | Execution location | Useful for |
|---|---|---|
| Flink SQL queries and previews | Configured Flink query environment | Data reads, processing results and output schemas |
| Native SQL | Catalog database, connected through JDBC | Native queries and source or target data inspection |
Native SQL supports JDBC, Doris, StarRocks, Oracle and Kingbase catalog types, with matching database and driver versions. It uses database SQL semantics rather than arbitrary cross-system federation. Configure a dedicated read-only database account for production queries.
Queries access actual data systems. Choose appropriate accounts, data ranges and result sizes, and cancel continuous queries when finished. Validation and previews do not replace full reconciliation or business acceptance.
Resources and publishing
Resources include applications, catalogs, SQL connectors, pipeline connectors, formats and UDFs, with property parsing, grouping and dependency configuration.
- Save the draft and check SQL/YAML, artifacts, arguments and dependencies.
- Select a target and check Flink version, storage, identity and runtime parameters.
- Publish the draft and inspect build and submission details.
- Open job operations to confirm status, logs and output.
Draft history rollback restores development definitions. Running stateful jobs also require publishing, restart and checkpoint/savepoint compatibility decisions. Git builds and history do not imply a complete approval or automatic environment promotion workflow.
Operations and diagnosis
Investigate status, runtime instances, release versions, metrics, logs and checkpoints together. Startup reconciliation and lost-connection checks help restore observation of external jobs. Source, runtime and target failures still require separate diagnosis.
Identify whether a failure happened during build, submission, initialization, processing or output before selecting an action. Confirm available state, compatible versions and expected data behavior before migration or recovery.
Batch jobs can be scheduled. Validate trigger times, timezone and failure handling against the configured schedule, and include monitoring and alert dependencies in deployment planning.
AI assistance and team access
Optional AI assistance can use selected schemas and job context to generate SQL/CDC, explain validation errors and investigate runtime problems. Review, validate and test generated output. Before using an external model, establish which SQL, metadata, errors and diagnostic context can be sent.
Workspaces, permissions and operation records organize team access. Configure database account permissions separately from platform permissions; exact roles and publishing controls depend on the installed release.
Explore streaming lakehouse development or book a demo.