Catalogs and metadata
Catalogs are the shared data context for streaming lakehouse development and AI assistance. Realfuture connects built-in and external catalogs to a workspace so teams can explore and reuse databases, tables, views and functions.
This screen from the product documentation uses the earlier Awestream name.
What catalogs manage
| Catalog | Purpose |
|---|---|
| Built-in StreamPark catalog | Store and reuse metadata objects managed by the platform |
| JDBC, Oracle and Kingbase | Discover database tables and inspect their schemas |
| Kafka catalog | Filter topics and map supported message formats to table schemas |
| Paimon and Fluss | Browse lakehouse and streaming-storage tables and reuse their definitions |
| Doris and StarRocks | Browse analytics databases and tables for development and supported queries |
| Hive | Access databases and table metadata through the configured Hive catalog |
Available object operations depend on the catalog type. Where supported, teams can also create and reuse views and function definitions.
Catalogs manage structure and objects; connectors provide runtime reads and writes. A catalog integration does not imply every connector direction or CDC behavior is supported.
Connect and reuse
- Create a catalog in the workspace and configure its type, connection, authentication and required resources.
- Browse databases and tables; check columns, types, keys and DDL.
- Reference existing objects in SQL completion, validation, ETL or queries; attach relevant tables as context for AI development.
- Refresh metadata after schema changes and check affected jobs.
Form and SQL operations depend on the catalog. Some external catalogs primarily provide discovery. Their capabilities can differ from the built-in catalog.
This example uses a catalog named streampark; use the name configured in your workspace:
SHOW CATALOGS;
USE CATALOG streampark;
SHOW DATABASES;
USE default;
SHOW TABLES;
Inspect an existing table before querying or processing it. Use catalog.database.table to identify an object by its full path.
Kafka discovery
The Kafka catalog can filter topics by exact name, wildcard or regular expression. It can infer schemas from supported structured message formats; raw messages use a single string column. Metadata and additional columns can supply fields needed by a job.
Inferred schemas describe sampled data. Check empty topics, missing fields and varying message types; a sample does not guarantee the structure of future messages.
Schema changes and SQL lineage
The platform provides metadata refresh and corresponding schema override and alignment operations. Distinguish logical metadata in the platform from the actual external schema: not every edit changes the external system.
Catalog schema management and CDC schema evolution serve different purposes. Catalog operations keep development definitions usable. CDC schema evolution handles supported source schema changes as a running pipeline writes to its target, according to its configured policy. Refreshing a catalog alone does not update a running pipeline or its target tables.
SQL lineage describes table and column origins for supported INSERT/CTAS statements. It does not cover arbitrary JARs, external programs or every enterprise governance requirement.
Queries and publishing
Catalogs provide shared development context across sources, lake tables and analytics systems, including Doris and StarRocks. Accounts, SQL semantics and execution environments depend on the query path. Published jobs also need compatible connectors, dependencies and runtime configuration.
Continue with development and publishing, streaming lakehouse development and compatibility.