Skip to content
Navigation Menu

IBM Cloud

  • CatalogCatalog
  • Cost EstimatorCost Estimator
    • HelpHelp
      • Docs
  • Log in
  • Sign up
  • Catalog
  • Cost Estimator
  • Help
    • Docs

  • Navigation settings

Error

Change theme

This feature is in early stage, some parts of the platform might not fully support different themes yet.

  1. Catalog

Data Engine (previously SQL Query)

Read, analyze, store and stream data in Cloud Object Storage with ANSI SQL. Manage table metadata in a Hive metastore compatible catalog.

  • Date of last update: 12/05/2022
  • Docs
  • API docs
Type
  • Service
Provider
  • IBM
Last updated
  • 12/05/2022
Category
  • Databases
Compliance
  • IAM-enabled
Related links
  • API docs
  • Docs
  • Terms

Pricing plans

PlanFeatures and capabilitiesPricing

Type
  • Service
Provider
  • IBM
Last updated
  • 12/05/2022
Category
  • Databases
Compliance
  • IAM-enabled
Related links
  • API docs
  • Docs
  • Terms

Summary

Data Engine is IBM Cloud's central service for data lakes. It provides stream ingestion, data preparation, ETL, and data query from Object Storage and Kafka. It also manages tables and views in a catalog that is compatible with Hive metastore and other big data engines and services can connect to it. Data Engine supports full standard ANSI SQL to submit work as serverless jobs. There is no infrastructure to manage. The service is highly available, offers a Multi-AZ deployment, and autoscales based on your workload.

Features and capabilities

Transform data

Transform data to prepare it for all disciplines of AI by changing data content, granularity, format, and layout on disk.

Quick SQL access and easy data exploration

Query server logs, click-stream data, IoT messages, or raw analytics events using SQL directly on the files where you store them in Cloud Object Storage. Run ad-hoc SQL queries against fresh data before the data has moved through the pipeline and into the reporting stack.

Querying of archived data

Use SQL to query structured data archived from legacy systems, databases (for example, NoSQL), and/or data warehouses in Cloud Object Storage.

Integrate with your applications

Run SQLs from your application using the rich API ecosystem, comprising REST API, Python SDK, Node SDK, and IBM Cloud Function interface.

Stream data

Use SQL to stream data from Kafka into Cloud Object Storage using a permanent running job.

Catalog tables and views

Use an external Hive metastore catalog to organise your tables and views in a central catalog within the IBM Cloud.

Getting support

If you're experiencing issues with this product, go to the IBM Cloud Support Center and navigate to creating a case. Use the All products option to search for this product to continue creating the case or to find more information about getting support. Third party and community supported products might direct you to a support process outside of IBM Cloud.

Summary

Data Engine (previously SQL Query)

    Already have an account? Log in