IBM Cloud Docs
Provisioning an IBM Analytics Engine serverless instance

Provisioning an IBM Analytics Engine serverless instance

You can create a serverless IBM Analytics Engine service instance:

Note that you are not able to define certain limitation and quota settings while provisioning a serverless instance. These values are predefined. See Limits and quotas for Analytics Engine instances for a list of these settings and their values.

You must have access to either the IBM Cloud® us-south (Dallas) or the eu-de (Frankurt) region.

Creating a service instance from the IBM Cloud console

You can create an instance using the IBM Cloud console. To understand the concepts behind provisioning settings in the UI, see Architecture and concepts in serverless instances.

To create an IBM Analytics Engine instance:

  1. Log into the IBM Cloud® console.

  2. Click Sevices and select the category Analytics.

  3. Search for Analytics Engine and then click on the tile to open the service instance creation page.

  4. Choose the location in which you want the service instance to be deployed. Currently, us-south and eu-de are the only supported regions.

  5. Select a plan. Currently, Standard Serverless for Apache Spark is the only supported serverless plan.

  6. Configure the instance by entering a name of your choice, selecting a resource group and adding tags.

  7. Select the default Spark runtime. You can choose between Spark 3.3, and Spark 3.4 runtimes. The runtime pre-installs the spatio-temporal, data skipping and Paquet modular encryption packages by default.

  8. Select the IBM Cloud Object Storage instance from your account that you want to use as the instance home to store instance related data.

  9. Add Spark configuration values to override default Apache Spark settings.

  10. Click Create to provision the service instance in the background.

    The newly created service is listed in your IBM Cloud® resource list under *Services.

Creating a service instance using the IBM Cloud command-line interface

To create a service instance using the IBM Cloud® command-line interface:

  1. Download and configure the IBM Cloud® CLI. Follow the instructions in Getting started with the IBM Cloud® CLI.

  2. Set the API endpoint for your region and log in:

    ibmcloud api https://{DomainName}
ibmcloud login

  3. Get the list of the resource groups for your account and select one of the returned resource group as the target resource group in which to create the IBM Analytics Engine serverless instance:

    ibmcloud resource groups
ibmcloud target -g <resource_group_name>

  4. Create a service instance:

    ibmcloud resource service-instance-create <service_instance_name> ibmanalyticsengine <plan_name> <region> -p @<path_to JSON file with cluster parameters>

    For example, for the Dallas region:

    ibmcloud resource service-instance-create MyServiceInstance ibmanalyticsengine standard-serverless-spark us-south -p @provision.json

    You can give the service instance any name you choose. Note that currently, standard-serverless-spark is the only supported serverless plan and us-south and eu-de the only supported regions.

    The provision.json file contains the provisioning parameters for the instance you want to create.

    The endpoint to your IBM Cloud Object Storage instance in the payload JSON file should be the direct endpoint. Direct endpoints provide better performance than public endpoints and do not incur charges for any outgoing or incoming bandwidth.

    This is a sample of what the provision.json file can look like. See Architecture and concepts in serverless instances for a description of the provisioning parameters in the payload.

    Note that Spark 3.3, and Spark 3.4 are supported. If you don't specify a default Spark runtime version when you create a service instance, Spark 3.3 is taken by default.

    {
  "default_runtime": {
    "spark_version": "3.3"
    },
  "instance_home": {
    "region": "us-south",
    "endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
    "hmac_access_key": "<your-hmac-access-key",
    "hmac_secret_key": "<your-hmac-secret-key"
    },
  "default_config": {
    "key1": "value1",
    "key2": "value2"
    }
}

    The IBM Cloud® response to the create instance command:

    Creating service instance MyServiceInstance in resource group Default of account <your account name> as <your user name>...
OK
Service instance MyServiceInstance was created.

Name:                MyServiceInstance
ID:                  crn:v1:staging:public:ibmanalyticsengine:us-south:a/d628eae2cc7e4373bb0c9d2229f2ece5:1e32e***-afd9-483a-b1**-724ba5cf4***::
GUID:                1e32e***-afd9-483a-b1**-724ba5cf4***
Location:            us-south
State:               provisioning
Type:                service_instance
Sub Type:
Service Endpoints:   public
Allow Cleanup:       false
Locked:              false
Created at:          2021-11-29T07:20:40Z
Updated at:          2021-11-29T07:20:42Z
Last Operation:
                    Status    create in progress
                    Message   Started create instance operation

    Make a note of the instance ID from the output. You will need the instance ID when you call instance management or Spark application management APIs. See Spark application REST API.

  5. Track instance readiness.

Creating a service instance using the Resource Controller REST API

An IBM Analytics Engine serverless instance must reside in an IBM Cloud® resource group. As a first step towards creating an IBM Analytics Engine serverless instance through the Resource Controller REST API, you need to have the resource group ID and serverless plan ID close at hand.

To create a service instance using the Resource Controller REST API:

  1. Get the resource group ID by logging into the IBM Cloud® CLI and running the following command:

    ibmcloud resource groups

    Sample result:

    Retrieving all resource groups under account <Account details..>
OK
Name      ID      Default Group   State
Default   XXXXX   true            ACTIVE

  2. Use the following resource plan ID for the Standard Serverless for Apache Spark plan:

    8afde05e-5fd8-4359-a597-946d8432dd45

  3. Get the IAM token by performing the following steps.

  4. Create an instance using the Resource Controller REST API:

    curl -X POST https://resource-controller.cloud.ibm.com/v2/resource_instances/
--header "Authorization: Bearer $token" -H 'Content-Type: application/json' -d @provision.json

    The provision.json file contains the provisioning parameters for the instance you want to create. See Architecture and concepts in serverless instances for a description of the provisioning parameters in the payload.

    Note that Spark 3.3, and Spark 3.4 are supported. If you don't specify a default Spark runtime version when you create a service instance, Spark 3.3 is taken by default.

    This is a sample of what the provision.json file can look like:

    {
  "name": "your-service-instance-name",
  "resource_plan_id": "8afde05e-5fd8-4359-a597-946d8432dd45",
  "resource_group": "resource-group-id",
  "target": "us-south",
  "parameters": {
    "default_runtime": {
      "spark_version": "3.3"
        },
        "instance_home": {
          "region": "us-south",
          "endpoint": "s3.direct.us-south.cloud-object-storage.appdomain.cloud",
          "hmac_access_key": "your-access-key",
          "hmac_secret_key": "your-secret-key"
          }
    }
}

  5. Track instance readiness.

For more information on the Resource Controller REST API for creating an instance, see Create (provision) a new resource instance.

Tracking instance readiness

To run applications on a newly created serverless instance, the instance must be in active state.

To track instance readiness:

  1. Enter the following command:
    curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/{instance_id} -H "Authorization: Bearer $token"
    Sample response:
    {
  "id": "dc0e****-eab2-4t9e-9441-56620949****",
  "state": "created",
  "state_change_time": "2021-04-21T04:24:01Z",
  "default_runtime": {
    "spark_version": "3.3",
    "instance_home": {
      "provider": "ibm-cos",
      "type": "objectstore",
      "region": "us-south",
      "endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
      "bucket": "ae-bucket-do-not-delete-dc0e****-eab2-4t**-9441-566209499546",
      "hmac_access_key": "eH****g=",
      "hmac_secret_key": "4d********76"
    },
    "default_config": {
      "spark.driver.memory": "4g",
      "spark.driver.cores": 1
    }
  }
}
  2. Check the value of the "state" attribute. It must be active before you can start running applications in the instance.

Learn more

When provisioning serverless instances, follow the recommended Best practices.