Architecture and concepts in serverless instances

IBM Analytics Engine service instance

An IBM Cloud® service is cloud extension that provides ready-for-use functionality, such as database, messaging, and web software for running code, or application management or monitoring capabilities. Services usually do not require installation or maintenance and can be combined to create applications. An instance of a service is an entity that consists of resources that are reserved for a particular application or a service.

When you create an IBM Analytics Engine from the catalog, you will give the service instance a name of your choice, select the default Spark runtime you want to associate with the instance and provide the default Spark configuration to use with the instance. Additionally, you need have to specify the Instance home, which is the storage attached to the instance for instance related data only.

Note:

• When you create an IBM Analytics Engine service instance, no costs are incurred unless you have Spark applications running or the Spark history server is accessed.
• Costs are incurred if IBM Cloud Object Storage if accessed through public endpoints, and when you enable forwarding IBM Analytics Engine logs to IBM Log Analysis.
• There is a default limit on the number of service instances permitted per IBM Cloud® account and on the amount of CPU and memory that can be used in any given IBM Analytics Engine service instance. See Limits and quotas for Analytics Engine instances. If you need to adjust these limits, open an IBM Support ticket.
• There is no limit on the number of Spark applications that can be run in an IBM Analytics Engine service instance.

Default Spark runtime

At the time of instance provisioning, you can select the Spark version to be used. Currently, you can choose between Spark 3.3 and Spark 3.4. Spark 3.3 is considered as the default version.

The runtime includes open source Spark binaries and the configuration helps you to quickly proceed with the instance creation and run Spark applications in the instance. In addition to the Spark binaries, the runtime also includes the geospatial, data skipping, and Parquet modular encryption libraries.

Across all Spark runtime version, you can submit Spark applications written in the following languages:

• Scala
• Python
• R

The following table shows the Spark runtime version and runtime language version.

The language versions are upgraded periodically to keep the runtime free from any security vulnerabilities. You can always override the Spark runtime version when you submit an application. For details on what to add to the payload, see Passing the runtime Spark version when submitting an application.

Instance home

Instance home is the storage attached to the instance for instance related data only, such as custom application libraries and Spark history events. Currently, only IBM Cloud Object Storage is accepted for instance home. This instance can be an instance in your IBM Cloud® account or an instance from a different account.

When you provision an instance using the IBM Cloud console, the IBM Cloud Object Storage instances in your IBM Cloud® account are auto discovered and displayed in a list for you to select from. If no IBM Cloud Object Storage instances are found in your account, you can use the REST APIs to update instance home after instance creation.

You can't change instance home after instance creation. You can only edit the access keys.

Default Spark configuration

You can specify default Spark configurations at the time of provisioning an Analytics Engine instance (See Provisioning an IBM Analytics Engine serverless instance). The configurations are automatically applied to the Spark applications submitted on the instance. You can also update the configurations after creating the instance. You can edit the configuration from the Configuration section in the Analytics Engine Instance details page, Analytics Engine Rest APIs or IAE CLI . Values specified as instance level defaults can be overridden at the time of submitting Spark applications.

To learn more about the various Apache Spark configurations, see Spark Configuration.