Running jobs with Lithops framework

Lithops is an open source framework that designed to massively scale your Python applications. Lithops provides a simple push to the IBM Cloud® Code Engine experience so that you can focus on your Python code, while Lithops focuses on the deployment of your code at massive scale while monitoring executions, obtaining results, and much more. Lithops enables native Python integration with Code Engine by using the Lithops API. For more information about Lithops, see Lithops quick start guide.

Running your first flow by using the Lithops framework

Before you can run jobs that reference the Lithops framework, you must first install Lithops and set up a storage backend.

Before you begin

Install Code Engine CLI.
Create and work with a Code Engine project.
Find and set your kubeconfig environment variable.

Installing Lithops

Install Lithops.
Create the Lithops configuration file.
Install Docker (community edition) version.
Log in to Docker.
```
docker login
```

Setting up a storage backend for Lithops

Select a supported storage backend; for example, IBM Cloud Object Storage.

To set up IBM Cloud Object Storage,

Create an IBM Cloud Object Storage account.
Create a bucket in the region that you want to use.
In the side navigation, click Endpoints to find your API endpoint. You must copy both the public and private endpoints of the region where you created your bucket.
Create the credentials to access to your IBM Cloud Object Storage account. Choose one option

Option 1 (API key) for Lithops

From the IBM Cloud Object Storage navigation, click Service Credentials.
Click New credential + and provide a name and select a role.
Click Add to generate service credential.
Click View credentials and copy the apikey value.

Edit your lithops config file and add the following keys.

lithops:
    storage_backend: ibm_cos

ibm_cos:
    endpoint   : <REGION_ENDPOINT>  
    private_endpoint : <PRIVATE_REGION_ENDPOINT>
    api_key    : <API_KEY>

Option 2 (IBM Cloud Object Storage HMAC credentials) for Lithops

From the IBM Cloud Object Storage navigation, click Service Credentials.
Click New credential + and provide a name and select a role.
Click Advanced options and enable the option Include HMAC Credential.
Click Add to generate service credential.
Click View credentials and copy the access_key_id and secret_access_key values.

Edit your lithops config file and add the following keys:

lithops:
    storage_backend: ibm_cos

ibm_cos:
    endpoint   : <REGION_ENDPOINT>  
    private_endpoint : <PRIVATE_REGION_ENDPOINT>
    access_key    : <ACCESS_KEY_ID>
    secret_key    : <SECRET_KEY_ID>

Option 3 (IBM IAM API key) for Lithops

If you don't have an IAM API key, navigate to the IBM IAM dashboard.
Click Create an IBM Cloud API Key. Enter a name and optional description for your API key and click Create.
Copy the generated IAM API key (You can see the key only when you create it, so make sure to copy it).

Edit your lithops config file and add the following keys:

lithops:
    storage_backend: ibm_cos

ibm:
    iam_api_key: <IAM_API_KEY>

ibm_cos:
    endpoint   : <REGION_ENDPOINT>  
    private_endpoint : <PRIVATE_REGION_ENDPOINT>

Deploy your first Code Engine job by using Lithops

Run the following hello world example,

import lithops

iterdata = [1,2,3,4,5]

def my_map_function(data):
    return data + 1

if __name__ == '__main__':
    pw = lithops.function_executor(type = "serverless", backend='code_engine') 
    futures = pw.map(my_map_function, iterdata)
    print (pw.get_result())
    pw.clean()

Next steps for Lithops

You can find more examples and use cases, as well as ask questions, on the Lithops project page.