Creating and running a job that runs indefinitely

Learn how to create and run a Code Engine job that can run indefinitely and does not time out.

Typically, jobs are designed to run one time and exit with a maximum execution time.

However, suppose that you want to constantly poll a third-party data store. You can choose to create an application; however, the app port must remain open to handle HTTP requests. Instead, if you don't want to service HTTP requests, you can choose to create a job that runs without a maximum execution time and does not time out.

With Code Engine, you can choose the mode of your job. Use the default task mode for jobs where a maximum execution time applies. For these jobs, failed instances are restarted per the job retries limit.

If you want to create a job that can run indefinitely and does not time out, use daemon mode for your jobs. With this mode, runs of the job do not time out and any failed instances are automatically restarted indefinitely.

When you use daemon mode to run jobs, if you want Code Engine to automatically restart the job if there's a failure, make sure that your code stops with a nonzero exit code. If the container image that is associated with your daemon job ends with a 0 exit code, then Code Engine assumes that the job was stopped intentionally, and the job is not restarted. However, if the job fails for an unexpected reason, and the container image that is associated with your daemon job exits with a nonzero exit code, then Code Engine automatically restarts the daemon job. See Why does my daemon job not automatically restart?

With Code Engine, you pay for only the resources that you use. When your job runs in daemon mode, be aware that the job is always running until you delete the job.

Creating and running a job that runs indefinitely from the console

This example uses the IBM Cloud Code Engine samples; in particular, the helloworld sample. When you create a job that references the icr.io/codeengine/helloworld sample, and you also specify the mode as daemon, this job runs without a maximum execution time and failed instances are restarted indefinitely. If mode is not specified, by default, the job runs in task mode.

Create your job
1. Open the Code Engine console.
2. Select Let's go.
3. Select Job.
4. Enter a name for the job; for example, myjob.
5. Select a project from the list of available projects. You can also create a new one. Note that you must have a selected project to create a job.
6. Choose the code to run. You can create your job from an image in a public registry, private registry, or in IBM Cloud Container Registry. You can also create your job from repository or local source code. For this example, use the Code Engine sample image, icr.io/codeengine/helloworld.
7. In the Resources & scaling section, specify daemon for the mode in the Resiliency settings. When you specify daemon mode, job runs for this job can run indefinitely, and they do not time out. There is no retry limit for these job runs. Any job runs for these jobs are automatically restarted without limit.
8. Click Create to create the job.
From the Jobs page, click the Jobs tab to view a list of defined jobs. Click the name of your job to open the configuration.
Run the job.
1. Click Submit job to open the Submit job dialog. Review and optionally change default configuration values such as instances, CPU, memory, number of job retries, and job timeout. For more information about these options, see Options for creating and running a job.
2. Click Submit job to run your job. The system displays the status of the instances of your job on the job details page. If any of the instances of your job failed to run, these instances are automatically retried indefinitely.
(optional) View logs of the job run. From the console, you can view job logs from the console for your job runs. Notice in the log output for the helloworld sample, that the JOB_MODE environment variable displays the mode of the job.

Creating and running a job that runs indefinitely with the CLI

Before you begin

Set up your Code Engine CLI environment.
Create and work with a project.

This example uses the IBM Cloud Code Engine samples; in particular, the helloworld sample. When you create a job that references the icr.io/codeengine/helloworld sample, and you also specify --mode daemon, this job runs without a maximum execution time and failed instances are restarted indefinitely. If mode is not specified, by default, the job runs in task mode.

Create the myjob-daemon job, which uses the sample image icr.io/codeengine/helloworld. To run this sample image as a job in daemon mode, specify --mode daemon.

ibmcloud ce job create --name myjob-daemon --image icr.io/codeengine/helloworld --mode daemon

Example output

Creating job 'myjob-daemon'...

The following table summarizes the options that are used with the job create command in this example. For more information about the command and its options, see the ibmcloud ce job create command.

Command description
Option	Description
`--name`	The name of the job. Use a name that is unique within the project. This value is required. The name must begin and end with a lowercase alphanumeric character. The name must be 63 characters or fewer and can contain letters, numbers, and hyphens (-).
`--image`	The name of the image that is used for runs of the job.
`--mode`	The mode for runs of the job. Valid values are `task` (default) and `daemon`. In `daemon` mode, there is no timeout value and failed instances are restarted indefinitely.

(optional) Use the job get command to display information about your job.

ibmcloud ce job get --name myjob-daemon

Example output

[...]
Name:          myjob-daemon
ID:            abcdefgh-abcd-abcd-abcd-1a2b3c4d5e6f
Project Name:  myproject
Project ID:    01234567-abcd-abcd-abcd-abcdabcd1111
Age:           2d15h
Created:       2022-04-14T16:10:11-04:00

Image:                icr.io/codeengine/helloworld
Resource Allocation:
  CPU:     1
  Memory:  4G

Runtime:
    Mode:                  daemon
    Array Indices:         0
    Array Size:            1
    Max Execution Time:    7200
    Retry Limit:           3
[...]

Run your job. This example command runs the myjobrun-daemon job run that is based on the myjob-daemon job configuration.
```
ibmcloud ce jobrun submit --name myjobrun-daemon --job myjob-daemon
```

(optional) View the details of your job run.

ibmcloud ce jobrun get --name myjobrun-daemon

Example output

Getting jobrun 'myjobrun-daemon'...
Getting instances of jobrun 'myjobrun-daemon'...
Getting events of jobrun 'myjobrun-daemon'...
For troubleshooting information visit: https://cloud.ibm.com/docs/codeengine?topic=codeengine-troubleshoot-job.
Run 'ibmcloud ce jobrun events -n myjobrun-daemon' to get the system events of the job run instances.
Run 'ibmcloud ce jobrun logs -f -n myjobrun-daemon' to follow the logs of the job run instances.
OK

Name:          myjobrun-daemon
ID:            abcdefgh-abcd-abcd-abcd-1a2b3c4d5e6f
Project Name:  myproject
Project ID:    01234567-abcd-abcd-abcd-abcdabcd1111
Age:           2d15h
Created:       2022-04-14T16:10:11-04:00

Job Ref:              myjob-daemon
Image:                icr.io/codeengine/helloworld
Resource Allocation:
  CPU:                1
  Ephemeral Storage:  400M
  Memory:             4G

Runtime:
    Mode:                  daemon
    Array Indices:         0
    Array Size:            1
    JOP_ARRAY_SIZE Value:  1
    Max Execution Time:    7200
    Retry Limit:           3

Status:
  Instance Statuses:
    Running:  1
  Conditions:
    Type     Status  Last Probe  Last Transition
    Pending  True    27s         27s
    Running  True    4s          4s

Events:
  Type    Reason   Age               Source                Messages
  Normal  Updated  4s (x3 over 26s)  batch-job-controller  Updated JobRun "myjobrun-daemon"

Instances:
  Name                 Running  Status   Restarts  Age
  myjobrun-daemon-0-0  1/1      Running  0         26s

(optional) View logs for the job run. With the CLI, you can view job logs for all instances of a job run or for a particular job run instance. The following example displays job logs for all instances of the myjobrun-daemon job run.

ibmcloud ce jobrun logs --name myjobrun-daemon

Example output

Getting logs for all instances of job run 'myjobrun-daemon'...
Getting jobrun 'myjobrun-daemon'...
Getting instances of jobrun 'myjobrun-daemon'...
OK

myjobrun-daemon-0-0/myjob-daemon:
Hello from helloworld! I'm a batch job! Index: 0

Hello World from:
. ___  __  ____  ____
./ __)/  \(    \(  __)
( (__(  O )) D ( ) _)
.\___)\__/(____/(____)
.____  __ _   ___  __  __ _  ____
(  __)(  ( \ / __)(  )(  ( \(  __)
.) _) /    /( (_ \ )( /    / ) _)
(____)\_)__) \___/(__)\_)__)(____)

Some Env Vars:
--------------
CE_DOMAIN=us-south.codeengine.appdomain.cloud
CE_JOB=myjob-daemon
CE_JOBRUN=myjobrun-daemon
CE_SUBDOMAIN=glxo4k7nj7d
HOME=/root
HOSTNAME=myjobrun-daemon-0-0
JOB_INDEX=0
JOB_MODE=daemon
KUBERNETES_PORT=tcp://172.21.0.1:443
KUBERNETES_PORT_443_TCP=tcp://172.21.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=172.21.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=172.21.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
z=Set env var 'SHOW' to see all variables

Now that your job is created in daemon mode, the job runs indefinitely. Because there is no timeout and failed instances are restarted indefinitely, the --maxexecutiontime and --retrylimit options are not allowed when you run a job in daemon mode.

Notice that the JOB_MODE environment variable displays the mode of the runs of the job.

Stopping a job that runs indefinitely

When you run a job in daemon mode, it can run indefinitely, since it runs without a maximum execution time and any failed instances are automatically retried indefinitely. To stop a job that is in daemon mode, delete the specific job run or delete the job.

Stopping a job that runs indefinitely from the console

To delete a specific job run, go to the job page for your specific job. Go to the Job runs tab. Delete the job run that you want to remove.
To delete a job, go to the Jobs page, click the name of the job that you want to remove. Deleting a job deletes the job, its configuration, and any job runs associated with the job.

Stopping a job that runs indefinitely with the CLI

To delete a specific job run with the CLI, use the jobrun delete command. You can optionally use the -f option to force the delete of a job run without confirmation.
```
ibmcloud ce jobrun delete --name myjobrun-daemon -f
```
Example output
```
Deleting job run 'myjobrun-daemon'...
OK
```
To delete a job with the CLI, use the job delete command. You can optionally use the -f option to force the delete of a job without confirmation. When you delete a job, any job runs for this job are also deleted. The following example deletes the myjob-daemon job,
```
ibmcloud ce job delete --name myjob-daemon -f
```
Example output
```
Deleting job 'myjob-daemon'...
OK
```