Deploy microservices with Red Hat OpenShift on IBM Cloud

Initialize a Cloud Shell

The Red Hat OpenShift Container Platform CLI exposes commands for managing your applications, as well as lower level tools to interact with each component of your system. The CLI is available using the oc command.

To avoid installing the command line tools, the recommended approach is to use the IBM Cloud Shell.

IBM Cloud Shell is a cloud-based shell workspace that you can access through your browser. It's preconfigured with the full IBM Cloud CLI and many plug-ins and tools that you can use to manage apps, resources, and infrastructure.

In this step, you'll use the IBM Cloud shell and configure oc to point to the cluster assigned to you.

1. When the cluster is ready, click the button (next to your account) in the upper right corner to launch a Cloud shell. Make sure you don't close this window/tab.

2. Check the version of the OpenShift CLI:

   oc version
   

   The version needs to be at minimum 4.12.x, otherwise install the latest version by following these instructions.

3. Validate your cluster is shown when listing all clusters:

   ibmcloud oc clusters
   

4. Initialize the oc command environment by replacing the placeholder <your-cluster-name>:

   ibmcloud oc cluster config -c <your-cluster-name> --admin
   

5. Verify the oc command is working:

   oc get projects
   

Create Project

A project is a collection of resources managed by a DevOps team. An administrator will create the project and the developers can create applications that can be built and deployed.

1. Navigate to the Red Hat OpenShift web console by clicking the OpenShift web console button in the selected Cluster.
2. On the left navigation pane, under the Administrator perspective, select Home > Projects view to display all the projects.
3. Create a new project by clicking Create Project. In the pop up Name the project example-health, leave Display Name and Description blank, click Create.
4. The new project's Project Details page is displayed. Observe that your context is Administrator > Home > Projects on the left and Projects > Project details > example-health on the top.

Build and Deploy Application

1. Switch from the Administrator to the Developer perspective. Your context should be Developer > +Add on the left and Project: example-health on the top.

   

2. Let's build and deploy the application by selecting Import from Git.
3. Enter the repository https://github.com/IBM-Cloud/patient-health-frontend.git in the Git Repo URL field.
   • Note the green check Builder image detected and the Node.js 16 (UBI 8).
   • Note that the builder image automatically detected the language Node.js. If not detected, select Node.js from the provided list.
   • Builder Image Version leave at the default.
   • Application Name delete all of the characters and leave it empty (this will default to the Name)
   • Name : patient-health-frontend.
   • Click the Resource type link and choose DeploymentConfig.
   • Leave defaults for other selections.
4. Click Create at the bottom of the window to build and deploy the application.

View Application

1. You should see the app you just deployed. Notice that you are in the Topology view of the example-health project in the Developer perspective. All applications in the project are displayed.

2. Select the node patient-health-frontend to bring up the details view of the DeploymentConfig. Note the DC next to patient-health-frontend. The Pods, Builds, Services and Routes are visible.

   

   • Pods: Your Node.js application containers
   • Builds: The auto-generated build that created a Docker image from your Node.js source code, deployed it to the Red Hat OpenShift container registry, and kicked off your deployment config
   • Services: Tells Red Hat OpenShift how to access your Pods by grouping them together as a service and defining the port to listen to
   • Routes: Exposes your services to the outside world using the LoadBalancer provided by the IBM Cloud network

3. Click on View Logs next to your completed Build. This shows you the process that Red Hat OpenShift took to install the dependencies for your Node.js application and build/push a Docker image. The last entry should looks like this:

   Successfully pushed image-registry.openshift-image-registry.svc:5000/example-health/patient-health-frontend@sha256:f9385e010144f36353a74d16b6af10a028c12d005ab4fc0b1437137f6bd9e20a
   Push successful
   

4. Click back to the Topology and select your app again.

5. Click on the URL under Routes to visit your application. Enter any string for username and password, for instance test:test because the app is running in demonstration mode.

The Node.js app has been deployed to Red Hat OpenShift Container Platform. To recap:

• The "Example Health" Node.js application was deployed directly from GitHub into your cluster.
• The application was examined in the Red Hat OpenShift on IBM Cloud console.
• A Build Configuration was created - a new commit can be both built and deployed by clicking Start Build in the Builds section of the application details.

Simulate Load on the Application

Create a script to simulate load.

1. Make sure you're connected to the project where you deployed your app.

   oc project example-health
   

2. Retrieve the public route to access your application:

   oc get routes
   

   Output looks similar to this, note your value for Host:

   NAME         HOST/PORT                                                                                                 PATH      SERVICES     PORT       TERMINATION   WILDCARD
   patient-health-frontend   patient-health-frontend-example-health.roks07-872b77d77f69503584da5a379a38af9c-0000.eu-de.containers.appdomain.cloud             patient-health-frontend   8080-tcp                 None
   

3. Define a variable with the host:

   HOST=$(oc get routes -o json | jq -r '.items[0].spec.host')
   

4. Verify access to the application. It outputs patient information:

   curl -s -L http://$HOST/info
   

   Output should look like:

   $ curl -s -L http://$HOST/info
   {"personal":{"name":"Ralph DAlmeida","age":38,"gender":"male","street":"34 Main Street","city":"Toronto","zipcode":"M5H 1T1"},"medications":["Metoprolol","ACE inhibitors","Vitamin D"],"appointments":["2018-01-15 1:00 - Dentist","2018-02-14 4:00 - Internal Medicine","2018-09-30 8:00 - Pediatry"]}
   

5. Run the following script which will endlessly send requests to the application and generates traffic:

   while sleep 0.2; do curl --max-time 2 -s -L http://$HOST/info >/dev/null; echo -n "."
   done
   

   To stop the script, hit CTRL + c on your keyboard

Red Hat OpenShift on IBM Cloud Logging

Since there is only one pod, viewing the application logs will be straight forward.

1. Ensure that you're in the Topology view of the Developer perspective.

2. Navigate to your Pod by selecting your app.

3. Click on View Logs next to the name of the Pod under Pods to see streaming logs from your running application. If you're still generating traffic, you should see log messages for every request being made.

   

Red Hat OpenShift on IBM Cloud Terminal

One of the great things about Kubernetes is the ability to quickly debug your application pods with SSH terminals. This is great for development, but generally is not recommended in production environments. Red Hat OpenShift makes it even easier by allowing you to launch a terminal directly in the dashboard.

1. Switch from the Logs tab to the Terminal tab.
2. Run the following Shell commands:

Red Hat OpenShift on IBM Cloud Monitoring

When deploying new apps, making configuration changes, or simply inspecting the state of your cluster, the project-scope dashboard gives a Developer clear insights.

1. Access the dashboard in the Developer perspective by clicking Observe on the left side menu.
2. You can also dive in a bit deeper by clicking the Events tab. Events are useful for identifying the timeline of events and finding potential error messages. When tracking the state of a new rollout, managing existing assets, or even something simple like exposing a route, the Events view is critical in identifying the timeline of activity. This becomes even more useful when considering that multiple operators may be working against a single cluster.

Almost all actions in Red Hat OpenShift result in an event being fired in this view. As it is updated real-time, it's a great way to track changes to state.

Dashboards

Red Hat OpenShift comes with predefined dashboards to monitor your projects.

1. Get started by switching from the Developer perspective to the Administrator perspective:
2. Navigate to Observe > Dashboards in the left-hand bar.
3. Select Kubernetes / Compute Resources / Namespace (Pods) from the dropdown and Namespace to example-health.
4. Notice the CPU and Memory usage for your application. In production environments, this is helpful for identifying the average amount of CPU or Memory your application uses, especially as it can fluctuate through the day. Auto-scaling is one way to handle fluctuations and will be demonstrated a little later.

Metrics

Red Hat OpenShift provides a web interface to run queries and examine the metrics visualized on a plot. This functionality provides an extensive overview of the cluster state and enables you to troubleshoot problems.

1. Navigate to Observe > Metrics.

2. Enter the following expression and click Run queries. You should see the value and the graph associated with the query.

   sum(container_cpu_usage_seconds_total{container="patient-health-frontend"})
   

   

Enable Resource Limits

Before autoscaling maximum CPU and memory resource limits must be established.

The dashboards earlier showed you that the load was consuming anywhere between ".002" to ".02" cores. This translates to 2-20 "millicores". To be safe, let's bump the higher-end up to 30 millicores. In addition, the data showed that the app consumes about 25-65 MB of RAM. The following steps will set the resource limits in the deploymentConfig

1. Make sure the script to generate traffic is running.

2. Switch to the Administrator perspective.

3. Navigate to Workloads > DeploymentConfigs.

4. Select the example-health project.

5. From the Actions menu (the three vertical dots) of patient-health-frontend, choose Edit DeploymentConfig.

   

6. Under the YAML view, find the section spec > template > spec > containers, add the following resource limits into the empty resources. Replace the resources {}, and ensure the spacing is correct -- YAML uses strict indentation.

             resources:
               limits:
                 cpu: 30m
                 memory: 100Mi
               requests:
                 cpu: 3m
                 memory: 40Mi
   

   Here is a snippet after you have made the changes:

          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            limits:
              cpu: 30m
              memory: 100Mi
            requests:
              cpu: 3m
              memory: 40Mi
          terminationMessagePath: /dev/termination-log
   

7. Save to apply the changes.

8. Verify that the replication controller has been changed by navigating to Events tab:

   

Enable Autoscaler

Now that resource limits are configured, the pod autoscaler can be enabled.

By default, the autoscaler allows you to scale based on CPU or Memory. Pods are balanced between the minimum and maximum number of pods that you specify. With the autoscaler, pods are automatically created or deleted to ensure that the average CPU usage of the pods is below the CPU request target as defined. In general, you probably want to start scaling up when you get near 50-90% of the CPU usage of a pod. In our case, 1% can be used with the load being provided.

1. Navigate to Administrator perspective Workloads > HorizontalPodAutoscalers, then click Create HorizontalPodAutoscaler.

   

   Replace the contents of the editor with this yaml:

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: patient-hpa
     namespace: example-health
   spec:
     scaleTargetRef:
       apiVersion: apps.openshift.io/v1
       kind: DeploymentConfig
       name: patient-health-frontend
     minReplicas: 1
     maxReplicas: 10
     metrics:
       - type: Resource
         resource:
           name: cpu
           target:
             averageUtilization: 1
             type: Utilization
   

2. Click Create.

Test Autoscaler

If you're not running the script to simulate load, the number of pods should stay at 1.

1. Check by opening the Overview page of the deployment config. Click Workloads > DeploymentConfigs and click patient-health-frontend and make sure the Details panel is selected.

2. Start simulating load (see previous section to simulate load on the application).

   

   It can take a few minutes for the autoscaler to make adjustments.

That's it! You now have a highly available and automatically scaled front-end Node.js application. Red Hat OpenShift is automatically scaling your application pods since the CPU usage of the pods greatly exceeded 1% of the resource limit, 30 millicores.

Autoscaling from the command line

You can also can delete and create resources like autoscalars with the command line.

1. Start by verifying the context is your project:

   oc project example-health
   

2. Get the autoscaler that was created earlier:

   oc get hpa
   

3. Delete the autoscaler made earlier:

   oc delete hpa/patient-hpa
   

4. Create a new autoscaler with a max of 9 pods:

   oc autoscale deploymentconfig/patient-health-frontend --name patient-hpa --min 1 --max 9 --cpu-percent=1
   

5. Revisit the Workloads > DeploymentConfigs Details page for patient-health-frontend deployment and watch it work.

Enable the IBM Cloud Operator

Let's understand exactly how Operators work. In the first exercise, you used a builder to deploy a simple application using a DeploymentConfig a default resource type that come with Red Hat OpenShift. A custom resource definition allows you to create resource types that do not come preinstalled with Red Hat OpenShift on IBM Cloud such an IBM Cloud service. Operators manage the lifecycle of resources and create Custom Resource Descriptors, CRDs, allowing you to manage custom resources the native "Kubernetes" way.

1. In the Administrator perspective, and click Operators > OperatorHub.
2. Find the IBM Cloud Operator, and click Install.
3. Keep the default options and click Install.
4. After a few seconds installed operator - ready for use should be displayed.

Create a Cloudant Service and Bind using the CRDs

Click to open it. Scroll down to the Prerequisites section.

An API key with the appropriate permissions to create a IBM Cloudant database is required in this section. The API key is going to be stored in a Kubernetes Secret resource. This will need to be created using the shell. There are instructions in the Prerequisites section of the installed operator. Steps:

1. Use the same resource group and region that is associated with your cluster.

   ibmcloud target -g <resource_group> -r <region>
   

   To see the the resource groups in your account, run ibmcloud resource groups command

2. Verify that the resource group and region matches your cluster. The following command should return your cluster.

   ibmcloud oc cluster ls
   

Output looks something like this:

$ ibmcloud oc cluster ls
OK
Name      ID                     State    Created        Workers   Location   Version                  Resource Group Name   Provider
osmicro   ck68svdd0vvcfs6ad9ag   normal   18 hours ago   2         Dallas     4.12.26_1562_openshift   default               vpc-gen2

3. Use the helper script provided by IBM to create the following resources:

   • IBM Cloud API key that represents you and your permissions to use IBM Cloud
   • Kubernetes Secret named secret-ibm-cloud-operator in the default namespace. This secret has the keys api-key and region. The operator will use this data to create the cloudant service instance.
   • Kubernetes ConfigMap resource with the name config-ibm-cloud-operator in the default namespace to hold the region and resource group

   Use the supplied curl command:

   curl -sL https://raw.githubusercontent.com/IBM/cloud-operators/master/hack/configure-operator.sh | bash
   

4. Back in the Red Hat OpenShift web console, click the Create service under the Service tab on the Installed Operators of the IBM Cloud Operator page and select YAML view to bring up the yaml editor.

5. Make the suggested substitutions where the serviceClass is cloudantnosqldb and the plan can be lite or standard (only one lite plan is allowed per account). Replace <your-initials>:

   apiVersion: ibmcloud.ibm.com/v1
   kind: Service
   metadata:
     annotations:
      ibmcloud.ibm.com/self-healing: enabled
     name: <your-initials>-cloudant-service
     namespace: example-health
   spec:
     serviceClass: cloudantnosqldb
     plan: standard
   

6. Click Create to create a IBM Cloudant database instance. Your context should be Operators > Installed Operators > IBM Cloud Operator in the Administrator perspective with Project: example-health in the Service panel.

7. Click on the service just created, <your-initials>-cloudant-service and over time the state field will change from provisioning to Online meaning it is good to go.

8. Create a Binding resource and a Secret resource for the cloudant Service resource just created. Navigate back to Operators > Installed Operators > IBM Cloud Operator > Binding tab. Open the Binding tab, click Create Binding and select YAML View. Create a cloudant-binding associated with the serviceName <your-initials>-cloudant-service, (this is the the name provided for the Service created earlier).

   apiVersion: ibmcloud.ibm.com/v1
   kind: Binding
   metadata:
     name: cloudant-binding
     namespace: example-health
   spec:
     serviceName: <your-initials>-cloudant-service
   

9. Optionally dig a little deeper to understand the relationship between the Red Hat OpenShift resources: Service, service Binding, binding Secret and the IBM Cloud resources: Service, service Instance and the instance's Service credentials. Using the cloud shell:

   ibmcloud resource service-instances --service-name cloudantnosqldb
   

   YOURINITIALS=<your-initials>
   

   ibmcloud resource service-instance $YOURINITIALS-cloudant-service
   

   ibmcloud resource service-keys --instance-name $YOURINITIALS-cloudant-service --output json
   

   Output looks something like this:

   youyou@cloudshell:~$ ibmcloud resource service-instances --service-name cloudantnosqldb
   Retrieving instances with type service_instance in all resource groups in all locations under ..
   OK
   Name                           Location   State    Type
   <your-initials>-cloudant-service               us-south   active   service_instance
   youyou@cloudshell:~$ ibmcloud resource service-instance <your-initials>-cloudant-service
   Retrieving service instance <your-initials>-cloudant-service in all resource groups under ...
   OK
   
   Name:                  <your-initials>-cloudant-service
   ID:                    crn:v1:bluemix:public:cloudantnosqldb:us-south:a/0123456789507a53135fe6793c37cc74:SECRET
   GUID:                  SECRET
   Location:              us-south
   Service Name:          cloudantnosqldb
   Service Plan Name:     standard
   Resource Group Name:   Default
   State:                 active
   Type:                  service_instance
   Sub Type:
   Created at:            2020-05-06T22:39:25Z
   Created by:            youyou@us.ibm.com
   Updated at:            2020-05-06T22:40:03Z
   Last Operation:
                       Status       create succeeded
                       Message      Provisioning is complete
                       Updated At   2020-05-06 22:40:03.04469305 +0000 UTC
   
   youyou@cloudshell:~$ ibmcloud resource service-keys --instance-name $YOURINITIALS-cloudant-service --output json
   [
       {
           "guid": "01234560-902d-4078-9a7f-20446a639aeb",
           "id": "crn:v1:bluemix:public:cloudantnosqldb:us-south:a/0123456789507a53135fe6793c37cc74:SECRET",
           "url": "/v2/resource_keys/01234560-902d-4078-9a7f-20446a639aeb",
           "created_at": "2020-05-06T23:03:43.484872077Z",
           "updated_at": "2020-05-06T23:03:43.484872077Z",
           "deleted_at": null,
           "name": "cloudant-binding",
           "account_id": "0123456789507a53135fe6793c37cc74",
           "resource_group_id": "01234567836d49029966ab5be7fe50b5",
           "source_crn": "crn:v1:bluemix:public:cloudantnosqldb:us-south:a/0123456789507a53135fe6793c37cc74:SECRET",
           "state": "active",
           "credentials": {
               "apikey": "SECRET",
               "host": "SECRET",
               "iam_apikey_description": "Auto-generated for key SECRET",
               "iam_apikey_name": "cloudant-binding",
               "iam_role_crn": "SECRET",
               "iam_serviceid_crn": "SECRET",
               "password": "SECRET",
               "port": 443,
               "url": "https://01234SECRET",
               "username": "01234567-SECRET"
           },
           "iam_compatible": true,
           "resource_instance_url": "/v2/resource_instances/SECRET",
           "crn": "crn:v1:bluemix:public:cloudantnosqldb:us-south:a/0123456789507a53135fe6793c37cc74:SECRET"
       }
   ]
   

Deploy the Node.js Patient Backend Database App

Now you'll create the Node.js app that will populate your Cloudant DB with patient data. It will also serve data to the front-end application deployed earlier.

1. Make sure you're your context is the project example-health:

   oc project example-health
   

2. The following new-app commmand will make a build configuration and Deployment Configuration. The following demonstrates the CLI invocation of the add application (remember using the GUI console for the frontend):

   oc new-app --name=patient-health-backend --as-deployment-config centos/nodejs-10-centos7~https://github.com/IBM-Cloud/patient-health-backend
   

3. Back in the console, and in the Topology view of the Developer perspective, open the patient-health-backend app and wait for the build to complete. Notice that the Pod is failing to start. Click on the Pod logs to see:

   > node app.js
   
   /opt/app-root/src/app.js:23
            throw("Cannot find Cloudant credentials, set CLOUDANT_URL.")
            ^
   Cannot find Cloudant credentials, set CLOUDANT_URL.
   

4. Let's fix this by setting the environment variable of the DeploymentConfig to the cloudant-binding secret created earlier in the operator binding section. Navigate to the deployment config for the patient-health-backend app by clicking the app, and then selecting the name next to DC:

   

5. Go to the Environment tab, click Add from ConfigMap or Secret and create a new environment variable named CLOUDANT_URL. Choose the cloudant-binding secret, then choose url for the Key. Click Save.

   

6. Go back to the Topology tab, and click the patient-health-backend. Check out the Pods section, which should indicate Running shortly. Click on View logs next to the running pod and notice the databases created.

Configure Patient Health Frontend App to use Patient Health Backend App

The patient-health-frontend application has an environment variable for the backend microservice url.

1. Set the API_URL environment variable to default in the frontend DeploymentConfig. Navigate to the deployment config for the patient-health-frontend app by clicking the frontend app in the Topology view, and then selecting the name next to DC:

2. Go to the Environment tab, and in the Single values (env) section add a name API_URL and value default. Click Save then Reload. This will result in a connection to http://patient-health-backend:8080/ which you can verify by looking at the pod logs. You can verify this is the correct port by scanning for the Pod Template / Containers / Port output of this command:

   oc describe dc/patient-health-backend
   

Your application is now backed by the mock patient data in the Cloudant DB! You can now log-in using any user-id/password in the Cloudant DB, use "opall:opall".

1. In a real-world application, these passwords should not be stored as plain-text. To review the patients (and alternate logins) in the Cloudant DB, navigate to your services in IBM Cloud Resource List. Click <your-initials>-cloudant-service.
2. Launch the Cloudant dashboard by clicking on Launch Dashboard button and then click the patients db.
3. Click through the different patients you can log in as.

Verify that the Log Analysis agent is deployed successfully

Verify that the Log Analysis-agent pods on each node are in a Running status.

oc get pods -n ibm-observe

The deployment is successful when you see one or more Log Analysis pods:

someone@cloudshell:~$ oc get pods -n ibm-observe
NAME                 READY     STATUS    RESTARTS   AGE
logdna-agent-mdgdz   1/1       Running   0          86s
logdna-agent-qlqwc   1/1       Running   0          86s

The number of Log Analysis pods equals the number of worker nodes in your cluster.

• All pods must be in a Running state
• stdout and stderr are automatically collected and forwarded from all containers. Log data includes application logs and worker logs.
• By default, the Log Analysis agent pod that runs on a worker collects logs from all namespaces on that node.

After the agent is configured logs from this cluster will be visible in the Log Analysis web UI covered in the next section. If after a period of time you cannot see logs, check the agent logs.

To check the logs that are generated by a Log Analysis agent, run the following command:

oc logs logdna-agent-<ID> -n ibm-observe

Where ID is the ID for a Log Analysis agent pod.

For example,

oc logs logdna-agent-mdgdz -n ibm-observe

Launch the Log Analysis webUI

Launch the web UI within the context of a Log Analysis instance, from the IBM Cloud UI.

1. Navigate to Red Hat OpenShift on IBM Cloud clusters
2. Click on your cluster and verify the Overview tab on the left is selected
3. In the Integrations section next to Logging, click the Launch button.

The Log Analysis UI should open in a new tab.

Create a custom view

In Log Analysis, you can configure custom views to monitor a subset of data. You can also attach an alert to a view to be notified of the presence or absence of log lines.

In the Log Analysis web UI notice the log entries are displayed with a predefined format. You can modify in the User Preferences section how the information in each log line is displayed. You can also filter logs and modify search settings, then bookmark the result as a view. You can attach and detach one or more alerts to a view. You can define a custom format for how your lines are shown in the view. You can expand a log line and see the data parsed.

Simulate Load on the Application

With the application now connected to a database for its data, to simulate load we will generate requests to the database using a patient id that we added to the database ef5335dd-db17-491e-8150-20ce24712b06.

1. Make sure you're connected to the project where you deployed your app.

   oc project example-health
   

2. Define a variable with the host:

   HOST=$(oc get routes -o json | jq -r '.items[0].spec.host')
   

3. Verify access to the application. It outputs patient information:

   curl -s -L "http://$HOST/info?id=ef5335dd-db17-491e-8150-20ce24712b06"
   

   Output should look like:

   $ curl -L "http://$HOST/info?id=ef5335dd-db17-491e-8150-20ce24712b06"
   {"personal":{"name":"Opal Larkin","age":22,"street":"805 Bosco Vale","city":"Lincoln","zipcode":"68336"},"medications":["Cefaclor ","Amoxicillin ","Ibuprofen ","Trinessa ","Mirena ","Naproxen sodium "],"appointments":["2009-01-29 10:46 - GENERAL PRACTICE","1999-07-01 10:46 - GENERAL PRACTICE","2001-12-27 10:46 - GENERAL PRACTICE","2005-01-06 10:46 - GENERAL PRACTICE","2004-01-01 10:46 - GENERAL PRACTICE","1999-09-30 10:46 - GENERAL PRACTICE","2018-10-29 10:46 - GENERAL PRACTICE","2012-02-16 10:46 - GENERAL PRACTICE","2015-11-23 10:46 - GENERAL PRACTICE","2000-03-30 10:46 - GENERAL PRACTICE","1999-04-29 10:46 - GENERAL PRACTICE","2015-01-07 10:46 - GENERAL PRACTICE","1999-02-25 10:46 - GENERAL PRACTICE","2010-07-23 10:46 - GENERAL PRACTICE","2008-01-24 10:46 - GENERAL PRACTICE","2004-05-24 10:46 - GENERAL PRACTICE","1999-01-21 10:46 - GENERAL PRACTICE","2015-03-05 10:46 - GENERAL PRACTICE","2002-06-27 10:46 - GENERAL PRACTICE","2000-06-29 10:46 - GENERAL PRACTICE","2005-01-06 10:46 - GENERAL PRACTICE","2015-01-10 10:46 - GENERAL PRACTICE","2000-12-28 10:46 - GENERAL PRACTICE","2016-06-02 10:46 - GENERAL PRACTICE","2016-03-10 10:46 - GENERAL PRACTICE","2013-09-08 10:46 - GENERAL PRACTICE","2011-02-10 10:46 - GENERAL PRACTICE","2013-02-21 10:46 - GENERAL PRACTICE","2003-04-30 10:46 - GENERAL PRACTICE","2004-07-23 10:46 - GENERAL PRACTICE","2006-01-12 10:46 - GENERAL PRACTICE","2002-12-26 10:46 - GENERAL PRACTICE","1999-12-30 10:46 - GENERAL PRACTICE","2017-01-04 10:46 - GENERAL PRACTICE","2018-03-22 10:46 - GENERAL PRACTICE","2010-02-04 10:46 - GENERAL PRACTICE","2009-11-29 10:46 - GENERAL PRACTICE","2013-02-26 10:46 - GENERAL PRACTICE","2003-02-04 10:46 - GENERAL PRACTICE","2003-03-01 10:46 - GENERAL PRACTICE","2000-04-15 10:46 - GENERAL PRACTICE","2001-06-28 10:46 - GENERAL PRACTICE","2007-01-18 10:46 - GENERAL PRACTICE","2018-08-30 10:46 - GENERAL PRACTICE","2017-03-16 10:46 - GENERAL PRACTICE","2014-02-27 10:46 - GENERAL PRACTICE","2000-09-27 10:46 - GENERAL PRACTICE"]}
   

4. Run the following script which will endlessly send requests to the application and generates traffic:

   while sleep 0.2; do curl --max-time 2 -s -L "http://$HOST/info?id=ef5335dd-db17-491e-8150-20ce24712b06" >/dev/null; echo -n "."
   done
   

   To stop the script, hit CTRL + c on your keyboard

View events with the default format

1. In the Log Analysis web UI, click the Views icon.
2. Select Everything to see all the events. It can take a few minutes for the load on the application to be visible.

   

Customize your default view

In the User Preferences, you can modify the order of the data fields that are displayed per line.

1. Click your profile icon in the bottom left and select User Preferences.
2. Select Log Format.
3. Modify the Line Format section to match your requirements. Drag boxes around. Click Done. For example, add %app after the timestamp.

   

Create a custom view to monitor logs

You can select the events that are displayed through a view by applying a search query in the search bar, selecting values in the search area, or a combination of both. You can save that view for reuse later.

1. In the Log Analysis web UI, filter out the logs for the sample app that you have deployed in the cluster in previous steps. Click in the search bar at the bottom and enter the following query: app:patient-health-frontend.

2. Filter out log lines to display only lines that are tagged as debug lines. Add in the search bar the following query: level:debug and hit enter. The view will show lines that meet the filter and search criteria.

   

3. Click Unsaved view. Select Save as new view.

   

   1. Enter the name of the view. Use the following format: <Enter your user name> patientUI. For example, yourname patientui.
   2. Enter a category. Use the following format: <Enter your user name>. For example, yourname Then click Add this as a new view category.
   3. Click Save view.

4. A new view appears on the left navigation panel.

Analyze a log line

At any time, you can view each log line in context.

Complete the following steps:

1. Click the Views icon.

2. Select Everything or a view.

3. Identify a line in the log that you want to explore.

4. Expand the log line to display information about line identifiers, tags, and labels.

5. Click View in Context to see the log line in context of other log lines from that host, app, or both. This is a very useful feature when you want to troubleshoot a problem.

   

6. A new pop up window opens. In the window, choose one of the following options:

   • By Everything to see the log line in the context of all log records (everything) that are available in the Log Analysis instance
   • By source to see the log line in the context of the log lines for the same source
   • By App to see the log line in the context of the log lines of the app
   • By Source and App to see the log line in the combined context of the app and source

   Then click Continue in New Viewer to get the view in a different page. You might need to scroll down to get this option.

   Tip: Open a view per type of context to troubleshoot problems.

   

7. Expand the selected log and click Copy to clipboard to copy the message field to the clipboard. Notice that when you copy the log record you get less information than what it is displayed in the view. To get a line with all the fields, you must export data from a custom view.

8. When you are finished, close the line.

View a subset of the events by applying a timeframe

In a view, you can search events that are displayed through a view for a specific timeframe.

You can apply a timestamp by specifying an absolute time, a relative time, or a time range.

Complete the following steps to jump to a specific time:

1. Launch the Log Analysis web UI.
2. Click the Views icon.
3. Select your custom view.
4. Enter a time query. Choose any of the following options:
   • Enter a relative time such as 1 hour ago. Type ENTER

     

   • Enter an absolute time to jump to a point in time in your events such as January 27 10:00am
   • You can also enter a time range such as yesterday 10am to yesterday 11am, last fri 4:30pm to 11/12 1 AM, last wed 4:30pm to 23/05 1 AM, or May 20 10am to May 22 10am. Make sure to include to to separate the initial timestamp from the end timestamp

You might get the error message: Your request is taking longer than expected, try refreshing your browser after a few minutes of delay to allow logs to flow into the service. Also, ensure that the the timeframe selected is likely to have events available for display. It may be required to change the time query, and retry.

Create a dashboard

You can create a dashboard to monitor your app graphically through interactive graphs. For example, you can use graphs to analyze patterns and trends over a period of time.

Index fields are created on a regular schedule. Currently it is done at 00:01 UTC (midnight). The following steps that require fields will not be possible until this process completes.

Complete the following steps to create a dashboard to monitor logs from the lab's sample app:

1. In the Log Analysis web UI, click the Boards icon.

2. Select NEW BOARD to create a new dashboard.

3. Select the Field All lines under Graph a field.

4. Select the Filter app:patient-health-frontend.

   

5. Click Add Graph.

   

6. Note the view that displays the count of logs lines for the frontend app. Click the graph in a peak of data at the time that you want to see logs, and then click Show logs.

   A new page opens with the relevant log entries. Click the browser's back button when done with the log lines to return to the graph.

7. Add breakdowns to analyze the data by applying additonal filtering criteria.

   

   1. Click View breakdowns.
   2. Select Histogram and level.Click Add Breakdown.

   

8. Name the dashboard by hitting the pencil Edit Board button next to the New Board N name".

   • Enter patientui as the name of the dashboard
   • Enter a category, for example, yourname then click Add this as a new board category
   • Click Save

A new category appears on the left navigation panel.

Create a screen to monitor your app

You can create a screen to monitor your app graphically through metrics (counters), operational KPIs (gauges), tables, and time-shifted graphs (graphs that you can use to analyze patterns and trends for comparison analysis).

Complete the following steps to create a dashboard to monitor logs from the lab's sample app:

1. In the Log Analysis web UI, click the screens icon.

2. Select NEW SCREEN.

3. Add a count of the patient health frontend log lines for the last two weeks:

   • Click Add Widget at the top and select Count
   • Click the newly created widget to reveal the configuration fields for the widget on the right
   • In the Data section
     • Select the field app, and set the value to patient-health-frontend
     • Keep Operation at the default Counts
     • In the Duration drop down select Last 2 Weeks
   • In the Appearance section
     • In the Label text box type patient-health-frontend

   The widget should look similar to the following one:

   

4. Add a gauge that records the debug lines for the patient-health-frontend for the last day.

   • Click Add Widget at the top and select Gauge
   • Click the newly created widget to reveal the configuration fields for the widget on the right
   • In the Data section
     • Select the field app, and set the value to patient-health-frontend
     • Click Advanced Filtering and in the text box type level:debug
     • Keep the Duration set to the default Last 1 day
   • In the Appearance section
     • In the Label text box type patient-health-frontend

5. Add a table of logs by namespace.

   • Click Add Widget at the top and select Table
   • Click the newly created widget to reveal the configuration fields for the widget on the right
   • In the Data section
     • Select the field Group By and choose namespace from the drop down
   • In the Data Format section
     • Select the field Number of Rows and choose 10 from the drop down

6. Drag the table to improve the presentation. Verify the screen resembles the following:

   

7. Save the screen. Select Save Screen.

   If you do not save the screen, you lose all your widgets.

Find more about Log Analysis in the IBM Cloud documentation.

Verify that the Monitoring agent is deployed successfully

Verify that the sysdig-agent pods on each node have a Running status.

Run the following command:

oc get pods -n ibm-observe

Example output:

NAME                 READY     STATUS    RESTARTS   AGE
sysdig-agent-qrbcq   1/1       Running   0          1m
sysdig-agent-rhrgz   1/1       Running   0          1m

View pre-defined monitoring views and dashboards

Use views and dashboards to monitor your infrastructure, applications, and services. You can use pre-defined dashboards. You can also create custom dashboards through the Web UI or programmatically. You can backup and restore dashboards by using Python scripts.

The following table lists the different types of pre-defined dashboards:

View the Monitoring dashboard

1. Navigate to Red Hat OpenShift on IBM Cloud clusters and notice the Red Hat OpenShift clusters
2. Click on your cluster and verify the Overview tab on the left is selected
3. In the Integrations section next to Monitoring, click the Launch button.

Initial data may NOT be available on newly created Monitoring instances.

• After a few minutes, raw data will be displayed
• After about an hour, indexing will provides the detail required to proceed with this tutorial

1. Under the Dashboards section, select Kubernetes > Pod Status & Performance to view raw metrics for all workloads running on the cluster.
2. Set the namespace filter to example-health to focus on the pods of your application.
3. Under Dashboards on the left pane, expand Applications in Dashboard Templates. Then select HTTP to get a global view of the cluster HTTP load.

Explore the cluster and the node capacity

1. Select Dashboards, check out the two dashboard templates:

   • Containers > Container Resource Usage
   • Host Infrastructure > Host Resource Usage

2. Select Kubernetes > Pod Rightsizing & Workload Capacity Optimization template. This dashboard helps you to optimize your infrastructure and better control cluster spend by ensure pods are sized correctly. Understand if you can free up resources by reducing memory and/or CPU requests.

Explore the Application

1. Select Dashboards and the template Kubernetes > Workload Status & Performance.

   A detailed dashboard showing all the pods in the cluster.

2. Create a customized dashboard and then scope it to a specific namespace.

   • In the upper right click Copy to my Dashboards and name it Workload Status & Performanceapp example-health
   • Click Create and Open to create your own dashboard.
   • Edit the dashboard scope.
   • Set the filter for kube_namespace_name, is, example-health.
   • Click Save.

   The dashboard now shows information focused on the example-health namespace.

   Scroll down to the TimeCharts for HTTP Requests, Latency, Error, ... to understand the performance of the application.

   

Find more about IBM Cloud Monitoring in the IBM Cloud documentation.