Use the Log Analysis web UI

Before you begin

• Make sure you have a logging instance that is collecting logs.

  • You can have a logging agent configured to collect logs from a VPC instance. See Logging in Linux VPC server instances.

  • You can have a logging agent configured to collect logs from a Kubernetes cluster. See Logging in Kubernetes clusters (cluster-level logging).

• Check you have permissions to launch the logging web UI. See Granting IAM policies to a user to launch the web UI.

Launch the IBM Log Analysis web UI

1. Log in to your IBM Cloud account.

   Click Log in to IBM Cloud to sign in to the IBM Cloud.

   After you log in with your user ID and password, the IBM Cloud console opens.

2. Select your account.

3. Click the Menu icon > Observability.

4. Click Logging.

5. For your instance, click Open dashboard. The logging web UI will be displayed.

Create a custom view

In IBM Log Analysis, you can configure custom views to monitor a subset of data. For example, you configure your cluster to forward specific logs to the logging instance and you can filter those logs for analysis.

This tutorial will show you how to create a view that shows only a specific log.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Click the EVERYTHING view.

   

3. Click Apps and select the log you want to view.

4. Click Apply.

   The view changes to only show your selected logs and the tab changes to Unsaved View.

   

5. Click Unsaved View > Save as new view.

6. Enter a Name for your view. Optionally, you can select a Category and Alert value. Click Save View.

   Your view is listed under your selected category. If you didn't select a category, it will be listed under UNCATEGORIZED.

Customize the log format

When you launch the IBM Log Analysis web UI, log entries are displayed in a predefined format. You can modify how the log entries are displayed.

These configuration changes will affect all defined views.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Click the Settings icon .

3. Click USER PREFERENCES.

4. Click Log Format.

5. Modify the log format as desired.

   Change the log viewer text size by using the slider.

   To add items to the log view, drag the available items and add them in your desired order to the view.

6. Click Done to save your changes. To cancel without making changes, press Esc.

Customize the data displayed for a view

You can customize the fields that are displayed in a custom view.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Make sure you have a custom view defined.

3. Select the view to change in the navigation. Your view will be listed under the category where it was created, or under UNCATEGORIZED if you didn't specify a category when the view was created. For example, the following shows a view named "My View" that was created without an assigned category.

   

4. Click the selected view name. The following example shows the view named "My View".

   

5. Click the view name. The menu is displayed. Click Edit view properties.

   

6. In the Custom Template you can change how you want the log line formatted for the view by including static text and fields from the log.

   • To determine the possible fields, click the down arrow on a log line in the view.

     

   • Click Extract Fields

     The log fields that can be used are displayed in the Reference line along with the specific values from the selected log entry. For example:

     {"logSourceCRN":"crn:v1:bluemix:public:containers-kubernetes:us-south:a/xxxxx:yyyyy::","saveServiceCopy":true,"message":"Cluster yyyyy health status set to All Workers Normal"}
     

     In this case, the following fields can be used in the Custom Template: logSourceCRN and message.

     The saveServiceCopy field value cannot be used because it is a boolean value. Boolean field values will not display in a Custom Template.

   • In the Custom Template, specify the fields and static text in the way you want them displayed. Fields need to be enclosed in double braces {{field}} or in BASH format ${field}. All other specified characters will be rendered as plain text.

     For example, if you have log entries similar to the following:

     Nov 16 13:59:02 containers-kubernetes crn:v1:bluemix:public:containers-kubernetes:us-south:a/xxxxx:yyyyy:: Cluster yyyyy health status set to All Workers Normal
     

     And you want them formatted like this:

     SOURCE: Nov 16 13:59:02 containers-kubernetes crn:v1:bluemix:public:containers-kubernetes:us-south:a/xxxxx:yyyyy::
     MESSAGE: Cluster yyyyy health status set to All Workers Normal
     

     And the extracted reference line is:

     {"logSourceCRN":"crn:v1:bluemix:public:containers-kubernetes:us-south:a/xxxxx:yyyyy::","saveServiceCopy":true,"message":"Cluster yyyy health status set to All Workers Normal"}
     

     Then your Custom Template would be:

     SOURCE: {{logSourceCRN}}
     MESSAGE: {{message}}
     

     The original log line will be displayed followed by the custom formatting.

Create a parsing rule

There are occasions when you might want to reference information that is available in log records. Parsing rules maps information from a log line to a field that can be used in other configuration settings.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Click the Settings icon .

3. Click PARSING.

4. Click Create Template. The template wizard opens.

5. For this tutorial, for Choose a sample log line to begin building your template select Find an existing log line. Enter a search term that will match a field in your log. See Customize the data displayed for a view for information on extracting log fields.

6. Scroll to the end of the page and click Build Parsing Template. A Reference Line for your log records is displayed.

7. Click the the Pencil icon and enter a name for your parsing rule.

   If you name a template before you click Build Parsing Template, the name is not changed.

8. Choose the Extract values by delimiter extractor.

9. For Delimiter specify a , (comma). A preview of the output delimited by a comma is presented for your review.

10. Choose one of the entries. For example, if your log contains a field called message you might select that line.

11. For Choose an Operator select Extract Values by Delimiter.

12. For Delimiter specify a delimiter that makes sense for your selected entry. For example, a reasonable delimiter might be a : (colon). A preview of the delimited output is presented for your review.

13. Select an entry containing a date value.

14. For Choose an Operator select Trim Value. Using the index range specify a starting and ending character position so only the date is retained in the displayed output.

15. For Choose an Operator select Capture in Field to set the name of the field that will capture this information.

16. For Field Name specify a name that you will use to reference this type of information.

17. At the end of the page click Proceed to Validation.

18. For Set a query and verify parsed output with different sample lines enter a search that will return multiple records. For example, message.

19. If all log lines returned match the rule you specified, click Mark as Valid for each and then click Activate at the end of the page.

    Until you validate all results, the Activate option will be disabled.

The rule will take affect in approximately 15 minutes.

Analyze the log line

You might want to view a log line in context.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Click the Views icon .

3. Select a log view. You might want to use the custom view you created.

4. Identify a log line you want to explore.

5. Open the log entry by clicking the down arrow. Information about line identifiers, tags, and labels is displayed.

   .

6. Click . The log line is displayed in context of other log lines from that host, app, or both. This information is helpful when troubleshooting a problem.

7. Explore the selected log line by using the following options.

   By Everything displays the log line in context of all log records that are available in the IBM Log Analysis instance.

   By source displays the log line in context of the log lines for the same source.

   By App displays the log line in context of the log lines of the app.

   By Source & App displays the log line in the combined context of the app and source.

8. Click Continue in New Viewer to open the view as a new Unsaved View. Keeping multiple views can help troubleshoot problems.

Filtering to a specific timeframe

You can search for events occurring in a specified timeframe.

The timeframe can be an absolute time, relative time, or a time range.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Click the Views icon .

3. Select a log view. You might want to use the custom view you created.

4. In the Jump to timeframe field enter a timeframe to filter the log.

   1. Specify a specific date or date time. For example, Nov 16 will jump to log entries starting with that date. You can also specify a date and time. For example, Nov 16 22:29.

   2. Specify a value relative to the current time. For example, 1 day ago or 5 hours ago.

   3. Specify a date, or date and time range. For example, Nov 14 to Nov 15 or Nov 14 12:00 to Nov 14 23:59. Ranges can also be relative to the current time. For example, Yesterday 10am to Yesterday 11am.

If you get a “Your request is taking longer than expected, try refreshing your browser in a bit as we try to catch up” error message, the timeframe you specified might not have any available data. Change your timeframe and retry.

Create dashboard

A dashboard provides interactive graphs that help you monitor your app. For example, graphs can help you analyze patterns and trends over a period of time.

The following will help you create a dashboard.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Click the Boards icon .

3. Click NEW BOARD.

4. Click Add Graph.

5. Select app For Field.

6. Select ANY for Field Value.

7. Click Add Graph. A trend line showing the count of app log events is listed.

8. Click the graph line and click Show logs to see the log at point in time.

9. Open the subplot view and select Histogram for Choose breakdown type and host for SYSTEM FIELD.

10. Click Add Breakdown. A histogram of the data by host is added to the dashboard.

11. Click Add and add an additional histogram and specify verb for SYSTEM FIELD.

12. Try adding additional plots by clicking Add Plot.

13. Multiple plot lines can be hidden or displayed on the graph. Select your plot and click Show to show the line or Hide to hide it.

14. Click the Filter icon to filter the graph by a specific field's data. For example, host:myhost.

15. Click the Pencil icon and enter a name for your dashboard. Specify a Category if desired and click Save.

If you don't specify a Category your dashboard will be listed under UNCATEGORIZED. If you are editing an existing dashboard, changes will be saved when exiting the dashboard without having to click Save.

Create a widget in a screen

Using the screen option you can create a screen of widgets that you can use to monitor your app using metrics (counters), operational KPIs (gauges), tables, and graphs that can be used to analyze patterns and trends.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Click the Screens icon .

3. Click NEW SCREEN.

4. To save the screen, click Save Screen. Enter the following and then click Save.

   Name: The name you want to give to your screen.

   Category: Select an existing category or enter a new category name. If adding a new category, click Add this as a new screen category to add the category to the list.

5. Click Add Widget and select Count. A widget is added to your screen.

6. Click the widget. The configuration for the widget is displayed.

7. Change the Field and Field Value to change what is counted. For example, you might change Field to host and Field Value to a host in your account. The widget will now count the log messages in the past day. You can also change the Duration of time counted.

8. Change the Label of the widget to something meaningful to you.

9. Click Save Screen > Save to save your changes.

10. Add a table widget by clicking Add Widget > Table.

11. Click the table widget. The configuration for the widget is displayed.

12. Change the Field and Field Value to change what is displayed in the table. In Group By, specify the field used to group the data . For example, you might change Field to host and Field Value to a host in your account. You then might want to group that data by app. The widget will display a table of the counts for the host grouped by app during the past day. You can also change the Duration of time counted and specify a Label for your widget.

    If you need a table with more rows, change the Data Format Number of Rows to your preferred value.

13. You can resize any of the widgets by clicking the widget and clicking and dragging one of the anchors. Widgets can be moved by clicking the widget and dragging it to your desired location on the screen.

14. Click Save Screen > Save to save your changes.

If you do not save your screen, you will lose all your changes.

Add alert to custom view

You can configure an alert to be sent by email, Slack, PagerDuty or as a Webhook when specific criteria is met.

1. Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI.

2. Make sure you have a custom view defined..

3. Select the view to change in the navigation. Your view will be listed under the category where it was created, or under UNCATEGORIZED if you didn't specify a category when the view was created. For example, the following shows a view named "My View" that was created without an assigned category.

   

4. Click the selected view name. The following example shows the view named "My View".

   

5. Click the view name. The menu is displayed. Click Attach an alert

   

6. Click View-specific alert.

7. Click Email.

8. Configure the alert.

   

   1. Specify when the alert will be triggered, and an email sent. For this example specify When 5 or more matches appear within 30 seconds.

   2. Select the email addresses of the Recipients who should receive the alert.

   3. Click Save Alert.

9. Check the configured email for an alert.

10. When you are ready to disable the alert, click the view name and in the menu click Detach alerts.

Next steps

Want to start fresh? Remove the following resources that you created as a part of this tutorial:

• Delete any views you will not need.
• Delete any dashboards and screens you will not need.
• Delete any parsing rules you will not need.
• Detach any alerts you will not need.