Truck Tracker: An IoT Pattern Using OpenShift and other IBM Cloud Services

Combining IBM Cloud services to produce a variety of insights.

Truck Tracker is a service that receives data from a fleet of trucks as they travel across the country. It could be any data (e.g., fuel consumption, driver details, ambient temperature, or anything else that can be measured as you cruise). For the purposes of this post, it will be the truck's ID and position (lat/long coordinates). Truck Tracker will store the data in various data stores and also show you the current position of the trucks on a map.

Some app users need this data to run analytics. For example, to find out how many miles the fleet has traveled in the last 24 hours or how much fuel it has consumed. For that, all data points can be stored in a reliable document store like IBM Cloudant, as well as searched and aggregated at will. IBM Cloudant is a fully managed, distributed database optimized for heavy workloads and fast-growing web and mobile apps.

Other users might need to know where the trucks currently are. For this, you only need the last known position of each truck. In this case, Redis is a perfect solution — a fast key-value store where a key is the unique truck ID and the value is its last known position.

IBM Cloud® Databases for Redis is a fully managed in-memory data structure store that can be used as a database, cache and message broker.

There may be other, yet unknown, uses for the data that require it to be accessed or stored in different ways.

So your trucks are flying around the country and data is pouring out of them. How do you make sure everyone gets the data they need? Enter IBM Event Streams, an event-streaming platform that helps you build smart apps that can react to events as they happen. It is a fully managed service built on open-source Apache Kafka.

Event Streams can take in the data from the trucks (data producers) and then serve it to the various applications that will use it (data consumers).

Finally, you will need to run your Truck Tracker service somewhere, and for that we will use Red Hat OpenShift on IBM Cloud, a fast and secure way to containerize and deploy enterprise workloads in Kubernetes clusters. We will provision an Red Hat OpenShift on IBM Cloud cluster, deploy our applications to it and make them available to the database services that need access to it. We will also make a public-facing application to view the positions of trucks on a map. Read more about Red Hat OpenShift on IBM Cloud.

What you will build

This is the high-level system that we are going to build:

In a real-life scenario, your data generators (the truck's IoT devices) would have to find a way to communicate with the Event Streams application, probably using the MQTT protocol. We will simulate our trucks traveling around the country and sending data by using a script (producer.js) that will be generating a truck location every second by reading from a predefined set of lat/long pairs of a couple of "road trips" (LAToDallas.json and boulderToNYC.json) and feeding it into Event Streams.

This tutorial should take you about an hour to complete. It will not be cost-free because some of the IBM services we will use do not come with a free tier, but if you deprovision the services after completing it, you should not have to pay more than a few dollars.

What you will need

Before you begin, it's a good idea to install some necessary productivity tools:

An IBM Cloud pay-as-you-go account (make sure it is logged in to your account.)
The IBM CLI - the command line interface to interact with the IBM API. You will need to make sure it has the following plug-ins:
- openshift (oc)
- container registry (cr)
Make sure you have access to a Mac or Linux terminal.
The Kubernetes CLI - a command line interface for running commands against Kubernetes clusters.
Terraform - automates your resource provisioning
Docker We will be using Docker to create the images that will run your code inside your cluster — make sure you are logged into your Docker account
jq - a lightweight and flexible command-line JSON processor
Git - a free and open source distributed version control system
Node.js and npm

Step-by-step instructions

Step 1: Obtain an API key to deploy infrastructure to your account

Follow the steps in this document to create an API key and make a note of it for Step 2.

Step 2: Clone the repo and cd into the Terraform directory

git clone https://github.com/IBM-Cloud/truck-tracker.git
cd truck-tracker/terraform

Create a document called terraform.tfvars with the following fields:

ibmcloud_api_key = "<your_api_key_from_step_1>"
region = "eu-gb"
redis_password  = "<make_up_a_password>"

The terraform.tfvars document contains variables that you may want to keep secret so it is ignored by the GitHub repository.

Step 3: Create the infrastructure

TL;DR — run the Terraform script:

terraform init 
terraform apply --auto-approve

The Terraform folder contains a number of simple scripts:

main.tf tells Terraform to use the IBM Cloud.
variables.tf contains the variable definitions whose values will be populated from terraform.tfvars.
cloudant.tf creates a standard Cloudant DB and some credentials that we will use later to access it.
redis.tf creates the Redis instance and some credentials that we will use later to access it.
eventstreams.tf creates the Event Streams instance and some credentials that we will use later to access it.
registry.tf creates the Container Registry that will hold your container images.
iam.tf creates the access key that is needed to interact with the Container Registry.
vpc.tf creates the VPC (Virtual Private Cloud) infrastructure that will be needed to deploy an OpenShift cluster. This includes a VPC, subnets and public gateways in three different availability zones.
cos.tf create a single Cloud Object Storage instance that is needed by OpenShift to manage the cluster.
openshift.tf creates a cluster of virtual machines inside your VPC, where you will deploy your applications.

It will take several minutes for the databases and other resources to be ready, but you should now have some VPC resources, a cluster of virtual machines, an IBM Cloud® Databases for Redis instance, a Cloudant database and an Event Streams instance in your account, as well as a Container Registry namespace for your container images. You can check by visiting the Resources section of your IBM Cloud account.

Step 4: Create container images and deploy to OpenShift

At this point, all your infrastructure is ready to receive data. Now we have to build and deploy the different modules that will produce and consume the data, as well as the application that will display the truck position on a map.

Run the build script — but before you do, are you logged into the IBM Cloud CLI and Docker?

Go into the root of the project and type the following:

./build.sh

Step 5: Watch your trucks truckin'

The end of the script will output a URL — something like https:// some.thing.eu-gb.appdomain.cloud.

If you visit the URL that has been generated, you should be able to see a map of the US and, after a few seconds, several markers on the map that represent your trucks:

The Truck Tracker Wide View — Figure 3. All-Truck View

If you zoom in on one of these markers you will see it moving along the road. As the fleet manager, you now know where all your trucks are.

The Truck Tracker Zoom View — Figure 4. Single-Truck View

What you just did

The build.sh script

The build.sh script does a number of things:

It builds a docker image for each of the services you need. The producer, consumers and web application are all in different folders. Each folder has its own simple Dockerfile file that is used by the script to create a container image, which is then uploaded to the IBM Container Registry.
It creates a bunch of kubernetes secrets with all the credentials that are going to be needed to access the different IBM services (IBM Cloudant, Redis and Event Streams).
It deploys the application code (and the secretes) to the Red Hat OpenShift on IBM Cloud cluster using the instructions contained in the deployment.yml script.

The deployment.yml script

This script tells Openshift what to deploy to the machines in the Openshift cluster. In our case this is:

Data producers (trucktrackerproducer*).
Data consumers (trucktrackerredisconsumer and trucktrackercloudantconsumer)
Web application (trucktrackerweb)

In all cases, they pull an image from the container registry and get fed some environment variables with the credentials they require to access external services.

The Truck Tracker system

The Truck Tracker system is a set of simple Node.js scripts that use four main packages:

@ibm-cloud/cloudant to connect to IBM Cloudant and read/write data.
Redis to connect to the Redis instance and read/write data.
kafkajs to connect to the Event Streams instance.
Express to enable a simple web server that allows interaction with the data.

There are five main files:

server.js: This runs the web server and communicates with Redis. It is getting regular calls from the front end to the /data route. It connects to Redis, downloads an object with the ID and position of every truck and returns that to the front end.
index.html: This is the one and only page of the front end application. When it loads, it shows a map of the US from the awesome Open Street Map service. Then it is on a five-second loop where it makes an HTTP request to the backend (server.js), retrieves all the trucks and their positions and loops through them to plot their latest position on the map. Each truck has a map marker whose lat/long gets updated with every cycle.
producer.js: This takes in a location file and a truck_id from the script invocation and then loops through all the data points in the location file. Each data point is turned into a message with the truck ID and lat/long and posted once a second to Event Streams. For added fun, it picks a random starting point along the route and turns around and comes back when it reaches the end of the location file.
cloudantConsumer.js: This script creates a trucktracker database in Cloudant if it does not already exist and then polls Event Streams at regular intervals for more data. When data comes in, it gets written in bulk to Cloudant.
redisConsumer.js: This script polls Event Streams at regular intervals for data. When data comes in, it gets written into a Redis "hash" where the hash key is the truck ID and the value is the truck data (lat/long). This hash arrangement allows all the truck data to be pulled down by the server.js script in one go. The great thing is that if for some reason one of your consumers crashes, it will pick up where it left off without losing any data, because Event Streams knows the last data point it served to each consumer. Decoupling data-producing systems from data-consuming systems in this way is a good design practice that increases resilience and fault tolerance.

Summary

In this tutorial, we have combined a number of IBM services to model a realistic IoT pattern where Internet-enabled devices on trucks are producing data that gets consumed by various applications for different purposes.

Because of the managed nature of all these services, as an application developer, you can focus on solving your business problems and leave us to do the heavy lifting of managing the infrastructure for you, thereby saving potentially months of work on non-core activities like server provisioning, software deployment, patching, security and others.

If you followed this tutorial, remember to de-provision your infrastructure to stop incurring charges. On your terminal, do the following:

cd terraform/
terraform destroy

If you want to take the next step in your developer journey, check out some of our trial offers.