Viewing flow log objects

Flow logs object header fields (per Object StorageS object)

Flow log fields

In most cases, you can find the direction field by comparing the vNIC’s private IP with the source and destination IPs. However, the field is convenient for queries.

Example flow log object

    {
        "version": "0.0.1",
        "collector_crn": "crn",
        "attached_endpoint_type": "vnic",
        "network_interface_id": "crn",
        "instance_crn": "crn",
        "vpc_crn": "crn"
        "capture_end_time": "2008-09-15T15:53:00Z",
        "capture_start_time": "2008-09-15T15:00:00Z",
        "state": "ok",
        "number_of_flow_logs" : 1,
        "flow_logs": [
            {
                "start_time": "2008-09-15T15:53:00",
                "end_time": "2008-09-15T15:40:00",
                "direction": "O",
                "action": "accepted",
                "initiator_ip": "1.2.3.4",
                "initiator_port": 20033,
                "target_ip": "5.6.7.8",
                "target_port": 80,
                "transport_protocol": 6,
                "ether_type": "IPv4",
                "was_initiated": true,
                "was_terminated": false,
                "bytes_from_initiator": 12000,
                "packets_from_initiator": 2212,
                "bytes_from_target": 323232,
                "packets_from_target": 3232
                "cumulative_packets_from_initiator": 2212,
                "cumulative_packets_from_target": 3232,
                "cumulative_bytes_from_target": 323232,
                "cumulative_bytes_from_initiator": 12000,
            }
        ],
    }

Analyzing flow logs by using IBM Cloud SQL Query

You can analyze flow logs with SQL that uses IBM Cloud® Data Engine by using the path to the objects on Object Storage in the FROM clause of your SQL query. (For an example, see Step 3 in Viewing flow log objects. However, for a more convenient way of querying, it is recommended that you create table and view definitions that give you a flattened view on your flows. For more information about table catalog functions and benefits, see Getting started with the catalog.

The view flattens the flow log data structure; thus, making it easier to analyze your flows.

For more elaborate and repeatable analysis, such as when you want to collaborate and share your log analysis, it is recommended that you use IBM Cloud® Data Engine through Jupyter Notebooks (for example, through IBM Watson Studio). For a generic starter and demo notebook for IBM Cloud® Data Engine that you can use as basis for your work, see Using IBM Cloud SQL Query.

Important: Replace these variables in the following steps:

• bucket - The bucket where your flow logs are stored.
• region - The region alias of the bucket that holds your flow logs.

To analyze flow logs, follow these steps:

1. Define a table for flow logs:

   -- Table definition for flow logs
   CREATE TABLE FLOW(
    version string,
    collector_crn string,
    attached_endpoint_type string,
    network_interface_id string,
    instance_crn string,
    capture_start_time timestamp,
    capture_end_time timestamp,
    number_of_flow_logs int,
    flow_logs array<struct<
         start_time: string,
         end_time: string,
         connection_start_time: string,
         direction: string,
         action: string,
         initiator_ip: string,
         target_ip: string,
         initiator_port: int,
         target_port: int,
         transport_protocol: int,
         ether_type: string,
         was_initiated: boolean,
         was_termined: boolean,
         bytes_from_initiator: long,
         packets_from_initiator: long,
         bytes_from_target: long,
         packets_from_target: long,
         cumulative_bytes_from_initiator: long,
         cumulative_packets_from_initiator: long,
         cumulative_bytes_from_target: long,
         cumulative_packets_from_target: long
        >>,
      account string,
      region string,
       `vpc-id` string,
       `subnet-id` string,
       `endpoint-type` string,
       `instance-id` string,
       `vnic-id` string,
       `record-type` string,
       year int,
       month int,
       day int,
       hour int,
       `stream-id` string
   ) USING JSON
   LOCATION cos://<region>/<bucket>/ibm_vpc_flowlogs_v1/
   

2. Define a view for convenient flow log queries:

   CREATE VIEW FLOW_FLAT AS
   WITH EXPLODED_FLOW as (
       SELECT version,collector_crn,attached_endpoint_type,network_interface_id, instance_crn, capture_start_time, capture_end_time, `vnic-id`, `record-type`, year, month, day, hour, `stream-id`, explode(flow_logs) as flow FROM FLOW)
   SELECT version, collector_crn, attached_endpoint_type, network_interface_id, instance_crn, capture_start_time, capture_end_time, `vnic-id`, `record-type`, year, month, day, hour, `stream-id`, flow.*
   FROM EXPLODED_FLOW
   

You can verify that flow log data is being collected by using IBM Cloud Monitoring. For more information, see Monitoring flow logs for VPC metrics.

SELECT * FROM FLOW_FLAT WHERE CAST(capture_start_time AS date) = DATE("2020-07-24") LIMIT 100

SELECT hour(capture_start_time) as hour,
    network_interface_id,
    initiator_ip,
    target_port
FROM FLOW_FLAT
WHERE
    transport_protocol = 6 AND
    action = "accepted" AND
    CAST(capture_start_time AS date) = DATE("2020-07-24")
SORT BY hour
LIMIT 100

SELECT target_ip,
    SUM(packets_from_initiator) AS `packets_received`,
    SUM(packets_from_target) AS `packets_sent`
FROM
    FLOW_FLAT
WHERE
    target_port = 443 AND
    day = day(current_date() - INTERVAL 7 days) AND
    month = month(current_date() - INTERVAL 7 days) AND
    year = year(current_date() - INTERVAL 7 days)
GROUP BY target_ip
ORDER BY `packets_received` DESC LIMIT 10

SELECT target_ip,
    SUM(bytes_from_initiator) AS `bytes`
FROM
    FLOW_FLAT
WHERE
    capture_start_time > current_timestamp() - INTERVAL 1 hours AND
    hour = hour(current_date() - INTERVAL 1 hours) AND
    day = day(current_date() - INTERVAL 1 hours) AND
    month = month(current_date() - INTERVAL 1 hours) AND
    year = year(current_date() - INTERVAL 1 hours)
GROUP BY target_ip
ORDER BY `bytes` DESC LIMIT 5

Example solution: Analyzing flow logs

You can download an example solution of how to use IBM Log Analysis to analyze flow logs from the IBM-Cloud vpc-flowlogs Github repository. This project (Readme file) shows how to use a trigger function to read a flow log Object Storage object and write it to IBM Log Analysis.

Viewing generated flow log files from the Object Storage bucket

If you use the Lite (free) pricing plan for IBM Cloud Object Storage, you cannot use an SQL query to confirm the creation of flow log objects in the Object Storage bucket without purchasing the Standard pricing plan. As an alternative, you can access generated flow log object files directly from the Object Storage bucket to verify that the files were created successfully and that the objects are being generated.

To view generated flow log files from the Object Storage bucket, follow these steps:

1. To ensure that the flow logs are being captured, check the Object Storage bucket.

2. Navigate in the folder to find your VPC or the resource where you configured monitoring.

3. Drill-down into the resource to find the network resource that you are monitoring.

   If configured correctly, you should see several gzip files for each log entry.

Using SQL queries, you can flatten and view these objects or any other type of view of all these files. In our case, we want to ensure that data is being captured. Download one of the files to view its contents to ensure that traffic is being captured.