Viewing flow log objects

Flow log data format

Flow log traffic summaries contain the following information:

• Byte/packet counts, separately for RX (receive) and TX (transmit).
• Whether a connection starts or stops in the time window.

Because a flow log reflects network traffic in a limited window of time, a long-running connection might cause multiple flow logs to be emitted. For every connection processed by a vNIC, a time-ordered sequence of flow logs emits (not including failures). These flow logs appear in one or more Object Storage objects.

The initiator_ip in a flow log is defined to be the source_ip that appears in the first packet of its connection that reached the vNIC. At the implementation level, this packet is typically the one that causes the new connection to be added to the connection table. Similarly, the target_ip in the flow log is set to the dest_ip field in that same packet.

If both the connection initiator vNIC and connection target vNIC enable flow logs, the initiator and target IP and ports fields are identical in the flow logs on both vNICs.

Example: Consider a client that is sending an HTTP request to a web server. The flow log for this HTTP request on the client-side vNIC has the client’s IP address as initiator_ip. The corresponding flow log on the server-side vNIC has the same initiator_ip.

The start_time and end_time in a flow log reflects:

• Capture time - The time that data path elements were queried for traffic counters.
• Data path time - The time as maintained in the data path element itself.

It's possible that the flow log does not reflect all traffic (for example, in the data path) between a flow log start_time and end_time. In other words, it might be that packets sent and received by the vNIC toward the end of the capture window are reflected only in a flow log with the later start_time window.

Flow logs reflect actual traffic on connections: If traffic does not occur on a connection in a capture window, no flow log appears for it in the Object Storage object for that window. Meaning that the sequence of flow logs for a connection might be mapped to a sequence of non-consecutive Object Storageobjects.

The flow logs array within an object does not need to be sorted in any specific order.

Rejected traffic:

• A specific flow log summarizes either rejected or accepted traffic, never both.
• The sequence of flow logs for a connection might have time-window overlaps between accepted and rejected flow logs.
• Rejected packets that are not associated with an existing connection (for example, a security group, or network ACL, denies the connection initiation).
• Rejected packets on an existing connection (for example, traffic that does not match the TCP state-machine, or a network ACL rule changed mid-connection).
• Rejected packets are written to separate flow logs based on whether they are incoming or outgoing.

Flows are tagged as rejected if their packets were blocked by security group or network ACL rules.

Flow log object format

Flow logs are written to the user-specified Object Storage bucket in the following naming convention:

ibm_vpc_flowlogs_v1/account={account}/region={region}/vpc-id={vpc-id}/subnet-id={subnet-id}/endpoint-type=vnics/instance-id={vsi-id}/vnic-id={vnic-id}/record-type={all|ingress|egress}/year={xxxx}/month={yy}/day={zz}/hour={hh}/stream-id={stream-id}/{sequence-number}.gz

Where:

• {stream-id} is a date and time string in ISO 8601 format yyyymmddThhmmssZ (Coordinated Universal Time), defined as the time the first object in the directory was created.
• {sequence-number} is a running counter of objects within the stream, represented as an eight-character, zero-filled field (%08d).
• {all|ingress|egress} shows the type of traffic the flow includes.
• {xxxx} is the year in four-digit format.
• {yy} is the month in two-digit, zero-leading format.
• {zz} is the day of the month in two-digit, zero-leading format.
• {hh} is the hour in zero-leading format (for example, 00-24).

Any instances of URL-unsafe characters in the path are URL-encoded.

A flow log's Object Storage object contains a single valid JSON object. The Object Storage object is .gz compressed.

The object header fields that are specified in the following table are written to the Object Storage object metadata.

    {
        "version": "0.0.1",
        "collector_crn": "crn",
        "attached_endpoint_type": "vnic",
        "network_interface_id": "crn",
        "instance_crn": "crn",
        "vpc_crn": "crn"
        "capture_end_time": "2008-09-15T15:53:00Z",
        "capture_start_time": "2008-09-15T15:00:00Z",
        "state": "ok",
        "number_of_flow_logs" : 1,
        "flow_logs": [
            {
                "start_time": "2008-09-15T15:53:00",
                "end_time": "2008-09-15T15:40:00",
                "direction": "O",
                "action": "accepted",
                "initiator_ip": "1.2.3.4",
                "initiator_port": 20033,
                "target_ip": "5.6.7.8",
                "target_port": 80,
                "transport_protocol": 6,
                "ether_type": "IPv4",
                "was_initiated": true,
                "was_terminated": false,
                "bytes_from_initiator": 12000,
                "packets_from_initiator": 2212,
                "bytes_from_target": 323232,
                "packets_from_target": 3232
                "cumulative_packets_from_initiator": 2212,
                "cumulative_packets_from_target": 3232,
                "cumulative_bytes_from_target": 323232,
                "cumulative_bytes_from_initiator": 12000,
            }
        ],
    }