Using Calico network policies to control traffic on Classic clusters

Learn how to use Calico policies to allow network traffic from and to certain IP addresses.

Note that the follow steps are for Classic clusters with Classic LoadBalancers.

By default, Kubernetes NodePort, LoadBalancer, and Ingress services make your app available on all public and private cluster network interfaces. The allow-node-port-dnat default Calico policy permits incoming traffic from NodePort, network load balancer (NLB), and Ingress application load balancer (ALB) services to the app pods that those services expose. Kubernetes uses destination network address translation (DNAT) to forward service requests to the correct pods.

However, for security reasons, you might need to allow traffic to the networking services from certain source IP addresses only. You can use Calico Pre-DNAT policies to allow or block traffic from or to certain IP addresses. Pre-DNAT policies prevent specified traffic from reaching your apps because they are applied before Kubernetes uses regular DNAT to forward traffic to pods. When you create Calico Pre-DNAT policies, you choose whether to allow or block source IP addresses. For most scenarios, allowing specific traffic provides the most secure configuration because all traffic is blocked except traffic from known, permitted source IP addresses. denying specific traffic is typically useful only in scenarios such as preventing an attack from a small set of IP addresses.

In this scenario, you play the role of a networking administrator for a PR firm, and you notice some unusual traffic that hits your apps. The lessons in this tutorial walk you through creating a sample web server app, exposing the app by using a network load balancer (NLB) service, and protecting the app from unwanted unusual traffic with both allowlist and blocklist Calico policies.

Objectives

Learn to block all incoming traffic to all node ports by creating a high-order Pre-DNAT policy.
Learn to allow specific source IP addresses to access the NLB public IP and port by creating a low-order Pre-DNAT policy. Lower-order policies override higher-order policies.
Learn to block specific source IP addresses from accessing the NLB public IP and port by creating a low-order Pre-DNAT policy.

Audience

This tutorial is intended for software developers and network administrators who want to manage network traffic to an app.

Prerequisites

Create a classic cluster with at least 3 worker nodes. Single worker node clusters don’t have the required resources to complete this tutorial. This tutorial is not available for VPC clusters.
Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.
Install and configure the Calico CLI.
Ensure that you have the following IBM Cloud IAM access policies for IBM Cloud Kubernetes Service:
- Any platform access role
- The Writer or Manager service access role

Deploy an app and expose it by using an NLB

The first lesson shows you how your app is exposed from multiple IP addresses and ports, and where public traffic is coming into your cluster.

Start by deploying a sample web server app to use throughout the tutorial. The echoserver web server shows data about the connection that is made to the cluster from the client, and you can test access to the PR firm's cluster. Then, expose the app by creating a network load balancer (NLB) 1.0 service. An NLB 1.0 service makes your app available over both the NLB service IP address and the worker nodes' node ports.

Want to use an Ingress application load balancer (ALB)? Instead of creating an NLB in steps 3 and 4, create a service for the web server app and create an Ingress resource for the web server app. Then get the public IPs of your ALBs by running ibmcloud ks ingress alb ls --cluster <cluster_name> and use these IPs throughout the tutorial in place of the <loadbalancer_IP>.

The following image shows how the web server app is exposed to the internet by the public node port and public NLB at the end of Lesson 1.

At the end of Lesson 1, the web server app is exposed to the internet by the public node port and public NLB. — Figure 1. The web server app is exposed to the internet by the public node port and public NLB.

Deploy the sample web server app. When a connection is made to the web server app, the app responds with the HTTP headers that it received in the connection.
```
kubectl apply -f https://raw.githubusercontent.com/IBM-Cloud/kube-samples/master/deploy-apps-clusters/webserver.yaml
```

Verify that the web server app pods have a STATUS of Running.

kubectl get pods -o wide

Example output

NAME                         READY     STATUS    RESTARTS   AGE       IP               NODE
webserver-855556f688-6dbsn   1/1       Running   0          1m        172.30.xxx.xxx   10.176.48.78
webserver-855556f688-76rkp   1/1       Running   0          1m        172.30.xxx.xxx   10.176.48.78
webserver-855556f688-xd849   1/1       Running   0          1m        172.30.xxx.xxx   10.176.48.78

To expose the app to the public internet, create an NLB 1.0 service configuration file called webserver-lb.yaml in a text editor.

apiVersion: v1
kind: Service
metadata:
  labels:
    run: webserver                               
  name: webserver-lb
spec:
  type: LoadBalancer
  selector:
    run: webserver
  ports:
  - name: webserver-port
    protocol: TCP
    port: 8080
    targetPort: 8080 # Optional. By default, the `targetPort` is set to match the `port` value unless specified otherwise.

Deploy the NLB.

kubectl apply -f filepath/webserver-lb.yaml

Verify that you can publicly access the app that is exposed by the NLB from your computer.

Get the public EXTERNAL-IP address of the NLB.

kubectl get svc -o wide

Example output

NAME           CLUSTER-IP       EXTERNAL-IP        PORT(S)        AGE       SELECTOR
webserver-lb   172.21.xxx.xxx   169.xx.xxx.xxx     80:31024/TCP   2m        run=webserver

Create a cheat sheet text file, and copy the NLB IP into the text file. The cheat sheet helps you more quickly use values in later lessons.

Verify that you can publicly access the external IP for the NLB.

curl --connect-timeout 10 <loadbalancer_IP>:80

The following example output confirms that the NLB exposes your app on the 169.1.1.1 public NLB IP address. The webserver-855556f688-76rkp app pod received the curl request.

Hostname: webserver-855556f688-76rkp
Pod Information:
    -no pod information available-
Server values:
    server_version=nginx: 1.13.3 - lua: 10008
Request Information:
    client_address=10.176.XX.XX
    method=GET
    real path=/
    query=
    request_version=1.1
    request_scheme=http
    request_uri=http://169.1.1.1:8080/
Request Headers:
    accept=*/*
    host=169.1.1.1
    user-agent=curl/7.54.0
Request Body:
    -no body in request-

Verify that you can publicly access the app that is exposed by the node port from your computer. An NLB service makes your app available over both the NLB service IP address and the worker nodes' node ports.

Get the node port that the NLB assigned to the worker nodes. The node port is in the 30000 - 32767 range.

kubectl get svc -o wide

In the following example output, the node port is 31024:

NAME           CLUSTER-IP       EXTERNAL-IP        PORT(S)        AGE       SELECTOR
webserver-lb   172.21.xxx.xxx   169.xx.xxx.xxx     80:31024/TCP   2m        run=webserver

For classic clusters, get the Public IP address of a worker node. For VPC clusters, get the Private IP address instead.

ibmcloud ks worker ls --cluster <cluster_name>

Example output

ID                                                 Public IP        Private IP     Machine Type        State    Status   Zone    Version   
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w1   169.xx.xxx.xxx   10.176.48.67   u3c.2x4.encrypted   normal   Ready    dal10   1.29_1513*   
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w2   169.xx.xxx.xxx   10.176.48.79   u3c.2x4.encrypted   normal   Ready    dal10   1.29_1513*   
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w3   169.xx.xxx.xxx   10.176.48.78   u3c.2x4.encrypted   normal   Ready    dal10   1.29_1513*

Copy the public IP of the worker node and the node port into your text cheat sheet to use in later lessons.
Verify that you can access the public IP address the worker node through the node port. Note: Because worker nodes in VPC clusters don't have a public IP address, you can access an app through a NodePort only if you are connected to your private VPC network, such as through a VPN connection. Then, you can use the worker node's private IP address and NodePort: <worker_private_IP>:<NodePort>.
```
curl  --connect-timeout 10 <worker_IP>:<NodePort>
```
The following example output confirms that the request to your app came through the private IP address 10.1.1.1 for the worker node and the 31024 node port. The webserver-855556f688-xd849 app pod received the curl request:
```
Hostname: webserver-855556f688-xd849
Pod Information:
    -no pod information available-
Server values:
    server_version=nginx: 1.13.3 - lua: 10008
Request Information:
    client_address=1.1.1.1
    method=GET
    real path=/
    query=
    request_version=1.1
    request_scheme=http
    request_uri=http://10.1.1.1:8080/
Request Headers:
    accept=*/*
    host=10.1.1.1:31024
    user-agent=curl/7.60.0
Request Body:
    -no body in request-
```

At this point, your app is exposed from multiple IP addresses and ports. Most of these IPs are internal to the cluster and can be accessed only over the private network. Only the public node port and public NLB port are exposed to the public internet.

Next, you can start creating and applying Calico policies to block public traffic.

Block all incoming traffic to all node ports

To secure the PR firm's cluster, you must block public access to both the NLB service and node ports that are exposing your app. Start by blocking access to node ports.

The following image shows how traffic is permitted to the NLB but not to node ports at the end of Lesson 2:

At the end of Lesson 2, the web server app is exposed to the internet by public NLB only. — Figure 2. The webserver app is exposed to the internet by public NLB only.

In a text editor, create a high-order Pre-DNAT policy called deny-nodeports.yaml to deny incoming TCP and UDP traffic from any source IP to all node ports.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: deny-nodeports
spec:
  applyOnForward: true
  preDNAT: true
  ingress:
  - action: Deny
    destination:
      ports:
      - 30000:32767
    protocol: TCP
    source: {}
  - action: Deny
    destination:
      ports:
      - 30000:32767
    protocol: UDP
    source: {}
  selector: ibm.role=='worker_public'
  order: 1100
  types:
  - Ingress

Apply the policy.

Linux:

calicoctl apply -f filepath/deny-nodeports.yaml

Windows and OS X:

calicoctl apply -f filepath/deny-nodeports.yaml --config=filepath/calicoctl.cfg

Example output

Successfully applied 1 'GlobalNetworkPolicy' resource(s)

Using the values from your cheat sheet, verify that you can't publicly access the worker node public IP address and node port.
```
curl  --connect-timeout 10 <worker_IP>:<NodePort>
```
The connection times out because the Calico policy you created is blocking traffic to node ports.
```
curl: (28) Connection timed out after 10016 milliseconds
```
Change the externalTrafficPolicy of the LoadBalancer you created in the previous lesson from Cluster to Local. Local ensures that the source IP of your system is preserved when you curl the external IP of the LoadBalancer in the next step.
```
kubectl patch svc webserver-lb -p '{"spec":{"externalTrafficPolicy":"Local"}}'
```
Using the value from your cheat sheet, verify that you can still publicly access the NLB external IP address.
```
curl --connect-timeout 10 <loadbalancer_IP>:80
```
Example output
```
Hostname: webserver-855556f688-76rkp
Pod Information:
    -no pod information available-
Server values:
    server_version=nginx: 1.13.3 - lua: 10008
Request Information:
    client_address=1.1.1.1
    method=GET
    real path=/
    query=
    request_version=1.1
    request_scheme=http
    request_uri=http://<loadbalancer_IP>:8080/
Request Headers:
    accept=*/*
    host=<loadbalancer_IP>
    user-agent=curl/7.54.0
Request Body:
    -no body in request-
```
In the Request Information section of the output, the source IP address is, for example, client_address=1.1.1.1. The source IP address is the public IP of the system that you're using to run curl. Otherwise, if you are connecting to the internet through a proxy or VPN, the proxy or VPN might be obscuring your system's actual IP address. In either case, the NLB sees your system's source IP address as the client IP address.
Copy your system's source IP address (client_address=1.1.1.1 in the previous step output) into your cheat sheet to use in later lessons.

Great! At this point, your app is exposed to the public internet from the public NLB port only. Traffic to the public node ports is blocked. Your cluster is partially locked down from unwanted traffic.

Next, you can create and apply Calico policies to allow traffic from only certain source IPs.

Allow incoming traffic from a specific IP to the NLB

You now decide to completely lock down traffic to the PR firm's cluster and test access by allowing only your own computer's IP address.

First, in addition to the node ports, you must block all incoming traffic to the NLB exposing the app. Then, you can create a policy that allows your system's IP address. At the end of Lesson 3, all traffic to the public node ports and NLB is blocked and only traffic from your allowed system IP is allowed.

The web server app is exposed by public NLB to your system IP only. — Figure 3. The webserver app is exposed by public NLB to your system IP only.

In a text editor, create a high-order Pre-DNAT policy called deny-lb-port-80.yaml to deny all incoming TCP and UDP traffic from any source IP to the NLB IP address and port. Replace <loadbalancer_IP> with the NLB public IP address from your cheat sheet.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: deny-lb-port-80
spec:
  applyOnForward: true
  preDNAT: true
  ingress:
  - action: Deny
    destination:
      nets:
      - <loadbalancer_IP>/32
      ports:
      - 80
    protocol: TCP
    source: {}
  - action: Deny
    destination:
      nets:
      - <loadbalancer_IP>/32
      ports:
      - 80
    protocol: UDP
    source: {}
  selector: ibm.role=='worker_public'
  order: 800
  types:
  - Ingress

Apply the policy.

Linux:

calicoctl apply -f filepath/deny-lb-port-80.yaml

Windows and OS X:

calicoctl apply -f filepath/deny-lb-port-80.yaml --config=filepath/calicoctl.cfg

Using the value from your cheat sheet, verify that you now can't access the public NLB IP address. The connection times out because the Calico policy you created is blocking traffic to the NLB.
```
curl --connect-timeout 10 <loadbalancer_IP>:80
```

In a text editor, create a low-order Pre-DNAT policy called allowlist.yaml to allow traffic from your system's IP to the NLB IP address and port. Using the values from your cheat sheet, replace <loadbalancer_IP> with the public IP address of the NLB and <client_address> with the public IP address of your system's source IP. If you can't remember your system IP, you can run curl ifconfig.co.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allowlist
spec:
  applyOnForward: true
  preDNAT: true
  ingress:
  - action: Allow
    destination:
      nets:
      - <loadbalancer_IP>/32
      ports:
      - 80
    protocol: TCP
    source:
      nets:
      - <client_address>/32
  selector: ibm.role=='worker_public'
  order: 500
  types:
  - Ingress

Apply the policy.

Linux:

calicoctl apply -f filepath/allowlist.yaml

Windows and OS X:

calicoctl apply -f filepath/allowlist.yaml --config=filepath/calicoctl.cfg

Your system's IP address is now allowed.

Using the value from your cheat sheet, verify that you now can access the public NLB IP address.
```
curl --connect-timeout 10 <loadbalancer_IP>:80
```
If you have access to another system that has a different IP address, try to access the NLB from that system.
```
curl --connect-timeout 10 <loadbalancer_IP>:80
```
The connection times out because that system's IP address isn't allowed.

At this point, all traffic to the public node ports and NLB is blocked. Only traffic from your allowed system IP is allowed.

Deny incoming traffic from specific IPs to the NLB

In the previous lesson, you blocked all traffic and allowed only a few IPs. That scenario works well for testing purposes when you want to limit access to only a few controlled source IP addresses. However, the PR firm has apps that need to be widely available to the public. You need to make sure that all traffic is permitted except for the unusual traffic you are seeing from a few IP addresses. A denylist is useful in a scenario like this one because it can help you prevent an attack from a small set of IP addresses.

In this lesson, block traffic from your own system's source IP address. At the end of Lesson 4, all traffic to the public node ports is blocked, and all traffic to the public NLB is allowed. Only traffic from your specific system IP to the NLB is blocked.

The web server app is exposed by public NLB to the internet. Only traffic from your system IP is blocked. — Figure 4. The webserver app is exposed by public NLB to the internet.

Clean up the allowlist policies you created in the previous lesson.

Linux:

calicoctl delete GlobalNetworkPolicy deny-lb-port-80

calicoctl delete GlobalNetworkPolicy allowlist

Windows and OS X:

calicoctl delete GlobalNetworkPolicy deny-lb-port-80 --config=filepath/calicoctl.cfg

calicoctl delete GlobalNetworkPolicy allowlist --config=filepath/calicoctl.cfg

Now, all incoming TCP and UDP traffic from any source IP to the NLB IP address and port is permitted again.

To deny all incoming TCP and UDP traffic from your system's source IP address to the NLB IP address and port, create a low-order pre-DNAT policy called blocklist.yaml in a text editor. Using the values from your cheat sheet, replace <loadbalancer_IP> with the public IP address of the NLB and <client_address> with the public IP address of your system's source IP.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: blocklist
spec:
  applyOnForward: true
  preDNAT: true
  ingress:
  - action: Deny
    destination:
      nets:
      - <loadbalancer_IP>/32
      ports:
      - 80
    protocol: TCP
    source:
      nets:
      - <client_address>/32
  - action: Deny
    destination:
      nets:
      - <loadbalancer_IP>/32
      ports:
      - 80
    protocol: UDP
    source:
      nets:
      - <client_address>/32
 selector: ibm.role=='worker_public'
 order: 500
 types:
 - Ingress

Apply the policy.

Linux:

calicoctl apply -f filepath/blocklist.yaml

Windows and OS X:

calicoctl apply -f filepath/blocklist.yaml --config=filepath/calicoctl.cfg

Your system's IP address is now blocked.

Using the value from your cheat sheet, verify from your system that you can't access the NLB IP because your system's IP is blocked.
```
curl --connect-timeout 10 <loadbalancer_IP>:80
```
At this point, all traffic to the public node ports is blocked, and all traffic to the public NLB is allowed. Only traffic from your specific system IP to the NLB is blocked.

Great work! You successfully controlled traffic into your app by using Calico Pre-DNAT policies to block source IPs.

Logging blocked traffic from specific IPs to the NLB

In the previous lesson, you blocked traffic from your system IP to the NLB. In this lesson, you can learn how to log the denied traffic requests.

In this example scenario, the PR firm you work for wants you to set up a logging trail for any unusual traffic that is continuously being denied by one of your network policies. To monitor the potential security threat, you set up logging to record every time that your blocklist policy denies an attempted action on the NLB IP.

Create a Calico NetworkPolicy named log-denied-packets. This log policy uses the same selector as the blocklist policy, which adds this policy to the Calico Iptables rule chain. By using a lower-order number, such as 300, you can ensure that this rule is added to the Iptables rule chain before the blocklist policy. Packets from your IP are logged by this policy before they try to match the blocklist policy rule and are denied.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
    name: log-denied-packets
 spec:
  applyOnForward: true
  preDNAT: true
  ingress:
  - action: Log
    destination:
      nets:
      - <loadbalancer_IP>/32
      ports:
      - 80
    protocol: TCP
    source:
      nets:
      - <client_address>/32
  - action: Log
    destination:
      nets:
      - <loadbalancer_IP>/32
      ports:
      - 80
    protocol: UDP
    source:
      nets:
      - <client_address>/32
  selector: ibm.role=='worker_public'
  order: 300
  types:
  - Ingress

Apply the policy.

Linux:

calicoctl apply -f /log-denied-packets.yaml

Windows and OS X:

calicoctl apply -f /log-denied-packets.yaml --config=<filepath>/calicoctl.cfg

Generate log entries by sending requests from your system IP to the NLB IP. These request packets are logged before they are denied.
```
curl --connect-timeout 10 <loadbalancer_IP>:80
```

Check for log entries that are written to the /var/log/syslog path. The log entry looks similar to the following.

Sep 5 14:34:40 <worker_hostname> kernel: [158271.044316] calico-packet: IN=eth1 OUT= MAC=08:00:27:d5:4e:57:0a:00:27:00:00:00:08:00 SRC=192.XXX.XX.X DST=192.XXX.XX.XX LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52866 DF PROTO=TCP SPT=42962 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

Nice! You set up logging so that blocked traffic can be monitored more easily.

If you want to clean up the blocklist and the log policies:

Clean up the blocklist policy.

Linux:

calicoctl delete GlobalNetworkPolicy blocklist

Windows and OS X:

calicoctl delete GlobalNetworkPolicy blocklist --config=filepath/calicoctl.cfg

Clean up the log policy.

Linux:

calicoctl delete GlobalNetworkPolicy log-denied-packets

Windows and OS X:

calicoctl delete GlobalNetworkPolicy log-denied-packets --config=filepath/calicoctl.cfg

What's next?

Read more about controlling traffic with network policies.
For more example Calico network policies that control traffic to and from your cluster, you can check out the Calico tutorials.