Centralize communication through a VPC Transit Hub and Spoke architecture - Part one

Objectives

• Understand the concepts behind a VPC based hub and spoke model.
• Understand the implementation of a firewall-router and a transit VPC environment.
• Understand VPC ingress and egress routing.
• Identify and optionally resolve asymmetric routing issues.
• Connect VPCs via a Transit Gateway.

Before you begin

This tutorial requires:

• terraform to use Infrastructure as Code to provision resources,
• python to optionally run the pytest commands,
• Implementing a firewall-router will require that you enable IP spoofing checks,
• An SSH key to connect to the virtual servers. If you don't have an SSH key, follow the instructions for creating a key for VPC.

See the prerequisites for a few options including a Dockerfile to easily create the prerequisite environment.

In addition:

• Check for user permissions. Be sure that your user account has sufficient permissions to create and manage all the resources in this tutorial. See the list of:
  • required permissions for VPC.
  • required permissions for creating Transit Gateway.
  • required permissions for IP spoofing checks.

IP Address and Subnet Layout

In this step you will provision the VPC network resources. Carefully plan by designing an addressing plan for a VPC and use non-overlapping CIDR blocks.

It is tempting to divide up the CIDR space first by VPC but this complicates routing. Instead think of an availability zone as a single CIDR block and each VPC as consuming a slice of it.

This diagram shows just zone 1 in more detail. The subnet sizes and layout are identical in the other zones:

Above the enterprise is on the left and the IBM Cloud on the right. In the IBM Cloud for simplictiy a single zone is depicted for the transit VPC and Spoke 0. Notice the CIDR blocks do not overlap and VPCs all consume a CIDR block in each zone:

• The on-premises CIDR is 192.168.0.0/16.
• The zones in this multi-zone region are 10.*.0.0/16. The second digit: 1, 2, 3 is the zone number (shown for Dallas/us-south):
  • 10.1.0.0/16, zone 1, Dallas 1, us-south-1.
  • 10.2.0.0/16, zone 2, Dallas 2, us-south-2.
  • 10.3.0.0/16, zone 3, Dallas 3, us-south-3.
• The transit VPC consumes CIDRs 10.*.15.0/24:
  • 10.1.15.0/24, zone 1.
  • 10.2.15.0/24, zone 2.
  • 10.3.15.0/24, zone 3.
• Spoke 0 consumes 10.*.0.0/24 or CIDRs:
  • 10.1.0.0/24, zone 1.
  • 10.2.0.0/24, zone 2.
  • 10.3.0.0/24, zone 3.
• The subnet CIDRs further divide the /24 into /26.

The subnets in the transit and spoke are for the different resource types:

• worker - network accessible compute resources VPC instances, load balancers, Red Hat OpenShift, etc. VPC instances are demonstrated in this tutorial.
• dns - DNS Services location appliances used in part two.
• vpe - VPE for VPC used in part two.
• fw - firewall-router VPC instances (only in transit).

Provision VPC network resources

1. The companion GitHub Repository has the source files to implement the architecture. In a desktop shell clone the repository:

   git clone https://github.com/IBM-Cloud/vpc-transit
   cd vpc-transit
   

2. The config_tf directory contains configuration variables that you are required to configure.

   cp config_tf/template.terraform.tfvars config_tf/terraform.tfvars
   

3. Edit config_tf/terraform.tfvars and use the comments in that file as your guide.

4. Since it is important that each layer is installed in the correct order and some steps in this tutorial will install multiple layers a shell command ./apply.sh is provided. The following will display help:

   ./apply.sh
   

5. You could apply all of the layers configured by executing ./apply.sh : :. The colons are shorthand for first (or config_tf) and last (power_tf). The -p prints the layers:

   ./apply.sh -p : :
   

   It will look something like:

   directories: config_tf enterprise_tf transit_tf spokes_tf transit_spoke_tgw_tf test_instances_tf test_lbs_tf enterprise_link_tf firewall_tf transit_ingress_tf spokes_egress_tf all_firewall_tf all_firewall_asym_tf dns_tf vpe_transit_tf vpe_spokes_tf power_tf
   

6. If you don't already have one, obtain a Platform API key and export the API key for use by Terraform:

   export IBMCLOUD_API_KEY=YourAPIKEy
   

7. In this first step apply in config_tf, enterprise_tf, transit_tf, spokes_tf and transit_spoke_tgw_tf:

   ./apply.sh : transit_spoke_tgw_tf
   

The VPCs and subnets have been created. The transit VPC and spoke VPCs have been connected through a provisioned Transit Gateway. Open the Virtual Private Clouds in the browser. Open the transit VPC and note the CIDR blocks for address prefixes and subnets. Examine the enterprise and spoke VPCs as well. Open Transit Gateway and click the transit gateway to see the connection between the transit and spoke VPCs.

Create test instances

VPC Virtual Server Instances, VSIs, are provisioned to test the network connectivity. A test instance will be added to each of the worker subnets (one per zone) in the enterprise, transit and each of the spokes. If the default configuration of 3 zones and 2 spokes is used then 12 instances will be provisioned.

1. Create the test instances

   ./apply.sh test_instances_tf
   

It can be enlightening to explore the resources created at each step in the IBM Cloud console. Optionally open the Virtual Private Clouds. On the left click on the Virtual server instances and notice the instances that were created.

Testing

This tutorial will add communication paths one layer at a time. A pytest test suite will be used to exhaustively tests communication paths. By the end of the tutorial all of the tests are expected to pass.

It is not required for the reader to use pytest to verify the results. Follow along in the tutorial, apply the layers, and trust the results described in the tutorial. The reader can still explore the VPC resources like VSIs, subnets and route tables after they are created.

Each pytest test will SSH to one of the instances and perform a type of connectivity test, like executing a curl command to one of the other instances. The default SSH environment is used to log in to the instances. If you see unexpected test results try the pytest troubleshooting section.

1. Run the zone 1 curl tests in the suite by using the -m (markers) flag. Choose the tests marked with curl, lz1 (left zone 1) and rz1 (right zone 1).

   Your expected results are: Connectivity within a VPC, like enterprise <-> enterprise will be PASSED. Connectivity between transit and spokes will be Passed. Cross VPC from enterprise -> transit or spokes will be FAILED.

   pytest -m "curl and lz1 and rz1"
   

   Below is an example output:

   root@ea28970e0897:/usr/src/app# pytest -m "curl and lz1 and rz1"
   ===================================================== test session starts ======================================================
   platform linux -- Python 3.12.3, pytest-8.1.1, pluggy-1.4.0 -- /usr/local/bin/python
   cachedir: .pytest_cache
   rootdir: /usr/src/app
   configfile: pytest.ini
   testpaths: py
   plugins: xdist-3.5.0
   collected 36 items / 20 deselected / 16 selected
   
   py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-enterprise-z1-worker] PASSED                                   [  6%]
   py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-transit-z1-worker] FAILED                                      [ 12%]
   py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-spoke0-z1-worker] FAILED                                       [ 18%]
   py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-spoke1-z1-worker] FAILED                                       [ 25%]
   py/test_transit.py::test_curl[l-transit-z1-worker -> r-enterprise-z1-worker] FAILED                                      [ 31%]
   py/test_transit.py::test_curl[l-transit-z1-worker -> r-transit-z1-worker] PASSED                                         [ 37%]
   py/test_transit.py::test_curl[l-transit-z1-worker -> r-spoke0-z1-worker] PASSED                                          [ 43%]
   py/test_transit.py::test_curl[l-transit-z1-worker -> r-spoke1-z1-worker] PASSED                                          [ 50%]
   py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-enterprise-z1-worker] FAILED                                       [ 56%]
   py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-transit-z1-worker] PASSED                                          [ 62%]
   py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-spoke0-z1-worker] PASSED                                           [ 68%]
   py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-spoke1-z1-worker] PASSED                                           [ 75%]
   py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-enterprise-z1-worker] FAILED                                       [ 81%]
   py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-transit-z1-worker] PASSED                                          [ 87%]
   py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-spoke0-z1-worker] PASSED                                           [ 93%]
   py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-spoke1-z1-worker] PASSED                                           [100%]
   
   =================================================== short test summary info ====================================================
   FAILED py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-transit-z1-worker] - assert False
   FAILED py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-spoke0-z1-worker] - assert False
   FAILED py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-spoke1-z1-worker] - assert False
   FAILED py/test_transit.py::test_curl[l-transit-z1-worker -> r-enterprise-z1-worker] - assert False
   FAILED py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-enterprise-z1-worker] - assert False
   FAILED py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-enterprise-z1-worker] - assert False
   ========================================= 6 failed, 10 passed, 20 deselected in 38.76s =========================================
   

A change to the network configuration can take a couple of test runs for the underlying VPC network system to become consistent. If you do not see the expected results initially be prepared to run the test again a couple of times.

The r- and l- stand for right and left. The middle part of the name identifies enterprise, transit, spoke0, spoke1, ... The z1, z2, ... identify the zone. The test will SSH to the left instance. On the left instance the connectivity to the right instance is attempted. The test_curl performs a curl connectivity on the left instance to the right instance.

In summary the test test_curl[l-enterprise-z1 -> r-transit-z1] will:

1. SSH to a test instance in enterprise zone 1.
2. Execute a curl to transit zone 1.
3. Assert the return string contains the ID of transit zone 1 to mark pass or fail.

The README.md in the companion GitHub Repository has more details and the source code.

Connect Enterprise to Transit via Direct Link and Transit Gateway

Provision a IBM Cloud® Direct Link using Transit Gateway.

IBM Cloud® Direct Link is a high speed secure data path for connecting an enterprise to the IBM Cloud. In this tutorial Transit Gateway is used for distribution. The use of Transit Gateway is optional for an on-premises connection.

The enterprise in this tutorial is simulated with another VPC. Connecting this simulated enterprise (actually another VPC) via the Transit Gateway will ensure an experience very close to what you would experience with a Direct Link.

1. Apply the enterprise_link_tf layer:

   ./apply.sh enterprise_link_tf
   

2. Run the zone 1 curl tests in the suite my using the -m (markers) flag. Choose the tests marked with curl, lz1 (left zone 1) and rz1 (right zone 1).

   Your expected results are: Connectivity within a VPC, transit <-> spoke(s), enterprise <-> transit, spoke(s) <-> spoke(s) pass but enterprise <-> spoke(s) fail.

   pytest -m "curl and lz1 and rz1"
   

Connect Enterprise to Spoke(s) via Transit NFV Firewall-Router

The incentive for a transit VPC for enterprise <-> cloud traffic is typically to route, inspect, monitor and log network traffic. In this step a firewall-router appliance will be installed in each zone of the transit VPC.

NFV Router

Provision the firewall-router appliances. An ingress route table for Transit Gateway has been added to the transit VPC as indicated by the dotted lines. A subnet has been created in each of the zones of the transit VPC to hold the firewall-router.

Connectivity from the enterprise to a spoke is achieved through a Network Function Virtualization, NFV, firewall-router instance in the transit VPC. In production you can choose one from the catalog or bring your own. This demonstration will use an Ubuntu stock image with kernel iptables set up to forward all packets from the source to destination. In this tutorial, no firewall inspection is performed.

The Terraform configuration will configure the firewall-router instance with allow_ip_spoofing. You must enable IP spoofing checks before continuing.

1. Apply the firewall_tf layer:

   ./apply.sh firewall_tf
   

2. Run the test suite.

   Your expected results are: Connectivity within a VPC, enterprise -> transit, enterprise <-> spoke same zone pass. But all transit -> spoke, all transit -> enterprise fail due to asymmetric routing issues.

   pytest -m "curl and lz1 and (rz1 or rz2)"
   

   Part two of this tutorial will route all VPC <-> different VPC traffic through the firewall-router and resolve these issues. But first it is important to learn what is happening.

Removing the firewall for Transit destination traffic

The IBM Cloud VPC uses the industry standard state-based routing for secure TCP connection tracking. It requires that the TCP connections use the same path on the way in as the way out. One exception is Direct Server Return used by routers like Network Load Balancers. It allows incoming connections from the enterprise to pass through the firewall to the transit test instance and then return directly to the originator.

This does not help with the traffic originating in the transit test instance passing through the Transit Gateway then back through ingress routing to the firewall-router. This connection gets stuck at the firewall-router (3) and will not get forwarded back to the worker as shown in red below. Traffic transit -> enterprise and transit -> spoke are failing.

One possible solution is to stop sending traffic destined to the transit VPC to the firewall-router. The wide ingress routes for the transit are currently routing traffic to the firewall-router. More specific routes can be added for the transit to Delegate to the default behavior - send directly to the intended destination instead of the firewall-router.

This diagram shows the traffic flow that is desired for this step. Only the enterprise <-> spoke is passing through the firewall:

1. enterprise <-> transit
2. spoke <-> transit
3. spoke <-> spoke
4. enterprise <--transit firewall-router--> spoke

This routing can be achieved by adding these routes to the transit ingress route table:

1. To observe the current value of the ingress route table visit the Routing tables for VPC in the IBM Cloud console. Select the transit VPC from the drop down and then select the tgw-ingress routing table.

2. Make the changes to the routing table by applying the transit_ingress layer:

./apply.sh transit_ingress_tf

1. Refresh the browser display of the routing table to observe the new routes.

2. Run the test suite.

   Your expected results are: All tests will result in PASSED.

   pytest -m "curl and lz1 and (rz1 or rz2)"
   

It is interesting to note how cross zone traffic flows between the enterprise and spokes in the configuration. Enterprise sends traffic to the correct zone and through the firewall-router via ingress routing in the transit VPC. The Transit Gateway has learned that 192.168.0.0/16 is available in all zones and will route to the transit VPC using the advertised enterprise routes in the same zone as the spoke as shown in the diagram below:

Routing Summary

Basic routing is complete:

• enterprise <-> transit
• transit <-> spoke(s)
• enterprise <--(transit firewall-router)--> spoke

Production Notes and Conclusions

The VPC reference architecture for IBM Cloud for Financial Services has much more detail on securing workloads in the IBM Cloud.

Some obvious changes to make:

• CIDR blocks were chosen for clarity and ease of explanation. The Availability Zones in the Multi zone Region could be 10.0.0.0/10, 10.64.0.0/10, 10.128.0.0/10 to conserve address space. The address space for Worker nodes could be expanded at the expense of firewall, DNS and VPE space.
• Security Groups for each of the network interfaces for worker VSIs, Virtual Private Endpoint Gateways, DNS Locations and firewalls should all be carefully considered.
• Network Access Control Lists for each subnet should be carefully considered.

Floating IPs were attached to all test instances to support connectivity tests via SSH. This is not required or desirable in production.

Implement context-based restrictions rules to further control access to all resources.

In this tutorial you created a hub VPC and a set of spoke VPCs. You identified the required Availability Zones for the architecture and created a set of subnets in the VPCs. You created a transit VPC firewall-router in each zone to forwards traffic. Test instances were used to verify connectivity and identify potential problems. Routing table routes were used to identify the traffic paths required.

Remove resources

It is not required to remove the resources if you plan to continue with the second part of this tutorial.

Execute terraform destroy in all directories in reverse order using the ./apply.sh command:

./apply.sh -d : spokes_egress_tf

Expand the tutorial

You are encouraged to continue on to part two of this tutorial where all cross VPC traffic is routed through the firewall-router, VPE for VPC and DNS are examined.

Your architecture will likely be different than the one presented but will likely be constructed from the fundamental components discussed here. Ideas to expand this tutorial:

• Integrate incoming public Internet access using IBM Cloud® Internet Services.
• Add flow log capture in the transit.
• Put each of the spokes in a separate account in an enterprise.
• Force some of the spoke to spoke traffic through the firewall and some not through the firewall.
• Replace the worker VSIs with Red Hat OpenShift on IBM Cloud and VPC load balancer.
• Force all out bound traffic through the firewall in the transit VPC and through Public gateways .

Related content

• IBM Cloud for Financial Services
• How to deploy isolated workloads across multiple locations and regions
• Public front end and private backend in a Virtual Private Cloud,
• Network Function Virtualization
• Private hub and spoke with transparent VNF and spoke-to-spoke traffic