Lab · Task 3.4 · about 60 minutes
Peering Is Not Transitive
Services in this lab
- Amazon VPCin scope
- AWS Transit Gatewayin scope
- Amazon EC2in scope
Lesson for this lab: High-Performing and Scalable Network Architectures
Goal
Build three VPCs joined by two peering connections, prove that the outer two cannot reach each other through the middle VPC, then add a transit gateway and see the same path reachable.
Before you start
- An AWS account you can experiment in, ideally a sandbox account with a budget alert. Don't use the root user; sign in with an IAM Identity Center user or an IAM role that can manage VPCs, subnets, network interfaces, transit gateways, peering connections and Reachability Analyzer.
- AWS CLI v2 configured for that identity, and one Region chosen for the whole lab (for example
us-east-1). Run every command in one terminal session so the shell variables carry over. - No instances are needed. Reachability Analyzer analyzes network interfaces, so this lab creates an elastic network interface in each VPC rather than launching EC2 instances.
Resources this lab creates (the clean-up list): three VPCs, three subnets, three security groups, three network interfaces, two VPC peering connections, one transit gateway with three VPC attachments, one Reachability Analyzer path and its analyses. Every resource except the analyses is tagged Lab=lab34.
Steps
0 of 8 steps done
The VPCs use non-overlapping CIDR blocks, which VPC peering requires. Each subnet sits in the same Availability Zone.
AZ=$(aws ec2 describe-availability-zones --query 'AvailabilityZones[0].ZoneName' --output text) for N in 1 2 3; do VPC=$(aws ec2 create-vpc --cidr-block 10.$N.0.0/16 --query Vpc.VpcId --output text) aws ec2 create-tags --resources $VPC --tags Key=Name,Value=lab34-vpc$N Key=Lab,Value=lab34 SUBNET=$(aws ec2 create-subnet --vpc-id $VPC --cidr-block 10.$N.1.0/24 --availability-zone $AZ \ --query Subnet.SubnetId --output text) aws ec2 create-tags --resources $SUBNET --tags Key=Lab,Value=lab34 eval "VPC$N=$VPC; SUBNET$N=$SUBNET" done echo $VPC1 $VPC2 $VPC3 $SUBNET1 $SUBNET2 $SUBNET3You should see three
vpc-IDs and threesubnet-IDs. (Theevalline stores each ID in a numbered shell variable,VPC1throughVPC3andSUBNET1throughSUBNET3.)Each security group allows TCP 22 only from the private 10.0.0.0/8 range. That is enough for the analysis to pass the security group check, and you never connect to anything.
for N in 1 2 3; do eval "V=\$VPC$N; S=\$SUBNET$N" SG=$(aws ec2 create-security-group --group-name lab34-sg$N --description "lab 3.4" \ --tag-specifications 'ResourceType=security-group,Tags=[{Key=Lab,Value=lab34}]' \ --vpc-id $V --query GroupId --output text) aws ec2 authorize-security-group-ingress --group-id $SG --protocol tcp --port 22 --cidr 10.0.0.0/8 ENI=$(aws ec2 create-network-interface --subnet-id $S --groups $SG \ --tag-specifications 'ResourceType=network-interface,Tags=[{Key=Lab,Value=lab34}]' \ --query NetworkInterface.NetworkInterfaceId --output text) eval "SG$N=$SG; ENI$N=$ENI" done echo $ENI1 $ENI2 $ENI3Three
eni-IDs are printed.Each peering connection is one-to-one. Create it, accept it in the same account, and check that it is active.
PCX_12=$(aws ec2 create-vpc-peering-connection --vpc-id $VPC1 --peer-vpc-id $VPC2 \ --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Lab,Value=lab34}]' \ --query VpcPeeringConnection.VpcPeeringConnectionId --output text) aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id $PCX_12 PCX_23=$(aws ec2 create-vpc-peering-connection --vpc-id $VPC2 --peer-vpc-id $VPC3 \ --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Lab,Value=lab34}]' \ --query VpcPeeringConnection.VpcPeeringConnectionId --output text) aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id $PCX_23 aws ec2 describe-vpc-peering-connections --vpc-peering-connection-ids $PCX_12 $PCX_23 \ --query 'VpcPeeringConnections[].Status.Code'The last command returns
["active", "active"].Each VPC's main route table gets a route to every CIDR it should reach directly. VPC 1 and VPC 3 are not peered, so neither gets a route to the other. VPC 2 is the middle VPC and has a route to both.
RT1=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=$VPC1 Name=association.main,Values=true \ --query 'RouteTables[0].RouteTableId' --output text) RT2=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=$VPC2 Name=association.main,Values=true \ --query 'RouteTables[0].RouteTableId' --output text) RT3=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=$VPC3 Name=association.main,Values=true \ --query 'RouteTables[0].RouteTableId' --output text) aws ec2 create-route --route-table-id $RT1 --destination-cidr-block 10.2.0.0/16 --vpc-peering-connection-id $PCX_12 aws ec2 create-route --route-table-id $RT2 --destination-cidr-block 10.1.0.0/16 --vpc-peering-connection-id $PCX_12 aws ec2 create-route --route-table-id $RT2 --destination-cidr-block 10.3.0.0/16 --vpc-peering-connection-id $PCX_23 aws ec2 create-route --route-table-id $RT3 --destination-cidr-block 10.2.0.0/16 --vpc-peering-connection-id $PCX_23Notice what is missing: there is no route in
$RT3for10.1.0.0/16. That is deliberate.EXPECTED FAILURE. Create a Reachability Analyzer path from the VPC 3 interface to the VPC 1 interface on TCP port 22, then analyze it.
cat > source-filter.json <<'EOF' { "DestinationPortRange": { "FromPort": 22, "ToPort": 22 } } EOF NIP=$(aws ec2 create-network-insights-path --source $ENI3 --destination $ENI1 --protocol TCP \ --tag-specifications 'ResourceType=network-insights-path,Tags=[{Key=Lab,Value=lab34}]' \ --filter-at-source file://source-filter.json \ --query NetworkInsightsPath.NetworkInsightsPathId --output text) NIA1=$(aws ec2 start-network-insights-analysis --network-insights-path-id $NIP \ --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) sleep 60 aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $NIA1 \ --query 'NetworkInsightsAnalyses[0].[Status,NetworkPathFound,Explanations[].ExplanationCode]' --output jsonExpect a status of
succeeded,NetworkPathFoundset tofalse, and an explanation code that points at the route table, most likelyNO_ROUTE_TO_DESTINATION(the route table has no applicable route to the destination). The route table for VPC 3's subnet has no route to10.1.0.0/16, so the packet has nowhere to go, even though VPC 2 has a route to VPC 1. This is the trap from the lesson: VPC peering is not transitive, and VPC 2 cannot carry the packet on to VPC 1. If your analysis reports a different code, open the Explanations list indescribe-network-insights-analysesand find the component that blocked the path.A transit gateway is a regional hub. Its default route table learns each attached VPC's CIDR automatically, because default association and propagation are on.
TGW=$(aws ec2 create-transit-gateway --description "lab34 hub" \ --options DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable \ --tag-specifications 'ResourceType=transit-gateway,Tags=[{Key=Lab,Value=lab34}]' \ --query TransitGateway.TransitGatewayId --output text) for N in 1 2 3; do eval "V=\$VPC$N; S=\$SUBNET$N" eval "TA$N=$(aws ec2 create-transit-gateway-vpc-attachment --transit-gateway-id $TGW --vpc-id $V \ --subnet-ids $S --tag-specifications 'ResourceType=transit-gateway-attachment,Tags=[{Key=Lab,Value=lab34}]' \ --query TransitGatewayVpcAttachment.TransitGatewayAttachmentId --output text)" done aws ec2 describe-transit-gateway-vpc-attachments --filters Name=transit-gateway-id,Values=$TGW \ --query 'TransitGatewayVpcAttachments[].State'Repeat the last command until all three states read
available. Attachments usually take a few minutes. Don't continue before then, or the routes in the next step will fail.Each VPC now sends the other two VPCs' CIDR blocks to the hub. Where a peering route already exists for a CIDR, replace it. Where none exists, create one. A CIDR can have only one route per route table, so this step replaces the peering routes.
aws ec2 replace-route --route-table-id $RT1 --destination-cidr-block 10.2.0.0/16 --transit-gateway-id $TGW aws ec2 create-route --route-table-id $RT1 --destination-cidr-block 10.3.0.0/16 --transit-gateway-id $TGW aws ec2 replace-route --route-table-id $RT2 --destination-cidr-block 10.1.0.0/16 --transit-gateway-id $TGW aws ec2 replace-route --route-table-id $RT2 --destination-cidr-block 10.3.0.0/16 --transit-gateway-id $TGW aws ec2 create-route --route-table-id $RT3 --destination-cidr-block 10.1.0.0/16 --transit-gateway-id $TGW aws ec2 replace-route --route-table-id $RT3 --destination-cidr-block 10.2.0.0/16 --transit-gateway-id $TGWNo peering route remains that matches a transit gateway route, so the hub now carries all traffic between the VPCs.
Reuse the path you created earlier, so the before and after results are comparable.
NIA2=$(aws ec2 start-network-insights-analysis --network-insights-path-id $NIP \ --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) sleep 60 aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $NIA2 \ --query 'NetworkInsightsAnalyses[0].[Status,NetworkPathFound]' --output text aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $NIA2 \ --query 'NetworkInsightsAnalyses[0].ForwardPathComponents[].Component.Id' --output textThe first line reads
succeeded True. The second command lists the components on the forward path. Your transit gateway ID should appear in that list, which shows the path passed through the hub.
Verify
describe-vpc-peering-connectionsshows both peering connections asactive. They still exist, but no route uses them after the transit gateway step.- The first analysis reports
NetworkPathFoundasFalse, and the second reportsTrue. aws ec2 describe-transit-gateway-attachments --filters Name=transit-gateway-id,Values=$TGW --query 'TransitGatewayAttachments[].State'lists threeavailableattachments.aws ec2 describe-route-tables --route-table-ids $RT3 --query 'RouteTables[0].Routes[].[DestinationCidrBlock,TransitGatewayId]' --output textlists10.1.0.0/16and10.2.0.0/16with your transit gateway ID as the target.
Clean up
Delete everything in dependency order, and check at the end that nothing tagged Lab=lab34 remains. Each step depends on the one before it.
-
Delete the Reachability Analyzer path. Deleting a path also deletes all its analyses.
aws ec2 delete-network-insights-path --network-insights-path-id $NIPConsole: VPC > Reachability Analyzer, select the path, then Delete.
-
Delete the transit gateway VPC attachments, and wait until each reports
deletedbefore deleting the transit gateway.for A in $TA1 $TA2 $TA3; do aws ec2 delete-transit-gateway-vpc-attachment --transit-gateway-attachment-id $A; done until [ -z "$(aws ec2 describe-transit-gateway-vpc-attachments --filters Name=transit-gateway-id,Values=$TGW \ --query "TransitGatewayVpcAttachments[?State!='deleted'].State" --output text)" ]; do sleep 15 doneDeleted attachments stay listed for a while with state
deleted, so the loop waits until no attachment is in any other state. Then delete the transit gateway:aws ec2 delete-transit-gateway --transit-gateway-id $TGWConsole: VPC > Transit gateway attachments, then VPC > Transit gateways.
-
Delete the two peering connections. Any route that still points at a deleted peering connection becomes a blackhole route, and it disappears when its route table is deleted in step 4.
aws ec2 delete-vpc-peering-connection --vpc-peering-connection-id $PCX_12 aws ec2 delete-vpc-peering-connection --vpc-peering-connection-id $PCX_23 -
Delete the network interfaces, then the security groups, then the subnets, then the VPCs. Each VPC's main route table goes with its VPC.
for E in $ENI1 $ENI2 $ENI3; do aws ec2 delete-network-interface --network-interface-id $E; done for G in $SG1 $SG2 $SG3; do aws ec2 delete-security-group --group-id $G; done for S in $SUBNET1 $SUBNET2 $SUBNET3; do aws ec2 delete-subnet --subnet-id $S; done for V in $VPC1 $VPC2 $VPC3; do aws ec2 delete-vpc --vpc-id $V; done -
Check that nothing billable or tagged remains.
aws ec2 describe-transit-gateways --filters Name=tag:Lab,Values=lab34 \ --query 'TransitGateways[?State!=`deleted`].TransitGatewayId' aws ec2 describe-vpcs --filters Name=tag:Lab,Values=lab34 --query 'Vpcs[].VpcId' aws ec2 describe-network-insights-paths --filters Name=tag:Lab,Values=lab34 --query 'NetworkInsightsPaths[].NetworkInsightsPathId' aws ec2 describe-security-groups --filters Name=tag:Lab,Values=lab34 --query 'SecurityGroups[].GroupId' aws ec2 describe-vpc-peering-connections --filters Name=tag:Lab,Values=lab34 --query 'VpcPeeringConnections[?Status.Code!=`deleted`].VpcPeeringConnectionId'Each command must return an empty list. The last two also confirm that the security groups and peering connections are gone. If the transit gateway still shows
deleting, wait a minute and check again. Then remove the localsource-filter.jsonfile.
Sources
- https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-basics.html
- https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-quotas.html
- https://docs.aws.amazon.com/vpc/latest/reachability/getting-started-cli.html
- https://docs.aws.amazon.com/vpc/latest/reachability/explanation-codes.html
Last reviewed