Skip to content
Certification Guide

Lab · Task 3.4 · about 60 minutes

Peering Is Not Transitive

Services in this lab

  • Amazon VPCin scope
  • AWS Transit Gatewayin scope
  • Amazon EC2in scope

Lesson for this lab: High-Performing and Scalable Network Architectures

Goal

Build three VPCs joined by two peering connections, prove that the outer two cannot reach each other through the middle VPC, then add a transit gateway and see the same path reachable.

Before you start

  • An AWS account you can experiment in, ideally a sandbox account with a budget alert. Don't use the root user; sign in with an IAM Identity Center user or an IAM role that can manage VPCs, subnets, network interfaces, transit gateways, peering connections and Reachability Analyzer.
  • AWS CLI v2 configured for that identity, and one Region chosen for the whole lab (for example us-east-1). Run every command in one terminal session so the shell variables carry over.
  • No instances are needed. Reachability Analyzer analyzes network interfaces, so this lab creates an elastic network interface in each VPC rather than launching EC2 instances.

Resources this lab creates (the clean-up list): three VPCs, three subnets, three security groups, three network interfaces, two VPC peering connections, one transit gateway with three VPC attachments, one Reachability Analyzer path and its analyses. Every resource except the analyses is tagged Lab=lab34.

Steps

0 of 8 steps done

  1. The VPCs use non-overlapping CIDR blocks, which VPC peering requires. Each subnet sits in the same Availability Zone.

    AZ=$(aws ec2 describe-availability-zones --query 'AvailabilityZones[0].ZoneName' --output text)
    for N in 1 2 3; do
      VPC=$(aws ec2 create-vpc --cidr-block 10.$N.0.0/16 --query Vpc.VpcId --output text)
      aws ec2 create-tags --resources $VPC --tags Key=Name,Value=lab34-vpc$N Key=Lab,Value=lab34
      SUBNET=$(aws ec2 create-subnet --vpc-id $VPC --cidr-block 10.$N.1.0/24 --availability-zone $AZ \
        --query Subnet.SubnetId --output text)
      aws ec2 create-tags --resources $SUBNET --tags Key=Lab,Value=lab34
      eval "VPC$N=$VPC; SUBNET$N=$SUBNET"
    done
    echo $VPC1 $VPC2 $VPC3 $SUBNET1 $SUBNET2 $SUBNET3
    

    You should see three vpc- IDs and three subnet- IDs. (The eval line stores each ID in a numbered shell variable, VPC1 through VPC3 and SUBNET1 through SUBNET3.)

  2. Each security group allows TCP 22 only from the private 10.0.0.0/8 range. That is enough for the analysis to pass the security group check, and you never connect to anything.

    for N in 1 2 3; do
      eval "V=\$VPC$N; S=\$SUBNET$N"
      SG=$(aws ec2 create-security-group --group-name lab34-sg$N --description "lab 3.4" \
        --tag-specifications 'ResourceType=security-group,Tags=[{Key=Lab,Value=lab34}]' \
        --vpc-id $V --query GroupId --output text)
      aws ec2 authorize-security-group-ingress --group-id $SG --protocol tcp --port 22 --cidr 10.0.0.0/8
      ENI=$(aws ec2 create-network-interface --subnet-id $S --groups $SG \
        --tag-specifications 'ResourceType=network-interface,Tags=[{Key=Lab,Value=lab34}]' \
        --query NetworkInterface.NetworkInterfaceId --output text)
      eval "SG$N=$SG; ENI$N=$ENI"
    done
    echo $ENI1 $ENI2 $ENI3
    

    Three eni- IDs are printed.

  3. Each peering connection is one-to-one. Create it, accept it in the same account, and check that it is active.

    PCX_12=$(aws ec2 create-vpc-peering-connection --vpc-id $VPC1 --peer-vpc-id $VPC2 \
      --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Lab,Value=lab34}]' \
      --query VpcPeeringConnection.VpcPeeringConnectionId --output text)
    aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id $PCX_12
    PCX_23=$(aws ec2 create-vpc-peering-connection --vpc-id $VPC2 --peer-vpc-id $VPC3 \
      --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Lab,Value=lab34}]' \
      --query VpcPeeringConnection.VpcPeeringConnectionId --output text)
    aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id $PCX_23
    aws ec2 describe-vpc-peering-connections --vpc-peering-connection-ids $PCX_12 $PCX_23 \
      --query 'VpcPeeringConnections[].Status.Code'
    

    The last command returns ["active", "active"].

  4. Each VPC's main route table gets a route to every CIDR it should reach directly. VPC 1 and VPC 3 are not peered, so neither gets a route to the other. VPC 2 is the middle VPC and has a route to both.

    RT1=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=$VPC1 Name=association.main,Values=true \
      --query 'RouteTables[0].RouteTableId' --output text)
    RT2=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=$VPC2 Name=association.main,Values=true \
      --query 'RouteTables[0].RouteTableId' --output text)
    RT3=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=$VPC3 Name=association.main,Values=true \
      --query 'RouteTables[0].RouteTableId' --output text)
    
    aws ec2 create-route --route-table-id $RT1 --destination-cidr-block 10.2.0.0/16 --vpc-peering-connection-id $PCX_12
    aws ec2 create-route --route-table-id $RT2 --destination-cidr-block 10.1.0.0/16 --vpc-peering-connection-id $PCX_12
    aws ec2 create-route --route-table-id $RT2 --destination-cidr-block 10.3.0.0/16 --vpc-peering-connection-id $PCX_23
    aws ec2 create-route --route-table-id $RT3 --destination-cidr-block 10.2.0.0/16 --vpc-peering-connection-id $PCX_23
    

    Notice what is missing: there is no route in $RT3 for 10.1.0.0/16. That is deliberate.

  5. EXPECTED FAILURE. Create a Reachability Analyzer path from the VPC 3 interface to the VPC 1 interface on TCP port 22, then analyze it.

    cat > source-filter.json <<'EOF'
    {
      "DestinationPortRange": { "FromPort": 22, "ToPort": 22 }
    }
    EOF
    NIP=$(aws ec2 create-network-insights-path --source $ENI3 --destination $ENI1 --protocol TCP \
      --tag-specifications 'ResourceType=network-insights-path,Tags=[{Key=Lab,Value=lab34}]' \
      --filter-at-source file://source-filter.json \
      --query NetworkInsightsPath.NetworkInsightsPathId --output text)
    NIA1=$(aws ec2 start-network-insights-analysis --network-insights-path-id $NIP \
      --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text)
    sleep 60
    aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $NIA1 \
      --query 'NetworkInsightsAnalyses[0].[Status,NetworkPathFound,Explanations[].ExplanationCode]' --output json
    

    Expect a status of succeeded, NetworkPathFound set to false, and an explanation code that points at the route table, most likely NO_ROUTE_TO_DESTINATION (the route table has no applicable route to the destination). The route table for VPC 3's subnet has no route to 10.1.0.0/16, so the packet has nowhere to go, even though VPC 2 has a route to VPC 1. This is the trap from the lesson: VPC peering is not transitive, and VPC 2 cannot carry the packet on to VPC 1. If your analysis reports a different code, open the Explanations list in describe-network-insights-analyses and find the component that blocked the path.

  6. A transit gateway is a regional hub. Its default route table learns each attached VPC's CIDR automatically, because default association and propagation are on.

    TGW=$(aws ec2 create-transit-gateway --description "lab34 hub" \
      --options DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable \
      --tag-specifications 'ResourceType=transit-gateway,Tags=[{Key=Lab,Value=lab34}]' \
      --query TransitGateway.TransitGatewayId --output text)
    for N in 1 2 3; do
      eval "V=\$VPC$N; S=\$SUBNET$N"
      eval "TA$N=$(aws ec2 create-transit-gateway-vpc-attachment --transit-gateway-id $TGW --vpc-id $V \
        --subnet-ids $S --tag-specifications 'ResourceType=transit-gateway-attachment,Tags=[{Key=Lab,Value=lab34}]' \
        --query TransitGatewayVpcAttachment.TransitGatewayAttachmentId --output text)"
    done
    aws ec2 describe-transit-gateway-vpc-attachments --filters Name=transit-gateway-id,Values=$TGW \
      --query 'TransitGatewayVpcAttachments[].State'
    

    Repeat the last command until all three states read available. Attachments usually take a few minutes. Don't continue before then, or the routes in the next step will fail.

  7. Each VPC now sends the other two VPCs' CIDR blocks to the hub. Where a peering route already exists for a CIDR, replace it. Where none exists, create one. A CIDR can have only one route per route table, so this step replaces the peering routes.

    aws ec2 replace-route --route-table-id $RT1 --destination-cidr-block 10.2.0.0/16 --transit-gateway-id $TGW
    aws ec2 create-route  --route-table-id $RT1 --destination-cidr-block 10.3.0.0/16 --transit-gateway-id $TGW
    aws ec2 replace-route --route-table-id $RT2 --destination-cidr-block 10.1.0.0/16 --transit-gateway-id $TGW
    aws ec2 replace-route --route-table-id $RT2 --destination-cidr-block 10.3.0.0/16 --transit-gateway-id $TGW
    aws ec2 create-route  --route-table-id $RT3 --destination-cidr-block 10.1.0.0/16 --transit-gateway-id $TGW
    aws ec2 replace-route --route-table-id $RT3 --destination-cidr-block 10.2.0.0/16 --transit-gateway-id $TGW
    

    No peering route remains that matches a transit gateway route, so the hub now carries all traffic between the VPCs.

  8. Reuse the path you created earlier, so the before and after results are comparable.

    NIA2=$(aws ec2 start-network-insights-analysis --network-insights-path-id $NIP \
      --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text)
    sleep 60
    aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $NIA2 \
      --query 'NetworkInsightsAnalyses[0].[Status,NetworkPathFound]' --output text
    aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $NIA2 \
      --query 'NetworkInsightsAnalyses[0].ForwardPathComponents[].Component.Id' --output text
    

    The first line reads succeeded True. The second command lists the components on the forward path. Your transit gateway ID should appear in that list, which shows the path passed through the hub.

Verify

  • describe-vpc-peering-connections shows both peering connections as active. They still exist, but no route uses them after the transit gateway step.
  • The first analysis reports NetworkPathFound as False, and the second reports True.
  • aws ec2 describe-transit-gateway-attachments --filters Name=transit-gateway-id,Values=$TGW --query 'TransitGatewayAttachments[].State' lists three available attachments.
  • aws ec2 describe-route-tables --route-table-ids $RT3 --query 'RouteTables[0].Routes[].[DestinationCidrBlock,TransitGatewayId]' --output text lists 10.1.0.0/16 and 10.2.0.0/16 with your transit gateway ID as the target.

Clean up

Delete everything in dependency order, and check at the end that nothing tagged Lab=lab34 remains. Each step depends on the one before it.

  1. Delete the Reachability Analyzer path. Deleting a path also deletes all its analyses.

    aws ec2 delete-network-insights-path --network-insights-path-id $NIP
    

    Console: VPC > Reachability Analyzer, select the path, then Delete.

  2. Delete the transit gateway VPC attachments, and wait until each reports deleted before deleting the transit gateway.

    for A in $TA1 $TA2 $TA3; do aws ec2 delete-transit-gateway-vpc-attachment --transit-gateway-attachment-id $A; done
    until [ -z "$(aws ec2 describe-transit-gateway-vpc-attachments --filters Name=transit-gateway-id,Values=$TGW \
      --query "TransitGatewayVpcAttachments[?State!='deleted'].State" --output text)" ]; do
      sleep 15
    done
    

    Deleted attachments stay listed for a while with state deleted, so the loop waits until no attachment is in any other state. Then delete the transit gateway:

    aws ec2 delete-transit-gateway --transit-gateway-id $TGW
    

    Console: VPC > Transit gateway attachments, then VPC > Transit gateways.

  3. Delete the two peering connections. Any route that still points at a deleted peering connection becomes a blackhole route, and it disappears when its route table is deleted in step 4.

    aws ec2 delete-vpc-peering-connection --vpc-peering-connection-id $PCX_12
    aws ec2 delete-vpc-peering-connection --vpc-peering-connection-id $PCX_23
    
  4. Delete the network interfaces, then the security groups, then the subnets, then the VPCs. Each VPC's main route table goes with its VPC.

    for E in $ENI1 $ENI2 $ENI3; do aws ec2 delete-network-interface --network-interface-id $E; done
    for G in $SG1 $SG2 $SG3; do aws ec2 delete-security-group --group-id $G; done
    for S in $SUBNET1 $SUBNET2 $SUBNET3; do aws ec2 delete-subnet --subnet-id $S; done
    for V in $VPC1 $VPC2 $VPC3; do aws ec2 delete-vpc --vpc-id $V; done
    
  5. Check that nothing billable or tagged remains.

    aws ec2 describe-transit-gateways --filters Name=tag:Lab,Values=lab34 \
      --query 'TransitGateways[?State!=`deleted`].TransitGatewayId'
    aws ec2 describe-vpcs --filters Name=tag:Lab,Values=lab34 --query 'Vpcs[].VpcId'
    aws ec2 describe-network-insights-paths --filters Name=tag:Lab,Values=lab34 --query 'NetworkInsightsPaths[].NetworkInsightsPathId'
    aws ec2 describe-security-groups --filters Name=tag:Lab,Values=lab34 --query 'SecurityGroups[].GroupId'
    aws ec2 describe-vpc-peering-connections --filters Name=tag:Lab,Values=lab34 --query 'VpcPeeringConnections[?Status.Code!=`deleted`].VpcPeeringConnectionId'
    

    Each command must return an empty list. The last two also confirm that the security groups and peering connections are gone. If the transit gateway still shows deleting, wait a minute and check again. Then remove the local source-filter.json file.

Sources

Last reviewed