How to trace packets in OVN

2022-06-03 4698 words 23 minutes

Contents

This article documents how to trace packets in RHOSP16 with ml2_ovn with examples for DVR and non DVR environments

Note

This document is WIP

Environment Setup Instance

In this guide I’m going to be using a Cirros instance with a two Neutron networks, one tenant and one external connected by a router. You can use the floating_ip-tenant-network-instance.yaml ansible playbook to deploy it.

Deploying Test Instance

Source the overcloudrc file, download the playbook and make any necessary edits for your network configuration:

source overcloudrc
curl https://gitlab.com/lewisdenny/ansible/-/raw/master/deploying_openstack_instances/floating_ip-tenant-network-instance.yaml -O
ansible-playbook floating_ip-tenant-network-instance.yaml

Gather Required Information

First we need to collect some information about the environment, instance and network.

Nova Instance information

Using the openstack server show command we can view the hypervisor the instance is running on, the libvirt name and the IP addresses configured:

openstack server show \
  -c OS-EXT-SRV-ATTR:host \
  -c OS-EXT-SRV-ATTR:instance_name \
  -c name \
  -c addresses \
  -f yaml \
  cirros-floating_ip

OS-EXT-SRV-ATTR:host: compute-0.redhat.local
OS-EXT-SRV-ATTR:instance_name: instance-00000002
addresses: pri_network=172.20.1.201, 10.0.0.152
name: cirros-floating_ip

Network information

With the ip address’s we can gather all the information we need about the floating IP, internal and external network

Internal Network

Network Show

openstack network show \
  -c name \
  -c id \
  -c provider:segmentation_id \
  -c router:external \
  -c subnets \
  -f yaml \
  pri_network

id: 205ed5bf-e1cd-4622-a2dc-59d5578e5fa1
name: pri_network
provider:segmentation_id: 52953
router:external: false
subnets:
- 51df2626-a3f5-4b27-bcea-397cda60a7cd

Note

We can see the network our instance is connected to is not set as external router:external: false, the packets will need to be routed out another network through a router.

Internal Port

openstack port list \
  -f yaml \
  --fixed-ip ip-address=172.20.1.201

- Fixed IP Addresses:
  - ip_address: 172.20.1.201
    subnet_id: 51df2626-a3f5-4b27-bcea-397cda60a7cd
  ID: 880e395b-d40d-40ad-b97d-52e90ab987c4
  MAC Address: fa:16:3e:33:2b:77
  Name: ''
  Status: ACTIVE

Floating IP

Floating IP details

Let’s get the port information for the attached floating IP:

openstack floating ip list \
  --long \
  -f yaml \
  -c "Fixed IP Address" \
  -c "Floating IP Address" \
  -c "Router" \
  -c "Floating Network" \
  -c "Port" \
  -c "ID" \
  --floating-ip-address 10.0.0.152

- Fixed IP Address: 172.20.1.201
  Floating IP Address: 10.0.0.152
  Floating Network: a74c89ff-4f4a-4f92-8feb-d848e9620f6a
  ID: 1f9acd22-b165-4d1c-9cfa-c8460d4f291e
  Port: 880e395b-d40d-40ad-b97d-52e90ab987c4
  Router: 43af05c8-0efc-42aa-a28f-2a89f6e26f40

Floating IP Port

openstack port list \
  -f yaml \
  --fixed-ip ip-address=10.0.0.152

- Fixed IP Addresses:
  - ip_address: 10.0.0.152
    subnet_id: 53f99136-a2e3-4da5-aee9-f5e136137663
  ID: dc4e7dac-ef83-4b57-a0f7-de6b285d6637
  MAC Address: fa:16:3e:2c:15:a4
  Name: ''
  Status: N/A

External Network

Network Show

From the floating IP list command above we can see the external network UUID is a74c89ff-4f4a-4f92-8feb-d848e9620f6a so let’s get the details

openstack network show \
  -c name \
  -c id \
  -c provider:segmentation_id \
  -c provider:physical_network \
  -c provider:network_type \
  -c router:external \
  -c subnets \
  -f yaml \
  a74c89ff-4f4a-4f92-8feb-d848e9620f6a

id: a74c89ff-4f4a-4f92-8feb-d848e9620f6a
name: ext_network
provider:network_type: flat
provider:physical_network: datacentre
provider:segmentation_id: null
router:external: true
subnets:
- 53f99136-a2e3-4da5-aee9-f5e136137663

Note

We can see this network has the external flag set to true router:external: true

Router

Router Show

Also from the openstack floating ip list command above we know the router we are looking for is 43af05c8-0efc-42aa-a28f-2a89f6e26f40. You can see below that this router is connected to both the internal and external subnet

 openstack router show \
  -c external_gateway_info \
  -c id \
  -c interfaces_info \
  -c name \
  -f yaml \
  43af05c8-0efc-42aa-a28f-2a89f6e26f40

external_gateway_info:
  enable_snat: true
  external_fixed_ips:
  - ip_address: 10.0.0.234
    subnet_id: 53f99136-a2e3-4da5-aee9-f5e136137663
  network_id: a74c89ff-4f4a-4f92-8feb-d848e9620f6a
id: 43af05c8-0efc-42aa-a28f-2a89f6e26f40
interfaces_info:
- ip_address: 172.20.1.254
  port_id: 01932525-1310-4b29-a166-017fb4128bed
  subnet_id: 51df2626-a3f5-4b27-bcea-397cda60a7cd
name: router

Libvirt Network information

Now that we have collected all the instance and Neutron networking information we can login to the compute node hosting the instance and take a look at it’s interface(s):

ssh heat-admin@overcloud-novacompute-0.ctlplane
sudo su -
podman exec -it nova_libvirt virsh domiflist instance-00000002

 Interface        Type     Source   Model    MAC
----------------------------------------------------------------
 tap880e395b-d4   bridge   br-int   virtio   fa:16:3e:33:2b:77

Note

Notice there is only one interface attached to the instance but two IP addresses assigned to the instance, this is because the FIP is handled inside OVN and the OpenFlow rules NAT the packets from the FIP to the internal IP configured on the attached interface we can see in libvirt.

Recap on Collected information

#Instance:
OS-EXT-SRV-ATTR:host: compute-0.redhat.local
OS-EXT-SRV-ATTR:instance_name: instance-00000002
addresses: pri_network=172.20.1.201, 10.0.0.152
name: cirros-floating_ip

#Internal Network:
id: 205ed5bf-e1cd-4622-a2dc-59d5578e5fa1
name: pri_network
provider:segmentation_id: 52953
router:external: false
subnets:
- 51df2626-a3f5-4b27-bcea-397cda60a7cd

#Port:
- Fixed IP Addresses:
  - ip_address: 172.20.1.201
    subnet_id: 51df2626-a3f5-4b27-bcea-397cda60a7cd
  ID: 880e395b-d40d-40ad-b97d-52e90ab987c4
  MAC Address: fa:16:3e:33:2b:77
  Name: ''
  Status: ACTIVE

#External FIP Network:
- Fixed IP Address: 172.20.1.201
  Floating IP Address: 10.0.0.152
  Floating Network: a74c89ff-4f4a-4f92-8feb-d848e9620f6a
  ID: 1f9acd22-b165-4d1c-9cfa-c8460d4f291e
  Port: 880e395b-d40d-40ad-b97d-52e90ab987c4
  Router: 43af05c8-0efc-42aa-a28f-2a89f6e26f40

#Port:
- Fixed IP Addresses:
  - ip_address: 10.0.0.152
    subnet_id: 53f99136-a2e3-4da5-aee9-f5e136137663
  ID: dc4e7dac-ef83-4b57-a0f7-de6b285d6637
  MAC Address: fa:16:3e:2c:15:a4
  Name: ''
  Status: N/A

#Router:
external_gateway_info:
  enable_snat: true
  external_fixed_ips:
  - ip_address: 10.0.0.234
    subnet_id: 53f99136-a2e3-4da5-aee9-f5e136137663
  network_id: a74c89ff-4f4a-4f92-8feb-d848e9620f6a
id: 43af05c8-0efc-42aa-a28f-2a89f6e26f40
interfaces_info:
- ip_address: 172.20.1.254
  port_id: 01932525-1310-4b29-a166-017fb4128bed
  subnet_id: 51df2626-a3f5-4b27-bcea-397cda60a7cd
name: router

#External Network:
id: a74c89ff-4f4a-4f92-8feb-d848e9620f6a
name: ext_network
provider:network_type: flat
provider:physical_network: datacentre
provider:segmentation_id: null
router:external: true

#Libvirt Instance:
instance:tap-interface: tap880e395b-d4
instance:mac-address: fa:16:3e:33:2b:77
instance:bridge: br-int

Generate Traffic to the Instance

In this guide we will be using ping from outside the director node but it could be from anywhere.

Confirm if DVR is enabled

This will determine where the floating IP will be listening

To confirm if DVR is enabled check crudini --get /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ml2_conf.ini ovn enable_distributed_floating_ip on the controller node https://docs.openstack.org/networking-ovn/train/configuration/ml2_conf.html#ovn.enable_distributed_floating_ip

With enable_distributed_floating_ip enabled the NAT action for FIP will no longer be done on the centralized gateway but be done locally on the compute node itself.

We can also confirm the external_mac field is set for the ip address in the ovn database with ovn-nbctl lr-nat-list neutron-43af05c8-0efc-42aa-a28f-2a89f6e26f40

TYPE             EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
dnat_and_snat    10.0.0.152                          172.20.1.201          fa:16:3e:2c:15:a4    880e395b-d40d-40ad-b97d-52e90ab987c4
snat             10.0.0.234                          172.20.1.0/24

This is configured in THT by including the neutron-ovn-dvr-ha.yaml template:

environments/services/neutron-ovn-dvr-ha.yaml
28:  NeutronEnableDVR: true

environments/services/neutron-ovn-ha.yaml
21:  NeutronEnableDVR: False

Even though the router is hosted on a controller with the enable-chassis-as-gw CMS option set the FIP traffic is handled locally

If you ping the router ip address you will see the packets on the external NIC of the highest priority chassis, if you ping the FIP or out of the instance it doesn’t hit the router

Trace packet to FIP in DVR environment

Identify external network

So now we have confirmed the environment is using DVR we know that pinging the FIP should be handed locally on the compute node.

We now need to check what interface the packet will come in on. We can check two ways, from the host in ovs and from Neutron.

Checking the bridge in Neutron:

openstack network agent list -f yaml --host compute-0.redhat.local

- Agent Type: OVN Controller agent
  Alive: true
  Availability Zone: ''
  Binary: ovn-controller
  Host: compute-0.redhat.local
  ID: 225690b7-ee5f-4382-b8a1-70166c05ed5f
  State: true
- Agent Type: OVN Metadata agent
  Alive: true
  Availability Zone: ''
  Binary: networking-ovn-metadata-agent
  Host: compute-0.redhat.local
  ID: a2d82774-211d-5c67-adfc-17d56e0af77c
  State: true

Check either of them for the bridge-mappings

openstack network agent show -f yaml -c configuration  a2d82774-211d-5c67-adfc-17d56e0af77c

configuration:
  bridge-mappings: datacentre:br-ex,tenant:br-isolated
  chassis_name: 225690b7-ee5f-4382-b8a1-70166c05ed5f

Checking the bridge in OVS DB on the host

ovs-vsctl get open . external_ids

{hostname=compute-0.redhat.local, ovn-bridge=br-int, ovn-bridge-mappings="datacentre:br-ex,tenant:br-isolated", ovn-encap-ip="172.17.2.66", ovn-encap-type=geneve, ovn-match-northd-version="false", ovn-monitor-all="true", ovn-openflow-probe-interval="60", ovn-remote="tcp:172.17.1.12:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="225690b7-ee5f-4382-b8a1-70166c05ed5f"}

We can see in ovn-bridge-mappings that datacentre is mapped to br-ex

Checking what NIC lives in the external bridge

We can check what ports are in br-ex in OVS:

ovs-vsctl list-ports br-ex

ens5
patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int

Verifying

ens5 is the only NIC in the bridge so lets check it with tcpdump:

tcpdump -nnei ens5 -c 1 icmp and src host 10.0.0.75

04:48:52.070550 52:54:00:be:f5:b1 > fa:16:3e:2c:15:a4, ethertype IPv4 (0x0800), length 98: 10.0.0.75 > 10.0.0.152: ICMP echo request, id 35503, seq 3671, length 64

We can see both the request and reply so we know it’s working.

OVN

OVN Theory

Now we know the packet is coming in on ens5 inside br-ex the packet needs to br-int where the instance tap interface lives, we can confirm that with ovs-vsctl port-to-br tap880e395b-d4

ovs-vsctl port-to-br tap880e395b-d4

br-int

From inside br-ex packets will go over the patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int port to br-int
Inside br-int we can see the peer of this port patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c
The packet will then be processed inside br-int, have the nat rule applied then be routed to the tap interface.
We can’t tcpdump patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int but we can see the requests on patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c using ovs-tcpdump

ovs-tcpdump -nne -i patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c

04:52:50.022690 52:54:00:be:f5:b1 > fa:16:3e:2c:15:a4, ethertype IPv4 (0x0800), length 98: 10.0.0.75 > 172.20.1.201: ICMP echo request, id 35503, seq 3904, length 64

Note

tcpdump is not able to see packets clearly on the patch because a patch port is not the same as a veth pair. It is not used to “send” and “receive” traffic. When we process an upcall and have to determine what actions to perform on a packet, a patch_port is is basically an equivalent to “continue processing OpenFlow flows on this other bridge”.

Once the packet has made it past the NAT and is inside br-int we can tcpdump the tap interface of the instance, here we can see that the src ip is still the same but the dst host has now been translated to the pri_network ip address configured inside the instance.

tcpdump -nnei tap880e395b-d4 -c 1 icmp and src host 10.0.0.75

04:59:28.806839 fa:16:3e:75:01:fc > fa:16:3e:33:2b:77, ethertype IPv4 (0x0800), length 98: 10.0.0.75 > 172.20.1.201: ICMP echo request, id 35503, seq 4294, length 64

If we take a look inside the instance we can see the pri_network ip address configured

ip -o a

1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever
2: eth0    inet 172.20.1.201/24 brd 172.20.1.255 scope global eth0\       valid_lft forever preferred_lft forever
2: eth0    inet6 fe80::f816:3eff:fe33:2b77/64 scope link \       valid_lft forever preferred_lft forever

./tcpdump -nni eth0 -c 1 icmp and src host 10.0.0.75

05:10:36.947243 IP 10.0.0.75 > 172.20.1.201: ICMP echo request, id 35503, seq 4949, length 64

OVN logical Elements

So far we know about how the packet should go through the physical interface, bridges and OVS ports but let’s now take a look at the OVN elements involved

Note

To run these commands on the host you will need to apply the host-alias’

if [ -f "/var/run/docker.pid" ]; then export containerTool=docker; else export containerTool=podman; fi

export SBDB=$(ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g')
export NBDB=$(ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g' | sed -e 's/6642/6641/g')
alias ovn-sbctl='$containerTool exec ovn_controller ovn-sbctl --db=$SBDB'
alias ovn-nbctl='$containerTool exec ovn_controller ovn-nbctl --db=$NBDB'
alias ovn-trace='$containerTool exec ovn_controller ovn-trace --db=$SBDB'
alias ovn-appctl='$containerTool exec ovn_controller ovn-appctl'
alias ovn-detrace='cat >/tmp/trace && $containerTool cp /tmp/trace ovn_controller:/tmp/trace && $containerTool exec -it ovn_controller bash -c "ovn-detrace --ovnsb=$SBDB --ovnnb=$NBDB </tmp/trace"'

Tip

if you would like to know more about OVN architecture you can check out the ovn-architecture man page!

[root@compute-0 heat-admin]# ovn-nbctl show

switch 7c69f1da-2968-4ef1-accc-2bb40859aed1 (neutron-a74c89ff-4f4a-4f92-8feb-d848e9620f6a) (aka ext_network)
    port provnet-189127e8-3244-41e2-b989-9497ae5d044c
        type: localnet
        addresses: ["unknown"]
    port 65ff31a4-4106-474c-8c65-709890507931         <--- The port to router from the external network -------+
        type: router                                                                                           |
        router-port: lrp-65ff31a4-4106-474c-8c65-709890507931                                                  |
    port 33f20266-58b3-4801-bd08-f9f5567ca201                                                                  |
        type: localport                                                                                        |
        addresses: ["fa:16:3e:28:38:30 10.0.0.150"]                                                            |
switch 016fd8ab-1bb8-4716-b426-91be1d001560 (neutron-205ed5bf-e1cd-4622-a2dc-59d5578e5fa1) (aka pri_network)   |
    port 01932525-1310-4b29-a166-017fb4128bed         <--- The port to router from the internal network ---+   |
        type: router                                                                                       |   |
        router-port: lrp-01932525-1310-4b29-a166-017fb4128bed                                              |   |
    port 5de3fb32-850f-43be-93ae-1c6448cfaff9                                                              |   |
        type: localport                                                                                    |   |
        addresses: ["fa:16:3e:8f:bb:1a 172.20.1.1"]                                                        |   |
    port 880e395b-d40d-40ad-b97d-52e90ab987c4         <--- The instance internal IP port                   |   |
        addresses: ["fa:16:3e:33:2b:77 172.20.1.201"]                                                      |   |
router 3aefc84e-a009-4e42-b8f5-c0c49f4302e0 (neutron-43af05c8-0efc-42aa-a28f-2a89f6e26f40) (aka router)    |   |
    port lrp-01932525-1310-4b29-a166-017fb4128bed     <--- The routers port to the internal network -------+   |
        mac: "fa:16:3e:75:01:fc"                                                                               |
        networks: ["172.20.1.254/24"]                                                                          |
    port lrp-65ff31a4-4106-474c-8c65-709890507931     <--- The routers port to the external network -----------+             
        mac: "fa:16:3e:2a:f7:4a"
        networks: ["10.0.0.234/24"]
        gateway chassis: [4249e953-391e-4a9e-9851-7a961caca01a e734d771-e0e1-45c1-8b06-4ea7c693778d 10b1f04f-d83d-43ce-9f2d-bd6c32fec0a3]
    nat 2bfa771b-b573-4c49-a808-bc8cbf1de2aa          <--- The instance FIP nat rule
        external ip: "10.0.0.152"
        logical ip: "172.20.1.201"
        type: "dnat_and_snat"
    nat fcc5acdf-e888-4bb0-a794-de181eced554
        external ip: "10.0.0.234"
        logical ip: "172.20.1.0/24"
        type: "snat"

Using `ovn-trace`

Using the ovn-trace command we can simulate a packet going through these logical network devices.

Let’s create an ovn-trace command to see what the OVN logical flows want to do with our ICMP packet, for this we will need some of the information we have captured so far:

switch: OVN switch for our external network 
inport: OVN port of our provnet in br-ex
ip.ttl: Time to live for the packet
icmp4.type: Type of ICMP packet
  
Following information from tcpdump instance the instance or on the tap device:
eth.src: Source MAC 
eth.dst: Destination MAC
ip4.src: Source IP
ip4.dst: Destination IP

Now that we have gathered the required information we can execute the ovn-trace command:

ovn-trace --ct=new --summary neutron-a74c89ff-4f4a-4f92-8feb-d848e9620f6a 'inport == "provnet-189127e8-3244-41e2-b989-9497ae5d044c" && eth.src == 52:54:00:be:f5:b1 && eth.dst == fa:16:3e:2c:15:a4 && ip4.src == 10.0.0.75 && ip4.dst == 172.20.1.20 && ip.ttl == 32 && icmp4.type == 8'

Summary:

# icmp,reg14=0x1,vlan_tci=0x0000,dl_src=52:54:00:be:f5:b1,dl_dst=fa:16:3e:2c:15:a4,nw_src=10.0.0.75,nw_dst=172.20.1.20,nw_tos=0,nw_ecn=0,nw_ttl=32,icmp_type=8,icmp_code=0                  
ingress(dp="ext_network", inport="provnet-189127") {
    next;
    next;
    next;
    outport = "65ff31";
    output;
    egress(dp="ext_network", inport="provnet-189127", outport="65ff31") {
        next;
        output;
        /* output to "65ff31", type "patch" */;
        ingress(dp="router", inport="lrp-65ff31") {
            xreg0[0..47] = fa:16:3e:2a:f7:4a;
            next;
            reg9[2] = 1;
            next;
            next;
            reg7 = 0;
            next;
            ip.ttl--;
            reg8[0..15] = 0;
            reg0 = ip4.dst;
            reg1 = 172.20.1.254;
            eth.src = fa:16:3e:75:01:fc;
            outport = "lrp-019325";
            flags.loopback = 1;
            next;
            next;
            reg8[0..15] = 0;
            next;
            next;
            get_arp(outport, reg0);
            /* No MAC binding. */
            next;
            arp { eth.dst = ff:ff:ff:ff:ff:ff; arp.spa = reg1; arp.tpa = reg0; arp.op = 1; output; };                                                                                       
            arp {
                eth.dst = ff:ff:ff:ff:ff:ff;
                arp.spa = reg1;
                arp.tpa = reg0;
                arp.op = 1;
                output;
                egress(dp="router", inport="lrp-65ff31", outport="lrp-019325") {                                                                                                             
                    reg9[4] = 0;                                                                                                                                                            
                    next;               
                    output;                                                                                                                                                                 
                    /* output to "lrp-019325", type "patch" */;
                    ingress(dp="pri_network", inport="019325") {
                        next;
                        next;
                        next;
                        reg0[8] = 1;
                        reg0[9] = 1;                                     
                        next;
                        outport = "_MC_flood_l2";
                        output;                
                        multicast(dp="pri_network", mcgroup="_MC_flood_l2") {
                            egress(dp="pri_network", inport="019325", outport="5de3fb") {
                                next;
                                next;
                                reg0[8] = 1;
                                reg0[9] = 1;
                                next;
                                output;
                                /* output to "5de3fb", type "localport" */;
                            };
                            egress(dp="pri_network", inport="019325", outport="880e39") {
                                next;
                                next;   
                                reg0[8] = 1;
                                reg0[9] = 1;
                                next;
                                output;
                                /* output to "880e39", type "" */;
                            };
                        };
                    };             
                };               
            };   
        };                                                                                                                                                                                  
    };           
};

Detailed:

# icmp,reg14=0x1,vlan_tci=0x0000,dl_src=52:54:00:be:f5:b1,dl_dst=fa:16:3e:2c:15:a4,nw_src=10.0.0.75,nw_dst=172.20.1.20,nw_tos=0,nw_ecn=0,nw_ttl=32,icmp_type=8,icmp_code=0                  

ingress(dp="ext_network", inport="provnet-189127")
--------------------------------------------------
 0. ls_in_port_sec_l2 (northd.c:5501): inport == "provnet-189127", priority 50, uuid 79aa1bf4
    cookie=0x79aa1bf4, duration=140902.997s, table=8, n_packets=119565, n_bytes=11970319, idle_age=0, priority=50,reg14=0x1,metadata=0x1 actions=resubmit(,9)                               
    next;
 6. ls_in_pre_lb (northd.c:5648): ip && inport == "provnet-189127", priority 110, uuid bc24152f                                                                                             
    cookie=0xbc24152f, duration=140902.997s, table=14, n_packets=1199, n_bytes=146846, idle_age=130, priority=110,ipv6,reg14=0x1,metadata=0x1 actions=resubmit(,15)                         
    cookie=0xbc24152f, duration=140902.997s, table=14, n_packets=114794, n_bytes=11673449, idle_age=0, priority=110,ip,reg14=0x1,metadata=0x1 actions=resubmit(,15)                         
    next;
18. ls_in_arp_rsp (northd.c:7663): inport == "provnet-189127", priority 100, uuid b7f7bdd2
    cookie=0xb7f7bdd2, duration=140902.997s, table=26, n_packets=119565, n_bytes=11970319, idle_age=0, priority=100,reg14=0x1,metadata=0x1 actions=resubmit(,27)                            
    next;
24. ls_in_l2_lkup (northd.c:8404): eth.dst == fa:16:3e:2c:15:a4 && is_chassis_resident("880e39"), priority 50, uuid 75015d29                                                                
    cookie=0x75015d29, duration=140902.997s, table=32, n_packets=114789, n_bytes=11672935, idle_age=0, priority=50,metadata=0x1,dl_dst=fa:16:3e:2c:15:a4 actions=set_field:0x3->reg15,resubmit(,37)
    outport = "65ff31";
    output;

egress(dp="ext_network", inport="provnet-189127", outport="65ff31")
-------------------------------------------------------------------
 0. ls_out_pre_lb (northd.c:5651): ip && outport == "65ff31", priority 110, uuid d6c6d4ce
    cookie=0xd6c6d4ce, duration=140902.997s, table=40, n_packets=1199, n_bytes=146846, idle_age=130, priority=110,ipv6,reg15=0x3,metadata=0x1 actions=resubmit(,41)                         
    cookie=0xd6c6d4ce, duration=140902.997s, table=40, n_packets=114786, n_bytes=11672809, idle_age=0, priority=110,ip,reg15=0x3,metadata=0x1 actions=resubmit(,41)                         
    next;
 9. ls_out_port_sec_l2 (northd.c:5599): outport == "65ff31", priority 50, uuid 08cb3f9d
    cookie=0x8cb3f9d, duration=140902.997s, table=49, n_packets=118356, n_bytes=11822749, idle_age=0, priority=50,reg15=0x3,metadata=0x1 actions=resubmit(,64)                              
    output;
    /* output to "65ff31", type "patch" */

ingress(dp="router", inport="lrp-65ff31")
-----------------------------------------
 0. lr_in_admission (northd.c:10609): eth.dst == fa:16:3e:2c:15:a4 && inport == "lrp-65ff31" && is_chassis_resident("880e39"), priority 50, uuid 6d9f6bda                                   
    cookie=0x6d9f6bda, duration=140902.998s, table=8, n_packets=120412, n_bytes=12024237, idle_age=0, priority=50,reg14=0x1,metadata=0x3,dl_dst=fa:16:3e:2c:15:a4 actions=set_field:0xfa163e2af74a0000000000000000/0xffffffffffff0000000000000000->xxreg0,resubmit(,9)
    xreg0[0..47] = fa:16:3e:2a:f7:4a;
    next;
 1. lr_in_lookup_neighbor (northd.c:10752): 1, priority 0, uuid be266fbc
    cookie=0xbe266fbc, duration=140912.106s, table=9, n_packets=275414, n_bytes=42164894, idle_age=0, priority=0,metadata=0x3 actions=set_field:0x4/0x4->xreg4,resubmit(,10)                
    reg9[2] = 1;
    next;
 2. lr_in_learn_neighbor (northd.c:10761): reg9[2] == 1 || reg9[3] == 0, priority 100, uuid 09a713c1                                                                                        
    cookie=0x9a713c1, duration=140912.111s, table=10, n_packets=275415, n_bytes=42164980, idle_age=0, priority=100,reg9=0/0x8,metadata=0x3 actions=resubmit(,11)                            
    cookie=0x9a713c1, duration=140912.111s, table=10, n_packets=2, n_bytes=84, idle_age=65535, priority=100,reg9=0x4/0x4,metadata=0x3 actions=resubmit(,11)                                 
    next;
10. lr_in_ip_routing_pre (northd.c:10984): 1, priority 0, uuid 9476be26
    cookie=0x9476be26, duration=140912.108s, table=18, n_packets=270957, n_bytes=41909216, idle_age=0, priority=0,metadata=0x3 actions=set_field:0/0xffffffff->xxreg1,resubmit(,19)
    reg7 = 0;
    next;
11. lr_in_ip_routing (northd.c:9525): ip4.dst == 172.20.1.0/24, priority 74, uuid 80987fbc
    cookie=0x80987fbc, duration=140912.108s, table=19, n_packets=116850, n_bytes=11875081, idle_age=0, priority=74,ip,metadata=0x3,nw_dst=172.20.1.0/24 actions=dec_ttl(),set_field:0/0xffff00000000->xreg4,move:NXM_OF_IP_DST[]->NXM_NX_XXREG0[96..127],set_field:0xac1401fe0000000000000000/0xffffffff0000000000000000->xxreg0,set_field:fa:16:3e:75:01:fc->eth_src,set_field:0x3->reg15,set_field:0x1/0x1->reg10,resubmit(,20)
    ip.ttl--;
    reg8[0..15] = 0;
    reg0 = ip4.dst;
    reg1 = 172.20.1.254;
    eth.src = fa:16:3e:75:01:fc;
    outport = "lrp-019325";
    flags.loopback = 1;
    next;
12. lr_in_ip_routing_ecmp (northd.c:11059): reg8[0..15] == 0, priority 150, uuid 8845f34f
    cookie=0x8845f34f, duration=140912.108s, table=20, n_packets=270647, n_bytes=41868236, idle_age=0, priority=150,reg8=0/0xffff,metadata=0x3 actions=resubmit(,21)
    next;
13. lr_in_policy (northd.c:11192): 1, priority 0, uuid 435046ed
    cookie=0x435046ed, duration=140912.111s, table=21, n_packets=270647, n_bytes=41868236, idle_age=0, priority=0,metadata=0x3 actions=set_field:0/0xffff00000000->xreg4,resubmit(,22)
    reg8[0..15] = 0;
    next;
14. lr_in_policy_ecmp (northd.c:11194): reg8[0..15] == 0, priority 150, uuid 5bb8b7ab
    cookie=0x5bb8b7ab, duration=140912.108s, table=22, n_packets=270647, n_bytes=41868236, idle_age=0, priority=150,reg8=0/0xffff,metadata=0x3 actions=resubmit(,23)
    next;
15. lr_in_arp_resolve (northd.c:11228): ip4, priority 0, uuid f8dcc0a9
    cookie=0xf8dcc0a9, duration=140912.106s, table=23, n_packets=151741, n_bytes=29791667, idle_age=0, priority=0,ip,metadata=0x3 actions=push:NXM_NX_REG0[],push:NXM_NX_XXREG0[96..127],pop:NXM_NX_REG0[],set_field:00:00:00:00:00:00->eth_dst,resubmit(,66),pop:NXM_NX_REG0[],resubmit(,24)
    get_arp(outport, reg0);
    /* No MAC binding. */
    next;
19. lr_in_arp_request (northd.c:11865): eth.dst == 00:00:00:00:00:00 && ip4, priority 100, uuid 9b0c9364
    cookie=0x9b0c9364, duration=140912.108s, table=27, n_packets=2, n_bytes=159, idle_age=65535, priority=100,ip,metadata=0x3,dl_dst=00:00:00:00:00:00 actions=controller(userdata=00.00.00.00.00.00.00.00.00.19.00.10.80.00.06.06.ff.ff.ff.ff.ff.ff.00.00.00.1c.00.18.00.20.00.40.00.00.00.00.00.01.de.10.80.00.2c.04.00.00.00.00.00.1c.00.18.00.20.00.60.00.00.00.00.00.01.de.10.80.00.2e.04.00.00.00.00.00.19.00.10.80.00.2a.02.00.01.00.00.00.00.00.00.ff.ff.00.10.00.00.23.20.00.0e.ff.f8.25.00.00.00)
    arp { eth.dst = ff:ff:ff:ff:ff:ff; arp.spa = reg1; arp.tpa = reg0; arp.op = 1; output; };

arp
---
    eth.dst = ff:ff:ff:ff:ff:ff;
    arp.spa = reg1;
    arp.tpa = reg0;
    arp.op = 1;
    output;

egress(dp="router", inport="lrp-65ff31", outport="lrp-019325")
--------------------------------------------------------------
 0. lr_out_chk_dnat_local (northd.c:13100): 1, priority 0, uuid 1f08f333
    cookie=0x1f08f333, duration=140912.111s, table=40, n_packets=272163, n_bytes=41817187, idle_age=0, priority=0,metadata=0x3 actions=set_field:0/0x10->xreg4,resubmit(,41)
    reg9[4] = 0;
    next;
 6. lr_out_delivery (northd.c:11922): outport == "lrp-019325", priority 100, uuid 44be9e99
    cookie=0x44be9e99, duration=140912.111s, table=46, n_packets=116850, n_bytes=11875081, idle_age=0, priority=100,reg15=0x3,metadata=0x3 actions=resubmit(,64)
    output;
    /* output to "lrp-019325", type "patch" */

ingress(dp="pri_network", inport="019325")
------------------------------------------
 0. ls_in_port_sec_l2 (northd.c:5501): inport == "019325", priority 50, uuid 267bdf53
    cookie=0x267bdf53, duration=140912.119s, table=8, n_packets=116850, n_bytes=11875081, idle_age=0, priority=50,reg14=0x2,metadata=0x2 actions=resubmit(,9)
    next;
 5. ls_in_pre_acl (northd.c:5748): eth.mcast, priority 110, uuid 628f2ef2
    cookie=0x628f2ef2, duration=140912.117s, table=13, n_packets=5, n_bytes=812, idle_age=65535, priority=110,metadata=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,14)
    next;
 6. ls_in_pre_lb (northd.c:5829): eth.mcast, priority 110, uuid c4b7c973
    cookie=0xc4b7c973, duration=140912.111s, table=14, n_packets=6, n_bytes=902, idle_age=65535, priority=110,metadata=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,15)
    cookie=0xc4b7c973, duration=140902.998s, table=14, n_packets=10, n_bytes=420, idle_age=65535, priority=110,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,15)
    next;
 8. ls_in_acl_hint (northd.c:6013): !ct.trk, priority 5, uuid 963f4597
    cookie=0x963f4597, duration=140912.116s, table=16, n_packets=119567, n_bytes=11989845, idle_age=0, priority=5,ct_state=-trk,metadata=0x2 actions=set_field:0x100000000000000000000000000/0x100000000000000000000000000->xxreg0,set_field:0x200000000000000000000000000/0x200000000000000000000000000->xxreg0,resubmit(,17)
    reg0[8] = 1;
    reg0[9] = 1;
    next;
24. ls_in_l2_lkup (northd.c:7151): eth.src == {fa:16:3e:75:01:fc, fa:16:3e:2c:15:a4} && (arp.op == 1 || nd_ns), priority 75, uuid 9cccfac4
    cookie=0x9cccfac4, duration=140902.998s, table=32, n_packets=0, n_bytes=0, idle_age=65535, priority=75,icmp6,metadata=0x2,dl_src=fa:16:3e:75:01:fc,nw_ttl=255,icmp_type=135,icmp_code=0 actions=set_field:0x8005->reg15,resubmit(,37)
    cookie=0x9cccfac4, duration=140902.998s, table=32, n_packets=0, n_bytes=0, idle_age=65535, priority=75,arp,metadata=0x2,dl_src=fa:16:3e:75:01:fc,arp_op=1 actions=set_field:0x8005->reg15,resubmit(,37)
    cookie=0x9cccfac4, duration=140902.998s, table=32, n_packets=0, n_bytes=0, idle_age=65535, priority=75,arp,metadata=0x2,dl_src=fa:16:3e:2c:15:a4,arp_op=1 actions=set_field:0x8005->reg15,resubmit(,37)
    cookie=0x9cccfac4, duration=140902.998s, table=32, n_packets=0, n_bytes=0, idle_age=65535, priority=75,icmp6,metadata=0x2,dl_src=fa:16:3e:2c:15:a4,nw_ttl=255,icmp_type=135,icmp_code=0 actions=set_field:0x8005->reg15,resubmit(,37)
    outport = "_MC_flood_l2";
    output;

multicast(dp="pri_network", mcgroup="_MC_flood_l2")
---------------------------------------------------

    egress(dp="pri_network", inport="019325", outport="5de3fb")
    -----------------------------------------------------------
         0. ls_out_pre_lb (northd.c:5830): eth.mcast, priority 110, uuid 6b7a6265
            cookie=0x6b7a6265, duration=140912.112s, table=40, n_packets=0, n_bytes=0, idle_age=65535, priority=110,metadata=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,41)
            cookie=0x6b7a6265, duration=140902.999s, table=40, n_packets=110, n_bytes=18840, idle_age=1436, priority=110,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,41)
            next;
         1. ls_out_pre_acl (northd.c:5750): eth.mcast, priority 110, uuid 62987828
            cookie=0x62987828, duration=140912.118s, table=41, n_packets=1, n_bytes=90, idle_age=65535, priority=110,metadata=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,42)
            next;
         3. ls_out_acl_hint (northd.c:6013): !ct.trk, priority 5, uuid b5b858c7
            cookie=0xb5b858c7, duration=140912.117s, table=43, n_packets=2739, n_bytes=116228, idle_age=31, priority=5,ct_state=-trk,metadata=0x2 actions=set_field:0x100000000000000000000000000/0x100000000000000000000000000->xxreg0,set_field:0x200000000000000000000000000/0x200000000000000000000000000->xxreg0,resubmit(,44)
            reg0[8] = 1;
            reg0[9] = 1;
            next;
         9. ls_out_port_sec_l2 (northd.c:5623): eth.mcast, priority 100, uuid 2f470ec7
            cookie=0x2f470ec7, duration=140912.112s, table=49, n_packets=1, n_bytes=90, idle_age=65535, priority=100,metadata=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,64)
            cookie=0x2f470ec7, duration=140902.999s, table=49, n_packets=2418, n_bytes=294532, idle_age=130, priority=100,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,64)
            output;
            /* output to "5de3fb", type "localport" */

    egress(dp="pri_network", inport="019325", outport="880e39")
    -----------------------------------------------------------
         0. ls_out_pre_lb (northd.c:5830): eth.mcast, priority 110, uuid 6b7a6265
            cookie=0x6b7a6265, duration=140912.112s, table=40, n_packets=0, n_bytes=0, idle_age=65535, priority=110,metadata=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,41)
            cookie=0x6b7a6265, duration=140902.999s, table=40, n_packets=110, n_bytes=18840, idle_age=1436, priority=110,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,41)
            next;
         1. ls_out_pre_acl (northd.c:5750): eth.mcast, priority 110, uuid 62987828
            cookie=0x62987828, duration=140912.118s, table=41, n_packets=1, n_bytes=90, idle_age=65535, priority=110,metadata=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,42)
            next;
         3. ls_out_acl_hint (northd.c:6013): !ct.trk, priority 5, uuid b5b858c7
            cookie=0xb5b858c7, duration=140912.117s, table=43, n_packets=2739, n_bytes=116228, idle_age=31, priority=5,ct_state=-trk,metadata=0x2 actions=set_field:0x100000000000000000000000000/0x100000000000000000000000000->xxreg0,set_field:0x200000000000000000000000000/0x200000000000000000000000000->xxreg0,resubmit(,44)
            reg0[8] = 1;
            reg0[9] = 1;
            next;
         9. ls_out_port_sec_l2 (northd.c:5623): eth.mcast, priority 100, uuid 2f470ec7
            cookie=0x2f470ec7, duration=140912.112s, table=49, n_packets=1, n_bytes=90, idle_age=65535, priority=100,metadata=0x2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,64)
            cookie=0x2f470ec7, duration=140902.999s, table=49, n_packets=2418, n_bytes=294532, idle_age=130, priority=100,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,64)
            output;
            /* output to "880e39", type "" */

OVS

OVS architecture

Now let’s take a look at OVS architecture with ovs-vsctl show

ovs-vsctl show

Full output:

69084434-b528-4a8e-81e2-a6fde56dea33
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-ex
        fail_mode: standalone
        Port br-ex
            Interface br-ex
                type: internal
        Port patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int
            Interface patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c}
        Port ens5
            Interface ens5
    Bridge br-isolated
        fail_mode: standalone
        Port br-isolated
            Interface br-isolated
                type: internal
        Port ens4
            Interface ens4
        Port vlan50
            tag: 50
            Interface vlan50
                type: internal
        Port vlan20
            tag: 20
            Interface vlan20
                type: internal
        Port vlan30
            tag: 30
            Interface vlan30
                type: internal
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port tap205ed5bf-e0
            Interface tap205ed5bf-e0
        Port ovn-e734d7-0
            Interface ovn-e734d7-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.17.2.140"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port br-int
            Interface br-int
                type: internal
        Port patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c
            Interface patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c
                type: patch
                options: {peer=patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int}
        Port ovn-c62de1-0
            Interface ovn-c62de1-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.17.2.48"}
        Port ovn-10b1f0-0
            Interface ovn-10b1f0-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.17.2.45"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port tap880e395b-d4
            Interface tap880e395b-d4
        Port ovn-4249e9-0
            Interface ovn-4249e9-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.17.2.136"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
    ovs_version: "2.15.4"

Shortened output:

69084434-b528-4a8e-81e2-a6fde56dea33
    Bridge br-isolated
    [...]
    Bridge br-ex
        fail_mode: standalone
        Port ens5                      <--- Packet enters here 
            Interface ens5
        Port br-ex
            Interface br-ex
                type: internal
        Port patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int        -------------------+
            Interface patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int                  |
                type: patch                                                                         |
                options: {peer=patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c}        |
    Bridge br-int                                                                                   |
        fail_mode: secure                                                                           |
        datapath_type: system                                                                       |
        Port patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c       <-------------------+
            Interface patch-br-int-to-provnet-189127e8-3244-41e2-b989-9497ae5d044c
                type: patch
                options: {peer=patch-provnet-189127e8-3244-41e2-b989-9497ae5d044c-to-br-int}
        Port tap205ed5bf-e0
            Interface tap205ed5bf-e0
        Port tap880e395b-d4
            Interface tap880e395b-d4   <---- Packet ends here
        Port br-int
            Interface br-int
                type: internal
        Port ovn-e734d7-0
        [...]
        Port ovn-c62de1-0
        [...]
        Port ovn-10b1f0-0
        [...]
        Port ovn-4249e9-0
        [...]
    ovs_version: "2.15.4"

Using ovs-appctl ofproto/trace

Now that we have had a good look at the ovn-trace and OVS overview, let’s look at ovs-appctl ofproto/trace.

Note

Remember, OVN is programing OVS so the OVN logical flows could be fine but there could be something missing in OVS OpenFlow

First we need the Open vSwitch test package which provides the ovs-tcpundump command:

$ dnf install openvswitch2.15-test

Tip

openvswitch2.15-test is in the fast-datapath-for-rhel-8-x86_64-rpms repo

With the Open vSwitch test package installed we now have access to the ovs-tcpundump command. When given the hex output of a packet from tcpdump it will convert the output into a single hexadecimal string, this can then be used with ovs-appctl ofproto/trace

flow=$(tcpdump -nXXi tap880e395b-d4 -c 1 icmp and src host 10.0.0.75 | ovs-tcpundump)

echo $flow
c8d3ffa6820ffa163efdd3b20800450000542df64000400152f4ac150ba801010101080054ec3602d4fcb651e1c200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

With the hexedecimal packet captured we can pass it into ovs-appctl ofproto/trace

$ ovs-appctl ofproto/trace br-int in_port=`ovs-vsctl get Interface tapecd8fb74-f4 ofport` $flow

Flow: icmp,in_port=1,vlan_tci=0x0000,dl_src=52:54:00:be:f5:b1,dl_dst=fa:16:3e:2c:15:a4,nw_src=10.0.0.75,nw_dst=10.0.0.152,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0

bridge("br-ex")
---------------
 1. priority 0
    NORMAL
     -> forwarding to learned port

bridge("br-int")
----------------
 0. in_port=7,vlan_tci=0x0000/0x1000, priority 100, cookie 0x93fb036d
    set_field:0x8->reg11
    set_field:0x7->reg12
    set_field:0x1->metadata
    set_field:0x1->reg14
    resubmit(,8)
 8. reg14=0x1,metadata=0x1, priority 50, cookie 0x79aa1bf4
    resubmit(,9)
 9. metadata=0x1, priority 0, cookie 0xc4339339
    resubmit(,10)
10. metadata=0x1, priority 0, cookie 0x13b355fd
    resubmit(,11)
11. metadata=0x1, priority 0, cookie 0xba29016d
    resubmit(,12)
12. metadata=0x1, priority 0, cookie 0x42ecbd8f
    resubmit(,13)
13. metadata=0x1, priority 0, cookie 0x4e0f6a59
    resubmit(,14)
14. ip,reg14=0x1,metadata=0x1, priority 110, cookie 0xbc24152f
    resubmit(,15)
15. metadata=0x1, priority 0, cookie 0xc119fe5f
    resubmit(,16)
16. metadata=0x1, priority 65535, cookie 0x62d724ec
    resubmit(,17)
17. metadata=0x1, priority 65535, cookie 0x37d41ac7
    resubmit(,18)
18. metadata=0x1, priority 0, cookie 0xc69bbb1e
    resubmit(,19)
19. metadata=0x1, priority 0, cookie 0xbfbbcdc5
    resubmit(,20)
20. metadata=0x1, priority 0, cookie 0xb56cc401
    resubmit(,21)
21. metadata=0x1, priority 0, cookie 0xdac8fa19
    resubmit(,22)
22. metadata=0x1, priority 0, cookie 0xa69e779a
    resubmit(,23)
23. metadata=0x1, priority 0, cookie 0x53a1d404
    resubmit(,24)
24. metadata=0x1, priority 0, cookie 0xe35dea33
    resubmit(,25)
25. metadata=0x1, priority 0, cookie 0x8c286a3a
    resubmit(,26)
26. reg14=0x1,metadata=0x1, priority 100, cookie 0xb7f7bdd2
    resubmit(,27)
27. metadata=0x1, priority 0, cookie 0xac3d35b5
    resubmit(,28)
28. metadata=0x1, priority 0, cookie 0xe6eaf52d
    resubmit(,29)
29. metadata=0x1, priority 0, cookie 0x46587fef
    resubmit(,30)
30. metadata=0x1, priority 0, cookie 0x713c29df
    resubmit(,31)
31. metadata=0x1, priority 0, cookie 0xdd401e7c
    resubmit(,32)
32. metadata=0x1,dl_dst=fa:16:3e:2c:15:a4, priority 50, cookie 0x75015d29
    set_field:0x3->reg15
    resubmit(,37)
37. priority 0
    resubmit(,38)
38. reg15=0x3,metadata=0x1, priority 100, cookie 0x1aa420e7
    set_field:0x8->reg11
    set_field:0x7->reg12
    resubmit(,39)
39. priority 0
    set_field:0->reg0
    set_field:0->reg1
    set_field:0->reg2
    set_field:0->reg3
    set_field:0->reg4
    set_field:0->reg5
    set_field:0->reg6
    set_field:0->reg7
    set_field:0->reg8
    set_field:0->reg9
    resubmit(,40)
40. ip,reg15=0x3,metadata=0x1, priority 110, cookie 0xd6c6d4ce
    resubmit(,41)
41. metadata=0x1, priority 0, cookie 0x7343ece1
    resubmit(,42)
42. metadata=0x1, priority 0, cookie 0xa654b191
    resubmit(,43)
43. metadata=0x1, priority 65535, cookie 0x672d4804
    resubmit(,44)
44. metadata=0x1, priority 65535, cookie 0x638d95bd
    resubmit(,45)
45. metadata=0x1, priority 0, cookie 0xc2c7f022
    resubmit(,46)
46. metadata=0x1, priority 0, cookie 0xc61bdac7
    resubmit(,47)
47. metadata=0x1, priority 0, cookie 0xd2a97c09
    resubmit(,48)
48. metadata=0x1, priority 0, cookie 0x809bca24
    resubmit(,49)
49. reg15=0x3,metadata=0x1, priority 50, cookie 0x8cb3f9d
    resubmit(,64)
64. priority 0
    resubmit(,65)
65. reg15=0x3,metadata=0x1, priority 100, cookie 0x1aa420e7
    clone(ct_clear,set_field:0->reg11,set_field:0->reg12,set_field:0->reg13,set_field:0x2->reg11,set_field:0x1->reg12,set_field:0x3->metadata,set_field:0x1->reg14,set_field:0->reg10,set_field:0->reg15,set_field:0->reg0,set_field:0->reg1,set_field:0->reg2,set_field:0->reg3,set_field:0->reg4,set_field:0->reg5,set_field:0->reg6,set_field:0->reg7,set_field:0->reg8,set_field:0->reg9,resubmit(,8))
    ct_clear
    set_field:0->reg11
    set_field:0->reg12
    set_field:0->reg13
    set_field:0x2->reg11
    set_field:0x1->reg12
    set_field:0x3->metadata
    set_field:0x1->reg14
    set_field:0->reg10
    set_field:0->reg15
    set_field:0->reg0
    set_field:0->reg1
    set_field:0->reg2
    set_field:0->reg3
    set_field:0->reg4
    set_field:0->reg5
    set_field:0->reg6
    set_field:0->reg7
    set_field:0->reg8
    set_field:0->reg9
    resubmit(,8)
 8. reg14=0x1,metadata=0x3,dl_dst=fa:16:3e:2c:15:a4, priority 50, cookie 0x6d9f6bda
    set_field:0xfa163e2af74a0000000000000000/0xffffffffffff0000000000000000->xxreg0
    resubmit(,9)
 9. metadata=0x3, priority 0, cookie 0xbe266fbc
    set_field:0x4/0x4->xreg4
    resubmit(,10)
10. reg9=0/0x8,metadata=0x3, priority 100, cookie 0x9a713c1
    resubmit(,11)
11. metadata=0x3, priority 0, cookie 0x5a82ae42
    resubmit(,12)
12. ip,reg10=0/0x1,reg14=0x1,metadata=0x3,nw_dst=10.0.0.152, priority 100, cookie 0x379c99db
    ct(table=13,zone=NXM_NX_REG11[0..15],nat)
    nat
     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 13.
     -> Sets the packet to an untracked state, and clears all the conntrack fields.

Final flow: unchanged
Megaflow: recirc_id=0,eth,icmp,in_port=1,dl_src=52:54:00:be:f5:b1,dl_dst=fa:16:3e:2c:15:a4,nw_src=10.0.0.0/25,nw_dst=10.0.0.152,nw_ttl=64,nw_frag=no
Datapath actions: ct(zone=2,nat),recirc(0x29)

===============================================================================
recirc(0x29) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
Replacing src/dst IP/ports to simulate NAT:
 Initial flow: 
 Modified flow: 
===============================================================================

Flow: recirc_id=0x29,ct_state=new|trk,ct_zone=2,eth,icmp,reg0=0xfa16,reg1=0x3e2af74a,reg9=0x4,reg11=0x2,reg12=0x1,reg14=0x1,metadata=0x3,in_port=7,vlan_tci=0x0000,dl_src=52:54:00:be:f5:b1,dl_dst=fa:16:3e:2c:15:a4,nw_src=10.0.0.75,nw_dst=10.0.0.152,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0

bridge("br-ex")
---------------
    thaw
        Resuming from table 13
13. metadata=0x3, priority 0, cookie 0x9357b0ca
    resubmit(,14)
14. ip,reg14=0x1,metadata=0x3,nw_dst=10.0.0.152, priority 100, cookie 0xdf0c7ebe
    ct(commit,table=15,zone=NXM_NX_REG11[0..15],nat(dst=172.20.1.201))
    nat(dst=172.20.1.201)
     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 15.
     -> Sets the packet to an untracked state, and clears all the conntrack fields.
Final flow: recirc_id=0x29,eth,icmp,reg0=0xfa16,reg1=0x3e2af74a,reg9=0x4,reg11=0x2,reg12=0x1,reg14=0x1,metadata=0x3,in_port=7,vlan_tci=0x0000,dl_src=52:54:00:be:f5:b1,dl_dst=fa:16:3e:2c:15:a4,nw_src=10.0.0.75,nw_dst=10.0.0.152,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
Megaflow: recirc_id=0x29,eth,ip,in_port=7,nw_dst=10.0.0.152,nw_frag=no
Datapath actions: ct(commit,zone=2,nat(dst=172.20.1.201)),recirc(0x61)

===============================================================================
recirc(0x61) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
Replacing src/dst IP/ports to simulate NAT:
 Initial flow: nw_src=10.0.0.75,tp_src=8,nw_dst=10.0.0.152,tp_dst=0
 Modified flow: nw_src=10.0.0.75,tp_src=8,nw_dst=172.20.1.201,tp_dst=0
===============================================================================

Flow: recirc_id=0x61,ct_state=new|trk,ct_zone=2,eth,icmp,reg0=0xfa16,reg1=0x3e2af74a,reg9=0x4,reg11=0x2,reg12=0x1,reg14=0x1,metadata=0x3,in_port=7,vlan_tci=0x0000,dl_src=52:54:00:be:f5:b1,dl_dst=fa:16:3e:2c:15:a4,nw_src=10.0.0.75,nw_dst=172.20.1.201,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0

bridge("br-ex")
---------------
    thaw
        Resuming from table 15
15. metadata=0x3, priority 0, cookie 0x4918c237
    resubmit(,16)
16. metadata=0x3, priority 0, cookie 0xe8dfa444
    resubmit(,17)
17. metadata=0x3, priority 0, cookie 0xacc28f1
    resubmit(,18)
18. metadata=0x3, priority 0, cookie 0x9476be26
    set_field:0/0xffffffff->xxreg1
    resubmit(,19)
19. ip,metadata=0x3,nw_dst=172.20.1.0/24, priority 74, cookie 0x80987fbc
    dec_ttl()
    set_field:0/0xffff00000000->xreg4
    move:NXM_OF_IP_DST[]->NXM_NX_XXREG0[96..127]
     -> NXM_NX_XXREG0[96..127] is now 0xac1401c9
    set_field:0xac1401fe0000000000000000/0xffffffff0000000000000000->xxreg0
    set_field:fa:16:3e:75:01:fc->eth_src
    set_field:0x3->reg15
    set_field:0x1/0x1->reg10
    resubmit(,20)
20. reg8=0/0xffff,metadata=0x3, priority 150, cookie 0x8845f34f
    resubmit(,21)
21. metadata=0x3, priority 0, cookie 0x435046ed
    set_field:0/0xffff00000000->xreg4
    resubmit(,22)
22. reg8=0/0xffff,metadata=0x3, priority 150, cookie 0x5bb8b7ab
    resubmit(,23)
23. reg0=0xac1401c9,reg15=0x3,metadata=0x3, priority 100, cookie 0xf1d842b1
    set_field:fa:16:3e:33:2b:77->eth_dst
    resubmit(,24)
24. metadata=0x3, priority 0, cookie 0x1e44ea85
    resubmit(,25)
25. metadata=0x3, priority 0, cookie 0x6eeaec3d
    resubmit(,26)
26. metadata=0x3, priority 0, cookie 0x3c5cb9fd
    resubmit(,27)
27. metadata=0x3, priority 0, cookie 0xa860b399
    resubmit(,37)
37. priority 0
    resubmit(,38)
38. reg15=0x3,metadata=0x3, priority 100, cookie 0x97e1a3d3
    set_field:0x2->reg11
    set_field:0x1->reg12
    resubmit(,39)
39. priority 0
    set_field:0->reg0
    set_field:0->reg1
    set_field:0->reg2
    set_field:0->reg3
    set_field:0->reg4
    set_field:0->reg5
    set_field:0->reg6
    set_field:0->reg7
    set_field:0->reg8
    set_field:0->reg9
    resubmit(,40)
40. metadata=0x3, priority 0, cookie 0x1f08f333
    set_field:0/0x10->xreg4
    resubmit(,41)
41. metadata=0x3, priority 0, cookie 0x4f5e98be
    resubmit(,42)
42. metadata=0x3, priority 0, cookie 0xdf025dc6
    resubmit(,43)
43. metadata=0x3, priority 0, cookie 0x9788ea5b
    resubmit(,44)
44. metadata=0x3, priority 0, cookie 0xa0c6d643
    resubmit(,45)
45. metadata=0x3, priority 0, cookie 0x62b8d6d5
    resubmit(,46)
46. reg15=0x3,metadata=0x3, priority 100, cookie 0x44be9e99
    resubmit(,64)
64. reg10=0x1/0x1,reg15=0x3,metadata=0x3, priority 100, cookie 0x97e1a3d3
    push:NXM_OF_IN_PORT[]
    set_field:ANY->in_port
    resubmit(,65)
    65. reg15=0x3,metadata=0x3, priority 100, cookie 0x97e1a3d3
            clone(ct_clear,set_field:0->reg11,set_field:0->reg12,set_field:0->reg13,set_field:0x4->reg11,set_field:0x5->reg12,set_field:0x2->metadata,set_field:0x2->reg14,set_field:0->reg10,set_field:0->reg15,set_field:0->reg0,set_field:0->reg1,set_field:0->reg2,set_field:0->reg3,set_field:0->reg4,set_field:0->reg5,set_field:0->reg6,set_field:0->reg7,set_field:0->reg8,set_field:0->reg9,resubmit(,8))
            ct_clear
            set_field:0->reg11
            set_field:0->reg12
            set_field:0->reg13
            set_field:0x4->reg11
            set_field:0x5->reg12
            set_field:0x2->metadata
            set_field:0x2->reg14
            set_field:0->reg10
            set_field:0->reg15
            set_field:0->reg0
            set_field:0->reg1
            set_field:0->reg2
            set_field:0->reg3
            set_field:0->reg4
            set_field:0->reg5
            set_field:0->reg6
            set_field:0->reg7
            set_field:0->reg8
            set_field:0->reg9
            resubmit(,8)
         8. reg14=0x2,metadata=0x2, priority 50, cookie 0x267bdf53
            resubmit(,9)
         9. metadata=0x2, priority 0, cookie 0xc4339339
            resubmit(,10)
        10. metadata=0x2, priority 0, cookie 0x13b355fd
            resubmit(,11)
        11. metadata=0x2, priority 0, cookie 0xba29016d
            resubmit(,12)
        12. metadata=0x2, priority 0, cookie 0x42ecbd8f
            resubmit(,13)
        13. ip,reg14=0x2,metadata=0x2, priority 110, cookie 0x995dac96
            resubmit(,14)
        14. ip,reg14=0x2,metadata=0x2, priority 110, cookie 0x91db794f
            resubmit(,15)
        15. metadata=0x2, priority 0, cookie 0xc119fe5f
            resubmit(,16)
        16. ct_state=-trk,metadata=0x2, priority 5, cookie 0x963f4597
            set_field:0x100000000000000000000000000/0x100000000000000000000000000->xxreg0
            set_field:0x200000000000000000000000000/0x200000000000000000000000000->xxreg0
            resubmit(,17)
        17. metadata=0x2, priority 0, cookie 0x26c1a9b0
            resubmit(,18)
        18. metadata=0x2, priority 0, cookie 0xc69bbb1e
            resubmit(,19)
        19. metadata=0x2, priority 0, cookie 0xbfbbcdc5
            resubmit(,20)
        20. metadata=0x2, priority 0, cookie 0xb56cc401
            resubmit(,21)
        21. metadata=0x2, priority 0, cookie 0xdac8fa19
            resubmit(,22)
        22. metadata=0x2, priority 0, cookie 0xa69e779a
            resubmit(,23)
        23. metadata=0x2, priority 0, cookie 0x53a1d404
            resubmit(,24)
        24. metadata=0x2, priority 0, cookie 0xe35dea33
            resubmit(,25)
        25. metadata=0x2, priority 0, cookie 0x8c286a3a
            resubmit(,26)
        26. metadata=0x2, priority 0, cookie 0xc64dfb55
            resubmit(,27)
        27. metadata=0x2, priority 0, cookie 0xac3d35b5
            resubmit(,28)
        28. metadata=0x2, priority 0, cookie 0xe6eaf52d
            resubmit(,29)
        29. metadata=0x2, priority 0, cookie 0x46587fef
            resubmit(,30)
        30. metadata=0x2, priority 0, cookie 0x713c29df
            resubmit(,31)
        31. metadata=0x2, priority 0, cookie 0xdd401e7c
            resubmit(,32)
        32. metadata=0x2,dl_dst=fa:16:3e:33:2b:77, priority 50, cookie 0x22d61661
            set_field:0x3->reg15
            resubmit(,37)
        37. priority 0
            resubmit(,38)
        38. reg15=0x3,metadata=0x2, priority 100, cookie 0xc3d19a43
            set_field:0x3->reg13
            set_field:0x4->reg11
            set_field:0x5->reg12
            resubmit(,39)
        39. priority 0
            set_field:0->reg0
            set_field:0->reg1
            set_field:0->reg2
            set_field:0->reg3
            set_field:0->reg4
            set_field:0->reg5
            set_field:0->reg6
            set_field:0->reg7
            set_field:0->reg8
            set_field:0->reg9
            resubmit(,40)
        40. metadata=0x2, priority 0, cookie 0xa6171594
            resubmit(,41)
        41. ip,metadata=0x2, priority 100, cookie 0x45849d62
            set_field:0x1000000000000000000000000/0x1000000000000000000000000->xxreg0
            resubmit(,42)
        42. ip,reg0=0x1/0x1,metadata=0x2, priority 100, cookie 0xc034e1da
            ct(table=43,zone=NXM_NX_REG13[0..15])
            drop
             -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 43.
             -> Sets the packet to an untracked state, and clears all the conntrack fields.
    pop:NXM_OF_IN_PORT[]
     -> NXM_OF_IN_PORT[] is now 7

Final flow: recirc_id=0x61,ct_state=new|trk,ct_zone=2,eth,icmp,reg10=0x1,reg11=0x2,reg12=0x1,reg14=0x1,reg15=0x3,metadata=0x3,in_port=7,vlan_tci=0x0000,dl_src=fa:16:3e:75:01:fc,dl_dst=fa:16:3e:33:2b:77,nw_src=10.0.0.75,nw_dst=172.20.1.201,nw_tos=0,nw_ecn=0,nw_ttl=63,icmp_type=8,icmp_code=0
Megaflow: recirc_id=0x61,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=7,dl_src=52:54:00:be:f5:b1,dl_dst=fa:16:3e:2c:15:a4,nw_dst=172.20.1.201,nw_ttl=64,nw_frag=no
Datapath actions: ct_clear,set(eth(src=fa:16:3e:75:01:fc,dst=fa:16:3e:33:2b:77)),set(ipv4(ttl=63)),ct(zone=3),recirc(0x62)

===============================================================================
recirc(0x62) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
===============================================================================

Flow: recirc_id=0x62,ct_state=new|trk,ct_zone=3,eth,icmp,reg0=0x1,reg11=0x4,reg12=0x5,reg13=0x3,reg14=0x2,reg15=0x3,metadata=0x2,in_port=ANY,vlan_tci=0x0000,dl_src=fa:16:3e:75:01:fc,dl_dst=fa:16:3e:33:2b:77,nw_src=10.0.0.75,nw_dst=172.20.1.201,nw_tos=0,nw_ecn=0,nw_ttl=63,icmp_type=8,icmp_code=0

bridge("br-ex")
---------------
    thaw
        Resuming from table 43
43. ct_state=+new-est+trk,metadata=0x2, priority 7, cookie 0x569cdcd6
    set_field:0x80000000000000000000000000/0x80000000000000000000000000->xxreg0
    set_field:0x200000000000000000000000000/0x200000000000000000000000000->xxreg0
    resubmit(,44)
44. icmp,reg0=0x80/0x80,reg15=0x3,metadata=0x2, priority 2002, cookie 0x167fc61a
    set_field:0x2000000000000000000000000/0x2000000000000000000000000->xxreg0
    resubmit(,45)
45. metadata=0x2, priority 0, cookie 0xc2c7f022
    resubmit(,46)
46. metadata=0x2, priority 0, cookie 0xc61bdac7
    resubmit(,47)
47. ip,reg0=0x2/0x2002,metadata=0x2, priority 100, cookie 0xb3d00a03
    ct(commit,zone=NXM_NX_REG13[0..15],nat(src),exec(set_field:0/0x1->ct_label))
    nat(src)
    set_field:0/0x1->ct_label
     -> Sets the packet to an untracked state, and clears all the conntrack fields.
    resubmit(,48)
48. ip,reg15=0x3,metadata=0x2,dl_dst=fa:16:3e:33:2b:77,nw_dst=172.20.1.201, priority 90, cookie 0x60a4ec
    resubmit(,49)
49. reg15=0x3,metadata=0x2,dl_dst=fa:16:3e:33:2b:77, priority 50, cookie 0x1524c8d1
    resubmit(,64)
64. priority 0
    resubmit(,65)
65. reg15=0x3,metadata=0x2, priority 100, cookie 0xc3d19a43
    output:5

Final flow: recirc_id=0x62,eth,icmp,reg0=0x283,reg11=0x4,reg12=0x5,reg13=0x3,reg14=0x2,reg15=0x3,metadata=0x2,in_port=ANY,vlan_tci=0x0000,dl_src=fa:16:3e:75:01:fc,dl_dst=fa:16:3e:33:2b:77,nw_src=10.0.0.75,nw_dst=172.20.1.201,nw_tos=0,nw_ecn=0,nw_ttl=63,icmp_type=8,icmp_code=0
Megaflow: recirc_id=0x62,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=ANY,dl_src=fa:16:3e:75:01:fc,dl_dst=fa:16:3e:33:2b:77,nw_dst=172.20.1.201,nw_frag=no
Datapath actions: ct(commit,zone=3,label=0/0x1,nat(src)),10

Non DVR

#todo

Notes:

L3HA mode is always enabled, DVR is enabled on top of it. in L3HA mode FIP is hosted on the highest prio chassis, traffic to FIP will go through the gw chassis and then through the tunnel to the instance on the compute node

Each chassis is given a specific priority for the router’s gateway and priority increases with increasing value ( i.e. 1 < 2 < 3 …). The highest prioritized chassis hosts gateway port. Other chassis are selected as slaves. https://docs.openstack.org/networking-ovn/latest/contributor/design/l3_ha_rescheduling.html

Contents