I have two remote ESXi server in different datacenter. On both I installed 2 VM pfsense with a site2site VPN with OpenVPN to connect the 2 subnet 192.168.1.0/24 and 192.168.2.0/24 over internet.
I have a lot of VM on these two ESXi connected each other via the above subnets. All can ping each other except one.
This is the situation:
Site A: 192.168.10.0/24
Site B: 192.168.20.0/24
Pfsense A: 192.168.10.1/24
Pfsense B: 192.168.20.1/24
VM A1: eth1 192.168.10.2/24 and eth0 1.1.1.1/32
VM A2: 192.168.10.3/24 (single IF with default GTW 192.168.10.1
VM B1: eth0 192.168.20.2/24 and eth1 2.2.2.2/32
VM A1 route -n:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 11.11.11.11 0.0.0.0 UG 101 0 0 eth0
11.11.11.11 0.0.0.0 255.255.255.255 UH 100 0 0 eth0
1.1.1.1 0.0.0.0 255.255.255.255 UH 100 0 0 eth0
192.168.20.0 192.168.10.1 255.255.255.0 UG 100 0 0 eth1
192.168.10.0 0.0.0.0 255.255.255.0 U 100 0 0 eth1
VM B1 route -n:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 22.22.22.22 0.0.0.0 UG 101 0 0 eth1
22.22.22.22 0.0.0.0 255.255.255.255 UH 100 0 0 eth1
2.2.2.2 0.0.0.0 255.255.255.255 UH 100 0 0 eth1
192.168.10.0 192.168.20.1 255.255.255.0 UG 100 0 0 eth0
192.168.20.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
(11.11.11.11 and 22.22.22.22 is the default gateway given from the provider, I need this configuration to connect from remote on that IP).
The strange thing is:
- I can ping A1 and A2 from B1.
- I can ping B1 from A2.
- I CAN'T ping B1 from A1.
On tcpdump on pfsense, pinging from A1 to B1, I notice that A1 use 1.1.1.1 as IP, instead 192.168.10.2, while B1 and A2 use 192.168.20.2 and 192.168.10.3:
tcpdump on pfsense A, ping pfsense A from A1:
19:41:33.739531 IP 192.168.10.2 > 192.168.10.1: ICMP echo request, id 26186, seq 1, length 64
19:41:33.739565 IP 192.168.10.1 > 192.168.10.2: ICMP echo reply, id 26186, seq 1, length 64
tcpdump on pfsense A, ping pfsense B from A1:
19:41:41.065664 IP 1.1.1.1 > 192.168.20.1: ICMP echo request, id 26193, seq 1, length 64
tcpdump on pfsense A, ping B1 from A1:
19:41:41.065664 IP 1.1.1.1 > 192.168.20.2: ICMP echo request, id 26193, seq 1, length 64
tcpdump on pfsense A, ping B1 from A2:
19:42:24.368952 IP 192.168.10.3 > 192.168.20.2: ICMP echo request, id 31618, seq 1, length 64
19:42:24.383237 IP 192.168.20.2 > 192.168.10.3: ICMP echo reply, id 31618, seq 1, length 64
tcpdump on pfsense B, ping A1 from B1:
19:46:05.538819 IP 192.168.20.2 > 192.168.10.2: ICMP echo request, id 12244, seq 4, length 64
19:46:05.552069 IP 192.168.10.2 > 192.168.20.2: ICMP echo reply, id 12244, seq 4, length 64
A1 and B1 are two Centos 7.3 with network manager. The route was added with nmcli.
Why B1 can reach A1 but A1 not? They have the same route!
--EDIT-- I found the problem: There was a SNAT rule on my A1 iptables. I excluded 192.168.10.2 from translation and now it work.
