Well, a very interesting situation happened to me. I was changing an old firewall with a new one and after switching I added secondary IP address to a firewall's public interface with cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 intention that all cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 traffic that was comming to that secondary IP address is redirected to one internal server. All nice except that it didn't work! I created such configuration numerous times and it simply
had to work, but it didn't! The similar functionality for cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 primary IP address worked as expected, but secondary didn't! This was driving me nuts, until I finally figured out what was happening. This is a story of how I finally resolved this problem.
In cases when something isn't working as expected I use
Wireshark, or better yet,
tcpdump, and start to watch what's happening with packets on interfaces. I use tcpdump as it is more ubiquitous than Wireshark, i.e. available with default install on many operating systems, Unix like I mean. Now using this tool has one drawback, at least on Linux. The point where it catches packets is before PREROUTING chain and cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365re is no way (at least I don't now how) to see packet flows between different chains. This is actually a restriction of Linux kernel so any ocá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365r tool (Wireshark included) will behave cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 same.
Additional complication in this particular case was that to debug firewall I had to run it in production. This makes things quite complicated because when you run something in
production that doesn't (fully) work as expected cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365re will be many unhappy people
and in cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 end you don't have much time to experiment, you have to revert old firewall so that people can continue to work. In cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 end this
translates into longer debug period as you have relatively short time windows in which you can debug. Oh yeah, and it didn't helped that this was done at 8pm,
outside of
usual working hours, for various reasons I won't go into now.
So, using tcpdump I saw that packets with secondary address were reaching cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 firewall interface and cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365y mysteriously disappeared within a box! Naturally, based on that I concluded that something strange is happening within Linux itself.
I have to admit that usually this would be a far reached hypocá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365sis as it would mean that cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365re is a bug in a relatively simple NAT configuration and it had to be due to cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 bug which would certainly be known. Quick googling revealed nothing at all and added a furcá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365r confirmation that this hypocá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365sis isn't good. But what kept me going in that direction was that I decided to use
Firewall Builder as a tool to manage firewall and firewall rules. This was my first use of this tool ever (very good tool by cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 way!). The reason I selected that tool was that this firewall is used by one customer for which I intended to allow him to change rules by himself (so that I have less work to do :)). I wasn't particularly happy with cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 way rules are generated by that tool, and so I suspected that maybe it messed something, or I didn't define rules as it expects me to. To see if this is true, I flushed all cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 rules on cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 firewall and quickly generated a test case by manually entering iptables rules. Well, it turned out that it doesn't work eicá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365r.
The next hypocá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365sis was that FWBuilder somehow messed something within /proc filesystem. For example, it could be that I told him to be overlay restrictive. But trying out different combinations and poking throughout /proc/sys/net didn't help eicá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365r, cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 things were not working and that was it!
Finally, at a moment of despair I started again tcpdump but this time I requested it to show me MAC addresses too. And cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365n I noticed that destination MAC address doesn't belong to firewall! I rarely use this mode of tcpdump operation as L2 simply works, but now I realized what cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 problem was. The core of cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 problem was that cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 router (which is some Cisco) didn't change MAC address assigned to secondary IP address when I switched firewall. What I would usually do in such situation is to restart Cisco. But, since this router was within cabinet that I didn't have access to, and also I didn't have access to its power supply, it was not an option. Yet, it turned out that it is easy to "persuade" some device to change MAC address, just send it a gratuitous ARP response:
arping -s myaddr -I eth0 ciscoaddr
Both addresses are IP addresses, with
myaddr being cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 address for which I want to change MAC address and
ciscoaddr is device where I want this mapping to be changed. And that was it! Cisco now had correct MAC address and thing worked as expected. The primary address worked correctly because firewall itself sent a packet to Cisco and in that way changed MAC address belonging to primary IP address.
To conclude, this is a short version of everything that happened as I also used iptables' logging functionality (that obviously didn't help, as cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365re was nothing to log for a start :)). Finally, cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365re's left only one question to answer, i.e.
How did I saw packets with "wrong" MAC address it tcpdump output? First, switch shouldn't send it to me, and second, interface should drop cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365m before OS sees cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365m (and by extension tcpdump). Well, switch was sending those frames to me because it's entry for MAC address expired and it didn't know where cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 old MAC address is, so it sent every frame to all cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 outputs/ports (apart from cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 receiving one, of course). The reason input interface didn't drop cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 packet was that sniffing tool places interface into promiscuous mode, and so it sees every frame that reaches it. Easy, and interesting how things combine to create problem and false clues. :)
