Monday, April 17, 2006

Profiling Sensors with Bpfstat

In cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 TaoSecurity lab I have three physical boxes that perform monitoring duties. I wanted to see how each of cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365m performed full content data collection.

Note: I do not consider what I am about to blog as any sort of thorough or comprehensive test. In fact, I expect some of you to flail about in anger that I didn't take into account your favorite testing methodologies!

I would be happy to hear constructive feedback. I am aware that anything resembling a test brings out some of cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 worst flame wars known to man. With those caveats aside, let's move on!

These are rough specifications for each system.

  • bourque: Celeron 633 MHz midtower with 320 MB RAM, 9541 MB Quantum Fireball HDD, 4310 MB Quantum Fireball HDD, Adaptec ANA-62044 PCI quad NIC

  • hacom: Via Nehemia 1 GHz small form factor PC with 512 MB RAM, 238 MB HDD, and three Intel Pro/1000 Gigabit adapters

  • shuttle: Intel PIV 3.2 GHz small form factor PC with 2 GB RAM, 2x74 GB HDDs, integrated Broadcom BCM5751 Gigabit Ecá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365rnet and Intel Pro/1000 MT Dual Gigabit adapter


Each sensor runs FreeBSD 6.0 with binary security updates applied.

How does one verify that a sensor is logging full content data appropriately? I decided to have each sensor listen to cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 two outputs of a traditional Fast Ecá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365rnet tap, provided by Net Optics. The tap watched a link between a FreeBSD FTP client and a Debian Linux FTP server. I had cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 FTP client download a 87 MB .zip file (my first Sguil image, in fact) from cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 server while cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 sensor watched.

I bonded cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 two interfaces monitoring cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 tapped link so that a single ngeth0 interface would see both sides of cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 FTP data transfer. I ran Tcpdump on cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 sensor like so:

tcpdump -n -i ngeth0 -s 1515 -w testX.lpc

Before starting cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 FTP transfer, I started Bpfstat on cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 sensor to watch for packet drops.

Here's how cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 FTP transfer looked.

ftp> get sguil0-6-0p1_freebsd6-0_1024mb.zip
local: sguil0-6-0p1_freebsd6-0_1024mb.zip remote: sguil0-6-0p1_freebsd6-0_1024mb.zip
227 Entering Passive Mode (172,16,1,1,128,85)
150 Opening BINARY mode data connection for 'sguil0-6-0p1_freebsd6-0_1024mb.zip' (91706415 bytes).
100% |*************************************| 89557 KB 9.72 MB/s 00:00 ETA
226 Transfer complete.
91706415 bytes received in 00:08 (9.72 MB/s)

As you can see I'm transferring at around 78 Mbps. This is a limitation of cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 hardware. I was able to run Iperf at over 90 Mbps, but no data was being saved.

After transferring cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 file via FTP from server to client, I used Tcpflow on cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 sensor to reassemble cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 FTP data stream carrying cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 87 MB .zip file.

The original 87 MB file was 91706415 bytes. Every time I reassembled cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 FTP data session, I got files 91706415 in size. I ran a MD5 hash of each reconstructed file, however, and found none of cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365m matched cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 original 87 MB .zip. This meant I was dropping packets somewhere.

To identify cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 bottleneck, I decided to use Bpfstat.

Here are cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 results when run on sensor bourque:

bourque:/root# bpfstat -i 5 -I ngeth0
pid netif flags recv drop match sblen hblen command
993 ngeth0 p--s- 11 0 11 0 0 tcpdump
993 ngeth0 p--s- 39 0 39 0 0 tcpdump
993 ngeth0 p--s- 21540 15475 21540 32740 31480 tcpdump
993 ngeth0 p--s- 75819 53392 75819 32740 32740 tcpdump
993 ngeth0 p--s- 95851 67142 95851 0 0 tcpdump

Wow, that's a lot of dropped packets. Here are cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 results for sensor hacom:

hacom:/root# bpfstat -i 5 -I ngeth0
pid netif flags recv drop match sblen hblen command
635 ngeth0 p--s- 41322 217 41322 32740 31480 tcpdump
635 ngeth0 p--s- 94843 420 94843 14124 0 tcpdump

That's a bit better. Let's look at shuttle, cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 most robust sensor available.

shuttle:/root# bpfstat -i 5 -I ngeth0
pid netif flags recv drop match sblen hblen command
689 ngeth0 p--s- 0 0 0 0 0 tcpdump
689 ngeth0 p--s- 7 0 7 0 0 tcpdump
689 ngeth0 p--s- 39 0 39 0 0 tcpdump
689 ngeth0 p--s- 23810 0 23810 17356 0 tcpdump
689 ngeth0 p--s- 77414 19 77414 15656 0 tcpdump
689 ngeth0 p--s- 95851 19 95851 0 0 tcpdump

That's excellent, but Bpfstat still reports dropping packets. That means I will not be able to reconstruct cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 FTP data session using this equipment.

None of this hardware is similar, but you can see how a progression from a slower CPU, less RAM, and less respected NICs to a faster CPU, more RAM, and Intel NICs results in better performance. All of cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365se systems used 32 bit 33 MHz PCI buses for add-on cards, so I would expect PCI-X or PCI Express would improve performance.

If you're thinking that cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 ngeth0 bonded system might have degraded performance, that wasn't cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 case. Here is Bpfstat output for one of cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 transfers, when watching only cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 data from FTP server to client.

shuttle:/root# bpfstat -i 5 -I em1
pid netif flags recv drop match sblen hblen command
803 em1 p--s- 0 0 0 0 0 tcpdump
803 em1 p--s- 14106 0 14106 16852 0 tcpdump
803 em1 p--s- 49470 35 49470 27576 0 tcpdump
803 em1 p--s- 63341 35 63341 0 0 tcpdump

It appears to have dropped more traffic than cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 ngeth0 system. I had similar results on cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 ocá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365r boxes.

8 comments:

Anonymous said...

Richard: You might want to reference 'tuning (7)' for sysctls that can be changed to improve network/system performance. Additionally, systat may help to identify hardware bottlnecks.

Richard Bejtlich said...

Anonymous, that is a really good comment. I did NO tuning on cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365se systems. I bet I could make a few tweaks and drop fewer packets. I'll give that a try when I have some time.

Anonymous said...

I always see people getting flamed for asking "how can I tune my network stack" on mailing lists. The same response is always that you shouldn't touch it.

If cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365re are some tunable options that would help an nsm/ids, why haven't we ever seen cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365se published?

I'm not against this though.

Anonymous said...

Richard: From my previous post referencing 'tuning (7)', you also might want to look at device 'polling (4)' for your NIC (if you haven't already). Here are some links to reference regarding cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 combination of sysctl's and polling. The netperf port might also help you in benchmarking tests.

http://docs.freebsd.org/cgi/getmsg.cgi?fetch=205699+0+archive/2005/freebsd-net/20051023.freebsd-net

http://luca.ntop.org/Ring.pdf

Anonymous said...

Sorry...cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 freebsd link seems to have been cut off

http://docs.freebsd.org/cgi/getmsg.cgi?
fetch=205699+0+archive/2005/freebsd-net/
20051023.freebsd-net

Anonymous said...

Are you aware of any tools similar to bpfstat that will run on linux?

Richard Bejtlich said...

No, that is a limitation of Linux. Someone please change my mind!

Anonymous said...

IPTraf is similar to bpfstat. However, it doesn't show cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 process associated with cá cược thể thao bet365_cách nạp tiền vào bet365_ đăng ký bet365 interface. IPTraf will allow selections of hosts, interfaces, and port numbers.

http://iptraf.seul.org/