Help us out readers: How would you sniff and store network traffic?

Having successfully averted the XP-ocalypse at the Wirliyatjarrayi Learning Centre in the remote Australian desert community of Willowra, Vulture South's operatives felt pleased with themselves. As regular readers will know, our encounter with the facility, run by the Batchelor Institute of Indigenous Tertiary Education, led …

[Khaptain]

Wireshark maybe

Wireshark would be my first choice but what exactly are your needs.

Do you just want to make a general sniff or is it more application oriented sniffing that you are after.

As an example : Are you looking to sniff out the various layers of TCP/IP or are you after the URLs that IE is browsing ?.

How much space/resources do you have : Is it only the client machines hard disk, or is there also some kind of NAS ?

Do you need this traffic sent real time ?

Do you need 24/24 worth of sniffing or just periodic snapshots ?

[Woodgie]

Re: Wireshark maybe

These are all good questions which can be summarised by the question: "What is your aim?"

Do you want the info for troubleshooting? Compliance reasons? To catch someone you suspect of doing something naughty? How you answer these should dictate how much importance you put on each aspect of the task.

For instance if you're trying to understand who is using the most bandwidth and which sites they're using then it MIGHT be a better solution to upgrade to a router/firewall that can gather these statistics for you if you don't have one already (I haven't read the other article yet). Most home routers these days and certainly the lower cost 'business' routers seem to have this functionality.

You might find that your firewall/router also has the ability to do more detailed log capture/packet analysis and send the results to a log server which could be one of the internal PCs which you SSH into and SCP the files from OR you could get it to log to YOUR server, wherever that might be.

There are many options. And the more I think of it the more I can come up with but the key question remains: "What is your aim?"

[Anonymous Coward]

You can only sniff broadcast traffic (software such as Wireshark) on your own layer 2 or traffic passing across the sniffer's NIC (s).

Remember the numerous suggestions you've obviously ignored about using a caching proxy? A chimp could setup smoothwall. It runs nicely on any old Pc with 2 network cards (and runs well in virtualbox). All you need is a PC with 2 network cards.

[Woodgie]

I know you said "...Installing a server to do the sniffing probably isn't an option..." so I'm going to take the 'probably' bit and run with it. Also, in the time it takes me to type this up ninety-eleven other people are bound to have said the same thing.

I think this is probably exactly the kind of thing a Raspberry Pi running Wireshark would excel at. It'd allow you to SSH in to SCP the dump files for analysis, it's small with low power requirements and importantly it's very low cost.

It's true though that if you wanted to capture ALL traffic on the local layer 2 segment then you'd have to have (as I understand it) a switch that can mirror traffic from other ports to the one the Pi is attached to otherwise all you'll be able to pick up is broadcast traffic. Also, with regard to inbound and outbound traffic you'd need to find a way to use the Pi as a router and pass the out/in bound traffic over NICs attached to it which would mean using a USB to Ethernet adapter and I have no experience using one of them with a Pi.

[Terafirma-NZ]

I will go with Woodgie's comment about still going to site but with different gear. I say this as putting together a full report of all applications in use from a wireshark capture will take longer than 2 days and that is after you have pulled all the capture files back over the slow link that along might take more than the 2 days depending on the depth of capture and amount of local traffic.

Ask Palo Alto network for a demo unit to trial on your network to see what applications are in use. You then get a report from the box and Palo Alto will also generate a nice PDF report for you detailing all the info. You might have to swing a few sentences about the job as this is normally for lead generating in the sales team. But you may even swing it for a Reg review.

Or even better if the links are slow you can just purchase a PA200 unit and leave it there gaining all the features it provides. (they are pretty low cost)

Installing it is just a simple power and connect network in and network out running in wire-tap mode with allow all rule (default from factory); so you could freight it there and talk someone through it with some photo's (assuming video calling is out of the question).

no I'm not an employee just a happy customer.

[jake]

"Are there other requirements we've overlooked?"

Yep. You are missing a cognizant engineer who understands both networking & system security. Until you have that, you are spinning your wheels. There are no substitutes for proper education in any given field (which surely you are aiming for overall?).

[cleidh_mor]

TCPdump

If you're running a Linux based wifi router, tcpdump might be available on the router, which would be ideal. Otherwise, run it on the problem PC and log the results to text file.

You didn't mention whether this was for long term analysis or troubleshooting, so it's difficult to say how much storage you'll need, but tcpdump is pretty efficient and by generating text files, compression will work well. You can also filter very efficiently with tcpdump to minimise the volume of data you are storing.

[Tom Wood]

Re: TCPdump

Indeed. Wireshark and tcpdump both use libpcap for the actual capture of traffic. tcpdump is a command line frontend, ideal if you want to run a packet capture for a period of time then analyze it later (perhaps with Wireshark). Wireshark allows real-time capture and analysis (using libpcap) but also offline analysis of saved captures.

libpcap allows highly configurable capture filters to reduce the size of the packet capture by discarding stuff you don't need (such as filter by protocols, hosts, only capture packet headers and discard the payload, etc).

To capture traffic from all the PCs though you either need to run a capture on each individual PC or on the switch/router/gateway. You can only intercept the traffic where it passes and ordinarily one PC won't see traffic destined for another PC.

[Andrew Commons]

Start simple

"the Learning Centre's WLAN and satellite WAN are both slow"

Latency will be the killer for the satellite WAN and some of this may be reflected in the local WLAN performance if 'local' requests are making it out local environment. DNS would be one candidate I guess.

So start by understanding what traffic is going through the satellite gateway, no need to sniff the traffic yet, just get connection logs if possible from the device itself, this should give you a good idea of what is going on with just src/dest ip and ports.

How you do this will depend on the capabilities of the gateway, best case is very careful use of remote administration access or using the VNC capabilities to get internal administration access otherwise you are facing big time latency - two day round-trip.

[Benno]

What about the option of something on the other side of the WAN link? Can you arrange to pay for a service at the ISP to provide more detailed usage statistics? A transparent proxy might do the trick.

That is, assuming the WAN traffic is what you're interested in. If it's LAN, then you probably need a router with flow features (e.g. Cisco NetFlow) and the respective flow-analyzer. As a 'simpler' version, you could implement a device with ACL's that logged packet counts against 'allow' rules.

[Anonymous Coward]

Is the switch managed?

Is the switch a managed one - one with some sort of administration interface - where you could mirror all the traffic to a given port (also called span port or the like)?

If so you can setup one of the PC (even remotely) to capture traffic during a given period (that should representative of the traffic), and everything will happen at the remote site.

If PC are Windows one, you can use WinPCap + Wireshark (command line tools available), Microsoft Netowrk Monitor or any other tool to capture and then analyze the traffic. Wireshark works on Linux too, but there are other tools as well.

You could also set filters on both capture and analysis phase to see only the outgoing traffic ignoring the LAN one, and then maybe have scripts uploading you files when there is little use of the network, i.e. at night.

If the switch isn't managed, it's much more difficult - you'd need to install additional hardware to intercept the traffic you need to analyze. An hub between the router and the switch (if neither can deliver traffic to another device) can be used to capture the traffic between the devices, but a PC needs to be connected to hub to receive traffic. Inspecting traffic among PCs may be harder, though.

[duhmb]

span port or inline will be needed

If you cannot see all the traffic betwixt switch and satellite you're buggered.

Use ntop to see all traffic once you can plug in to a spanned port or better yet between the two.

If you can put something between the two have it cache web traffic as a pass through.

Ntop is web based so light.

Any crappy old PC can run it.

If you can get hold of smooth wall with proxy you should be right.

[ciaran]

Re: span port or inline will be needed

Ntop is great. Span port can suck as configuration goes, and I would't want to configure one remotely. Inline configuration is good - particurlarly combined with the NAT and a Firewall. An ethernet tap is great, but I've never tried it with Ntop. An ethernet tap is a small hardware contraption that seperately splices into the "in" and "out" stream of an ethernet link. So you need a box with 2 ethernet ports to record the trafic from each direction. Plus a third to actually communicate.

[Chris J]

MikroTik SwOS devices appear to offer port mirroring, and are cheap.

Combine with NtopNG running on raspberry pi perhaps to get overview of what is being sent and received in real time?

[pblakez]

I would start with opendns to see what traffic they are using, pretty easy and cloud based

sniffing locally only after you you figure out what you want to monitor on the local network

[Anonymous Coward]

You have a PhD from Batchelor, or a Bach from Masters?

I have a Bachelor, from Batchelor, and a Masters from Masters. Where do you think I should do a PhD?

[HKmk23]

Over engineering? Perhaps a low tech solution?

While there are some excellent suggestions above which will fulfill your requirement, as you do not state the reason for need I would suggest a simple keylogger on each machine will enable you to analyse traffic to your hearts content.

[Anonymous Coward]

Re: Over engineering? Perhaps a low tech solution?

A keylogger can capture very sensitive information and put you in legal troubles. Also a keylogger doesn't tell you what happens at the network level (and an AV can spot it and block it...)

While capturing packets you can decide to capture only part of the packet itself, i.e. just the transport information and not the whole payload.

When you start to capture and inspect traffic you have to be very careful about what you do and how you do it.

[Anonymous Coward]

I'm with HKmk23... kinda

From my experience, i think it is pointless to analyse traffic anywhere on the network except at the gateway, mainly because that is where you'll see simultaneous hits on the bandwidth.

if you're looking for a per pc analysis, I'd go with a simple software install, something a little more sophisticated than a keylogger, something like eblaster for instance, which will generate your traffic reports by email on a remotely configurable schedule

[TheresaJayne]

how about something a little more elegant, install Oracle Virtual Box or similar onto one of the machines, and install Kali Linux.

Once running you could ssh into it and even give it it's own IP.

It has tools that can Monitor and Scan all manner of Connections.

Or if you had a machine you could put as a potential firewall then you could always install a copy of the FREE firewall software Untangle.