Netflow

In a previous post I described what is netflow and when do you use it. This post describes how to use netflow with nfdump and nfsen.

Netflow with nfdump and nfsen

Command line and web interface

Having netflow is great but of course you’d like a way to view your netflow data. I’m covering the nfdump and nfsen tools.

nfdump is the command line interface whereas nfsen is the web interface. Both tools can be used together. In fact, nfsen is a web wrapper around the nfdump command line. What’s more, the nfsen web interface always outputs the corresponding command and options that you have to use to re-produce the same output via command line.

Installation of nfdump

Before you can start with nfdump you will need a couple of Linux prerequisites:

sudo apt-get install flex
sudo apt-get install librrd-dev
sudo apt-get install librrds-perl
sudo apt-get install libmailtools-perl
sudo apt-get install libsocket6-perl

Note that if you do not already have a LAMP installation you might also want to install Apache and PHP. This is needed for nfsen.

Then download nfdump and nfsen in /usr/local/src and extract them.

The nfdump process needs its own user. For simplicity you can use user netflow. This user will have to be part of the www-data group to allow nfsen access to the netflow data.

sudo useradd netflow
sudo vigr 
 -> have it look like this
   www-data:x:33:netflow

Now enter the nfdump directory, run the configure script, make and build the binaries.

cd /usr/local/src/nfdump-1.6.13
./configure --enable-nfprofile --enable-nftrack --enable-sflow --enable-readpcap --enable-nfpcapd
make
sudo make install

The configure script uses a couple of options, this is what they do :

• –enable-nfprofile : needed for nfsen, build nfprofile;
• –enable-nftrack : needed for portracker (nfsen module);
• –enable-sflow : build sflow collector sfcpad;
• –enable-readpcap : build nfcapd collector to read from pcap file instead of network data;
• –enable-nfpcapd : build nfpcapd collector to create netflow data from interface or pcap data

Configuration and installation of nfsen

For nfsen you have to have a working web server with PHP support. Enter the nfsen directory, make a copy of the default config file etc/nfsen-dist.conf to etc/nfsen.conf and update the file.

cd /usr/local/src/nfsen-1.3.6p1/    
cp etc/nfsen-dist.conf etc/nfsen.conf

In nfsen.conf you have to do these changes

$WWWUSER  = "www-data";
$WWWGROUP = "www-data";

$PERL_HAS_MEMLEAK=1;

If you’re not happy with the default proposed datadirectory then change the setting $BASEDIR.

Now you have to configure the netflow sources, enable plugins and set the e-mail server for the alert e-mails. Open the nfsen.conf again. We’ll add the source we created before with pmacctd.

     %sources = (
         'local'       => { 'port' => '9001', 'IP' => '127.0.0.1', 'col' => '#0000ff' }
    );

The porttracker plugin allows you to get quick overviews on the most used network ports.

    @plugins = (
        # profile    # module
        # [ '*',     'demoplugin' ],
         [ '*', 'PortTracker' ],
    );

Netflow is capable to send you alerts by e-mail. To have this working properly you have to configure the e-mail server and the sender.

    $MAIL_FROM   = 'changeme@example.com';
    $SMTP_SERVER = 'localhost';

Now you have to run the install script to create the necessary directories and files. In the nfsen directory do this

sudo ./install.pl etc/nfsen.conf

You will be asked to set what Perl version to use and when all goes well the output should conclude with ‘setup done’. In order to get the porttracker plugin working you need to take some extra steps.

touch /data/nfsen/profiles-stat/hints
chown netflow:www-data /data/nfsen/profiles-stat/hints

Finally you should also tweak your Apache configuration. The main script used by nfsen is nfsen.php. It makes sense to set this script as the default script to execute when accessing the nfsen website.

Ideally you have this website run over SSL, preferably restricting access with a client SSL certificate.

<VirtualHost _default_:443>
        ServerName nf.mydomain.be
        DocumentRoot /var/www/nfsen
        DirectoryIndex nfsen.php
...
</VirtualHost>

Adding sources to nfsen

If you want to add an additional netflow source to nfsen you will have to add it to the nfsen.conf file. After adding it you have to go through the install process again.

sudo ./install.pl etc/nfsen.conf

This will also stop and restart the nfsen processes.

Starting and stopping of nfsen

The nfsen command in $BINDIR is also used to start and stop NfSen

bin/nfsen start
bin/nfsen stop

Working of nfdump and nfsen

Basically how nfdump and nfsen works is that it uses different nfcapd processes to read the netflow data from the network and store the data into files. The files are organized in a time based fashion in a dedicated directory (most often /data/nfsen) and frequently rotated (typically every 5 minutes). This results in

/data/nfsen/profiles-data/<profile-name>/<source-name>/<year>/<month>/<day>/nfcapd.<year><month><day><hour>20
/data/nfsen/profiles-data/<profile-name>/<source-name>/<year>/<month>/<day>/nfcapd.<year><month><day><hour>25
/data/nfsen/profiles-data/<profile-name>/<source-name>/<year>/<month>/<day>/nfcapd.<year><month><day><hour>30

Every query started via the web interface of nfsen will continue to run until its finished, regardless if you tried to stop the web-process. If you want to cancel a long running nfsen query you will have to kill (on the command line) the corresponding nfdump command.

Using nfsen

Overview page

Once you have configured all of the above it’s time to have a look at the capabilities of nfsen. The default web interface has a couple of tabs. By default you’ll get an overview of the traffic per day, week, month mapped out per flow/s, packets/s and bytes/s.

Details page

But eventually you’ll want to click on the Details tab. That is the page that will allow you to get the most out of netflow.

The details page consists of three major blocks

• Graphs : a visual overview of flows, packets and bytes;
• Statistics : a statistical overview of flows, packets and bytes;
• Query : a query form

Details – graphs

The visual overview of the number of flows, packets and bytes gives you immediate access on what’s going on. The details page has a couple of options to fine tune what you are seeing.

• 1 : do a query based on a single timeslice or a time window. If you choose for time window you are able to move the sliders (7) to the start and end of the desired timeframe, for a single timeslice you can move the slider to the left or right;
• 2 : display information for a day, a week or another timeframe. Note that this is only the visual representation, it does not influence the time window of your query;
• 3 : move the graphs to the end or the beginning of a timeframe;
• 4 : display the data based on the number of bytes;
• 5 : display the data based on the number of packets;
• 6 : filter on only TCP, UDP, ICMP, other or all the traffic;
• 7 : the time slider that allows you to set the time window.

The next part of the details page contains the statistics for the different sources.

The last part of the details page contains the part that you will probably be using the most. It’s the query part that allows you to query the netflow data.

• 1 : the nfdump filter;
• 2 : the netflow sources;
• 3 : set the sortorder;
• 4 : get flow data, host data, …;
• 5 : aggregation options.

The use of the nfsen web interface, together with a couple of examples, is described in detail on the nfsen website.

Profiles

The profiles in nfsen are a powerful feature to build a specific view on your data. A profile is defined by its name, type and one or more profile filters. It’s an ideal solution if for example you want to

• have a view specific for the HTTP and HTTPs traffic;
• have a view for traffic coming from a specific network source;
• have a view for traffic with specific TCP flags.

Alerts

Nfsen has a feature that allows you to get e-mail alerts when an nfdump query is matched. This alerting is an ideal solution if you want to get notices when a connection has been detected towards for example a C2 server.

You can set a number of thresholds and conditions before the alerts triggers.

An existing alert also holds a graphical overview of traffic that matched the nfdump query.

Conclusion

Netflow data, especially when analyzed via nfdump or nfsen, is an excellent source to have both an overview of what is going on on your network and detect malicious (network) activity. It requires some training to getting used to extract actionable data but especially for network based activity it is a very useful source of data.

If you do not have access to network devices you can still use netflow data based on network traffic that is exported by Linux or BSD hosts.