flowgrep - TCP/UDP/IP stream 'grep' tool

"the world's cheapest IPS" -- a friend
"cool, ngrep but better ;)" -- another friend

ABOUT

flowgrep is a basic IDS/IPS tool written in python as a way to help you investigate and manage your network. it works by sniffing traffic, reassembling TCP streams, and IP and UDP fragments into single packets, and allowing you to "grep" through their payloads using regular expressions. the quality of the regular expression engine is similar to Perl's. think of it as a marriage of tcpflow, tcpkill, and ngrep.

flowgrep's usage is quite simple:

$ flowgrep.py -h
./flowgrep: TCP stream/UDP/IP payload 'grep' utility
    Usage: ./flowgrep OPTIONS [FILTER]

    where OPTIONS are any of the following:
       -a [pattern]     match any stream with pattern
       -c [pattern]     match client stream with pattern
       -D [num]         distance score for libdistance-based match
       -d [device]      input device 
       -E [name]        string distance algorithm to use
                        (one of: levenshtein, damerau, hamming, jaccard)
       -e [string]      string to compare against for distance-based matches
       -F [file]        obtain server patterns from file, one per line
       -f [file]        obtain client patterns from file, one per line
       -i               case insensitive match
       -k               kill matched stream (TCP only)
       -l [dir]         log matched flows relative to dir (default: .)
       -r [file]        input file (in pcap(3) format)
       -s [pattern]     match server stream with pattern
       -u [username]    run as username (default: nobody)
       -V               print version information and exit
       -v               select non-matching input
       -x               print logged filenames (for use with xargs(1))

    [FILTER]            pcap(3) filter expression
 
      UDP and IP payloads will test any pattern (no stream to test).

you can specify any string or regular expression to look for and optionally set the stream to log or be killed (-l or -k, respectively). note that the kill option only works on TCP streams. you can look for expressions in the client or the server (or both) streams. note that for UDP and IP, both expressions are tested since no "stream" is established. you need root permissions to use this effectively since you are sniffing the network.

logged streams are written out as two streams for TCP streams or a single file for UDP and IP payloads that match. the payload for any matched stream is written out. files are named 'time-source-sport-dest-dport-proto' in the local directory. examples:

-rw-r--r--  1 nobody  jose    40 Oct 20 21:09 1098320995-192.168.10.84-22-192.168.10.190-28318-tcp
-rw-r--r--  1 nobody  jose     7 Oct 20 21:09 1098320995-192.168.10.190-28318-192.168.10.84-22-tcp

note that the files may contain binary data, or malicous data, so handle them with care. non-UDP and non-TCP payloads are not written using a port number, and the protocol is "protoN", where "N" is the number.

USES OF FLOWGREP

you can do measurements of particular traffic, you can build a very cheap IPS device (ie for mail-based worm infestations), disrupt spammers, or you can just sniff on your coworkers.

specific instances:

be a jerk and kill everyone's web sessions, flowgrep -s "HTTP/1.1 200 OK" -k
be a manager and detect and stop SSH on unauthorized ports, flowgrep -i -a "ssh-" -k tcp not port 22
kill HTTP tunnels by stopping non-HTTP traffic on port 80, flowgrep -v -i -c "^GET|^POST|^PUT|^HEAD" -k tcp port 80.

"By now everyone should appreciate just how powerful and useful Flowgrep can be." Richard Bejtlich on his blog on 28 Jan, 2005 (see below).

LICENSE

BSD 3-clause

AUTHOR

jose nazario <jose@monkey.org>

DOWNLOAD

20 october, 2004. version 0.5
20 october, 2004. version 0.6
added a configurable log directory, timestamps to filenames, and compiled regexp statements (to improve performance).
21 october, 2004. version 0.6.1
fixed a UDP callback typo
22 october, 2004. version 0.6.2
typo in getopt
27 october, 2004. version 0.7
added Makefile, setup.py. installation is now a snap ... changed how -v operates, now it acts a lot more like grep (ie negate the match). added a manpage. fixed matching the end or beginning of line anchors in regexps.
1 november, 2004. version 0.7.1
add support for specific devices (flowgrep -d) and input files (-r; broken due to pynids 0.1 limitation, patch sent to author). install as flowgrep, not flowgrep.py.
4 november, 2004. version 0.7.2
use the right method to call nids_killtcp() from pynids ...
7 november, 2004. version 0.7.3
add -F/-f to load patterns from a file (like grep does), add -V (like grep), add -u (like snort), allow multiple -a/-c/-s declarations, add -x (for combinations with xargs).
12 may, 2005. version 0.8a
typo fix in manpage (thank you laurent)
10 sept, 2005. version 0.9: flowgrep 0.9
add preliminary libdistance support, requested by l. sheng at hitb. fix a bug in matching TCP streams with -v.

REQUIREMENTS AND INSTALLATION

pynids, http://pilcrow.madison.wi.us/pynids/, version 0.3 or later
python 2.2 or later, http://www.python.org/
root permissions (to read /dev/bpf*)
optional: libdistance version 0.2.1 or later for fuzzy string matching

TODO

implement -A, -B, -C
find, fix bugs

THINGS THAT REFERENCE FLOWGREP

SIMILAR

ngrep http://ngrep.sourceforge.net/
tcpflow http://www.circlemud.org/~jelson/software/tcpflow/
dsniff http://monkey.org/~dugsong/dsniff/