Re: Parsing tcpdump files

["Steve Bernard" <sbernard_(_at_)_gmu_(_dot_)_edu>]

Joe,

Thanks for the suggestions. Up until now I've been doing basically what 
you have suggested, but with small files (300MB), and using 
BASH/grep/etc. for the parsing. Importing everything into a RDBMS was my 
first strategy but that isn't feasible right now. I intend to store the 
reporting data in MySQL but I don't have the disk space to duplicate 
everything in the RDBMS. I was hoping that there were some tools for 
parsing the data in binary format because converting to ASCII adds about 
35% to the file sizes, and my customer prefers the data to be in 
tcpdump's binary format. I still need to compare the file sizes using 
hex format instead of ASCII. Unless I hear other suggestions I plan on 
going forward using Perl or Python.

Thanks,

Steve


Joseph C. Bender wrote:
> On Saturday 01 March 2003 08:30 pm, Steve Bernard wrote:
>
> > Jack,
> >
> > I'm supporting several granted network engineering and security analysis
> > research projects. Each project has specific data requirements and
> > capabilities. To facilitate this I need to perform string parsing, data
> > aggregation/sub-setting, statistical analysis, and reporting. They will
> > each do much more on their own but, this is what is required at my end. I
> > anticipate the capture files being around 1GB each.
>
> 	Well, I just had to do some looking at a 12 hour capture of some database 
> server traffic.
>
> I did the capture with Tcpdump, then did my initial sorting with tcpdump 
> reading the file with filtering for each major grouping of info I needed.
>
> Because I was looking for something, I then pulled each file into Ethereal to 
> better see the captures, and display-filter the data in question.
>
> However, it would seem that you'd want to write the ascii output of tcpdump to 
> files, then using $PARSING_LANGUAGE_OF_CHOICE to generate your numbers, or 
> pull the info into a RDBMS for using some major tool.