Analyzing all that data: Techniques for sifting haystacks and finding needles

Jose Nazario, Ann Arbor, MI
Presented at Hack in the Box, Kuala Lumpur, Malaysia, September 29, 2005

Previously, gathering data was a difficult task, and so simple data analysis techniques worked well. now with access to information increasing, and the need to get an even broader coverage of events, making sense of mountains of data has never been more pressing. The great risk in this scenario is missing an indicator or losing data.

This presentation will introduce you to a number of techniques for making sense of large collections of data, including sorting and clustering techniques, fuzzy matching, and trend analysis. These techniques have applicability in numerous applications, such as mail filtering and network event analysis.

Slides: [PDF]

Related code