Jose Nazario

here you'll find my blog, projects, pictures, my wiki, and much more. what you wont find is a strong set of design skills.

 Recent Blog Posts

bugspot in python

i saw this discussion on hackernews, which links to a google blog post about how they try and predict bug hotspots based on past activity. pretty simple approach, not all that ineffective, either.

someone on HN had a working ruby tool for git in a few hours, but those are tools i don't use (git or ruby). so i wrote a version for subversion in python, which is below. my output is based on theirs since it made a lot of sense. one thing i think i may do is add support for arbitrary strings for the match conditions (e.g. whatever terms you team uses) and also a max "age" or timeline to start at.

#!/usr/bin/env python
"""
bugspot-svn.py 
copyright (c) 2011 jose nazario , all rights reserved
license: 2 clause BSD
"""
# see
# http://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.html
def svnlog_parser(input):
    assert type(input) is file
    data = []
    while True:
        line = input.readline()
        if line.startswith('-'*30):
            if data:
                yield ''.join(data)
                data = []
        if line == '': raise StopIteration
        else:
            data.append(line)
if __name__ == '__main__':
    import math
    import os
    import re
    import sys
    import time
    try: 
        print 'Scanning %s' % sys.argv[1]
    except IndexError:
        print >> sys.stderr, 'Usage: %s /path/to/repo' % sys.argv[0]
        sys.exit(1)
                           
    s = svnlog_parser(os.popen('cd %s && svn log -v' % sys.argv[1]))
    message_matchers = [ re.compile(x, re.I) for x in ('fixes', 'fixed', 'closes', 'bug\w?#\d+', ) ]
    hotspots = {}
    messages = []
    times = []
    for m in s: 
        paths = []
        lines = m.split('\n')
        i = 0
        for line in lines:
            if line.startswith('-'*20):
                # seperator
                i += 1
                continue
            if i == 1:
                # revision | who | timestamp | N lines
                i += 1
                timestamp = ' '.join(line.split(' | ')[2].split()[:2])
                timestamp = int(time.strftime('%s', time.strptime(timestamp, '%Y-%m-%d %H:%M:%S')))
                times.append(timestamp)
                continue
            if line == 'Changed paths:': 
                # blah
                i += 1
                continue
            try:
                # actual files changed
                if line[3] in ('D', 'M', 'A'):
                    i += 1
                    paths.append(line.split(' ', 1)[1])
                    continue
            except IndexError:
 pass
            # and everything else is the changelog
            msg = ' '.join(lines[i:])
        for matcher in message_matchers:
            if matcher.findall(msg):
                messages.append(msg)
                for path in paths:
                    path = path.strip()
                    l = hotspots.get(path, [])
                    l.append(timestamp)
                    hotspots[path] = l
                break
    start = min(times)
    end = max(times)
    def score(ts):
        s = 0
        for t in ts:
            t = (float(t)-start)/(end-start)
            s += 1/(1+(math.e**(-12*t+12)))
        return s
    hotspots = [ (score(y),x) for x,y in hotspots.iteritems() ]
    hotspots.sort()
    hotspots.reverse()
    hotspots = [ (y,x) for x,y in hotspots ]
    hotspots = filter(lambda x: x[1] > 0.001, hotspots)
    print 'Found %d bugfix commits, with %d hotspots' % (len(messages), len(hotspots))
    print
    print 'Fixes:'
    for msg in messages:
        print '    - %s' % msg
    print
    print 'Hotspots:'
    for path, n in hotspots:
        print '    %.3f - %s' % (n, path)

Scanning /Users/jose/code/phoneyc/trunk
Found 6 bugfix commits, with 2 hotspots
Fixes:
    -  fix quoting issues 
    -  fix arg length 
    -  [phoneyc] support for  RTSP MPEG4 SP Control ActiveX Control "MP4Prefix" Property Buffer Overflow vuln module, exploit demo  
    -  [phoneyc] found an exploit for QvodCtrl 
at SecFocus,
 add.  fix: - add CLSID for QvodCtrl 
 - look for URL and url  - XXX case independent handling of methods etc? - proper length check - object instantiation can be done with name, not just id  
    -  [phoneyc] - handle the  redirect stuff as an href - fix up URLs that lack a needed trailing '/'  
    -  import order fixup fix sgmllib exception namespace  
Hotspots:
    0.463 - /phoneyc/trunk/honeyclient.py
    0.099 - /phoneyc/trunk/modules/jscript/NCTAudioFile2.js

i'll be testing it on larger codebases soon, i developed it against the svn repo of phoneyc.

if you use it, please let me know how you find it. i'll happily accept patches, too.

     [link]      Saturday, Dec 17, 2011 @ 08:42am

passphrase generator

inspired by xkcd 936 on password generators, i wrote a simple passphrase generator in python. you can see it below.

#!/usr/bin/env python
import random
import sys
try: passlen = int(sys.argv[1])
except IndexError:
 passlen = 4          # default to 4
r = random.Random()
words = [ x.strip() for x in open('/usr/share/dict/words', 'r').readlines() ]
for x in xrange(passlen):
    print r.choice(words),
print

usage is simple: just specify how long you want the passphrase to be. if you want klingon, spanish, etc you can alter the list of words.

some examples:

Dix:bin jose$ ./passphrase 
Euplocomi Protomycetales hypodermal reissuer
Dix:bin jose$ ./passphrase 
hidromancy precompounding reascertainment gnomical
Dix:bin jose$ ./passphrase 6
orthographize sicknessproof velvetbreast holocephalian hypochlorhydric harshly

hope this is useful to someone.

     [link]      Saturday, Aug 27, 2011 @ 09:01am

The App Store as a Security Model

Like it or not, the "app store" model is a likely future scenario of computing which really improves security by reducing the attackable space on a user's computer. When paired with cloud computing there's a retro-future aspect to this. We're all headed right back to mainframes and dumb terminals, but this time with shiner devices. The app store model here refers not just to Apple's "App Store" but anyone with a marketplace for installable applications that they gate.

This model has hit it big again, lately, with the iPad and various complaints that Apple's way too restrictive in their relationships with developers. There's a bit of truth to peoples' claustrophobia around this along with the unfairness complaints about Apple's recent behavior. There is another angle to this, and that's reducing the attack surface.

When you think about what a lot of people need to do, it's play some media, handle email, chat and facebook, and surf the web. It's not to do things like manage antivirus or a personal firewall, worry about the latest scareware, etc. The bad guys on the Internet thrive in this complex, confusing, arbitrary purpose computing environment. While you may need to create the next Google, lots of people don't, they just need to work, to communicate, and to enjoy.

The app store model reduces the attack surface are in a few ways. First, there is a strict way to get code to run on the system, including signing and gating through the actual application store that does the download and installation. Secondly, apps are presumably vetted (although, I doubt, for security weaknesses or malicious options), meaning rogue apps could be blocked from entry. Third, telcos and OS folks can always revoke an app's runnable status.

This ignores, as noted, any failure to screen for hidden rogue actions in an app, security flaws (accidental or intentional), or even bribery to get an app into the app store.

Couple this to the cloud, where you have something similar: fewer options, determined by someone else, offered to you as a choice. The cloud is where resources can be allocated to protect your documents. With this mix you now have a model that feels increasingly boxed in for some folks, but is really quite liberating.

What this model does, as far as security is concerned, is move a large chunk of the risk to central, manageable locations: the app store gateway, and the cloud operations folks. Presumably these should protect their own turf, and by extension you, but that's a bit of a stretch and we know it wont always happen that way. I expect some significant issues with app stores and the cloud in the coming years, but I also anticipate that we'll wind up with this model widely adopted, and with significant security benefits to come. Imagine a world where Zeus and Koobface can't arbitrarily infect your computer. Threat vectors will change but never go away.

As for me, I use the cloud but I still use a general purpose OS on my netbook and laptop. I like to innovate, and a closed platform like the iPhone OS isn't supportive of that. That said, I'll probably ditch my iPhone when my renewal comes up, it's simply not exciting enough in terms of its innovation any more. Better apps are being written elsewhere.

     [link]      Friday, Apr 16, 2010 @ 02:35pm

 Recent Wiki Changes

abc.php_00 on Thursday, Dec 16, 2010 @ 12:01am
abc.php on Wednesday, Dec 15, 2010 @ 11:59pm
wiki:recentchanges on Sunday, Aug 22, 2010 @ 07:17am
wiki:payloadrecoveryscript on Tuesday, Jun 29, 2010 @ 02:46pm
wiki:recipes on Tuesday, Mar 31, 2009 @ 07:58pm
phishingcorpus on Wednesday, Oct 01, 2008 @ 09:01pm
wiki:wild_blueberry_tart on Sunday, Sep 14, 2008 @ 03:43pm
wiki:cold_potato_salad on Thursday, Sep 04, 2008 @ 11:07am
wiki:recipes on Thursday, Sep 04, 2008 @ 11:06am
wiki:almond_cookies on Thursday, Sep 04, 2008 @ 11:06am