Released: pathod 0.3

I've just released pathod 0.3, which beefs up pathoc's fuzzing capabilities, improves the spec language and includes lots of bugfixes and other small tweaks. Get it while it's hot!

Better fuzzing

A major focus of this release is to improve pathoc's capabilities as a basic fuzzing tool. I've had fun breaking webservers with pathoc, and it's even come in handy in my Day Job. Here's a quick summary of how things have changed.

  • The -x flag tells pathoc to explain its requests. This prints out an expanded pathoc query specification, with all randomly generated content and query modifications resolved. If you trigger an exception, you can precisely replay the offending query using this explanation.
  • The options for outputting requests and responses have been expanded hugely. First, the -q and -r flags tell pathoc to dump complete records of requests and responses respectively. This data is sniffed by instrumenting the socket, so is canonical regardless of our ability to interpret returned data. The -x option makes pathod dump this data in hexdump format (otherwise unprintable characters are escaped).
  • A number of options have been added to let you ignore expected responses. -C takes a comma-separated list of response codes to ignore. -T ignores server timeouts. This lets you hone in on the exceptional responses that you care about, and ignore the lrest.

Language improvements

  • I've simplified response specifications by making the response message specification a standard component with the "r" mnemonic.
  • I've added the "u" mnemonic to request specifications, as a shortcut for specifying the User-Agent header:
get:/:u"My Weird User-Agent"

We also have a small library of representative User-Agent strings that can be used instead of specifying your own. For example, this specifies the GoogleBot User-Agent string:

get:/:ug

The list of available shortcuts are in the docs, and can be listed from the commandline using the --show-uas flag to pathoc:

> ./pathoc --show-uas
User agent strings:
   a android
   l blackberry
   b bingbot
   c chrome
   f firefox
   g googlebot
   i ie9
   p ipad
   h iphone
   s safari

A few months ago, I announced pathod, a pathological HTTP daemon. The project started as a testing tool to let me craft standards-violating HTTP responses while working on mitmproxy. It soon became a free-standing project, and has turned out to be incredibly useful in security testing, exploit delivery and general creative mischief. In the last release, I added pathoc - pathod's malicious client-side twin. It does for HTTP requests what pathod does for HTTP responses, and uses the same hyper-terse specification language.

In this post, I show how pathoc can be used as a very simple fuzzer, by finding issues in a number of major pure-Python webservers. None of the tested servers failed catastrophically - they all caught the unexpected exception and continued serving requests. None the less, I think it's reasonable to say that we've triggered a bug if a) the server returns an 500 Internal Server Error response or terminates the connection abnormally, and b) we see a traceback in our logs. In fact, by this definition, I found bugs in every pure-Python server I tested.

All of the problems I list below are simple failures of validation - what they have in common is that somewhere in the project code is called with input that it doesn't expect and can't handle. This matters - in fact, I'd argue that the majority of security problems fall in this category. It's interesting to ponder why this type of issue is so ubiquitous in Python servers. I have no doubt that part the answer lies in Python's use of exceptions - errors that would be explicit in other languages can be implicit in Python, and code that seems clean and intuitive might in fact be buggy. I think this is especially relevant right now, given the recent flurry of discussion surrounding the Go language and its error handling. It's pretty instructive to read Russ Cox's recent riposte to this post criticizing Go's explicit approach, while looking at the bugs below. I love Python and I think it's a fine language, but I also think the designers of Go probably made the right choice.

Basic fuzzing with pathoc

My methodology for these tests was very simple indeed. I launched each server in turn, and used pathod to fire corrupted GET requests at the daemon until I saw an error. I then looked at the logs, and boiled the distinct cases down to a minimal pathoc specification by hand. This exercises a rather shallow set of features in the server software - mostly parsing of the HTTP lead-in and request headers. It's possible to give software a much, much deeper workout with pathoc, but I'll leave that for a future post.

My pathoc fuzzing command looked something like this:

pathoc -n 1000 -p 8080 -t 1 localhost 'get:/:b@10:ir,"\x00"'

The most important flags here are -n, which tells pathoc to make 1000 consecutive requests, and -t, which tells pathoc to time out after one second (necessary to prevent hangs when daemons terminate improperly). The request specification itself breaks down as follows:

get Issue a GET request
/ ... to the path /
b@10 ... with a body consisting of 10 random bytes
ir,"\x00" ... and inject a NULL byte at a random location.

It's that last clause - the random injection - that makes the difference between simply crafting requests and basic fuzzing. Every time a new request is issued, the injection occurs at a different location. I varied the injected character between a NULL byte, a carriage return and a random alphabet letter. Each exposed different errors in different servers. For a complete description of the specification language, see the online docs.

Results

For each bug, I've given a traceback and a minimal pathoc call to trigger the issue. The tracebacks have been edited lightly to shorten file paths and remove irrelevances like timestamps.

pathoc -p 8080 localhost 'get:/:b@10:h"Content-Length"="x"'
ENGINE ValueError("invalid literal for int() with base 10: 'x'",)
Traceback (most recent call last):
  File "cherrypy/wsgiserver/wsgiserver2.py", line 1292, in communicate
    req.parse_request()
  File "cherrypy/wsgiserver/wsgiserver2.py", line 591, in parse_request
    success = self.read_request_headers()
  File "cherrypy/wsgiserver/wsgiserver2.py", line 711, in read_request_headers
    if mrbs and int(self.inheaders.get("Content-Length", 0)) > mrbs:
ValueError: invalid literal for int() with base 10: 'x'
pathoc -p 8080 localhost 'get:/:i4,"\r"
ENGINE TypeError("argument of type 'NoneType' is not iterable",)
Traceback (most recent call last):
  File "cherrypy/wsgiserver/wsgiserver2.py", line 1292, in communicate
    req.parse_request()
  File "cherrypy/wsgiserver/wsgiserver2.py", line 580, in parse_request
    success = self.read_request_line()
  File "cherrypy/wsgiserver/wsgiserver2.py", line 644, in read_request_line
    if NUMBER_SIGN in path:
TypeError: argument of type 'NoneType' is not iterable
pathoc -p 8080 localhost 'get:/:b@10:h"Content-Length"="x"'
[E 120927 11:42:26 iostream:307] Uncaught exception, closing connection.
    Traceback (most recent call last):
      File "tornado/iostream.py", line 304, in wrapper
        callback(*args)
      File "tornado/httpserver.py", line 254, in _on_headers
        content_length = int(content_length)
    ValueError: invalid literal for int() with base 10: 'x'
[E 120927 11:42:26 ioloop:435] Exception in callback 
    Traceback (most recent call last):
      File "tornado/ioloop.py", line 421, in _run_callback
        callback()
      File "tornado/iostream.py", line 304, in wrapper
        callback(*args)
      File "tornado/httpserver.py", line 254, in _on_headers
        content_length = int(content_length)
    ValueError: invalid literal for int() with base 10: 'x'
pathoc -p 8080 localhost 'get:/:h"h\r\n"="x"'
[E iostream:307] Uncaught exception, closing connection.
    Traceback (most recent call last):
      File "tornado/iostream.py", line 304, in wrapper
        callback(*args)
      File "tornado/httpserver.py", line 236, in _on_headers
        headers = httputil.HTTPHeaders.parse(data[eol:])
      File "tornado/httputil.py", line 127, in parse
        h.parse_line(line)
      File "tornado/httputil.py", line 113, in parse_line
        name, value = line.split(":", 1)
    ValueError: need more than 1 value to unpack
[E ioloop:435] Exception in callback 
    Traceback (most recent call last):
      File "tornado/ioloop.py", line 421, in _run_callback
        callback()
      File "tornado/iostream.py", line 304, in wrapper
        callback(*args)
      File "tornado/httpserver.py", line 236, in _on_headers
        headers = httputil.HTTPHeaders.parse(data[eol:])
      File "tornado/httputil.py", line 127, in parse
        h.parse_line(line)
      File "tornado/httputil.py", line 113, in parse_line
        name, value = line.split(":", 1)
    ValueError: need more than 1 value to unpack
pathoc -p 8080 localhost 'get:/:b@10:h"Content-Length"="x"'
[HTTPChannel,4,127.0.0.1] Unhandled Error
    Traceback (most recent call last):
      File "twisted/python/log.py", line 84, in callWithLogger
        return callWithContext({"system": lp}, func, *args, **kw)
      File "twisted/python/log.py", line 69, in callWithContext
        return context.call({ILogContext: newCtx}, func, *args, **kw)
      File "twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
    ---  ---
      File "twisted/internet/selectreactor.py", line 150, in _doReadOrWrite
        why = getattr(selectable, method)()
      File "twisted/internet/tcp.py", line 199, in doRead
        rval = self.protocol.dataReceived(data)
      File "twisted/protocols/basic.py", line 564, in dataReceived
        why = self.lineReceived(line)
      File "twisted/web/http.py", line 1558, in lineReceived
        self.headerReceived(self.__header)
      File "twisted/web/http.py", line 1580, in headerReceived
        self.length = int(data)
    exceptions.ValueError: invalid literal for int() with base 10: 'x'
pathoc -p 8080 localhost 'get:"/\0"'
Exception happened during processing of request from ('127.0.0.1', 54029)
Traceback (most recent call last):
  File "lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "lib/python2.7/SocketServer.py", line 638, in __init__
    self.handle()
  File "python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "lib/python2.7/SimpleHTTPServer.py", line 44, in do_GET
    f = self.send_head()
  File "lib/python2.7/SimpleHTTPServer.py", line 68, in send_head
    if os.path.isdir(path):
  File "lib/python2.7/genericpath.py", line 41, in isdir
    st = os.stat(s)
TypeError: must be encoded string without NULL bytes, not str
pathoc -p 8080 localhost 'get:/:i16," "'
ERROR:waitress:uncaptured python exception, closing channel 
 
(
    :list index out of range 
        [lib/python2.7/asyncore.py|read|83] 
        [lib/python2.7/asyncore.py|handle_read_event|444]
        [lib/python2.7/site-packages/waitress/channel.py|handle_read|169]
        [lib/python2.7/site-packages/waitress/channel.py|received|186]
        [lib/python2.7/site-packages/waitress/parser.py|received|99]
        [lib/python2.7/site-packages/waitress/parser.py|parse_header|158]
        [lib/python2.7/site-packages/waitress/parser.py|get_header_lines|247]
)
Edit: The first version of this post had examples that were due to the test WSGI application, not waitress. I've replaced them with the traceback above, which has been reformatted for clarity.
pathoc -p 8080 localhost 'get:/:h"Host"="n\r\0"'
Traceback (most recent call last):
  File "flask/app.py", line 1518, in __call__
    return self.wsgi_app(environ, start_response)
  File "flask/app.py", line 1507, in wsgi_app
    return response(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 1082, in __call__
    app_iter, status, headers = self.get_wsgi_response(environ)
  File "werkzeug/wrappers.py", line 1070, in get_wsgi_response
    headers = self.get_wsgi_headers(environ)
  File "werkzeug/wrappers.py", line 986, in get_wsgi_headers
    headers['Location'] = location
  File "werkzeug/datastructures.py", line 1132, in __setitem__
    self.set(key, value)
  File "werkzeug/datastructures.py", line 1097, in set
    self._validate_value(_value)
  File "werkzeug/datastructures.py", line 1065, in _validate_value
    raise ValueError('Detected newline in header value.  This is '
ValueError: Detected newline in header value.  This is a potential security problem

I recently wrote a series of posts using the Hilbert curve to visualize binaries, culminating in a gallery showing regions of high entropy in malware.

The fact that the Hilbert curve has excellent locality preservation means that one dimensional features are preserved (as much as they can be) in the two-dimensional layout. This lets us visually pick out features of interest, and makes it possible, for instance, to quickly identify different malware packers just based on their layout characteristics.

An obvious next step is to ask if it's possible to extend this idea to let us visually compare binaries, creating a sort of visual diff. Unfortunately, we now bump our heads against the limitations of space-filling curve visualization. I made the animation below after a recent conversation along these lines, and I think it illustrates the main issues nicely. It shows a single contiguous stretch of data (the black area) being shifted progressively through a binary. At each timestep, the only thing that changes is the starting location of the data block:

Two things are immediately clear:

  • The block of data doesn't retain its shape at different offsets - identical stretches of data can look totally different depending on their locations.
  • There's no way to quickly see where in the binary a piece of information lies. Unless you are very familiar with the particular curve and know its exact orientation, you can't say, for instance, when the data block lies a third of the way through the binary.

It's often worthwhile to trade off these things for locality preservation, but it definitely scotches certain use cases. I do wonder if it might be possible to tune the trade-off somewhat - sacrificing some locality preservation for better shape retention and offset estimation. I've toyed with some ideas along these lines (see the unrolled layouts in the binary visualization post), but I still don't have a satisfying solution. If anyone out there knows of one, drop me a line.

It's become quite a popular parlor game to guess who is responsible for the recent Antisec UDID leak. I've now seen no less than six separate apps named as the probable source (two of which came from Marco Arment). Before we pick the next culprit, I think it's worth taking a step back to consider the list of things we don't know:

  • We don't know that we're dealing with just one source. The Antisec dump may well be an amalgam of data from various sources.
  • We don't know that we're looking for just one app, or even a set of apps by one developer. The leak may well come from one of the myriad of 3rd party services which could be included in thousands of apps.
  • We don't know that Antisec is being truthful about the scale of the database, or the additional data they claim is associated with the UDID/APNS records.
  • We certainly don't know that the data was filched from an FBI laptop or that the NCFTA was in any way involved.

Given all of these unknowns, I think a simple process-of-elimination approach to tracking down the leak will probably be fruitless, or worse, result in the finger being pointed at even more innocent parties. The one entity that may already have the answer to this question is Apple. They have a list of a million affected UDIDs, and they presumably have records of all apps that have ever used the associated push tokens. Given a large and precise sample like this, it should be possible to find the origin(s) of the leak reasonably easily. Indeed, if Apple is on the ball they may already have done this.

Now for some frank speculation of my own. Let's assume for a moment that Antisec has been entirely truthful about the data, and that we're dealing with a single source. In that case, we're looking for:

  • ... an app or third-party service integrated into multiple apps
  • ... with 12 million or more users
  • ... that is APNS-enabled
  • ... which also gathers user data like real names and zip codes.

I'll throw my hat in the ring and say that my money is on a third-party service, not a single app. If my hunch is right, the list of possible culprits is actually rather short.

Something I've been worrying about for a long time has just happened: Antisec has leaked a database with more than a million UDIDs. The UDID issue has been a bit of a white whale of mine - I've written many blog posts about it and spent more hours than I care to think negotiating responsible disclosure with companies misusing UDIDs. Let's recap some of the posts I've written about this:

  • In May 2011, just before its sale to Gree was announced, I showed that OpenFeint was misusing UDIDs in a way that allowed you to link a UDID to a user's identity, geolocation and Facebook and Twitter accounts. I didn't discuss it openly at the time, you could also completely take over an OpenFeint account, and access chat, forums, friends lists, and more using just a UDID. This resulted in a class-action lawsuit against OpenFeint, which has since petered out.

  • Later that month, I published a survey looking at how UDIDs are used in practice. The data is now slightly out of date, but shows just how widely UDIDs are used and misused.

  • In September 2011, I published the most troubling news so far, which paradoxically also got the least coverage in the press. I looked at all the gaming social networks on IOS - basically OpenFeint and its competitors - and found catastrophic mismanagement by nearly everyone. The vulnerabilities ranged from de-anonymization, to takeover of the user's gaming social network account, to the ability to completely take over the user's Facebook and Twitter accounts using just a UDID.

As serious these problems are, I'm afraid it's just the tip of the iceberg. Negotiating disclosure and trying to convince companies to fix their problems has taken literally months of my time, so I've stopped publishing on this issue for the moment. It's disheartening to say it, but some of the companies mentioned in my posts still have unfixed problems (they were all notified well in advance of any publication). I will also note ominously that I know of a number of similar vulnerabilities elsewhere in the IOS app ecosystem that I've just not had the time to pursue.

When speaking to people about this, I've often been asked "What's the worst that can happen?". My response was always that the worst case scenario would be if a large database of UDIDs leaked... and here we are.

Defiler

I've been living out of a bag for the last 3 weeks, working hard on a series of intense but fun audits. After running in high gear for a while I find that I need a mental palate cleanser - something to help me refocus and stop me from getting snowblind. I then grab my camera, strap on my macro rig, and walk out the door to try to catch the local wildlife in the act. It's become a bit of a game - the aim is to catch creatures in their natural setting and leave them completely undisturbed when I go, with no posing, prodding or other disturbances. Getting a usable shot of a 5mm target sitting on a twig swaying in the wind is a fun challenge.

Today I find myself in Sydney, working in a part of the town that is shot through with unreasonably beautiful walking tracks. The place is also blessed with a huge diversity of invertebrate life that makes my adopted home town seem barren by comparison. I walked along a nearby track until I found a quiet, leafy spot, geared up, and leopard-crawled through the underbrush. Not long after, I came face-to-face with this imposing little chap sitting on the tip of a fern frond.

This is a Lymantriid caterpillar of some variety, probably one of the tussock moths native to Australia. "Lymantria" means "defiler" - some species of this family can cause huge damage to foliage, and are considered to be destructive pests. So much so, that when a single male Gypsy Moth (Lymantria dispar) was discovered in Hamilton, New Zealand, they sprayed the entire city with a caterpillar-specific bacterial insecticide.

No need for drastic measures with this particular fellow, though - he's native to this ecosystem, and the only pest is me and my camera. He was head down munching away when I found him, and paid absolutely no attention to me when I moved in close to get these shots. He's got reason to be cocksure, too - those tufts of hair on his back contain hollow, poison-filled spines that can cause a pretty unpleasant reaction when touched.

An few hours exploring and photographing is a very effective brain-cleaner, leaving me ready to deal with spiny, venomous defilers of the digital variety.

I've just pushed pathod 0.2 out the door. This is a huge release, with many new features:

  • pathoc, pathod's evil client-side twin.
  • libpathod.test, a framework for using pathod in your unit tests.
  • Improved mini language, including many new abilities and improvements.
  • A rewrite of the networking core.

The project also has a new website at pathod.net. Yes, pathod is now self-hosting, so you can try out both pathod and pathoc specifications right on the website. There's also a new public pathod instance, which I'm sure everyone will use entirely responsibly.

get it here

I've just released pathod, a pathological HTTP/S daemon useful for testing and torturing HTTP clients. At its core is a tiny, terse language for crafting HTTP responses. It also has a built-in web interface that lets you play with the response spec language, inspect logs, and access pathod's full help document.

The rest of this post is a quick teaser showing some of pathod's abilities. See the detailed documentation on the pathod site if you want more.

The simplest possible response

The easiest way to craft a response is to specify it directly in the request URL. Lets start with the simplest possible example. Start pathod, and then visit this URL:

http://localhost:9999/p/200

The "/p/" path is the location of the response generator in pathod's default configuration - everything after that a response specification in pathod's mini-language. The general form of a response spec is as follows:

code[MESSAGE]:[colon-separated list of features]

In this case, we're specifying only the HTTP response code - that is, an HTTP 200 OK with no headers and no content, resulting in a response like this:

HTTP/1.1 200 OK

Specifying features

One example of a "feature" is a response header. Lets embellish our response by adding one:

200:h"Etag"="foo"

The first letter of the feature - "h", in this case - is a mnemonic indicating the type of feature we're adding. The full response to this spec looks like this:

HTTP/1.1 200 OK
Etag: foo

Both "Etag" and "foo" are Value Specifiers, a syntax used throughout the response specification language. In this case they are literal values, as indicated by the fact that they are quoted strings. The Value Specification syntax also lets us load values from files or generate random data. For instance, here is a specification that generates 100k of random binary data for the header value:

200:h"Etag"=@100k

Now, binary data in the header value will probably break things in interesting ways, but is unlikely to be read by the client as a valid (but over-long) value. To see if the client really drops off its perch if we feed it a single 100k header, we have to constrain the random data. Here's the same response, but with data generated only from ASCII letters:

200:h"Etag"=@100k,ascii_letters

pathod has a large number of built-in character classes from which random data can be generated.

Pauses and Disconnects

Next, we can disrupt the communications in various ways. At the moment, this means adding pauses and disconnects to a response. Let's start with an HTTP 404 response with a body consisting of a 100k of random binary data:

404:b@100k

Here's the same response, but with a 120 second pause after sending 100 bytes:

404:b@100k:p120,100

And, the same response again, but with hard disconnect after sending 100 bytes:

404:b@100k:d100

Instead of specifying a time explicitly, we can ask pathod to just randomly disconnect at a time of its choosing:

404:b@100k:dr

That's it for the teaser - hopefully it's enough to entice you into looking at pathod's full documentation.

What's next?

pathod is an "airport project" - the first draft was written in its entirety during a 40-hour trip back home from New York (I drew a bad lot in stopovers). I've now firmed it up a bit, but there's still work to be done. In the next month, mitmproxy's test suite will move to pathod, after which there will be a simple, well-documented way to unit test. I also plan to build out the JSON API (which is used to drive pathod in test suites), and expand the mini-language with convenient ways to generate pathological cookies, authentication headers, SSL errors, and cache control.