Birb - Faster Web Discovery

Web discovery?

Penetration testing on web applications often involves some scanning for API endpoints, directories and other such elements.

This kind of brute-force enumeration can become quite time-consuming for some cases. Similarly, the diversity of server implementations and framework/application behavior makes it particularly difficult to have a "universal" scanner - some manual tuning is almost always required to achieve the best possible results.

 

The Birb approach

Birb is a very fast web discovery tool.

Written in C, it uses parallel processing to go through a wordlist as fast as possible, consistently hitting the highest possible request rates allowed by the server.

In other words, Birb requires very little resources from the user's system, and is only limited by the network capabilities and processing power of the target.

 

How it works

To achieve this, Birb forks into as many processes as requested by the user, splitting the wordlist into chunks for each process. Scanner processes therefore have close to zero adherence between each other, thus maximizing parallelism.

Thanks to this, Birb can easily run with more than 100 simultaneous processes on modern medium-range laptops.

Birb also makes use of HTTP pipelining to send as many requests as defined by the user in bulk, before the responses even start coming in. Some servers accept only one request at a time, requiring the user to limit Birb through command-line arguments, but more modern/powerful servers may allow upwards of 300 requests at a time (the default value for Birb).

Finally, Birb doesn't rely on any pre-made HTTP library - instead featuring its own HTTP implementation, optimized for itself.

 

Get it

The latest version of Birb can be downloaded here: birb-1.13.0

It requires Linux or WSL (Windows Subsystem for Linux), and pretty much nothing else.

 

 

Usage

Birb is run with a set of arguments to control various aspects such as number of processes, number of bulk requests to pipeline, scan filters, etc...

Keys can then be used at runtime (during the scan) to achieve various effects.

Any time a directory is encountered, Birb will descend into it and re-scan the wordlist within it. At any point, the "p" key can be used to ignore the current directory and step back up one level, while the "x" key will step back all the way to the root directory. Alternatively, the "e" key will cancel the scan session and exit.

Basic

In its most common form, Birb is usually run with only a wordlist and a target:

$ birb path/to/wordlist.txt https://target.example/

Tuning

Performance can often be improved by tweaking the "-b/--bulk" and "-p/--procs" command-line parameters.

$ birb -b 10 -p 10 path/to/wordlist.txt https://target.example/

Filtering

Whenever we have some false positives (uninteresting results), we can use Birb's filtering engine which offers two types of filters:

Content-Length filtering (-l SIZE) can be used to ignore any responses which feature a body of exactly SIZE bytes.

Regex filtering (-r REGEX) can be used to filter out any response which matches the REGEX regular expression (PCRE format). The expression is applied to the entire response, including HTTP status code, headers and body.

Multiple filters can be combined together for best results.

$ birb -r 'HTTP/1.. 502 ' -r 'does not exist' -l 218 path/to/wordlist.txt https://target.example/

Advanced

Below is the complete help, which can be obtained from the "birb --help" command.

$ birb --help
Usage: birb [OPTION...] WORDLIST URL
Birb - Web Discovery Tool

Birb uses a wordlist to interact with HTTP servers. Two modes of operation are
available:

 * URL discovery (Normal mode) - Words are used to construct URLs, allowing
Birb to discover directories and endpoints within websites.

 * Body-fuzzing - Words are used to alter the request body, while the URL never
changes. To use body-fuzzing, specify a payload containing the fuzz marker
([birb:fuzz]) using the '-d' argument.

The following keys can be used at runtime:
 [p] Step back one directory level
 [x] Step back to root directory
 [e] Abort and exit

  -b, --bulk=REQS            Send out REQS requests before even getting a
                             response (default: 300)
  -c, --cert=CERT_FILE       Use CERT_FILE as client certificate (PKCS12)
  -d, --body=DATA            Add an HTTP body to the request (And enable
                             body-fuzzing if '[birb:fuzz]' is present)
  -e, --ext=EXTENSION        Add EXTENSION to list of extensions to be tested
                             for every word
  -f, --path=REGEX           Skip any path that matches REGEX
  -h, --header=HEADER        Set HTTP header HEADER in request
  -i, --skip=WORD            Skip a word from the wordlist
  -l, --clen=LENGTH          Filter out responses where HTTP content-length is
                             LENGTH
  -m, --method=METH          HTTP verb to use (default: GET)
  -n, --noenc                Don't URL-encode words
  -o, --log=LOG_FILE         Save log output to LOG_FILE
  -p, --procs=PROCS          Parallelize scanning using PROCS processes
                             (default: 1)
  -r, --regex=REGEX          Filter out responses where REGEX matches
  -s, --suffix=SUFFIX        Append SUFFIX after every word
  -t, --retry=COUNT          Try COUNT times before giving up on a test
                             (Default: 3)
  -v, --verbose              Produce verbose output
  -w, --delay=SECS           Wait SECS seconds before each request
  -x, --dump=DIR             Dump interesting responses to files in DIR
  -z, --cpwd=CERT_PASSWORD   Use CERT_PASSWORD as the password for the client
                             certificate (-c)
  -?, --help                 Give this help list
      --usage                Give a short usage message
  -V, --version              Print program version

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

Report bugs to <eresse@dooba.io>.