Birb - Faster Web Discovery
Penetration testing on web applications often involves some scanning for API endpoints, directories and other such elements.
This kind of brute-force enumeration can become quite time-consuming for some cases. Similarly, the diversity of server implementations and framework/application behavior makes it particularly difficult to have a "universal" scanner - some manual tuning is almost always required to achieve the best possible results.
The Birb approach
Birb is a very fast web discovery tool.
Written in C, it uses parallel processing to go through a wordlist as fast as possible, consistently hitting the highest possible request rates allowed by the server.
In other words, Birb requires very little resources from the user's system, and is only limited by the network capabilities and processing power of the target.
How it works
To achieve this, Birb forks into as many processes as requested by the user, splitting the wordlist into chunks for each process. Scanner processes therefore have close to zero adherence between each other, thus maximizing parallelism.
Thanks to this, Birb can easily run with more than 100 simultaneous processes on modern medium-range laptops.
Birb also makes use of HTTP pipelining to send as many requests as defined by the user in bulk, before the responses even start coming in. Some servers accept only one request at a time, requiring the user to limit Birb through command-line arguments, but more modern/powerful servers may allow upwards of 300 requests at a time (the default value for Birb).
Finally, Birb doesn't rely on any pre-made HTTP library - instead featuring its own HTTP implementation, optimized for itself.
The latest version of Birb can be downloaded here: birb-1.13.0
It requires Linux or WSL (Windows Subsystem for Linux), and pretty much nothing else.
Birb is run with a set of arguments to control various aspects such as number of processes, number of bulk requests to pipeline, scan filters, etc...
Keys can then be used at runtime (during the scan) to achieve various effects.
Any time a directory is encountered, Birb will descend into it and re-scan the wordlist within it. At any point, the "p" key can be used to ignore the current directory and step back up one level, while the "x" key will step back all the way to the root directory. Alternatively, the "e" key will cancel the scan session and exit.
In its most common form, Birb is usually run with only a wordlist and a target:
$ birb path/to/wordlist.txt https://target.example/
Performance can often be improved by tweaking the "-b/--bulk" and "-p/--procs" command-line parameters.
$ birb -b 10 -p 10 path/to/wordlist.txt https://target.example/
Whenever we have some false positives (uninteresting results), we can use Birb's filtering engine which offers two types of filters:
Content-Length filtering (-l SIZE) can be used to ignore any responses which feature a body of exactly SIZE bytes.
Regex filtering (-r REGEX) can be used to filter out any response which matches the REGEX regular expression (PCRE format). The expression is applied to the entire response, including HTTP status code, headers and body.
Multiple filters can be combined together for best results.
$ birb -r 'HTTP/1.. 502 ' -r 'does not exist' -l 218 path/to/wordlist.txt https://target.example/
Below is the complete help, which can be obtained from the "birb --help" command.
$ birb --help Usage: birb [OPTION...] WORDLIST URL Birb - Web Discovery Tool Birb uses a wordlist to interact with HTTP servers. Two modes of operation are available: * URL discovery (Normal mode) - Words are used to construct URLs, allowing Birb to discover directories and endpoints within websites. * Body-fuzzing - Words are used to alter the request body, while the URL never changes. To use body-fuzzing, specify a payload containing the fuzz marker ([birb:fuzz]) using the '-d' argument. The following keys can be used at runtime: [p] Step back one directory level [x] Step back to root directory [e] Abort and exit -b, --bulk=REQS Send out REQS requests before even getting a response (default: 300) -c, --cert=CERT_FILE Use CERT_FILE as client certificate (PKCS12) -d, --body=DATA Add an HTTP body to the request (And enable body-fuzzing if '[birb:fuzz]' is present) -e, --ext=EXTENSION Add EXTENSION to list of extensions to be tested for every word -f, --path=REGEX Skip any path that matches REGEX -h, --header=HEADER Set HTTP header HEADER in request -i, --skip=WORD Skip a word from the wordlist -l, --clen=LENGTH Filter out responses where HTTP content-length is LENGTH -m, --method=METH HTTP verb to use (default: GET) -n, --noenc Don't URL-encode words -o, --log=LOG_FILE Save log output to LOG_FILE -p, --procs=PROCS Parallelize scanning using PROCS processes (default: 1) -r, --regex=REGEX Filter out responses where REGEX matches -s, --suffix=SUFFIX Append SUFFIX after every word -t, --retry=COUNT Try COUNT times before giving up on a test (Default: 3) -v, --verbose Produce verbose output -w, --delay=SECS Wait SECS seconds before each request -x, --dump=DIR Dump interesting responses to files in DIR -z, --cpwd=CERT_PASSWORD Use CERT_PASSWORD as the password for the client certificate (-c) -?, --help Give this help list --usage Give a short usage message -V, --version Print program version Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options. Report bugs to <firstname.lastname@example.org>.