Skip to content

Commit d615b95

Browse files
maroofimaroofi
committed
update
1 parent 601485f commit d615b95

File tree

4 files changed

+691
-221
lines changed

4 files changed

+691
-221
lines changed

‎README.md

+82-30
Original file line numberDiff line numberDiff line change
@@ -2,25 +2,33 @@
22

33
### bulkDNS: A fast DNS scanner for large-scale Internet measurement
44

5-
Using **bulkDNS**, you can scan millions of domain names in a few minutes. The scanner has been designed to be fast with a very small footprint.
5+
Using **bulkDNS**, you can scan millions of domain names in a few minutes. The scanner has been designed to be fast with a very small footprint. It also supports customized scan scenarios through Lua scripting.
66

77
The output of bulkDNS is a detailed JSON structure (example at the end of the page) which can be parsed both by command-line (e.g., by `jq`) or any programming language.
88

99
### Menu
1010

1111
* [How to compile bulkDNS](#How-to-compile)
12+
* [Compile without Lua](#Compile-without-Lua)
13+
* [Compile with Lua for customized scan scenarios](#Compile-with-Lua-for-customized-scan-scenarios)
1214
* [Supported Resource Records (RRs)](#Supported-Resource-Records)
1315
* [List of Switches](#List-of-Switches)
1416
* [Example Output](#Example-Output)
1517
* [Notes](#Notes)
16-
* [A note on threads](#A-note-on-threads)
18+
* [A note on threads and concurrency](#A-note-on-threads-and-concurrency)
1719
* [A note on names and conventions](#Names-and-output-convention)
1820
* [Hex representation of the output](#Hex-represantaion-of-the-output)
1921
* [FAQ](#FAQ)
2022

2123

2224
### How to compile
2325

26+
You have two options to compile bulkDNS. If You just want to use the scanner for scanning resource records like `A`, `AAAA`, `NS`, `MX`, etc, You can compile the scanner without Lua which is very easy. However, if you want to use `--server-mode` option or you have your own scan scenarios in mind that is more complicated than a simple resource record, then you must compile _bulkDNS_ with Lua support. Here is the instruction for both cases:
27+
28+
#### Compile without Lua
29+
30+
This is the first case (just scanning resource records)
31+
2432
You need to have `jansson` and `pthread` installed.
2533

2634
```bash
@@ -42,12 +50,49 @@ make
4250

4351
The compiled output is inside the `bin` directory.
4452

53+
#### Compile with Lua for customized scan scenarios
54+
55+
In this case, you need to have `lua5.4`, `pthread` and `jansson` installed. Here is the procedure:
56+
```bash
57+
# install lua binary
58+
sudo apt install lua5.4
59+
60+
# install lua lib
61+
sudo apt install liblua5.4-dev
62+
63+
# install the dependencies
64+
sudo apt install libpthread-stubs0-dev libjansson-dev
65+
66+
# after installing lua, if you run pkg-config like this:
67+
pkg-config --cflags --libs lua5.4
68+
# you should see an output like this:
69+
# -I/usr/include/lua5.4 -llua5.4
70+
71+
# Now clone the repository
72+
git clone --recurse-submodules https://github.com/maroofi/bulkDNS.git
73+
74+
# and make with the following commands
75+
cd bulkDNS
76+
make with-lua
77+
78+
```
79+
The compiled output is inside the `bin` directory.
80+
81+
In case the `pkg-config` commands gives you a different output for Lua headers and library locations, then
82+
you must specify the values in the make file like this:
83+
84+
```bash
85+
make LUALIB=<your-lua-lib-name> LUAINCDIR=<your-path-to-lua-include-dir> with-lua
86+
```
87+
88+
That's all!
89+
90+
4591
### Supported Resource Records
4692

4793
Currently, bulkDNS supports the following 17 RRs:
4894

49-
**A**, **AAAA**, **NS**, **RRSIG**, **SOA**, **MX**, **SRV**, **PTR**, **HINFO**, **TXT**, **CNAME**, **URI**,
50-
**NID**, **L32**, **L64**, **LP**, **CAA**
95+
**A**, **AAAA**, **NS**, **RRSIG**, **SOA**, **MX**, **SRV**, **PTR**, **HINFO**, **TXT**, **CNAME**, **URI**, **NID**, **L32**, **L64**, **LP**, **CAA**
5196

5297
It also supports adding **EDNS0** (**DNSSEC-OK** and **NSID**) to queries.
5398

@@ -70,27 +115,31 @@ All the RRs and EDNS0 are implemented based on the following RFCs (Some implemen
70115
[Help]
71116
72117
Summary:
73-
BulkDNS scanner based on sdns low-level DNS library.
74-
75-
./bulkdns [OPTIONS] <INPUT|FILE>
76-
--udp-only Only query using UDP connection (Default will follow TCP)
77-
--set-do Set DNSSEC OK (DO) bit in queries (default is no DO)
78-
--noedns Do not support EDNS0 in queries (Default supports EDNS0)
79-
--set-nsid The packet has NSID in edns0
80-
--threads=<param> How many threads should be used (it's pthreads, and default is 300)
81-
-t <param>, --type=<param> Resource Record type (A, AAAA, NS, etc). Default is 'A'
82-
-c <param>, --class=<param> RR Class (IN, CH). Default is 'IN'
83-
-r <param>, --resolver=<param> Resolver IP address to send the query to (default 1.1.1.1)
84-
-p <param>, --port=<param> Resolver port number to send the query to (default 53)
85-
-o <param>, --output=<param> Output file name (default is the terminal with stdout)
86-
-e <param>, --error=<param> where to write the error (default is terminal with stderr)
87-
-h , --help Print this help message
118+
Bulk DNS scanner based on sdns low-level DNS library.
88119
89-
We currently supports the following RR:
90-
A, AAAA, NS, RRSIG, SOA, MX, SRV, URI, PTR,
91-
HINFO, TXT, CNAME, NID, L32, L64, LP, CAA
92-
Supported DNS classes: IN, CH
120+
./bulkdns [OPTIONS] <INPUT|FILE>
93121
122+
-t <param>, --type=<param> Resource Record type (Default is 'A')
123+
-c <param>, --class=<param> RR Class (IN, CH). Default is 'IN'
124+
-r <param>, --resolver=<param> Resolver IP address to send the query to (default 1.1.1.1)
125+
-p <param>, --port=<param> Resolver port number to send the query to (default 53)
126+
-e <param>, --error=<param> where to write the error (default is terminal with stderr)
127+
-o <param>, --output=<param> Output file name (default is the terminal with stdout)
128+
--lua-script=<param> Lua script to be used either for scan or server mode
129+
--bind-ip=<param> IP address to bind (default 127.0.0.1 for scan mode, 0.0.0.0 for server-mode)
130+
--timeout=<param> Timeout of the socket (default is 5 seconds)
131+
--concurrency=<param> How many concurrent requests should we send (default is 1000)
132+
--udp-only Only query using UDP connection (Default will follow TCP)
133+
--set-do Set DNSSEC OK (DO) bit in queries (default is no DO)
134+
--set-nsid The packet has NSID in edns0
135+
--noedns Do not support EDNS0 in queries (Default supports EDNS0)
136+
--server-mode Run bulkDNS in server mode
137+
-h, --help Print this help message
138+
139+
bulkDNS currently supports the following RRs:
140+
A, AAAA, NS, RRSIG, SOA, MX, SRV, URI, PTR,
141+
HINFO, TXT, CNAME, NID, L32, L64, LP, CAA
142+
Supported DNS classes: IN, CH
94143
```
95144

96145
### Example Output
@@ -148,20 +197,21 @@ We try to keep the output as close as possible to DNS RFC standards.
148197

149198
### Notes
150199

151-
#### A note on threads
200+
#### A note on threads and concurrency
152201

153202
* bulkDNS is capable of scanning 1,000,000 (1M) domain names in around 5 minutes with less than 1% of errors using default number of threads (300). That means you can
154203
scan the whole domain name system in less than one day.
155204
This makes it probably the most practical (and maybe fastest) DNS scanner. It does not have any requirements in terms of CPU or RAM. As all other network scanners,
156205
the bottleneck is always the network bandwidth, firewalls and the remote recursive resolver. We recommend using Cloudflare quad one (1.1.1.1) as the resolver since
157206
it has no limit in terms of the number of queries. However, you can also run your own recursive resolver to do the job. If you decrease the number of threads, you can
158-
also use google quad eight (8.8.8.8) which has 1,500 queries/second limit.
207+
also use google quad-eight (8.8.8.8) which has 1,500 queries/second limit.
208+
209+
* Using `--concurrency` option, you can increase or decrease the number of concurrent requests based on your network and your experience. It's important to note that if you set `--concurrency=1000`, it means you ask for openning 1,000
210+
sockets (which means binding to 1,000 ports) at the same time.
159211

160-
* Using `--threads` option, you can increase or decrease the number of threads based on your network and your experience. However, at some point, more threads will probably
161-
make the scanner even slower (threads competing each other to acquire the lock).
212+
* If you are running the scanner on Linux, the maximum number of open files is 1024 by default. So if you plan to set
213+
the `--concurrency` to a value greater than 1000, then you need to increse the limit of open files using `ulimit -n` commands.
162214

163-
* If you are running the scanner on Linux, the maximum number of open files is 1024 by default which is three times more than the default number of threads in bulkDNS.
164-
However, if you plan to run bulkDNS with more threads, you may want to increase the number of open files using the `ulimit -n` commands.
165215

166216

167217
#### Names and output convention
@@ -211,10 +261,12 @@ In the above example the `cpu` is the hex represantation of `some-kinda-cpu` and
211261

212262
### FAQ
213263
1. Why another scanner?
214-
- Because I feel like it
264+
- Because It's fun!
215265
1. Why not using CMake in the project?
216266
- I don't know CMake
217267
2. Is there any similar project like this?
218268
- The only comparable project to this one (that I'm aware of) is zmap/zdns.
219269
3. Can I pass a domain name (e.g., `ns1.google.com` as the resolver)?
220270
- No. The resolver must be an IPv4 address. We pass this value to `inet_addr()` function which accepts an IPv4.
271+
4. How fast it can scan domain names?
272+
- It highly depends on your network and the (remote) resolver you use.

‎include/scanner.h

+28-6
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,8 @@
44
#include <stdlib.h>
55
#include <netinet/in.h>
66
#include <pthread.h>
7-
7+
#include <poll.h>
8+
#include <signal.h>
89
#include <cmdparser.h>
910
#include <cqueue.h>
1011

@@ -14,6 +15,8 @@
1415

1516
#define BULKDNS_MAX_QUEUE_SIZE 1000000
1617

18+
#define BULKDNS_MAX_SOCKET_FOR_POLL 32
19+
1720

1821
struct scanner_input {
1922
int udp_only; // should we send only udp queries?
@@ -31,7 +34,7 @@ struct scanner_input {
3134
FILE * OUTPUT; // output file handle
3235
FILE * ERROR; // error file handle
3336
FILE * INPUT; // input file handle
34-
unsigned int threads; // number of threads to perform the scan
37+
unsigned int concurrency; // number of concurrent requests (This is the number of open sockets/ports)
3538
unsigned int server_mode; // should we work in server mode instead of active scan
3639
char * lua_file; // Lua file to use either in server mode or custom scan
3740
char * bind_ip; // this is the IP address we want to bind to in server-mode
@@ -42,9 +45,21 @@ struct thread_param {
4245
struct scanner_input * si;
4346
pthread_mutex_t lock;
4447
cqueue_ctx * qinput;
45-
cqueue_ctx * qoutput;
48+
cqueue_ctx * queue_tcp;
4649
};
4750

51+
typedef struct{
52+
int * sock_list;
53+
int num_sock;
54+
struct thread_param * tp;
55+
}scan_mode_receiver_param;
56+
57+
58+
typedef struct {
59+
void * item;
60+
int udp_sock;
61+
struct sockaddr_in server;
62+
}scan_mode_worker_item;
4863

4964

5065
// server-mode structure definition
@@ -78,17 +93,24 @@ typedef struct{
7893

7994

8095
//server-mode function declaration
81-
96+
void handle_read_socket(int sockfd, char * mem_result, struct thread_param * tp);
97+
int udp_socket_send(char * tosend_buffer, size_t tosend_len, int sockfd, struct sockaddr_in server);
8298
void server_mode_to_log(const char * msg, FILE* fd);
8399
void server_mode_run_all(server_mode_server_param *smsp);
84100
void switch_server_mode(struct scanner_input *);
85101

86102

87103
// scan mode function declaration
104+
void * tcp_routine_handler(void * ptr);
105+
void * scan_receiver_routine(void * ptr);
106+
void * read_item_from_queue(struct thread_param * tp);
88107

108+
int init_udp_socket(struct scanner_input * si);
109+
#ifdef COMPILE_WITH_LUA
89110
void * scan_lua_worker_routine(void * ptr);
90-
void dns_routine_scan(void*, struct scanner_input * si, char * mem_result);
91-
int perform_lookup_udp(char * tosend_buffer, size_t tosend_len, char ** toreceive_buffer, size_t * toreceive_len, struct scanner_input * si);
111+
#endif
112+
void dns_routine_scan(scan_mode_worker_item*, struct scanner_input * si, char * mem_result);
113+
int perform_lookup_udp(char * tosend_buffer, size_t tosend_len, char ** toreceive_buffer, size_t * toreceive_len, struct scanner_input * si, int sockfd);
92114
int perform_lookup_tcp(char * tosend_buffer, size_t tosend_len, char ** toreceive_buffer, size_t * toreceive_len, struct scanner_input * si);
93115
void *scan_worker_routine(void * ptr);
94116
int convert_type_to_int(char * type);

‎src/cmdparser.c

+53-21
Original file line numberDiff line numberDiff line change
@@ -294,34 +294,66 @@ void arg_free(ARG_PARSED_ARGS * parsed){
294294

295295
void arg_show_help(PARG_CMDLINE cmd, int argc, char ** argv){
296296
fprintf(stdout, "[Help]\n\n");
297-
PARG_CMD_OPTION tmp = cmd->cmd_option;
298-
const char * has_param = "<param>";
299-
const char * no_param = "";
300-
const char * check_param;
301-
char equal = '=';
302297
fprintf(stdout, "Summary:\n");
303298
fprintf(stdout, "%s", cmd->summary);
304299
fprintf(stdout, "\n\n");
305300
fprintf(stdout, "%s [OPTIONS] ", argv[0]);
306301
fprintf(stdout, "%s\t", cmd->accept_file?"<INPUT|FILE>":"");
307-
fprintf(stdout, "\n");
308-
tmp = cmd->cmd_option;
309-
while (tmp->tag != NULL){
310-
check_param = tmp->has_param == 1?has_param:no_param;
311-
equal = tmp->has_param == 1?'=':' ';
312-
char shrt = tmp->short_option == 0?' ':tmp->short_option;
313-
char dash = tmp->short_option == 0?' ':'-';
314-
int has_shrt = tmp->short_option == 0?0:1;
315-
char ddash[3] = {'-', '-', 0};
316-
int put_ddash = strcmp(tmp->long_option, "") == 0?0:1;
317-
char virgul = has_shrt && put_ddash?',':' ';
318-
char * pddash = put_ddash?ddash:"\t";
319-
fprintf(stdout, "\t%c%c %s%c %s%s%c%s\t%s\n",dash, shrt, has_shrt?check_param:"", virgul,pddash, tmp->long_option, equal, check_param, tmp->help);
320-
tmp++;
321-
}
322-
fprintf(stdout, "\nWe currently supports the following RR:\n");
302+
fprintf(stdout, "\n\n");
303+
304+
fprintf(stdout, "\t-t <param>, --type=<param>\t\t");
305+
fprintf(stdout, "Resource Record type (Default is 'A')\n");
306+
307+
fprintf(stdout, "\t-c <param>, --class=<param>\t\t");
308+
fprintf(stdout, "RR Class (IN, CH). Default is 'IN'\n");
309+
310+
fprintf(stdout, "\t-r <param>, --resolver=<param>\t\t");
311+
fprintf(stdout, "Resolver IP address to send the query to (default 1.1.1.1)\n");
312+
313+
fprintf(stdout, "\t-p <param>, --port=<param>\t\t");
314+
fprintf(stdout, "Resolver port number to send the query to (default 53)\n");
315+
316+
fprintf(stdout, "\t-e <param>, --error=<param>\t\t");
317+
fprintf(stdout, "where to write the error (default is terminal with stderr)\n");
318+
319+
fprintf(stdout, "\t-o <param>, --output=<param>\t\t");
320+
fprintf(stdout, "Output file name (default is the terminal with stdout)\n");
321+
322+
fprintf(stdout, "\t--lua-script=<param>\t\t\t");
323+
fprintf(stdout, "Lua script to be used either for scan or server mode\n");
324+
325+
fprintf(stdout, "\t--bind-ip=<param>\t\t\t");
326+
fprintf(stdout, "IP address to bind (default 127.0.0.1 for scan mode, 0.0.0.0 for server-mode)\n");
327+
328+
fprintf(stdout, "\t--timeout=<param>\t\t\t");
329+
fprintf(stdout, "Timeout of the socket (default is 5 seconds)\n");
330+
331+
fprintf(stdout, "\t--concurrency=<param>\t\t\t");
332+
fprintf(stdout, "How many concurrent requests should we send (default is 1000)\n");
333+
334+
fprintf(stdout, "\t--udp-only\t\t\t\t");
335+
fprintf(stdout, "Only query using UDP connection (Default will follow TCP)\n");
336+
337+
fprintf(stdout, "\t--set-do\t\t\t\t");
338+
fprintf(stdout, "Set DNSSEC OK (DO) bit in queries (default is no DO)\n");
339+
340+
fprintf(stdout, "\t--set-nsid\t\t\t\t");
341+
fprintf(stdout, "The packet has NSID in edns0\n");
342+
343+
fprintf(stdout, "\t--noedns\t\t\t\t");
344+
fprintf(stdout, "Do not support EDNS0 in queries (Default supports EDNS0)\n");
345+
346+
fprintf(stdout, "\t--server-mode\t\t\t\t");
347+
fprintf(stdout, "Run bulkDNS in server mode\n");
348+
349+
fprintf(stdout, "\t-h, --help\t\t\t\t");
350+
fprintf(stdout, "Print this help message\n");
351+
352+
fprintf(stdout, "\nbulkDNS currently supports the following RRs:\n");
323353
fprintf(stdout, "\tA, AAAA, NS, RRSIG, SOA, MX, SRV, URI, PTR,\n");
324354
fprintf(stdout, "\tHINFO, TXT, CNAME, NID, L32, L64, LP, CAA\n");
325355
fprintf(stdout, "Supported DNS classes: IN, CH\n\n\n");
356+
fprintf(stdout, "Please report bugs to:\n");
357+
fprintf(stdout, "\thttps://github.com/maroofi/bulkDNS\n\n");
326358
}
327359

0 commit comments

Comments
 (0)