Hiding In Plain Sight Continued

Let’s look at exactly how to configure GoWitness to blend in with normal internet traffic. The settings we’ll use aren’t complicated, but each one serves a specific purpose in making our scanning look unremarkable.

The most important setting is the user agent string. GoWitness (version 3.0.5) lets you set this with the –user-agent flag. Here’s what a typical Censys.io crawler’s user agent looks like:

--chrome-user-agent "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)”

Another option is to use Googlebot’s user agent:

--chrome-user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Google has such a massive scanning infrastructure that it’s harder to verify whether a request really came from them. Plus, everyone expects to see Google crawling their site.

The next crucial setting is the delay between requests. GoWitness can make multiple requests simultaneously, but too many parallel connections is another red flag. The –threads flag controls this:

--threads 2

Two concurrent connections is usually safe. More than four may start to look suspicious.

Putting it all together, here’s the full command I use:

gowitness-3.0.5-windows-amd64.exe scan file -f test.txt --chrome-user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" --threads 2 --timeout 15

I added –timeout 15 at the end because real crawlers don’t wait forever for slow pages. Fifteen seconds seems reasonable than the default which is sixty seconds.

These settings won’t make you completely invisible. But they’ll make your scanning look like routine crawler behavior rather than targeted reconnaissance. The key is consistency; your timing and user agent strings should all support your cover story.

Remember that looking normal is more important than looking stealthy. Real web crawlers don’t try to hide. They operate in the open, following predictable patterns. The more closely you mimic these patterns, the less likely you are to attract attention.

The best test is to scan your own web server and look at the logs. Do the requests stand out? Do they match the patterns you see from legitimate crawlers? If you can’t tell your scan apart from normal crawler traffic, you’ve probably got the settings right.

Tomorrow, we’ll take a look at how to see what some of the more popular web crawlers look like.