We browse only 5% of the internet. Here’s what happens on the other 95%

Photo credit: Ministerio TIC Colombia.

There’s a reason why the dark web is seen as the seedy underbelly of the internet.

It’s rife with criminal activity – including child pornography, illegal drugs, and contract hitmen. Trying to get an accurate picture of what all takes place there is an insurmountable task as the system is designed to be anonymous and encrypted.

The dark web is a subgroup of the deep, or hidden web, explains Avi Kasztan, CEO of SixGill. The hidden web is the part of the internet that’s not accessible by search engines. Avi claims it makes up “95 to 96 percent” of the total internet landscape.

The Tor network was originally developed by the US navy.

Most of the hidden web is fairly innocuous. It includes things like databases, pages behind paywalls, and registration forms. But dig a little deeper and you’ll start stumbling across the grisly stuff.

“People [can] use the hidden web to do bad things,” outlines Avi. “But the dark web is a completely different story. It’s much, much more chilling. It’s been made anonymous and encrypted on purpose.”

Avi knows what he’s talking about. His startup, Sixgill, crawls the dark web’s social networks (yes, you read that right) to detect and defuse threats before they’re carried out. The startup’s team comprises of former intelligence officials and cyber security experts. It recently pocketed US$5 million to scale operations.

Avi Kasztan, CEO of Sixgill.

Dark origins

The most popular way to access the dark web is through the anonymous browser Tor, developed by the US navy to facilitate secure communications. Avi says the project had “very good intentions” as it tried to enable people to communicate anonymously in places where there was no freedom of speech.

In 2004, the navy released the code for Tor under a free, public license and scientists Roger Dingledine and Nick Mathewson were given funding to continue its development. In 2006, both Roger and Nick, along with five others, founded The Tor Project, a research-education non profit aimed at helping democracy advocates in authoritarian states.

The original developers and organizations like the Electronic Frontier Foundation maintain that Tor is principally used by journalists, activists, and whistleblowers. However, the Tor network also started becoming associated with criminal activity. It’s unclear when exactly that happened. What’s sure is that cryptocurrencies like Bitcoin have helped the underground economy to flourish.

To provide some context, a study by researchers at Carnegie Mellon indicates the size of the underground economy in the dark web to be in the region of US$100 million – with the overwhelming majority of transactions conducted in Bitcoin.

Penetrating the network

Scary, but true