Crawling the Dark Web to Detect the Next Market

Over the past few days, the internet has been abuzz with talk of the recent blows dealt by law enforcement to two major dark web “marketplaces”, AlphaBay and Hansa market; and the subsequent suicide of Alexander Cazes – the Canadian programmer-turned-criminal mastermind behind AlphaBay, who ended his own life in a Thai prison while awaiting extradition.

These latest developments have been dominating the conversation in cyber security and law enforcement circles, not to mention outraged users of these marketplaces on places such as Reddit. Since we also have more than a fleeting interest in dark web data, we’ve prepared this summary of what happened so far – and how crawling and indexing the dark web can detect the next permutation of these illicit marketplaces, which is bound to pop up sooner rather than later.

Not Your Father’s Marketplace

Let’s start at the very beginning – what are these dark web sites, and why have they garnered so much attention from law enforcement?
Simply put, these innocuous-sounding marketplaces are the places where the dark web conducts its business, connecting between buyers and sellers of products or services.

Unfortunately, said dealings are usually fairly sordid – with users taking advantage of the relative anonymity of the dark web and cryptocurrency such as Bitcoin to peddle wares that range from illicit substances to forged documents to alleged hitmen for hire (which turned out to be scams more often than not).

Shattered Dreams

However, law enforcement has not remained idle in the face of this unlawful activity, and has taken several (highly publicized) steps to curb it.
After the FBI shut down the Silk Road, which was the largest of these marketplaces until 2013, the focus shifted to AlphaBay – a bustling hub of illegal activity that hosted some $1 billion worth of transactions, mostly in drugs and stolen credit card details, and which was seized by authorities earlier this month.

This FBI takedown notice is all that remains of the once-thriving AlphaBay marketplace

Shortly thereafter it was announced that a similar market called Hansa, to which users reportedly flocked after AlphaBay was shut down, had been operated by Dutch police for the past month after its two German operators were arrested. The identity of many of those who engaged in illegal activities is known to law authorities, with arrests already being made and likely more to follow.

One of the largest remaining marketplaces, known as Dream Market, is still active – but already there is much paranoia among users that it too has been “compromised” and is being monitored by the authorities.

Powerful Tech and Old Fashioned Police Work

So the past month has definitely been a fruitful one for the authorities in the ongoing cat-and-mouse game between law enforcement and the lawless wasteland of the darknet marketplaces. Whether these victories have a strategic impact or remain tactical is yet to be seen, but from a cybersecurity perspective it’s clearly interesting to look at some of the methods used to identify and prevent illegal activity on the dark web.

From what has been published, it is clear that authorities used both technological means – to pinpoint the location of servers, cryptocurrency trails and encrypted messages – as well as traditional police work in their efforts to take down the two marketplaces.

These dual forces led Dutch investigators to the servers in Lithuania that hosted the Hansa Market, which they proceeded to duplicate and monitor transactions in real-time. The recklessness of AlphaBay boss Cazes – who left a trail that included a public Hotmail address, millions of unreported income in his bank account, and was logged in to the AlphaBay admin panel when he was arrested – gave FBI agents access to a breadth of illegal transactions and allowed them to trace the real identity of many of the people involved.

Shedding Light on the Dark Web

Finally, an important piece of the puzzle when it comes to monitoring the dark web and identifying illegal activity – either to prevent it from happening or as part of an investigation of a crime that has already occurred – is the ability to crawl, index and analyze data from the dark web in real-time.

What does this mean? The dark web is disparate, decentralized and populated by a highly suspicious user base. And while law enforcement has successfully managed to penetrate these barriers with the use of undercover agents and sting operations, illicit activity tends to reappear in new guises and shapes, requiring constant, automated surveillance of the dark web’s hidden recesses.

This is where crawling and scraping come in – identifying new onions in the dark web as they appear, indexing them and enabling cybersecurity professionals to algorithmically analyze this data and identify threats. Tools such as our own dark web crawler transforms darknet websites into machine-readable data, creating a data feed that allows cyber analysts to identify traces and patterns of illegal activity within existing marketplaces – and after these are shut down, to discover the new marketplaces that are growing in popularity and replacing them.

It is this combination of comprehensive coverage, advanced technological means, and human intelligence that produces the small and big victories against cybercrime on the dark web. And the increasingly sophisticated means used to hide the illegal activities that happen there. Webz.io gives cybersecurity companies the most comprehensive coverage of both the dark and visible web – along with granular filtering, automated source discovery and incredibly easy integration.

Ready to explore data at scale? Start your 10-day free trial today, or talk with a data expert to learn more about Webz’s solutions.