Digital Shadows firm develops a search engine for the Deep Web

Security firm Digital Shadows has realized a search engine for the Deep Web that according the company CEO “sucks in pages in real time”.

The deep web is the part of the world wide web not indexed by common search engines, it represents almost all of the content available on the Internet. Law enforcement, intelligence agencies and scientists are spending a great effort to try to index also this hidden part of the web. Recently the DARPA announced the Memex search engine, a program launched in the 2014 to design advanced search tools that could be also used to scan the deep web. Another project focused on the searchers in the deep web is Flashpoint, which is research tool focused on threats from the dark web, for example, every form of extremism.

The Memex  search engine was started to allow search of not indexed content, an operation that in the majority of cases is still run manually the by Intelligence Agencies.

Today we discuss about a similar project conducted by a British cyber security firm, Digital Shadows, that has developed a search engine to search for content in the Tor network..

“Basically, it’s a Google for Tor,” explained Alistair Paterson, CEO of Digital Shadows.

Mr Paterson explained that the search engine developed by his company is the most comprehensive to date.

Mr Paterson says the search engine “sucks in pages in real time and analyses them. We have some analysts direction. It’s helpful we have two Russian speakers as well. So the system is running all the time, but it’s directed by humans.” reported Tom Cheshire, Technology Correspondent for SkyNews.

Digital Shadows developed the Deep Web search engine to offer its services to private firms to help them identifying cyber threats or any other illegal activity that could represent a threat.

“In one instance, the system automatically found a bank employee who was offering customers’ log in details for online banking for £50 a pop. Digital Shadows took that information to the bank, which tracked down the employee.” added Cheshire.

It’s clear that the Deep Web represents just one of the possible sources that a threat intelligence firm can scan for its investigation. Cyber threats are becoming even more sophisticated and cyber criminals exploit any technology in order to monetize their efforts, that’s why threat intelligence is becoming a fundamental discipline to prevent the menaces.

Pierluigi Paganini

(Security Affairs –  Deep Web, Threat Intelligence)