The good and the bad of the Deep Web

Pierluigi Paganini September 17, 2012


Article Published on The Hacker New Magazine – September Edition “Security in a serious way”


The Deep Web (or Invisible web) is the set of information resources on the World Wide Web not reported by normal search engines, according a raw estimation of some security experts clear web represents only a small portion of the overall web content, the remaining part is unknown to the majority of web users.

Ordinary web users are literally shocked when understand the existence of the Deep Web, a network of interconnected systems, not indexed, having a size hundreds of times higher than the current web, around 500 times.

To explain the Deep Web I use to cite the definition provided by the founder of BrightPlanet, Mike Bergman, that compared searching on the Internet today to dragging a net across the surface of the ocean: a great deal may be caught in the net, but there is a wealth of information that is deep and therefore missed.


Who and why could be interested in the Deep Web? Is the Deep Web the reign of cybercrime? Is it legal surf in anonymity?

Professionals have several advantages to surf through Deep Web and the conviction that it represents a parallel world of illicit activities is profoundly wrong.

Let’s start with the consideration that illicit activities are daily arranged on clear web such as in the Deep Web, in many cases we have read of platforms used to spread and sell malware on the ordinary web and we all know that is quite simple to find any kind of objects, also illegal, on the clear web.

But what primarily distinguishes the clear web from the Deep Web? Of course, when we speak of hidden web we can think of a dark world characterized by the possibility to surf, under specific conditions, in total anonymity. This aspect makes very desirable the Deep Web for cyber criminals that in short time is moving all their activities in the dark world.

But consider also that the Deep Web is the privileged channel used by governments to exchange documents secretly, for journalists to bypass censorship of several states and also dissidents to avoid the control of authoritarian regimes… and these are just a few samples of not illicit use of the resources of the deep web.

How is possible that resources located on the web are not visible and which are the content of the hidden web?

Ordinary search engines use software called “crawlers” to find content on the web, they are computer programs that browse the World Wide Web in a methodical, automated manner and are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches.

This technique is ineffective in finding the hidden resources on the Web that could be classified into the following categories:

  • Dynamic content: dynamic pages which are returned in response to a submitted query or accessed only through a form, especially if open-domain input elements (such as text fields) are used; such fields are hard to navigate without domain knowledge.
  • Unlinked content: pages which are not linked to by other pages, which may prevent Web crawling programs from accessing the content. This content is referred to as pages without backlinks (or inlinks).
  • Private Web: sites that require registration and login (password-protected resources).
  • Contextual Web: pages with content varying for different access contexts (e.g., ranges of client IP addresses or previous navigation sequence).
  • Limited access content: sites that limit access to their pages in a technical way (e.g., using the Robots Exclusion Standard, CAPTCHAs, or no-cache Pragma HTTP headers which prohibit search engines from browsing them and creating cached copies).
  • Scripted content: pages that are only accessible through links produced by JavaScript as well as content dynamically downloaded from Web servers via Flash or Ajax solutions.
  • Non-HTML/text content: textual content encoded in multimedia (image or video) files or specific file formats not handled by search engines.
  • Text content using the Gopher protocol and files hosted on FTP that are not indexed by most search engines. Engines such as Google do not index pages outside of HTTP or HTTPS.

The  Tor Network, how to preserve the anonymity?

Tor is the acronym of “The onion router”, a system implemented to enable online anonymity as part of a project sponsored the US Naval Research Laboratory from 2004 to 2005 and successively supported by the Electronic Frontier Foundation.

Actually the software is under development and maintenance of Tor Project. A user that navigate using Tor it’s difficult to trace ensuring his privacy because the data are encrypted multiple times passing through nodes, Tor relays, of the network.

Tor client software routes Internet traffic through a worldwide volunteer network of servers hiding user’s information eluding any activities of monitoring.

How does Tor network work?

Imagine a typical scenario where Alice desire to be connected with Bob using the Tor network. Let’s see step by step how it is possible.

She makes an unencrypted connection to a centralized directory server containing the addresses of Tor nodes. After receiving the address list from the directory server the Tor client software will connect to a random node (the entry node), through an encrypted connection. The entry node would make an encrypted connection to a random second node which would in turn do the same to connect to a random third Tor node. The process goes on until it involves a node (exit node) connected to the destination.

Consider that during Tor routing, in each connection, the Tor node is randomly chosen and the same node cannot be used twice in the same path. To ensure anonymity the connections have a fixed duration. Every ten minutes to avoid statistical analysis that could compromise the user’s privacy, the client software changes the entry node

Up to now we have considered an ideal situation in which a user accesses the network only to connect to another. To further complicate the discussion, in a real scenario, the node Alice could in turn be used as a node for routing purposes with other established connections between other users.

A malevolent third-party would not be able to know which connection is initiated as a user and which as node making impossible the monitoring of the communications.

Is it legal to surf in anonymity? Right to anonymity – Legal implications

Every day, all our web actions leave traces of ourselves and of our way of life through the storing of massive amounts of personal data in databases on the internet, all these information composes our digital identity, our representation in the cyber space.

Users are “entities” in the cyberspace, built also with the correlation of data that increasingly escapes the control of the owner, anyone can theoretically “expropriate” of our digital identity.

Today tracking user activities on internet are one of the primary interests of private companies and Governments, business and political motivations are pushing for the development of monitoring and surveillance systems.

Anonymous communications have an important place in our political and social discourse, many individuals desire to hide their identities because they may be concerned about political or economic retribution harassment or even threats to their lives.

Anonymity is derived from the Greek word anonymia, meaning “without a name”, in the common usage the term refers to the state of an individual’s personal identity, or personally identifiable information, being publicly unknown.

In internet the anonymity is guaranteed when IP addresses cannot be tracked, due this reason it has been assisted in the creation of Anonymizing services such as I2P – The Anonymous Network or Tor address. The anonymizing services are based on the concept of distribution of routing information, during a transmission in fact is not known prior the path between source and destination and every node of the network manage minimal information to route the packets to the next hop without conserving history on the path, the introduction of encryption algorithms makes impossible the wiretapping of the information and the composition of the original messages.

The Supreme Court of the United States has ruled repeatedly that the right to anonymous free speech is protected by the First Amendment. A much-cited 1995 Supreme Court ruling in McIntyre v. Ohio Elections Commission reads:

Protections for anonymous speech are vital to democratic discourse. Allowing dissenters to shield their identities frees them to express critical minority views . . . Anonymity is a shield from the tyranny of the majority. . . . It thus exemplifies the purpose behind the Bill of Rights and of the First Amendment in particular: to protect unpopular individuals from retaliation . . . at the hand of an intolerant society.

Many institutions and foundations, such as The Electronic Frontier Foundation, are spending a great effort to protect the rights to on-line anonymity.  As one court observed in a case handled by EFF along with the ACLU of Washington:

“[T]he free exchange of ideas on the Internet is driven in large part by the ability of Internet users to communicate anonymously.”

US First Amendment settled that the right to speak anonymously, the Supreme Court has held,

“Anonymity is a shield from the tyranny of the majority,” that “exemplifies the purpose” of the First Amendment: “to protect unpopular individuals from retaliation…at the hand of an intolerant society.”

Court pronunciations establish the duty of government to guard against undue hindrances to political conversations and the exchange of ideas, a vigilant review that

“must be undertaken and analyzed on a case-by-case basis”.

US laws establish a right to Speak Anonymously on the Internet and also right to Read Anonymously on the Internet, ensuring the principle of free internet ideological confrontation and the right to free movement of information.

 “People are permitted to interact pseudonymously and anonymously with each other so long as those acts are not in violation of the law. This ability to speak one’s mind without the burden of the other party knowing all the facts about one’s identity can foster open communication and robust debate.”

The technological developments of recent years caused high attention to the legal and technological possibility to maintain the online anonymity, especially in the face of the multiplication of resources internet monitoring.

The right to internet anonymity is also covered by European legislation that recognizes the fundamental right to data protection, freedom of expression, freedom of impression. The European Union Charter of Fundamental Rights recognizes in Article. 8 (Title II: “Freedoms”) the right of everyone to protection of personal data concerning him.

The right to privacy is now essentially the individual’s right to have and to maintain control over information about him.

Sailing in the dark

After this necessary parenthesis on Tor network routing we are ready to enter the Deep Web simply using the Tor software from the official website of the project. Tor is able to work on all the existing platforms and many add-ons make simple they integration with existing applications, including web browsers. Despite the network has been projected to protect user’s privacy, to be really anonymous it’s suggested to go through a VPN.

A better mode to navigate inside the deep web is to use the Tails OS distribution which is bootable from any machine don’t leave a trace on the host. Once the Tor Bundle is installed it comes with its own portable Firefox version, ideal for anonymous navigation because an appropriate control of installed plugins, in the commercial version in fact common plugins could expose our identity.

Well once inside the deep web we must understand that the navigation is quite different from ordinary web, every research is more complex due the absence of indexing of the content.

A user that start its navigation in the Deep Web have to know that a common way to list the content is to adopt collection of wiki and BBS-like sites which have the main purpose to aggregate links categorizing them in more suitable groups of consulting. Another difference that user has to take in mind is that instead of classic extensions (e.g. .com, .gov) the domains in the Deep Web generally end with the .onion suffix. Following a short list of links that have made famous the Deep Web published on Pastebin

Cleaned Hidden Wiki should be a also a good starting point for the first navigations


Be careful, some content are labeled with common used tag such as CP= child porn, PD is pedophile, stay far from them.

The Deep Web is considered the place where everything is possible, you can find every kind of material and services for sale, most of them illegal. The hidden web offers to cybercrime great business opportunity, hacking services, malware, stolen credit cards, weapons.

We all know the potentiality of the e-commerce in ordinary web and its impressive growth in last couple of years, well now imagine the Deep Web market that is more than 500 times bigger and where there is no legal limits on the odds to sell. We are facing with amazing business controlled by cyber criminals organizations.

The dark business

As said the hidden web is considerable a wide marked covered by anonymity, a condition that make it attractive for the cybercrime industry that is moving its business in a region of cyber space where is really difficult to trace sellers and acquires, whatever goods they exchange.

Majority of Deep Web know is just because they have read about the possibility to acquire weapons, malware and drugs in total security avoiding the control of law enforcement and far from any kind of limitations. In effect in several market place present in the dark web it is possible to acquire illegal odds and the press has made great advertising on this aspect, that is the type of news that people love to.

One of the most famous dark market is without doubt Silk Road web site, an online marketplace where the majority of products are derived from illegal activities. Of course it’s not the only one, many other markets are managed to address specify products, believe me, many of them are terrifying.

Most transactions on the Deep Web accept BitCoin currency for payments allowing the purchase of any kind of products preserving the anonymity of the transaction, encouraging the development of trade in respect to any kind of illegal activities. We are facing with an autonomous system that advantage the exercise of criminal activities while ensuring the anonymity of transactions and the inability to track down the criminals.

Recently the Carnegie Mellon computer security professor Nicolas Christin published research on Silk Road and its business model, it seems that the market is able to realize $22 Million In Annual Sales only related to the drug market.  Total revenue made by the sellers has been estimated around USD 1.9 million per month, an incredible business also for the Silk Road operators that receive about USD 143,000 per month in commissions.

The experts have examined over 24,400 separate items sold on the popular site demonstrating that Silk Road is mainly used as drugs market, very interesting also the composition of the sellers that for obvious needs leaves within a couple of weeks the site to appear in second time.

The study has analyzed the evolution of the market in the last months demonstrating the increasing of the business may be obtained also thanks to the aura of mystery that many media give the Deep Web.

The number of sellers of any kind of drugs is passed from 300 in February to around 570 in August as reported in the following graph:

Which are the most sold products?

The study has grouped the product in categories and has revealed that the “most wanted” items are drugs, following is proposed the list of the Top 20 categories in terms of items available.

Most sellers leave the site fairly quickly, but a core of about 4% of them have been on the site for the entire duration of our study, the majority of sellers are only on the site for less than two months, may be because they leave the site once sold the products or because they move “into stealth mode as soon as they have established a large enough customer base”.

The experts and law enforcement are conscious that are facing with an anomalous market where identities are hidden, payments difficult to trace, where no advertising is made and where the access to the “market place” implies anonymizing tool such as a Tor client. Despite all this consideration the study has revealed a string growth of the business, the market appears in expansion and number of sellers that use it is dramatically increased.

Christin declared : “It’s a stable marketplace, and overall it’s growing steadily.”

But many users on the site have worried for possible infiltration made by law enforcement, another source of concerns is that several of its high-profile sellers have disappeared. The possibility to infiltrate a similar market is concrete and market place such as Silk Road represents in my opinion a moderate risks for the worldwide community. The most problematic aspects of similar business is that they are controlled by criminal organization but the figure proposed are far from to justify a massive Government intervention,  the problem is how much hidden services like this are in the dark web?

But Deep Web is also famous because is the place where is relative simple to acquire malware and similar agents to realize cyber fraud, one of the most requested article are bot agents to be able to compose a botnet without particular knowledge.

Recently I read of a botnet offering from the deep web describes many interesting technical characteristics. With just $8000 and three IP addresses it is possible to set up C&C, and get a personalized copy of the bot that has a hardcoded/obfuscated max of 10k zombies. I have no idea is the offer is real but is high probable that similar offers are daily available on deep web, we can imagine the impact on this wave of malware in the cyber space.

We have explained several time that a new model of business is growing around malware sells, old style criminals are investing in technology to expand their activities, they are requesting support and material to realize complex frauds, I introduced the term C2C (cybercrime to cybercrime) to describe the phenomenon of support provided by new cyber criminality  to ordinary crime.

One of the most famous malware sold in clear web which is “migrating” to the deep web is the Citadel trojan, based on  the Zeus experience has evolved becoming one of the most interesting cyber criminal project. Security experts have found an excellent customer relationship management (CRM) model implemented by its creators. Thanks a malware evolution dictated by market needs, the trojan has evolved in time, many instances have been detected with different powerful features developed for specific clients.

The creators of the agent have structured an efficient service for the sale ( with sale price of nearly $2,500) and the supply of improvement services for the trojan through social network platforms.

But just one of the strengths of the model, the opportunity to get in touch with the  creators of the virus, paradoxically, could stop the spread of the dreaded malware.

So how to protect anonymity of the creators maintaining a malware as service selling model?

Deep web gives a great opportunity, that’s why Citadel’s authors will probably migrate to the hidden web, trying to avoid the controls of law enforcement.

The need to restrict the audience of prospective customers could restrict the global business preserving its vitality. The anonymity is a need for cyber criminal, we have assisted to the proliferation of encrypted instant messaging communications and of VPN service providers, all to avoid to be spied on.

Cyber crime is characterized by a technical soul that is pushing the implementation of new hidden services deployed in the dark web, we are assisting to the consolidation of the black market, brokers can set up auctions to sell new malware and zero-day vulnerabilities ensuring the anonymity of the parties.

Deep Web … a powerful analysis tool

We have seen that Deep Web thanks to anonymity and its dimensions represents a great opportunity for cyber criminal business it is also a powerful analysis tool.

The Tor Metrics Portal gives a set of useful the instruments to monitors the workload of the TOR networks, it proposes a complete collection of tools and documentation for statistical analysis regarding the activities of relays and bridges. These metrics could also be used for intelligence purpose, for example analyzing principal network metrics it is possible to investigate on the application of monitoring system inside a country for censorship purpose. Recently in many area of the planet similar systems have been used to suppress media protest and to persecute dissidents, avoiding the circulation of unconformable information outside the country. It is happened for example in Syria and in Iran, country where the control of the web is a major concern of the government. These situations are expression of a political sufferance of a country and could give a further element of evaluation to the analysts.

Analyzing the number of access to the Tor Network over the time it has been possible for example to discover how The Ethiopian Telecommunication Corporation, unique telecommunication service provider of the country, has deployed for testing purpose a Deep Packet Inspection (DPI) of all Internet traffic.

Using the metrics it was possible to identify the introduction of the filtering system as displayed in the following graphs.

It’s simple to note that in the last week of May the Tor Network was not accessible from the country even with trying to use bridged access, evidence of the presence of filtering system for Deep Packet Inspection.

Deep Web is not the hell

After this overview I desire to explain to the readers that despite the Deep Web provide an environment to protect their privacy there are several conditions and different type of attacks that could expose user’s identity. Governments are increasing their capability to monitor the hidden network, mainly trying to infiltrating them with spying services. In more than one occasion we have read of several U.S. cyber units totally dedicated to the monitoring if the Deep Web.

We have said that Deep Web is a creature desired by governments to allow to operate in totally anonymity, of course this aspect has been also exploited by cyber criminals, hacktivists and normal people who desire to defend their privacy, for this reason, institutions and agencies of every countries have promoted project to develop new monitoring systems and at same time they have started a misinformation campaign against the this parallel and hidden world.

The governments want you stay far from hidden web, because they cannot spy on you, the crime is present in deep web as in the clear web of course the anonymity granted by deep web could encourage and facilitate criminal activities but at same time it represent an obstacle to the criminal that for example desire to steal sensible information of the users or spy on them.

Meanwhile on the clear web we are able to find many reports produced by security firms on cyber criminal activities and related earnings, we know relatively little about the profits related to the Deep Web that is characterized by a size and a criminal turnover dramatically greater than the one in the dark web.

In the meantime … don’t believe to those that say you that Deep Web is the reign of the evil, because they are trying simply to defend their secrets keeping you away from that place.

[adrotate banner=”9″]

About the Author : Pierluigi Paganini, Security Specialist

CEH – Certified Ethical Hacker, EC Council

Security Affairs (  )

Email : [email protected]


(Security Affairs – Deep Web) 

you might also like

leave a comment