The Deep Web Part 1: Introduction to the Deep Web and how to wear clothes online!

Pierluigi Paganini October 15, 2012

Article published on The Malta Indipendent on October 14th 2012

Ron Kelson,

Pierluigi Paganini,

Fabian Martin,

David Pace,

Benjamin Gittins


According to Cisco, by the end of this year, there will be more Internet-connected mobile devices than people on Earth!  Not surprisingly there is a lot of interesting content being generated…

According to Netcraft, there are over 190 million active websites, and according to the WorldWideWebSize daily estimate, the Indexed Web contains at least 8.42 billion pages.  Yet… this is just the tip of the iceberg!

The lesser known “Deep Web” is the set of information resources on the World Wide Web not indexed by normal search engines.  According to rough estimates by principal researchers and security experts, it is around 500 times greater than the overall clear (generally accessible/indexed) web content, and it is unknown to the vast majority of web users!

To better describe the Deep Web we cite the definition provided by the founder of Bright-Planet, Mike Bergman, which equates searching on the Internet today to dragging a net across the surface of the ocean: a great deal may be caught in the net, but there is a wealth of information that is deep, and therefore missed.

How is it possible that resources located on the web are not visible?   Ordinary search engines use automated software called “crawlers” or “robots” to search for content on the web.  These robots browse the web in a methodical, automated manner, storing the content of discovered pages in massive databases that can be searched.  The biggest problem is that not all content can be discovered by crawling from one page to the next. Crawlers are ineffective at finding:

  1. Public information accessible in large databases that can only be searched through dynamic queries.
  2. Website pages which are not linked to by other pages.
  3. Scripted content accessible through links produced by JavaScript as well as content dynamically downloaded from Web servers via Flash or Ajax solutions.
  4. Private Web (password-protected resources).
  5. Content stored on other “Networks” that run over the Internet:  a) Text content using the Gopher protocol, b) Files hosted on “File Transfer Protocol” (FTP) and c) Information hosted on anonymizing networks, such as Tor – The onion router.

There are many, many fascinating aspects to the Deep Web…  For example, we may like to learn how to legitimately access more of this public data that is hidden from view, and hard to find, due to the limitations of today’s search engines.  On the flip side, with all the massive data collection going on around the world, we may like to use the Deep Web to protect ourselves.


Your legitimate right to data privacyToday, tracking user activities on the Internet is of interest for private companies and many Governments.  Business and political motives are pushing the ever increasing development of monitoring and surveillance systems.  Too often, Governments have a tendency to simplistically equate more “surveillance” with greater “security” (for who?).  Too often businesses rationalise privacy invasion as “business intelligence” to give them a market edge.

Truly anonymous communications have an important place in our political and social discourse.  Many individuals may have the legitimate desire to avoid disclosing their identities because they are concerned about political or economic retribution, harassment, or even threats to their lives.  You don’t walk around the street telling every stranger your name, so why should it be any different on the Internet?

For example, lets consider the political discourse: “The Spirit of the Laws” written by Montesquieu in 1748.  Montesquieu was a French *social commentator* and *political thinker* who lived during the Enlightenment.  He is famous for his articulation of the theory of separation of powers, which is taken for granted in modern discussions of government, and implemented in many constitutions throughout the world.  This work was originally published anonymously partly because Montesquieu’s works were subject to censorship.

Shortly after publication, the Catholic Church banned this work in 1751 and included it on the Index of Prohibited Books!  Had the Catholic Church’s actions been effective, Montesquieu’s work, pleading in favor of a constitutional system of government and the separation of powers, the ending of slavery, the preservation of civil liberties and the law, and the idea that political institutions ought to reflect the social and geographical aspects of each community, would not have been available.  Social progress would have been snuffed out due to National Security concerns in the 1750’s!

The Supreme Court of the United States has ruled repeatedly that the right to anonymous free speech is protected by the First Amendment:  Anonymity is a shield from the tyranny of the majority,” that “exemplifies the purpose” of the First Amendment: “to protect unpopular individuals from retaliation…at the hand of an intolerant society.”

Court pronunciations establish the duty for government to guard against undue hindrances to political conversations and the exchange of ideas, a vigilant review that US laws establish the right to Speak Anonymously on the Internet, and also right to Read Anonymously on the Internet, ensuring the principle of free Internet ideological confrontation, and the right to free movement of information:

“People are permitted to interact pseudonymously and anonymously with each other so long as those acts are not in violation of the law. This ability to speak one’s mind without the burden of the other party knowing all the facts about one’s identity can foster open communication and robust debate.”

The right to Internet anonymity is also covered by European legislation which recognizes the fundamental right to data protection, freedom of expression, freedom of impression.  The European Union Charter of Fundamental Rights recognizes in Article. 8 (Title II: “Freedoms”) the right of everyone to protection of personal data concerning him.

The Internet was not built with security in mind:  It was not built to protect your privacy.  According to Brian Snow, former Technical Director of the Information Assurance Directorate of the United States National Security Agency:

“The creators of the Internet knew that MALICE was a serious issue.”  …  “However, the creators of the Internet pushed security aside due to the perceived difficulties, or cost, and that is the start of our problems today. To put it bluntly, the Internet was not built to address the known risks.  By design, the Internet naïvely relies on the honesty of every network user, and places far too little emphasis on healthy mutual suspicion!  The cost and risks were not eliminated — rather they were both shifted away from the designers and the manufacturers, and transferred to the Global user base.  You and me pick up the check.”  

According to Vint Cerf, one of the founders of the Internet,

“A new version of the Internet might be the best way to defend against cyber attacks”.  Vint goes on to say: “Co-founder Bob Kahn and I did not think enough about security when we built the framework for the web”.

To put it simply, browsing around the Internet today is like going out into the streets with no clothes on!   Lets explore how to get dressed properly before surfing the Internet!

Preserving your privacy on the InternetUse the Tor network!  “The onion router” (Tor) is free software and an open network that helps you defend against a form of network surveillance that threatens personal freedom and privacy, confidential business activities and relationships, and traffic analysis.

Tor protects your online Internet activities by first encrypting (putting your clothes on) and then bouncing your communications around a distributed network of (900+) relays run by volunteers all around the world.  By doing this it prevents malicious people watching your Internet connection from learning what sites you visit, and it prevents the sites you visit from learning your physical location!

Why is this important?  Have you ever visited a new website, and somehow the advertising on that site knows what you like? Or it somehow knows the subjects you have recently searched?  Tor protects you from this type of online stalking.  Thankfully, Tor works with many of your existing applications, including web browsers and instant messaging clients!

Who are using anonymizing networks today, and whyHere are just a few typical examples of Tor usage:

  1. Normal people, like you and your family, use Tor to protect themselves, their children and their dignity while using the Internet.  Tor helps to protect your privacy from tracking websites, unscrupulous marketers and identity thieves.  Individuals also use Tor for socially sensitive communication: chat rooms and web forums for physical abuse survivors, or people with embarrassing medical illnesses (like AIDS).
  2.  Journalists and their contacts to maintain anonymity avoiding persecution of authoritarian regimes.
  3.  Activists and Whistleblowers to anonymously report abuses from danger zones.
  4. Business executives, to avoid resources accessed being tracked by competitors, and to increase the level of protection against cyber espionage especially during procurement, or when senior management performs remote access from “external and dangerous locations” (e.g. accessing company resources during a business trip in countries such as China).
  5. IT Professionals to test and verify their infrastructures.

How do I get it!  Tor is free and available for Windows, Mac, Linux/Unix, and Android!   You can download and install it from here: .  Tor is one of the best ways to protect yourself from malicious surveillance by corporations and others malicious parties online.  We strongly recommend wearing clothes on the Internet:  Try installing and using Tor today!

A word of wisdom:  The Internet provides a level of anonymity which is higher than when interacting face-to-face in person.  Tor permits an even higher level of anonymity to be achieved with correct use.  On the Internet and on Tor’s Networks, as in all activities in life,  please apply the Golden Rule:  Always act with integrity, and treat others with respect and compassion.  Please use Tor responsibly as a constructive tool for personal and social development.


About the Authors:

Pierluigi Paganini, Deep web expert and Security Specialist CISO Bit4ID Srl, a CEH Certified Ethical Hacker, EC Council and Founder of Security Affairs ( ). Pierluigi Paganini is a co-author (with Richard Amores) of the soon to be published book – “The Deep Dark Web: The hidden world” which extensively covers all aspects of the Deep Web.

David Pace is Project Manager of the ICT Gozo Malta Project, and a freelance IT Consultant

Prof. Fabian Martins, ( ) Banking security expert and Product Development Manager at Scopus Tecnologia, ) owned by Bradesco Bank Group.

Ron Kelson is Vice Chair of the ICT Gozo Malta Project and CEO of Synaptic Laboratories Limited [email protected] .

Benjamin Gittins is CTO of Synaptic Laboratories Limited. [email protected]


ICT Gozo Malta is a joint collaboration between the Gozo Business Chamber and Synaptic Labs, part funded in 2011 by the Malta Government, Ministry for Gozo, Eco Gozo Project, and a prize winner in the 2012 Malta Government National Enterprise Support Awards. links to free cyber awareness resources for all age groups.   To promote Maltese ICT, we encourage all ICT Professionals to register on the ICT GM Skills Register and keep aware of developments, both in Cyber security and other ICT R&D initiatives in Malta and Gozo.   For further details contact David Pace at [email protected] or phone +356 79630221 .


you might also like

leave a comment