• Home
  • Cyber Crime
  • Cyber warfare
  • APT
  • Data Breach
  • Deep Web
  • Digital ID
  • Hacking
  • Hacktivism
  • Intelligence
  • Internet of Things
  • Laws and regulations
  • Malware
  • Mobile
  • Reports
  • Security
  • Social Networks
  • Terrorism
  • ICS-SCADA
  • POLICIES
  • Contact me
MUST READ

Hackers deploy DripDropper via Apache ActiveMQ flaw, patch systems to evade detection

 | 

A Scattered Spider member gets 10 years in prison

 | 

FBI: Russia-linked group Static Tundra exploit old Cisco flaw for espionage

 | 

US CERT/CC warns of flaws in Workhorse Software accounting software used by hundreds of municipalities in Wisconsin

 | 

DOJ takes action against 22-year-old running RapperBot Botnet

 | 

Google fixed Chrome flaw found by Big Sleep AI

 | 

Pharmaceutical firm Inotiv discloses ransomware attack. Qilin group claims responsibility for the hack

 | 

A hacker tied to Yemen Cyber Army gets 20 months in prison

 | 

Exploit weaponizes SAP NetWeaver bugs for full system compromise

 | 

Allianz Life security breach impacted 1.1 million customers

 | 

U.S. CISA adds Trend Micro Apex One flaw to its Known Exploited Vulnerabilities catalog

 | 

AI for Cybersecurity: Building Trust in Your Workflows

 | 

Taiwan Web Infrastructure targeted by APT UAT-7237 with custom toolset

 | 

New NFC-Driven Android Trojan PhantomCard targets Brazilian bank customers

 | 

Cisco fixed maximum-severity security flaw in Secure Firewall Management Center

 | 

'Blue Locker' Ransomware Targeting Oil & Gas Sector in Pakistan

 | 

Hackers exploit Microsoft flaw to breach Canada ’s House of Commons

 | 

Norway confirms dam intrusion by Pro-Russian hackers

 | 

Zoom patches critical Windows flaw allowing privilege escalation

 | 

Manpower data breach impacted 144,180 individuals

 | 
  • Home
  • Cyber Crime
  • Cyber warfare
  • APT
  • Data Breach
  • Deep Web
  • Digital ID
  • Hacking
  • Hacktivism
  • Intelligence
  • Internet of Things
  • Laws and regulations
  • Malware
  • Mobile
  • Reports
  • Security
  • Social Networks
  • Terrorism
  • ICS-SCADA
  • POLICIES
  • Contact me
  • Home
  • Breaking News
  • Hacking
  • How hackers use Robots txt to harvest information

How hackers use Robots txt to harvest information

Pierluigi Paganini May 19, 2015

The penetration tester Thiebauld Weksteen has published an interesting analysis to explaine the importance of robots.txt for the hacking activities.

Thiebauld Weksteen, a penetration tester from Melbourne is advising system administrators that robots.txt can give precious details to hackers, when it comes to attacks, because robots.txt as the capability to tell search engines which directories can and cannot be crawled on a web server.

Weksteen, a security expert at Securus Global explained that system administrators offer clues to hackers related the storage of sensitive assets and information by including the path in the robots.txt file. The analysis of robots.txt could help the intruder to target the attack, instead of trying to strike blindly.

“In the simplest cases, it (robots.txt) will reveal restricted paths and the technology used by your servers, ” From a defender perspective, two common fallacies remain; that robots.txt somewhat is acting as an access control mechanism [and that] content will only be read by search engines and not by humans.”

In the same way that portal administrators don’t invest enough time in patching vulnerabilities, robots.txt normally shows the lack of caring, regularly reporting lists of text to disallow.

robots txt

Reconnaissance is an essential activity for a penetration tester, by harvesting robots.txt it is possible to get precious information for the sensitive assets.

“During the reconnaissance stage of a web application testing, the tester usually uses a list of known subdirectories to brute force the server and find hidden resources.

Depending on the uptake of certain web technologies, it needs to be refreshed on a regular basis. As you may see, the directive disallow gives an attacker precious knowledge on what may be worth looking at. Additionally, if that is true for one site, it is worth checking for another. ”

This is done by using a crawling technique, and as reported by Weksteen, on a total of 59,558 crawled website, 59,436 sent us a response.

“This is a good indicator of the freshness of Common Crawl results. Of these, 37,431 responded with a HTTP status of 200. Of these, 35,376 returned something that looked like a proper robots.txt (i.e., matching at least one standard directive).”

Administrators need to spend more time in hardening their systems, and if possible excluding assets based in generic terms, not specific ones.

Let me suggest you to read the analysis published by Weksteen at the following address
http://thiébaud.fr/robots.txt.html

About the Author Elsio Pinto

Elsio Pinto (@high54security) is at the moment the Lead Mcafee Security Engineer at Swiss Re, but he also as knowledge in the areas of malware research, forensics, ethical hacking. He had previous experiences in major institutions being the European Parliament one of them. He is a security enthusiast and tries his best to pass his knowledge. He also owns his own blog http://high54security.blogspot.com/

Edited by Pierluigi Paganini

(Security Affairs – Robots.txt, hacking)


facebook linkedin twitter

Hacking penetration testing Pierluigi Paganini Robots.txt security Security Affairs

you might also like

Pierluigi Paganini August 21, 2025
Hackers deploy DripDropper via Apache ActiveMQ flaw, patch systems to evade detection
Read more
Pierluigi Paganini August 21, 2025
A Scattered Spider member gets 10 years in prison
Read more

leave a comment

newsletter

Subscribe to my email list and stay
up-to-date!

    recent articles

    Hackers deploy DripDropper via Apache ActiveMQ flaw, patch systems to evade detection

    Malware / August 21, 2025

    A Scattered Spider member gets 10 years in prison

    Cyber Crime / August 21, 2025

    FBI: Russia-linked group Static Tundra exploit old Cisco flaw for espionage

    Intelligence / August 21, 2025

    US CERT/CC warns of flaws in Workhorse Software accounting software used by hundreds of municipalities in Wisconsin

    Security / August 21, 2025

    DOJ takes action against 22-year-old running RapperBot Botnet

    Cyber Crime / August 20, 2025

    To contact me write an email to:

    Pierluigi Paganini :
    pierluigi.paganini@securityaffairs.co

    LEARN MORE

    QUICK LINKS

    • Home
    • Cyber Crime
    • Cyber warfare
    • APT
    • Data Breach
    • Deep Web
    • Digital ID
    • Hacking
    • Hacktivism
    • Intelligence
    • Internet of Things
    • Laws and regulations
    • Malware
    • Mobile
    • Reports
    • Security
    • Social Networks
    • Terrorism
    • ICS-SCADA
    • POLICIES
    • Contact me

    Copyright@securityaffairs 2024

    We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
    Cookie SettingsAccept All
    Manage consent

    Privacy Overview

    This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities...
    Necessary
    Always Enabled
    Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
    Non-necessary
    Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
    SAVE & ACCEPT