Reconnaissance and OSINT: Ethical Information Gathering

Before anyone can protect an organisation, they must see it the way an outsider does. Reconnaissance is the process of gathering information about a target; OSINT (open-source intelligence) is the practice of doing it using publicly available sources. Attackers use these techniques to plan; defenders and authorised testers use them to find out what the world can already learn about them. In this lesson you will learn the techniques, the legal and ethical limits and how to practise safely on your own domain.

Passive vs Active Reconnaissance

TypeWhat it meansRisk of detectionExamples
PassiveCollecting information without interacting directly with the target’s systemsVery lowSearch engines, public records, certificate transparency logs, social media, archives
ActiveInteracting with the target’s systems to learn about themHigher; leaves logsPort scans, banner grabbing, directory probing

Active reconnaissance requires explicit authorisation. Even passive collection must respect privacy law and the terms of the sites you use; see Ethical Hacking: Law and Ethics.

What Can Be Learned From Public Sources

  • Domains and subdomains (including forgotten test or admin sites).
  • IP ranges and hosting providers.
  • Technology in use: web servers, frameworks, content management systems, versions.
  • Email addresses and naming conventions (such as first.last@company).
  • Employees and roles from professional networks and company pages.
  • Leaked or exposed data: code repositories, documents, credentials in public breach corpuses.
  • Physical locations, vendors and partners.
  • Job advertisements that reveal technology stacks and internal tools.

Every piece of this can help an attacker craft a convincing phishing message or find a forgotten server. That is why reducing your public footprint is part of defence.

Core Techniques and Safe Practice

The commands below use example.com, a domain reserved for documentation. When practising, substitute a domain you own.

1. WHOIS: who registered a domain?

whois example.com

Shows registrar, dates and sometimes contacts. Many registries now redact personal details for privacy. Review what your domain exposes.

2. DNS records

dig +short A example.com        # IPv4 addresses
dig +short AAAA example.com     # IPv6 addresses
dig +short MX example.com       # mail servers
dig +short NS example.com       # name servers
dig +short TXT example.com      # text records: SPF, verification tokens
dig +short TXT _dmarc.example.com

MX and TXT records reveal email providers and third-party services. Defensively, check that SPF and DMARC are published and that nothing sensitive sits in TXT records.

3. Certificate transparency

Every publicly trusted certificate is logged publicly. Searching the logs for your domain (for example on crt.sh) lists hostnames for which certificates were issued, often exposing subdomains you forgot existed. Use it to inventory and clean up your own attack surface, and to spot unexpected certificates issued for your name.

4. Search-engine research

Advanced search operators help find content that was indexed by accident. Run them for your own site to see what is exposed:

site:yourdomain.example filetype:pdf
site:yourdomain.example intitle:"index of"
site:yourdomain.example inurl:admin

If you discover documents or admin pages that should be private, remove them, restrict access and request de-indexing.

5. Internet-wide scanners and asset search engines

Services such as Shodan and Censys catalogue internet-facing devices and services. Searching for your own organisation’s IP ranges shows what the world sees: exposed databases, remote desktop, outdated software or camera feeds. Treat them as a mirror for your own exposure, not as a tool for exploring other people’s systems.

6. Web archives and technology fingerprinting

The Internet Archive’s Wayback Machine shows historical versions of pages, sometimes including removed contact lists or old admin paths. HTTP headers and page source reveal server software and frameworks:

curl -I https://example.com
curl -s https://example.com | grep -i -E "generator|wordpress|powered"

Hiding version banners and unneeded headers removes easy clues, but remember that patching matters more than hiding.

7. Code repositories and secrets

Developers sometimes commit passwords, API keys or private keys to public repositories. Defenders should scan their own organisations’ repositories with secret-scanning tools (for example, Gitleaks or built-in platform scanning) and rotate any exposed credential immediately; see Secure Coding and DevSecOps.

8. People-focused OSINT

Professional networking profiles, conference talks and social media reveal organisation structure, travel and interests. Authorised social-engineering tests use this information carefully; the privacy of individuals must be respected, and the scope must say what is allowed. Never use OSINT to stalk, harass or expose private individuals.

Practice Lab: Footprint Your Own Domain

If you do not own a domain, use your own blog address or a free subdomain you control. Create a short worksheet and fill it in:

  1. Run the WHOIS and DNS commands above. Record registrar, name servers, mail provider and any TXT records.
  2. Search certificate transparency logs for your domain. List all hostnames. Are there any you had forgotten?
  3. Run the three site: searches. Does anything appear that should not?
  4. Check the HTTP response headers of your site (curl -I). What do they reveal about your server and software?
  5. Review your public profiles. Would a stranger be able to learn your employer, location and routines?
  6. Write a remediation list: what will you remove, restrict or monitor?

A sample worksheet

ItemFindingRiskAction
Subdomain test.yourdomain.example in CT logsForgotten staging serverMediumDecommission or protect with authentication
Server header shows exact versionVersion disclosureLowRemove version banner; patch regularly
No DMARC recordEmail spoofing possibleMediumPublish SPF, DKIM and DMARC

Reducing Your Exposure

  1. Maintain an asset inventory, and decommission unused hosts and subdomains.
  2. Enable WHOIS privacy where appropriate; use role-based email addresses for registration.
  3. Remove metadata from public documents (author names, software versions, internal paths).
  4. Configure web servers not to reveal unnecessary banners, and block directory listing.
  5. Scan repositories for secrets and enable prevention features.
  6. Brief employees about what to share publicly, especially on professional networks.
  7. Monitor for leaked credentials and look-alike domains registered to impersonate you.

Legal and Ethical Boundaries

  • Passive collection from genuinely public sources is generally lawful, but laws on personal data, scraping and terms of service vary by country.
  • Do not log in with leaked credentials, access private areas or probe systems without authorisation. Using stolen data is a crime even if it is easy to find.
  • Keep the information you collect secure and share it only with those who need it.
  • Stay within your scope even when something interesting appears next door.

Practice Exercises

  1. Explain the difference between passive and active reconnaissance, and give an example of each that you could do safely in your lab.
  2. List five pieces of information that could help an attacker write a convincing phishing email to your organisation.
  3. Find the MX and TXT records for a domain you own and explain what each reveals.
  4. Why might a certificate transparency log reveal systems that were meant to be private?
  5. Draft three rules for an OSINT policy: what your team may collect, how it must be stored and what is off limits.

Mentor tip: Run an OSINT review of your own organisation twice a year. The surprises you find (forgotten servers, old documents, over-sharing profiles) are the same ones an attacker would find on day one.

Educational use only: perform reconnaissance only on your own assets or within an authorised scope, and respect privacy and the law.

Key takeaways
  • Reconnaissance gathers information; passive methods avoid touching the target, active methods require permission.
  • WHOIS, DNS, certificate transparency, search operators and asset search engines reveal an organisation’s footprint.
  • Defenders use the same techniques to shrink their attack surface and find leaks.
  • Respect privacy, law and scope; never use leaked credentials.
  • Regularly footprint your own organisation and fix what you find.

Related Lessons