diff --git a/README.md b/README.md
index bdaa243fc5..f845ff3159 100644
--- a/README.md
+++ b/README.md
@@ -3,10 +3,9 @@
[](https://raw.githubusercontent.com/smicallef/spiderfoot/master/LICENSE)
[](https://www.python.org)
-[](https://github.com/smicallef/spiderfoot/releases/tag/v3.5)
+[](https://github.com/smicallef/spiderfoot/releases/tag/v4.0)
[](https://github.com/smicallef/spiderfoot/actions?query=workflow%3A"Tests")
[](https://github.com/smicallef/spiderfoot/commits/master)
-[](https://libraries.io/github/smicallef/spiderfoot)
[](https://codecov.io/github/smicallef/spiderfoot)
[](https://twitter.com/spiderfoot)
[](https://discord.gg/vyvztrG)
@@ -15,13 +14,14 @@
SpiderFoot has an embedded web-server for providing a clean and intuitive web-based interface but can also be used completely via the command-line. It's written in **Python 3** and **GPL-licensed**.
-
+
### FEATURES
- Web based UI or CLI
- Over 200 modules (see below)
-- Python 3
+- Python 3.7+
+- YAML-configurable [correlation engine](/correlations/README.md) with [37 pre-defined rules](/correlations)
- CSV/JSON/GEXF export
- API key export/import
- SQLite back-end for custom querying
@@ -31,7 +31,7 @@ SpiderFoot has an embedded web-server for providing a clean and intuitive web-ba
- TOR integration for dark web searching
- Dockerfile for Docker-based deployments
- Can call other tools like DNSTwist, Whatweb, Nmap and CMSeeK
-- Actively developed since 2012!
+- [Actively developed since 2012!](https://medium.com/@micallst/lessons-learned-from-my-10-year-open-source-project-4a4c8c2b4f64)
### WANT MORE?
@@ -41,13 +41,13 @@ Need more from SpiderFoot? Check out [SpiderFoot HX](https://www.spiderfoot.net/
- Multiple targets per scan
- Multi-user collaboration
- Authenticated and 2FA
-- Correlations
- Investigations
- Customer support
- Third party tools pre-installed & configured
- Drive it with a fully RESTful API
- TOR integration built-in
- Screenshotting
+- Bring your own Python SpiderFoot modules
- Feed scan data to Splunk, ElasticSearch and REST endpoints
See the full set of differences between SpiderFoot HX and the open source version [here](https://www.spiderfoot.net/open-source-vs-hx/).
@@ -95,9 +95,9 @@ To install and run SpiderFoot, you need at least Python 3.7 and a number of Pyth
#### Stable build (packaged release):
```
-$ wget https://github.com/smicallef/spiderfoot/archive/v3.5.tar.gz
-$ tar zxvf v3.5.tar.gz
-$ cd spiderfoot-3.5
+$ wget https://github.com/smicallef/spiderfoot/archive/v4.0.tar.gz
+$ tar zxvf v4.0.tar.gz
+$ cd spiderfoot-4.0
$ pip3 install -r requirements.txt
$ python3 ./sf.py -l 127.0.0.1:5001
```
@@ -117,236 +117,251 @@ Check out the [documentation](https://www.spiderfoot.net/documentation) and our
Whether you're a contributor, user or just curious about SpiderFoot and OSINT in general, we'd love to have you join our community! SpiderFoot now has a [Discord server](https://discord.gg/vyvztrG) for seeking help from the community, requesting features or just general OSINT chit-chat.
+### WRITING CORRELATION RULES
+
+We have a comprehensive write-up and reference of the correlation rule-set introduced in SpiderFoot 4.0 [here](/correlations/README.md).
+
+Also take a look at the [template.yaml](/correlations/template.yaml) file for a walk through. The existing [37 rules](/correlations) are also quite readable and good as starting points for additional rules.
+
### MODULES / INTEGRATIONS
SpiderFoot has over 200 modules, most of which *don't require API keys*, and many of those that do require API keys *have a free tier*.
-| Name | Description | Link | Type |
-|:---------| :-----------| :----------|:-------|
-AbstractAPI|Look up domain, phone and IP address information from AbstractAPI.|[https://app.abstractapi.com/](https://app.abstractapi.com/)|Tiered API
-abuse.ch|Check if a host/domain, IP address or netblock is malicious according to Abuse.ch.|[https://www.abuse.ch](https://www.abuse.ch)|Free API
-AbuseIPDB|Check if an IP address is malicious according to AbuseIPDB.com blacklist.|[https://www.abuseipdb.com](https://www.abuseipdb.com)|Tiered API
-Abusix Mail Intelligence|Check if a netblock or IP address is in the Abusix Mail Intelligence blacklist.|[https://abusix.org/](https://abusix.org/)|Tiered API
-Account Finder|Look for possible associated accounts on nearly 200 websites like Ebay, Slashdot, reddit, etc.|N/A|Internal
-AdBlock Check|Check if linked pages would be blocked by AdBlock Plus.|[https://adblockplus.org/](https://adblockplus.org/)|Tiered API
-AdGuard DNS|Check if a host would be blocked by AdGuard DNS.|[https://adguard.com/](https://adguard.com/)|Free API
-Ahmia|Search Tor 'Ahmia' search engine for mentions of the target.|[https://ahmia.fi/](https://ahmia.fi/)|Free API
-AlienVault IP Reputation|Check if an IP or netblock is malicious according to the AlienVault IP Reputation database.|[https://cybersecurity.att.com/](https://cybersecurity.att.com/)|Free API
-AlienVault OTX|Obtain information from AlienVault Open Threat Exchange (OTX)|[https://otx.alienvault.com/](https://otx.alienvault.com/)|Tiered API
-Amazon S3 Bucket Finder|Search for potential Amazon S3 buckets associated with the target and attempt to list their contents.|[https://aws.amazon.com/s3/](https://aws.amazon.com/s3/)|Free API
-Apple iTunes|Search Apple iTunes for mobile apps.|[https://itunes.apple.com/](https://itunes.apple.com/)|Free API
-Archive.org|Identifies historic versions of interesting files/pages from the Wayback Machine.|[https://archive.org/](https://archive.org/)|Free API
-ARIN|Queries ARIN registry for contact information.|[https://www.arin.net/](https://www.arin.net/)|Free API
-Azure Blob Finder|Search for potential Azure blobs associated with the target and attempt to list their contents.|[https://azure.microsoft.com/en-in/services/storage/blobs/](https://azure.microsoft.com/en-in/services/storage/blobs/)|Free API
-Bad Packets|Obtain information about any malicious activities involving IP addresses found|[https://badpackets.net](https://badpackets.net)|Commercial API
-Base64 Decoder|Identify Base64-encoded strings in URLs, often revealing interesting hidden information.|N/A|Internal
-BGPView|Obtain network information from BGPView API.|[https://bgpview.io/](https://bgpview.io/)|Free API
-Binary String Extractor|Attempt to identify strings in binary content.|N/A|Internal
-BinaryEdge|Obtain information from BinaryEdge.io Internet scanning systems, including breaches, vulnerabilities, torrents and passive DNS.|[https://www.binaryedge.io/](https://www.binaryedge.io/)|Tiered API
-Bing (Shared IPs)|Search Bing for hosts sharing the same IP.|[https://www.bing.com/](https://www.bing.com/)|Tiered API
-Bing|Obtain information from bing to identify sub-domains and links.|[https://www.bing.com/](https://www.bing.com/)|Tiered API
-Bitcoin Finder|Identify bitcoin addresses in scraped webpages.|N/A|Internal
-Bitcoin Who's Who|Check for Bitcoin addresses against the Bitcoin Who's Who database of suspect/malicious addresses.|[https://bitcoinwhoswho.com/](https://bitcoinwhoswho.com/)|Tiered API
-BitcoinAbuse|Check Bitcoin addresses against the bitcoinabuse.com database of suspect/malicious addresses.|[https://www.bitcoinabuse.com/](https://www.bitcoinabuse.com/)|Free API
-Blockchain|Queries blockchain.info to find the balance of identified bitcoin wallet addresses.|[https://www.blockchain.com/](https://www.blockchain.com/)|Free API
-blocklist.de|Check if a netblock or IP is malicious according to blocklist.de.|[http://www.blocklist.de/en/index.html](http://www.blocklist.de/en/index.html)|Free API
-BotScout|Searches BotScout.com's database of spam-bot IP addresses and e-mail addresses.|[https://botscout.com/](https://botscout.com/)|Tiered API
-botvrij.eu|Check if a domain is malicious according to botvrij.eu.|[https://botvrij.eu/](https://botvrij.eu/)|Free API
-BuiltWith|Query BuiltWith.com's Domain API for information about your target's web technology stack, e-mail addresses and more.|[https://builtwith.com/](https://builtwith.com/)|Tiered API
-C99|Queries the C99 API which offers various data (geo location, proxy detection, phone lookup, etc).|[https://api.c99.nl/](https://api.c99.nl/)|Commercial API
-CallerName|Lookup US phone number location and reputation information.|[http://callername.com/](http://callername.com/)|Free API
-Censys|Obtain host information from Censys.io.|[https://censys.io/](https://censys.io/)|Tiered API
-Certificate Transparency|Gather hostnames from historical certificates in crt.sh.|[https://crt.sh/](https://crt.sh/)|Free API
-CertSpotter|Gather information about SSL certificates from SSLMate CertSpotter API.|[https://sslmate.com/certspotter/](https://sslmate.com/certspotter/)|Tiered API
-CINS Army List|Check if a netblock or IP address is malicious according to Collective Intelligence Network Security (CINS) Army list.|[https://cinsscore.com/](https://cinsscore.com/)|Free API
-CIRCL.LU|Obtain information from CIRCL.LU's Passive DNS and Passive SSL databases.|[https://www.circl.lu/](https://www.circl.lu/)|Free API
-CleanBrowsing.org|Check if a host would be blocked by CleanBrowsing.org DNS content filters.|[https://cleanbrowsing.org/](https://cleanbrowsing.org/)|Free API
-CleanTalk Spam List|Check if a netblock or IP address is on CleanTalk.org's spam IP list.|[https://cleantalk.org](https://cleantalk.org)|Free API
-Clearbit|Check for names, addresses, domains and more based on lookups of e-mail addresses on clearbit.com.|[https://clearbit.com/](https://clearbit.com/)|Tiered API
-CloudFlare DNS|Check if a host would be blocked by CloudFlare DNS.|[https://www.cloudflare.com/](https://www.cloudflare.com/)|Free API
-CoinBlocker Lists|Check if a domain appears on CoinBlocker lists.|[https://zerodot1.gitlab.io/CoinBlockerListsWeb/](https://zerodot1.gitlab.io/CoinBlockerListsWeb/)|Free API
-CommonCrawl|Searches for URLs found through CommonCrawl.org.|[http://commoncrawl.org/](http://commoncrawl.org/)|Free API
-Comodo Secure DNS|Check if a host would be blocked by Comodo Secure DNS.|[https://www.comodo.com/secure-dns/](https://www.comodo.com/secure-dns/)|Tiered API
-Company Name Extractor|Identify company names in any obtained data.|N/A|Internal
-Cookie Extractor|Extract Cookies from HTTP headers.|N/A|Internal
-Country Name Extractor|Identify country names in any obtained data.|N/A|Internal
-Credit Card Number Extractor|Identify Credit Card Numbers in any data|N/A|Internal
-Crobat API|Search Crobat API for subdomains.|[https://sonar.omnisint.io/](https://sonar.omnisint.io/)|Free API
-Cross-Referencer|Identify whether other domains are associated ('Affiliates') of the target by looking for links back to the target site(s).|N/A|Internal
-CRXcavator|Search CRXcavator for Chrome extensions.|[https://crxcavator.io/](https://crxcavator.io/)|Free API
-Custom Threat Feed|Check if a host/domain, netblock, ASN or IP is malicious according to your custom feed.|N/A|Internal
-CyberCrime-Tracker.net|Check if a host/domain or IP address is malicious according to CyberCrime-Tracker.net.|[https://cybercrime-tracker.net/](https://cybercrime-tracker.net/)|Free API
-Darksearch|Search the Darksearch.io Tor search engine for mentions of the target domain.|[https://darksearch.io/](https://darksearch.io/)|Free API
-Debounce|Check whether an email is disposable|[https://debounce.io/](https://debounce.io/)|Free API
-Dehashed|Gather breach data from Dehashed API.|[https://www.dehashed.com/](https://www.dehashed.com/)|Commercial API
-Digital Ocean Space Finder|Search for potential Digital Ocean Spaces associated with the target and attempt to list their contents.|[https://www.digitalocean.com/products/spaces/](https://www.digitalocean.com/products/spaces/)|Free API
-DNS Brute-forcer|Attempts to identify hostnames through brute-forcing common names and iterations.|N/A|Internal
-DNS Common SRV|Attempts to identify hostnames through brute-forcing common DNS SRV records.|N/A|Internal
-DNS for Family|Check if a host would be blocked by DNS for Family.|[https://dnsforfamily.com/](https://dnsforfamily.com/)|Free API
-DNS Look-aside|Attempt to reverse-resolve the IP addresses next to your target to see if they are related.|N/A|Internal
-DNS Raw Records|Retrieves raw DNS records such as MX, TXT and others.|N/A|Internal
-DNS Resolver|Resolves hosts and IP addresses identified, also extracted from raw content.|N/A|Internal
-DNS Zone Transfer|Attempts to perform a full DNS zone transfer.|N/A|Internal
-DNSDB|Query FarSight's DNSDB for historical and passive DNS data.|[https://www.farsightsecurity.com](https://www.farsightsecurity.com)|Tiered API
-DNSDumpster|Passive subdomain enumeration using HackerTarget's DNSDumpster|[https://dnsdumpster.com/](https://dnsdumpster.com/)|Free API
-DNSGrep|Obtain Passive DNS information from Rapid7 Sonar Project using DNSGrep API.|[https://opendata.rapid7.com/](https://opendata.rapid7.com/)|Free API
-DroneBL|Query the DroneBL database for open relays, open proxies, vulnerable servers, etc.|[https://dronebl.org/](https://dronebl.org/)|Free API
-DuckDuckGo|Query DuckDuckGo's API for descriptive information about your target.|[https://duckduckgo.com/](https://duckduckgo.com/)|Free API
-E-Mail Address Extractor|Identify e-mail addresses in any obtained data.|N/A|Internal
-EmailCrawlr|Search EmailCrawlr for email addresses and phone numbers associated with a domain.|[https://emailcrawlr.com/](https://emailcrawlr.com/)|Tiered API
-EmailFormat|Look up e-mail addresses on email-format.com.|[https://www.email-format.com/](https://www.email-format.com/)|Free API
-EmailRep|Search EmailRep.io for email address reputation.|[https://emailrep.io/](https://emailrep.io/)|Tiered API
-Emerging Threats|Check if a netblock or IP address is malicious according to EmergingThreats.net.|[https://rules.emergingthreats.net/](https://rules.emergingthreats.net/)|Free API
-Error String Extractor|Identify common error messages in content like SQL errors, etc.|N/A|Internal
-Ethereum Address Extractor|Identify ethereum addresses in scraped webpages.|N/A|Internal
-Etherscan|Queries etherscan.io to find the balance of identified ethereum wallet addresses.|[https://etherscan.io](https://etherscan.io)|Free API
-F-Secure Riddler.io|Obtain network information from F-Secure Riddler.io API.|[https://riddler.io/](https://riddler.io/)|Commercial API
-File Metadata Extractor|Extracts meta data from documents and images.|N/A|Internal
-Flickr|Search Flickr for domains, URLs and emails related to the specified domain.|[https://www.flickr.com/](https://www.flickr.com/)|Free API
-Focsec|Look up IP address information from Focsec.|[https://focsec.com/](https://focsec.com/)|Tiered API
-FortiGuard Antispam|Check if an IP address is malicious according to FortiGuard Antispam.|[https://www.fortiguard.com/](https://www.fortiguard.com/)|Free API
-Fraudguard|Obtain threat information from Fraudguard.io|[https://fraudguard.io/](https://fraudguard.io/)|Tiered API
-FullContact|Gather domain and e-mail information from FullContact.com API.|[https://www.fullcontact.com](https://www.fullcontact.com)|Tiered API
-FullHunt|Identify domain attack surface using FullHunt API.|[https://fullhunt.io/](https://fullhunt.io/)|Tiered API
-Github|Identify associated public code repositories on Github.|[https://github.com/](https://github.com/)|Free API
-GLEIF|Look up company information from Global Legal Entity Identifier Foundation (GLEIF).|[https://search.gleif.org/](https://search.gleif.org/)|Tiered API
-Google Maps|Identifies potential physical addresses and latitude/longitude coordinates.|[https://cloud.google.com/maps-platform/](https://cloud.google.com/maps-platform/)|Tiered API
-Google Object Storage Finder|Search for potential Google Object Storage buckets associated with the target and attempt to list their contents.|[https://cloud.google.com/storage](https://cloud.google.com/storage)|Free API
-Google SafeBrowsing|Check if the URL is included on any of the Safe Browsing lists.|[https://developers.google.com/safe-browsing/v4/lookup-api](https://developers.google.com/safe-browsing/v4/lookup-api)|Free API
-Google|Obtain information from the Google Custom Search API to identify sub-domains and links.|[https://developers.google.com/custom-search](https://developers.google.com/custom-search)|Tiered API
-Gravatar|Retrieve user information from Gravatar API.|[https://secure.gravatar.com/](https://secure.gravatar.com/)|Free API
-Grayhat Warfare|Find bucket names matching the keyword extracted from a domain from Grayhat API.|[https://buckets.grayhatwarfare.com/](https://buckets.grayhatwarfare.com/)|Tiered API
-Greensnow|Check if a netblock or IP address is malicious according to greensnow.co.|[https://greensnow.co/](https://greensnow.co/)|Free API
-grep.app|Search grep.app API for links and emails related to the specified domain.|[https://grep.app/](https://grep.app/)|Free API
-Greynoise|Obtain information from Greynoise.io's Enterprise API.|[https://greynoise.io/](https://greynoise.io/)|Tiered API
-HackerOne (Unofficial)|Check external vulnerability scanning/reporting service h1.nobbd.de to see if the target is listed.|[http://www.nobbd.de/](http://www.nobbd.de/)|Free API
-HackerTarget|Search HackerTarget.com for hosts sharing the same IP.|[https://hackertarget.com/](https://hackertarget.com/)|Free API
-Hash Extractor|Identify MD5 and SHA hashes in web content, files and more.|N/A|Internal
-HaveIBeenPwned|Check HaveIBeenPwned.com for hacked e-mail addresses identified in breaches.|[https://haveibeenpwned.com/](https://haveibeenpwned.com/)|Commercial API
-Host.io|Obtain information about domain names from host.io.|[https://host.io](https://host.io)|Tiered API
-Hosting Provider Identifier|Find out if any IP addresses identified fall within known 3rd party hosting ranges, e.g. Amazon, Azure, etc.|N/A|Internal
-Human Name Extractor|Attempt to identify human names in fetched content.|N/A|Internal
-Hunter.io|Check for e-mail addresses and names on hunter.io.|[https://hunter.io/](https://hunter.io/)|Tiered API
-Hybrid Analysis|Search Hybrid Analysis for domains and URLs related to the target.|[https://www.hybrid-analysis.com](https://www.hybrid-analysis.com)|Free API
-IBAN Number Extractor|Identify International Bank Account Numbers (IBANs) in any data.|N/A|Internal
-Iknowwhatyoudownload.com|Check iknowwhatyoudownload.com for IP addresses that have been using torrents.|[https://iknowwhatyoudownload.com/en/peer/](https://iknowwhatyoudownload.com/en/peer/)|Tiered API
-Instagram|Gather information from Instagram profiles.|[https://www.instagram.com/](https://www.instagram.com/)|Free API
-IntelligenceX|Obtain information from IntelligenceX about identified IP addresses, domains, e-mail addresses and phone numbers.|[https://intelx.io/](https://intelx.io/)|Tiered API
-Interesting File Finder|Identifies potential files of interest, e.g. office documents, zip files.|N/A|Internal
-Internet Storm Center|Check if an IP address is malicious according to SANS ISC.|[https://isc.sans.edu](https://isc.sans.edu)|Free API
-ipapi.com|Queries ipapi.com to identify geolocation of IP Addresses using ipapi.com API|[https://ipapi.com/](https://ipapi.com/)|Tiered API
-ipapi.co|Queries ipapi.co to identify geolocation of IP Addresses using ipapi.co API|[https://ipapi.co/](https://ipapi.co/)|Tiered API
-IPInfo.io|Identifies the physical location of IP addresses identified using ipinfo.io.|[https://ipinfo.io](https://ipinfo.io)|Tiered API
-IPQualityScore|Determine if target is malicious using IPQualityScore API|[https://www.ipqualityscore.com/](https://www.ipqualityscore.com/)|Tiered API
-ipregistry|Query the ipregistry.co database for reputation and geo-location.|[https://ipregistry.co/](https://ipregistry.co/)|Tiered API
-ipstack|Identifies the physical location of IP addresses identified using ipstack.com.|[https://ipstack.com/](https://ipstack.com/)|Tiered API
-JsonWHOIS.com|Search JsonWHOIS.com for WHOIS records associated with a domain.|[https://jsonwhois.com](https://jsonwhois.com)|Tiered API
-Junk File Finder|Looks for old/temporary and other similar files.|N/A|Internal
-Keybase|Obtain additional information about domain names and identified usernames.|[https://keybase.io/](https://keybase.io/)|Free API
-Koodous|Search Koodous for mobile apps.|[https://koodous.com/apks/](https://koodous.com/apks/)|Free API
-Leak-Lookup|Searches Leak-Lookup.com's database of breaches.|[https://leak-lookup.com/](https://leak-lookup.com/)|Free API
-LeakIX|Search LeakIX for host data leaks, open ports, software and geoip.|[https://leakix.net/](https://leakix.net/)|Free API
-Maltiverse|Obtain information about any malicious activities involving IP addresses|[https://maltiverse.com](https://maltiverse.com)|Free API
-MalwarePatrol|Searches malwarepatrol.net's database of malicious URLs/IPs.|[https://www.malwarepatrol.net/](https://www.malwarepatrol.net/)|Tiered API
-MetaDefender|Search MetaDefender API for IP address and domain IP reputation.|[https://metadefender.opswat.com/](https://metadefender.opswat.com/)|Tiered API
-Mnemonic PassiveDNS|Obtain Passive DNS information from PassiveDNS.mnemonic.no.|[https://www.mnemonic.no](https://www.mnemonic.no)|Free API
-multiproxy.org Open Proxies|Check if an IP address is an open proxy according to multiproxy.org open proxy list.|[https://multiproxy.org/](https://multiproxy.org/)|Free API
-MySpace|Gather username and location from MySpace.com profiles.|[https://myspace.com/](https://myspace.com/)|Free API
-NameAPI|Check whether an email is disposable|[https://www.nameapi.org/](https://www.nameapi.org/)|Tiered API
-NetworksDB|Search NetworksDB.io API for IP address and domain information.|[https://networksdb.io/](https://networksdb.io/)|Tiered API
-NeutrinoAPI|Search NeutrinoAPI for phone location information, IP address information, and host reputation.|[https://www.neutrinoapi.com/](https://www.neutrinoapi.com/)|Tiered API
-numverify|Lookup phone number location and carrier information from numverify.com.|[http://numverify.com/](http://numverify.com/)|Tiered API
-Onion.link|Search Tor 'Onion City' search engine for mentions of the target domain using Google Custom Search.|[https://onion.link/](https://onion.link/)|Free API
-Onionsearchengine.com|Search Tor onionsearchengine.com for mentions of the target domain.|[https://as.onionsearchengine.com](https://as.onionsearchengine.com)|Free API
-Onyphe|Check Onyphe data (threat list, geo-location, pastries, vulnerabilities) about a given IP.|[https://www.onyphe.io](https://www.onyphe.io)|Tiered API
-Open Bug Bounty|Check external vulnerability scanning/reporting service openbugbounty.org to see if the target is listed.|[https://www.openbugbounty.org/](https://www.openbugbounty.org/)|Free API
-Open Passive DNS Database|Obtain passive DNS information from pdns.daloo.de Open passive DNS database.|[http://pdns.daloo.de/](http://pdns.daloo.de/)|Free API
-OpenCorporates|Look up company information from OpenCorporates.|[https://opencorporates.com](https://opencorporates.com)|Tiered API
-OpenDNS|Check if a host would be blocked by OpenDNS.|[https://www.opendns.com/](https://www.opendns.com/)|Free API
-OpenNIC DNS|Resolves host names in the OpenNIC alternative DNS system.|[https://www.opennic.org/](https://www.opennic.org/)|Free API
-OpenPhish|Check if a host/domain is malicious according to OpenPhish.com.|[https://openphish.com/](https://openphish.com/)|Free API
-OpenStreetMap|Retrieves latitude/longitude coordinates for physical addresses from OpenStreetMap API.|[https://www.openstreetmap.org/](https://www.openstreetmap.org/)|Free API
-Page Information|Obtain information about web pages (do they take passwords, do they contain forms, etc.)|N/A|Internal
-PasteBin|PasteBin search (via Google Search API) to identify related content.|[https://pastebin.com/](https://pastebin.com/)|Tiered API
-PGP Key Servers|Look up e-mail addresses in PGP public key servers.|N/A|Internal
-PhishStats|Check if a netblock or IP address is malicious according to PhishStats.|[https://phishstats.info/](https://phishstats.info/)|Free API
-PhishTank|Check if a host/domain is malicious according to PhishTank.|[https://phishtank.com/](https://phishtank.com/)|Free API
-Phone Number Extractor|Identify phone numbers in scraped webpages.|N/A|Internal
-Port Scanner - TCP|Scans for commonly open TCP ports on Internet-facing systems.|N/A|Internal
-Project Honey Pot|Query the Project Honey Pot database for IP addresses.|[https://www.projecthoneypot.org/](https://www.projecthoneypot.org/)|Free API
-ProjectDiscovery Chaos|Search for hosts/subdomains using chaos.projectdiscovery.io|[https://chaos.projectdiscovery.io](https://chaos.projectdiscovery.io)|Commercial API
-Psbdmp|Check psbdmp.cc (PasteBin Dump) for potentially hacked e-mails and domains.|[https://psbdmp.cc/](https://psbdmp.cc/)|Free API
-Pulsedive|Obtain information from Pulsedive's API.|[https://pulsedive.com/](https://pulsedive.com/)|Tiered API
-PunkSpider|Check the QOMPLX punkspider.io service to see if the target is listed as vulnerable.|[https://punkspider.io/](https://punkspider.io/)|Free API
-Quad9|Check if a host would be blocked by Quad9 DNS.|[https://quad9.net/](https://quad9.net/)|Free API
-Recon.dev|Search Recon.dev for subdomains.|[https://recon.dev](https://recon.dev)|Free API
-ReverseWhois|Reverse Whois lookups using reversewhois.io.|[https://www.reversewhois.io/](https://www.reversewhois.io/)|Free API
-RIPE|Queries the RIPE registry (includes ARIN data) to identify netblocks and other info.|[https://www.ripe.net/](https://www.ripe.net/)|Free API
-RiskIQ|Obtain information from RiskIQ's (formerly PassiveTotal) Passive DNS and Passive SSL databases.|[https://community.riskiq.com/](https://community.riskiq.com/)|Tiered API
-Robtex|Search Robtex.com for hosts sharing the same IP.|[https://www.robtex.com/](https://www.robtex.com/)|Free API
-Scylla|Gather breach data from Scylla API.|[https://scylla.so/](https://scylla.so/)|Free API
-searchcode|Search searchcode for code repositories mentioning the target domain.|[https://searchcode.com/](https://searchcode.com/)|Free API
-SecurityTrails|Obtain Passive DNS and other information from SecurityTrails|[https://securitytrails.com/](https://securitytrails.com/)|Tiered API
-Seon|Queries seon.io to gather intelligence about IP Addresses, email addresses, and phone numbers|[https://seon.io/](https://seon.io/)|Commercial API
-SHODAN|Obtain information from SHODAN about identified IP addresses.|[https://www.shodan.io/](https://www.shodan.io/)|Tiered API
-Similar Domain Finder|Search various sources to identify similar looking domain names, for instance squatted domains.|N/A|Internal
-Skymem|Look up e-mail addresses on Skymem.|[http://www.skymem.info/](http://www.skymem.info/)|Free API
-SlideShare|Gather name and location from SlideShare profiles.|[https://www.slideshare.net](https://www.slideshare.net)|Free API
-Snov|Gather available email IDs from identified domains|[https://snov.io/](https://snov.io/)|Tiered API
-Social Links|Queries SocialLinks.io to gather intelligence from social media platforms and dark web.|[https://sociallinks.io/](https://sociallinks.io/)|Commercial API
-Social Media Profile Finder|Tries to discover the social media profiles for human names identified.|[https://developers.google.com/custom-search](https://developers.google.com/custom-search)|Tiered API
-Social Network Identifier|Identify presence on social media networks such as LinkedIn, Twitter and others.|N/A|Internal
-SORBS|Query the SORBS database for open relays, open proxies, vulnerable servers, etc.|[http://www.sorbs.net/](http://www.sorbs.net/)|Free API
-SpamCop|Check if a netblock or IP address is in the SpamCop database.|[https://www.spamcop.net/](https://www.spamcop.net/)|Free API
-Spamhaus Zen|Check if a netblock or IP address is in the Spamhaus Zen database.|[https://www.spamhaus.org/](https://www.spamhaus.org/)|Free API
-spur.us|Obtain information about any malicious activities involving IP addresses found|[https://spur.us/](https://spur.us/)|Commercial API
-SpyOnWeb|Search SpyOnWeb for hosts sharing the same IP address, Google Analytics code, or Google Adsense code.|[http://spyonweb.com/](http://spyonweb.com/)|Tiered API
-Spyse|Search Spyse.com Internet assets registry for information about domains, IP addresses, host info, potential vulnerabilities, passive DNS, etc.|[https://spyse.com](https://spyse.com)|Tiered API
-SSL Certificate Analyzer|Gather information about SSL certificates used by the target's HTTPS sites.|N/A|Internal
-Steven Black Hosts|Check if a domain is malicious (malware or adware) according to Steven Black Hosts list.|[https://github.com/StevenBlack/hosts](https://github.com/StevenBlack/hosts)|Free API
-Strange Header Identifier|Obtain non-standard HTTP headers returned by web servers.|N/A|Internal
-Subdomain Takeover Checker|Check if affiliated subdomains are vulnerable to takeover.|N/A|Internal
-Sublist3r PassiveDNS|Passive subdomain enumeration using Sublist3r's API|[https://api.sublist3r.com](https://api.sublist3r.com)|Free API
-SURBL|Check if a netblock, IP address or domain is in the SURBL blacklist.|[http://www.surbl.org/](http://www.surbl.org/)|Free API
-Talos Intelligence|Check if a netblock or IP address is malicious according to TalosIntelligence.|[https://talosintelligence.com/](https://talosintelligence.com/)|Free API
-TextMagic|Obtain phone number type from TextMagic API|[https://www.textmagic.com/](https://www.textmagic.com/)|Tiered API
-ThreatCrowd|Obtain information from ThreatCrowd about identified IP addresses, domains and e-mail addresses.|[https://www.threatcrowd.org](https://www.threatcrowd.org)|Free API
-ThreatFox|Check if an IP address is malicious according to ThreatFox.|[https://threatfox.abuse.ch](https://threatfox.abuse.ch)|Free API
-ThreatMiner|Obtain information from ThreatMiner's database for passive DNS and threat intelligence.|[https://www.threatminer.org/](https://www.threatminer.org/)|Free API
-TLD Searcher|Search all Internet TLDs for domains with the same name as the target (this can be very slow.)|N/A|Internal
-Tool - CMSeeK|Identify what Content Management System (CMS) might be used.|[https://github.com/Tuhinshubhra/CMSeeK](https://github.com/Tuhinshubhra/CMSeeK)|Tool
-Tool - DNSTwist|Identify bit-squatting, typo and other similar domains to the target using a local DNSTwist installation.|[https://github.com/elceef/dnstwist](https://github.com/elceef/dnstwist)|Tool
-Tool - Nmap|Identify what Operating System might be used.|[https://nmap.org/](https://nmap.org/)|Tool
-Tool - WAFW00F|Identify what web application firewall (WAF) is in use on the specified website.|[https://github.com/EnableSecurity/wafw00f](https://github.com/EnableSecurity/wafw00f)|Tool
-Tool - WhatWeb|Identify what software is in use on the specified website.|[https://github.com/urbanadventurer/whatweb](https://github.com/urbanadventurer/whatweb)|Tool
-TOR Exit Nodes|Check if an IP adddress or netblock appears on the Tor Metrics exit node list.|[https://metrics.torproject.org/](https://metrics.torproject.org/)|Free API
-TORCH|Search Tor 'TORCH' search engine for mentions of the target domain.|[https://torchsearch.wordpress.com/](https://torchsearch.wordpress.com/)|Free API
-Trashpanda|Queries Trashpanda to gather intelligence about mentions of target in pastesites|[https://got-hacked.wtf](https://got-hacked.wtf)|Tiered API
-Trumail|Check whether an email is disposable|[https://trumail.io/](https://trumail.io/)|Free API
-Twilio|Obtain information from Twilio about phone numbers. Ensure you have the Caller Name add-on installed in Twilio.|[https://www.twilio.com/](https://www.twilio.com/)|Tiered API
-Twitter|Gather name and location from Twitter profiles.|[https://twitter.com/](https://twitter.com/)|Free API
-UCEPROTECT|Check if a netblock or IP address is in the UCEPROTECT database.|[http://www.uceprotect.net/](http://www.uceprotect.net/)|Free API
-URLScan.io|Search URLScan.io cache for domain information.|[https://urlscan.io/](https://urlscan.io/)|Free API
-Venmo|Gather user information from Venmo API.|[https://venmo.com/](https://venmo.com/)|Free API
-ViewDNS.info|Reverse Whois lookups using ViewDNS.info.|[https://viewdns.info/](https://viewdns.info/)|Tiered API
-VirusTotal|Obtain information from VirusTotal about identified IP addresses.|[https://www.virustotal.com/](https://www.virustotal.com/)|Tiered API
-VoIP Blacklist (VoIPBL)|Check if an IP address or netblock is malicious according to VoIP Blacklist (VoIPBL).|[https://voipbl.org/](https://voipbl.org/)|Free API
-VXVault.net|Check if a domain or IP address is malicious according to VXVault.net.|[http://vxvault.net/](http://vxvault.net/)|Free API
-Web Analytics Extractor|Identify web analytics IDs in scraped webpages and DNS TXT records.|N/A|Internal
-Web Framework Identifier|Identify the usage of popular web frameworks like jQuery, YUI and others.|N/A|Internal
-Web Server Identifier|Obtain web server banners to identify versions of web servers being used.|N/A|Internal
-Web Spider|Spidering of web-pages to extract content for searching.|N/A|Internal
-WhatCMS|Check web technology using WhatCMS.org API.|[https://whatcms.org/](https://whatcms.org/)|Tiered API
-Whoisology|Reverse Whois lookups using Whoisology.com.|[https://whoisology.com/](https://whoisology.com/)|Commercial API
-Whois|Perform a WHOIS look-up on domain names and owned netblocks.|N/A|Internal
-Whoxy|Reverse Whois lookups using Whoxy.com.|[https://www.whoxy.com/](https://www.whoxy.com/)|Commercial API
-WiGLE|Query WiGLE to identify nearby WiFi access points.|[https://wigle.net/](https://wigle.net/)|Free API
-Wikileaks|Search Wikileaks for mentions of domain names and e-mail addresses.|[https://wikileaks.org/](https://wikileaks.org/)|Free API
-Wikipedia Edits|Identify edits to Wikipedia articles made from a given IP address or username.|[https://www.wikipedia.org/](https://www.wikipedia.org/)|Free API
-XForce Exchange|Obtain IP reputation and passive DNS information from IBM X-Force Exchange.|[https://exchange.xforce.ibmcloud.com/](https://exchange.xforce.ibmcloud.com/)|Tiered API
-Yandex DNS|Check if a host would be blocked by Yandex DNS.|[https://yandex.com/](https://yandex.com/)|Free API
-Zetalytics|Query the Zetalytics database for hosts on your target domain(s).|[https://zetalytics.com/](https://zetalytics.com/)|Tiered API
-Zone-H Defacement Check|Check if a hostname/domain appears on the zone-h.org 'special defacements' RSS feed.|[https://zone-h.org/](https://zone-h.org/)|Free API
+| Name | Description | Type |
+|:---------| :-----------|:-------|
+[AbstractAPI](https://app.abstractapi.com/)|Look up domain, phone and IP address information from AbstractAPI.|Tiered API
+[abuse.ch](https://www.abuse.ch)|Check if a host/domain, IP address or netblock is malicious according to Abuse.ch.|Free API
+[AbuseIPDB](https://www.abuseipdb.com)|Check if an IP address is malicious according to AbuseIPDB.com blacklist.|Tiered API
+[Abusix Mail Intelligence](https://abusix.org/)|Check if a netblock or IP address is in the Abusix Mail Intelligence blacklist.|Tiered API
+Account Finder|Look for possible associated accounts on nearly 200 websites like Ebay, Slashdot, reddit, etc.|Internal
+[AdBlock Check](https://adblockplus.org/)|Check if linked pages would be blocked by AdBlock Plus.|Tiered API
+[AdGuard DNS](https://adguard.com/)|Check if a host would be blocked by AdGuard DNS.|Free API
+[Ahmia](https://ahmia.fi/)|Search Tor 'Ahmia' search engine for mentions of the target.|Free API
+[AlienVault IP Reputation](https://cybersecurity.att.com/)|Check if an IP or netblock is malicious according to the AlienVault IP Reputation database.|Free API
+[AlienVault OTX](https://otx.alienvault.com/)|Obtain information from AlienVault Open Threat Exchange (OTX)|Tiered API
+[Amazon S3 Bucket Finder](https://aws.amazon.com/s3/)|Search for potential Amazon S3 buckets associated with the target and attempt to list their contents.|Free API
+[Apple iTunes](https://itunes.apple.com/)|Search Apple iTunes for mobile apps.|Free API
+[Archive.org](https://archive.org/)|Identifies historic versions of interesting files/pages from the Wayback Machine.|Free API
+[ARIN](https://www.arin.net/)|Queries ARIN registry for contact information.|Free API
+[Azure Blob Finder](https://azure.microsoft.com/en-in/services/storage/blobs/)|Search for potential Azure blobs associated with the target and attempt to list their contents.|Free API
+[Bad Packets](https://badpackets.net)|Obtain information about any malicious activities involving IP addresses found|Commercial API
+Base64 Decoder|Identify Base64-encoded strings in URLs, often revealing interesting hidden information.|Internal
+[BGPView](https://bgpview.io/)|Obtain network information from BGPView API.|Free API
+Binary String Extractor|Attempt to identify strings in binary content.|Internal
+[BinaryEdge](https://www.binaryedge.io/)|Obtain information from BinaryEdge.io Internet scanning systems, including breaches, vulnerabilities, torrents and passive DNS.|Tiered API
+[Bing (Shared IPs)](https://www.bing.com/)|Search Bing for hosts sharing the same IP.|Tiered API
+[Bing](https://www.bing.com/)|Obtain information from bing to identify sub-domains and links.|Tiered API
+Bitcoin Finder|Identify bitcoin addresses in scraped webpages.|Internal
+[Bitcoin Who's Who](https://bitcoinwhoswho.com/)|Check for Bitcoin addresses against the Bitcoin Who's Who database of suspect/malicious addresses.|Tiered API
+[BitcoinAbuse](https://www.bitcoinabuse.com/)|Check Bitcoin addresses against the bitcoinabuse.com database of suspect/malicious addresses.|Free API
+[Blockchain](https://www.blockchain.com/)|Queries blockchain.info to find the balance of identified bitcoin wallet addresses.|Free API
+[blocklist.de](http://www.blocklist.de/en/index.html)|Check if a netblock or IP is malicious according to blocklist.de.|Free API
+[BotScout](https://botscout.com/)|Searches BotScout.com's database of spam-bot IP addresses and e-mail addresses.|Tiered API
+[botvrij.eu](https://botvrij.eu/)|Check if a domain is malicious according to botvrij.eu.|Free API
+[BuiltWith](https://builtwith.com/)|Query BuiltWith.com's Domain API for information about your target's web technology stack, e-mail addresses and more.|Tiered API
+[C99](https://api.c99.nl/)|Queries the C99 API which offers various data (geo location, proxy detection, phone lookup, etc).|Commercial API
+[CallerName](http://callername.com/)|Lookup US phone number location and reputation information.|Free API
+[Censys](https://censys.io/)|Obtain host information from Censys.io.|Tiered API
+[Certificate Transparency](https://crt.sh/)|Gather hostnames from historical certificates in crt.sh.|Free API
+[CertSpotter](https://sslmate.com/certspotter/)|Gather information about SSL certificates from SSLMate CertSpotter API.|Tiered API
+[CINS Army List](https://cinsscore.com/)|Check if a netblock or IP address is malicious according to Collective Intelligence Network Security (CINS) Army list.|Free API
+[CIRCL.LU](https://www.circl.lu/)|Obtain information from CIRCL.LU's Passive DNS and Passive SSL databases.|Free API
+[CleanBrowsing.org](https://cleanbrowsing.org/)|Check if a host would be blocked by CleanBrowsing.org DNS content filters.|Free API
+[CleanTalk Spam List](https://cleantalk.org)|Check if a netblock or IP address is on CleanTalk.org's spam IP list.|Free API
+[Clearbit](https://clearbit.com/)|Check for names, addresses, domains and more based on lookups of e-mail addresses on clearbit.com.|Tiered API
+[CloudFlare DNS](https://www.cloudflare.com/)|Check if a host would be blocked by CloudFlare DNS.|Free API
+[CoinBlocker Lists](https://zerodot1.gitlab.io/CoinBlockerListsWeb/)|Check if a domain appears on CoinBlocker lists.|Free API
+[CommonCrawl](http://commoncrawl.org/)|Searches for URLs found through CommonCrawl.org.|Free API
+[Comodo Secure DNS](https://www.comodo.com/secure-dns/)|Check if a host would be blocked by Comodo Secure DNS.|Tiered API
+Company Name Extractor|Identify company names in any obtained data.|Internal
+Cookie Extractor|Extract Cookies from HTTP headers.|Internal
+Country Name Extractor|Identify country names in any obtained data.|Internal
+Credit Card Number Extractor|Identify Credit Card Numbers in any data|Internal
+[Crobat API](https://sonar.omnisint.io/)|Search Crobat API for subdomains.|Free API
+Cross-Referencer|Identify whether other domains are associated ('Affiliates') of the target by looking for links back to the target site(s).|Internal
+[CRXcavator](https://crxcavator.io/)|Search CRXcavator for Chrome extensions.|Free API
+Custom Threat Feed|Check if a host/domain, netblock, ASN or IP is malicious according to your custom feed.|Internal
+[CyberCrime-Tracker.net](https://cybercrime-tracker.net/)|Check if a host/domain or IP address is malicious according to CyberCrime-Tracker.net.|Free API
+[Darksearch](https://darksearch.io/)|Search the Darksearch.io Tor search engine for mentions of the target domain.|Free API
+[Debounce](https://debounce.io/)|Check whether an email is disposable|Free API
+[Dehashed](https://www.dehashed.com/)|Gather breach data from Dehashed API.|Commercial API
+[Digital Ocean Space Finder](https://www.digitalocean.com/products/spaces/)|Search for potential Digital Ocean Spaces associated with the target and attempt to list their contents.|Free API
+DNS Brute-forcer|Attempts to identify hostnames through brute-forcing common names and iterations.|Internal
+DNS Common SRV|Attempts to identify hostnames through brute-forcing common DNS SRV records.|Internal
+[DNS for Family](https://dnsforfamily.com/)|Check if a host would be blocked by DNS for Family.|Free API
+DNS Look-aside|Attempt to reverse-resolve the IP addresses next to your target to see if they are related.|Internal
+DNS Raw Records|Retrieves raw DNS records such as MX, TXT and others.|Internal
+DNS Resolver|Resolves hosts and IP addresses identified, also extracted from raw content.|Internal
+DNS Zone Transfer|Attempts to perform a full DNS zone transfer.|Internal
+[DNSDB](https://www.farsightsecurity.com)|Query FarSight's DNSDB for historical and passive DNS data.|Tiered API
+[DNSDumpster](https://dnsdumpster.com/)|Passive subdomain enumeration using HackerTarget's DNSDumpster|Free API
+[DNSGrep](https://opendata.rapid7.com/)|Obtain Passive DNS information from Rapid7 Sonar Project using DNSGrep API.|Free API
+[DroneBL](https://dronebl.org/)|Query the DroneBL database for open relays, open proxies, vulnerable servers, etc.|Free API
+[DuckDuckGo](https://duckduckgo.com/)|Query DuckDuckGo's API for descriptive information about your target.|Free API
+E-Mail Address Extractor|Identify e-mail addresses in any obtained data.|Internal
+[EmailCrawlr](https://emailcrawlr.com/)|Search EmailCrawlr for email addresses and phone numbers associated with a domain.|Tiered API
+[EmailFormat](https://www.email-format.com/)|Look up e-mail addresses on email-format.com.|Free API
+[EmailRep](https://emailrep.io/)|Search EmailRep.io for email address reputation.|Tiered API
+[Emerging Threats](https://rules.emergingthreats.net/)|Check if a netblock or IP address is malicious according to EmergingThreats.net.|Free API
+Error String Extractor|Identify common error messages in content like SQL errors, etc.|Internal
+Ethereum Address Extractor|Identify ethereum addresses in scraped webpages.|Internal
+[Etherscan](https://etherscan.io)|Queries etherscan.io to find the balance of identified ethereum wallet addresses.|Free API
+File Metadata Extractor|Extracts meta data from documents and images.|Internal
+[Flickr](https://www.flickr.com/)|Search Flickr for domains, URLs and emails related to the specified domain.|Free API
+[Focsec](https://focsec.com/)|Look up IP address information from Focsec.|Tiered API
+[FortiGuard Antispam](https://www.fortiguard.com/)|Check if an IP address is malicious according to FortiGuard Antispam.|Free API
+[Fraudguard](https://fraudguard.io/)|Obtain threat information from Fraudguard.io|Tiered API
+[F-Secure Riddler.io](https://riddler.io/)|Obtain network information from F-Secure Riddler.io API.|Commercial API
+[FullContact](https://www.fullcontact.com)|Gather domain and e-mail information from FullContact.com API.|Tiered API
+[FullHunt](https://fullhunt.io/)|Identify domain attack surface using FullHunt API.|Tiered API
+[Github](https://github.com/)|Identify associated public code repositories on Github.|Free API
+[GLEIF](https://search.gleif.org/)|Look up company information from Global Legal Entity Identifier Foundation (GLEIF).|Tiered API
+[Google Maps](https://cloud.google.com/maps-platform/)|Identifies potential physical addresses and latitude/longitude coordinates.|Tiered API
+[Google Object Storage Finder](https://cloud.google.com/storage)|Search for potential Google Object Storage buckets associated with the target and attempt to list their contents.|Free API
+[Google SafeBrowsing](https://developers.google.com/safe-browsing/v4/lookup-api)|Check if the URL is included on any of the Safe Browsing lists.|Free API
+[Google](https://developers.google.com/custom-search)|Obtain information from the Google Custom Search API to identify sub-domains and links.|Tiered API
+[Gravatar](https://secure.gravatar.com/)|Retrieve user information from Gravatar API.|Free API
+[Grayhat Warfare](https://buckets.grayhatwarfare.com/)|Find bucket names matching the keyword extracted from a domain from Grayhat API.|Tiered API
+[Greensnow](https://greensnow.co/)|Check if a netblock or IP address is malicious according to greensnow.co.|Free API
+[grep.app](https://grep.app/)|Search grep.app API for links and emails related to the specified domain.|Free API
+[GreyNoise](https://greynoise.io/)|Obtain IP enrichment data from GreyNoise|Tiered API
+[HackerOne (Unofficial)](http://www.nobbd.de/)|Check external vulnerability scanning/reporting service h1.nobbd.de to see if the target is listed.|Free API
+[HackerTarget](https://hackertarget.com/)|Search HackerTarget.com for hosts sharing the same IP.|Free API
+Hash Extractor|Identify MD5 and SHA hashes in web content, files and more.|Internal
+[HaveIBeenPwned](https://haveibeenpwned.com/)|Check HaveIBeenPwned.com for hacked e-mail addresses identified in breaches.|Commercial API
+Hosting Provider Identifier|Find out if any IP addresses identified fall within known 3rd party hosting ranges, e.g. Amazon, Azure, etc.|Internal
+[Host.io](https://host.io)|Obtain information about domain names from host.io.|Tiered API
+Human Name Extractor|Attempt to identify human names in fetched content.|Internal
+[Hunter.io](https://hunter.io/)|Check for e-mail addresses and names on hunter.io.|Tiered API
+[Hybrid Analysis](https://www.hybrid-analysis.com)|Search Hybrid Analysis for domains and URLs related to the target.|Free API
+IBAN Number Extractor|Identify International Bank Account Numbers (IBANs) in any data.|Internal
+[Iknowwhatyoudownload.com](https://iknowwhatyoudownload.com/en/peer/)|Check iknowwhatyoudownload.com for IP addresses that have been using torrents.|Tiered API
+[Instagram](https://www.instagram.com/)|Gather information from Instagram profiles.|Free API
+[IntelligenceX](https://intelx.io/)|Obtain information from IntelligenceX about identified IP addresses, domains, e-mail addresses and phone numbers.|Tiered API
+Interesting File Finder|Identifies potential files of interest, e.g. office documents, zip files.|Internal
+[Internet Storm Center](https://isc.sans.edu)|Check if an IP address is malicious according to SANS ISC.|Free API
+[ipapi.co](https://ipapi.co/)|Queries ipapi.co to identify geolocation of IP Addresses using ipapi.co API|Tiered API
+[ipapi.com](https://ipapi.com/)|Queries ipapi.com to identify geolocation of IP Addresses using ipapi.com API|Tiered API
+[IPInfo.io](https://ipinfo.io)|Identifies the physical location of IP addresses identified using ipinfo.io.|Tiered API
+[IPQualityScore](https://www.ipqualityscore.com/)|Determine if target is malicious using IPQualityScore API|Tiered API
+[ipregistry](https://ipregistry.co/)|Query the ipregistry.co database for reputation and geo-location.|Tiered API
+[ipstack](https://ipstack.com/)|Identifies the physical location of IP addresses identified using ipstack.com.|Tiered API
+[JsonWHOIS.com](https://jsonwhois.com)|Search JsonWHOIS.com for WHOIS records associated with a domain.|Tiered API
+Junk File Finder|Looks for old/temporary and other similar files.|Internal
+[Keybase](https://keybase.io/)|Obtain additional information about domain names and identified usernames.|Free API
+[Koodous](https://koodous.com/apks/)|Search Koodous for mobile apps.|Free API
+[LeakIX](https://leakix.net/)|Search LeakIX for host data leaks, open ports, software and geoip.|Free API
+[Leak-Lookup](https://leak-lookup.com/)|Searches Leak-Lookup.com's database of breaches.|Free API
+[Maltiverse](https://maltiverse.com)|Obtain information about any malicious activities involving IP addresses|Free API
+[MalwarePatrol](https://www.malwarepatrol.net/)|Searches malwarepatrol.net's database of malicious URLs/IPs.|Tiered API
+[MetaDefender](https://metadefender.opswat.com/)|Search MetaDefender API for IP address and domain IP reputation.|Tiered API
+[Mnemonic PassiveDNS](https://www.mnemonic.no)|Obtain Passive DNS information from PassiveDNS.mnemonic.no.|Free API
+[multiproxy.org Open Proxies](https://multiproxy.org/)|Check if an IP address is an open proxy according to multiproxy.org open proxy list.|Free API
+[MySpace](https://myspace.com/)|Gather username and location from MySpace.com profiles.|Free API
+[NameAPI](https://www.nameapi.org/)|Check whether an email is disposable|Tiered API
+[NetworksDB](https://networksdb.io/)|Search NetworksDB.io API for IP address and domain information.|Tiered API
+[NeutrinoAPI](https://www.neutrinoapi.com/)|Search NeutrinoAPI for phone location information, IP address information, and host reputation.|Tiered API
+[numverify](http://numverify.com/)|Lookup phone number location and carrier information from numverify.com.|Tiered API
+[Onion.link](https://onion.link/)|Search Tor 'Onion City' search engine for mentions of the target domain using Google Custom Search.|Free API
+[Onionsearchengine.com](https://as.onionsearchengine.com)|Search Tor onionsearchengine.com for mentions of the target domain.|Free API
+[Onyphe](https://www.onyphe.io)|Check Onyphe data (threat list, geo-location, pastries, vulnerabilities) about a given IP.|Tiered API
+[Open Bug Bounty](https://www.openbugbounty.org/)|Check external vulnerability scanning/reporting service openbugbounty.org to see if the target is listed.|Free API
+[Open Passive DNS Database](http://pdns.daloo.de/)|Obtain passive DNS information from pdns.daloo.de Open passive DNS database.|Free API
+[OpenCorporates](https://opencorporates.com)|Look up company information from OpenCorporates.|Tiered API
+[OpenDNS](https://www.opendns.com/)|Check if a host would be blocked by OpenDNS.|Free API
+[OpenNIC DNS](https://www.opennic.org/)|Resolves host names in the OpenNIC alternative DNS system.|Free API
+[OpenPhish](https://openphish.com/)|Check if a host/domain is malicious according to OpenPhish.com.|Free API
+[OpenStreetMap](https://www.openstreetmap.org/)|Retrieves latitude/longitude coordinates for physical addresses from OpenStreetMap API.|Free API
+Page Information|Obtain information about web pages (do they take passwords, do they contain forms, etc.)|Internal
+[PasteBin](https://pastebin.com/)|PasteBin search (via Google Search API) to identify related content.|Tiered API
+PGP Key Servers|Look up e-mail addresses in PGP public key servers.|Internal
+[PhishStats](https://phishstats.info/)|Check if a netblock or IP address is malicious according to PhishStats.|Free API
+[PhishTank](https://phishtank.com/)|Check if a host/domain is malicious according to PhishTank.|Free API
+Phone Number Extractor|Identify phone numbers in scraped webpages.|Internal
+Port Scanner - TCP|Scans for commonly open TCP ports on Internet-facing systems.|Internal
+[Project Honey Pot](https://www.projecthoneypot.org/)|Query the Project Honey Pot database for IP addresses.|Free API
+[ProjectDiscovery Chaos](https://chaos.projectdiscovery.io)|Search for hosts/subdomains using chaos.projectdiscovery.io|Commercial API
+[Psbdmp](https://psbdmp.cc/)|Check psbdmp.cc (PasteBin Dump) for potentially hacked e-mails and domains.|Free API
+[Pulsedive](https://pulsedive.com/)|Obtain information from Pulsedive's API.|Tiered API
+[PunkSpider](https://punkspider.io/)|Check the QOMPLX punkspider.io service to see if the target is listed as vulnerable.|Free API
+[Quad9](https://quad9.net/)|Check if a host would be blocked by Quad9 DNS.|Free API
+[Recon.dev](https://recon.dev)|Search Recon.dev for subdomains.|Free API
+[ReverseWhois](https://www.reversewhois.io/)|Reverse Whois lookups using reversewhois.io.|Free API
+[RIPE](https://www.ripe.net/)|Queries the RIPE registry (includes ARIN data) to identify netblocks and other info.|Free API
+[RiskIQ](https://community.riskiq.com/)|Obtain information from RiskIQ's (formerly PassiveTotal) Passive DNS and Passive SSL databases.|Tiered API
+[Robtex](https://www.robtex.com/)|Search Robtex.com for hosts sharing the same IP.|Free API
+[Scylla](https://scylla.so/)|Gather breach data from Scylla API.|Free API
+[searchcode](https://searchcode.com/)|Search searchcode for code repositories mentioning the target domain.|Free API
+[SecurityTrails](https://securitytrails.com/)|Obtain Passive DNS and other information from SecurityTrails|Tiered API
+[Seon](https://seon.io/)|Queries seon.io to gather intelligence about IP Addresses, email addresses, and phone numbers|Commercial API
+[SHODAN](https://www.shodan.io/)|Obtain information from SHODAN about identified IP addresses.|Tiered API
+Similar Domain Finder|Search various sources to identify similar looking domain names, for instance squatted domains.|Internal
+[Skymem](http://www.skymem.info/)|Look up e-mail addresses on Skymem.|Free API
+[SlideShare](https://www.slideshare.net)|Gather name and location from SlideShare profiles.|Free API
+[Snov](https://snov.io/)|Gather available email IDs from identified domains|Tiered API
+[Social Links](https://sociallinks.io/)|Queries SocialLinks.io to gather intelligence from social media platforms and dark web.|Commercial API
+[Social Media Profile Finder](https://developers.google.com/custom-search)|Tries to discover the social media profiles for human names identified.|Tiered API
+Social Network Identifier|Identify presence on social media networks such as LinkedIn, Twitter and others.|Internal
+[SORBS](http://www.sorbs.net/)|Query the SORBS database for open relays, open proxies, vulnerable servers, etc.|Free API
+[SpamCop](https://www.spamcop.net/)|Check if a netblock or IP address is in the SpamCop database.|Free API
+[Spamhaus Zen](https://www.spamhaus.org/)|Check if a netblock or IP address is in the Spamhaus Zen database.|Free API
+[spur.us](https://spur.us/)|Obtain information about any malicious activities involving IP addresses found|Commercial API
+[SpyOnWeb](http://spyonweb.com/)|Search SpyOnWeb for hosts sharing the same IP address, Google Analytics code, or Google Adsense code.|Tiered API
+[Spyse](https://spyse.com)|Search Spyse.com Internet assets registry for information about domains, IP addresses, host info, potential vulnerabilities, passive DNS, etc.|Tiered API
+SSL Certificate Analyzer|Gather information about SSL certificates used by the target's HTTPS sites.|Internal
+[StackOverflow](https://www.stackecxchange.com)|Search StackOverflow for any mentions of a target domain. Returns potentially related information.|Tiered API
+[Steven Black Hosts](https://github.com/StevenBlack/hosts)|Check if a domain is malicious (malware or adware) according to Steven Black Hosts list.|Free API
+Strange Header Identifier|Obtain non-standard HTTP headers returned by web servers.|Internal
+Subdomain Takeover Checker|Check if affiliated subdomains are vulnerable to takeover.|Internal
+[Sublist3r PassiveDNS](https://api.sublist3r.com)|Passive subdomain enumeration using Sublist3r's API|Free API
+[SURBL](http://www.surbl.org/)|Check if a netblock, IP address or domain is in the SURBL blacklist.|Free API
+[Talos Intelligence](https://talosintelligence.com/)|Check if a netblock or IP address is malicious according to TalosIntelligence.|Free API
+[TextMagic](https://www.textmagic.com/)|Obtain phone number type from TextMagic API|Tiered API
+[ThreatCrowd](https://www.threatcrowd.org)|Obtain information from ThreatCrowd about identified IP addresses, domains and e-mail addresses.|Free API
+[ThreatFox](https://threatfox.abuse.ch)|Check if an IP address is malicious according to ThreatFox.|Free API
+[ThreatMiner](https://www.threatminer.org/)|Obtain information from ThreatMiner's database for passive DNS and threat intelligence.|Free API
+TLD Searcher|Search all Internet TLDs for domains with the same name as the target (this can be very slow.)|Internal
+[Tool - CMSeeK]([https://github.com/Tuhinshubhra/CMSeeK](https://github.com/Tuhinshubhra/CMSeeK))|Identify what Content Management System (CMS) might be used.|Tool
+[Tool - DNSTwist]([https://github.com/elceef/dnstwist](https://github.com/elceef/dnstwist))|Identify bit-squatting, typo and other similar domains to the target using a local DNSTwist installation.|Tool
+[Tool - nbtscan]([http://www.unixwiz.net/tools/nbtscan.html](http://www.unixwiz.net/tools/nbtscan.html))|Scans for open NETBIOS nameservers on your target's network.|Tool
+[Tool - Nmap]([https://nmap.org/](https://nmap.org/))|Identify what Operating System might be used.|Tool
+[Tool - Nuclei]([https://nuclei.projectdiscovery.io/](https://nuclei.projectdiscovery.io/))|Fast and customisable vulnerability scanner.|Tool
+[Tool - onesixtyone]([https://github.com/trailofbits/onesixtyone](https://github.com/trailofbits/onesixtyone))|Fast scanner to find publicly exposed SNMP services.|Tool
+[Tool - Retire.js]([http://retirejs.github.io/retire.js/](http://retirejs.github.io/retire.js/))|Scanner detecting the use of JavaScript libraries with known vulnerabilities|Tool
+[Tool - snallygaster]([https://github.com/hannob/snallygaster](https://github.com/hannob/snallygaster))|Finds file leaks and other security problems on HTTP servers.|Tool
+[Tool - testssl.sh]([https://testssl.sh](https://testssl.sh))|Identify various TLS/SSL weaknesses, including Heartbleed, CRIME and ROBOT.|Tool
+[Tool - TruffleHog]([https://github.com/trufflesecurity/truffleHog](https://github.com/trufflesecurity/truffleHog))|Searches through git repositories for high entropy strings and secrets, digging deep into commit history.|Tool
+[Tool - WAFW00F]([https://github.com/EnableSecurity/wafw00f](https://github.com/EnableSecurity/wafw00f))|Identify what web application firewall (WAF) is in use on the specified website.|Tool
+[Tool - Wappalyzer]([https://www.wappalyzer.com/](https://www.wappalyzer.com/))|Wappalyzer indentifies technologies on websites.|Tool
+[Tool - WhatWeb]([https://github.com/urbanadventurer/whatweb](https://github.com/urbanadventurer/whatweb))|Identify what software is in use on the specified website.|Tool
+[TOR Exit Nodes](https://metrics.torproject.org/)|Check if an IP adddress or netblock appears on the Tor Metrics exit node list.|Free API
+[TORCH](https://torchsearch.wordpress.com/)|Search Tor 'TORCH' search engine for mentions of the target domain.|Free API
+[Trashpanda](https://got-hacked.wtf)|Queries Trashpanda to gather intelligence about mentions of target in pastesites|Tiered API
+[Trumail](https://trumail.io/)|Check whether an email is disposable|Free API
+[Twilio](https://www.twilio.com/)|Obtain information from Twilio about phone numbers. Ensure you have the Caller Name add-on installed in Twilio.|Tiered API
+[Twitter](https://twitter.com/)|Gather name and location from Twitter profiles.|Free API
+[UCEPROTECT](http://www.uceprotect.net/)|Check if a netblock or IP address is in the UCEPROTECT database.|Free API
+[URLScan.io](https://urlscan.io/)|Search URLScan.io cache for domain information.|Free API
+[Venmo](https://venmo.com/)|Gather user information from Venmo API.|Free API
+[ViewDNS.info](https://viewdns.info/)|Identify co-hosted websites and perform reverse Whois lookups using ViewDNS.info.|Tiered API
+[VirusTotal](https://www.virustotal.com/)|Obtain information from VirusTotal about identified IP addresses.|Tiered API
+[VoIP Blacklist (VoIPBL)](https://voipbl.org/)|Check if an IP address or netblock is malicious according to VoIP Blacklist (VoIPBL).|Free API
+[VXVault.net](http://vxvault.net/)|Check if a domain or IP address is malicious according to VXVault.net.|Free API
+Web Analytics Extractor|Identify web analytics IDs in scraped webpages and DNS TXT records.|Internal
+Web Framework Identifier|Identify the usage of popular web frameworks like jQuery, YUI and others.|Internal
+Web Server Identifier|Obtain web server banners to identify versions of web servers being used.|Internal
+Web Spider|Spidering of web-pages to extract content for searching.|Internal
+[WhatCMS](https://whatcms.org/)|Check web technology using WhatCMS.org API.|Tiered API
+[Whoisology](https://whoisology.com/)|Reverse Whois lookups using Whoisology.com.|Commercial API
+Whois|Perform a WHOIS look-up on domain names and owned netblocks.|Internal
+[Whoxy](https://www.whoxy.com/)|Reverse Whois lookups using Whoxy.com.|Commercial API
+[WiGLE](https://wigle.net/)|Query WiGLE to identify nearby WiFi access points.|Free API
+[Wikileaks](https://wikileaks.org/)|Search Wikileaks for mentions of domain names and e-mail addresses.|Free API
+[Wikipedia Edits](https://www.wikipedia.org/)|Identify edits to Wikipedia articles made from a given IP address or username.|Free API
+[XForce Exchange](https://exchange.xforce.ibmcloud.com/)|Obtain IP reputation and passive DNS information from IBM X-Force Exchange.|Tiered API
+[Yandex DNS](https://yandex.com/)|Check if a host would be blocked by Yandex DNS.|Free API
+[Zetalytics](https://zetalytics.com/)|Query the Zetalytics database for hosts on your target domain(s).|Tiered API
+[Zone-H Defacement Check](https://zone-h.org/)|Check if a hostname/domain appears on the zone-h.org 'special defacements' RSS feed.|Free API
### DOCUMENTATION
diff --git a/VERSION b/VERSION
index b246d5c178..90d8b300db 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-SpiderFoot 3.5
+SpiderFoot 4.0.0
diff --git a/sfcli.py b/sfcli.py
index b9fe49e4fc..a1abbd4339 100755
--- a/sfcli.py
+++ b/sfcli.py
@@ -53,7 +53,7 @@ class bcolors:
class SpiderFootCli(cmd.Cmd):
- version = "3.5.0"
+ version = "4.0.0"
pipecmd = None
output = None
modules = []
diff --git a/spiderfoot/__version__.py b/spiderfoot/__version__.py
index 6c44de1513..d165c4420f 100644
--- a/spiderfoot/__version__.py
+++ b/spiderfoot/__version__.py
@@ -1,3 +1,3 @@
-VERSION = (3, 5, 0)
+VERSION = (4, 0, 0)
__version__ = '.'.join(map(str, VERSION))