Skip to content

Commit

Permalink
Refactor documentation
Browse files Browse the repository at this point in the history
* This commit changes most of the markdown files from the user documentation
  to restructured text
* The autogen.py module was updated to write rst format for the Feeds
  file
* Some of the content was refactored: some of the instroduction was moved to
  the index page of the documentation. The User-Guide was renamed
  to 'Configuration and Management'.
* The user related guides are now in the subdirectory `user`, the
  development related content is in the subdirectory `dev`
* The architecture document and the autogenerated harmonization field
  document are still markdown

Closes certtools#1636
Birger Schacht committed Oct 29, 2020
1 parent 5cec1d8 commit 2910dc1
Showing 34 changed files with 3,403 additions and 3,730 deletions.
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -23,8 +23,8 @@ src/
# sphinx
docs/source
docs/_build
docs/guides/Feeds.md
docs/guides/Harmonization-fields.md
docs/user/feeds.rst
docs/dev/harmonization-fields.md

# Debian build filed
debian/files
File renamed without changes
File renamed without changes
File renamed without changes.
File renamed without changes
File renamed without changes
37 changes: 18 additions & 19 deletions docs/autogen.py
Original file line number Diff line number Diff line change
@@ -74,34 +74,26 @@ def feeds_docs():
with open(os.path.join(BASEDIR, 'intelmq/etc/feeds.yaml')) as fhandle:
config = yaml.safe_load(fhandle.read())

toc = ""
for provider in sorted(config['providers'].keys()):
provider_link = provider.replace('.', '')
provider_link = provider_link.replace(' ', '-')
toc += "- [%s](#%s)\n" % (provider, provider_link.lower())

output = """# Available Feeds
output = """Feeds
======
The available feeds are grouped by the provider of the feeds.
For each feed the collector and parser that can be used is documented as well as any feed-specific parameters.
To add feeds to this file add them to `intelmq/etc/feeds.yaml` and then run
`intelmq/bin/intelmq_gen_feeds_docs.py` to generate the new content of this file.
<!-- TOC depthFrom:2 depthTo:2 withLinks:1 updateOnSave:1 orderedList:0 -->
%s
To add feeds to this file add them to `intelmq/etc/feeds.yaml` and then rebuild the documentation.
<!-- /TOC -->\n
.. contents ::
""" % toc
"""

for provider, feeds in sorted(config['providers'].items(), key=lambda x: x[0]):

output += "## %s\n\n" % provider
output += f"{provider}\n"
output += "-"*len(provider) + "\n"

for feed, feed_info in sorted(feeds.items(), key=lambda x: x[0]):

output += "### %s\n\n" % feed
output += f"{feed}\n"
output += "^"*len(feed) + "\n"

if feed_info.get('public'):
output += info("public", "yes" if feed_info['public'] else "no")
@@ -122,7 +114,7 @@ def feeds_docs():

for bot, bot_info in sorted(feed_info['bots'].items(), key=lambda x: x[0]):

output += "#### %s\n\n" % bot.title()
output += "**%s**\n\n" % bot.title()

output += info("Module", bot_info['module'])
output += info("Configuration Parameters")
@@ -141,10 +133,17 @@ def feeds_docs():
if isinstance(value, (list, tuple)) and value:
value = '["%s"]' % '", "'.join(value)

output += "* * `%s`: `%s`\n" % (key, value)
output += " * `%s`: `%s`\n" % (key, value)

output += '\n'

output += '\n'

return output


if __name__ == '__main__': # pragma: no cover
with open('guides/Harmonization-fields.md', 'w') as handle:
handle.write(harm_docs())
with open('user/feeds.rst', 'w') as handle:
handle.write(feeds_docs())
10 changes: 7 additions & 3 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -54,7 +54,11 @@ def resolve_any_xref(self, env, fromdocname, builder, target, node, contnode):
# The full version, including alpha/beta/rc tags
release = '2.3.0'


rst_prolog = """
.. |intelmq-users-list-link| replace:: `IntelMQ Users Mailinglist <https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users>`__
.. |intelmq-developers-list-link| replace:: `IntelMQ Developers Mailinglist <https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev>`__
.. |intelmq-manager-github-link| replace:: `IntelMQ Manager <https://github.com/certtools/intelmq-manager>`__
"""
# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
@@ -116,9 +120,9 @@ def run_apidoc(_):


def run_autogen(_):
with open('guides/Harmonization-fields.md', 'w') as handle:
with open('dev/harmonization-fields.md', 'w') as handle:
handle.write(autogen.harm_docs())
with open('guides/Feeds.md', 'w') as handle:
with open('user/feeds.rst', 'w') as handle:
handle.write(autogen.feeds_docs())


File renamed without changes.
207 changes: 207 additions & 0 deletions docs/dev/data-harmonization.rst

Large diffs are not rendered by default.

82 changes: 82 additions & 0 deletions docs/dev/feeds-wishlist.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
###############
Feeds whishlist
###############

This is a list with various feeds, which are either currently not supported or the usage is not clearly documented in IntelMQ.

If you want to **contribute** documenting how to configure existing bots in order to collect new feeds or by creating new parsers, here is a list of potentially interesting feeds.
See `Feeds documentation <Developers-Guide.md#feeds-documentation>`_ for more information on this.

This list evolved from the issue `Contribute: Feeds List (#384) <https://github.com/certtools/intelmq/issues/384>`_.

- A list of feeds
- `threatfeeds.io <https://threatfeeds.io>`_
- `TheCyberThreat <http://thecyberthreat.com/cyber-threat-intelligence-feeds/>`_

- Some third party intelmq bots: `NRDCS' IntelMQ fork <https://github.com/NRDCS/intelmq/tree/certlt/intelmq/bots>`_

- List of potentially interesting data sources:
- `Abuse.ch SSL Blacklists <https://sslbl.abuse.ch/blacklist/>`_
- `Adblock Plus Malwaredomains <https://easylist-msie.adblockplus.org/malwaredomains_full.tpl>`_
- `apivoid IP Reputation API <https://www.apivoid.com/api/ip-reputation/>`_
- `APWG's ecrimex <https://www.ecrimex.net>`_
- `Bad IPs <https://www.badips.com>`_
- `Berkeley <https://security.berkeley.edu/aggressive_ips/ips>`_
- `Binary Defense <https://www.binarydefense.com/>`_
- `Bot Invaders Realtime tracker <http://www.marc-blanchard.com/BotInvaders/index.php>`_
- `Botscout Last Caught <http://botscout.com/last_caught_cache.htm>`_
- `Carbon Black Feeds <https://github.com/carbonblack/cbfeeds>`_
- `CERT.pl Phishing Warning List <http://hole.cert.pl/domains/>`_
- `Chaos Reigns <http://www.chaosreigns.com/spam/>`_
- `Critical Stack <https://intel.criticalstack.com>`_
- `Cruzit <http://www.cruzit.com/xwbl2txt.php>`_
- `Cyber Crime Tracker <http://cybercrime-tracker.net/all.php>`_
- `DNS DB API <https://api.dnsdb.info>`_
- `Dyn DNS <http://security-research.dyndns.org/pub/>`_
- `Facebook Threat Exchange <https://developers.facebook.com/docs/threat-exchange>`_
- `FilterLists <https://filterlists.com>`_
- `Firehol IPLists <https://iplists.firehol.org/>`_
- `Google Webmaster Alerts <https://www.google.com/webmasters/>`_
- `GPF Comics DNS Blacklist <https://www.gpf-comics.com/dnsbl/export.php>`_
- `Greensnow <https://blocklist.greensnow.co/greensnow.txt>`_
- `HP Feeds <https://github.com/rep/hpfeeds>`_
- `IBM X-Force Exchange <https://exchange.xforce.ibmcloud.com/>`_
- `ISC SANS <https://isc.sans.edu/ipsascii.html>`_
- `ISightPartners <http://www.isightpartners.com/>`_
- `Joewein <http://www.joewein.net>`_
- `Malshare <https://malshare.com/>`_
- `Malware Config <http://malwareconfig.com>`_
- `Malware DB (cert.pl) <https://mwdb.cert.pl/>`_
- `MalwareDomainList <http://www.malwaredomainlist.com/zeuscsv.php>`_
- `MalwareDomains <http://www.malwaredomainlist.com/hostslist/yesterday_urls.php>`_
- `MalwareInt <http://malwareint.com>`_
- `Manity Spam IP addresses <http://www.dnsbl.manitu.net/download/nixspam-ip.dump.gz>`_
- `Marc Blanchard DGA Domains <http://www.marc-blanchard.com/BotInvaders/index.php>`_
- `MaxMind Proxies <https://www.maxmind.com/en/anonymous_proxies>`_
- `mIRC Servers <http://www.mirc.com/servers.ini>`_
- `Monzymerza <https://github.com/monzymerza/parthenon>`_
- `Multiproxy <http://multiproxy.org/txt_all/proxy.txt>`_
- `MVPS <http://mvps.org>`_
- `Null Secure <http://nullsecure.org>`_
- `OpenBugBounty <https://www.openbugbounty.org/>`_
- `Payload Security <http://payload-security.com>`_
- `Project Honeypot (#284) <http://www.projecthoneypot.org/list_of_ips.php?rss=1>`_
- `ShadowServer Sandbox API <http://www.shadowserver.org/wiki/pmwiki.php/Services/Sandboxapi>`_
- `Shodan search API <https://shodan.readthedocs.io/en/latest/tutorial.html#searching-shodan>`_
- `Snort <http://labs.snort.org/feeds/ip-filter.blf>`_
- `Spamhaus BGP feed (BGPf) <https://www.spamhaus.org/bgpf/>`_
- `SteveBlack Hosts File <https://github.com/StevenBlack/hosts>`_
- `TheCyberThreat <http://thecyberthreat.com/cyber-threat-intelligence-feeds/>`_
- `The Haleys <http://charles.the-haleys.org/ssh_dico_attack_hdeny_format.php/hostsdeny.txt>`_
- `Threat Crowd <https://www.threatcrowd.org/feeds/hashes.txt>`_
- `Threat Grid <http://www.threatgrid.com/>`_
- `Threatstream <https://ui.threatstream.com/>`_
- `TOR Project Exit addresses <https://check.torproject.org/exit-addresses>`_
- `TotalHash <http://totalhash.com>`_
- `UCE Protect <http://wget-mirrors.uceprotect.net/>`_
- `URI BL <http://rss.uribl.com/index.shtml>`_
- `Virustotal <https://www.virustotal.com/gui/home/search>`_
- `virustream <https://github.com/ntddk/virustream>`_
- `VoIP Blacklist <http://www.voipbl.org/update/>`_
- `Wordpress Callback Domains <http://callbackdomains.wordpress.com>`_
- `YourCMC <http://vmx.yourcmc.ru/BAD_HOSTS.IP4>`_
Loading

0 comments on commit 2910dc1

Please sign in to comment.