Documentation
The DBpedia community uses a flexible and extensible framework to extract different kinds of structured information from Wikipedia. The DBpedia extraction framework is written using Scala 2.8. The framework is available from the DBpedia Github repository (GNU GPL License). The change log may reveal more recent developments. More recent configuration options can be found here: https://github.com/dbpedia/extraction-framework/wiki
When the above link is 100% complete, we will remove this section.
For historic reasons, the documentation of the old, superseded PHP-based DBpedia extraction framework is still available here.
Before you can start developing, you need to take care of some prerequisites:
- DBpedia Extraction Framework Get the most recent revision from the Github repository.
git clone git://github.com/dbpedia/extraction-framework.git (read only)- Java Development Kit The DBpedia extraction framework uses Java. Get the most recent JDK from http://java.com/.
- The DBpedia extraction framework currently requires at least Java 7 (JDK v1.7.0) for full functionality.
- You can compile and run it with an earlier JDK by deleting or blanking the following two files. (The launchers_purge-download_and_purge-extract_in the_dump_module won't work, but they are not vitally necessary.)
core/src/main/scala/org/dbpedia/extraction/util/Rich Path.scaladump/src/main/scala/org/dbpedia/extraction/dump/clean/Clean.scala- Maven is used for project management and build automation. Get it from: http://maven.apache.org/. Either Maven 2 or 3 should work.
This is enough to compile and run the DBpedia extraction framework. The required input files — the wikimedia dumps — will be downloaded by extractor code if configured to do so (see below).
If you'd like to use an IDE for coding, there are a number of options:
- IntelliJ IDEA Currently the most stable IDE for developing with Scala. To get the most recent Scala Plugin get the current early access version and install the Scala plugin from the official repository. Please follow the DBpedia & IntelliJ Quick Start Guide
- Eclipse: Please follow the DBpedia & Eclipse Quick Start Guide
- Netbeans also offers a Scala plugin.
The DBpedia extraction framework is structured into different modules
- Core Module : Contains the core components of the framework.
- Dump extraction Module : Contains the DBpedia dump extraction application.
The Live, Wiktionary, and Server modules are not necessary for running the extraction and can be disabled from the pom.xml for that use case.
Components
- Source : The Source package provides an abstraction over a source of Media Wiki pages.
- WikiParser : The Wiki Parser package specifies a parser, which transforms an Media Wiki page source into an Abstract Syntax Tree (AST).
- Extractor : An Extractor is a mapping from a page node to a graph of statements about it.
- Destination : The Destination package provides an abstraction over a destination of RDF statements.
In addition to the core components, a number of utility packages offers essential functionality to be used by the extraction code:
- Ontology Classes used to represent an ontology. Methods for both, reading and writing ontologies are provided. All classes are located in the namespace org.dbpedia.extraction.ontology
- DataParser Parsers to extract data from nodes in the abstract syntax tree. All classes are located in the namespace org.dbpedia.extraction.dataparser
- Util Various utility classes. All classes are located in the namespace org.dbpedia.extraction.util
For details about a package follow the links.
You can find the complete scaladoc here
More recent configuration options can be found here: https://github.com/dbpedia/extraction-framework/wiki/Extraction-Instructions.
When the above link is 100% complete, we will remove this section.
The framework is undergoing a lot of refactoring, and the following sections are not 100% correct. For now, you should use the 'dump' branch for bump-based extraction.
$ git clone git://github.com/dbpedia/extraction-framework.git
$ cd extraction-framework
$ mvn clean install
$ cd dump
$ ../run download config=download.properties.file
$ ../run extraction extraction.properties.file
The extraction and download property files contain many documentation comments and can be easily adapted to your needs
All configurations are read from a Java properties file named dump/config.properties. After a fresh checkout you will need to copy it from the .default and modify according to your needs (e.g. like removing unwanted languages to be extracted).
The following properties are available:
-
dumpDir The directory where the dumps are located. The DEF expects to see subdirectories of the type «
enwiki/[date]" inside -
updateDumps If true, the extraction framework will download every dump which is either missing or not up-to-date. If you want to use your own dumps or don't want the framework to update the dumps, set it to false and be sure that (uncompressed) dumps are available in
dumpDir/<lang>/<date>/<lang>wiki-<date>-pages-articles.xml. - outputDir The output directory.
- languages The languages of the Wikipedia dumps to be extracted.
-
extractors The extractor classes to be used for the extraction. See Available Extractors. Language specific extractors can be configured using a property of the format
extractors.{wikiCode}, e.g.,extractors.en
Before you can start the extraction you need to install the framework into your maven repository by running mvn install from the extraction
directory.
The dump extraction is started by running mvn scala:run from the directory extraction/dump.
Abstracts are not generated by the Simple Wiki Parser, they are produced by a local wikipedia clone using a modified mediawiki installation.
In order to generate clean abstracts from Wikipedia articles, one needs to render wiki templates as they would be rendered in the original Wikipedia instance. So in order for the DBpedia Abstract Extractor to work, a running Media Wiki instance with Wikipedia data in a database is necessary.
See:
Extraction on Ubuntu.
Internationalization
If you leave the updateDumps setting to false you could download the dump you are interested with from http://dumps.wikimedia.org/backup-index.html, pick the latest complete one from <lang>wiki (e.g., itwiki) and choose the pages-articles.xml.bz2 one (e.g., itwiki-20120226-pages-articles.xml.bz2). The input file must be placed in dumpDir/<lang>/<date> (e.g., /srv/dbpedia/dumps/it/201220226/itwiki-20120226-pages-articles.xml.bz2, if dumpDir is /srv/dbpedia/dumps).
This module is intended for testing the framework.
There are two Scala classes that configure the parameters of the server:
- In
org.dbpedia.extraction.server.Configuration, you can configure the possible languages and URL to the mappings wiki API. * Inorg.dbpedia.extraction.server.ExtractionManagerin the functionloadExtractor, you can configure the extractors that should be used by the extraction server. See Available Extractors.
Before you can start the server you need to install the framework into your maven repository by running mvn install from the extraction directory.
The extraction server is started by running mvn scala:run from the directory extraction/server. The standard port is 9999.
A browser window should open in which you can specify the language and the URI that you would like to extract.