Overview

Spring Hadoop Sample Applications

These sample applications are meant to be run as an 'executable' out of the box, with no compilation or build steps required. There is a shell, based on the Spring Shell, that allows you to start/stop the server and to run sample applications that showcase various features of Spring Hadoop.

There are three categories of applications

1. Simple 'main' apps
2. Spring Integration based event-stream processing apps
3. Spring Batch based data import/export/ETL apps

To get started using these applications, please refer to the individual documentation pages for each category.

Structure of the event-stream applications.

The structure of the event-stream processing apps follow a simple underlying convention that will allow you to modify the app to your needs. In the ./src/main/config directory of the server project contains the following directories

1. input-adapters
2. processors
3. output-adapters
4. properties
5. A 'top level application directory, e.g. syslog-hdfs.

The input-adapter directory contains a definition for a Spring Integration inbound channel adapter with the name 'inputAdapter' This can be any SI channel adapter and it should place its incoming message on a channel named 'processorInput'.

The processors directory contains a definition for a SI processing chain. By convention it is setup to read messages from the channel processorInput and put them on a channel named 'processorOutput'. This where you can place a great deal of your custom processing logic.

The output-adapters directory contains a definition for a SI outbound channel adapter that the end of a SI processing chain that read from a channel named 'processorOutput'.

The directory properties contains property files with externalized property values that are used to parametrize the configuration of each component.

The top level application directory is what pulls together the inbound-adapter, processor, outbound-adapter and property files into a single application. The modularity of this approach allows you to swap in and out different adapters and processors. An example of top level application that processes a syslog event stream and writes to HDFS is shown below.

 <context:property-placeholder 
          location="properties/syslog.properties,properties/hadoop.properties,properties/hdfs.properties"/>
 
 <import resource="../input-adapters/syslog-udp.xml"/>
 
 <import resource="../processors/noop.xml"/>
 
 <import resource="../output-adapters/hdfs.xml"/>

The event-stream applications are administratively controlled using the JMX operations that Spring Integration has but exported over HTTP/JSON using the Jolokia Project. Commands in the shell interact with these endpoints.

Structure of the batch/ETL applications.

The batch applications follow the structure of Spring Batch Admin applications, In the src/main/resources/META-INF/spring/batch directory are the following sub-directories

1. jobs
2. override

In the jobs directory you place Spring Batch configuration files to define batch jobs and in the override directory you place configuration information unique to your environment such as database configuration. In the sample applications an embedded H2 database is used.

The batch applications are administratively controlled over HTTP/JSON using the Spring Batch Admin project. The admin commands are exposed in the shell allowing you to start/stop/list batch jobs.