This example shows you how to use OCI Data Flow to process data in OCI Object Store and save the results to Oracle ADW or ATP.
Before you begin:
-
Ensure your tenant is configured for Data Flow by following instructions
-
Provision an ADW or ATP instance.
-
Create a wallet for your ADW/ATP instance.
-
Store the wallet password in a secret within the OCI Secrets Service.
-
Deploy the Oracle JDBC driver to the repo subdirectory, if it does not already exist.
- Download a Java 8 compatible (ojdbc8) JDBC driver
- Install the required JARs into your local environment
On Mac/Linux:
mvn deploy:deploy-file -Durl=file://$(pwd)/repo -Dfile=ojdbc8-full/ojdbc8.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
mvn deploy:deploy-file -Durl=file://$(pwd)/repo -Dfile=ojdbc8-full/ucp.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
mvn deploy:deploy-file -Durl=file://$(pwd)/repo -Dfile=ojdbc8-full/oraclepki.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
mvn deploy:deploy-file -Durl=file://$(pwd)/repo -Dfile=ojdbc8-full/osdt_cert.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
mvn deploy:deploy-file -Durl=file://$(pwd)/repo -Dfile=ojdbc8-full/osdt_core.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
On Windows:
mvn deploy:deploy-file -Durl=file://%cd%/repo -Dfile=ojdbc8-full/ojdbc8.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
mvn deploy:deploy-file -Durl=file://%cd%/repo -Dfile=ojdbc8-full/ucp.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
mvn deploy:deploy-file -Durl=file://%cd%/repo -Dfile=ojdbc8-full/oraclepki.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
mvn deploy:deploy-file -Durl=file://%cd%/repo -Dfile=ojdbc8-full/osdt_cert.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
mvn deploy:deploy-file -Durl=file://%cd%/repo -Dfile=ojdbc8-full/osdt_core.jar -DgroupId=com.oracle -DartifactId=$file -Dpackaging=jar -Dversion=18.3
Upload a sample CSV file to OCI object store.
Customize src/main/java/example/Example.java
to set values for walletPath, user, passwordOcid and tnsName.
mvn package
Test the Application Locally (recommended) You can test the application locally using spark-submit:
spark-submit --class example.Example target/loadadw-1.0-SNAPSHOT.jar
oci os object put --bucket-name <mybucket> --file target/loadadw-1.0-SNAPSHOT.jar
oci data-flow application create \
--compartment-id <your_compartment> \
--display-name "Load ADW Java"
--driver-shape VM.Standard2.1 \
--executor-shape VM.Standard2.1 \
--num-executors 1 \
--spark-version 2.4.4 \
--file-uri oci://<bucket>@<namespace>/target/loadadw-1.0-SNAPSHOT.jar \
--language Java \
--class-name example.Example
Make note of the Application OCID produced by this command
oci data-flow run create \
--compartment-id <your_compartment> \
--application-id <application_ocid> \
--display-name "Load ADW Java"