LAB4_Streaming

🎓 LAB4 - Streaming from Pulsar to Cassandra

_ ==================> _

---

✅ STEP 1: Create a sink

1. Select Sinks > Add in the menu on the left side

2. In the SINKS box select cassandra-enhanced, you might notice than Pulsar already manage a lot of different technologies. Today we will use the SINK implemented by Datastax.

3. In field cloud.secureConnectBundle upload the secureBundle zip file you downloaded on previous lab.

4. In field auth.username provide the astra DB user name astraUser

5. In field auth.password provide the astra DB user name astraPassword1

This is what the user interface looks like for those first fields.

Now scroll to the bottom of the page to keep entering the following settings:

6. In field topics provides the topic name we created in LAB2 astra

7. Set field verbose to true, it is useful to show the log on the user interface.

8. Click on the banner click for extra settings (yes this is a button) we will form inside the create keys.

9. Create key topic.astra.pulsar.products.mapping with the value id=value.id,description=value.value and then click ADD KEY.

ℹ️ Information
---------------
This key describes the mapping between Cassandra Table and pulsar Topic. 
- Key NAME follows pattern `<topic>.<topic_name>.<cassandra_keyspace_name>.<cassandra_table_name>.mapping`. 
- Key VALUE follows pattern `<colum_name>=value.<json_field_path>, ...`

10. Create key ignoreErrors with the value ALL and then click ADD KEY. If a message does not match with the expecting schema/mapping the subscription Pulsar => Cassandra will loop on this message by default (idea is to not loose messages). Here in dev we can afford raising errors.

11. Below the custom keys table locate the check box Auto Ack ans disable it.

12. In the **Input Topics (at least one) section change default Topic value from cassandra-enhancedSink-input to astra. You do not need to change the namespace, simply click ADD TOPIC.

13. You can finally click ADD in the very bottom to create your sink.

This is a lot of steps ! This is what look like the form when you completed steps from 6 to 13

✅ STEP 2: Check that the sink in working in Kesque

• It will take a few seconds for the sink to start. You will first get a message Sink successfully created but running instances are still 0.

• This is what it looks like when the sink has started. Instance is 1 and running is also 1

• Now select tab Instances to see the details and number of messages that are processed in real time. 100 in the following screenshot

✅ STEP 3: Check that the sink in working in Astra DB

Alright ! This is now the moment of truth. Back to the CQL Console in Astra DB execute again the following statement:

select * from pulsar.products;

👏👏 Et VOILA you now have messages coming into the table, congratulation ! 👏👏 This is the end of hands-on for today, hope you enjoyed we have more resources to share with you in the last section, see you there.