Getting started with Pulsar IO

Connecting your systems with Pulsar using Pulsar IO

Pulsar IO is a feature of Pulsar that enables you to easily create and manage connectors that interface with external systems, such as databases and other messaging systems.

Setup

In order to run Pulsar IO connectors, you’ll need to have a binary distribution of pulsar locally.

Managing connectors

Pulsar connectors can be managed using the source and sink commands of the pulsar-admin CLI tool.

Running sources

You can use the create

You can submit a sink to be run in an existing Pulsar cluster using a command of this form:

$ ./bin/pulsar-admin sink create --className  <classname> --jar <jar-location> --tenant test --namespace <namespace> --name <sink-name> --inputs <input-topics>

Here’s an example command:

bin/pulsar-admin source create --className  org.apache.pulsar.io.twitter.TwitterFireHose --jar ~/application.jar --tenant test --namespace ns1 --name twitter-source --destinationTopicName twitter_data

Instead of submitting a source to run on an existing Pulsar cluster, you alternatively can run a source as a process on your local machine:

bin/pulsar-admin source localrun --className  org.apache.pulsar.io.twitter.TwitterFireHose --jar ~/application.jar --tenant test --namespace ns1 --name twitter-source --destinationTopicName twitter_data

Running Sinks

You can submit a sink to be run in an existing Pulsar cluster using a command of this form:

./bin/pulsar-admin sink create --className  <classname> --jar <jar-location> --tenant test --namespace <namespace> --name <sink-name> --inputs <input-topics>

Here’s an example command:

./bin/pulsar-admin sink create --className  org.apache.pulsar.io.cassandra --jar ~/application.jar --tenant test --namespace ns1 --name cassandra-sink --inputs test_topic

Instead of submitting a sink to run on an existing Pulsar cluster, you alternatively can run a sink as a process on your local machine:

./bin/pulsar-admin sink localrun --className  org.apache.pulsar.io.cassandra --jar ~/application.jar --tenant test --namespace ns1 --name cassandra-sink --inputs test_topic

Available connectors

At the moment, the following connectors are available for Pulsar:

Name Java class
Aerospike sink org.apache.pulsar.io.aerospike.AerospikeStringSink
Cassandra sink org.apache.pulsar.io.cassandra.CassandraStringSink
Kafka source org.apache.pulsar.io.kafka.KafkaStringSource
Kafka sink org.apache.pulsar.io.kafka.KafkaStringSink
RabbitMQ source org.apache.pulsar.io.rabbitmq.RabbitMQSource
Twitter Firehose source org.apache.pulsar.io.twitter.TwitterFireHose