Skip to main content

Get started in Kubernetes

This section guides you through every step of installing and running Apache Pulsar with Helm on Kubernetes quickly, including

  • Install the Apache Pulsar on Kubernetes using Helm
  • Start and stop Apache Pulsar
  • Create topics using pulsar-admin
  • Produce and consume messages using Pulsar clients
  • Monitor Apache Pulsar status with Prometheus and Grafana

For deploying a Pulsar cluster for production usage, please read the documentation on how to configure and install a Pulsar Helm chart.

Prerequisite​

  • Kubernetes server 1.14.0+
  • kubectl 1.14.0+
  • Helm 3.0+
tip

For the following steps, step 2 and step 3 are for developers and step 4 and step 5 are for administrators.

Step 0: Prepare a Kubernetes cluster​

Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow the instructions to prepare a Kubernetes cluster.

We use Minikube in this quick start guide.

  1. Create a kubernetes cluster on Minikube.


    minikube start --memory=8192 --cpus=4 --kubernetes-version=<k8s-version>

    The <k8s-version> can be any Kubernetes version supported by your minikube installation. Example: `v1.16.1.

  2. Set kubectl to use Minikube.


    kubectl config use-context minikube

  3. In order to use the Kubernetes Dashboard with local Kubernetes cluster on Minikube, enter the command below:


    minikube dashboard

    The command automatically triggers opening a webpage in your browser.

Step 1: Install Pulsar Helm Chart​

  1. Clone the Pulsar Helm chart repository.


    git clone https://github.com/apache/pulsar-helm-chart
    cd pulsar-helm-chart

  2. Run prepare_helm_release.sh to create secrets required for installing Apache Pulsar Helm chart. The username pulsar and password pulsar are used for logging into Grafana dashboard and Pulsar Manager.


    ./scripts/pulsar/prepare_helm_release.sh \
        -n pulsar \
        -k pulsar-mini \
        --control-center-admin pulsar \
        --control-center-password pulsar \
        -c

  3. Use the Pulsar Helm chart to install a Pulsar cluster to Kubernetes.


    helm install \
        --values examples/values-minikube.yaml \
        pulsar-mini charts/pulsar

  4. Check the status of all pods.


    kubectl get pods -n pulsar

    If all pods start up successfully, you can see STATUS changes to Running or Completed.

    Output


    NAME                                         READY   STATUS      RESTARTS   AGE
    pulsar-mini-bookie-0                         1/1     Running     0          9m27s
    pulsar-mini-bookie-init-5gphs                0/1     Completed   0          9m27s
    pulsar-mini-broker-0                         1/1     Running     0          9m27s
    pulsar-mini-grafana-6b7bcc64c7-4tkxd         1/1     Running     0          9m27s
    pulsar-mini-prometheus-5fcf5dd84c-w8mgz      1/1     Running     0          9m27s
    pulsar-mini-proxy-0                          1/1     Running     0          9m27s
    pulsar-mini-pulsar-init-t7cqt                0/1     Completed   0          9m27s
    pulsar-mini-pulsar-manager-9bcbb4d9f-htpcs   1/1     Running     0          9m27s
    pulsar-mini-toolset-0                        1/1     Running     0          9m27s
    pulsar-mini-zookeeper-0                      1/1     Running     0          9m27s

  5. Check the status of all services in the namespace pulsar.


    kubectl get services -n pulsar

    Output


    NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
    pulsar-mini-bookie           ClusterIP      None             <none>        3181/TCP,8000/TCP             11m
    pulsar-mini-broker           ClusterIP      None             <none>        8080/TCP,6650/TCP             11m
    pulsar-mini-grafana          LoadBalancer   10.106.141.246   <pending>     3000:31905/TCP                11m
    pulsar-mini-prometheus       ClusterIP      None             <none>        9090/TCP                      11m
    pulsar-mini-proxy            LoadBalancer   10.97.240.109    <pending>     80:32305/TCP,6650:31816/TCP   11m
    pulsar-mini-pulsar-manager   LoadBalancer   10.103.192.175   <pending>     9527:30190/TCP                11m
    pulsar-mini-toolset          ClusterIP      None             <none>        <none>                        11m
    pulsar-mini-zookeeper        ClusterIP      None             <none>        2888/TCP,3888/TCP,2181/TCP    11m

Step 2: Use pulsar-admin to create Pulsar tenants/namespaces/topics​

pulsar-admin is the CLI tool for Pulsar. In this step, you can use pulsar-admin to create resources including tenants, namespaces, and topics.

  1. Enter the toolset container.


    kubectl exec -it -n pulsar pulsar-mini-toolset-0 -- /bin/bash

  2. In the toolset container, create a tenant named apache.


    bin/pulsar-admin tenants create apache

    Then you can list the tenants to see if the tenant is created successfully.


    bin/pulsar-admin tenants list

    You should see a similar output as below. The tenant apache has been successfully created.


    "apache"
    "public"
    "pulsar"

  3. In the toolset container, create a namespace named pulsar in the tenant apache.


    bin/pulsar-admin namespaces create apache/pulsar

    Then you can list the namespaces of tenant apache to see if the namespace is created successfully.


    bin/pulsar-admin namespaces list apache

    You should see a similar output as below. The namespace apache/pulsar has been successfully created.


    "apache/pulsar"

  4. In the toolset container, create a topic test-topic with 4 partitions in the namespace apache/pulsar.


    bin/pulsar-admin topics create-partitioned-topic apache/pulsar/test-topic -p 4

  5. In the toolset container, list all the partitioned topics in the namespace apache/pulsar.


    bin/pulsar-admin topics list-partitioned-topics apache/pulsar

    Then you can see all the partitioned topics in the namespace apache/pulsar.


    "persistent://apache/pulsar/test-topic"

Step 3: Use Pulsar client to produce and consume messages​

You can use the Pulsar client to create producers and consumers to produce and consume messages.

By default the Helm chart expose the Pulsar cluster through a Kubernetes LoadBalancer. In Minikube, you can use the following command to get the IP address of the proxy service.


kubectl get services -n pulsar | grep pulsar-mini-proxy

You will see a similar output as below.


pulsar-mini-proxy            LoadBalancer   10.97.240.109    <pending>     80:32305/TCP,6650:31816/TCP   28m

This output tells what are the node ports that Pulsar cluster's binary port and http port are exposed to. The port after 80: is the http port while the port after 6650: is the binary port.

Then you can find the ip address of your minikube server by running the following command.


minikube ip

At this point, you will get the service urls to connect to your Pulsar client.


webServiceUrl=http://$(minikube ip):<exposed-http-port>/
brokerServiceUrl=pulsar://$(minikube ip):<exposed-binary-port>/

Then proceed with the following steps:

  1. Download the Apache Pulsar tarball from downloads page.

  2. Decompress the tarball based on your download file.


    tar -xf <file-name>.tar.gz

  3. Expose PULSAR_HOME.

    (1) Enter the directory of the decompressed download file.

    (2) Expose PULSAR_HOME as the environment variable.


    export PULSAR_HOME=$(pwd)

  4. Configure the Pulsar client.

    In the ${PULSAR_HOME}/conf/client.conf file, replace webServiceUrl and brokerServiceUrl with the service urls you get from the above steps.

  5. Create a subscription to consume messages from apache/pulsar/test-topic.


    bin/pulsar-client consume -s sub apache/pulsar/test-topic  -n 0

  6. Open a new terminal. In the new terminal, create a producer and send 10 messages to the test-topic topic.


    bin/pulsar-client produce apache/pulsar/test-topic  -m "---------hello apache pulsar-------" -n 10

  7. Verify the results.

    • From producer side

      Output

      The messages have been produced successfully.


      18:15:15.489 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 10 messages successfully produced

    • From consumer side

      Output

      At the same time, you can receive the messages as below.


      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------
      ----- got message -----
      ---------hello apache pulsar-------

Step 4: Use Pulsar Manager to manage the cluster​

Pulsar Manager is a web-based GUI management tool for managing and monitoring Pulsar.

  1. By default, the Pulsar Manager is exposed as a separate LoadBalancer. You can open the Pulsar Manager UI using the following command:


    minikube service pulsar-mini-pulsar-mananger

  2. The pulsar manager UI will be open in your browser. You can use username pulsar and password pulsar to log into Pulsar Manager.

  3. In Pulsar Manager UI, you can create an environment.

    • Click New Environment button in the top-left corner.
    • Type pulsar-mini for the field Environment Name in the popup window.
    • Type http://pulsar-mini-broker:8080 for the field Service URL in the popup window.
    • Click Confirm button in the popup window.

  4. After successfully creating an environment, you will be redirected to the tenants page of that environment. Then you can create tenants, namespaces and topics using Pulsar Manager.

Step 5: Use Prometheus and Grafana to monitor the cluster​

Grafana is an open-source visualization tool, which can be used for visualizing time series data into dashboards.

  1. By default, the Grafana is exposed as a separate LoadBalancer. You can open the Grafana UI using the following command:


    minikube service pulsar-mini-grafana -n pulsar

  2. The Grafana UI will be open in your browser. You can use username pulsar and password pulsar to log into Grafana Dashboard.

  3. You will be able to view dashboards for different components of a Pulsar cluster.

Was this helpful?
💡 Suggest changes🛟 Get support