Skip to main content

What’s New in Apache Pulsar 2.11

Technoboy-, momo-jun
· 4 min read

The Apache Pulsar community releases version 2.11! 61 contributors provided feature enhancements and fixes that delivered 1617 commits. Thanks for all your contributions.

Highlights of the 2.11 release​

  • Upgrade Pulsar server Java build version to 17 (PIP-156)
  • Support chunking with shared subscriptions (PR-16202)
  • Support pluggable topic factory (PR-12235)
  • Support configurable compression type for ManagedCursorInfo to balance storage resource usage (PR-14542)

This blog documents the most noteworthy changes in this release. For the complete list, including all feature enhancements and bug fixes, check out the Pulsar 2.11 Release Notes.

Notable feature enhancements and fixes​

Broker​

Store subscription properties within metadata service (PR-15757)​

Issue

Restarting the broker or unloading topics may lead to the loss of subscription properties since they are not stored in the metadata service.

Resolution

Store the subscription properties within the metadata service (ManagedCursorInfo).

Support configurable compression type for ManagedCursorInfo (PR-14542)​

Issue

When the cursor data expands, the data size increases and it takes a lot of time to pull the data.

Resolution

Introduce various compression algorithms to support cursor compression.

Reference documentation: managedCursorInfoCompressionType

Sync Pulsar metadata across multiple clouds (PR-16425)​

Issue

Synchronizing configuration metadata (policies) is a critical path to sharing tenant/namespace/topic policies among clusters and administering policies uniformly across all clusters. However, syncing the metadata store (global zookeeper) among Pulsar clusters deployed on separate cloud platforms is not supported in earlier versions.

Resolution

Use two system topics metadataSyncEventTopic and configurationmetadataSyncEventTopic along with a metadata synchronizer to sync metadata among clusters deployed on different cloud platforms and persist local topic policies with it.

Reference documentation

Producer​

Fix client memory limit currentUsage leak in ProducerImpl (PR-16837)​

Issue

The client memory limit currentUsage leaks in ProducerImpl and causes the producer’s send rate to be slow.

Resolution

Release the memory when there are failed pending batch messages.

Release semaphore before discarding messages in batchMessageContainer (PR-17019)​

Issue

There is a race condition in batchMessageContainer when discarding the messages and releasing the semaphore, which causes the semaphore to be held.

Resolution

Release the semaphore before discarding the messages in batchMessageContainer.

Consumer​

Support consumer client memory limit (PR-15216)​

Issue

It takes time for an application with a large number of producers/consumers to select an appropriate value for its queue size. The same applies to topics with many partitions.

Resolution

  1. Block the expansion of the consumer receiver’s queue size if the memory usage exceeds 75%.
  2. Trigger the shrinking of the consumer receiver’s queue size if the memory usage exceeds 95%.

Support adding interceptors for readers (PR-14729)​

Issue

Pulsar supports adding interceptors for producers and consumers to implement message tracing, but there is no way to add interceptors for readers.

Resolution

  1. Add a new interface ReaderInterceptor to customize the reader interceptor.
  2. Support setting reader interceptor and auto-update partition configurations by using ReaderBuilder.

User documentation: Create a reader with an interceptor

Function​

Support Record as a new output type of Functions (PR-16041)​

Issue

In earlier versions, if you want to dynamically set an output topic, message properties, or change the output schema in a Pulsar function, the only way is to create a function that returns Void. It would be more intuitive to return a structure like Record that carries this information.

Resolution

Add a utility method newOutputRecordBuilder to Context that returns a FunctionRecord builder initialized with the information from the source record. The builder methods can be used to override these values as needed.

User documentation: Use Record as function output

Tiered storage​

Add a universal S3 provider for the offloader (PR-15710)​

Issue

Pulsar supports cloud storage compatible with S3 APIs, such as AWS and Aliyun, through existing offloaders, but the forced registration of specific metadata limits their use.

Resolution

Provide a more general offloader S3 to serve more S3-compatible storage, which uses pure JClouds S3 metadata and allows overriding the default JClouds properties through system properties.

User documentation: Use S3 offloader with Pulsar

Proxy​

Support PrometheusRawMetricsProvider for Pulsar proxies (PR-14681)​

Issue

The metrics of plugins could not be exposed to Prometheus for monitoring.

Resolution

Add PrometheusRawMetricsProvider in Pulsar proxies so that plugins can add their metrics to Prometheus.

What’s Next?​

If you are interested in learning more about Pulsar 2.11, you can download and try it out now!

For more information about the Apache Pulsar project and current progress, visit the Pulsar website, follow the project on Twitter @apache_pulsar, and join Pulsar Slack!