Apache Pulsar 4.0.0

2024-10-21

Changes in 4.0.0 release since 3.3.0 release preparation was started. For the complete list, check the full changelog and full PR list.

What's New in Apache Pulsar 4.0? covers topics such as:

• Enhanced Key_Shared subscription implementation
• Secure Docker runtime based on Alpine Linux and Java 21
• Enhanced Quality of Service controls for multi-tenant deployments
• Rate limiting as a foundation for multi-tenant resource management
• Upgrade paths from previous versions
• Client compatibility and Importance of upgrading Java clients to address CVE-2024-47561

PIPs accepted since 3.0.0 release

PIPs accepted in 4.0.0 release

• PIP-327: Support force topic loading for unrecoverable errors (#21752)
• PIP-337: SSL Factory Plugin to customize SSL Context and SSL Engine generation (#22016)
• PIP-347: add role field in consumer's stat (#22564)
• PIP-348: Trigger offload on topic load stage (#22650)
• PIP-349: Add additionalSystemCursorNames ignore list for TTL check (#22651)
• PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis (#22688)
• PIP-351: Additional options for Pulsar-Test client to support KeyStore based TLS (#22694)
• PIP-352: Event time based topic compactor (#22710)
• PIP-353: Improve transaction message visibility for peek-messages (#22746)
• PIP-354: apply topK mechanism to ModularLoadManagerImpl (#22765)
• PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22778)
• PIP-356: Support Geo-Replication starts at earliest position (#22806)
• PIP-357: Correct the conf name in load balance module. (#22823)
• PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. (#22889)
• PIP-359: Support custom message listener executor for specific subscription (#22902)
• PIP-360: Add admin API to display Schema metadata (#22913)
• PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete. (#22940)
• PIP-364: Introduce a new load balance algorithm AvgShedder (#22946)
• PIP-366: Support to specify different config for Configuration and Local Metadata Store (#23033)
• PIP-367: Propose a Contributor Repository for Pulsar (#23061)
• PIP-368: Support lookup based on the lookup properties (#23075)
• PIP-369: Flag based selective unload on changing ns-isolation-policy (#23116)
• PIP-370: configurable remote topic creation in geo-replication (#23124)
• PIP-374: Visibility of messages in receiverQueue for the consumers (#23235)
• PIP-376: Make topic policies service pluggable (#23248)
• PIP-378: Add ServiceUnitStateTableView abstraction (ExtensibleLoadMangerImpl only) (#23300)
• PIP-379: Key_Shared Draining Hashes for Improved Message Ordering (#23309)
  • Documentation: Preserving order of message delivery by key
• PIP-381: Handle large PositionInfo state (#23328)
• PIP-383: Support granting/revoking permissions for multiple topics (#23355)
• PIP-384: ManagedLedger interface decoupling (#23363)

PIPs accepted in 3.3.0 release

• PIP-315: Configurable max delay limit for delayed delivery (#21798)
• PIP-321: Introduce allowed-cluster at the namespace level (#21648)
• PIP-324: Alpine Docker images (#21716)
• PIP-325: Add command to abort transaction (#21731)
• PIP-326: Create a BOM to ease dependency management (#21747)
• PIP-329: Strategy for maintaining the latest tag to Pulsar docker images (#21872)
• PIP-330: getMessagesById gets all messages (#21873)
• PIP-342: Support OpenTelemetry metrics in Pulsar client (#22178)
• PIP-343: Use picocli instead of jcommander (#22181)
• PIP-344: Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) (#22182)
• PIP-335: Oxia metadata plugin (#22009)
• PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22071)
• PIP-282: Change definition of the recently joined consumers position (#20776)

PIPs accepted in 3.2.0 release

• PIP-323: Complete Backlog Quota Telemetry (#21709)
• PIP-322: Pulsar Rate Limiting Refactoring (#21680)
• PIP-320: OpenTelemetry Scaffolding (#21635)
• PIP-318: Don't retain null-key messages during topic compaction (#21541)
• PIP-315: Configurable max delay limit for delayed delivery (#21490)
• PIP-313: Support force unsubscribe using consumer api (#21452)
• PIP-312: Use StateStoreProvider to manage state in Pulsar Functions endpoints (#21438)
• PIP-307: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager (#20748)
• PIP-307-dup: Support subscribing multi topics for WebSocket (#21390)
• PIP-305: Customize DNS servers to use for Pulsar Client (#21352)
• PIP-303: Add optional parameters for getPartitionedStats (#21228)
• PIP-302: Introduce refreshAsync API for TableView (#21271)
• PIP-301: Introduce LoadBalanceResources to unify the load-date CRUD (#21129)
• PIP-300: Add custom dynamic configuration for plugins (#21127)
• PIP-299: Stop dispatch messages if the individual acks will be lost in the persistent storage (#21118)
• PIP-297: Support terminating Function & Connector with the fatal exception (#21079)
• PIP-293: Delete config disableBrokerInterceptors (#20993)
• PIP-290: Provide a way to implement WSS E2E encryption and not need to expose the private key to the WebSocket Proxy (#20923)
• PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index (#20867)
• PIP-277: Add current option in the Clusters list cmd (#20614)
• PIP-264: Enhanced OTel-based metric system #21080

PIPs accepted in 3.1.0 release

• PIP-275: Introduce topicOrderedExecutorThreadNum to deprecate numWorkerThreadsForNonPersistentTopic (#20504)
• PIP-255: Make the partition assignment strategy pluggable (#19806)
• PIP-278: Support pluggable topic compaction service (#20624)

Library updates and dependency changes

• [cleanup][misc] Remove classifier from netty-transport-native-unix-common dependency (#22951)
• [fix] Upgrade Alpine packages at build time to fix CVE-2023-4236 (#22763)
• [improve] Upgrade to Oxia client 0.3.0 (#22807)
• [fix] Upgrade to Oxia 0.3.1 (#23048)
• [fix][metadata] Upgrade Oxia to 0.3.2 (#23140)
• [feat][meta] Bump oxia java version from 0.3.2 to 0.4.5 (#23277)
• [fix] Bump commons-io:commons-io from 2.8.0 to 2.14.0 (#23393)
• [fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go (#22261)
• [fix] Bump io.airlift:aircompressor from 0.20 to 0.27 (#22819)
• [fix] Bump io.grpc from 1.56.0 to 1.56.1 (#23276)
• [fix][sec] Drop hdfs2 support, Upgrade hadoop3 to 3.4.0 and dnsjava to 3.6.2 to address CVE-2024-25638 (#23411)
• [fix][sec] Upgrade Avro to 1.11.4 to address CVE-2024-47561 (#23394)
• [fix][sec] Upgrade Bouncycastle libraries to address CVEs (#22826)
• [fix][sec] Upgrade postgresql version to avoid CVE-2024-1597 (#22635)
• [fix][sec] Upgrade protobuf-java to 3.25.5 (#23356)
• [fix][sec] Upgrade vertx to 4.5.10 to address CVE-2024-8391 (#23338)
• [fix][sec] Upgrade jackson to 2.17.2 (#23174)
• [improve] Upgrade IPAddress to 5.5.0 (#22886)
• [improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835)
• [improve] Upgrade Pulsar Python client in docker image to 3.5.0 (#23377)
• [improve] Upgrade opensearch sink to client 2.16 and tests to use server 2.16.0 (#23312)
• [improve] Use Google re2/j library for user provided regexes (#22829)
• [improve] Install openssl in the docker image to fix compatibility with Apache Pulsar Helm chart (#23362)
• [improve][misc] Set Alpine base image to 3.20 instead of 3.19.1 (#22941)
• [improve][misc] Upgrade Bookkeeper to 4.17.1 (#22962)
• [improve][misc] Upgrade Jetty to 9.4.56.v20240826 (#23405)
• [improve][misc] Upgrade Netty to 4.1.113 and netty-tcnative to 2.0.66 (#23255)
• [improve][misc] Upgrade OTel library to 1.38.0 version (#22825)
• [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
• [improve][misc] Bump RoaringBitmap version to 1.0.6 (#22920)

Broker

• [feat] PIP-352: Event time based compaction (#22517)

• [feat][broker] PIP-264: Add OpenTelemetry broker connection metrics (#22931)

• [feat][broker] PIP-264: Add OpenTelemetry broker replicator metrics (#22972)

• [feat][broker] PIP-264: Add OpenTelemetry consumer metrics (#22693)

• [feat][broker] PIP-264: Add OpenTelemetry managed cursor metrics (#23000)

• [feat][broker] PIP-264: Add OpenTelemetry managed ledger metrics (#22987)

• [feat][broker] PIP-264: Add OpenTelemetry metadata store stats (#22952)

• [feat][broker] PIP-264: Add OpenTelemetry producer metrics (#22882)

• [feat][broker] PIP-264: Add broker web executor metrics (#22816)

• [feat][broker] PIP-264: Add managed ledger cache metrics (#22898)

• [feat][broker] PIP-264: Add replication subscription stats (#23026)

• [feat][broker] PIP-264: Add schema registry metrics (#22624)

• [feat][broker] PIP-264: Add transaction metrics (#22970)

• [feat][broker] PIP-264: Enable OpenTelemetry reusable data memory mode (#22914)

• [feat][broker] PIP-321 Introduce allowed-cluster at the namespace level (#22378)

• [feat][broker] PIP-368: Support lookup based on the lookup properties (#23223)

• [feat][client] PIP-374: Visibility of messages in receiverQueue for the consumers (#23303)

• [feat][misc] PIP-264: Add OpenTelemetry HTTP rate limiting filter metric (#23042)

• [feat][misc] PIP-264: Add OpenTelemetry authentication and token metrics (#23016)

• [feat][misc] PIP-264: Add OpenTelemetry messaging rate limit metrics (#23035)

• [feat][misc] PIP-264: Copy OpenTelemetry resource attributes to Prometheus labels (#23005)

• [improve][broker] PIP-327: Support force topic loading for unrecoverable errors (#21759)

• [improve][broker] PIP-347: Add role field in consumer's stat (#22562)

• [improve][broker] PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22779)

• [improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)

• [improve][broker] PIP-357: Correct the conf name in load balance module. (#22824)

• [improve][broker] PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. (#22888)

• [improve][broker] PIP-360 Add admin API to display Schema metadata (#22938)

• [improve][broker] PIP-364: Introduce a new load balance algorithm AvgShedder (#22949)

• [improve][broker] PIP-366: Support to specify different config for Configuration and Local Metadata Store (#23041)

• [improve][broker] PIP-376: Make topic policies service pluggable (#23319)

• [improve][broker] PIP-379: Enable the use of the classic implementation of Key_Shared / Shared with feature flag (#23424)

• [improve][broker] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering (#23352)

• [improve][broker] PIP-379: Snapshot hash range assignments only in AUTO_SPLIT ordered mode (#23423)

• [improve][broker] PIP-383: Support granting/revoking permissions for multiple topics (#23372)

• [improve][broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication (#23169)

• [improve][broker] Part 2 of PIP-370: add metrics "pulsar_replication_disconnected_count" (#23213)

• [improve][broker] PIP-354: apply topK mechanism to ModularLoadManagerImpl (#22753)

• [Fix][broker] Limit replication rate based on bytes (#22674)

• [feat] Use producer name and sequence number as fallback key in Key_Shared implementation (#23219)

• [fix] Key_Shared mode consumption latency when low traffic (#23340)

• [fix] Make operations on individualDeletedMessages in lock scope (#22966)

• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)

• [fix] Removing out of the box option for Java serde in functions (#22832)

• [fix] StatsOutputStream: add string write function (#308) (#23227)

• [fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)

• [fix][admin] Tenant AdminRoles can not contains whitespace in the beginning or end. (#22450)

• [fix][admin][part-1] Clearly define REST API on Open API (#22774)

• [fix][admin][part-2] Clearly define REST API on Open API for Namespaces@v2 (#22775)

• [fix][admin][part-3]Clearly define REST API on Open API for Topics (#22782)

• [fix][admin][part-4] Clearly define REST API on Open API (#22783)

• [fix][broker] Add principal name into info log to enhance analysis and troubleshooting (#23257)

• [fix][broker] Allow broker to handle non-recoverable schema error only if SchemaLedgerForceRecovery flag is enabled (#23428)

• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)

• [fix][broker] Avoid introducing delay when there are delayed messages or marker messages (#23343)

• [fix][broker] Avoid orphan ledgers in BucketDelayedDeliveryTracker (#22802)

• [fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)

• [fix][broker] Cancel possible pending replay read in cancelPendingRead (#23384)

• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)

• [fix][broker] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064)

• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)

• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)

• [fix][broker] Execute the pending callbacks in order before ready for incoming requests (#23266)

• [fix][broker] Fail fast if the extensible load manager failed to start (#23297)

• [fix][broker] Fix 'Disabled replicated subscriptions controller' logic and logging (#23142)

• [fix][broker] Fix AvgShedder strategy check (#23156)

• [fix][broker] Fix Broker was failing to create producer with broken schema ledger (#23395)

• [fix][broker] Fix Broker was failing to load stats-internal with broken schema ledger (#22845)

• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)

• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)

• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)

• [fix][broker] Fix ack hole in cursor for geo-replication (#20931)

• [fix][broker] Fix authenticate order in AuthenticationProviderList (#23111)

• [fix][broker] Fix broker OOM when upload a large package. (#22989)

• [fix][broker] Fix brokers still retry start replication after closed the topic (#23237)

• [fix][broker] Fix compatibility issues for PIP-344 (#23136)

• [fix][broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented (#22684)

• [fix][broker] Fix cursor should use latest ledger config (#22644)

• [fix][broker] Fix delivery-test with unnecessary invocation-count (#23399)

• [fix][broker] Fix doc of ThresholdShedder and remove useless method. (#22798)

• [fix][broker] Fix geo-replication admin client url (#22584)

• [fix][broker] Fix incomplete NAR file extraction which prevents broker from starting (#23274)

• [fix][broker] Fix namespace unload might be blocked too long with extensible load manager (#23433)

• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)

• [fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector (#23327)

• [fix][broker] Fix retry backoff for PersistentDispatcherMultipleConsumers (#23284)

• [fix][broker] Fix shadow topics cannot be consumed when the entry is not cached (#23147)

• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)

• [fix][broker] Fix system topic can not be loaded up if it contains data offloaded (#23279)

• [fix][broker] Fix the broker registering might be blocked for long time (#23371)

• [fix][broker] Fix the broker registery cannot recover from the metadata node deletion (#23359)

• [fix][broker] Fix the bug that elected leader thinks it's a follower (#23138)

• [fix][broker] Fix thread unsafe access on the bundle range cache for load manager (#23217)

• [fix][broker] Fix topic policies cannot be queried with extensible load manager (#23326)

• [fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)

• [fix][broker] Fix typo in PersistentTopicsBase (#22904)

• [fix][broker] Fix typos in pulsar-broker and tiered-storage. (#23415)

• [fix][broker] Fix unloadNamespaceBundlesGracefully can be stuck with extensible load manager (#23349)

• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)

• [fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)

• [fix][broker] Handle the case when getOwnedServiceUnits fails gracefully (#23119)

• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)

• [fix][broker] Internal reader of __change_events can not started after metadata store session rebuilt (#23018)

• [fix][broker] Introduce the last sent position to fix message ordering issues in Key_Shared (PIP-282) (#21953)

• [fix][broker] Let Pending ack handler can retry to init when encounters a metadata store error (#23153)

• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)

• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)

• [fix][broker] Partitioned shadow topic not work properly (#22797)

• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)

• [fix][broker] Remove blocking calls from Subscription.getStats (#23088)

• [fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)

• [fix][broker] Skip reading entries from closed cursor. (#22751)

• [fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)

• [fix][broker] Support large number of unack message store for cursor recovery (#9292)

• [fix][broker] The topic might reference a closed ledger (#22860)

• [fix][broker] Topic can never be loaded up due to broker maintains a failed topic creation future (#23184)

• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)

• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)

• [fix][broker] disable loadBalancerDirectMemoryResourceWeight by default (#22821)

• [fix][broker] disable loadBalancerMemoryResourceWeight by default (#22820)

• [fix][broker] fix can not cleanup heartbeat data if scaling down broker (#22750)

• [fix][broker] fix compile error for PersistentStickyKeyDispatcherMultipleConsumers (#23055)

• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)

• [fix][broker] fix exception may hidden and result in stuck when topic loading (#23102)

• [fix][broker] fix pulsar-admin topics stats-internal caused a BK client thread a deadlock (#23258)

• [fix][broker] fix replicated namespaces filter in filterAndUnloadMatchedNamespaceAsync (#23100)

• [fix][broker] fix replicated subscriptions for transactional messages (#22452)

• [fix][broker] fix the log format error in the AdminResource (#23278)

• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)

• [fix][broker] fix typo and useless local variable. (#22704)

• [fix][broker] fix unload bundle count metric. (#22895)

• [fix][broker] maintain last active info in memory only. (#22794)

• [fix][broker] normalize path (#23438)

• [fix][broker] rename to changeMaxReadPositionCount (#22656)

• [fix][broker] replace loadSheddingPipeline with loadSheddingStrategy. (#22786)

• [fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)

• [fix][broker] support peek-message for compressed and encrypted messages (#23234)

• [fix][broker] timeout when broker registry hangs and monitor broker registry (ExtensibleLoadManagerImpl only) (#23382)

• [fix][broker] topics failed to delete after remove cluster from replicated clusters set and caused OOM (#23360)

• [fix][broker] type cast on exceptions in exceptionally can lead to lost calls (#23117)

• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)

• [fix][broker] fix irregular method name (#23108)

• [fix][broker] A failed consumer/producer future in ServerCnx can never be removed (#23123)

• [fix][broker] Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)

• [fix][meta] Check if metadata store is closed in RocksdbMetadataStore (#22852)

• [fix][meta] Oxia metadta store: Convert error to MetadataStoreException if operation failed (#23154)

• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)

• [fix][ml] Fix race conditions in RangeCache (#22789)

• [fix][ml] Managed ledger should recover after open ledger failed (#23368)

• [fix][ml] Remove duplicated field initialization of ML (#22676)

• [fix][ml] Remove unnecessary return in ManagedLedgerImpl (#23418)

• [fix][ml]Fix subscription props could be lost in case of missing ledger during recovery (#22637)

• [fix][ml][PIP-327] fix recover from ledger when ledgerForceRecovery is true (#23426)

• [fix][offload] Break the fillbuffer loop when met EOF (#22722)

• [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache (#22679)

• [fix][schema] Error checking schema compatibility on a schema-less topic via REST API (#22720)

• [improve][admin] PIP-369 Change default value of unload-scope in ns-isolation-policy set (#23253)

• [improve][admin] PIP-369 Introduce unload flag in ns-isolation-policy set call (#23120)

• [improve][broker] Add ServiceUnitStateTableView (ExtensibleLoadManagerImpl only) (#23301)

• [improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)

• [improve][broker] Add callback parameters to the SendCallback.sendComplete (#23196)

• [improve][broker] Add limits for Key_Shared Subscription look ahead in dispatching (#23231)

• [improve][broker] Add log to track issue when handleGetTopicsOfNamespace (#23434)

• [improve][broker] Add retry for start service unit state channel (ExtensibleLoadManagerImpl only) (#23230)

• [improve][broker] Add subscription prefix for internal reader (#23044)

• [improve][broker] Avoid subscription fenced error with consumer.seek whenever possible (#23163)

• [improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)

• [improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)

• [improve][broker] Close protocol handlers before unloading namespace bundles (#22728)

• [improve][broker] Decouple ManagedLedger interfaces from the current implementation (#23311)

• [improve][broker] Do not call cursor.isCursorDataFullyPersistable if disabled dispatcherPauseOnAckStatePersistentEnabled (#22729)

• [improve][broker] Don't use "recently joined consumers" rules for Key_Shared in STICKY mode (#23275)

• [improve][broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only) (#23144)

• [improve][broker] Fail fast when it failed to create LoadSheddingStrategy instance (#22827)

• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)

• [improve][broker] Improve CPU resources usege of TopicName Cache (#23052)

• [improve][broker] Improve exception for topic does not have schema to check (#22974)

• [improve][broker] Improve pulsar_topic_load_failed metric to record correct failed time (#23199)

• [improve][broker] Make cluster metadata init command support metadata config path (#23269)

• [improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)

• [improve][broker] Optimize high CPU usage when consuming from topics with ongoing txn (#23189)

• [improve][broker] Optimize message payload traffic for ShadowReplicator (#23236)

• [improve][broker] Optimize performance for checking max topics when the topic is a system topic (#23185)

• [improve][broker] Optimize the performance of individual acknowledgments (#23072)

• [improve][broker] Reduce number of OpenTelemetry consumer attributes (#22837)

• [improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)

• [improve][broker] Reducing the parse of MessageMetadata in compaction (#23285)

• [improve][broker] Register the broker to metadata store without version id compare (#23298)

• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)

• [improve][broker] Remove ConcurrentOpenHashMap and ConcurrentOpenHashSet (#23329)

• [improve][broker] Remove unused method CompactionRecord.reset (#22670)

• [improve][broker] Replace ConcurrentOpenHashMap with ConcurrentHashMap in BrokerService (#23320)

• [improve][broker] Replace ConcurrentOpenHashMap with ConcurrentHashMap in Topic classes (#23322)

• [improve][broker] Reschedule reads with increasing backoff when no messages are dispatched (#23226)

• [improve][broker] Reuse topic OpenTelemetry attributes (#22876)

• [improve][broker] Should notify bundle ownership listener onLoad event when ServiceUnitState start (ExtensibleLoadManagerImpl only) (#23152)

• [improve][broker] Support customized shadow managed ledger implementation (#23179)

• [improve][broker] Support to specify auth-plugin, auth-parameters and tls-enable arguments when init cluster metadata (#23087)

• [improve][broker] Trigger offload on topic load (#22652)

• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)

• [improve][broker] avoid creating new objects when intercepting (#22790)

• [improve][broker] check system topic is used before configuring system topic (ExtensibleLoadManagerImpl only) (#23381)

• [improve][broker] don't do load shedding when metadata service not available (#23040)

• [improve][broker] high CPU usage caused by list topics under namespace (#23049)

• [improve][broker] log exception in MessageDeduplication#takeSnapshot (#22994)

• [improve][broker] make system topic distribute evenly. (#22953)

• [improve][broker] Change the log level to reduce repeated error logs (#23192)

• [improve][broker] Ensure namespace deletion doesn't fail (#22627)

• [improve][broker]Reuse method getAvailableBrokersAsync (#23099)

• [improve][broker][PIP-379] Add observability stats for "draining hashes" (#23429)

• [improve][broker][PIP-379] Don't replace a consumer when there's a collision (#23441)

• [improve][broker][PIP-379] Improve hash collision handling by restoring consumer when other leaves (#23447)

• [improve][misc] Replace dependencies on PositionImpl with Position interface (#22891)

• [improve][broker][PIP-384] Decouple Bookkeeper client from ManagedLedgerStorage and enable multiple ManagedLedgerFactory instances (#23313)

• [improve][broker][break change] Do not create partitioned DLQ/Retry topic automatically (#22705)

• [cleanup][broker] Remove PersistentSubscription.getStats (#23095)

• [cleanup][broker] Remove warn logs when changing the state from Owned to Free (Extensible LB) (#22708)

• [cleanup][broker] delete ConcurrentSortedLongPairSet. (#23202)

• [cleanup][broker] remove DeviationShedder. (#22800)

• [improve] Allow to construct Oxia metadata-store directly from a client instance (#22756)

• [improve] Refactored BK ClientFactory to return futures (#22853)

• [improve][meta] Fix invalid use of drain API and race condition in closing metadata store (#22585)

• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)

• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)

• [improve][offload] Allow to disable the managedLedgerOffloadDeletionLagInMillis (#22689)

• [improve][offload] Replace usage of shaded class in OffsetsCache (#22683)

• [improve][schema] Improve Incompatible Schema Exception error message in ServerCnx.java (#23344)

• [improve][txn] Take first snapshot before persisting the first transactional message (#21406)

Observability

• [Enhancement] Added a metric for geo replication for tracking replicated subscriptions snapshot timeouts (#22381)
• [feat][broker] Add broker health check status into prometheus metrics (#20147)
• [improve][admin] Improve BrokerStats.allocatorStats (#23242)
• [improve][broker] Add msgInReplay subscription stat and metric to improve Key_Shared observability (#23224)
• [improve][log] Print source client addr when enabled haProxyProtocolEnabled (#22686)

Client

• [cleanup][client] Remove unneeded sync scope in TransactionImpl (#22932)
• [fix] DLQ to handle bytes key properly (#23172)
• [fix][client] Add more info while throwing reader creation exception (#23401)
• [fix][client] Copy orderingKey to retry letter topic and DLQ messages and fix bug in copying (#23182)
• [fix][client] Create the retry producer async (#23157)
• [fix][client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry (#22639)
• [fix][client] Fix broker/Client CPU reaching 100% during retriable connection failure (#23251)
• [fix][client] Fix client handle unknown exception during message-decryption and apply decryption action accordingly (#23256)
• [fix][client] Fix client to handle permits for discarded and failed decrypt batch-message (#23068)
• [fix][client] Fix concurrent lookup with properties might have different results (#23260)
• [fix][client] Fix failover consumer-listener stuck with cumulative ack and epoch time (#23345)
• [fix][client] Fix for early hit beforeConsume for MultiTopicConsumer (#23141)
• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
• [fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted (#22854)
• [fix][client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed (#22858)
• [fix][client] Fix the javadoc for ConsumerBuilder.isAckReceiptEnabled (#23452)
• [fix][client] Fix timeout handling in Pulsar Admin client (#23128)
• [fix][client] PIP-344 Do not create partitioned metadata when calling pulsarClient.getPartitionsForTopic(topicName) (#22206)
• [fix][client] TransactionCoordinatorClient support retry (#23081)
• [fix][client] the nullValue in msgMetadata should be true by default (#22372)
• [improve][client] Add exception handle for client send error (#23038)
• [improve][client] Add log when can't add message to the container (#21924)
• [improve][client] Add maxConnectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder (#22541)
• [improve][client] Deprecate MessageIdUtils.getOffset and MessageIdUtils.getMessageId (#22747)
• [improve][client] Don't print info logs for each schema loaded by client (#23206)
• [improve][client] Increase default Java client connectionMaxIdleSeconds to 60 seconds (#23430)
• [improve][client] PIP-344 support feature flag supportsGetPartitionedMetadataWithoutAutoCreation (#22773)
• [improve][client] improve the class GetTopicsResult (#22766)
• [improve][client] Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover (#23129)
• [improve][client] PIP-359:Support custom message listener executor for specific subscription (#22861)

Pulsar IO and Pulsar Functions

• [fix][fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes (#22910)
• [fix][fn] Support compression type and crypto config for all producers in Functions and Connectors (#22950)
• [fix][io] Upgrade mssql server docker tag in DebeziumMsSqlContainer (#23318)
• [improve] Validate user paths in Functions utils (#22833)
• [improve][fn] Add support for overriding additionalJavaRuntimeArguments with PF_additionalJavaRuntimeArguments env (#23130)
• [improve][fn] Make producer cache bounded and expiring in Functions/Connectors (#22945)
• [improve][io] The JDBC connector supports JSON substructure schema (#23043)

Others

• [improve] PIP-337: Implement SSL Factory Plugin to customize SSL Context and SSL Engine generation (#23110)
• [improve][misc] Improve AES-GCM cipher performance (#23122)
• [improve][misc] Optimize TLS performance by omitting extra buffer copies (#23115)
• [improve][misc] Set format_version=5, checksum=kxxHash in Bookkeeper RocksDB configs (#23175)
• [improve][proxy] Add debug logs for proxy cnx management (#23037)
• [improve][proxy] Reuse authentication instance in pulsar-proxy (#23113)
• [feat] Add scripts for updating BK RocksDB ini files (#23178)
• [feat][ws] Add support for initialSubscriptionPosition in Websocket Consumer (#23013)
• [fix] Reapply shell script parameter passthrough fix #22867 reverted in #22921 (#22923)
• [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
• [fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
• [fix][cli] Fix Pulsar-Client CLI to print metadata of message including encryption metadata (#23347)
• [fix][cli] Fix Pulsar-Client to allow consume encrypted messages with appropriate crypto-failure-action (#23346)
• [fix][cli] Fix expiration of tokens created with "pulsar tokens create" (#22815)
• [fix][cli] Fix healthcheck script pulsar-zookeeper-ruok.sh (#22873)
• [fix][cli] Fix the pulsar-daemon parameter passthrough syntax (#22905)
• [fix][cli] Refactor scripts to detect major Java version and pass correct add-opens parameters (#22927)
• [fix][cli] Remove defaultValue of tenant/namespace for CmdFunctions (#22764)
• [fix][cli] Remove deprecated "-client" JVM arg (#23370)
• [improve][cli] Add compactor type option for compaction tool (#23446)
• [improve][cli] PIP-353: Improve transaction message visibility for peek-message (#22762)
• [improve][cli] Supports creating tokens with additional headers (#22690)
• [improve][cli] Use LoadManagerReport instead of Object (#22850)
• [fix][conf] fix configuration name and typo. (#22822)
• [fix][conf] fix typo for TransferShedder. (#22757)
• [fix][doc] Fix some typos in pip (#23288)
• [fix][doc] Fix the doc for the message redelivery backoff (#22855)
• [fix][doc] fix the class name of transaction exception. (#22687)
• [fix][log] Do not print error log if tenant/namespace does not exist when calling get topic metadata (#23291)
• [fix][log] Do not print warn log when concurrently publishing and switching ledgers (#23209)
• [fix][misc] Add proper nslookup (included in bind-tools) to docker image (#22878)
• [fix][misc] Correct the comment of tlsAllowInsecureConnection in ClusterDataImpl class. (#22919)
• [fix][misc] Correct the description of patternAutoDiscoveryPeriod (#22615)
• [fix][misc] Disable JFR based telemetry collection since it's not used (#22869)
• [fix][misc] Log Conscrypt security provider initialization warnings at debug level (#23364)
• [fix][misc] Remove RoaringBitmap dependency from pulsar-common (#23008)
• [fix][misc] Rename netty native libraries in pulsar-client-admin-shaded (#22954)
• [fix][misc] Topic name from persistence name should decode local name (#22879)
• [improve][doc] Add Pulsar Geo-Replication considerations to the PIP template (#23145)
• [improve] Configure Rocksdb to use musl libc flavor of the native library (#23375)
• [improve] Validate range of argument before long -> int conversion (#22830)
• [cleanup] Cleanup some duplicated code (#23204)
• [improve][pip] Improve PIP process and voting timeline (#23387)

Tests, Build & CI

• [fix][test] Address flaky GetPartitionMetadataMultiBrokerTest (#23456)
• [fix][test] Fix ReplicatorRateLimiterTest (#23369)
• [fix][test] Fix TableViewBuilderImplTest NPE and infinite loop (#22924)
• [fix][test] Fix flaky AuthorizationTest.testGetListWithGetBundleOp (#22713)
• [fix][test] Fix flaky GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic (#23259)
• [fix][test] Fix flaky SubscriptionSeekTest.testSeekIsByReceive (#23170)
• [fix][test] Fix flaky UnloadSubscriptionTest.testMultiConsumer (#23243)
• [fix][test] Fix flaky test LeaderElectionTest.revalidateLeaderWithinSameSession (#22383)
• [fix][test] Fix flaky test ManagedLedgerTest.testDeleteCurrentLedgerWhenItIsClosed (#23437)
• [fix][test] Fix flaky test testShadowWrites (#22745)
• [fix][test] Fixed many tests of pulsar-proxy are not running (#23118)
• [fix][test] Revert the modification to NonDurableSubscriptionTest caused by a mistake in the PR#23129 (#23168)
• [fix][test] Update OpenTelemetry receiver endpoint in integration test (#22998)
• [fix][test] fix testGetMetrics in ExtensibleLoadManagerImplTest. (#22864)
• [fix][test] Fix flaky test increase the delay time (#23046)
• [cleanup][test] delete invalid method in PrometheusMetricsTest (#23218)
• [fix] Fix compile failing (#23070)
• [fix] JWT CLI util should force the token validation (#22831)
• [fix][build] Add basic support for vscode-java and Eclipse IDE (#23448)
• [fix][build] Add curl command for pulsar image (#22732)
• [fix][build] Add re2/j dependency to pulsar-common and client shading (#22884)
• [fix][build] Disable flaky test BrokerRegistryIntegrationTest (#23367)
• [fix][build] Fix CVE-2024-2511 by upgrading to OpenSSL in docker image (#22731)
• [fix][build] Fix problem where git.commit.id.abbrev is missing in image tagging (#23337)
• [fix][build] Fix pulsar-client-python installation on ARM arch (#22733)
• [fix][build] Fixed creation of packages-storage directory in docker image (#22730)
• [fix][build] Remove duplicate dependencies in pom.xml (#23440)
• [fix][build] Remove unnecessary Oracle maven repository from pom.xml (#23132)
• [fix][ci] Fix OWASP Dependency Check download by using NVD API key (#22999)
• [fix][ci] Fix jacoco code coverage report aggregation (#22964)
• [fix][ci] Fix snappy-java native lib fails to load in x86 alpine (#22804)
• [fix][ci] Pin aquasecurity/trivy-action@0.26.0 since master is broken (#23431)
• [fix][ci] Replace removed macos-11 with macos-latest in GitHub Actions (#22965)
• [improve][build] Drop deprecated pulsar-client-1x and pulsar-client-2x-shaded modules (#23450)
• [improve][build] Improve docker-push (#22702)
• [improve][build] Move docker-push profile to submodule (#23093)
• [improve][build] Remove invalid relativePath definitions in pom.xml files (#22741)
• [improve][build] Require Java 17 or Java 21 for building Pulsar (#22875)
• [improve][build] Support custom image and label names (#22703)
• [improve][build] Support git worktree working directory while building docker images (#22851)
• [improve][build] Update maven-wrapper (mvnw) to recent stable version 3.3.2 (#23410)
• [improve][build] Upgrade dependency-check-maven-plugin to 10.0.2 (#23012)
• [improve][build] Upgrade docker-maven-plugin to 0.45.0 (#23091)
• [improve][build] Use amazoncorretto:21-alpine image instead of apk installation (#22973)
• [improve][misc] Replace rename-netty-native-libs.sh script with renaming with maven-shade-plugin (#22957)
• [improve][ci] Add arm64 image build (#22755)
• [improve][ci] Continue Pulsar CI build even when Trivy scanner fails (#23397)
• [improve][ci] Migrate from Gradle Enterprise to Develocity (#22880)
• [improve][ci] Move some flaky ExtensibleLoadManager tests to flaky group until they are fixed (#23414)
• [improve][ci] Switch to Java 21 as default JVM version for CI (#23373)
• [improve][ci] Switch to use DEVELOCITY_ACCESS_KEY from GRADLE_ENTERPRISE_ACCESS_KEY (#23090)
• [improve][ci] Upgrade/Downgrade test (#22988)
• [improve][test] Add a test to guarantee the TNX topics will not be replicated (#22721)
• [improve][test] Restart broker in SimpleProducerConsumerTest if ns deletion times out (#22734)