Skip to main content

Apache Pulsar 2.10.0

2022-04-13​

Important notice​

  • Remove -XX:-ResizePLAB JVM option which degrades performance on JDK11 #12940
  • Enable TCP keepAlive flag on the sockets #12982
  • Reduce the time it takes for namespace bundle unloading to time out #12995
  • Align configurations defaults between default file and Java object (broker.conf, proxy.conf, websocket.conf) #13272
  • [PIP 118] Do not restart brokers when ZooKeeper session expires as default #13341
  • [PIP 119] Enable consistent hashing by default on KeyShared subscriptions dispatcher #13352
  • [PIP 120] Enable client memory limit controller by default #13344
  • Make Docker images non-root, by default, and OpenShift compliant #13376
  • [PIP 122] Change loadBalancer default loadSheddingStrategy to ThresholdShedder #13733
  • Fix netcat returning early for probe #14088

PIPs​

  • [PIP 79] Add lazy-loading feature to PartitionedProducer #10279
  • [PIP 84] Pulsar client: Redeliver command add epoch #10478
  • [PIP 86] Pulsar Functions: Preload and release external resources #13205
  • [PIP 92] Topic policy across multiple clusters #12517
  • [PIP 104] Add new consumer type: TableView #12838
  • [PIP-105] Support pluggable entry filter in Dispatcher #12869 #12970 #12979
  • [PIP 106] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility #12536
  • [PIP 107] Introduce chunk message ID #12403
  • [PIP 110] Support Topic metadata - PART-1 create topic with properties #12818
  • [PIP 121] Pulsar cluster level auto failover on client side #13316
  • [PIP-124] Create init subscription before sending message to DLQ #13355
  • [PIP-130] Apply redelivery backoff policy for ack timeout #13707
  • [PIP 131] Resolve produce chunk messages failed when topic level maxMessageSize is set #13599
  • [PIP 135] Include MetadataStore backend for Etcd #13225

Broker​

  • [PIP 45] Pluggable metadata interface
    • Add BookKeeper metadata adapter based on MetadataStore #12770
    • Add Rocksdb metadata store #12776
    • Convert BookieRackAffinityMapping to use MetadataStore #12841
    • Allow to configure metadata store URL in broker.conf #13077
    • Remove old ZK caches implementations #13075
    • Allow to start bookie with Pulsar metadata store backend #13296
    • Remove ZooKeeperClientFactory #13303
    • Use reference counting in RocksDB metadata store #13309
    • Allow configuring metadata store URL in proxy.conf #13777
    • Allow configuring metadata store URL in functions_worker.yml #13782
    • Add configuration metadata store url setting in WebSocket #13786
    • Allow configuring configurationMetadataStore when initializing cluster metadata #13889
    • Using the consistent metadata store scheme name #13937
    • Allow configuring metadataStoreUrl in pulsar-perf managed-ledger #14145
    • Deprecate zookeeper settings #14147
    • Fix metadata cache inconsistency on doing refresh #14283
  • Support broker level dispatch rate limiter #11325
  • Support setting geo-replication clusters on topic level #12136
  • Add Multi-tiered storage key to broker.conf #12173
  • Support configuration to rate-limit dispatching on batch message #12294
  • Support splitting the largest bundle of the namespace #12361
  • Support creating a subscription by specifying the earliest or latest position #12872
  • Support roll-over ledgers for inactive topics #13073
  • Support graceful shutdown for Broker #14114
  • Transparent batching of ZK operations #13043
  • Add uniform load shedder strategy to distribute traffic uniformly across brokers #12902
  • Provide option to split bundle based on load #12378
  • Use AuthorizationService#grantPermissionAsync to grant topic permission #12515
  • Only store authentication data after authentication is complete #12077
  • Allow to have different instances LocalMemoryMetadataStore that share the same state #12390
  • Allow GetTopicsOfNamespace op with consume permission #12600
  • Add local filesystem backend for package manager #12708
  • Add stop replicator producer logic when start replicator cluster failed #12724
  • Apply PolicyHierarchyValue to inactiveTopicPolicies #12687
  • Fix can not get leader broker in follower brokers #11353
  • Fix broker gc log options #11285
  • Fix NPE of ZkBookieRackAffinityMapping #11947
  • Fix prefix setting in JWT authn and avoid multi calls for the getProperty #12132
  • Fix missed check exit code of stop before calling start #12368
  • Fix getting children of parent nodes in LocalMemoryMetadataStore #12491
  • Fix collection get bug in ResourceGroupService #12499
  • Fix deadlock in metadata-store callback thread #12753
  • Improve exceptions thrown when handling the schema resource #12155
  • Trim topic name #12453
  • Avoid unnecessary recalculation of maxSubscriptionsPerTopic in AbstractTopic #12658
  • Optimize isValidPath check in MetadataStore #12663
  • Close connection after receiving unexpected SendCommand #12780
  • Ensure cache is refreshed (and not just invalidated) after a store write #12788
  • Optimize topicMaxMessageSize with topic local cache. #12830
  • Optimize blocking backlogQuotaCheck to non-blocking in ServerCnx#handleProducer #12874
  • Only refresh metadata if path is already in cache after write. #12896
  • Optimize put and get methods in AbstractMetadataStore #12916
  • Fix zk-node leak of admin path #12972
  • Optimize TopicPolicy#deduplicationEnabled with HierarchyTopicPolicies #13000
  • Fix race condition in FaultInjectionMetadataStore#programmedFailure #13007
  • Fix usage of PULSAR_EXTRA_OPTS/BOOKIE_EXTRA_OPTS in startup scripts #13025
  • Consider topics in pulsar/system namespace as system topics #13050
  • Fix wrong result for looking up a non-exist topic by rest api #13055
  • Key_Shared dispatcher with no connected consumers should be recreated if allowOutOfOrderDelivery changes #13063
  • Make load-balancer config dynamic for the runtime tuning #13074
  • Optimize TopicPolicy#maxProducersPerTopic with HierarchyTopicPolicies #13082
  • Load balancer supports disabling max-session for bundle split #13108
  • Optimize TopicPolicies#subscriptionTypesEnabled with HierarchyTopicPolicies #13121
  • Log thread dump when Zookeeper session expires to help detect possible deadlocks #13124
  • Add removeMaxConsumersPerSubscription method for v1 namespace #13192
  • Fixe error value for 1MB in metrics #13213
  • Optimize TopicPolicies#messageTTLInSeconds with HierarchyTopicPolicies #13241
  • Add schema compatibility strategy on topic level #13297
  • Optimize TopicPolicies#maxConsumerPerTopic with HierarchyTopicPolicies #13361
  • Optimize TopicPolicies#replicationClusters with HierarchyTopicPolicies #13495
  • Fix compatibility issue with other metadata store implementations in resources #13394
  • Fix race condition in stopping replicator while it is starting #13412
  • Support backlog quota across multiple clusters #13445
  • Return null instead of RestException when getting bookieAffinityGroup #13462
  • Fix failed to get Partitioned metadata for health checking #13525
  • Optimize TopicPolicies#maxConsumersPerSubscription with HierarchyTopicPolicies #13548
  • Fix raw use of generic types in OwnershipCache #13592
  • Optimize TopicPolicies#maxUnackedMessagesOnConsumer with HierarchyTopicPolicies #13618
  • Optimize TopicPolicies#delayedDelivery Enabled and TickTimeMillis with HierarchyTopicPolicies #13649
  • Optimize TopicPolicies#compactionThreshold with HierarchyTopicPolicies #13710
  • Add deleteSubscriptionDispatchRate method for v1 namespace #13711
  • Fix NPE of checkReplication #13720
  • Optimize deduplicationSnapshotIntervalSeconds with HierarchyTopicPolicies #13769
  • Use shared executors for broker and geo-replication clients #13839
  • Fix call sync method in async rest api for
    • for internalDeletePartitionedTopic #13805
    • for internalCreateSubscription. #13873
    • for internalGetBacklogSizeByMessageId #13871
    • for internalTriggerCompaction #13853
    • for internalGetManagedLedgerInfo #13847
    • for internalGetSubscriptions #13846
    • for internalUnloadTopic #13845
    • for internalGetLastMessageId #13882
    • for internalExpireMessagesByTimestamp #13880
    • for internalResetCursorOnPosition #13879
    • for internalExpireMessagesByPosition #13878
    • for internalGetMessageById. #13876
    • for internalSetReplicatedSubscriptionStatus #13887
    • for internalGetPartitionedStats #13886
    • for onPoliciesUpdate method #13885
    • for internalGetPartitionedStatsInternal #13884
    • for internalTerminatePartitionedTopic #13890
    • for internalSkipAllMessages #13901
    • for preValidation #13962
    • for internalSetReplicationClusters and internalRemoveReplicationClusters #13961
  • Make PulsarAuthorizationProvider#grantPermissionAsync actually async #13897
  • Fix no value present when updating dispatch rate #13900
  • Optimize retentionPolicies with HierarchyTopicPolicies #13909
  • Improve 403 response message wih consume permission on namespace #13912
  • Fix arithmetic exception for uniform load shedder #13914
  • Add threshold for each unload round for uniform load shedder #13915
  • Fix call sync method in async rest api for internalDeleteSubscription #13884 #13917
  • Change BrokersBase api getActiveBrokers and getLeaderBroker to pure async. #13935
  • Do not create missing topic when loading namespace #13948
  • Only reply to client when code completes producerFuture #13949
  • Improve error logging for topic not found #13950
  • Make validateTenantOperation method async in PulsarWebResource #14008
  • Optimize maxUnackedMessagesOnSubscription with HierarchyTopicPolicies #14011
  • Set default value of applied to false in getSchemaCompatibilityStrategy #14012
  • Make validateTopicPolicyOperation method async in PulsarWebResource #14024
  • Fix producerFuture.completeExceptionally not called before sendErrorResponse #14025
  • Make triggerOffload method async #14027
  • Make offloadStatus method async #14029
  • Make validateNamespacePolicyOperation method async in PulsarWebResource #14033
  • Make internalSkipMessages method async #14045
  • Make PersistentTopicsBase#internalSetBacklogQuota async #14051
  • Adjust the validation for policy schemaCompatibilityStrategy #14061
  • Remove Persistent Topics v3 API - use custom media type instead #14117
  • Fix race condition in PulsarLedgerIdGenerator#generateShortLedgerId #14118
  • Optimize topic policy with HierarchyTopicPolicies about subscriptionDispatchRate #14151
  • Make BrokerBase#deleteDynamicConfiguration to pure async method #14163
  • Set default value of applied to false on topic policy #14181
  • Change broker producer fence log level #14196
  • Remove duplicated filter for UniformLoadShedder#findBundlesForUnloading #14198
  • Adjust topic exists check logic in http lookup process #14199
  • Fix NPE of internalExpireMessagesByTimestamp #14243
  • Fix rackaware placement policy does not take effect after delete rack configuration #14248
  • Fix print error log when server return redirect (http code 307) #14259
  • Optimize topic policy with HierarchyTopicPolicies about publishRate #14267
  • Fix ack-hole and backlog for persistent-replicator #14282
  • Fix NPE in internalSkipMessages #14297
  • Validate blank advertised listener name #14306
  • Let entries expire in the metadata caches #14154
  • Fix transaction system topic loop creation #12749
  • Fix topic transaction buffer handle null snapshot #12758
  • Optimize changeToCloseState method #14277

Clients​

  • [C] Add missing includes in reader_configuration.h #12966
  • [C] Add pulsar_client_subscribe_multi_topics and pulsar_client_subscribe_pattern #12965

Pulsar IO and Pulsar Functions​

  • [Functions] Prevent NPE while stopping a non started Pulsar LogAppender #12643
  • [Functions] Allow configuring different implementations for Pulsar functions state store #12646
  • [Functions] Clean os even when statusFuture complete exceptionally #12767
  • [Functions] Override inactive_topic_policies in Pulsar Functions namespace creation #13048
  • [Functions] Add possibility to pass additional JVM arguments to the function JVM (additionalJavaRuntimeArguments) #13282
  • [Functions] Fix getTlsTrustChainBytes not work when functions worker not run with broker #13875
  • [Functions] Fix distributed log metadata not correctly initialized #13891
  • [Functions] Remove sensitive information from log #14159
  • [IO Connector] Pass client builder if no service url provided to debezium connector #12145
  • [IO Connector] Fix: "Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed for user 'sa'.." in MS SQL integration test 12374
  • [IO Connector] Fix the marshal and unmarshal the sink config #12625
  • [IO Connector] Support renaming the file that has been processed #13373
  • [IO Connector] Support Fixed and ENUM datatypes for ElasticSearch Sink #13800
  • [IO Connector] Implement --retain-key-ordering (KEY_SHARED subscription) for Sinks #14083

Observability​

  • [Broker] Expose broker bundles metrics to prometheus #12366
  • [Broker] Add publishRateLimitedTimes to topic metrics #13538
  • [Broker] Fix bundle metrics would overwrite loadbalance metrics #13641
  • [Broker] Fix managed cursor acknowledgment state metric names #13844
  • [Java Client] Add pending-queue size metrics to producer stats #12674

CLI​

  • [Pulsar Admin] Support a mapping from brokers to partitioned topics that each broker serves #11763
  • [Pulsar Admin] Support the termination of a partitioned topic #11893
  • [Pulsar Admin] Update command descriptions from old 'property/cluster/namespace' format to current 'tenant/namespace' format #10485
  • [Pulsar Admin] Add remove-subscription-types-enabled command for namespace #12392
  • [Pulsar Admin] Fix output format of string by pulsar-admin command #11878
  • [Pulsar Admin] Add a metric to get the earliest time in the backlog #12523
  • [Pulsar Admin] Reduce severity of log "refreshing key manager" in KeyManagerProxy #12594
  • [Pulsar Admin] Support getting a list of topics under a namespace bundle #12632
  • [Pulsar Admin] Add get-replicated-subscription-status command for topic #12891
  • [Pulsar Admin] Add remove-subscription-types-enabled command for topic #12983
  • [Pulsar Admin] Validate the size options in cmd for topic and namespace #13002
  • [Pulsar Admin] Support subscription across multiple clusters #13482
  • [Pulsar Admin] Support deduplication across multiple clusters #13487
  • [Pulsar Admin] Support message TTL across multiple clusters #13484
  • [Pulsar Admin] Support persistence policies across multiple clusters #13483
  • [Pulsar Admin] Support publish rate across multiple clusters #13496
  • [Pulsar Admin] Support max consumers across multiple clusters #13521
  • [Pulsar Admin] Support max producer across multiple clusters #13519
  • [Pulsar Admin] Support compaction threshold across multiple clusters #13513
  • [Pulsar Admin] Support offload policies across multiple clusters #13534
  • [Pulsar Admin] Support max unacked messages per consumer across multiple clusters #13547
  • [Pulsar Admin] Support max message size support across multiple clusters #13579
  • [Pulsar Admin] Support deduplication snapshot interval across multiple clusters #13578
  • [Pulsar Admin] Support delayed delivery policy across multiple clusters #13550
  • [Pulsar Admin] Support max unacked messages on subscription across multiple clusters #13549
  • [Pulsar Admin] Support replicator dispatch rate across multiple clusters #13624
  • [Pulsar Admin] Support max subscriptions per topic across multiple clusters #13623
  • [Pulsar Admin] Move schema compatibility strategy cmd from topics to topicPolicies #14225
  • [Pulsar CLI] Add restart command to pulsar-daemon #12279
  • [Pulsar CLI] Print log for standalone when configuration is failed to load #12280
  • [Client Tool] Support Disabling Replication #13659
  • [Perf Tool] Add auth for transaction perf #14271

Others​

  • [Proxy] Fix Pulsar Proxy to re-use authentication instance #12245
  • [Proxy] Fix auto-cert refresh when proxy connects to broker #14130
  • [WebSocket] Fix the batch message ack #12530
  • [Package Management] check service status before run commands #12847
  • [Package Management] Add package management filesystem storage into the distribution lib #13202
  • [Package Management] Fix the filesystem storage failure #13218
  • [Schema] Fix pulsar use json or avro primitive schema #12886
  • [Tiered Storage] Fix FileSystemManagedLedgerOffloader can not cleanup outdated ledger #12309
  • [Tiered Storage] Fix NoClassDefFoundError: com/google/inject/AbstractModule in pulsar-io/batch-data-generator and Jcloud offloader #14150
  • Add lua wireshark for pulsar #13564
  • Fix missing new line at the end of proxy.conf #14359

Library updates​

  • Upgrade function's Go client version to v0.7.0 #12839
  • Upgrade dependencies (guava and what brought in older guava) to get rid of the guava-related CVE-2018-10237 and CVE-2020-8908 #13716
  • Use dependencyManagement to enforce snakeyaml version to 1.30 #13722
  • Upgrade dependencies to get rid of pulsar-io/jdbc related CVE-2020-13692 #13753
  • Remove --illegal-access errors resulting from Google Guice (upgrade to 5.0.1 and JClouds to 2.4.0) #13810
  • Upgrade jakarta.el to 3.0.4 to get rid of CVE-2021-28170 #13943
  • Upgrade Netty to 4.1.73.Final #13981
  • Remove net.jodah.failsafe dependency (fix JDK17 build) #14124
  • Upgrade netty version to 4.1.74.Final #14257
Was this helpful?
💡 Suggest changes🛟 Get support