Skip to main content

Apache Pulsar 3.1.0

2023-08-10​

Important notice​

  • Fix the reason label of authentication metrics #20030

PIPs​

  • PIP-275: Introduce topicOrderedExecutorThreadNum to deprecate numWorkerThreadsForNonPersistentTopic #20504
  • PIP-255: Make the partition assignment strategy pluggable #20537
  • PIP-278: Support pluggable topic compaction service #20624

Broker​

  • Fix MessageDeduplication throw NPE when enabling broker deduplication and disabling namespace deduplication #20905
  • In replication scenarios, remote consumers could not be registered if there were no message sent #20888
  • Fix IndexOutOfBoundsException in the CompactedTopicUtils #20887
  • Add annotation for topic compaction strategy #20858
  • Fix inconsensus namespace policies by getPoliciesIfCached #20855
  • Pass bytesToRead when reading compacted entries #20850
  • Fix the built-in admin failed to delete a topic with a custom authz provider #20848
  • Avoid print redirect exception log when getting a list from bundle #20846
  • Inconsistent behaviour for topic auto_creation #20843
  • Follow up 4196 use PulsarByteBufAllocator handle OOM #20837
  • Add broker filter sync method back to guarantee the API compatibility #20826
  • Fix direct memory leak by delayed index OutOfDirectMemory #20823
  • Avoid infinite bundle unloading #20822
  • Add the MessageExpirer interface to make code clear #20800
  • Broker failed to load v1 namespace resources cache #20783
  • Cannot receive any messages after switching to a standby cluster #20767
  • Fix get topic policies as null during clean cache #20763
  • Fix typo in ServerCnx.java #20762
  • Avoid throwing RestException in BrokerService #20761
  • Call ServerCnxcloseProducer from correct thread #20747
  • Ignore openIDTokenIssuerTrustCertsFilePath conf when blank #20745
  • Support pluggable topic compaction service - part2 #20718
  • Revert "Skip loading broker interceptor when disableBrokerInterceptors is true 20422" #20710
  • Gracefully shutdown does not work with admin cli in standalone #20709
  • Improve CompactedTopicImpl lock #20697
  • Added the skipped message handler for ServiceUnitStateChannel #20677
  • Make ExtensibleLoadManagerImpl's broker filter pure async #20666
  • Add logs for topics or subscriptions when enabling rate limit #20660
  • Make get list from bundle Admin API async #20652
  • Fix the publish latency spike from the contention of MessageDeduplication #20647
  • Support pluggable topic compaction service - part1 #20645
  • Fix get owned service units NPE #20625
  • Topic policy cannot work well if the replay policy message has any exceptions #20613
  • Fix the publish latency spike issue with a large number of producers #20607
  • AuthenticationFilter.doFilter #20598
  • Fix NPE when resetting Replicator's cursor by position. #20597
  • Add consumer-id into the log when doing subscribe #20568
  • Release orphan replicator after topic closed #20567
  • New load balancer system topic should not be auto-created now #20566
  • Remove unused cache executor in PulsarService #20563
  • Handle get owned namespaces admin API in ExtensibleLoadManager #20552
  • Handle heartbeat namespace in ExtensibleLoadManager #20551
  • Validate authz earlier in delete subscription logic #20549
  • Should not throw NotFoundException when metadata already exists #20539
  • REST Client Producer fails with TLS only #20535
  • Fix redirect loop when using ExtensibleLoadManager and list in bundle admin API #20528
  • Restore solution for certain topic unloading race conditions #20527
  • Emit the namespace bundle listener event on extensible load manager #20525
  • Choose random thread for consumerFlow in PersistentDispatcherSingleActiveConsumer #20522
  • Disable EntryFilters for system topics #20514
  • Support get/remove permissions for AuthorizationProvider #20496
  • Improve the efficiency of checking message deletion #20490
  • Reformat property in generateResponseWithEntry #20481
  • Add metric prefix for topic_load_times #20472
  • Fix return the earliest position when query position by timestamp. #20457
  • Support revoking permission for AuthorizationProvider #20456
  • Replace String.intern() with Guava Interner #20432
  • Remove ineffective solution for reducing GC pressure #20428
  • Skip loading broker interceptor when disableBrokerInterceptors is true #20422
  • Avoid PersistentSubscription.expireMessages logic check backlog twice #20416
  • Change limitStatsLogging config default value to true #20409
  • PIP-255 Part-1: Add listener interface for namespace service #20406
  • Use immutable base cursor properties #20400
  • Fix partitioned __change_events topic is policy topic #20392
  • Do not expose bucketDelayedIndexStats #20383
  • Fix flaky test - testClusterMigrationWithReplica… #20379
  • Change some static fields referencing mutable objects to ordinary instance fields in BrokerService #20372
  • Fix pre-create non-partitioned system topics for load balance extension #20370
  • Fix memory leak when rebatchMessage #20369
  • Invalidate metadata children cache after key deleted #20363
  • Fix managedLedger.getConfig().getProperties().putAll(properties) NPE #20361
  • Update rest endpoint method names to avoid swagger conflict #20359
  • Fix broker load manager class filter NPE #20350
  • Copy BrokerEntryMetadata when rebatchMessage #20337
  • In Key_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck #20335
  • When creating a namespace anti-affinity group with the default local policies, use the bundle size supplied by the user. #20327
  • Fix skip message API when hole messages exist #20326
  • Gracefully shut down load balancer extension #20315
  • Fix class name typo PrecisPublishLimiter to "Precise" #20310
  • Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion #20304
  • Fix NPE cause by topic publish rate limiter. #20302
  • Remove Streaming Dispatcher Code #20279
  • Fix UnsupportedOperationException when updating topic properties. #20261
  • Fix ledger cachemiss size metric #20257
  • Fix default bundle size used while setting bookie affinity #20250
  • Fix the behavior of delayed message in Key_Shared mode #20233
  • Fix infinite ack of Replicator after topic is closed #20232
  • Producer created by replicator is not displayed in topic stats #20229
  • Use bulk operation instead of iteration #20219
  • Fix Return value of getPartitionedStats doesn't contain subscription type #20210
  • Improve knownBrokers update in ModularLoadManagerImpl #20196
  • Skip split bundle if only one broker #20190
  • Fix deadlock of metadata store #20189
  • Fix RoaringBitmap.contains can't check value 65535 #20176
  • Release EntryBuffer after parsing proto object #20170
  • Remove useless field in the DelayedMessageIndexBucketSegment.proto #20166
  • Optimization protobuf code in the bucket delayed tracker #20158
  • Move bitmap from lastMutableBucket to ImmutableBucket #20156
  • Producer/Consumer should call allowTopicOperationAsync #20142
  • Fix entry filter feature for the non-persistent topic #20141
  • Optimize delayed metadata index bitmap #20136
  • Fast fix infinite HTTP call getSubscriptions caused by wrong topicName #20131
  • Make timer execute immediately after load index #20126
  • Cache LedgerHandle in BookkeeperBucketSnapshotStorage #20117
  • Pin AppendIndexMetadataInterceptor to field in ManagedLedgerInterceptorImpl #20112
  • Close temporary open ledger in BookkeeperBucketSnapshotStorage #20111
  • Fix use error TimeUnit to record publish latency #20074
  • Fix the reason label of authentication metrics #20030
  • Save createIfMissing in TopicLoadingContext #19993
  • Log resource usage rate of brokers that need to be offloaded in ThresholdShedder #19983
  • Fix getPartitionedStats miss subscription's messageAckRate #19870
  • Fix wrong logic of method TopicName.getPartition(int index) #19841
  • Fix Update contains no change error when using --update-auth-data flag to update function/sink/source #19450
  • Add topic_load_failed metric #19236
  • Support getStats/update partitioned topic with -partition- #19235
  • Do not filter system topics while shedding. #18949
  • If ledger lost, cursor mark delete position can not forward #18620
  • Do not grant permission for each partition to reduce unnecessary zk metadata #18222
  • Support cgroup v2 by using jdk.internal.platform.Metrics in Pulsar Loadbalancer #16832

Pulsar IO and Pulsar Functions​

  • [IO] Not restart instance when kafka source poll exception. #20795
  • [IO] Add notifyError method on PushSource. #20791
  • [IO] elastic-search sink connector not support JSON.String schema #20741
  • [IO] Close the Kafka source connector got stuck #20698
  • [IO] Close the Kafka source connector if there is uncaught exception #20424
  • [IO] add protobuf ByteString to pulsar-io jdbc core #20259
  • [IO] KCA: handle kafka sources that use commitRecord #20121
  • [IO] Fix the wrong format of the logs #20907
  • [IO] Remove useless configuration for Pulsar SQL #20605
  • [IO] Some optimized points in 9224 #19027
  • [FN] Exit JVM when the main thread throws exceptions #20689
  • [FN] TLS args admin download command use zero arity #20513
  • [FN] Optimize string concatenation in user metrics gen #20499
  • [FN] Support customizing TLS config for function download command #20482
  • [FN] enable Go function token auth and TLS #20468
  • [FN] Reset idle timer correctly #20450
  • [FN] Add stateStorageURL and pulsarWebService URL to go InstanceConfig #20443
  • [FN] Go functions must retrieve consumers by non-partitioned topic ID #20413
  • [FN] Go functions need to use static grpcPort in k8s runtime #20404
  • [FN] Correct TLS cert config translation from broker to fn worker #20297
  • [FN] JavaInstanceStarter --tls_allow_insecure default to false #20267
  • [FN] Allow unknown fields in connectors config #20116
  • [FN] Use functions classloader in TopicSchema.newSchemaInstance() to fix ClassNotFoundException when using custom SerDe classes. (targeted for master) #20115
  • [FN] Upgrade Kotlin version from 1.4.32 to 1.8.20. (targeted for master) #20114
  • [FN] Check user metric len before iterating #20021
  • [FN] Support multiple input topics for Go runtime #20000
  • [FN] Make pulsar-admin support update py/go with package url #19897
  • [FN] Fix JavaInstanceStarter inferring type class name error #19896
  • [FN] Fix function update error #19895
  • [FN] Make KubernetesRuntime translate characters in function tenant, namespace, and name during function removal to avoid label errors #19584

Observability​

  • [Metrics] Add JVM start time metric #20381
  • [Metrics] Topic with double quote breaks the prometheus format #20230

CLI​

  • [Admin] Set ns level backlog quota does not take effect if retention exists #20690
  • [Admin] Remove duplicate topics name when deleteNamespace #20683
  • [Admin] Make admin.cli throw exception instead of silent catch #20530
  • [Admin] Pretty print bookies racks-placement command output #20516
  • [Admin] Make failed bin/pulsar-admin source command exit with code 1 (failed) instead of 0 (success) #20503
  • [Admin] Using AuthorizationService to revoke the permissions #20478
  • [Admin] Return BAD_REQUEST on cluster data is null for createCluster #20346
  • [Admin] Enhancing Transaction Buffer Stats and Introducing TransactionBufferInternalStats API #20330
  • [Admin] Fix examine messages if total message is zero #20152
  • [Admin] Make subscriptions on NonPersistentTopic non-durable #19741
  • [Admin] Report earliest msg in partitioned backlog #19465
  • [CLI] Fix the variable producerName of BatchMsgContainer is null #20819
  • [CLI] Fix error messages in CmdNameSpaces and CmdTopics #20701
  • [CLI] Dedicated docs tools #20700
  • [CLI] pulsar-client produce: allow to produce AVRO message from strings #20654
  • [CLI] Client: allow to specify KeyValue's schema key (Avro Key support) #20447
  • [CLI] Fulfill add-opens to function-localrunner also #20417
  • [CLI] Allow pulser-client consume create a replicated subscription #20316
  • [CLI] Add --cleanupSubscription to pulsar-admin #20028
  • [CLI] Fix logging noise while admin tool exit #19884
  • [CLI] PerformanceProducer do not produce expected number of messages #19775
  • [CLI] Pulsar-perf: refactor to reduce code duplication #19279

Others​

  • [TXN] Add log when can't send TcClientConnectRequest to broker #20880
  • [TXN] Use PulsarResource check for topic existence instead of brokerservice.getTopic() #20569
  • [TXN] Unwrap the completion exception. #20396
  • [TXN] Implement compatibility for transaction buffer segmented snapshot feature upgrade #20235
  • [TXN] Fix transaction is not aborted when send or ACK failed #20055
  • [TXN] fix consumer can receive aborted txn message when readType is replay #19815
  • [ML] cleanup numManagedLedgerWorkerThreads #20454
  • [ML] Fix ledger left in OPEN state when enabling inactiveLedgerRollOverTimeMs #20276
  • [ML] There are two same-named managed ledgers in the one broker #18688
  • [META] Bookie Info lost by notification race condition. #20642
  • [META] Adding the missed bookie id in the registration manager #20641
  • [META] Upgrade to jetcd to 0.7.5 #20339
  • [META] Switch to the metadata store thread after zk operation #20303
  • [META] deadlock of zkSessionWatcher when zkConnection loss #20122
  • [META] Support set metadata size threshold for compression #19561
  • [WS] Add cryptoKeyReaderFactoryClassName into the file websocket.conf #20840
  • [WS] Remove unnecessary ping/pong implementation #20733
  • [WS] Use allowTopicOperationAsync for authz checks #20299
  • [WS] Use async auth method to support OIDC #20238
  • [MISC] Stop logging full stack trace when SecurityUtility.loadConscryptProvider() fails to find Conscrypt with an unknown error #20705
  • [MISC] Fix typo in pip template #20556
  • [MISC] Use ubuntu 22.04 for Pulsar images #20475
  • [MISC] Upgrade Netty to 4.1.93.Final #20423
  • [misc] Consolidate certs in broker (and some proxy) tests #20353
  • [MISC] Catch TimeoutException when logging about time outs #20349
  • [Schema] Only handle exceptions when there has #20730
  • [Schema] Add admin cli for testCompatibility #19974
  • [Build] Fix generating javadoc issue #20922
  • [build] Do not bundle swagger dependencies #20687
  • [Build] Build function-go in golang image #20665
  • [Build] Configure git-commit-id-plugin to skip git describe #20550
  • [Build] Fix the pulsar-all image may use the wrong upstream image #20435
  • [Build] Fix publish image script #20305
  • [Build] Capture local build scans on ge.apache.org to benefit from deep build insights #20187

Library updates​

  • Upgrade Guava to 32.1.1 to address CVE-2023-2976 #20699
  • Upgrade snappy-java to address multiple CVEs #20604
  • Upgrade Jetty to address CVEs #20461
  • Suppress etcd CVE warnings #20342
  • Upgrade PyYaml version to 6.0.1 #20831
  • Upgrade jackson version to 2.15.0 for CVE-2022-1471 #20177
  • Upgrade prometheus client_golang to v1.12.2 to fix CVE-2022-21698 #20579
  • Upgrade Netty to 4.1.93.Final #20423
  • Upgrade to jetcd to 0.7.5 #20339
  • Upgrade Kotlin version from 1.4.32 to 1.8.20. (targeted for master) #20114

Documentation​

  • [feat][doc] add overview for broker load balancing #621
  • [feat][doc] add concepts for broker load balancing #644
  • [feat][doc] add use cases + features + benefits + types for broker load balancing #652
  • [feat][doc] add Quick Start for broker load balancing #660
  • [feat][doc] add migration tutorials for broker load balancing #657
  • [feat][doc] add metrics for broker load balancing #646
  • [feat][doc] add configurations for broker load balancing #661
  • [feat][doc] add development docs for broker load balancing #662
  • [improve][doc] clarify detailed behaviors for failover subscription type #633
  • [improve][doc] add docs for supporting reading config options from files in Python runner #544
  • [improve][doc] add solutions for debezium-source-mysql connection errors #649
  • [feat][doc] add docs for OpenID Connect Auth Provider #555
  • [feat][doc] add docs for openIDKeyIdCacheMissRefreshSeconds setting #580
  • [feat][doc] add docs for KubernetesServiceAccountTokenAuthProvider #570
  • [feat][doc] add client matrix page #513
  • [feature][doc] add docs for client memory limits #571
  • [feature][doc] add docs for CPP client table view #565
  • [improve][doc] add compatibility reminders for retry queues #651
  • [feat][doc] add docs for JVM metrics in Pulsar broker #567
  • [feat][doc] add docs for bucket delayed message tracker metrics #564
  • [fix][doc] exclude outdated versions of API documentation from search engine results #623
  • [feat][website] implement 'How does Pulsar work' homepage screen #614

For the complete list, including all features, enhancements, and bug fixes, check out the following resources.

A special thanks to the following contributors (alphabetic Github IDs) who contributed to Pulsar 3.1.0: Technoboy-, crossoverJie, lifepuzzlefun, poorbarcode, coderzc, liangyepianzhou, Demogorgon314, mattisonchao, BewareMyPower, yaalsn, zymap, lhotari, daziz, shibd, rdhabalia, eltociear, tisonkun, aloyszhang, michaeljmarshall, 315157973, JooHyukKim, zhanghaou, heesung-sn, codelipenghui, eolivelli, ethqunzhong, massakam, StevenLuMT, gaoran10, ericsyh, zuobiao-zhou, AnonHxy, xiaotongwang1, maanders-tibco, yuruguo, vineeth1995, flowchartsman, hanmz, nicoloboschi, Shawyeok, RobertIndie, nodece, TakaHiR07, hleecs, MMirelli, thetumbled, pgier, syk-coder, dragonls, bpereto, liqiangcc, clayburn, 1Jack2, asafm, tiny-rain, dlg99, gmiklos-ltg, cbornet, aymkhalil, jiangpengcheng, congbobo184, wangjialing218, dao-jun, csthomas1, teabot, labuladong, tjiuming, ziang123, shashrath.

Was this helpful?
💡 Suggest changes🛟 Get support