Skip to main content

Apache Pulsar 3.0.0

2023-05-02​

Important notice​

  • Upgrade the RocksDB version to 7.9.2 to keep sync with BookKeeper's RocksDB dependency in #20072
  • Introduce consistent hash ring to distribute the load of multiple topics subscription in #19502
  • Remove default 30s ackTimeout when setting DLQ policy on java consumer in #19486
  • Require authRole is proxyRole to set originalPrincipal in #19455
  • Update Elasticsearch sink idle cnx timeout to 30s in #19377
  • Move checked exception into builder when newTransaction. in #19356
  • TokenAuthenticationState: authenticate token only once in #19314
  • OneStageAuth State: move authn out of constructor in #19295
  • Update AuthenticationProvider to simplify HTTP Authn in #19197
  • Remove AuthorizationProvider methods deprecated in 2.7 and 2.9 in #19182
  • Deprecate blocking AuthorizationService, AuthorizationProvider methods in #19180
  • Reject create non existent persistent partitions. in #19086
  • Omit making a copy of CommandAck when there are no broker interceptors in #18997
  • Use correct file path for tls trust certificates in #18712
  • Enable certificate refresh for Quorum and Netty Servers in #18097
  • Remove timestamp from Prometheus metrics in #17419
  • Revert 5895: fix redeliveryCount in #17060
  • Fix producer/consume permission can’t get v1/schema in #16018
  • Autorecovery default reppDnsResolverClass to ZkBookieRackAffinityMapping in #15640
  • Allow to build and push multi-arch Docker images in #19432

PIPs​

  • PIP-160 Metrics stats of Transaction buffered writer #15370
  • PIP-167 Make it Configurable to Require Subscription Permission #15576
  • PIP-169 Support split bundle by flow or qps #16782
  • PIP-188: Cluster migration or Blue-Green cluster deployment support in Pulsar #16551
  • PIP-192: New Pulsar Broker Load Balancer #16691
  • PIP-193: Sink preprocessing Function #16739
  • PIP-195: New bucket based delayed message tracker #16763
  • PIP-214: Add broker level metrics statistics and expose to prometheus #18056
  • PIP-218: Consumer batchReceive() single partition every receive #18182
  • PIP-229: Add a common interface to get fields of the MessageIdData #18950
  • PIP-250: Add proxyVersion to CommandConnect #19623
  • PIP-257: Add Open ID Connect Support to Server Components #19771

Broker​

  • [Broker] incomplete PIP-97: Asynchronous Authentication Provider #12105
    • Implement for ServerCnx by @michaeljmarshall in #19409
    • Implement for ProxyConnection by @michaeljmarshall in #19292
  • [Broker] incomplete PIP-180: Shadow Topic, an alternative way to support readonly topic ownership. #16153
    • Add ShadowManagedLedgerImpl by @Jason918 in #18265
    • Support shadow topic creation. by @Jason918 in #17711
    • Add Shadow Replicator by @Jason918 in #17371
  • [Broker] PIP-192: New Pulsar Broker Load Balancer #16691
    • Fix getLastMessageId for compressed payload(And add compression and maxBatchSize for the load balance system topic) by @heesung-sn in #20087
    • Improve TransferSheder for overload outlier for large clusters by @heesung-sn in #20059
    • Update the lookup data path to support deployment and rollback by @Demogorgon314 in #19999
    • Filter the transfer dest broker by @Demogorgon314 in #19958
    • Updated metrics and cleanup broker selector by @heesung-sn in #19945
    • Support delete namespace bundle admin API by @Demogorgon314 in #19851
    • Improved Auto Unload Logic by @heesung-sn in #19813
    • Support broker isolation policy by @Demogorgon314 in #19592
    • Add large topic count filter by @Demogorgon314 in #19613
    • Added VersionId in ServiceUnitStateData by @heesung-sn in #19620
    • Added Deleted and Init states in ServiceUnitState by @heesung-sn in #19546
    • Added broker and top-bundles load reporters by @heesung-sn in #19471
    • Added TransferShedder by @heesung-sn in #18865
    • Added ServiceUnitStateCompactionStrategy by @heesung-sn in #19045
    • Added TopicCompactionStrategy for StrategicTwoPhaseCompactor and TableView. by @heesung-sn in #18195
    • Implement load data store by @Demogorgon314 in #18777
    • Implement broker registry for new load manager by @Demogorgon314 in #18810
    • Added ServiceUnitStateChannelImpl by @heesung-sn in #18489
  • [Broker] PIP-195: New bucket based delayed message tracker #16763
    • New bucket based delayed message tracker - interface&config&proto -part 1 by @coderzc in #17344
    • Implement delayed message index bucket snapshot (create/load) - part2 by @coderzc in #17611
    • Implement BookkeeperBucketSnapshotStorage - part3 by @coderzc in #17677
    • Support internal cursor properties - part4 by @coderzc in #17712
    • Implement BucketDelayedDeliveryTrackerFactory and load BucketDelayedDeliveryTracker - part6 by @coderzc in #17756
    • Implement delayed message bucket snapshot recover - part5 by @coderzc in #18420
    • Implement Filter out all delayed messages and skip them when reading messages from bookies - part7 by @coderzc in #19035
    • Fix cursor skip read by @coderzc in #19124
    • Implement delayed message index bucket snapshot(merge/delete) - part8 by @coderzc in #19138
    • Make BucketDelayedDeliveryTracker can retry snapshot operation & improve logs by @coderzc in #19577
    • Fix BucketDelayedDeliveryTracker merge issues by @coderzc in #19615
    • Cut off snapshot segment according to maxIndexesPerBucketSnapshotSegment by @coderzc in #19706
    • Add metrics for bucket delayed message tracker by @coderzc in #19716
    • Don't clean up BucketDelayedDeliveryTracker when all consumer disconnect by @coderzc in #19801
    • Add topicName and cursorName for ledger metadata of bucket snapshot by @coderzc in #19802
    • Make bucket merge operation asynchronous by @coderzc in #19873
    • Clear delayed message when unsubscribe & Make clear operation asynchronous by @coderzc in #19901
    • Merge multiple buckets at once by @coderzc in #19927
    • Ensure previous delayed index be removed from snapshotSegmentLastIndexTable & Make load operate asynchronous by @coderzc in #20086
    • Fix the issue of the future of clean-up delayed messages can't be completed by @coderzc in #20075
    • Support disabling delayed bucket merging. by @mattisonchao in #20155
  • [Broker] Pip-257: Add AuthenticationProviderOpenID by @michaeljmarshall in #19849
  • [Broker] Make LeastResourceUsageWithWeight thread safe by @lifepuzzlefun in #20159
  • [Broker] Supply download auth params when provided for k8s runtime by @michaeljmarshall in #20144
  • [Broker] AuthenticationProviderOpenID k8s error logs by @michaeljmarshall in #20135
  • [Broker] Implement authenticateAsync for AuthenticationProviderList by @michaeljmarshall in #20132
  • [Broker] Fix broker restart logic by @wolfstudy in #20113
  • [Broker] Refresh auth data if ProxyLookupRequests by @michaeljmarshall in #20067
  • [Broker] Fix flaky testCreateTopicWithZombieReplicatorCursor by @BewareMyPower in #20037
  • [Broker] Prevent range conflicts with Key Shared sticky consumers when TCP/IP connections get orphaned by @lhotari in #20026
  • [Broker] Fix leader broker log by @nicoloboschi in #19987
  • [Broker] Make LedgerOffloaderFactory can load the old nar. by @horizonzy in #19913
  • [Broker] Ignore and remove the replicator cursor when the remote cluster is absent by @BewareMyPower in #19972
  • [Broker] Fix DeadLetterProducer creation callback blocking client io thread. by @lifepuzzlefun in #19930
  • [Broker] Fix the loss of bundle stats data reported to zookeeper, when the updateStats method is executed by @lordcheng10 in #19887
  • [Broker] Fix can't send ErrorCommand when message is null value by @coderzc in #19899
  • [Broker] PIP-240: new public method unloadSubscription in PersistentTopic by @poorbarcode in #19737
  • [Broker] PIP-250: Add proxyVersion to CommandConnect by @michaeljmarshall in #19618
  • [Broker] Suppress error logging when message expiration fails by @massakam in #19778
  • [Broker] Allow proxy to pass same role for authRole and originalRole by @michaeljmarshall in #19557
  • [Broker] Make authentication refresh threadsafe by @michaeljmarshall in #19506
  • [Broker] Fix RetentionPolicies constructor by @nodece in #19777
  • [Broker] Fix create cluster with empty url by @nodece in #19762
  • [Broker] Fix potential exception cause the policy service init fail. by @Technoboy- in #19746
  • [Broker] Validate originalPrincipal earlier in ServerCnx by @michaeljmarshall in #19270
  • [Broker] Fix issue where msgRateExpired may not refresh forever by @massakam in #19759
  • [Broker] Fix index generator is not rollback after entries are failed added. by @gaozhangmin in #19727
  • [Broker] Change type of allowAutoTopicCreationType to TopicType by @yuruguo in #18814
  • [Broker] Store the original authentication data by @nodece in #19519
  • [Broker] GetLastMessageId returns a wrong batch index of last message if enabled read compacted by @poorbarcode in #18877
  • [Broker] PIP-188 support blue-green cluster migration [part-1] by @rdhabalia in #17962
  • [Broker] Embed stringified exception in placeholder in log message by @massakam in #19633
  • [Broker] Transaction buffer recover blocked by readNext by @poorbarcode in #18833
  • [Broker] Fix geo-replication admin by @nodece in #19548
  • [Broker] Counter of pending send messages in Replicator incorrect if schema future not complete by @poorbarcode in #19242
  • [Broker] Export pulsar_delayed_message_index_size_bytes in subscription level by @coderzc in #19595
  • [Broker] Store nonBlank clientVersions that have spaces by @michaeljmarshall in #19616
  • [Broker] Require authRole is proxyRole to set originalPrincipal by @michaeljmarshall in #19455
  • [Broker] Print stack trace when getting exception in ServerCnx by @wangjialing218 in #19593
  • [Broker] Make the service name resolver cache of PulsarWebResource expire after access by @poorbarcode in #19532
  • [Broker] OneStageAuth State: move authn out of constructor by @michaeljmarshall in #19295
  • [Broker] Forbid uploading BYTES schema with admin API by @labuladong in #18995
  • [Broker] Add test to verify authRole cannot change by @michaeljmarshall in #19430
  • [Broker] Topic load fail by ledger lost by @poorbarcode in #19444
  • [Broker] Terminate the async call chain when the condition isn't met for resetCursor by @lhotari in #19541
  • [Broker] Make ServerCnx#originalAuthData volatile by @michaeljmarshall in #19507
  • [Broker] Cleanup finalPosition null-check in asyncFindPosition by @AnonHxy in #19497
  • [Broker] ServerCnx: go to Failed state when auth fails by @michaeljmarshall in #19312
  • [Broker] Refactor update topic partitions endpoint. by @mattisonchao in #19166
  • [Broker] Fix mutex never released when trimming by @315157973 in #17911
  • [Broker] PIP-214 Add broker level metrics statistics and expose to prometheus by @yyj8 in #19047
  • [Broker] Fix delete system topic clean topic policy by @liangyepianzhou in #18823
  • [Broker] Support zookeeper read-only config. by @horizonzy in #19156
  • [Broker] Modify check waitingForPingResponse with volatile (#12615)" by @michaeljmarshall in #19439
  • [Broker] Expect msgs after server initiated CloseProducer by @michaeljmarshall in #19446
  • [Broker] Close transactionBuffer after MessageDeduplication#checkStatus failed by @tjiuming in #19157
  • [Broker] Topic could be in fenced state forever if deletion fails by @nicoloboschi in #19129
  • [Broker] Fix the incorrect total size if use ML interceptor by @poorbarcode in #19404
  • [Broker] Fix currency bug in BucketDelayedDeliveryTracker#recoverBucketSnapshot by @lhotari in #19394
  • [Broker] Added isActive in ManagedCursorImpl by @heesung-sn in #19341
  • [Broker] Replace sync method in async call chain of internalSetBacklogQuota by @lhotari in #19398
  • [Broker] Replace sync call in async call chain of AdminResource#internalCreatePartitionedTopic by @lhotari in #19399
  • [Broker] Reduce calls on metadata store / ZK event thread & Netty threads in PersistentTopic by @lhotari in #19388
  • [Broker] Release Netty buffer in finally block in ServerCnx#handleLastMessageIdFromCompactedLedger by @lhotari in #19395
  • [Broker] Replace sync method in NamespacesBase#internalDeleteNamespaceBundleAsync by @lhotari in #19391
  • [Broker] Fix concurrency bug in PersistentTopicsBase#internalGetReplicatedSubscriptionStatus by @lhotari in #19389
  • [Broker] Remove synchronous method call in async call chain in PersistentTopicsBase by @lhotari in #19387
  • [Broker] Update interceptor handler exception by @tuteng in #18940
  • [Broker] Add lowerBoundarySheddingEnabled conf by @crossoverJie in #19351
  • [Broker] Fix uncompleted future when getting the topic policies of a deleted topic by @liangyepianzhou in #18824
  • [Broker] Simplify extract entryMetadata code in filterEntriesForConsumer by @lifepuzzlefun in #18729
  • [Broker] Catch exception for brokerInterceptor by @aloyszhang in #19147
  • [Broker] Fix RetentionPolicies types mismatch. by @Technoboy- in #18114
  • [Broker] Make Consumer#equals more effective by @Shawyeok in #18662
  • [Broker] TokenAuthenticationState: authenticate token only once by @michaeljmarshall in #19314
  • [Broker] Fix namespace deletion if __change_events topic has not been created yet by @nicoloboschi in #18804
  • [Broker] Fix can not delete namespace by force by @liangyepianzhou in #18307
  • [Broker] Replace PulsarService Mockito spy solution for overriding getters in tests by @lhotari in #19323
  • [Broker] Fix issue where leader broker information isn't available after 10 minutes by @lhotari in #17401
  • [Broker] Add UncaughtExceptionHandler for every thread pool by @315157973 in #18211
  • [Broker] Replace authenticate with authenticateAsync by @michaeljmarshall in #19313
  • [Broker] Topics stats shows msgBacklog but there reality no backlog by @poorbarcode in #19275
  • [Broker] Only create extended partitions when updating partition number by @AnonHxy in #17349
  • [Broker] Let TokenAuthState update authenticationDataSource by @michaeljmarshall in #19282
  • [Broker] Eagerly Delete Offloaded Segments On Topic Deletion by @dlg99 in #17915
  • [Broker] Documentation for AuthenticationState contract by @michaeljmarshall in #19283
  • [Broker] Allow to use io_uring instead of epoll by @coderzc in #18385
  • [Broker] Using TopicName instead of String as the parameter for getTopic. by @Technoboy- in #17416
  • [Broker] Skip creating a subscription replication snapshot if no messages have been published after the topic gets activated on a broker by @lhotari in #16618
  • [Broker] Avoid bundle unload destination broker be set as an inactive broker. by @Nicklee007 in #19244
  • [Broker] Update AuthenticationProvider to simplify HTTP Authn by @michaeljmarshall in #19197
  • [Broker] Reduce GetReplicatedSubscriptionStatus local REST call by @AnonHxy in #16946
  • [Broker] Improve tls config on replication client and cluster cli by @nodece in #17295
  • [Broker] Persist correct markDeletePosition to prevent message loss by @michaeljmarshall in #18237
  • [Broker] Deprecate blocking AuthorizationService, AuthorizationProvider methods by @michaeljmarshall in #19180
  • [Broker] Allow user lookup topic name with -partition- but no metadata by @mattisonchao in #19171
  • [Broker] Add ref count for sticky hash to optimize the performance of Key_Shared subscription by @codelipenghui in #19167
  • [Broker] Copy proto command fields into final variables in ServerCnx by @michaeljmarshall in #18987
  • [Broker] Add parameter check for create/update cluster. by @mattisonchao in #19151
  • [Broker] In the trimDeletedEntries method, release the removed entry by @lordcheng10 in #18305
  • [Broker] Remove locallyAcquiredLock when removeOwnership by @leizhiyuan in #18197
  • [Broker] Support setting forceDeleteTenantAllowed dynamically by @Technoboy- in #18192
  • [Broker] Use ChannelVoidPromise to avoid useless promise objects creation by @nicoloboschi in #19141
  • [Broker] Fix ClusterDataImpl#clone and add test by @michaeljmarshall in #19126
  • [Broker] Fix invalid metric type "gauge " by @coderzc in #18129
  • [Broker] Fix ttl expiration block due to no-recoverable exception even if autoSkipNonRecoverableData=true by @aloyszhang in #19132
  • [Broker] Update the log print content of createSubscriptions by @Pomelongan in #18024
  • [Broker] Fix the order of resource close in the InMemoryDelayedDeliveryTracker by @coderzc in #18000
  • [Broker] Fix incorrect bundle split count metric by @Shawyeok in #17970
  • [Broker] Update authentication failed metrics report by @tuteng in #17787
  • [Broker] ManagedLedger: move to FENCED state in case of BadVersionException by @eolivelli in #17736
  • [Broker] Do not print error log with stacktrace for 404 by @codelipenghui in #19130
  • [Broker] Fix executeWithRetry result is null by @coderzc in #17694
  • [Broker] Fix can not revoke permission after update topic partition by @TakaHiR07 in #17393
  • [Broker] Add oshi library to help control OS resources. by @mattisonchao in #18990
  • [Broker] Fix missing return when internalGetReplicatedSubscriptionStatus by @HQebupt in #19054
  • [Broker] Throw AlreadyClosedException while request to a closed metadata store by @codelipenghui in #19055
  • [Broker] Fix thread safety issue in info-internal admin api for partitioned topics by @lhotari in #19021
  • [Broker] Only get consumer future when interceptor not null by @michaeljmarshall in #19022
  • [Broker] Use validateTopicOperationAsync methods in all cases by @lhotari in #19016
  • [Broker] Fix deadlock in PendingAckHandleImpl by @nicoloboschi in #18989
  • [Broker] Add javadoc for PulsarDecoder handle* method parameter lifecycle by @lhotari in #18998
  • [Broker] Fix direct memory leak in RawReaderImpl by @lhotari in #18928
  • [Broker] Make it Configurable to Require Subscription Permission by @michaeljmarshall in #15576
  • [Broker] Add config fsyncEnable for RocksdbMetadataStore by @coderzc in #18801
  • [Broker] Remove the unused method internalGetList from PersistentTopicsBase by @codelipenghui in #18775
  • [Broker] Fix duplicated schemas creation by @codelipenghui in #18701
  • [Broker] Use correct file path for tls trust certificates by @michaeljmarshall in #18712
  • [Broker] The interval of scheduled task should be greater than 0 by @yuruguo in #18728
  • [Broker] Improve shadow topic error message by @labuladong in #18709
  • [Broker] Combination of autocreate + forced delete of partitioned topic with active consumer leaves topic metadata inconsistent. by @dlg99 in #18193
  • [Broker] System topic writer/reader connection not counted by @yuruguo in #18603
  • [Broker] First judge whether config is null when standalone start by @yuruguo in #18681
  • [Broker] Fix thread-safety issue with currentLedgerEntries and currentLedgerSize fields by @lhotari in #17868
  • [Broker] Fix namespace not found will cause request timeout by @labuladong in #18512
  • [Broker] Avoid sharing the same random in multi-threads due to a performance issue by @Shawyeok in #18660
  • [Broker] System topic writer/reader connection not counted. by @Technoboy- in #18369
  • [Broker] Avoid unnecessary creation of BitSetRecyclable objects by @lordcheng10 in #17998
  • [Broker] Fix broker cache eviction of entries read by active cursors by @lhotari in #17273
  • [Broker] Add return for PersistentMessageExpiryMonitor#findEntryFailed by @Pomelongan in #17816
  • [Broker] Fix create ns by @nodece in #17864
  • [Broker] DnsResolverUtil.TTL should be greater than zero by @Technoboy- in #18565
  • [Broker] Make managedLedgerOffloadedReadPriority compatible with broker property by @ethqunzhong in #17803
  • [Broker] Remove legacy code by @aloyszhang in #18330
  • [Broker] Remove duplicate code to improve delete subscription by @nodece in #15347
  • [Broker] Fix issue where logs get truncated when Runtime.halt is used to terminate the process by @lhotari in #17589
  • [Broker] Add missing @Test annotation by @michaeljmarshall in #18528
  • [Broker] Unreasonable AuthenticationException reference by @zzccctv in #18502
  • [Broker] Remove useless argument for PersistentTopicBase.internalSkipAllMessagesForNonPartitionedTopicAsync by @poorbarcode in #18473
  • [Broker] Using handle instead of handleAsync to avoid using common pool thread by @Technoboy- in #17403
  • [Broker] In OverloadShedder, repeatedly filter local bundles by @lordcheng10 in #18463
  • [Broker] Fix OutOfDirectMemoryError in standalone with zk by @labuladong in #18337
  • [Broker] ENABLE_IO_URING property name by @coderzc in #18402
  • [Broker] Provide readable consumer stats time by @rdhabalia in #14251
  • [Broker] Fix delete_when_subscriptions_caught_up doesn't work while have active consumers by @codelipenghui in #18283
  • [Broker] Support autoSubscriptionCreation on topic level by @yuruguo in #17720
  • [Broker] Avoid object create in RedeliveryTracker.getRedeliveryCount by @lifepuzzlefun in #18352
  • [Broker] Remove check ack.getMessageIdsCount() == 1 for cumulative ack by @AnonHxy in #18333
  • [Broker] Fix repeated messages of shared streaming dispatcher by @poorbarcode in #18289
  • [Broker] Cleanup unused method: isSlowestReader by @lordcheng10 in #18306
  • [Broker] Get lowest PositionImpl from NavigableSet by @lifepuzzlefun in #18278
  • [Broker] Remove duplicate sendSuccess and sendError in PulsarCommandSender by @lifepuzzlefun in #18250
  • [Broker] Cleanup localStore from NamespaceResources by @AnonHxy in #18258
  • [Broker] Cleanup NamespaceService by @AnonHxy in #18259
  • [Broker] Make CompactedTopicImpl.findStartPointLoop work more efficiently by @poorbarcode in #17976
  • [Broker] Do not serialize field bundleStats by @thetumbled in #18150
  • [Broker] Fix flaky test: PrometheusMetricsTest.testDuplicateMetricTypeDefinitions by @nicoloboschi in #18077
  • [Broker] Avoid runtime check hasFilter in EntryFilterSupport by @AnonHxy in #18066
  • [Broker] Remove methods no longer used in AdminResource #7 by @Pomelongan in #17969
  • [Broker] Delete unused metrics of zk_write_latency and zk_read_latency by @liangyuanpeng in #17505
  • [Broker] ServerCnx: log at warning level when topic not found by @michaeljmarshall in #16225
  • [Broker] Set log level to warn when topic policy reader occurs AlreadyClosedException by @Technoboy- in #17841
  • [Broker] Add config to count filtered entries towards rate limits by @michaeljmarshall in #17686
  • [Broker] Fix unexpected subscription deletion caused by the cursor last active time not updated in time by @dragonls in #17573
  • [Broker] Remove unnecessary variable in PersistentTopic by @poorbarcode in #17511
  • [Broker] Make cursor properties support modify single value concurrently. by @coderzc in #17164
  • [Broker] Do not log stacktrace for 'Failed to flush mark-delete position' case by @eolivelli in #17432
  • [Broker] Refactor reflection method in delayed and service module by @HQebupt in #17347
  • [Broker] Remove unnecessary parameters(reusefuture) and related logic by @Pomelongan in #17378
  • [Broker] Fix NPE when updating topic properties by @Flowermin in #17352
  • [Broker] Refactor reflection method in loadbalance module by @HQebupt in #17313
  • [Broker] Make deletePersistence method async in Namespaces by @Pomelongan in #17206
  • [Broker] PIP-169: Support split bundle by flow or qps by @lordcheng10 in #16557
  • [Broker] Add Maven Modernizer plugin in pulsar-broker and fix violation in test: broker/admin by @youzipi in #16993
  • [Broker] Remove unused code by @lordcheng10 in #17036

Pulsar IO and Pulsar Functions​

  • [Functions] Supply download auth params when provided for k8s runtime by @michaeljmarshall in #20144
  • [Functions] Make /version return correct version by @michaeljmarshall in #20047
  • [Functions] Use AuthorizationServer more in Function Worker API by @michaeljmarshall in #19975
  • [Functions] Allow download link with basic auth by @vanhtuan0409 in #19750
  • [Functions] Fix k8s merge runtime opts bug by @michaeljmarshall in #19481
  • [Functions] Support .nar files without explicit directory entries by @lhotari in #19304
  • [Functions] Fix TLS configuration for fn worker to broker if auth disabled by @nicoloboschi in #19290
  • [Functions] Support reading config options from file in Function Python Runner by @tpiperatgod in #18951
  • [Functions] Support processingGuarantees "EFFECTIVELY_ONCE" in python function by @tpiperatgod in #18929
  • [Functions] Typo in method name by @tisonkun in #18844
  • [Functions] Support e2e cryption in python instance by @jiangpengcheng in #18738
  • [Functions] Support schema in python instance by @jiangpengcheng in #18432
  • [Functions] Support SubscriptionPosition in python instance by @jiangpengcheng in #18431
  • [Functions] KCA connectors: fix missing runtime dependencies by @nicoloboschi in #18370
  • [Functions] Fix function failed to start if no typeClassName provided in FunctionDetails by @freeznet in #18111
  • [Functions] Do not delete managed package when deleting function by @jiangpengcheng in #18030
  • [Functions] Fix builtin sink transformation on k8s by @cbornet in #18019
  • [Functions] JDBC Sink: add a flag to exclude non-declared fields by @nicoloboschi in #18008
  • [Functions] Fix the download of builtin Functions by @cbornet in #17877
  • [Functions] Fix K8S download function method with auth enabled by @nicoloboschi in #17597
  • [Functions] Add admin CLI command to get available built-in functions by @cbornet in #16822
  • [Functions] Remove unused code by @cbornet in #16472
  • [Connectors] Remove kafka-connect-avro-converter-shaded by @tisonkun in #19468
  • [Connectors] Keep new inputSpecs when updating sink configs by @aymkhalil in #19082
  • [Connectors] Upgrade Kafka client, connect runtime to 2.8.2 and Confluent version to 6.2.8 by @lhotari in #19010
  • [Connectors] Reduce dependencies of pulsar-io-common module by @nicoloboschi in #19000
  • [Connectors] Elasticsearch sink: Support loading config from secrets by @alpreu in #18986
  • [Connectors] KCA connectors: fix missing runtime dependencies by @nicoloboschi in #18370
  • [Connectors] JDBC sinks: implement JDBC Batch API by @nicoloboschi in #18017
  • [Connectors] Add option to copy Kafka headers to Pulsar properties by @aymkhalil in #17829
  • [Connectors] Upgrade hadoop client to 3.3.3 and 2.10.2 by @codelipenghui in #17727
  • [Connectors] Support Transform Function with LocalRunner by @cbornet in #17445
  • [Connectors] Fix OutputRecordSinkRecord getValue and getSchema by @cbornet in #17434
  • [Connectors] ElasticSearch Sink: add an option to copy key fields into the value by @nicoloboschi in #17117
  • [Connectors] Add support for a transform Function in Sinks by @cbornet in #16740
  • [Connectors] KCA: handle kafka's logical schemas by @dlg99 in #16485
  • [Connectors] Add support for full message synchronization by @shink in #16003

Observability​

  • [Metrics] Fix the partitioned publisher topic stat aggregation bug by @heesung-sn in #18807
  • [Metrics] Fix reporting pulsar_subscription_blocked_on_unacked_messages metric by @aymkhalil in #18621
  • [Metrics] Correctly set byte and message out totals per subscription by @pgier in #18451
  • [Metrics] Wrong metrics text generated when label_cluster specified by @poorbarcode in #17704
  • [Metrics] Remove timestamp from Prometheus metrics by @michaeljmarshall in #17419
  • [Metrics] Add metrics for BatchMetadataStore by @tjiuming in #17072
  • [Metrics] Add metadata store metrics by @tjiuming in #17041
  • [Metrics] PIP-160 Metrics stats of Transaction buffered writer by @poorbarcode in #16758
  • [Metrics] Add limitStatsLogging configuration when enabling bookkeeperClientExposeStatsToPrometheus by @TakaHiR07 in #16734
  • [Metrics] Ensure prometheus metrics are grouped by type (#8407, #13865) by @marksilcox in #15558
  • [Metrics] Add metrics for transaction by @tjiuming in #15140

CLI​

  • [Pulsar Admin] Use AuthenticationParameters for rest producer by @michaeljmarshall in #20046
  • [Pulsar Admin] Use AuthorizationServer more in Function Worker API by @michaeljmarshall in #19975
  • [Pulsar Admin] Fix incorrect state replication.connected on API partitioned-topic stat by @poorbarcode in #19942
  • [Pulsar Admin] Fix delete tenant by @nodece in #19925
  • [Pulsar Admin] Filter pending ack topic while deleting the namespace by @gaoran10 in #19719
  • [Pulsar Admin] Fix create-subscription command --subscription description by @yangl in #19591
  • [Pulsar Admin] Allow proxy to pass same role for authRole and originalRole by @michaeljmarshall in #19557
  • [Pulsar Admin] Fix async method composition in admin client's NamespacesImpl by @lhotari in #19397
  • [Pulsar Admin] Make the default value of param --get-subscription-backlog-size of admin API topics stats true by @poorbarcode in #19302
  • [Pulsar Admin] Fix race condition while updating partition number by @mattisonchao in #19199
  • [Pulsar Admin] Improve partitioned-topic condition evaluation by @gaozhangmin in #19015
  • [Pulsar Admin] InternalGetMessageById shouldn't be allowed on partitioned topic by @gaozhangmin in #19013
  • [Pulsar Admin] Optimize the logic of internalCreatePartitionedTopic by @Pomelongan in #18666
  • [Pulsar Admin] Set authentication when using loadConf in client and admin client by @onobc in #18358
  • [Pulsar Admin] Add offload managedLedgerOffloadThreshold RestAPI and CLI tools by @tjiuming in #18218
  • [Pulsar Admin] Passing force parameter to deleteNamespaceBundle when deleting namespaces by @coderzc in #18153
  • [Pulsar Admin] Returns 4xx error when pulsar-worker-service is disabled and trying to access it by @heesung-sn in #17901
  • [Pulsar Admin] Add command to list transaction coordinators by @nicoloboschi in #17522
  • [Pulsar Admin] Namespace CLI set-offload-policy's thresholdBytes support negative and 0 by @ethqunzhong in #17502
  • [Pulsar Admin] Make setPersistence method async in Namespaces by @Pomelongan in #17421
  • [Pulsar Admin] Make getBundleRange async by @Technoboy- in #17402
  • [Pulsar Admin] Improve backlogQuota endpoint to pure async. by @mattisonchao in #17383
  • [Pulsar Admin] Improve replicatorDispacherRate endpoint to pure async. by @mattisonchao in #17377
  • [Pulsar Admin] PulsarAdminBuilderImpl overrides timeout properties passed through config map by @Technoboy- in #17375
  • [Pulsar Admin] Fix Npe thrown by splitBundle by @gaozhangmin in #17370
  • [Pulsar Admin] Add cmd to remove topic properties by @yuruguo in #17337
  • [Pulsar Admin] Add api for update topic properties by @AnonHxy in #17238
  • [Pulsar Admin] Make deleteNamesapce api async by @gaozhangmin in #17230
  • [Pulsar Admin] Cleanup unused param in provisionPartitionedTopicPath by @AnonHxy in #17156
  • [Pulsar Admin] Reduce code duplication in admin client by @andrasbeni in #16377
  • [Pulsar Admin] Pulsar Admin: reduce code duplication - part 4 by @eolivelli in #13086
  • [Pulsar CLI] Fix Pulsar admin tool is ignoring tls-trust-cert path arg by @rdhabalia in #19696
  • [Pulsar CLI] Add read command to pulsar-client-tools by @massakam in #19298
  • [Pulsar CLI] Pulsar cli: use return code 1 for general error by @pgier in #18946
  • [Pulsar CLI] Pulsar-perf fails on jdk17 by @nicoloboschi in #18806
  • [Pulsar CLI] Fix timeout behavior of reader perf by @labuladong in #17882
  • [Pulsar CLI] Using separate TLS config on the compactor by @nodece in #17426
  • [Client Tool] Pulsar shell: ensure admin client is recycled or disposed by @nicoloboschi in #17619
  • [Client Tool] Pulsar-perf produce add possibility to set eventTime on messages by @eolivelli in #17189
  • [Client Tool] Pulsar-perf consume, do not fail in case of reading data older than 10 days by @eolivelli in #17160

Others​

  • [Proxy] Refresh auth data if ProxyLookupRequests by @michaeljmarshall in #20067
  • [Proxy] Only create ConnectionPool when needed by @michaeljmarshall in #20062
  • [Proxy] Fix connection read timeout handling in Pulsar Proxy by @lhotari in #20014
  • [Proxy] ProxyConnection should not call super.exceptionCaught by @michaeljmarshall in #19990
  • [Proxy] Remove unnecessary executor callback; use assert by @michaeljmarshall in #19670
  • [Proxy] PIP-250: Add proxyVersion to CommandConnect by @michaeljmarshall in #19618
  • [Proxy] ProxyWithAuthorizationTest remove SAN from test certs by @michaeljmarshall in #19594
  • [Proxy] Allow proxy to pass same role for authRole and originalRole by @michaeljmarshall in #19557
  • [Proxy] Fix using wrong client version in pulsar proxy by @RobertIndie in #19540
  • [Proxy] Fix JKS TLS transport by @nodece in #19485
  • [Proxy] Modify check waitingForPingResponse with volatile (#12615)" by @michaeljmarshall in #19439
  • [Proxy] Only go to connecting state once by @michaeljmarshall in #19331
  • [Proxy] PIP 97: Implement for ProxyConnection by @michaeljmarshall in #19292
  • [Proxy] Remove unused AuthenticationDataSource variable by @michaeljmarshall in #19278
  • [Proxy] Use ChannelVoidPromise to avoid useless promise objects creation by @nicoloboschi in #19141
  • [Proxy] Prevent DNS reverse lookup when physical address is an IP address by @lhotari in #19028
  • [Proxy] Fix logging in shutdown when proxy shutdown by @coderzc in #18622
  • [Proxy] Fix refresh client auth by @nodece in #17831
  • [Proxy] Update proxy lookup throw exception type by @tuteng in #17600
  • [Proxy] Fix producer info logging fails when proxyLogLevel > 0 by @zengguan in #17175
  • [Proxy] Support proxy limit maximum connections per IP by @mattisonchao in #17167
  • [Proxy] Remove empty else code block when initializing setMetadataStoreUrl by @aloyszhang in #17122
  • [Proxy] Use correct address for HAProxyMessage destination by @michaeljmarshall in #16045
  • [Schema] Fix delete schema command doc by @congbobo184 in #19411
  • [Schema] Fix schema get value doc problem by @congbobo184 in #17740
  • [Schema] Fix producer/consume permission can’t get v1/schema by @yuruguo in #16018

Library updates​

  • Upgrade the BookKeeper dependency to 4.16.1 by @hangc0276 in #20127
  • Upgrade snakeyaml version to 2.0 by @Technoboy- in #20085
  • Upgrade the RocksDB version to 7.9.2 to keep sync with BookKeeper's RocksDB dependency by @hangc0276 in #20072
  • Upgrade Vertx to match BK's version (otherwise BK does not work) by @dlg99 in #20070
  • Upgrade Spring dependency to 5.3.26 to fix OWASP Dependency Check by @lhotari in #20029
  • Upgrade lombok to 1.8.26 by @merlimat in #19426
  • Upgrade to zk 3.8.1 by @dlg99 in #19425
  • Upgrade wildfly-elytron (used by debezium) to fix CVE-2022-3143 by @dlg99 in #19333
  • Upgrade Netty to 4.1.87.Final by @lhotari in #19417
  • Upgrade Spotbugs to a version with JDK 17 compatibility by @lhotari in #19315
  • Upgrade kafka client to 3.4.0 to fix CVE-2023-25194 by @massakam in #19527
  • Upgrade dependency-check-maven plugin to fix broken OWASP check by @lhotari in #19170
  • Upgrade HBase to 2.4.15 by @sekikn in #19249
  • Upgrade ASF Maven parent pom version to 29 by @lhotari in #19165
  • Upgrade woodstox to 5.4.0 by @nicoloboschi in #19041
  • Upgrade scala-library to get rid of CVE-2022-36944 by @nicoloboschi in #18021
  • Upgrade jettison to 1.5.3 by @nicoloboschi in #19038
  • Upgrade Debezium to 1.9.7 by @dlg99 in #18465
  • Upgrade Confluent version to 6.2.8 by @lhotari in #19010
  • Upgrade JacksonXML to 2.13.4 by @nicoloboschi in #18020
  • Upgrade commons-text to 1.10.0 in the kinesis-sink by @nicoloboschi in #18093
  • Upgrade reload4j in file-system offloader by @nicoloboschi in #17716
  • Upgrade OpenSSL to version 1.1.1n by @RobertIndie in #17538
  • Upgrade to latest docbot action by @maxsxu in #17266
  • Upgrade hadoop client to 3.3.3 and 2.10.2 by @codelipenghui in #17727
  • Upgrade PrestoSQL to the first Trino version by @tisonkun in #16683

Documentation​

Here are the most noteworthy changes in this release.

  • Add fresh new docs for Pulsar admin API #504, #498, #493, #490, #471, #462, #403
  • Refactor the information architecture of client library docs #393
  • Refactor the information architecture of Schema topics #18242
  • Add docs for deploying a Pulsar cluster on the IBM cloud #399
  • Add observability docs #502
  • Add get started for transaction #501
  • Add docs and illustrations for failover #514
  • Add docs for access mode (producer) #537
  • Add docs for supporting reading config options from files in Python runner #544
  • Add docs for enabling replicated subscription #510
  • Add docs for env variables (CLI) #422
  • Add docs for schema support (Python functions) #486
  • Add docs for message dispatch throttling #386
  • Add docs for proxy version protocol #531

For the complete list, including all features, enhancements, and bug fixes, check out the following resources.

A special thanks to the following contributors (alphabetic Github IDs) who contributed to Pulsar 3.0.0: 0xflotus, Abhilash Mandaliya, Alexander Preuß, Ali Ahmed, AloysZhang, Alvaro, AlvaroStream, Andras Beni, Andrey Yegorov, Anonymitaet, Apurva007, Asaf Mesika, Ayman Khalil, Baodi Shi, Chris Bono, Christophe Bornet, Clay Johnson, Cong Zhao, Dave Duggins, Dave Fisher, Dave Maughan, David Dieruf, Dawn Leamon, Dezhi LIiu, Dianjin Wang, Elliot West, Enrico Olivelli, Eric Shen, Ethan Waldo, Fangbin Sun, Flowermin, GitHub, Guangning E, Hang Chen, Heesung Sohn, Hongten, HuangWei, HuangZeGui, Huanli Meng, JiangHaiting, Jiaqi Shen, Jiwei Guo, Kai, Kai Wang, Kengo Seki, Lan, Lari Hotari, Lei Zhiyuan, Leo fu, Leona Yoda, Li Li, Lin Zhao, LinChen, Lishen Yao, Mark Silcox, Marvin Cai, Masahiro Sakamoto, Mathieu Grenonville, Matteo Merli, Max Xu, Melker Narikka, Mendon Kissling, Mercurio, Michael Marshall, Naresh, Neng Lu, Nicolò Boschi, Nihar Rathod, Paul Gier, Penghui Li, Pradhyumnakashyap, Qiang Huang, Qiang Zhao, Rajan Dhabalia, Robert Barbey, Ruguo Yu, Rui Fu, SeasonPan, Shen Liu, Sherlock Xu, StevenLuMT, Tao Jiuming, Tuấn Vương, WJL3333, WangJialing, Xiangying Meng, Xiaoyu Hou, YANGLiiN, Yan Zhao, Yang Yang, YingQun Zhong, Yong Zhang, Yuanhao Ji, Yunze Xu, Yuri Mizushima, Zac Bentley, ZhangJian He, Zhangao, Zike Yang, Zixuan Liu, caption, codertmy, coderzc, congbo, crossoverJie, fengyubiao, feynmanlin, gaozhangmin, grayson, guan46, houxiaoyu, hrsakai, hzh0425, jiangpengcheng, ken, komalatammal, labuladong, laminar, laminar (Tian Fang), leixin, lifepuzzlefun, lixinyang, maheshnikam, mans2singh, mipper, momo-jun, nullscript, penghui, presenth7s, ran, sinan liu, thetumbled, tison, vineeth1995, wenbingshen, xiaokang, xiaolong ran, yangyijun, yingchang, youzipi, yws-tracy, zhaohaidao, zzccctv and 萧易客.

Was this helpful?
💡 Suggest changes🛟 Get support