Skip to main content

Apache Pulsar 2.11.1

2023-04-19

Broker

  • AbstractBatchedMetadataStore - use AlreadyClosedException instead of IllegalStateException #19284
  • Add ref count for sticky hash to optimize the performance of Key_Shared subscription #19167
  • Pass subName for subscription operations in ServerCnx #19184
  • Enable custom metadata stores #19208
  • Topics stats shows msgBacklog but there reality no backlog #19275
  • Fix open cursor with null-initialPosition result with earliest position #18416
  • Added isActive in ManagedCursorImpl #19341
  • Replaced checkBackloggedCursors with checkBackloggedCursor(single subscription check) upon subscription #19343
  • Close TransactionBuffer when MessageDeduplication#checkStatus failed #19289
  • Fix the incorrect total size if use ML interceptor #19404
  • Support deleting partitioned topics with the keyword -partition- #19230
  • Correct the prompt message #17009
  • Expect msgs after server initiated CloseProducer #19446
  • Fix Npe thrown by splitBundle #17370
  • Fix executeWithRetry result is null #17694
  • ManagedLedger: move to FENCED state in case of BadVersionException #17736
  • Fix can not revoke permission after update topic partition #17393
  • Allow deletion of a namespace that was left in deleted status #17592
  • Add UncaughtExceptionHandler for every thread pool #18211
  • Support setting ForceDeleteNamespaceAllowed dynamically #18181
  • Support setting forceDeleteTenantAllowed dynamically #18192
  • Remove locallyAcquiredLock when removeOwnership #18197
  • Update the log print content of createSubscriptions #18024
  • Unify time unit at dropping the backlog on a topic #17957
  • Fix mutex never released when trimming #17911
  • Fix getChildren in MemoryMetadataStore and EtcdMetadataStore #18172
  • Fix the order of resource close in the InMemoryDelayedDeliveryTracker #18000
  • Allow to configure and disable the size of lookahead for detecting fixed delays in messages #17907
  • Make BookieId work with PulsarRegistrationDriver #17922
  • In the trimDeletedEntries method, release the removed entry #18305
  • Make Consumer#equals more effective #18662
  • PulsarRegistrationClient - implement getAllBookies and follow BookieServiceInfo updates #18133
  • Fix PulsarRegistrationClient and ZkRegistrationClient not aware rack info problem #18672
  • Simplify extract entryMetadata code in filterEntriesForConsumer #18729
  • Fix uncompleted future when getting the topic policies of a deleted topic #18824
  • Close transactionBuffer after MessageDeduplication#checkStatus failed #19157
  • GetLastMessageId returns a wrong batch index of last message if enabled read compacted #18877
  • Fix multi invocation for ledger createComplete #18975
  • Fix txn coordinator recover handle committing and aborting txn race condition #19201
  • Reset individualDeletedMessagesSerializedSize after acked all messages #19428
  • Fix potential NPE cause future never complete #19415
  • Fix the return value of canConsumeAsync #19412
  • Close TransactionBuffer when creating persistent topic timeout #19384
  • Handle changeToReadyState failure correctly in TC client #19308
  • The atomicity of multiple fields of ml is broken #19346
  • MessagesConsumedCounter of NonDurableCursor was initialized incorrectly #19355
  • Improve PersistentMessageExpiryMonitor expire speed when ledger not existed #17842
  • Always send correct transaction id in end txn response #19137
  • Allow superusers to abort transactions #19467
  • Make ServerCnx#originalAuthData volatile #19507
  • Validate originalPrincipal earlier in ServerCnx #19270
  • Require authRole is proxyRole to set originalPrincipal #19455
  • ServerCnx broken after recent cherry-picks #19521
  • ServerCnx: go to Failed state when auth fails #19312
  • Add test to verify authRole cannot change #19430
  • Make authentication refresh threadsafe #19506
  • OneStageAuth State: move authn out of constructor #19295
  • Allow proxy to pass same role for authRole and originalRole #19557
  • Copy command fields and fix potential thread-safety in ServerCnx #19517
  • Topic load fail by ledger lost #19444
  • Incorrect service name selection logic #19505
  • Make the service name resolver cache of PulsarWebResource expire after access #19532
  • Simplify enabling Broker, WS Proxy hostname verification #19674
  • Improve tls config on replication client and cluster cli #19588
  • Fix geo-replication admin #19686
  • Filter system topic when getting topic list by binary proto #19667
  • Fixed history load not releasing #19560
  • Fix loadbalance score calculation problem #19420
  • Use shrink map for trackerCache #19534
  • Fix individual ack batch message with transaction abort redevlier duplicate messages #14327
  • Fix receive duplicated messages due to pendingAcks in PendingAckHandle #19581
  • Fail to update partition meta of topic due to ConflictException: subscription already exists for topic #17251
  • Fix catching ConflictException when update topic partition #17374
  • Only create extended partitions when updating partition number #17349
  • Create replicated subscriptions for new partitions when needed #18659
  • Fix race condition while updating partition number #19199
  • Topic close failure leaves subscription in a permanent fence state #19692
  • Fix potential exception cause the policy service init fail #19746
  • Allow version to start positive and grow by more than one #19503
  • Fix deadlock causes session notification not to work #19754
  • Fix issue where msgRateExpired may not refresh forever #19759
  • Delete topic failed if disabled system topic #19735
  • Fix index generator is not rollback after entries are failed added #19727
  • Store the original authentication data #19519
  • Copy subscription properties during updating the topic partition number #19800
  • Fix close borrowed executor #19761
  • Fixed race condition between ResourceLock update and invalidation #19817
  • Counter of pending send messages in Replicator incorrect if schema future not complete #19242
  • Add getState in transaction for client API #19834
  • Follow up #19230 to tighten the validation scope #19234
  • Authorize originalPrincipal when provided #19830
  • Follow up #19817, Fix race condition between ResourceLock update and invalidation #19859
  • Fix delete system topic clean topic policy #19835
  • Fix NPE when update topic policy #19875
  • Make the result of delete cursor is success if cursor is deleted #19825
  • Only validate superuser access if authz enabled #19989
  • Cleanup how superusers abort txns #19976
  • Return if AbstractDispatcherSingleActiveConsumer closed #19934
  • Use AuthenticationParameters for rest producer #20046
  • Fix uncompleted future when removing cursor #20050
  • Fix estimateBacklogFromPosition if position is greater than the greatest ledgerId #20069

Clients

  • [Java] Fix reader listener can't auto ack with pooled message #19354
  • [Java] Prevent DNS reverse lookup when physical address is an IP address #19028
  • [Java] Set fields earlier for correct ClientCnx initialization #19327
  • [Java] Improve docs and code quality about KeyValueSchema usages #17256
  • [Java] Add init capacity for messages in BatchMessageContainerImpl #17822
  • [Java] Support MAX_ACK_GROUP_SIZE configurable #18107
  • [Java] Fix exception when calling loadConf on a ConsumerBuilder that has a KeySharedPolicy #18345
  • [Java] Support LocalDateTime Conversion #18334
  • [Java] Fix pendingLookupRequestSemaphore leak #18219
  • [Java] Unify the acknowledge process for batch and non-batch message IDs #17833
  • [Java] Set authentication when using loadConf in client and admin client #18358
  • [Java] Fixes batch_size not checked in MessageId#fromByteArrayWithTopic #18405
  • [Java] Fix possible npe #18406
  • [Java] Change the get lastMessageId to debug level #18421
  • [Java] Fix failover/exclusive consumer with batch cumulate ack issue #18454
  • [Java] Avoid redelivering duplicated messages when batching is enabled #18486
  • [Java] For exclusive subscriptions, if two consumers are created repeatedly, the second consumer will block #18633
  • [Java] Fix the Windows absolute path not recognized in auth param string #18403
  • [Java] Broker address resolution wrong if connect through a multi-dns names proxy #19597
  • [Java] Fix memory leak if enabled pooled messages #19585
  • [Java] Fix authentication not update after changing the serviceUrl #19510
  • [Java] Fix async completion in ConsumerImpl#processPossibleToDLQ #19392
  • [Java] Moving get sequenceId into the sync code segment #19837
  • [Java] Fix DeadLetterProducer creation callback blocking client io thread #19930

Pulsar IO and Pulsar Functions

  • [Functions] Ensure prometheus metrics are grouped by type #15558
  • [Functions] Fix k8s merge runtime opts bug #19481
  • [Functions] Use AuthorizationServer more in Function Worker API #19975
  • [IO Connector] Debezium sources: Support loading config from secrets #19205
  • [IO Connector] Elasticsearch sink: Support loading config from secrets #19206
  • [IO Connector] Update Elasticsearch sink idle cnx timeout to 30s #19377
  • [IO Connector] DBC sink: allow any jdbc driver #17951
  • [IO Connector] ElasticSearch sink: align null fields behaviour #18577
  • [IO Connector] KCA sink: handle null values with KeyValue<Avro,Avro> schema #19861
  • [IO Connector] KCA: option to collapse partitioned topics #19923
  • [IO Connector] KCA: flag to force optional primitive schemas #19951

Observability

  • [Broker] Fix wrong metrics text generated when label_cluster specified #17704
  • [Broker] Add metrics for transaction #15140
  • [Broker] Remove timestamp from Prometheus metrics #17419
  • [Broker] Fix incorrect bundle split count metric #17970
  • [Broker] Update authentication failed metrics report #17787
  • [Function] Fix invalid metric type "gauge " #18129

CLI

  • [Pulsar Admin] Returns 4xx error when pulsar-worker-service is disabled and trying to access it #17901
  • [Pulsar Admin] PulsarAdminBuilderImpl overrides timeout properties passed through config map #17375
  • [Pulsar Admin] Improve partitioned-topic condition evaluation #19015
  • [Pulsar Admin] Make the default value of param --get-subscription-backlog-size of admin API topics stats true #19302
  • [Pulsar Admin] Make response code to 400 instead of 500 when deleting topic fails due to enabled geo-replication #19879
  • [Pulsar Admin] Fix delete tenant #19925
  • [Pulsar Admin] Fix incorrect state replication.connected on API partitioned-topic stat #19942
  • [Pulsar CLI] Quit PerformanceConsumer after receiving numMessages messages #17750
  • [Pulsar CLI] Check numMessages after incrementing counter #17826
  • [Pulsar CLI] Fix Pulsar admin tool is ignoring tls-trust-cert path arg #19696
  • [Pulsar CLI] Fix Broker crashed by too much memory usage of pulsar tools #20031

Others

  • [Proxy] Only go to connecting state once #19331
  • [Proxy] Fix using wrong client version in pulsar proxy #19569
  • [Proxy] Implement graceful shutdown for Pulsar Proxy #20011
  • [Proxy] Fix connection read timeout handling in Pulsar Proxy #20014
  • [WebSocket] Add ping support #19255
  • [Schema] Do not print error log with stacktrace for 404 #19130
  • [Tiered Storage] Don't cleanup data when offload met Metastore exception #17512
  • Fix transitive critical CVEs in file-system tiered storage #19957

Library updates

  • Bump snakeyaml to 1.32 for CVE-2022-38752 #17779
  • Upgrade jettison to 1.5.3 #19038
  • Upgrade woodstox to 5.4.0 #19041
  • Upgrade lombok to 1.8.26 #19426
  • Upgrade to zk 3.8.1 #19425
  • Upgrade bookkeeper to 4.15.4 #19812