Apache Pulsar 2.11.0

2023-01-11

Important notice

Starting from 2.11.0, the minimal version of JDK on the server side (broker, functions, connectors) is required to be 17. See Pulsar Runtime Java Version Recommendation and PIP-156 for more details.

• Rename Pulsar txn metrics to specify OpenMetrics #16581
• Rename Pulsar schema metrics to specify OpenMetrics #16610
• Rename Pulsar lb metrics to specify OpenMetrics #16611
• Remove jul-to-slf4j #16320
• Fix get non-persistent topics issue in Namespaces #16170
• Fix producer/consume permission can’t get schema #15956
• Add verification for configured default backlog quota and retention #15441
• Fix ineffective broker LoadBalance #15314
• Ensure the deletion consistency of topic and schema #14608
• Transaction cumulative ack redeliver change #14371
• Bump prometheus client version from 0.5.0 to 0.15.0 #13785
• [PIP-158] Split client TLS transport encryption from authentication #15634
• [PIP-155] Remove Python 2 support #15376
• [PIP-156] Build Pulsar Server on Java 17 #15264
• [PIP-162] Enable system topic by default #15619

PIPs

• [PIP 37] Support chunking with Shared subscription #16202
• [PIP-74] Pulsar client memory limits #14400
• [PIP-81] Split createNewMetadataLedger into multiple methods for reuse #15425
• [PIP 97] Update Authentication Interfaces to Include Async Authentication Methods #12104
• [PIP-100] Support pluggable topic factory #12235
• [PIP-105] Store Subscription properties #15757
• [PIP-117] Change Pulsar standalone defaults #15478
• [PIP-132] Include message header size when checking maxMessageSize for non-batch messages on the client side #14007
• [PIP-136] Sync Pulsar metadata across multiple clouds #16425
• [PIP-143] Support split bundle by specified boundaries #13796
• [PIP-145] Improve performance of regex subscriptions #14505
• [PIP-146] ManagedCursorInfo compression #14542
• [PIP 149] Make the REST Admin API fully async #14365
• [PIP-152] Support subscription level dispatch rate limiter setting #15295
• [PIP-153] Optimize metadataPositions in MLPendingAckStore #15137
• [PIP-160] Make transactions work more efficiently by aggregation operation for transaction log and pending ack store #15370
• [PIP-161] Exclusive Producer: ability to fence out an existing Producer #15488
• [PIP-163] Add lowWaterMark check before appending entry to TB #15424
• [PIP-166] Add MANUAL delivery semantics for Functions #16279
• [PIP-179] Support the admin API to check unknown request parameters #16135
• [PIP-181] Pulsar Shell #16250
• [PIP-184] Topic specific consumer priorityLevel #16715
• [PIP-187] Add API to analyze a subscription backlog and provide an accurate value #16545
• [PIP-189] No batching if only one message in batch #18548
• [PIP-201] Extensions mechanism for Pulsar Admin CLI tools #17158

Broker

• [PIP 81] Split the individual acknowledgments into multiple entries

  • Add range set wrapper to support record dirty ledgers #15607
  • Notifications for faster topic discovery #16062
  • Split createNewMetadataLedger into multiple methods for reuse #15425

• [PIP 105] Store Subscription properties

  • Store Subscription properties #15757
  • Pulsar Admin: create subscription with Properties #15503
  • Fix error on recycled SubscriptionPropertiesList #15335

• [PIP 160] Make transactions work more efficiently by aggregation operation for transaction log and pending ack store

  • Pending ack log store enables the batch feature #16707
  • Metrics stats of Transaction buffered writer #16758
  • Transaction buffered writer supports Timer #16727
  • Transaction log store enables the batch feature #16685
  • Protocol changes and configuration changes for transaction batch log #16617
  • Txn buffered writer for transaction log batch #16428

• [PIP 145] Improve performance of regex subscriptions

  • Enable evaluating subscription pattern on broker side #14804
  • Notifications for faster topic discovery #16062

• [PIP-156] Build Pulsar Server on Java 17

  • Build changes by dropping some required checks #15496
  • Build Pulsar Server on Java 17 #15264

• Add logs for why namespace bundle been split #19003

• Omit making a copy of CommandAck when there are no broker interceptors #18997

• Fix deadlock in PendingAckHandleImpl #18989

• Copy proto command fields into final variables in ServerCnx #18987

• Transaction pending ack store future not completely problem #18943

• Update interceptor handler exception #18940

• Do not require encryption on system topics #18898

• Fix incorrect Nic usage collected by pulsar #18882

• Fix PendingAckHandleImpl when pendingAckStoreProvider.checkInitializedBefore failed #18859

• Ignore the exception of creating namespace #18837

• Fixing can not delete namespace by force #18826

• Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer #18747

• System topic writer/reader connection not counted #18603

• DnsResolverUtil.TTL should be greater than zero #18565

• Persist correct markDeletePosition to prevent message loss #18237

• Fix delete_when_subscriptions_caught_up doesn't work while have active consumers #18320

• Read local cookie when start pulsar standalone #18260

• Remove the redundant judgment logic of ManagedCursorImpl #18205

• Fix unexpected behaviour by invoke PulsarConfigurationLoader#convertFrom #18816

• Fix the wrong behaviour when set overrideBrokerNicSpeedGbps #18818

• Fix the breaking change of standalone metadata initialization #18909

• Transaction buffer recover blocked by readNext #18969

• Fix duplicated schemas creation #18701

• Read local cookie when start pulsar standalone #18260

• The Pulsar standalone bookie is not getting passed the config from standalone.conf #18126

• Reduce unnecessary calling span() when filtering read entries #18106

• Change update schema auth from tenant to produce #18074

• Fixed delayed delivery after read operation error #18098

• Fix memory leak while Offloading ledgers #18500

• Cache invalidation due to concurrent access #18076

• Fix unable to start multiple bookies for BKCluster #18072

• Compare batch index when accumulating acks and updating batchDeletedIndexes #18042

• Limit the memory used by reads end-to-end #18245

• Consumer backlog eviction policy should not reset read position for consumer #18037

• Allow to configure and disable the size of lookahead for detecting fixed delays in messages #17907

• Fix create ns #17864

• Fix the wrong NIC speed rate unit #17890

• Fix numerical overflow bug while reading data from tiered storage #18595

• Support lower boundary shedding for ThresholdShedder #17456

• Avoid unnecessary creation of BitSetRecyclable objects #17998

• Fix broker cache eviction of entries read by active cursors #17273

• Fix issue where leader broker information isn't available after 10 minutes #17401

• Fix NPE when ResourceGroupService execute scheduled task #17840

• Make BookieId work with PulsarRegistrationDriver #17762

• Fix namespace backlog quota check with retention #17706

• Cleanup state when lock revalidation gets LockBusyException #17700

• Fix parsing partitionedKey with Base64 encode issue #17687

• Standalone Add param of --metadata-url for runing with metadata #17077

• Correctly set byte and message out totals per subscription #18451

• Implementation of Delayed Transaction Messages #17548

• Set revalidateAfterReconnection true for certain failures #17664

• Unregister topic policy listener if managed ledger close failed #17652

• Correctly handle list configuration values #17661

• Cancel the loadShedding task when closing pulsar service #17632

• Consumer can't consume messages because there has two sames topics in one broker #17526

• Prevent partitioned metadata lookup request when broker is closing #17315

• Prevent new connection request when broker is closing #17314

• Ledger handle leak when update schema #17283

• Add metrics for entry cache insertion, eviction #17248

• Do not send duplicate reads to BK/offloaders #17241

• Fix system service namespace create internal event topic #17867

• Fix getPositionAfterN infinite loop #17971

• Fix memory leak in case of error conditions in PendingReadsManager #17995

• After the broker is restarted, the cache dynamic configuration is invalid #17035

• Fix if dynamicConfig item in ZK do not exist in broker cause NPE #17705

• Fix the broker shutdown issue after Zookeeper node crashed #17909

• Fix broker irrational behavior when it is closing #17085

• ServerCnx: log at warning level when topic not found #16225

• Optimize getting ledger and entry id from entry #17108

• Topic policy reader can't recover when get any exception #17562

• Multiple consumer dispatcher stuck when unackedMessages greater than maxUnackedMessages #17483

• Fix memoryLimitController currentUsage and MaxQueueSize semaphore leak when batchMessageContainer add message exception #17276

• Fix arithmetic exception for LeastResourceUsageWithWeight strategy #17149

• Fix update topic remove properties #17231

• Fix ack with txn compute ackedCount error #17016

• Improve cursor.getNumberOfEntries if isUnackedRangesOpenCacheSetEnabled=true #17465

• Fix dispatch duplicated messages with Exclusive mode #17237

• Remove internalUpdateOffloadPolicies to keep the same behavior update topic policy #17236

• Fix update topic remove properties #17231

• Fix arithmetic exception for LeastResourceUsageWithWeight strategy #17149

• Fix pulsarLedgerIdGenerator can't delete index path when zk metadata store config rootPath #17192

• Avoid messages being repeatedly replayed with SHARED subscriptions (streaming dispatcher) #17163

• Add ServerCnx state check before server handle request #17084

• Pass subscriptionName to auth service #17123

• Support loadBalancerSheddingIntervalMinutes dynamic configuration #16408

• Fix out of order data replication #17154

• Fix schema does not replicate successfully #17049

• Optimize the shutdown sequence of broker service when it close #16756

• Modernizer managed ledger #16363

• Make deleteTopicPolicies serialized is executed when close topic #15811

• Streaming dispatcher stuck after reading the first entry with SHARED subscriptions #17143

• Fix calculate avg message per entry #17046

• Fix bundle-data metadata leak because bundle stats was not cleaned up #17095

• Duplicate ByteBuffer when Caching Backlogged Consumers #17105

• Fix offload read handle NPE #17056

• Increment topic stats outbound message counters and update rate after messages have been written to the TCP/IP connection #17043

• Move the state check forward #17020

• Follow up on #16968 to restore some behavior in PersistentDispatcherMultipleConsumers #17018

• Remove exception log when access status.html #17025

• Not allow terminating system topic #17006

• Fix memory leak if entry exists in cache #16996

• Remove unnecessary lock on the stats thread #16983

• Prevent StackOverFlowException in SHARED subscription #16968

• Improve naming for delete topic error #16965

• Fix ConcurrentModificationException when ModularLoadManagerImpl start #16953

• Skip mis-configured resource usage(>100%) in load balancer #16937

• Adapt basic authentication configuration with prefix #16935

• Change delete pending ack position from foreach to firstKey #16927

• Fix MaxQueueSize semaphore release leak in createOpSendMsg #16915

• PulsarLedgerManager to pass correct error code to BK client #16857

• Support start multiple bookies for BKCluster #16847

• Split TLS transport encryption support from authentication #16819

• Fix misleading -c option in pulsar standalone #16838

• Fix rack awareness cache expiration data race #16825

• Fix Repeated messages of shared dispatcher #16812

• Avoid IllegalStateException while client_version is not set #16788

• Optimize concurrent collection's shrink logic #16754

• Fix consumer does not abide by the max unacks limitation for Key_Shared subscription #16718

• Support clear old bookie data for BKCluster #16744

• Avoid ConcurrentModificationException for ModularLoadManagerImpl.cleanupDeadBrokersData() #16690

• Improve get the basic authentication config #16526

• Retry to delete the namespace if new topics created during the namespace deletion #16676

• Fix consumer does not abide by the max unacks limitation for Shared subscription #16670

• Add decode InputStream for Schema #16659

• Support for get token from HTTP params #16650

• Make defaultOffloader create after offloaderStats overrides #16638

• BadVersionException when splitting bundles, delay 100ms and try again #16612

• The configuration loadBalancerNamespaceMaximumBundles is invalid #16552

• PulsarLedgerManager: add missed return statement #16607

• Retry when DistributedIdGenerator has BadVersion error #16491

• Fixed error when delayed messages trackers state grows to >1.5GB #16490

• Fix RawReader hasMessageAvailable returns true when no messages #16443

• Fix No such ledger exception #16420

• Improve the package download process #16365

• Add config maxUnloadBundleNumPerShedding for UniformLoadShedder #16409

• Skip reading more entries for a pending read with no more entries #16400

• Recycle OpReadEntry in some corner cases #16399

• Add dynamic configuration for UniformLoadShedder #16391

• Fix RawReader out of order #16390

• Create the cursor ledger lazily to improve the subscribe performance #16389

• Release the entry in getEarliestMessagePublishTimeOfPos #16386

• Update CPU ResourceUsage before updating SystemResourceUsage usage #16366

• Use shared broker client scheduled executor provider #16338

• Fix etcd cluster error and add test for etcd cluster #16309

• Do not use IO thread for consumerFlow in Shared subscription #16304

• Provide new load balance placement strategy implementation based on the least resource usage with weight #16281

• Fix TopicTransactionBuffer ledger apend marker throw ManagedLedgerAlreadyClosedException #16265

• Avoid go through all the consumers to get the message ack owner #16245

• Reduce the consumers list sort by priority level #16243

• Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers #16241

• Fix NPE when invoke replaceBookie. #16239

• Fix getInternalStats occasional lack of LeaderInfo again #16238

• Fix NPE when drop backlog for time limit #16235

• Improve error msg when client is unauthorized #16224

• Fix compaction subscription acknowledge Marker msg issue #16205

• Fix subscribe dispatcher limiter not be initialized #16175

• Using handle instead of handleAsync to avoid using common pool thread #17403

• Use LinkedHashSet for config items of type Set to preserve elements order #16138

• Fix topic dispatch rate limiter not init on broker-level #16084

• Fix NPE when get /admin/v2/namespaces/public/default/maxTopicsPerNamespace #16076

• Add config to allow deliverAt time to be strictly honored #16068

• Add switch for enable/disable distribute bundles evenly in LoadManager #16059

• Fix thread safety issues in accessing ManagedCursorContainer.heap ArrayList #16049

• Make invalid namespace and topic name logs more descriptive #16047

• Terminate JVM when initialize-cluster-metadata command fails #16039

• Avoid storing MessageMetadata instances returned by peekMessageMetadata #15983

• Fix topic policies update not check message expiry #15941

• Fix reach max tenants error if the tenant already exists #15932

• Close transactionBufferClient before closing the internal Pulsar client #15922

• Remove topic -> namespace cache of LedgerOffloaderStatsImpl #15869

• Fix NPE when ledger id not found in OpReadEntry #15837

• Fix topic-level replicator rate limiter not init #15825

• Fix NPE in MessageDeduplication #15820

• Configure DLog Bookie, Pulsar, and Admin clients via pass through config #15818

• Fix the broker close hanged issue #15755

• Disable memory limit controller in internal Pulsar clients #15752

• Add timeout for unload namespace bundle #15719

• Support schemaValidationEnforced on topic level #15712

• Fix NPE when put value to RangeCache #15707

• Fast return if ack cumulative illegal #15695

• Fix creating producer failure when set backlog quota #15663

• Fix logging in shutdown when broker shutdown #15626

• Fix MetadataStoreException$NotFoundException while doing topic lookup #15633

• Use dispatchRateLimiterLock to update dispatchRateLimiter #15601

• Add .yaml suffix support for broker filter config file #15600

• Add newline to the end of standalone.conf #15591

• Create "standalone" cluster when starting standalone server #15583

• Close publishLimiter when disable it #15520

• Enable SO_REUSEADDR on listen sockets for broker,proxy #15517

• Fix to avoid TopicStatsImpl NPE even if producerName is null #15502

• Close connection if a ping or pong message cannot be sent #15382

• Introduce BitRateUnit for calculate bitrate #15435

• Add verification for configured default backlog quota and retention #15441

• Change unnecessary atomic to basic type #15369

• Support shrink for ConcurrentSortedLongPairSet #15354

• Use shrink map for message redelivery #15342

• Fix precision error in UsageUnit #15322

• Optimized namespace-level dispatcherRateLimiter update #15315

• Fix wrong unit of NIC speed on Linux #15304

• Fix totalEntries calculation problem in AbstractBaseDispatcher#filterEntriesForConsumere #15298

• Fix resource group does not report usage #15292

• Tidy up the system topic #15252

• Fix race condition between timeout and completion in OpAddEntry #15233

• Improve skipping of DNS resolution when creating AuthenticationDataHttp instance #15228

• Skip unnecessary DNS resolution when creating AuthenticationDataHttp instance #15221

• Fix MessageDeduplication#inactiveProducers may not be persistence correctly #15206

• Cancel fencedTopicMonitoringTask when topic closed normally #15202

• Fix npe when doCacheEviction #15184

• Make health check fail if dead locked threads are detected #15155

• Fix parameter saslJaasBrokerSectionName in broker.conf #15110

• Fix potential to add duplicated consumer #15051

• Fix rewind failed when redeliverUnacknowledgedMessages #15046

• Fix race condition in updating lastMarkDeleteEntry field #15031

• Avoid heartbeat topic to offload #15008

• Evict cache data according to the slowest markDeletedPosition #14985

• Return if reset in progress #14978

• Dynamic update broker-level subscribe-rate limter #14890

• Fix wrong state for non-durable cursor #14869

• Support advertised listeners for HTTP and HTTPS services #14839

• Support dynamic update between non-zero values of topicPublisherThrottlingTickTimeMillis #14782

• Support dynamic update between non-zero values of brokerPublisherThrottlingTickTimeMillis #14759

• Fix incorrect entryId in warn log when reading entry from tiered storage #14685

• Optimize PositionImpl toString, compareTo and hashCode methods #14678

• Improve readability of ManagedCursorImpl source code #14617

• Intercept ACK command by BrokerInterceptor #14616

• Add filteredMsgCount for pulsar-admin broker-stats topics #14531

• Fix NPE when subscription is already removed #14363

• Fix backpressure handling in Jetty web server configuration #14353

• Optimize find nics process #14340

• Support pass http auth status #14044

• Support caching to drain backlog consumers #12258

• Strict bookie affinity group strategy #12025

Clients

• [PIP 74] Pulsar client memory limits
  • Support auto scaled consumer receiver queue #14494
  • Support dynamic limit of consumer receiver queue #14400
  • Support consumer client memory limit #15216
• [Java] Fix multi-topic consumer stuck after redeliver messages #18491
• [Java] Fix NPE of MultiTopicsConsumerImpl due to race condition #18287
• [Java] Refactor SchemaHash to reduce call of hashFunction in SchemaHash #17948
• [Java] Fix pendingLookupRequestSemaphore leak when channel inactive #17856
• [Java] Unwrap completion exception for Lookup Services #17717
• [Java] Fixed ack failure in ReaderImpl due to null messageId #17728
• [Java] Fix scheduledExecutorProvider not shutdown #17527
• [Java] LastBatchSendNanoTime initialization #17058
• [Java] Fix inactive cnx channel causing the request to fail to time out and fail to return #17051
• [Java] Release semaphore before discarding messages in batchMessageContainer #17019
• [Java] Fix client memory limit currentUsage leak and semaphore release duplicated in ProducerImpl #16837
• [Java] Remove redundant check for chunked message TotalChunkMsgSize in ConsumerImpl #16797
• [Java] Reduce code duplication in admin client #16377
• [Java] Release memory usage for invalid messages #16835
• [Java] Fix subscription topic name error #16719
• [Java] Send CloseConsumer on client timeout #16616
• [Java] Add message key if exists to deadLetter messages #16615
• [Java] Make DeadLetterPolicy deserializable #16513
• [Java] Improve performance of multi-topic consumer with more than one IO thread #16336
• [Java] Add initialization for the OpSendMsg #16256
• [Java] Replace ScheduledExecutor to improve performance of message consumption #16236
• [Java] Fix large messages sometimes cannot be split into chunks #16196
• [Java] Only trigger the batch receive timeout when having pending batch receives requests #16160
• [Java] Fix the startMessageId can't be respected as the ChunkMessageID #16154
• [Java] Fix auto cluster failover can't resolve host bug #16152
• [Java] Switch to rely on Netty for Hostname Verification #15824
• [Java] Prevent the trigger from running concurrently in the memory limit controller #15737
• [Java] Fix messages sent by producers without schema cannot be decoded #15622
• [Java] Remove sensitive msg from consumer/producer stats log #15483
• [Java] Add pending messages information while printing the producer stats #15440
• [Java] Fix flaky BatchMessageTest by initializing lastBatchSendNanoTime #15406
• [Java] Check consumer schema null in advance #15327
• [Java] Fix inconsistent parameter of TopicPolicies.getSubscriptionDispatchRate #15293
• [Java] Fix typo in ConsumerBuilder #15194
• [Java] Fix performance regression with message listener #15162
• [Java] ConsumerBuilderImpl can not set null to deadLetterPolicy #14980
• [Java] Fix message publishing stuck when enabling batch #14870
• [Java] Return immutable data set when using TableView #14833
• [Java] Avoid timer task run before previous subscribe complete #14818
• [Java] Support Reader Interceptor #14729
• [Java] Support aggregate metrics for partition topic stats #18214
• [Java] Add api to get producer/consumer stats for partition topic #18212
• [Java] Optimize pause when creating sub consumers in multi-topic consumer #14566

Pulsar IO and Pulsar Functions

• [Functions] Fix function failed to start if no typeClassName provided in FunctionDetails #18111
• [Functions] Do not delete managed package when delete function #18030
• [Functions] Fix the download of builtin Functions #17877
• [Functions] Ensure InternalConfigurationData data model is compatible across different versions #17690
• [Functions] Use the schema set by the Function when it returns a Record #17142
• [Functions] Make mandatory to provide a schema in Context::newOutputRecordBuilder #17118
• [Functions] Add the ability to customize logging level for Go & Python functions #16939
• [Functions] Fixed error when user starts with the pulsar functions local runner #16565
• [Functions] Fix netty.DnsResolverUtil compat issue on JDK9+ for the function Runtimes #16423
• [Functions] Ensure bytes are a well-formed UTF-8 byte sequence when decoding the FunctionState bytes to string #16199
• [Functions] Support Record<?> as Function output type #16041
• [Functions] Configure DLog Bookie, Pulsar, and Admin clients via pass through config #15818
• [Functions] Disable memory limit controller in internal Pulsar clients #15752
• [Functions] Provide default error handler for function log appender #15728
• [Functions] Support disabling non-TLS service port #15328
• [Functions] Check executor null when close the FileSource #15247
• [Functions] Fix UserConfigFunction example #15240
• [Functions] Add a cache of versioned KeyValueSchemaImpl #15122
• [Functions] Add KeyStore support in WebSocket, Function Worker HTTPS Servers #15084
• [Functions] Handle NPE when getLeader returns null #15058
• [Functions] Remove internal dependency: pulsar-functions-instance #14925
• [Functions] Allow a Function<GenericObject,?> to access the original Schema of the Message and use it #14847
• [Functions] Fix pulsar-managed runtimes failed start function with package URL from package management service #14814
• [Functions] Set function channel to idle to prevent DNS resolution of deleted pod #14750
• [Functions] Remove extra call to gi.stats.incrTotalProcessedSuccessfully() #12316
• [Functions] Fix python instance not process zip file correctly #16697
• [Functions] Missing assertion in KubernetesRuntimeTest::verifyJavaInstance #16747
• [IO Connector] Only bundle kafka schema registry client #18931
• [IO Connector] Fix message without schema issue #18745
• [IO Connector] Kinesis sink: fix NPE with KeyValue schema and no value #17959
• [IO Connector] Provide a HTTP Sink #17581
• [IO Connector] IOConfigUtils support required and defaultValue annotations #16785
• [IO Connector] Fixed Kafka source config for pulsar-io-kafka when consumerConfigProperties is null #16731
• [IO Connector] Add reader config to pulsar-io-debezium and pulsar-io-kafka-connect-adaptor #16675
• [IO Connector] Support transactions for JDBC connector #16468
• [IO Connector] JDBC sinks: support upsert and row deletion #16448
• [IO Connector] Add getSourceConfig method on SourceContext #16305
• [IO Connector] KCA sinks: fix offset mapping when sanitizeTopicName=true #15950
• [IO Connector] Allow pulsar_tool_env.sh PULSAR_MEM to be Overridden #15868
• [IO Connector] JDBC sinks: support Avro specific datatypes #15845
• [IO Connector] Sink support custom acknowledge type #15491
• [IO Connector] Add getSinkConfig method to SinkContext #15482
• [IO Connector] Handle Avro collections native types (GenericData.Array and Utf8 map keys) #15432
• [IO Connector] Handle Avro collections native types (GenericData.Array and Utf8 map keys) #15430
• [IO Connector] Add hashed id support #15428
• [IO Connector] ElasticSearch Sink: option to output canonical key fields (JSON and Avro) #15426
• [IO Connector] Replaced "Device" with "measurement" #15416
• [IO Connector] Enable bulk flushing scheduling by default #15244
• [IO Connector] Set sinkType in config file #15174
• [IO Connector] Add a cache of versioned KeyValueSchemaImpl #15122
• [IO Connector] Support 4paradigm/OpenMLDB jdbc sink connector #15064
• [IO Connector] ElasticSearch Sink: option to disable SSL certificate validation #14997
• [IO Connector] Add FULL_MESSAGE_IN_JSON_EXPAND_VALUE message format to Kinesis sink #14929

Observability

• [Broker] Fix metrics string encoding #18138
• [Broker] Fix EntryFilter stats #17605
• [Broker] Add per-subscription EntryFilter metrics #16932
• [Broker] Expose topic level storage write and read rate metrics #16855
• [Broker] Rename Pulsar lb metrics to specify OpenMetrics #16611
• [Broker] Rename Pulsar schema metrics to specify OpenMetrics #16610
• [Broker] Rename Pulsar txn metrics to specify OpenMetrics #16581
• [Broker] Optimise msgOutCounter and bytesOutCounter #16214
• [Broker] Fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true #16219
• [Broker] Add metrics for pulsar web service thread pool #14742
• [Broker] Add support of PrometheusRawMetricsProvider for the Pulsar-Proxy #14681
• [Broker] Add filteredMsgCount for pulsar-admin broker-stats topics #14531
• [Broker] Improve /metrics endpoint performance #14453
• [Broker] Offloader metrics #13833
• [Broker] Improve /metrics endpoint performance #14453
• [Broker] Add metrics for InMemoryDelayedDeliveryTracker's memory usage #15867

CLI

• [Pulsar Admin] Fix NPE when get OffloadThreshold on namespace #18061
• [Pulsar Admin] Add SNI header when tlsHostnameVerification is not enabled #17543
• [Pulsar Admin] Unset namespace policy to improve deleting namespace #17033
• [Pulsar Admin] Reduce PartitionedStats local REST call #16916
• [Pulsar Admin] Make some police methods async in Namespaces #16881
• [Pulsar Admin] Expose the last consumed flow timestamp for consumer stats #16817
• [Pulsar Admin] Make some methods async in Namespaces #16814
• [Pulsar Admin] Dynamic configuration for check unknown request parameters #16781
• [Pulsar Admin] Make offload police methods async in Namespaces #16760
• [Pulsar Admin] Support the admin API to check unknown request #16577
• [Pulsar Admin] Make getBacklogQuotaMap method async in Namespaces #16504
• [Pulsar Admin] Make GetMessageIdByTimestamp pure async #16446
• [Pulsar Admin] Make splitNamespaceBundle and getTopicHashPositions async #16411
• [Pulsar Admin] Make AutoSubscriptionCreation async #16329
• [Pulsar Admin] Make SubscriptionExpirationTime method async #16328
• [Pulsar Admin] Make some method on permission async #16324
• [Pulsar Admin] Make unloadNamespaceBundle async #16313
• [Pulsar Admin] Make Namespaces.deleteNamespaceBundle async #16287
• [Pulsar Admin] Make deleteTopic method async #16232
• [Pulsar Admin] Make compactionStatus method async #16231
• [Pulsar Admin] Make terminate method async #16227
• [Pulsar Admin] Make getList async #16221
• [Pulsar Admin] Make getPartitionedTopicList method async #16217
• [Pulsar Admin] Improve documentation for unackedMessages stat #16213
• [Pulsar Admin] Make internalPeekNthMessage method async #16192
• [Pulsar Admin] Fix get non-persistent topics issue in Namespaces #16170
• [Pulsar Admin] Make getInternalStats method async #16141
• [Pulsar Admin] Fix get-publish-rete Admin API handle exception behavior #16001
• [Pulsar Admin] Release LookupRequestSemaphore before returning data #15974
• [Pulsar Admin] Fix producer/consume permission can’t get schema #15956
• [Pulsar Admin] Make publish rate and dispatch rate operation async #15946
• [Pulsar Admin] Support to get topic properties #15944
• [Pulsar Admin] New API to get subscription properties #16095
• [Pulsar Admin] Make some operation subscription dispatchRate methods in Namespaces async #15880
• [Pulsar Admin] Make some methods in SchemasResourceBase async #15821
• [Pulsar Admin] Make some operation replication clusters async #15760
• [Pulsar Admin] Make some methods of ClusterBase pure async #15685
• [Pulsar Admin] Allow creating builtin functions in pulsar-admin CLI #15671
• [Pulsar Admin] Enable users to specify TTL options in units other than seconds #15657
• [Pulsar Admin] Make some operation SubscribeRate methods in Namespaces async #15656
• [Pulsar Admin] Add transaction admin to get recover time in stats #15654
• [Pulsar Admin] Make some operation auto topic creation in Namespaces async #15621
• [Pulsar Admin] Add topic name and sub name for NotFound error message #15606
• [Pulsar Admin] Make some methods in ResourceQuotasBase async #15605
• [Pulsar Admin] Make some methods in TenantsBase async #15603
• [Pulsar Admin] Make some operation messageTTL methods in Namespaces async #15577
• [Pulsar Admin] Enable users to specify some time options in units other than seconds #15563
• [Pulsar Admin] Make some methods of ClusterBase pure async #15527
• [Pulsar Admin] Support filtering system topic when get list #15410
• [Pulsar Admin] Make some methods of ClusterBase pure async #15358
• [Pulsar Admin] Make some methods of ClusterBase pure async #15318
• [Pulsar Admin] Add admin api updateTransactionCoordinatorNumber #15296
• [Pulsar Admin] Put validateTopicOwnershipAsync before validateTopicOperationAsync #15265
• [Pulsar Admin] Remove duplication validateTopicOwnershipAsync #15120
• [Pulsar Admin] Fix pulsar-admin not prompting message when there is a 500 error #14856
• [Pulsar Admin] Add get active brokers api without cluster name #14702
• [Pulsar Admin] Grab contextual stacktrace for sync methods #14620
• [Pulsar Admin] Correct status message prompt #14603
• [Pulsar CLI] Fix NPE in admin-CLI topic stats command #18326
• [Pulsar CLI] Fix CLI client produce don't able to use multiple -m send multiple messages #18238
• [Pulsar CLI] Pulsar shell: allow to create a new config (--file) with a relative path #17675
• [Pulsar CLI] Pulsar shell: do not exit on user interrupt during commands #18615
• [Pulsar CLI] Pulsar shell: allow cloning an existing config #18608
• [Pulsar CLI] Pulsar shell: support relative paths #17648
• [Pulsar CLI] Pulsar shell, Pulsar admin, Pulsar client and Pulsar perf: support for Windows #17243
• [Pulsar CLI] Pulsar shell: add command to set/get property of a config #17651
• [Pulsar CLI] Fix Pulsar shell custom commands execution #17479
• [Pulsar CLI] CLI extensions: rename default location to 'cliextensions' #17435
• [Pulsar CLI] Exclude windows script from the docker image #17404
• [Pulsar CLI] Add a separate TLS transport encryption configuration #16930
• [Pulsar CLI] Fix incorrect description for producing messages #16876
• [Pulsar CLI] Use NoSplitter for subscription properties #16862
• [Pulsar CLI] Add TLS provider support #16700
• [Pulsar CLI] Support getEarliestTimeInBacklog at getPartitionedStats method #16388
• [Pulsar CLI] Add query options to the get topics in Namespace #16167
• [Pulsar CLI] Add cli cmd for subscription level dispatch-rate-limiter #15862
• [Pulsar CLI] Disable Pulsar client memory limit by default #15748
• [Pulsar CLI] Fix check on create function class name #15700
• [Pulsar CLI] Support filtering system topic when get list #15410
• [Pulsar CLI] Change the default for max-connections from 100 to 1 #15387
• [Pulsar CLI] Add short name for full name in admin cli #14301
• [Client Tool] Using int instead of long in python scripts #17215
• [Client Tool] Fix using directory '?' in the docker image #17185
• [Client Tool] Handle ctrl-d and exit the shell #17204
• [Client Tool] Add tlsTrustCertFilePath as CLI argument for pulsar-client-tool #16961
• [Perf Tool] Able to set maxLookupRequest for pulsar-perf #16967
• [Perf Tool] Pulsar-perf fails on jdk17 #18806

Others

• [Proxy] Fix refresh client auth #17831
• [Proxy] Update proxy lookup throw exception type #17600
• [Proxy] Add TLS transport encryption for broker client #16833
• [Proxy] Fix client service URL #16834
• [Proxy] Consolidate Netty channel flushes to mitigate syscall overhead #16372
• [Proxy] Do not preserve host when forwarding admin requests #16342
• [Proxy] Switch to rely on Netty for Hostname Verification #15824
• [Proxy] Support zero-copy of NIC to NIC on Proxy #15678
• [Proxy] Suggestion put brokerProxyAllowedTargetPorts in proxy.conf #15069
• [Proxy] Exit if proxy service fails to start #15076
• [Proxy] Add support of PrometheusRawMetricsProvider for the Pulsar-Proxy #14681
• [Proxy] Fail proxy startup if brokerServiceURL is missing scheme #14682
• [Proxy] Fix DNS server denial-of-service issue when DNS entry expires #15403
• [Proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler #15415
• [WebSocket] Support encryption in websocket proxy #16234
• [WebSocket] Fix MultiTopicReader#getConsumer ClassCastException #15534
• [WebSocket] Add KeyStore support in WebSocket, Function Worker HTTPS Servers #15084
• [WebSocket] Fix ClassCastException when user create MultiTopicReader #14316
• [Package Management] Fix the new path /data introduced regression #15367
• [Schema] Add decode InputStream for Schema #16659
• [Schema] Expose timestamp field for SchemaData&SchemaInfo #16380
• [Schema] Add classLoader field for SchemaDefinition #15915
• [Schema] Fix conversion of TimestampMillisConversion has no effect when Jsr310Conversion enabled #15863
• [Schema] Ensure the deletion consistency of topic and schema #14608
• [Tiered Storage] Shared subscription: improvement with offloaded ledgers #16417
• [Tiered Storage] Fix the wrong secret key name get from env #15814
• [Tiered Storage] Add pure S3 provider for the offloader #15710
• [Tiered Storage] Autorecovery default reppDnsResolverClass to ZkBookieRackAffinityMapping #15640
• [Tiered Storage] Reduce CPU usage when offloading ledgers #15063
• [Tiered Storage] Fix potential NPE in MockManagedLedger #15006
• [Tiered Storage] Add API to scan objects on Tiered Storage #14930
• [Tiered Storage] Fix incorrect entryId in warn log when reading entry from tiered storage #14685
• Include Pulsar shell in the released artifacts #18583
• Add lombok plugin to solve Javadoc issue #18520
• Remove cmd-line level test retries #16524
• Fix potentially incompatible Java opts caused by IS_JAVA_8 #15444
• Remove obsolete jdk11 and errorprone-jdk11 maven profiles #15505
• Remove internal dependency: pulsar-functions-instance #14925
• Enable retries for apt-get when building Pulsar docker image #14513
• Migrate Docker images and CI to Java 17 #14355

Library updates

• Remove versions that are handled by netty-bom #18629
• Upgrade Netty to 4.1.86.Final and Netty Tcnative to 2.0.54.Final #18599
• Upgrade jackson-databind to 2.13.4.2 to get rid of CVE-2022-42003 #18394
• Bump bookkeeper version to 4.15.3 #18455
• Upgrade jackson-databind to 2.13.4.2 to get rid of CVE-2022-42003 #18394
• Upgrade protobuf to 3.19.6 to get rid of CVE-2022-3171 #18086
• Bump commons-text to 1.10.0 fix CVE-2022-42889 #18053
• File tiered storage: upgrade jettison to get rid of CVE-2022-40149 #18022
• Upgrade scala-library to get rid of CVE-2022-36944 #18021
• Upgrade JacksonXML to 2.13.4 #18020
• Upgrade to Jetty to 9.4.48.v20220622 #16520
• Bump maven-dependency-plugin to 3.3.0 #16318
• Bump os-maven-plugin version from 1.4.1.Final to 1.7.0 #16308
• Bump presto.version to run PrestoServer on JDK11+ #16163
• Update fastjson version to 1.2.83 #16148
• Bump PyYAML from 5.3.1 to 5.4.1 to solve CVE-2020-14343 #15989
• Bump spring version to 5.3.20 to solve CVE-2022-22970 #15699
• Bump snappy zstd version to fix CompressorCodecBackwardCompatTest on Apple M1 #15698
• Athenz: do not use uber-jar and bump to 1.10.50 to remove jackson-databind shaded dependency #14884
• Remove --illegal-access errors resulting from Google Guice - Pulsar IO, Offloaders and Pulsar SQL - Bump Guice to 5.1.0 #14300
• Bump prometheus client version from 0.5.0 to 0.15.0 #13785
• Upgrade log4j2 version to 2.18.0 #16884

Documentation

This section only highlights the availability of docs serving existing features.

• Add docs about system topic #14795
• Add a comprehensive reference table about YAML configurations of Pulsar Functions #15389
• Add instructions about how to run CI from fork to the Contribution Guide #15535
• Add docs about basic authentication #15734
• Add docs about isolation deployments #15802
• Redesign the information architecture and improve the content of Pulsar Functions and add a new topic about how to get started #15975
• Add docs about anti-affinity namespace distribution across failure domains #16069
• Add an introductory table about BookKeeper recovery metrics #16554
• Publish REST API docs for lookup related operations #16621
• Add more concepts and user tasks about bookie isolation #16843
• Use 2.8.x/2.9.x/2.10.x doc set instead of version-specific doc set #17074

For the comprehensive list of doc-related improvements in 2.11.0, you can go to GitHub.

A special thanks to the following contributors (alphabetic Github IDs) who helped add the docs for Pulsar 2.11.0.

0xflotus, 315157973, 704237006, AlphaWang, alpreu, AlvaroStream, AnonHxy,Anonymitaet, asafm, cbornet, codelipenghui, coderzc, D-2-Ed, DaveDuggins, Demogorgon314, ericsyh, fantapsody, futeng, hangc0276, heesung-sn, hrsakai, HQebupt, horizonzy, Huanli-Meng, gaozhangmin, ikilobyte, Jason918, komalatammal, labuladong, larshp, lgxbslgx, lhotari, liangyepianzhou, lin-zhao, liudezhi2098, liuzhuang2017, Mans2singh, massakam, mattisonchao, maxsxu, mendonk, merlimat, michaeljmarshall, misselvexu, momo-jun, MTwz, nahguam,nicoloboschi, nodece, poorbarcode, Pradhyumnakashyap, rdhabalia, RobertIndie, samredai, Shawyeok, Sherlock113, shibd, Shoothzj, SignorMercurio, Technoboy-, tjiuming, tisonkun, urfreespace, wangjialing218, wenbingshen, youzipi, and zwOvO.