Skip to main content
Version: 5.0.x

Migration

Pulsar has 3 types of broker load balancers, that is, simple, modular, and extensible.

You can perform the following migrations.

MigrationWhen to use
Migrate from simple to modularIf you want to use the bundle unloading strategy of AvgShedder, OverloadShedder, ThresholdShedder, or UniformLoadShedder.
Migrate from modular to extensibleIf you want to use the bundle unloading strategy of TransferShedder.
Migrate from extensible to modularIf you want to use the bundle unloading strategy of AvgShedder, OverloadShedder, ThresholdShedder, or UniformLoadShedder.
note

It is not recommended to migrate from the modular or extensible to the simple broker load balancer since the simple broker load balancer is deprecated and no longer in use.

Considerations​

Before migrating from one broker load balancer type to another, review the relationship between broker load balancer type, Pulsar version, and bundle unloading strategy. You may need to upgrade Pulsar versions or update the bundle unloading strategy. Below are brief summaries.

This broker load balancer typeis available in this Pulsar version
SimpleAll versions
Modular1.7 and later
Extensible3.0 and later
This bundle unloading strategyis available for this broker load balancer typein available this Pulsar version
OverloadShedderModular1.18 and later
ThresholdShedderModular2.6 and later
UniformLoadShedderModular2.10 and later
AvgShedderModular3.0.6, 3.2.4, 3.3.1 and later (default since 5.0)
TransferShedderExtensible3.0 and later

Migrate from simple to modular broker load balancer​

You can migrate from the simple to the modular broker load balancer, by manually changing the configuration settings in the broker.conf file or by using the pulsar-admin tool.

Change broker.conf file​

  1. Access to the broker.conf file.

    vim apache-pulsar-5.0.0-M2/conf/broker.conf

  2. Change the broker load balancer by setting loadManagerClassName to ModularLoadManagerImpl in the broker.conf file.

    loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ModularLoadManagerImpl

  3. Restart the Pulsar cluster. The new setting will take effect after the restart.

Use pulsar-admin tool​

  1. Access the pulsar-admin tool.

    cd apache-pulsar-5.0.0-M2/bin

  2. Set --config to loadManagerClassName and --value to org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.

    pulsar-admin brokers update-dynamic-config \
    --config loadManagerClassName \
    --value org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl

    You do not need to restart the Pulsar cluster. The new settings will take effect after 1 to 2 minutes.

Migrate from modular to extensible broker load balancer​

You can migrate from the modular to the extensible broker load balancer, by manually changing settings in the broker.conf file. During the migration, the lookup and assignment will be redirected to the brokers with the extensible load balancer.

For a rolling migration on Pulsar 5.0 and later, first enable lookup redirection on the existing brokers. Introducing the new load manager without that flag leaves the two types handling lookups independently.

note

Do not change loadManagerClassName through pulsar-admin brokers update-dynamic-config for this migration. Although the setting is dynamic, changing it swaps the manager in place without reconfiguring the leader-election and load-reporting tasks initialized at startup or migrating bundle ownership between the two managers. Use broker.conf and a rolling upgrade of brokers; use the dynamic API to enable and disable migration redirection.

Change broker.conf file​

  1. Upgrade the Pulsar cluster to 3.0.0 or later versions.

  2. Access to the broker.conf file.

    vim apache-pulsar-5.0.0-M2/conf/broker.conf

  3. Change the following settings in the broker.conf file:

    loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl 

    loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder
    note

  4. Restart the Pulsar cluster. The new settings will take effect after the restart.

Migrate from extensible to modular broker load balancer​

You can migrate from the extensible to the modular broker load balancer, by manually changing the setting in the broker.conf file. During the migration, the lookup and assignment will be redirected to the brokers with the modular load balancer.

For a rolling migration on Pulsar 5.0 and later, first enable lookup redirection on the existing brokers, including when rolling back a migration.

note

Do not change loadManagerClassName through pulsar-admin brokers update-dynamic-config for this migration. Although the setting is dynamic, changing it swaps the manager in place without reconfiguring the leader-election and load-reporting tasks initialized at startup or migrating bundle ownership between the two managers. Use broker.conf and a rolling upgrade of brokers; use the dynamic API to enable and disable migration redirection.

Change broker.conf file​

  1. Access to the broker.conf file.

    vim apache-pulsar-5.0.0-M2/conf/broker.conf

  2. Change the following settings in the broker.conf file:

    • Update broker load balancer by setting loadManagerClassName to ModularLoadManagerImpl

    • Update bundle unloading strategy to AvgShedder (the default of the modular load balancer from Pulsar 5.0.0; the 5.0.0-M1/M2 milestones still default to ThresholdShedder), OverloadShedder, ThresholdShedder, or UniformLoadShedder based on your needs.

    loadManagerClassName=org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl

    loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder
    loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder
    note

    TransferShedder is only supported in the extensible broker load balancer, so you need to change TransferShedder to other bundle unloading strategies.

  3. Restart the Pulsar cluster. The new settings will take effect after the restart.

Enable lookup redirection during a rolling migration​

Pulsar 5.0 and later disable migration redirection by default (loadManagerMigrationEnabled=false). Before starting the first broker with a different load manager, enable it dynamically on the existing cluster:

pulsar-admin brokers update-dynamic-config --config loadManagerMigrationEnabled --value true

Wait for the setting to propagate before introducing the new type. This flag enables lookup redirection; change loadManagerClassName and its associated strategy settings in the broker configuration and restart the brokers, as described above. Do not change the load manager type through the dynamic configuration API for this migration.

While both load manager types are running, the most recently started broker determines the load manager type to which lookups are redirected, and each type assigns bundles only to brokers of its own type. The modular and extensible ownership stores remain separate. Plan enough capacity in the new pool for new assignments and expect additional ownership changes during migration. Follow Rolling upgrade of brokers to pause automatic rebalancing, control bundle movement, and check each replacement before proceeding.

After all brokers use the intended load manager, disable migration redirection:

pulsar-admin brokers update-dynamic-config --config loadManagerMigrationEnabled --value false

Pulsar 4.x performs this redirection without the flag. Choosing the extensible load manager's ownership backend (system topic or metadata store) is a separate migration; do not combine it with a load manager type change in the same rollout.

Was this helpful?
💡 Suggest changes🛟 Get support