Authentication and authorization in Pulsar
Pulsar supports a pluggable authentication mechanism that Pulsar clients can use to authenticate with brokers. Pulsar can also be configured to support multiple authentication sources.
Role tokens
In Pulsar, a role is a string, like admin
or app1
, that can represent a single client or multiple clients. Roles are used to control permission for clients to produce or consume from certain topics, administer the configuration for properties, and more.
The purpose of the authentication provider is to establish the identity of the client and then assign that client a role token. This role token is then used to determine what the client is authorized to do.
Authentication providers
Out of the box, Pulsar supports two authentication providers:
TLS client auth
In addition to providing connection encryption between Pulsar clients and brokers, Transport Layer Security (TLS) can be used to identify clients through a certificate signed by a trusted certificate authority.
Creating certificates
Creating TLS certificates for Pulsar involves creating a certificate authority (CA), broker certificate, and client certificate.
Certificate authority
The first step is to create the certificate for the CA. The CA will be used to sign both the broker and client certificates, in order to ensure that each party will trust the others.
Linux
$ CA.pl -newca
MacOS
$ /System/Library/OpenSSL/misc/CA.pl -newca
After answering the question prompts, this will store CA-related files in the ./demoCA
directory. Within that directory:
demoCA/cacert.pem
is the public certificate. It is meant to be distributed to all parties involved.demoCA/private/cakey.pem
is the private key. This is only needed when signing a new certificate for either broker or clients and it must be safely guarded.
Broker certificate
Once a CA certificate has been created, you can create certificate requests and sign them with the CA.
The following commands will ask you a few questions and then create the certificates. When asked for the common name, you need to match the hostname of the broker. You could also use a wildcard to match a group of broker hostnames, for example *.broker.usw.example.com
. This ensures that the same certificate can be reused on multiple machines.
$ openssl req \
-newkey rsa:2048 \
-sha256 \
-nodes \
-out broker-cert.csr \
-outform PEM
Convert the key to PKCS 8 format:
$ openssl pkcs8 \
-topk8 \
-inform PEM \
-outform PEM \
-in privkey.pem \
-out broker-key.pem \
-nocrypt
This will create two broker certificate files named broker-cert.csr
and broker-key.pem
. Now you can create the signed certificate:
$ openssl ca \
-out broker-cert.pem \
-infiles broker-cert.csr
At this point, you should have a broker-cert.pem
and broker-key.pem
file. These will be needed for the broker.
Client certificate
To create a client certificate, repeat the steps in the previous section, but did create client-cert.pem
and client-key.pem
files instead.
For the client common name, you need to use a string that you intend to use as the role token for this client, though it doesn’t need to match the client hostname.
Configure the broker for TLS
To configure a Pulsar broker to use TLS authentication, you’ll need to make some changes to the broker.conf
configuration file, which is located in the conf
directory of your Pulsar installation.
Add these values to the configuration file (substituting the appropriate certificate paths where necessary):
# Enable TLS and point the broker to the right certs
tlsEnabled=true
tlsCertificateFilePath=/path/to/broker-cert.pem
tlsKeyFilePath=/path/to/broker-key.pem
tlsTrustCertsFilePath=/path/to/cacert.pem
# Enable the TLS auth provider
authenticationEnabled=true
authorizationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
A full listing of parameters available in the conf/broker.conf
file, as well as the default values for those parameters, can be found in Broker Configuration.
Configure the discovery service
The discovery service used by Pulsar brokers needs to redirect all HTTPS requests, which means that it needs to be trusted by the client as well. Add this configuration in conf/discovery.conf
in your Pulsar installation:
tlsEnabled=true
tlsCertificateFilePath=/path/to/broker-cert.pem
tlsKeyFilePath=/path/to/broker-key.pem
Configure clients
For more information on Pulsar client authentication using TLS, see the following language-specific docs:
Configure CLI tools
Command-line tools like pulsar-admin
, pulsar-perf
, and pulsar-client
use the conf/client.conf
config file in a Pulsar installation.
You’ll need to add the following authentication parameters to that file to use TLS with Pulsar’s CLI tools:
serviceUrl=https://broker.example.com:8443/
authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
authParams=tlsCertFile:/path/to/client-cert.pem,tlsKeyFile:/path/to/client-key.pem
useTls=true
tlsAllowInsecureConnection=false
tlsTrustCertsFilePath=/path/to/cacert.pem
Athenz
Athenz is a role-based authentication/authorization system. In Pulsar, Athenz role tokens (aka z-tokens) can be used to establish the identify of the client.
Athenz authentication settings
In a decentralized Athenz system there is both an authoriZation Management System (ZMS) server and an authoriZation Token System (ZTS) server.
To begin, you need to set up Athenz service access control. You should create domains for the provider (which provides some resources to other services with some authentication/authorization policies) and the tenant (which is provisioned to access some resources in a provider). In this case, the provider corresponds to the Pulsar service itself and the tenant corresponds to each application using Pulsar (typically, a property in Pulsar).
Create the tenant domain and service
On the tenant side, you need to:
- Create a domain, such as
shopping
- Generate a private/public key pair
- Create a service, such as
some_app
, on the domain with the public key
Note that the private key generated in step 2 needs to be specified when the Pulsar client connects to the broker (see client configuration examples for Java and C++).
For more specific steps involving the Athenz UI, please refer to this doc.
Create the provider domain and add the tenant service to some role members
On the provider side, you need to:
- Create a domain, such as
pulsar
- Create a role
- Add the tenant service to members of the role
Note that in step 2 any action and resource can be specified since they are not used on Pulsar. In other words, Pulsar uses the Athenz role token only for authentication, not for authorization.
For more specific steps involving UI, please refer to this doc.
Configure the broker for Athenz
TLS encryption strongly recommended
Please note that using TLS encryption is strongly recommended when using Athenz as an authentication provider, as it can protect role tokens from being intercepted and reused (see also this doc).
In the conf/broker.conf
configuration file in your Pulsar installation, you need to provide the class name of the Athenz authentication provider as well as a comma-separated list of provider domain names.
# Add the Athenz auth provider
authenticationEnabled=true
authorizationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderAthenz
athenzDomainNames=pulsar
# Enable TLS
tlsEnabled=true
tlsCertificateFilePath=/path/to/broker-cert.pem
tlsKeyFilePath=/path/to/broker-key.pem
A full listing of parameters available in the conf/broker.conf
file, as well as the default values for those parameters, can be found in Broker Configuration.
Configure clients for Athenz
For more information on Pulsar client authentication using Athenz, see the following language-specific docs:
Configure CLI tools for Athenz
Command-line tools like pulsar-admin
, pulsar-perf
, and pulsar-client
use the conf/client.conf
config file in a Pulsar installation.
You’ll need to add the following authentication parameters to that file to use Athenz with Pulsar’s CLI tools:
# URL for the broker
serviceUrl=https://broker.example.com:8443/
# Set Athenz auth plugin and its parameters
authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationAthenz
authParams={"tenantDomain":"shopping","tenantService":"some_app","providerDomain":"pulsar","privateKey":"file:///path/to/private.pem","keyId":"v1"}
# Enable TLS
useTls=true
tlsAllowInsecureConnection=false
tlsTrustCertsFilePath=/path/to/cacert.pem
Authorization
In Pulsar, the authentication provider is charged with properly identifying clients and associating them with role tokens. Authorization is the process that determines what clients are able to do.
Authorization in Pulsar is managed at the property level, which means that you can have multiple authorization schemes active in a single Pulsar instance. You could, for example, create a shopping
property that has one set of roles and applies to a shopping application used by your company, while an inventory
property would be used only by an inventory application.
When working with properties, you can specify which of your Pulsar clusters your property is allowed to use. This enables you to also have cluster-level authorization schemes.
Creating a new property
A Pulsar property identifies a tenant and is typically provisioned by Pulsar instance administrators or by some kind of self-service portal.
Properties are managed using the pulsar-admin
tool. Here’s an example property creation command:
$ bin/pulsar-admin properties create my-property \
--admin-roles my-admin-role \
--allowed-clusters us-west,us-east
This command will create a new property my-property
that will be allowed to use the clusters us-west
and us-east
.
A client that successfully identified itself as having the role my-admin-role
would then be allowed to perform all administrative tasks on this property.
The structure of topic names in Pulsar reflects the hierarchy between properties, clusters, and namespaces:
Managing permissions
Permissions in Pulsar are managed at the namespace level (that is, within properties and clusters).
Grant permissions
You can grant permissions to specific roles for lists of operations such as produce
and consume
.
pulsar-admin
Use the grant-permission
subcommand and specify a namespace, actions using the --actions
flag, and a role using the --role
flag:
$ pulsar-admin namespaces grant-permission test-property/cl1/ns1 \
--actions produce,consume \
--role admin10
Wildcard authorization can be performed when authorizationAllowWildcardsMatching
is set to true
in broker.conf
.
e.g.
$ pulsar-admin namespaces grant-permission test-property/cl1/ns1 \
--actions produce,consume \
--role 'my.role.*'
Then, roles my.role.1
, my.role.2
, my.role.foo
, my.role.bar
, etc. can produce and consume.
$ pulsar-admin namespaces grant-permission test-property/cl1/ns1 \
--actions produce,consume \
--role '*.role.my'
Then, roles 1.role.my
, 2.role.my
, foo.role.my
, bar.role.my
, etc. can produce and consume.
Note: A wildcard matching works at the beginning or end of the role name only.
e.g.
$ pulsar-admin namespaces grant-permission test-property/cl1/ns1 \
--actions produce,consume \
--role 'my.*.role'
In this case, only the role my.*.role
has permissions.
Roles my.1.role
, my.2.role
, my.foo.role
, my.bar.role
, etc. cannot produce and consume.
REST API
POST
/admin/namespaces/:property/:cluster/:namespace/permissions/:role
Java
admin.namespaces().grantPermissionOnNamespace(namespace, role, getAuthActions(actions));
Get permission
You can see which permissions have been granted to which roles in a namespace.
pulsar-admin
Use the permissions
subcommand and specify a namespace:
$ pulsar-admin namespaces permissions test-property/cl1/ns1
{
"admin10": [
"produce",
"consume"
]
}
REST API
GET
/admin/namespaces/:property/:cluster/:namespace/permissions
Java
admin.namespaces().getPermissions(namespace);
Revoke permissions
You can revoke permissions from specific roles, which means that those roles will no longer have access to the specified namespace.
pulsar-admin
Use the revoke-permission
subcommand and specify a namespace and a role using the --role
flag:
$ pulsar-admin namespaces revoke-permission test-property/cl1/ns1 \
--role admin10
REST API
DELETE
/admin/namespaces/:property/:cluster/:namespace/permissions/:role
Java
admin.namespaces().revokePermissionsOnNamespace(namespace, role);
Superusers
In Pulsar you can assign certain roles to be superusers of the system. A superuser is allowed to perform all administrative tasks on all properties and namespaces, as well as to publish and subscribe to all topics.
Superusers are configured in the broker configuration file in conf/broker.conf
configuration file, using the superUserRoles
parameter:
superUserRoles=my-super-user-1,my-super-user-2
A full listing of parameters available in the conf/broker.conf
file, as well as the default values for those parameters, can be found in Broker Configuration.
Typically, superuser roles are used for administrators and clients but also for broker-to-broker authorization. When using geo-replication, every broker needs to be able to publish to other clusters’ topics.
Pulsar admin authentication
String authPluginClassName = "com.org.MyAuthPluginClass";
String authParams = "param1:value1";
boolean useTls = false;
boolean tlsAllowInsecureConnection = false;
String tlsTrustCertsFilePath = null;
ClientConfiguration config = new ClientConfiguration();
config.setAuthentication(authPluginClassName, authParams);
config.setUseTls(useTls);
config.setTlsAllowInsecureConnection(tlsAllowInsecureConnection);
config.setTlsTrustCertsFilePath(tlsTrustCertsFilePath);
PulsarAdmin admin = new PulsarAdmin(url, config);
To use TLS:
String authPluginClassName = "com.org.MyAuthPluginClass";
String authParams = "param1:value1";
boolean useTls = false;
boolean tlsAllowInsecureConnection = false;
String tlsTrustCertsFilePath = null;
ClientConfiguration config = new ClientConfiguration();
config.setAuthentication(authPluginClassName, authParams);
config.setUseTls(useTls);
config.setTlsAllowInsecureConnection(tlsAllowInsecureConnection);
config.setTlsTrustCertsFilePath(tlsTrustCertsFilePath);
PulsarAdmin admin = new PulsarAdmin(url, config);