Elastic Stack Azure Marketplace offering

Easily deploy the Elastic Stack of Elasticsearch, Kibana and Logstash to Azure.

This readme provides an overview of usage and features. For more comprehensive documentation, please refer to the Azure Marketplace and ARM template documentation

This repository consists of:

• src/mainTemplate.json - The main Azure Resource Management (ARM) template. The template itself is composed of many nested linked templates, with the main template acting as the entry point.
• src/createUiDefinition - UI definition file for our Azure Marketplace offering. This file produces an output JSON that the ARM template can accept as input parameters.

Building

After pulling the source, call the following once

npm install

to pull in all devDependencies. You may edit the build/allowedValues.json file, which the build uses to patch the ARM template and Marketplace UI definition. Then, run

npm run build

which will validate EditorConfig settings, lint JSON files, patch the template using build/allowedValues.json, and create a zip in the dist folder. For more details around developing the template, take a look at the Development README

Azure Marketplace

The Azure Marketplace Elastic Stack offering offers a simplified UI and installation experience over the full power of the ARM template.

It will always bootstrap an Elasticsearch cluster complete with a trial license of the Elastic Stack's platinum features.

Deploying through the Marketplace is great and easy way to get your feet wet for the first time with Elasticsearch on Azure, but in the long run, you'll want to deploy the templates directly from GitHub using the Azure CLI or PowerShell SDKs. Check out the CLI examples.

---

You can view the UI in developer mode by clicking here. If you feel something is cached improperly use this client unoptimized link instead

Reporting bugs

Have a look at this screenshot to see how you can navigate to the deployment error status message. Please create an issue with that message and in which resource it occured on our github issues

ARM template

The output from the Azure Marketplace UI is fed directly to the ARM deployment template. You can use the ARM template independently, without going through the Marketplace. In fact, there are many features in the ARM template that are not exposed within the Marketplace UI, such as configuring

• Azure Storage account to use with Azure Repository plugin for Snapshot/Restore
• Application Gateway to use for SSL/TLS and SSL offload

Check out our examples repository for examples of common scenarios and also take a look at the following blog posts for further information

• Spinning up a cluster with Elastic's Azure Marketplace template
• Elasticsearch and Kibana deployments on Azure
• SAML based Single Sign-On with Elasticsearch and Azure Active Directory

Elastic Stack features (formerly known as X-Pack)

Starting with Elasticsearch, Kibana and Logstash 6.3.0, The template deploys with Elastic Stack features bundled as part of the deployment, and includes the free features under the Basic license level. The xpackPlugins parameter determines whether a self-generated trial license is applied, offering a trial period of 30 days to the Platinum license features. A value of Yes applies a trial license, a value of No applies the Basic license. The license level applied determines the Elastic Stack features activated to use.

For Elasticsearch, Kibana and Logstash prior to 6.3.0, The xpackPlugins parameter determines whether X-Pack plugins are installed and a self-generated trial license is applied. In difference to 6.3.0 however, a value of No for xpackPlugins means that X-Pack plugins are not installed, and therefore does not provide the free features under the Basic license level, offering the Open Source features only. For these versions, you can install X-Pack plugins and register for a free Basic license to apply to the deployment, in order to use the free features available under the Basic license level.

Parameters

The ARM template accepts a lot of parameters, but don't fear! Most of them are optional and only used in conjunction with other parameters. Where a parameter value is not explicitly provided, it will take the default value defined in the template.

Parameter	Type	Description	Default Value
artifactsBaseUrl	string	The base url of the Elastic ARM template. Required	Raw content of the current branch
location	string	The location where to provision all the items in this template. Defaults to inheriting the location from the resource group. Any other value must be a valid Azure region.	[resourceGroup().location]
vmHostNamePrefix	string	The prefix to use for hostnames when naming virtual machines in the cluster. Hostnames are used for resolution of master nodes on the network, so if you are deploying a cluster into an existing virtual network containing an existing Elasticsearch cluster, be sure to set this to a unique prefix, to differentiate the hostnames of this cluster from an existing cluster. Can be up to 5 characters in length, must begin with an alphanumeric character and can contain alphanumeric and hyphen characters.	""
Elasticsearch related settings
esVersion	string	A valid supported Elasticsearch version for the target template version. See this list for supported versions. Required	Latest version supported by target template version
esClusterName	string	The name of the Elasticsearch cluster. Required	""
loadBalancerType	string	The load balancer to set up to access the cluster. Can be internal, external or gateway. By choosing internal, only an internal load balancer is deployed. Useful when connecting to the cluster happens from inside the Virtual Network By choosing external, both internal and external load balancers will be deployed. Kibana communicates with the cluster through the internal load balancer. By choosing gateway, Application Gateway will be deployed for load balancing, allowing a PKCS#12 archive (.pfx/.p12) containing the certificate and key to be supplied for SSL/TLS to and from Application Gateway, and providing SSL offload. An internal load balancer will also deployed. Application Gateway and Kibana communicate with the cluster through the internal load balancer. If you are setting up Elasticsearch or Kibana on a publicly available IP address, it is highly recommended to secure access to the cluster with a product like Elastic Stack Security, in addition to configuring SSL/TLS.	internal
xpackPlugins	string	Either Yes or No to install a trial license of the Elastic Stack features (formerly X-Pack) such as Monitoring, Security, Alerting, Graph, Machine Learning (5.5.0+) and SQL. If also installing Kibana, it will have Reporting and Profiler installed. A value of No for Elasticsearch and Kibana prior to 6.3.0, will include only the Open Source features. A value of No for Elasticsearch and Kibana 6.3.0+ will include the free Basic license features.	Yes
azureCloudPlugin	string	Either Yes or No to install the Azure Repository plugin for snapshot/restore. When set to Yes, at least azureCloudStorageAccountName must be specified to configure the plugin correctly.	No
azureCloudStorageAccountName	string	The name of an existing storage account to use for snapshots with Azure Repository plugin. Must be a valid Azure Storage Account name.	""
azureCloudStorageAccountResourceGroup	string	The name of an existing resource group containing the storage account azureCloudStorageAccountName to use for snapshots with Azure Repository plugin. Must be a valid Resource Group name.	""
esAdditionalPlugins	string	Additional Elasticsearch plugins to install. Each plugin must be separated by a semicolon. e.g. analysis-icu;mapper-attachments	""
esAdditionalYaml	string	Additional configuration for Elasticsearch yaml configuration file. Each line must be separated by a newline character \n e.g. "action.auto_create_index: +.*\nindices.queries.cache.size: 5%". This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment.	""
esHeapSize	integer	The size, in megabytes, of memory to allocate on each Elasticsearch node for the JVM heap. If unspecified, 50% of the available memory will be allocated to Elasticsearch heap, up to a maximum of 31744MB (~32GB). Take a look at the Elasticsearch documentation for more information. This is an expert level feature - setting a heap size too low, or larger than available memory on the Elasticsearch VM SKU will fail the deployment.	0
esHttpCertBlob	string	A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.	""
esHttpCertPassword	securestring	The password for the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be protected with a password. If using esHttpCaCertBlob, this password will be used to protect the generated PKCS#12 archive on each node. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.	""
esHttpCaCertBlob	string	A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for the HTTP layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.	""
esHttpCaCertPassword	securestring	The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be be protected with a password. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.	""
esTransportCaCertBlob	string	A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for Transport layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.	""
esTransportCaCertPassword	securestring	The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for Transport layer to Elasticsearch. Optional as the archive may not be be protected with a password. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.	""
esTransportCertPassword	securestring	The password to protect the generated PKCS#12 archive on each node. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.	""
samlMetadataUri	string	The URI from which the metadata file for the Identity Provider can be retrieved to configure SAML Single-Sign-On. For Azure Active Directory, this can be found in the Single-Sign-On settings of the Enterprise Application, and will look something like https://login.microsoftonline.com/<guid>/federationmetadata/2007-06/federationmetadata.xml?appid=<guid> Supported only for Elasticsearch 6.2.0+ Kibana must be installed X-Pack plugin must be installed with a level of license that enables the SAML realm. SSL/TLS must be configured for HTTP layer of Elasticsearch	""
samlServiceProviderUri	string	The public URI for the Service Provider to configure SAML Single-Sign-On. If samlMetadataUri is provided but no value is provided for samlServiceProviderUri, the public domain name for the deployed Kibana instance will be used. Supported only for Elasticsearch 6.2.0+ Kibana must be installed SSL/TLS must be configured for HTTP layer of Elasticsearch	""
Master node related settings
vmSizeMasterNodes	string	Azure VM size of dedicated master nodes. See this list for supported sizes. By default the template deploys 3 dedicated master nodes, unless dataNodesAreMasterEligible is set to Yes. Check that the size you choose is available in the region you choose.	Standard_D1
vmMasterNodeAcceleratedNetworking	string	Whether to enable accelerated networking for Master nodes, which enables single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. Valid values are Default: enables accelerated networking for VMs known to support it Yes: enables accelerated networking. No: does not enable accelerated networking	Default
Data node related settings
dataNodesAreMasterEligible	string	Either Yes or No to make all data nodes master eligible. This can be useful for small Elasticsearch clusters however, for larger clusters it is recommended to have dedicated master nodes. When Yes no dedicated master nodes will be provisioned.	No
vmSizeDataNodes	string	Azure VM size of the data nodes. See this list for supported sizes. Check that the size you choose is available in the region you choose.	Standard_D1
vmDataNodeAcceleratedNetworking	string	Whether to enable accelerated networking for Data nodes, which enables single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. Valid values are Default: enables accelerated networking for VMs known to support it Yes: enables accelerated networking. No: does not enable accelerated networking	Default
vmDataNodeCount	int	The number of data nodes you wish to deploy. Must be greater than 0.	3
Data node disk related settings
vmDataDiskCount	int	Number of managed disks to attach to each data node in RAID 0 setup. Must be equal to or greater than 0. If the number of disks selected is more than can be attached to the data node VM (SKU) size, the maximum number of disks that can be attached for the data node VM (sku) size will be used. Equivalent to taking min(vmDataDiskCount, max supported disks for data node VM size) When 1 disk is selected, the disk is not RAIDed. When 0 disks are selected, no disks will be attached to each data node. Instead, the temporary disk will be used to store Elasticsearch data. The temporary disk is ephemeral in nature and not persistent. Consult Microsoft Azure documentation on temporary disks to understand the trade-offs in using it for storage.	Maximum number supported disks for data node VM size
vmDataDiskSize	string	The disk size of each attached disk. Choose 32TiB, 16TiB, 8TiB, 4TiB, 2TiB, 1TiB, 512GiB, 256GiB, 128GiB, 64GiB or 32GiB. For Premium Storage, disk sizes equate to P80, P70, P60, P50, P40, P30, P20, P15, P10 and P6 storage disk types, respectively.	1TiB
storageAccountType	string	The storage account type of the attached disks. Choose either Default or Standard. The Default storage account type will be Premium Storage for VMs that support Premium Storage and Standard Storage for those that do not. Standard will use Standard Storage.	Default
Coordinating node related settings
vmClientNodeCount	int	The number of coordinating nodes to provision. Must be a positive integer. By default, the data nodes are added to the backend pool of the loadbalancer but if you provision coordinating nodes, these will be added to the loadbalancer instead. Coordinating nodes can be useful in offloading the gather process from data nodes and are necessary to scale an Elasticsearch cluster deployed with this template beyond 100 data nodes (the maximum number of VMs that can be added to a load balancer backend pool).	0
vmSizeClientNodes	string	Azure VM size of the coordinating nodes see this list for supported sizes. Check that the size you choose is available in the region you choose.	Standard_D1
vmClientNodeAcceleratedNetworking	string	Whether to enable accelerated networking for coordinating nodes, which enables single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. Valid values are Default: enables accelerated networking for VMs known to support it Yes: enables accelerated networking. No: does not enable accelerated networking	Default
Security related settings
adminUsername	string	Admin username used when provisioning virtual machines. Must be a valid Linux username i.e. avoid any of the following usernames for Ubuntu	""
authenticationType	string	The authentication type for the Admin user. Either password or sshPublicKey	password
adminPassword	securestring	When authenticationType is password this sets the OS level user's password	""
sshPublicKey	securestring	When authenticationType is sshPublicKey this sets the OS level sshKey that can be used to login.	""
securityBootstrapPassword	securestring	Security password for 6.x bootstrap.password key that is added to the keystore. If no value is supplied, a 13 character password will be generated using the ARM template uniqueString() function. The bootstrap password is used to seed the built-in users. Used only in 6.0.0+	""
securityAdminPassword	securestring	Security password Admin user. This is the built-in elastic user. should be a minimum of 12 characters, and must be greater than 6 characters.	""
securityKibanaPassword	securestring	Security password Kibana. This is the built-in kibana user. should be a minimum of 12 characters, and must be greater than 6 characters.	""
securityLogstashPassword	securestring	This is the built-in logstash_system user. should be a minimum of 12 characters, and must be greater than 6 characters.	""
securityBeatsPassword	securestring	This is the built-in beats_system user. Valid for Elasticsearch 6.3.0+ should be a minimum of 12 characters, and must be greater than 6 characters.	""
securityApmPassword	securestring	This is the built-in apm_system user. Valid for Elasticsearch 6.5.0+ should be a minimum of 12 characters, and must be greater than 6 characters.	""
securityRemoteMonitoringPassword	securestring	This is the built-in remote_monitoring_user user. Valid for Elasticsearch 6.5.0+ should be a minimum of 12 characters, and must be greater than 6 characters.	""
Kibana related settings
kibana	string	Either Yes or No to provision a machine with Kibana installed and a public IP address to access it.	Yes
vmSizeKibana	string	Azure VM size of the Kibana instance. See this list for supported sizes. Check that the size you select is available in the region you choose.	Standard_A2
vmKibanaAcceleratedNetworking	string	Whether to enable accelerated networking for Kibana, which enables single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. Valid values are Default: enables accelerated networking for VMs known to support it Yes: enables accelerated networking. No: does not enable accelerated networking	Default
kibanaCertBlob	string	A Base-64 encoded form of the certificate (.crt) in PEM format to secure HTTPS communication between the browser and Kibana.	""
kibanaKeyBlob	securestring	A Base-64 encoded form of the private key (.key) in PEM format to secure HTTPS communication between the browser and Kibana.	""
kibanaKeyPassphrase	securestring	The passphrase to decrypt the private key. Optional as the key may not be encrypted.	""
kibanaAdditionalYaml	string	Additional configuration for Kibana yaml configuration file. Each line must be separated by a \n newline character e.g. "server.name: \"My server\"\nkibana.defaultAppId: home". This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment.	""
Logstash related settings
logstash	string	Either Yes or No to provision a machine with Logstash installed.	No
vmSizeLogstash	string	Azure VM size of the Logstash instance. See this list for supported sizes. Check that the size you select is available in the region you choose.	Standard_D1
vmLogstashAcceleratedNetworking	string	Whether to enable accelerated networking for Logstash, which enables single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. Valid values are Default: enables accelerated networking for VMs known to support it Yes: enables accelerated networking. No: does not enable accelerated networking	Default
logstashHeapSize	integer	The size, in megabytes, of memory to allocate for the JVM heap for Logstash. If unspecified, Logstash will be configured with the default heap size for the distribution and version. Take a look at the Logstash documentation on profiling heap size for more information. This is an expert level feature - setting a heap size too low, or larger than available memory on the Logstash VM SKU will fail the deployment.	0
logstashConf	securestring	A Base-64 encoded form of a Logstash config file to deploy.	""
logstashKeystorePassword	securestring	The password to protect the Logstash keystore. If no value is supplied, a value will be generated using the ARM template uniqueString() function. Used only in 6.2.0+	""
logstashAdditionalPlugins	string	Additional Logstash plugins to install. Each plugin must be separated by a semicolon. e.g. logstash-input-heartbeat;logstash-input-twitter	""
logstashAdditionalYaml	string	Additional configuration for Logstash yaml configuration file. Each line must be separated by a newline character \n e.g. "pipeline.batch.size: 125\npipeline.batch.delay: 50". This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment.	""
Jumpbox related settings
jumpbox	string	Either Yes or No to optionally add a virtual machine with a public IP to the deployment, which you can use to connect and manage virtual machines on the internal network. NOTE: If you are deploying Kibana, the Kibana VM can act as a jumpbox, so a separate jumpbox VM is not needed.	No
Virtual network related settings
vNetNewOrExisting	string	Whether the Virtual Network is new or existing. An existing Virtual Network in another Resource Group in the same Location can be used.	new
vNetName	string	The name of the Virtual Network. The Virtual Network must already exist when using an existing Virtual Network	es-net
vNetExistingResourceGroup	string	The name of the Resource Group in which the Virtual Network resides when using an existing Virtual Network. Required when using an existing Virtual Network	""
vNetNewAddressPrefix	string	The address prefix when creating a new Virtual Network. Required when creating a new Virtual Network	10.0.0.0/24
vNetLoadBalancerIp	string	The internal static IP address to use when configuring the internal load balancer. Must be an available IP address on the provided vNetClusterSubnetName.	10.0.0.4
vNetClusterSubnetName	string	The name of the subnet to which Elasticsearch nodes will be attached. The subnet must already exist when using an existing Virtual Network	es-subnet
vNetNewClusterSubnetAddressPrefix	string	The address space of the subnet. Required when creating a new Virtual Network	10.0.0.0/25
vNetAppGatewaySubnetName	string	Subnet name to use for the Application Gateway. Required when selecting gateway for load balancing. The subnet must already exist when using an existing Virtual Network	es-gateway-subnet
vNetNewAppGatewaySubnetAddressPrefix	string	The address space of the Application Gateway subnet. Required when creating a new Virtual Network and selecting gateway for load balancing.	10.0.0.128/28
Application Gateway related settings
appGatewayTier	string	The tier of the Application Gateway, either Standard or WAF. Required when selecting gateway for load balancing.	Standard
appGatewaySku	string	The size of the Application Gateway. Choose Small, Medium or Large. When choosing appGatewayTier WAF, the size must be at least Medium. Required when selecting gateway for load balancing.	Medium
appGatewayCount	int	The number instances of the Application Gateway. Can be a value between 1 and 10. A minimum of 2 is recommended for production. Required when selecting gateway for load balancing.	2
appGatewayCertBlob	string	A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the certificate and key for Application Gateway. This certificate is used to secure HTTPS connections to and from the Application Gateway. Required when selecting gateway for load balancing.	""
appGatewayCertPassword	securestring	The password for the PKCS#12 archive (.p12/.pfx) containing the certificate and key for Application Gateway. Required when selecting gateway for load balancing.	""
appGatewayEsHttpCertBlob	securestring	The Base-64 encoded public certificate (.cer) used to secure the HTTP layer of Elasticsearch. Used by the Application Gateway to whitelist certificates used by the backend pool. Required when using esHttpCertBlob to secure the HTTP layer of Elasticsearch and selecting gateway for load balancing. X-Pack plugin must be installed	""
appGatewayWafStatus	string	The firewall status of the Application Gateway, either Enabled or Disabled. Required when selecting gateway for load balancing and using appGatewayTier WAF.	Enabled
appGatewayWafMode	string	The firewall mode of the Application Gateway, either Detection or Prevention. Required when selecting gateway for load balancing and using appGatewayTier WAF.	Detection

Web based deploy

The above button will take you to the autogenerated web based UI based on the parameters from the ARM template.

Command line deploy

You can deploy using the template directly from Github using the Azure CLI or Azure PowerShell

Azure CLI 1.0

Azure CLI 1.0 is no longer supported as the apiVersions of resources are newer than those supported by the last release. It's recommended to update to Azure CLI 2.0.

Azure CLI 2.0

1. Log into Azure

  az login

2. Create a resource group <name> in a <location> (e.g westeurope) where we can deploy too

az group create --name <name> --location <location>

3. Use our template directly from GitHub using --template-uri

az group deployment create \
  --resource-group <name> \
  --template-uri https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json \
  --parameters @parameters/password.parameters.json

where <name> refers to the resource group you just created.

Azure PowerShell

1. Log into Azure

Login-AzureRmAccount

2. Select a Subscription Id

Select-AzureRmSubscription -SubscriptionId "<subscriptionId>"

3. Define the parameters object for your deployment as a PowerShell hashtable. The keys correspond the parameters defined in the Parameters section

$clusterParameters = @{
    "artifactsBaseUrl"="https://raw.githubusercontent.com/elastic/azure-marketplace/master/src"
    "esVersion" = "7.1.1"
    "esClusterName" = "elasticsearch"
    "loadBalancerType" = "internal"
    "vmDataDiskCount" = 1
    "adminUsername" = "russ"
    "adminPassword" = "Password1234"
    "securityBootstrapPassword" = "Password1234"
    "securityAdminPassword" = "Password1234"     
    "securityKibanaPassword" = "Password1234"
    "securityLogstashPassword" = "Password1234"
    "securityBeatsPassword" = "Password1234"
    "securityApmPassword" = "Password1234"
    "securityRemoteMonitoringPassword" = "Password1234"
}

4. Create a resource group <name> in a <location> (e.g westeurope) where we can deploy too

New-AzureRmResourceGroup -Name "<name>" -Location "<location>"

5. Use our template directly from GitHub

New-AzureRmResourceGroupDeployment -Name "<deployment name>" -ResourceGroupName "<name>" -TemplateUri "https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json" -TemplateParameterObject $clusterParameters

Targeting a specific template version

You can target a specific version of the template by modifying the URI of the template and the artifactsBaseUrl parameter of the template to point to a specific tagged release.

Targeting a specific template version is recommended for repeatable production deployments.

For example, to target the 7.0.0 tag release with PowerShell

$templateVersion = "7.0.0"
$templateBaseUrl = "https://raw.githubusercontent.com/elastic/azure-marketplace/$templateVersion/src"

# minimum parameters required to deploy
$clusterParameters = @{
  "artifactsBaseUrl" = $templateBaseUrl
  "esVersion" = "7.0.0"
  "adminUsername" = "russ"
  "adminPassword" = "Password1234"
  "securityBootstrapPassword" = "Password1234"
  "securityAdminPassword" = "Password1234"
  "securityKibanaPassword" = "Password1234"
  "securityLogstashPassword" = "Password1234"
  "securityBeatsPassword" = "Password1234"
  "securityApmPassword" = "Password1234"
  "securityRemoteMonitoringPassword" = "Password1234"
}

$resourceGroup = "my-azure-cluster"
$location = "Australia Southeast"
$name = "my-azure-cluster"

New-AzureRmResourceGroup -Name $resourceGroup -Location $location
New-AzureRmResourceGroupDeployment -Name $name -ResourceGroupName $resourceGroup -TemplateUri "$templateBaseUrl/mainTemplate.json" -TemplateParameterObject $clusterParameters

Configuring TLS

It is strongly recommended that you secure communication using Transport Layer Security when using the template. The Elastic Stack security features can provide Basic Authentication, Role Based Access control, and Transport Layer Security (TLS) for both Elasticsearch and Kibana. For more details, please refer to the Security documentation.

For Elasticsearch versions 6.8.0+ (and less than 7.0.0), and 7.1.0+, the Elastic Stack security features that allow configuring TLS and role based access control are available in the free basic license tier. For all other versions, the Elastic Stack security features require a license level higher than basic; They can be configured with a trial license, which provides access to the Security features for 30 days.

TLS for Kibana

You can secure external access from the browser to Kibana with TLS by supplying a certificate and private key in PEM format with kibanaCertBlob and kibanaKeyBlob parameters, respectively.

TLS for Elasticsearch Transport layer

You can secure communication between nodes in the cluster with TLS on the Transport layer. Configuring TLS for the Transport layer requires xPackPlugins be set to Yes, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+.

You must supply a PKCS#12 archive with the esTransportCaCertBlob parameter (and optional passphrase with esTransportCaCertPassword) containing the CA cert which should be used to generate a certificate for each node within the cluster. An optional passphrase can be passed with esTransportCertPassword to encrypt the generated certificate on each node.

One way to generate a PKCS#12 archive containing a CA certificate and key is using Elastic's certutil command. The simplest command to generate a CA certificate is

./certutil ca

and follow the instructions.

TLS for Elasticsearch HTTP layer

You can secure external access to the cluster with TLS with an external loadbalancer or Application Gateway. Configuring TLS for the HTTP layer requires xPackPlugins be set to Yes, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+.

External load balancer

If you choose external as the value for loadBalancerType, you must either

• supply a PKCS#12 archive containing the key and certificate with the esHttpCertBlob parameter (and optional passphrase with esHttpCertPassword) containing the certs and private key to secure the HTTP layer. This certificate will be used by all nodes within the cluster, and

or

• supply a PKCS#12 archive containing the key and certificate with the esHttpCaCertBlob parameter (and optional passphrase with esHttpCaCertPassword) containing the CA which should be used to generate a certificate for each node within the cluster to secure the HTTP layer. Kibana will be configured to trust the CA and perform hostname verification for presented certificates. One way to generate a PKCS#12 archive is using Elastic's certutil command.

Application Gateway

If you choose gateway as the value for loadBalancerType, you must

• supply a PKCS#12 archive containing the key and certificate with the appGatewayCertBlob parameter (and optional passphrase with appGatewayCertPassword) to secure communication to Application Gateway. One way to generate a PKCS#12 archive is using Elastic's certutil command

Application Gateway performs SSL offload, so communication from Application Gateway to Elasticsearch is not encrypted with TLS by default. TLS to Application Gateway may be sufficient for your needs, but if you would like end-to-end encryption by also configuring TLS for Elasticsearch HTTP layer, you can

• supply a PKCS#12 archive containing the key and certificate with the esHttpCertBlob parameter (and optional passphrase with esHttpCertPassword) containing the certs and private key to secure the HTTP layer. This certificate will be used by all nodes within the cluster, and Kibana will be configured to trust the certificate CA (if CA certs are present within the archive). One way to generate a PKCS#12 archive is using Elastic's certutil command, and you must specify a --dns <name> argument with a name that matches that in the --name <name> argument.

and

• supply the public certificate in PEM format from the PKCS#12 archive passed with esHttpCertBlob parameter, using the appGatewayEsHttpCertBlob parameter. Application Gateway whitelists certificates used by VMs in the backend pool. This can be extracted from the PKCS#12 archive of the esHttpCertBlob parameter using openssl pkcs12

  openssl pkcs12 -in http_cert.p12 -out http_public_cert.cer -nokeys

  and provide the passphrase for the archive when prompted.

IMPORTANT: When configuring end-to-end encryption with Application Gateway, the certificate to secure the HTTP layer must include a x509v3 Subject Alternative Name extension with a DNS entry that matches the Subject CN, to work with Application Gateway's whitelisting mechanism. This can be checked using openssl x509

openssl x509 -in http_public_cert.cer -text -noout

which will output something similar to

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            // omitted for brevity ...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Elastic Certificate Tool Autogenerated CA
        Validity
            Not Before: Jul  5 02:37:40 2018 GMT
            Not After : Jul  4 02:37:40 2021 GMT
        Subject: CN=custom
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    // omitted for brevity ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                // omitted for brevity ...
            X509v3 Authority Key Identifier:
                // omitted for brevity ...

            X509v3 Subject Alternative Name:
                DNS:custom
            X509v3 Basic Constraints:
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         // omitted for brevity ...

Without this, Application Gateway will return 502 Bad Gateway errors, as the health probe for the backend pool will fail when the whitelisted certificate does not contain this certificate extension. You can typically understand if there is a problem with the key format when

1. TLS has been configured on the HTTP layer
2. Kibana is able to communicate to the cluster correctly but Application Gateway returns 502 errors.

This may not always be the case, but can be indicative. You should also check the description for Backend Health of the Application Gateway in the Azure portal.

Passing certificate parameters

Parameters such as esHttpCertBlob and kibanaCertBlob must be provided in Base-64 encoded form. A Base-64 encoded value can be obtained using

1. base64 on Linux, or openssl on Linux and MacOS

   base64

   httpCert=$(base64 http-cert.p12) 

   openssl

   httpCert=$(openssl base64 -in http-cert.p12)

   and including the value assigned to $httpCert in the parameters.json file as the value for certificate parameter passed to the Azure CLI command

2. PowerShell on Windows

   $httpCert = [Convert]::ToBase64String([IO.File]::ReadAllBytes("c:\http-cert.p12"))

   and then pass this in the template parameters object passed to the Azure PowerShell command

   $clusterParameters = @{
       # Other parameters skipped for brevity
       "esHttpCertBlob"= $httpCert
   }

License

This project is MIT Licensed and was originally forked from the Elasticsearch Azure quick start arm template