Atlas

Deploying an ARM template with Atlas

Today we’re going to take a look at using Atlas to deploy an Azure Resource Manager (arm) template. Deploying an arm template is a convenient and powerful way to create or modify multiple resources simultaneously on Azure. In this example we’ll be creating a storage account and blob container, but the same Atlas workflow will work for any template.

Upfront it should be said there are already several different ways you can do this. Azure DevOps has tasks which can deploy an ARM template. There are also Azure CLI shell commands and Azure PowerShell cmdlets which can be used to script this kind of scenario. So why would you choose to use Atlas when these options already exist?

I think that question comes down to what you’re trying to do overall. You may have found yourself in a situation where you are writing a lot of scripting code to which runs multiple azure templates intermixed with updating keyvault secrets and uploading customized blob files – all while using the outputs of one step as the parameters for the next. In those situations Atlas would be good to evaluate. It enables you to process complex JSON in a simple declarative manner as you chain together any number of operations.

Preparing to run examples

If you’d like to follow along and try this at home, first you’ll need a copy of Atlas. You can install that with the following command:

# to install for the first time:
dotnet tool install -g dotnet-atlas --add-source https://aka.ms/atlas-ci/index.json

# to update to the latest version:
dotnet tool update -g dotnet-atlas --add-source https://aka.ms/atlas-ci/index.json

The second thing you’ll need is to provide a few typical values like your tenant id and subscription id. You can provide these on the command line, like --set azure.tenant=YOUR-TENANT-ID, but it’s even more convenient to provide them in a values.yaml file in the current directory.

azure:
  tenant: # your tenantId
  subscription: # your subscriptionId

You can find your tenant id as the Directory ID in the Active Directory Properties blade on the Azure management portal. You can find your Subscription ID value in the Subscriptions blade on the same Azure management portal.

There is also an example workflow available which lists the tenant and subscription id combinations available to you.

atlas deploy https://github.com/Microsoft/Atlas/tree/master/examples/302-gather-info

Well, that’s enough prep work. Let’s jump right in by creating a resource group.

Step 1: Resource group creation

TL;DR You can run step1 from github.com/lodejard/workflows with the following command.

atlas deploy https://github.com/lodejard/workflows/tree/master/blog/arm-example/step1

Make a arm-example subfolder for the workflow and, based on the REST API docs, we can define the ResourceGroups_CreateOrUpdate details.

# https://docs.microsoft.com/en-us/rest/api/resources/resourcegroups/resourcegroups_createorupdate

method: PUT
url: https://management.azure.com/subscriptions/{{ azure.subscription }}/resourcegroups/{{ azure.resourceGroup }}?api-version=2018-02-01
auth:
  tenant: {{ azure.tenant }}
  resource: https://management.azure.com/
  client: 04b07795-8ddb-461a-bbee-02f9e1bf7b46 # Azure CLI
body:
  location: {{ azure.location }}

We will also need some default values for Azure region and resource group name.

azure:
  location: westus2
  resourceGroup: atlas-arm-example

Then we can call this operation to create the resource group.

operations:
- message: Create or update resource group
  request: ResourceGroups_CreateOrUpdate.yaml

That should be enough to run and see it work.

atlas deploy arm-example
Atlas version 0.1.3558601

  - Create or update resource group
PUT https://management.azure.com/subscriptions/3d2exxxx-xxxx-xxxx-xxxx-xxxxxxxx3c62/resourcegroups/atlas-arm-example?api-version=2018-02-01
OK https://management.azure.com/subscriptions/3d2exxxx-xxxx-xxxx-xxxx-xxxxxxxx3c62/resourcegroups/atlas-arm-example?api-version=2018-02-01 831ms

That’s not much so far, but creating a resource group is a necessary first step before you can create a deployment.

Step 2: Deploying the ARM template

TL;DR You can run step2 from github.com/lodejard/workflows with the following command.

atlas deploy --set storage.name=myuniquename1234 https://github.com/lodejard/workflows/tree/master/blog/arm-example/step2

For the next operation we’ll want to create an Azure ARM deployment. There is a REST API for that, of course, and based on it we’ll create this request definition.

# https://docs.microsoft.com/en-us/rest/api/resources/deployments/deployments_createorupdate

method: PUT
url: https://management.azure.com/subscriptions/{{ azure.subscription }}/resourcegroups/{{ azure.resourceGroup }}/providers/Microsoft.Resources/deployments/{{ azure.deploymentName }}?api-version=2018-02-01
auth:
  tenant: {{ azure.tenant }}
  resource: https://management.azure.com/
  client: 04b07795-8ddb-461a-bbee-02f9e1bf7b46 # Azure CLI
body: 
  properties:
    mode: Incremental
    {{{ yaml deployment indent=4 }}}

This operation will require a few more parameters, so let’s also add that to the workflow’s default values.yaml.

azure:
  location: westus2
  resourceGroup: atlas-arm-example
  deploymentName: atlas-deployment

And there is a new parameter that must be globally unique. Because there is no reasonable default, you’ll need to add that to the values.yaml in your current directory.

azure:
  tenant: # your tenantId
  subscription: # your subscriptionId
storage:
  name: # a valid storage account name that's not already in use

Now there are quite a few ways the workflow can provide additional body properties needed to create a deployment. The template can be provided indirectly in the properties.templateLink URI, or it can be embedded directly in the request body as properties.template JSON. When deploying with Atlas I prefer to keep the ARM template json file right in the workflow and embed it in the request body by importing it as a partial file.

For this example let’s use the azure-quickstart-templates repo. First copy the 101-storage-blob-container/azuredeploy.json and save it as an arm-example/azuredeploy.json file. Then add a second operation to the workflow which imports this template file and provides the required parameters.

operations:
- message: Create or update resource group
  request: ResourceGroups_CreateOrUpdate.yaml

- message: Create ARM template deployment
  values:
    deployment:
      template: {{> azuredeploy.json }}
      parameters:
        storageAccountName:
          value: ( storage.name )
  request: Deployments_CreateOrUpdate.yaml

You can run this again now and it will be successful, but it will finish suspiciously quickly. That’s because the template deployment is actually still running asynchronously on the server-side. If you read the _output/logs/002---Create-ARM-template-deployment.yaml file you can see in the response where that is indicated as providioningState: Accepted.

Because of that, the third and final thing we’ll want to do is poll the deployment until the provisioningState changes so we know if the deployment succeeded or failed. If we know that it failed we can also cause the workflow to throw an exception.

Step3: Monitoring deployment for success or failure

TL;DR You can run step3 from github.com/lodejard/workflows with the following command.

atlas deploy --set storage.name=myuniquename1234 https://github.com/lodejard/workflows/tree/master/blog/arm-example/step3

The request definition for this operation is virtually identical to the one which created the deployment entity. You’ll just need to change the method to GET and remove the body. Isn’t REST nice that way? 🙂

# https://docs.microsoft.com/en-us/rest/api/resources/deployments/deployments_get

method: GET
url: https://management.azure.com/subscriptions/{{ azure.subscription }}/resourcegroups/{{ azure.resourceGroup }}/providers/Microsoft.Resources/deployments/{{ azure.deploymentName }}?api-version=2018-02-01
auth:
  tenant: {{ azure.tenant }}
  resource: https://management.azure.com/
  client: 04b07795-8ddb-461a-bbee-02f9e1bf7b46 # Azure CLI
secret: result.body.properties.correlationId

The workflow.yaml is the only other file which needs updating. In the third and fourth operation you’ll see we’re using some additional properties like repeat, condition, and throw. Oh – and here’s another trick – there’s a {{secret arg}} template helper you can use to hide sensitive values from console output and log files. I don’t think the subscription id is sensitive, but I’ve been redacting it anyway so I’ll add it to the workflow to make that automatic. You can also declare secrets which in your request YAML files, like just above. That’s especially helpful for API calls like ListKeys.

But I digress. Here is the complete workflow which waits for a response and throws error details in case of failure.

# secrets - {{ secret azure.subscription }} {{ secret azure.tenant }}

operations:
- message: Create or update resource group
  request: ResourceGroups_CreateOrUpdate.yaml

- message: Create ARM template deployment
  values:
    deployment:
      template: {{> azuredeploy.json }}
      parameters:
        storageAccountName:
          value: ( storage.name )
  request: Deployments_CreateOrUpdate.yaml

- message: Monitoring deployment...
  request: Deployments_Get.yaml
  output:
    deployment: (result.body.properties)
  repeat:
    condition: ( contains(['Accepted', 'Running'], deployment.provisioningState) )
    delay: PT5S
    timeout: PT18M

- message: Deployment failed
  condition: (deployment.provisioningState != 'Succeeded')
  throw:
    message: Deployment failed
    details:
      provisioningState: (deployment.provisioningState)
      error: (deployment.error)
      correlationId: (deployment.correlationId)

This workflow should now be able to run successfully like this:

atlas deploy arm-example
Atlas version 0.1.3558601

  - Create or update resource group
PUT https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example?api-version=2018-02-01 774ms

  - Create ARM template deployment
PUT https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 1449ms

  - Monitoring deployment...
GET https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 82ms

  - Monitoring deployment...
GET https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 106ms

  - Monitoring deployment...
GET https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 82ms

  - Monitoring deployment...
GET https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 75ms
deployment:
  templateHash: "12144851514961987775"
  parameters:
    storageAccountName:
      type: String
      value: myuniquename1234
    containerName:
      type: String
      value: logs
    location:
      type: String
      value: westus2
  mode: Incremental
  provisioningState: Succeeded
  timestamp: 2018-10-16T23:36:24.3727647Z
  duration: PT12.9109667S
  correlationId: xxxxxxxx
  providers:
  - namespace: Microsoft.Storage
    resourceTypes:
    - resourceType: storageAccounts
      locations:
      - westus2
    - resourceType: storageAccounts/blobServices/containers
      locations:
      -
  dependencies:
  - dependsOn:
    - id: /subscriptions/xxxxxxxx/resourceGroups/atlas-arm-example/providers/Microsoft.Storage/storageAccounts/myuniquename1234
      resourceType: Microsoft.Storage/storageAccounts
      resourceName: myuniquename1234
    id: /subscriptions/xxxxxxxx/resourceGroups/atlas-arm-example/providers/Microsoft.Storage/storageAccounts/myuniquename1234/blobServices/default/containers/logs
    resourceType: Microsoft.Storage/storageAccounts/blobServices/containers
    resourceName: myuniquename1234/default/logs
  outputResources:
  - id: /subscriptions/xxxxxxxx/resourceGroups/atlas-arm-example/providers/Microsoft.Storage/storageAccounts/myuniquename1234
  - id: /subscriptions/xxxxxxxx/resourceGroups/atlas-arm-example/providers/Microsoft.Storage/storageAccounts/myuniquename1234/blobServices/default/containers/logs

We can also set an additional deployment parameter from the command line in order to see what a failed deployment looks like. Underscores are not allowed in blob container names, and there is a containerName parameter defined in the azuredeploy.json we can override.

atlas deploy arm-example --set deployment.parameters.containerName.value=bad_name
Atlas version 0.1.3558601

  - Create or update resource group
PUT https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example?api-version=2018-02-01 954ms

  - Create ARM template deployment
PUT https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 1439ms

  - Monitoring deployment...
GET https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 58ms

  - Monitoring deployment...
GET https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 201ms

  - Monitoring deployment...
GET https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 97ms

  - Monitoring deployment...
GET https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01
OK https://management.azure.com/subscriptions/xxxxxxxx/resourcegroups/atlas-arm-example/providers/Microsoft.Resources/deployments/atlas-deployment?api-version=2018-02-01 76ms

  - Deployment failed
Deployment failed
provisioningState: Failed
error:
  code: DeploymentFailed
  message: At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.
  details:
  - code: BadRequest
    message: >-
      {

        "error": {

          "code": "ContainerOperationFailure",

          "message": "The specifed resource name contains invalid characters.\nRequestId:e3d3371f-701e-00bb-0ca9-65e956000000\nTime:2018-10-16T23:42:49.6311746Z"

        }

      }
correlationId: xxxxxxxx

Microsoft.Atlas.CommandLine.Execution.OperationException: Deployment failed
   at Microsoft.Atlas.CommandLine.Commands.WorkflowCommands.ExecuteOperation(ExecutionContext context) in /var/lib/vstsagent/1/_work/1/s/src/Microsoft.Atlas.CommandLine/Commands/WorkflowCommands.cs:line 476
   at Microsoft.Atlas.CommandLine.Commands.WorkflowCommands.ExecuteOperations(ExecutionContext parentContext, IList`1 operations) in /var/lib/vstsagent/1/_work/1/s/src/Microsoft.Atlas.CommandLine/Commands/WorkflowCommands.cs:line 277
   at Microsoft.Atlas.CommandLine.Commands.WorkflowCommands.ExecuteCore(Boolean generateOnly) in /var/lib/vstsagent/1/_work/1/s/src/Microsoft.Atlas.CommandLine/Commands/WorkflowCommands.cs:line 241
   at Microsoft.Atlas.CommandLine.CommandLineApplicationExtensions.<>c__DisplayClass6_0`1.<OnExecute>b__1(TCommand cmd) in /var/lib/vstsagent/1/_work/1/s/src/Microsoft.Atlas.CommandLine/CommandLineApplicationExtensions.cs:line 118
   at Microsoft.Atlas.CommandLine.CommandLineApplicationExtensions.<>c__DisplayClass5_0`1.<OnExecute>b__0() in /var/lib/vstsagent/1/_work/1/s/src/Microsoft.Atlas.CommandLine/CommandLineApplicationExtensions.cs:line 98
   at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Microsoft.Atlas.CommandLine.Program.Main(String[] args) in /var/lib/vstsagent/1/_work/1/s/src/Microsoft.Atlas.CommandLine/Program.cs:line 52

And at that point we have the ability to deploy an arm template! This technique was used to deploy the DevOps agent pools used for Atlas’s own CI/CD pipelines, that workflow is the 401-linux-agent-pool example in the Atlas repo. That is actually a pretty good example of what was mentioned at the top of this article – after a storage account and keyvault is created the workflow proceeds to upload blob files, store secrets, and deploy a second template which creates the agent pool virtual machines.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.