Money is always important whether you are developing for yourself, your employer, or a client.

One way to save money in Azure is to shut down resources at the end of the day. As an Azure data engineer, it can be a tedious task that is easy to forget. There can be a lot of clicking through resources in the portal to get things shut down, especially if you have a lot of resources. Why not make our lives easier by scripting a solution?

Candidates for Shutdown

In this post, I’m going to talk about two resources that I usually shut down at the end of the workday while they are in active development: VMs and Synapse Analytic SQL Pools.

VMs

VMs are often used as development platforms during the day and are left idle throughout the night. While idle, the cost still accumulates. There are a few options available to shut down VMs. VMs can be shut down one by one manually via the portal.

There is also a great Start/Stop feature that can automatically shutdown your VMs at specific times. This is effective, but only if you work during fixed hours. The other option is shutting down via the Azure CLI which I will explain here.

Before going any further it is important to understand 2 VM states:

  • Stopped – The VM is powered down. This is equivalent to (and can be achieved by) shutting down the VM from the OS. Costs are the same as if it is running.
  • Stopped (Deallocated) – The compute resources are deallocated and you only pay for Azure Storage.

One caveat is that you will likely lose your dynamically assigned IP address when deallocating.

The portal Stop button moves the VM into the “Stopped (Deallocated)” state.

 

Azure CLI

Shutting down a VM via the Azure CLI gives the most control and flexibility. It is also an easy way to become familiar with the Azure CLI.

The commands covered today are

az vm stop

az vm deallocate

If you have a couple of static VMs you can just hard code these two commands and save them in a shell script and run them at the end of your workday. I recommend using the Azure Cloud Shell.

az vm stop --resource-group MyResourceGroup --name MyVm

az vm deallocate --name MyVm --no-wait --resource-group MyResourceGroup

If you have a few VMs in a resource group you can shut them all down with a commands like these:

az vm stop --ids $(az vm list -g MyResourceGroup --query "[].id" -o tsv)

az vm deallocate --ids $(az vm list -g MyResourceGroup --query "[].id" -o tsv)

Synapse SQL Pools

Synapse Analytics Workspaces (note: at the time of writing this is a preview feature and subject to change) have SQL and Spark Pools associated with them. If there are no workloads for your fixed performance level SQL pool you can safely scale it down or pause it overnight.

Just like VMs, this can be accomplished via the portal or via script.

The Azure CLI command to pause your SQL pool is:

az synapse sql pool pause --name sqlpoolcli1 --workspace-name testsynapseworkspace --resource-group rg

Dynamic Scripting

Up to now, you’ve seen that we can use Azure CLI to shut down VMs and Synapse SQL Pools via static scripts. This is very effective if you have one project and resources don’t change frequently. If you are working in multiple projects/resource groups bash scripting (via a UNIX terminal with Azure CLI installed) can be leveraged to use these commands dynamically.

The below script is one I developed to make it easier to shut down VMs and Synapse SQL Pools within a resource group. It is presented here in its current form but will be improved and expanded in my repository. Please feel free to use it and modify it to suit your needs. It will prompt you to select the resource group and list all possible SQL Pools and VMs to shut down. You can then select which resources to shutdown/pause. It will show the commands it will run and prompt you if you want to continue.

#!/bin/bash
## Author: Matt Szafir
## Description: This script is inteded to be used to shut down resources at the end of the day to save costs while developing in Azure
# Currently only doing VMs and Synapse SQL Pools
## Requirements: bash >= 4
# az cli version?
# az login already run
## TODO: add logging & perhaps integrate with a morning script
# add tagging functionality – where every resource w/ a tag could be shut down
# add script flags & help
# check if already paused / shut down (VM: "powerState": "VM deallocated")
# add more resource types
# use "&" for commands so they run in parallel
#### ENABLE SYNAPSE EXTENSION ####
printf '%s\n' "ENABLING SYNAPSE EXTENSION"
az extension add –name synapse
#### CONFIGURE RESOURCE GROUP ####
mapfile -t resourceGroupsArray < <( az group list –query "[].name" -o tsv )
j=1
for i in ${resourceGroupsArray[@]}; do
printf '%s\n' "${j}: ${i}"
j=$((j+1))
done
printf '%s\n' "Select Resource Group: [1 – ${#resourceGroupsArray[@]}]" &&
read rgNum
resourceGroup=${resourceGroupsArray[${rgNum}-1]}
printf '%s\n' "Configuring default resource group to "${resourceGroup}
az configure –defaults group=${resourceGroup}
#### VMs ####
cmd="az vm list –query [].name -o tsv"
mapfile -t VMArray < <( ${cmd} )
VMStopCommandsArray=()
for i in ${VMArray[@]}; do
VMStopCommandsArray+=("az vm stop –name "${i})
done
#### SYNAPSE POOLS ####
# Get workspaces
cmd="az synapse workspace list -o tsv –query [].name"
mapfile -t SynapseWorkspaceArray < <( ${cmd} )
# Loop through workspaces and build array of SQL Pools
SynapsePauseCommandsArray=()
SynapseSQLPoolArray=()
for i in ${SynapseWorkspaceArray[@]}; do
cmd="az synapse sql pool list –query [].name -o tsv –resource-group "${resourceGroup}" –workspace-name "${i}
mapfile -t SynapseSQLPoolTempArray < <( ${cmd} )
for j in ${SynapseSQLPoolTempArray[@]}; do
SynapsePauseCommandsArray+=("az synapse sql pool pause –name "${j}" –workspace-name "${i})
done
SynapseSQLPoolArray=("${SynapseSQLPoolArray[@]}" "${SynapseSQLPoolTempArray[@]}")
done
#### ASK USER WHICH RESOURCES TO SHUT DOWN ####
# combine arrays
allResourceArray=("${SynapseSQLPoolArray[@]}" "${VMArray[@]}")
allDeleteCommandsArray=("${SynapsePauseCommandsArray[@]}" "${VMStopCommandsArray[@]}")
j=1
for i in ${allResourceArray[@]}; do
printf '%s\n' "${j}: ${i}"
j=$((j+1))
done
if [ ${#allResourceArray[@]} -ne 0 ]; then
printf '%s\n' "Select the numbers cooresponding to resources above to shut down in comma separated list:" &&
read csvnums
else
printf '%s\n' "No Resources to shut down in "${resourceGroup}". Exiting"
return
fi
# Display pause commands for user
printf '%s\n' "About to run the following commands:"
for ((i=0; i< ${#allDeleteCommandsArray[@]}; i++))
do
if [[ $csvnums == *$(( ${i}+1 ))* ]]; then
printf '%s\n' "${allDeleteCommandsArray[$i]}"
if [[ ${allDeleteCommandsArray[$i]} == *"az vm stop"* ]]; then
printf '%s\n' "az vm deallocate –no-wait "${allDeleteCommandsArray[$i]:11:1000} # :1000 -> This is lazy
fi
fi
done
printf '%s\n' "Continue? (Y/N)" &&
read cont
if [ ${cont} == "Y" ]
then
for ((i=0; i< ${#allDeleteCommandsArray[@]}; i++))
do
if [[ ${csvnums} == *$(( ${i}+1 ))* ]]; then
printf '%s\n' "Running ${allDeleteCommandsArray[${i}]}"
# Run the command
( ${allDeleteCommandsArray[${i}]} )
# If we are stopping a VM we should also deallocate it
if [[ ${allDeleteCommandsArray[${i}]} == *"az vm stop"* ]]; then
printf '%s\n' "Running az vm deallocate –no-wait ${allDeleteCommandsArray[${i}]:11:1000}" # :1000 -> This is lazy
( az vm deallocate ${allDeleteCommandsArray[${i}]:11:1000} ) # :1000 -> This is lazy
fi
fi
done
printf '%s\n' "Done!"
return
else printf '%s\n' "Exiting"
fi
return

view raw
endOfDayShutdown.sh
hosted with ❤ by GitHub

Conclusion

I hope this helps you save time and money as you develop in Azure! Please reach out to me via the repository or on Twitter @MattSzafirPro.