Can I automate upgrading the Octopus Deploy server?

Knowledge Base
,
1

Right now, when a user wants to upgrade their Octopus Deploy server they have to manually RDP into the VM, download the latest Octopus Server MSI, backup the database, turn on maintenance mode, and then run the installer. With HA it gets even more complex, as you have to remote into all the servers and install the MSI on all the machines as fast as possible.

Is there a way to automate that?

2

Everyone’s upgrade requirements are different. Some customers I work with want to be on the latest version always. They automate a process to check our website once a day. Other customers only want to upgrade when they need to. Others want to stay on LTS. Some want the fast lane, but the last release of a major version (2019.1.11 instead of 2019.2.2).

HA adds another layer of complexity. TL;DR; it is something we have discussed but other items have been taking precedence.

In this process I am going to walk through how you can have one Octopus Deploy instance upgrade another instance. In this case, I am using an Octopus Cloud instance to upgrade my self-hosted instance running on my hypervisor in my home office.

On all my nodes on the self-hosted instance I installed a tentacle. The tentacles are used by the cloud instance. One of the tentacles is marked “Primary” while all the other ones are marked secondary.

image

image

In my Octopus Cloud instance, I created a new environment called “Maintenance.” I then created a new lifecycle which only has one environment in it, maintenance.

image
image.png272×562 35 KB

Please note: if you are using 2019.11 or higher, you should use runbooks instead of creating a new lifecycle.

I installed tentacles on the HA Nodes on my local infrastructure. For the main node (Octo01), I assigned it the roles HAServer-Primary and HAServer. On all other nodes, I assigned the roles HAServer and HAServer-Secondary (currently just Octo02). All nodes were assigned to the Maintenance environment.

My process checks a hot folder on a file share. I download an MSI and place it into that folder. I did this so I could download EAP releases from our build server when needed. After that first step, the process follows the same general rules we recommend when upgrading an HA server.

  1. Put server into maintenance mode. Maintenance mode is a DB setting. It will prevent any new tasks from being created by non-admins. It only needs to be done on the Primary node. The role is HAServer-Primary
  2. Drain and stop all nodes. Draining a node will stop any new task from being picked up. Stopping and draining nodes has to be done on all servers. The role is HAServer
  3. Backup SQL Server. It is database work. It only needs to be done on the Primary node. The role is HAServer-Primary. We have to stop the nodes because an upgrade could involve updating the database. I used the community step template “SQL - Backup Database” for this.
  4. Run the MSI on the primary node first. If there are any database changes to be made, we only want to run those changes once. The role is HAServer-Primary. The install will start up the Primary Node
  5. Run the MSI on all the secondary nodes. No database changes will occur with this install (they already happened). The role is HAServer-Secondary. The install will start up all the secondary nodes.
  6. Move the MSI out of the hot folder
  7. Take the server out of maintenance mode. Maintenance mode is a DB setting, it only needs to be done on the primary node. The role is HAServer-Primary.

image

Scripts for this process

Set Maintenance Mode

$OctopusAPIKey = $OctopusParameters["Global.Octopus.ApiKey"]  ## API Key of someone who has rights to set maintenance mode

$header = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$header.Add("X-Octopus-ApiKey", $OctopusAPIKey)
$header.Add("X-HTTP-Method-Override", "PUT")

$rawRequest = @{
  Id = "maintenance";
  IsInMaintenanceMode = $True;  ## Change this to false when you want to take out of maintenance mode
  Links = @{
    Self = "/api/maintenanceconfiguration";
  }
}
$jsonRequest = $rawRequest | ConvertTo-Json

Write-Host "Sending in the request $jsonRequest"

$maintenanceUrl = "https://local.octopusdemos.app/api/maintenanceconfiguration"
Write-Host "Setting maintenance mode $maintenanceUrl"
$maintenanceResponse = Invoke-RestMethod $maintenanceUrl -Headers $header -Method POST -Body $jsonRequest

Write-Host "Maintenance's response: $maintenanceResponse"

Script to drain and stop nodes

Set-Location "${env:ProgramFiles}\Octopus Deploy\Octopus" 

& .\octopus.server.exe node --instance="OctopusServer" --drain=true --wait=0 ##This will wait until all currently running tasks have finished
& .\octopus.server.exe service --instance="OctopusServer" --stop

Script to run the MSIs and take the instance out of drain mode

$upgradeMsiList = Get-ChildItem -Path \\192.168.0.233\data\octopusupgrade\* -Include *.msi

foreach ($msi in $upgradeMsiList){
    Write-Output "Installing MSI $msi" 
    $msiExitCode = (Start-Process -FilePath "msiexec.exe" -ArgumentList "/i $msi /quiet" -Wait -Passthru).ExitCode 
    Write-Output "Server MSI installer returned exit code $msiExitCode" 
}

Set-Location "${env:ProgramFiles}\Octopus Deploy\Octopus" 

& .\octopus.server.exe database --instance="OctopusServer" --upgrade
& .\octopus.server.exe service --instance="OctopusServer" --start
& .\octopus.server.exe node --instance="OctopusServer" --drain=false