297 lines
17 KiB
Markdown
297 lines
17 KiB
Markdown
|
h1. Converting Database-Connected Services To An AlkamiManifest Based Installer
|
|||
|
|
|
|||
|
|
> As a followup to the work done on DEV-120482 and lessons learned, we have moved the documentation for further work here to Confluence. Please continue to check this page for improved details as you work your tickets.
|
|||
|
|
|
|||
|
|
h3. Background and Business Case
|
|||
|
|
|
|||
|
|
EC2 based deploys at Alkami routinely take over 2.5 hours, sometimes creeping into 5 hours. The given window for installs is 4 hours, but we need a way to decrease those times to meet our new SLAs. As part of that effort, SRE and Vanguard have been working to introduce new patterns for deployments and deployment/release management (Vanguard in concert with Release Management).
|
|||
|
|
|
|||
|
|
The motto to-date has been this: "Change just one thing!" - [The Art of Troubleshooting|https://artoftroubleshooting.com/2012/01/10/change-just-one-thing/]. But now we reach the end of the road on changing one thing at a time as we feel all the other components parallelizable installation have been met with good success. With this document we are looking to finalize moving all packages to deployed to EC2 instances to use an AlkamiManifest and to be in a known package format.
|
|||
|
|
|
|||
|
|
Originally there was the assumption that the installer tool would produce all the required components for installation, including nuspec, but this was in a time prior to the introduction of manifests, and required too many areas of customization with added complexity at many steps. By moving to an AlkamiManifest based installer pattern, teams will have more direct control of their destiny, at the expense of a one-time setup cost, and a minor ongoing maintenance cost which would match any other maintenance for the legacy system (renaming items, moving the locations of dlls as built, etc). Additionally, this tooling does not adequately or correctly support dotnet core based services, and can not easily be made to do so without sufficient modification of all processes involved.
|
|||
|
|
|
|||
|
|
This adoption of an AlkamiManifest based installer allows the release engineering team to make decisions about what actions are minimally required, so that we can avoid longer "safety checks" imposed by choices made six or more years ago. It should be possible to speed up our processes by way of parallelization, and by introducing new tools that do not rely on the foundational need of chocolatey as an end-all be-all deployment mechanism.
|
|||
|
|
|
|||
|
|
The last area of service types to focus on have been the database-connected services. There will be a few further tickets aimed at looking for legacy components that still need to be moved over to AlkamiManifest based deliverables, as these are the areas that will cause us the greatest roadblock to deploying packages more smoothly. As this transition process is completed, the release engineering team will be looking to refuse to deploy any new packages that do not conform to this standard.
|
|||
|
|
|
|||
|
|
However, there is an *important callout to be noted*: Due to the way our filtering logic was written ([code here](link to repository)) there may be logic services on this list, which would not include the database associated logic.
|
|||
|
|
|
|||
|
|
h3. Instructions For Conversion
|
|||
|
|
|
|||
|
|
To update your Database-Connected Legacy Microservices, please use the following instructions. There is no available Visual Studio tooling as there was for Widgets and Providers, for this just run some commands on the command line.
|
|||
|
|
|
|||
|
|
Each ticket will contain a message such as the following:
|
|||
|
|
|
|||
|
|
> For your team <*team_name*> (ex: *AI)* we have found this list of projects that we believe need to be updated. They all seem to use the Alkami.MicroServices.Installer.Database|MasterDatabase nuget packages, and therefore should be updated to the Alkami.Installer.Services pattern.
|
|||
|
|
> {code}
|
|||
|
|
> Alkami.MicroServices.Aggregation.Service
|
|||
|
|
> Alkami.MicroServices.Aggregation.Service.Host
|
|||
|
|
> Alkami.MicroServices.AggregationProviders.Yodlee.Host
|
|||
|
|
> Alkami.MicroServices.FicoScore.Service.Host
|
|||
|
|
> Alkami.MicroServices.MyAccounts.Service.Host
|
|||
|
|
> Alkami.MicroServices.QBO.Service.Host
|
|||
|
|
> {code}
|
|||
|
|
> Foreach solution+project that contains a reference to Alkami.MicroServices.Installer.Database|MasterDatabase, convert the installer to Alkami.Installer.Services and add an AlkamiManifest file. You will need to update your produced manifest to include any database related dlls.
|
|||
|
|
|
|||
|
|
h1. Converting Database-Connected Services To An AlkamiManifest Based Installer
|
|||
|
|
|
|||
|
|
> As a followup to the work done on DEV-120482 and lessons learned, we have moved the documentation for further work here to Confluence. Please continue to check this page for improved details as you work your tickets.
|
|||
|
|
|
|||
|
|
h4. Determining that your projects need to be updated
|
|||
|
|
|
|||
|
|
You can determine if your csproj uses the above package with PowerShell by navigating to a solution folder and running the following command
|
|||
|
|
|
|||
|
|
{code}
|
|||
|
|
$csProjs = (Get-ChildItem *.csproj -recurse);
|
|||
|
|
foreach ($csProj in $csProjs) {
|
|||
|
|
$packagesConfigPath = (Join-Path (Split-Path $csProj -Parent) "packages.config");
|
|||
|
|
if (Test-Path $packagesConfigPath) {
|
|||
|
|
$lines = (Get-Content -Path $packagesConfigPath);
|
|||
|
|
foreach ($line in $lines) {
|
|||
|
|
if ($line.IndexOf('Alkami.MicroServices.Installer.') -gt -1) {
|
|||
|
|
Write-Host $csProj
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
{code}
|
|||
|
|
|
|||
|
|
h4. Steps to convert
|
|||
|
|
|
|||
|
|
In each solution for the referenced projects
|
|||
|
|
1. Navigate to the folder in a console window and run {{New-AlkamiManifest -Type Service}}
|
|||
|
|
** Ensure the file looks correct for your service, make any updates as necessary. Consult [https://confluence.alkami.com/display/SRE/Alkami.Installers+-+Manifests+-+Services] for appropriate details. For logic-only installers, remove references to `<migrations>` or `<db_role>` segments. Work with Vanguard for appropriate direction on dependencies when not clear.
|
|||
|
|
** If you only need database access, but do not have migrations, delete the migrations related nodes and use the `<db_role>` tag instead per the manifest.
|
|||
|
|
1. Remove any usages of Alkami.MicroServices.Installer.Logic from your project
|
|||
|
|
** This includes removing the prebuild event from your project
|
|||
|
|
1. Delete any of the following files\folders
|
|||
|
|
** configureInstaller.ps1
|
|||
|
|
** config.ps1
|
|||
|
|
** configOverrides.ps1
|
|||
|
|
1. Add a nuspec file for your project. A template is provided below. (In keeping with the spirit of updating packages, a new target folder name is used for the package/zip-file, per the documentation on Confluence: app)
|
|||
|
|
** Ensure you have a node for your migrations dll under the `<files>` element
|
|||
|
|
** Migrations dlls can be under the same /app root folder as your application or under a folder called `/migrations`, the tooling will look for the first matching filename in the root folder. [https://confluence.alkami.com/display/SRE/Alkami.Installers+-+Package+Structures+-+Services]
|
|||
|
|
1. Create a tools folder for (and only for) and include the chocolateyInstall/chocolateyUninstall PowerShell files (templates included below)
|
|||
|
|
1. Test by creating a new package using your new nuspec file (don't forget to supply your -version flag)
|
|||
|
|
** You should be able to install (nothing changes), uninstall (service is gone) and reinstall successfully and fairly quickly.
|
|||
|
|
** Chocolatey may "error" with red text when trying to stop the service before continuing with the install. This is unfortunately normal due to the way chocolatey works. You can prevent that by stopping the service yourself if you like.
|
|||
|
|
** You may need to close the Services console (services.msc) before testing, as this can on some environments cause issues.
|
|||
|
|
1. Create a PR for your changes
|
|||
|
|
1. Update your bamboo jobs as appropriate
|
|||
|
|
|
|||
|
|
h3. How long should this changeover take?
|
|||
|
|
|
|||
|
|
Manually, from start to end, approximately 15 minutes at the long-side of an estimate. At this point all teams have experience converting these projects, so speed should be fairly fast. There is some tooling at the bottom of this post that should be able to help you automate the changes so you only have to review and test/commit/PR.
|
|||
|
|
|
|||
|
|
h3. Alkami Bitbucket Repositories CSProj list with filters
|
|||
|
|
|
|||
|
|
This spreadsheet has all of the known csproj files as of capture date, according to repository, and may give you additional details that you can use like finding which solution a csproj filename above belongs to. There may be incorrect information which we would be happy to know about so we can resolve.
|
|||
|
|
|
|||
|
|
[https://docs.google.com/spreadsheets/d/1vAz-kV0hSsYz-jEFrHIKhjXDluBmofZB9ORbk6ImUls/edit#gid=94498051]
|
|||
|
|
|
|||
|
|
[Code template here] Supplied Nuspec Template
|
|||
|
|
{code:java}
|
|||
|
|
<?xml version="1.0" encoding="utf-8"?>
|
|||
|
|
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
|
|||
|
|
<metadata>
|
|||
|
|
<id></id>
|
|||
|
|
<version></version>
|
|||
|
|
<title></title>
|
|||
|
|
<authors>Alkami Technology, Inc.</authors>
|
|||
|
|
<owners>Alkami Technology, Inc.</owners>
|
|||
|
|
<projectUrl>https://confluence.alkami.com/display</projectUrl>
|
|||
|
|
<iconUrl>https://www.alkami.com/files/alkamilogo75x75.png</iconUrl>
|
|||
|
|
<licenseUrl>https://www.alkami.com/files/orblicense.html</licenseUrl>
|
|||
|
|
<requireLicenseAcceptance>false</requireLicenseAcceptance>
|
|||
|
|
<description></description>
|
|||
|
|
<copyright></copyright>
|
|||
|
|
<releaseNotes/>
|
|||
|
|
<tags></tags>
|
|||
|
|
<dependencies>
|
|||
|
|
<!-- Please do not list any dependencies here. Those should be in the AlkamiManifest -->
|
|||
|
|
</dependencies>
|
|||
|
|
</metadata>
|
|||
|
|
<files>
|
|||
|
|
<file src="bin\Release\*.*" target="app\" />
|
|||
|
|
<file src="tools\*.ps1" target="tools\" />
|
|||
|
|
</files>
|
|||
|
|
</package>
|
|||
|
|
{code}
|
|||
|
|
|
|||
|
|
[code template here] chocolateyInstall.ps1
|
|||
|
|
{code:java}
|
|||
|
|
[CmdletBinding()] Param()
|
|||
|
|
process {
|
|||
|
|
& C:\ProgramData\Alkami\Installer\Services\install.ps1 $PSScriptRoot;
|
|||
|
|
return;
|
|||
|
|
}
|
|||
|
|
{code}
|
|||
|
|
|
|||
|
|
[code template here] chocolateyUninstall.ps1
|
|||
|
|
{code:java}
|
|||
|
|
[CmdletBinding()] Param()
|
|||
|
|
process {
|
|||
|
|
& C:\ProgramData\Alkami\Installer\Services\uninstall.ps1 $PSScriptRoot;
|
|||
|
|
return;
|
|||
|
|
}
|
|||
|
|
{code}
|
|||
|
|
|
|||
|
|
h3. As a unified function
|
|||
|
|
|
|||
|
|
This function should be runnable from any parent directory. If run at the root of your git folder, will attempt to change ALL projects so found at any depth under. Don't forget to update your sem.ver
|
|||
|
|
|
|||
|
|
> *You will have to do manual modification after this script runs.*
|
|||
|
|
>
|
|||
|
|
> 1. You will have to declare the new nuspec file to include your migration dlls. Due to the variation across the breadth of over 400 Alkami services based projects, there is no easy way to know how to find your specific migrations package for your specific solution.
|
|||
|
|
>
|
|||
|
|
> This nuspec addition is as simple as adding a line to the location of your migrations dll as built.
|
|||
|
|
>
|
|||
|
|
> 2. You will have to modify the created AlkamiManifest to add the line for your migrations configuration.
|
|||
|
|
>
|
|||
|
|
> See notes above under steps to convert for further details and links
|
|||
|
|
|
|||
|
|
|
|||
|
|
h4. Modified code block from the previous installer to now handle all legacy installer types
|
|||
|
|
{code:java}
|
|||
|
|
function ConvertFrom-LegacyMicroserviceInstaller {
|
|||
|
|
<#
|
|||
|
|
.SYNOPSIS
|
|||
|
|
Used to attempt to clean up a solution for installing via the new process
|
|||
|
|
|
|||
|
|
.PARAMETER Path
|
|||
|
|
If you provide a path, will use that as the default working folder to look for csproj under.
|
|||
|
|
Defaults to the current folder and looks for all "possible" csproj to work with.
|
|||
|
|
|
|||
|
|
.PARAMETER Version
|
|||
|
|
If you provide a version, this will be included in the nuspec file directly. Defaults to "$version$"
|
|||
|
|
|
|||
|
|
.PARAMETER UseInstallerFolder
|
|||
|
|
If you pass this flag, your nuspec file will be created under a folder called Installer (this matches some teams expectations regarding Bamboo jobs)
|
|||
|
|
#>
|
|||
|
|
[CmdletBinding()]
|
|||
|
|
param(
|
|||
|
|
[string]$Path,
|
|||
|
|
[string]$Version = '$version$',
|
|||
|
|
[switch]$UseInstallerFolder
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
if ([string]::IsNullOrWhiteSpace($Path)) {
|
|||
|
|
$Path = (Get-Location).Path
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
$chocoInstallFile = @"
|
|||
|
|
process {
|
|||
|
|
& C:\ProgramData\Alkami\Installer\Services\install.ps1 `$PSScriptRoot;
|
|||
|
|
return;
|
|||
|
|
}
|
|||
|
|
"@
|
|||
|
|
$chocoUninstallFile = @"
|
|||
|
|
process {
|
|||
|
|
& C:\ProgramData\Alkami\Installer\Services\uninstall.ps1 `$PSScriptRoot;
|
|||
|
|
return;
|
|||
|
|
}
|
|||
|
|
"@
|
|||
|
|
|
|||
|
|
$csProjs = (Get-ChildItem -Path (Join-Path $Path "*.csproj") -Recurse)
|
|||
|
|
foreach ($csProj in $csProjs) {
|
|||
|
|
try {
|
|||
|
|
$projectFolder = (Split-Path $csProj -Parent)
|
|||
|
|
Write-Verbose "Attempting to work out of [$projectFolder]"
|
|||
|
|
$packagesConfigPath = (Join-Path $projectFolder "packages.config")
|
|||
|
|
if (Test-Path $packagesConfigPath) {
|
|||
|
|
$lines = (Get-Content -Path $packagesConfigPath)
|
|||
|
|
$foundLine = $false
|
|||
|
|
$writeLines = @()
|
|||
|
|
foreach ($line in $lines) {
|
|||
|
|
if ($line.IndexOf('Alkami.MicroServices.Installer.') -gt -1) {
|
|||
|
|
$foundLine = $true
|
|||
|
|
Write-Host $line
|
|||
|
|
} else {
|
|||
|
|
$writeLines += $line
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
if (!$foundLine) {
|
|||
|
|
Write-Verbose "No installer logic include found in this folder [$projectFolder]"
|
|||
|
|
} else {
|
|||
|
|
Write-Host "Processing [$projectFolder] for removing legacy installer logic"
|
|||
|
|
Set-Content -Path $packagesConfigPath -Value $writeLines
|
|||
|
|
|
|||
|
|
New-AlkamiManifest -Type Service -ProjectLocation $csproj -Destination $projectFolder -ErrorAction Ignore
|
|||
|
|
|
|||
|
|
if (Test-Path -Path (Join-Path $projectFolder "Installer")) {
|
|||
|
|
Remove-Item -Path (Join-Path $projectFolder "Installer") -Recurse -Force -ErrorAction SilentlyContinue | Out-Null
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
if (Test-Path -Path (Join-Path $projectFolder "InstallerOverrides")) {
|
|||
|
|
Remove-Item -Path (Join-Path $projectFolder "InstallerOverrides") -Recurse -Force -ErrorAction SilentlyContinue | Out-Null
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
$pathLeader = ""
|
|||
|
|
if ($UseInstallerFolder) {
|
|||
|
|
New-Item -Path (Join-Path $projectFolder "Installer") -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
|
|||
|
|
# This lets the nuspec properly reference things from the installer folder if that's how people have their solution configured.
|
|||
|
|
$pathLeader = "..\"
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
# ensure the manifest we just created is still valid, and then get the packageId so we can use it in a string replace
|
|||
|
|
$manifest = Get-PackageManifest -Path $projectFolder
|
|||
|
|
$packageId = $manifest.general.element
|
|||
|
|
$packageNuspec = @"
|
|||
|
|
<?xml version="1.0" encoding="utf-8"?>
|
|||
|
|
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
|
|||
|
|
<metadata>
|
|||
|
|
<id>$packageId</id>
|
|||
|
|
<version>$version</version>
|
|||
|
|
<title>$packageId</title>
|
|||
|
|
<authors>Alkami Technology, Inc.</authors>
|
|||
|
|
<owners>Alkami Technology, Inc.</owners>
|
|||
|
|
<projectUrl>https://confluence.alkami.com/display</projectUrl>
|
|||
|
|
<iconUrl>https://www.alkami.com/files/alkamilogo75x75.png</iconUrl>
|
|||
|
|
<licenseUrl>https://www.alkami.com/files/orblicense.html</licenseUrl>
|
|||
|
|
<requireLicenseAcceptance>false</requireLicenseAcceptance>
|
|||
|
|
<description>$packageId</description>
|
|||
|
|
<copyright>Alkami Technology, Inc. 2022</copyright>
|
|||
|
|
<releaseNotes/>
|
|||
|
|
<tags>Service</tags>
|
|||
|
|
<dependencies>
|
|||
|
|
<!-- Please do not list any dependencies here. Those should be in the AlkamiManifest -->
|
|||
|
|
</dependencies>
|
|||
|
|
</metadata>
|
|||
|
|
<files>
|
|||
|
|
<file src="$($pathLeader)bin\Release\*.*" target="app" />
|
|||
|
|
<file src="$($pathLeader)tools\*.ps1" target="tools" />
|
|||
|
|
<file src="$($pathLeader)AlkamiManifest.xml" target="" />
|
|||
|
|
</files>
|
|||
|
|
</package>
|
|||
|
|
"@
|
|||
|
|
if ($UseInstallerFolder) {
|
|||
|
|
Set-Content -Path (Join-Path (Join-Path $projectFolder "Installer") "$packageId.nuspec") -Value $packageNuspec -Force
|
|||
|
|
} else {
|
|||
|
|
Set-Content -Path (Join-Path $projectFolder "$packageId.nuspec") -Value $packageNuspec -Force
|
|||
|
|
}
|
|||
|
|
$toolsPath = (Join-Path $projectFolder "tools")
|
|||
|
|
if (!(Test-Path $toolsPath)) {
|
|||
|
|
(New-Item -Path $toolsPath -ItemType Directory -Force -ErrorAction Ignore) | Out-Null
|
|||
|
|
}
|
|||
|
|
Set-Content -Path (Join-Path $toolsPath "chocolateyInstall.ps1") -Value $chocoInstallFile -ErrorAction Ignore
|
|||
|
|
Set-Content -Path (Join-Path $toolsPath "chocolateyUninstall.ps1") -Value $chocoUninstallFile -ErrorAction Ignore
|
|||
|
|
|
|||
|
|
# idc, delete some stuff
|
|||
|
|
Remove-Item -Path (Join-Path $projectFolder "configureInstaller.ps1") -Force -ErrorAction Ignore
|
|||
|
|
Remove-Item -Path (Join-Path $projectFolder "config.ps1") -Force -ErrorAction Ignore
|
|||
|
|
Remove-Item -Path (Join-Path $projectFolder "configOverrides.ps1") -Force -ErrorAction Ignore
|
|||
|
|
|
|||
|
|
#clear prebuild from project
|
|||
|
|
$xml = [xml](Get-Content $csProj)
|
|||
|
|
@($xml.Project.PropertyGroup) | % { if ($null -ne $_.PreBuildEvent) { $_.PreBuildEvent = "" } }
|
|||
|
|
Save-XmlFile -xmlPath $csproj -Xml $xml
|
|||
|
|
}
|
|||
|
|
} else {
|
|||
|
|
Write-Verbose "No packages.config found in [$projectFolder], can not convert. Expected the packages.config to be at the same level as the csproj."
|
|||
|
|
}
|
|||
|
|
} catch {
|
|||
|
|
Write-Host "Error occurred, check logs for failure. Does your packages.config have both database and logic package includes? This function is not yet that smart to fix those."
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
{code}
|