Skip to content

Commit

Permalink
Changes to Release pipeline to include GarnetServer nuget and other c…
Browse files Browse the repository at this point in the history
…hanges to the process (#809)

* First test run for GarnetServer in release pipeline

* Disabled Zip files step

* Disable the task copying zipped files

* Debug message cleaned up and everything enabled EXCEPT for the final Garnet Release step and actual Push to Nuget.org

* Added -p:PackageVersion=$(name) to release yml and removed     <Version>1.0.38</Version>  from GarnetServer.csproj

* Trying nuget pack on Garnet Library

* Trying PACKAGE_VERSION variable

* Adding extracting version number from version.json

* Trying to use Build.BuildNumber instead of pulling from version.json

* Moved version.json to root (adjusted extract_version.ps1).

Have debug codestill in too.

* Updated to use Directory.Build.props

* Cleaned up debug messages and have all version implemented. Still have Push to Nuget and Push to GH Release disabled. Also Garnet Library pack uses .nuspec instead of dotnet pack

* Fixed GarnetServer to use GetVersion() to set the version based on the prefix in Directory.Build.props.

* Fixed bug in GetVersion

* Setting version variable which also set Build Name.

* Move setting build number to first step

* Fixed bug with setting version

* Fixed bug as the set version had to be after check out.

* Trying to set (build.updatebuildnumber) to version to see if it works in ADO pipeline output

* Trying to build without setting variable called "version" and instead seeing if it works when the BuildNumber is updated

* Trying Build.BuildNumber instead of the $version env var

* Packing Microsoft.Garnet using DotNet Pack and also moved the shared Nuget meta data to Directory.Builds.props.

* Remove Garnet.nuspec as it is not needed any more.

* Removed Garnet.nuspec from sln

* Add the dotnet pack line to release yml for the Garnet Library

* Fixed issue in nuget pack Garnet Library

* Left behind a hardcode path

* Added --no-build --no-restore to the dotnet pack so doesn't rebuild

* Removed Nuget install / authenticate and also trying to get README.md into library nuget package

* Fixed README.md part of the nuget package

* Trying to fix Readme package for Nuget

* Debug code left in ... removed it

* Works locally but not when ran in ADO. Trying to set WorkingDirectory to see if that helps.

* Added SymbolPackageFormat=snupkg  for the nuget package symbols

* Missing Title tag in the Directory.Build.props

* Set the version to 1.0.40 and enabled the GitHub Release and Nuget Push tasks.

* Changing case of readme.md to README.md

* All the semi minor changes (spacing etc) from the PR comments

* Added Version.props as the single place to update version instead of Directory.Build.props.  Push to GH Release and Push to Nuget ARE DISABLED for this check in to be able to test first

* Trying to get .pdb out of nuget packages other than the symbols package

* Another try at not having .pdb files in the nuget package

* Tyring include symbols in the csproj and the exclude pdb in Directory.Build.props

* some changes

* update

* Fixed GarnetServer to include the proper files in the Nuget package. Push to Nuget and Push to Github release are NOT enabled until we test through in actual pipeline

* Set library to NOT have pdb in nupkg and have server set up to have pdbs in nupkg since it is a dotnet tool

* Updated version to 41 and got the release pipeline all ready to merge where it WILL PUSH TO NUGET.ORG and create a GH Release.  This release has no pdb files in the nupkg package and only Garnet.Host.pdb in the .snupkg file.

* Changed from pdbs in .snupkg to embedding debug symbols into DLL.

* One final test before setting it to be merged

* Version bumped to 42.  Create Git Hub Release is now ENABLED and the Push to Nuget.org is now ENABLED.

* Moved IncludeSymbols and IncludeSource to Directory.Build.Props and set EmbedUntrackedSources to truefor Garnet.host

* Put IncludeSymbols, IncludeSourceand DebutType into same PropertyGroup since they are closerly related

---------

Co-authored-by: Badrish Chandramouli <[email protected]>
  • Loading branch information
darrenge and badrishc authored Nov 22, 2024
1 parent 7aa311f commit c84c8c7
Show file tree
Hide file tree
Showing 11 changed files with 129 additions and 144 deletions.
83 changes: 39 additions & 44 deletions .azure/pipelines/azure-pipelines-external-release.yml
Original file line number Diff line number Diff line change
@@ -1,22 +1,20 @@
######################################
# NOTE: Before running this pipeline to generate a new nuget package, update the version string in two places
# 1) update the name: string below (line 7) -- this is the version for the nuget package (e.g. 1.0.0)
# 2) update \libs\host\GarnetServer.cs readonly string version (~line 32) -- NOTE - these two values need to be the same
# 3) update the version in GarnetServer.csproj (~line 8)
# NOTE: Before running this pipeline to generate a the GitHub Release and new nuget packages, update the VersionPrefix value in Version.props file
# NOTE: When Version.props file is modified, this pipeline will automatically get triggered
######################################
name: 1.0.41
trigger:
branches:
include:
- main
paths:
include:
- .azure/pipelines/azure-pipelines-external-release.yml
- Version.props
resources:
repositories:
- repository: self
type: git
ref: refs/heads/main

jobs:
- job: Phase_1
displayName: Assessment
Expand All @@ -28,6 +26,13 @@ jobs:
clean: False
submodules: recursive
persistCredentials: True

- task: PowerShell@2
displayName: 'Extract version number from Version.props'
inputs:
filePath: .azure/pipelines/extract_version.ps1
workingDirectory: .azure/pipelines

- task: UseDotNet@2
displayName: Use .NET Core sdk 6.0.x - needed for code signing
inputs:
Expand All @@ -36,21 +41,11 @@ jobs:
displayName: Use .NET Core sdk 8.0.x
inputs:
version: 8.0.x
- task: NuGetToolInstaller@1
displayName: Nuget Tool Installer
inputs:
versionspec: '*'
checkLatest: true
- task: NuGetAuthenticate@1
displayName: NuGet Authenticate
- task: NuGetAuthenticate@1
displayName: 'NuGet Authenticate'

- task: DotNetCoreCLI@2
displayName: dotnet build
enabled: True
inputs:
projects: '**/Garnet.*.csproj'
projects: '**/Garnet*.csproj'
arguments: -c Release

- task: PowerShell@2
Expand All @@ -62,7 +57,6 @@ jobs:

- task: EsrpCodeSigning@5
displayName: Sign the binaries for nuget and zipped files
enabled: True
inputs:
ConnectedServiceName: 'GarnetCodeSigningOneCert'
AppRegistrationClientId: '19bbb452-ce7a-4b3d-bdc4-42a0090e797b'
Expand All @@ -71,7 +65,7 @@ jobs:
AuthCertName: 'Garnet-CodeSign-AuthCert'
AuthSignCertName: 'Garnet-CodeSigning-SigningCertificate'
FolderPath: .
Pattern: Garnet.server.dll,Garnet.client.dll,Garnet.common.dll,Garnet.cluster.dll,Garnet.host.dll,HdrHistogram.dll,Tsavorite.core.dll,Tsavorite.devices.AzureStorageDevice.dll,native_device.dll,GarnetServer.exe
Pattern: Garnet*.dll,Tsavorite*.dll,Garnet*.exe,HdrHistogram.dll,native_device.dll,*Lua.dll
signConfigType: inlineSignParams
inlineOperation: >-
[
Expand Down Expand Up @@ -99,31 +93,20 @@ jobs:
SessionTimeout: 20
VerboseLogin: true

- task: NuGetCommand@2
displayName: nuget pack Garnet
enabled: True
inputs:
command: custom
arguments: pack Garnet.nuspec -OutputDirectory $(Build.ArtifactStagingDirectory) -Symbols -SymbolPackageFormat snupkg -version $(Build.BuildNumber) -Verbosity Detailed

# Do after Nuget Pack so not part of Nuget Pack
- task: PowerShell@2
displayName: 'Zip the GarnetServer binaries'
- task: CmdLine@2
displayName: nuget pack Garnet Library
inputs:
filePath: .azure/pipelines/createbinaries.ps1
arguments: 2
workingDirectory: .azure/pipelines
workingDirectory: $(System.DefaultWorkingDirectory)
script: 'dotnet pack --no-build --no-restore --output $(Build.ArtifactStagingDirectory) -p:PackageVersion=$(Build.BuildNumber) /p:Configuration=Release libs/host/Garnet.host.csproj'

- task: CopyFiles@2
displayName: 'Copy Zipped Files to Artifacts dir: $(Build.artifactstagingdirectory)'
- task: CmdLine@2
displayName: nuget pack Garnet Server
inputs:
Contents: '**'
SourceFolder: '$(Build.SourcesDirectory)/main/GarnetServer/bin/Release/net8.0/publish/output'
TargetFolder: $(build.artifactstagingdirectory)
workingDirectory: $(System.DefaultWorkingDirectory)
script: 'dotnet pack --no-build --no-restore --output $(Build.ArtifactStagingDirectory) -p:PackageVersion=$(Build.BuildNumber) /p:Configuration=Release main/GarnetServer/GarnetServer.csproj'

- task: EsrpCodeSigning@5
displayName: Sign the NuGet Package
enabled: true
displayName: Sign the NuGet Packages
inputs:
ConnectedServiceName: 'GarnetCodeSigningOneCert'
AppRegistrationClientId: '19bbb452-ce7a-4b3d-bdc4-42a0090e797b'
Expand All @@ -132,7 +115,7 @@ jobs:
AuthCertName: 'Garnet-CodeSign-AuthCert'
AuthSignCertName: 'Garnet-CodeSigning-SigningCertificate'
FolderPath: $(Build.ArtifactStagingDirectory)
Pattern: Microsoft.Garnet.*.nupkg
Pattern: Microsoft.Garnet.*.nupkg, garnet-server.*.nupkg
signConfigType: inlineSignParams
inlineOperation: >-
[
Expand Down Expand Up @@ -160,13 +143,26 @@ jobs:
SessionTimeout: 20
VerboseLogin: true

# Do after Nuget Pack so not part of Nuget Pack
- task: PowerShell@2
displayName: 'Zip the GarnetServer binaries'
inputs:
filePath: .azure/pipelines/createbinaries.ps1
arguments: 2
workingDirectory: .azure/pipelines

- task: CopyFiles@2
displayName: 'Copy Zipped Files to Artifacts dir: $(Build.artifactstagingdirectory)'
inputs:
Contents: '**'
SourceFolder: '$(Build.SourcesDirectory)/main/GarnetServer/bin/Release/net8.0/publish/output'
TargetFolder: $(build.artifactstagingdirectory)

- task: PublishBuildArtifacts@1
displayName: 'Publish Artifact: drop'
enabled: True

- task: GitHubRelease@1
displayName: 'Create the GitHub release'
enabled: True
inputs:
action: 'create'
gitHubConnection: ADO_to_Github_ServiceConnection
Expand All @@ -190,8 +186,7 @@ jobs:
$(Build.ArtifactStagingDirectory)/*.7z
- task: NuGetCommand@2
displayName: 'Push to NuGet.org'
enabled: True
displayName: 'Push both packages to NuGet.org'
inputs:
command: push
packagesToPush: '$(Build.ArtifactStagingDirectory)/**/*.nupkg'
Expand Down
11 changes: 11 additions & 0 deletions .azure/pipelines/extract_version.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<#$f
.DESCRIPTION
This script pulls the version number from Version.props which is located in the root directory.
It then assigns the value to the BuildNumber which can be used in pipeline yml files
#>

$propsFile = Resolve-Path -Path "$PSScriptRoot/../../Version.props"
[xml]$xml = Get-Content -Path $propsFile
$version = $xml.Project.PropertyGroup.VersionPrefix
Write-Host "##vso[build.updatebuildnumber]$version"
47 changes: 46 additions & 1 deletion Directory.Build.props
Original file line number Diff line number Diff line change
@@ -1,14 +1,59 @@
<Project>

<PropertyGroup Label="Targets">
<PropertyGroup Label="Targets">
<TargetFrameworks>net8.0</TargetFrameworks>
</PropertyGroup>

<!-- Versioning property for builds and packages -->
<Import Project="Version.props" />

<PropertyGroup Label="Configuration">
<LangVersion>latest</LangVersion>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
</PropertyGroup>

<PropertyGroup Condition="'$(TF_BUILD)' == 'true' or '$(GITHUB_ACTIONS)' == 'true'">
<ContinuousIntegrationBuild>true</ContinuousIntegrationBuild>
</PropertyGroup>

<!-- Embeds debug symbols in the DLLs instead of separate pdb files -->
<PropertyGroup>
<DebugType>embedded</DebugType>
<IncludeSymbols>false</IncludeSymbols>
<IncludeSource>false</IncludeSource>
</PropertyGroup>

<ItemGroup>
<EmbeddedFiles Include="$(GeneratedAssemblyInfoFile)"/>
<EmbeddedFiles Include="$(TargetFrameworkMonikerAssemblyAttributesPath)"/>
</ItemGroup>

<ItemGroup>
<Content Include="../../website/static/img/garnet-logo-diamond.png" Pack="true" Visible="false" PackagePath="/" />
</ItemGroup>

<!-- Common properties for nuget packages -->
<PropertyGroup>
<Product>Garnet</Product>
<Authors>Microsoft</Authors>
<Company>Microsoft</Company>
<RepositoryType>git</RepositoryType>
<RepositoryUrl>https://github.com/microsoft/garnet.git</RepositoryUrl>
<PackageProjectUrl>https://microsoft.github.io/garnet</PackageProjectUrl>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageRequireLicenseAcceptance>True</PackageRequireLicenseAcceptance>
<Description>
Garnet is a remote cache-store from Microsoft Research, that offers strong performance (throughput and latency),
scalability, storage, recovery, cluster sharding, key migration, and replication features. Garnet uses the Redis RESP wire
protocol and can work with existing Redis clients.
</Description>
<PackageIcon>garnet-logo-diamond.png</PackageIcon>
<PackageReleaseNotes>See https://github.com/microsoft/garnet for details.</PackageReleaseNotes>
<Copyright>© Microsoft Corporation. All rights reserved.</Copyright>
<language>en-US</language>
<PackageTags>Garnet key-value store cache dictionary hashtable concurrent persistent remote cluster Redis RESP</PackageTags>
</PropertyGroup>

</Project>
67 changes: 0 additions & 67 deletions Garnet.nuspec

This file was deleted.

7 changes: 4 additions & 3 deletions Garnet.sln
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
Dockerfile.chiseled = Dockerfile.chiseled
Dockerfile.nanoserver = Dockerfile.nanoserver
Dockerfile.ubuntu = Dockerfile.ubuntu
Garnet.nuspec = Garnet.nuspec
.azure\pipelines\extract_version.ps1 = .azure\pipelines\extract_version.ps1
Version.props = Version.props
EndProjectSection
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Bitmap", "playground\Bitmap\Bitmap.csproj", "{B9FA7D45-6DAE-4D56-AD7E-BB6C987A58C5}"
Expand Down Expand Up @@ -98,15 +99,15 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SampleModule", "playground\
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "GarnetJSON", "playground\GarnetJSON\GarnetJSON.csproj", "{2C8F1F5D-31E5-4D00-A46E-F3B1D9BC098F}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MigrateBench", "playground\MigrateBench\MigrateBench.csproj", "{6B66B394-E410-4B61-9A5A-1595FF6F5E08}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "MigrateBench", "playground\MigrateBench\MigrateBench.csproj", "{6B66B394-E410-4B61-9A5A-1595FF6F5E08}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "hosting", "hosting", "{01823EA4-4446-4D66-B268-DFEE55951964}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Windows", "Windows", "{697766CD-2046-46D9-958A-0FD3B46C98D4}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Garnet.worker", "hosting\Windows\Garnet.worker\Garnet.worker.csproj", "{DF2DD03E-87EE-482A-9FBA-6C8FBC23BDC5}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Garnet.resources", "libs\resources\Garnet.resources.csproj", "{A48412B4-FD60-467E-A5D9-F155CAB4F907}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Garnet.resources", "libs\resources\Garnet.resources.csproj", "{A48412B4-FD60-467E-A5D9-F155CAB4F907}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Expand Down
6 changes: 6 additions & 0 deletions Version.props
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
<Project>
<!-- Versioning property for builds and packages -->
<PropertyGroup>
<VersionPrefix>1.0.42</VersionPrefix>
</PropertyGroup>
</Project>
12 changes: 11 additions & 1 deletion libs/host/Garnet.host.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
<AssemblyOriginatorKeyFile>../../Garnet.snk</AssemblyOriginatorKeyFile>
<DelaySign>false</DelaySign>
<EmbedUntrackedSources>true</EmbedUntrackedSources>
<PackageId>Microsoft.Garnet</PackageId>
<PackageReadmeFile>README.md</PackageReadmeFile>
</PropertyGroup>

<ItemGroup>
Expand All @@ -29,4 +31,12 @@
<InternalsVisibleTo Include="Garnet.test" Key="0024000004800000940000000602000000240000525341310004000001000100011b1661238d3d3c76232193c8aa2de8c05b8930d6dfe8cd88797a8f5624fdf14a1643141f31da05c0f67961b0e3a64c7120001d2f8579f01ac788b0ff545790d44854abe02f42bfe36a056166a75c6a694db8c5b6609cff8a2dbb429855a1d9f79d4d8ec3e145c74bfdd903274b7344beea93eab86b422652f8dd8eecf530d2" />
</ItemGroup>

</Project>
<ItemGroup>
<Content Include="..\host\bin\Release\net8.0\*.dll" Pack="true" PackagePath="lib\net8.0" />
<Content Include="..\host\bin\Release\net8.0\*.xml" Pack="true" PackagePath="lib\net8.0" />
<Content Include="..\host\bin\Release\net8.0\runtimes\**\*.so" Pack="true" PackagePath="runtimes" />
<Content Include="..\host\bin\Release\net8.0\runtimes\**\*.dll" Pack="true" PackagePath="runtimes" />
<None Include="..\..\README.md" Pack="true" PackagePath="/"/>
</ItemGroup>

</Project>
9 changes: 7 additions & 2 deletions libs/host/GarnetServer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text;
using System.Threading;
using Garnet.cluster;
Expand All @@ -28,8 +29,12 @@ namespace Garnet
/// </summary>
public class GarnetServer : IDisposable
{
// IMPORTANT: Keep the version in sync with .azure\pipelines\azure-pipelines-external-release.yml line ~7 and GarnetServer.csproj line ~8.
readonly string version = "1.0.41";
static readonly string version = GetVersion();
static string GetVersion()
{
var Version = Assembly.GetExecutingAssembly().GetName().Version;
return $"{Version.Major}.{Version.Minor}.{Version.Build}";
}

internal GarnetProvider Provider;

Expand Down
Loading

0 comments on commit c84c8c7

Please sign in to comment.