How should a REST source handle different ManifestVersions?

[jantari]

Brief description of your issue

Hi,

when the REST API server ingests manifests, it may find one version of a package in a v1.1.0 manifest and another version of the same program (PackageIdentifier) in a v1.2.0 manifest. This is the case with many manifests in the official winget-pkgs repo, e.g. Lapce.Lapce 0.1.3 vs the next release Lapce.Lapce 0.2.0.

The data from the v1.1.0 manifest is compatible with the v1.1.0 and v1.4.0 API schema.
The data from the v1.2.0 manifest needs a v1.4.0 API schema response.

So when both of these manifests are ingested by the REST server and a GET query is made to the /packageManifests/Lapce.Lapce endpoint, should the API response from the server:

1. Contain mixed API-schema 1.1.0 and 1.4.0 data
   This is what I do currently. This way no information from the newer manifests is lost and the older manifests are returned in an older API-format ensuring they aren't artificially made incompatible with older winget clients. This is what such a mixed-schema response looks like from my REST source, to illustrate:

   ❯ curl -skS https://localhost:8443/packageManifests/Lapce.Lapce | jq -rM
   {
     "Data": {
       "PackageIdentifier": "Lapce.Lapce",
       "Versions": [
         {
           "PackageVersion": "0.2.0",
           "DefaultLocale": {
             "PackageLocale": "en-US",
             "Publisher": "Lapce",
             "PublisherUrl": "https://lapce.dev/",
             "PublisherSupportUrl": "https://github.com/lapce/lapce/issues",
             "PrivacyUrl": "",
             "Author": "Lapce",
             "PackageName": "Lapce",
             "PackageUrl": "https://lapce.dev/",
             "License": "Apache-2.0",
             "LicenseUrl": "https://github.com/lapce/lapce/blob/master/LICENSE",
             "Copyright": "Copyright (C) 2022",
             "CopyrightUrl": "",
             "ShortDescription": "Lightning-fast and Powerful Code Editor written in Rust",
             "Description": "",
             "Moniker": "",
             "Tags": [
               "code",
               "coding",
               "develop",
               "development",
               "editing",
               "editor",
               "programming"
             ],
             "Agreements": null,
             "ReleaseNotes": "Features/Changes\n- #776: Implemented error lens.\n- #780: Add Fedora spec for packaging\n- #787: Add LSP stderr output to log\n- #809: Truncate plugin description with ellipsis when it is too long\n- #805: Add goto type definition support. (Where you can jump to the definition of the type a variable is of)\n- #800: Add alpine dev-container\n- #806: Add next/previous tab commands and keybindings. (Note that this is moving left/right in the tabs, not to the last used tab)\n- c5f1692: Make undo more granular\n- 69e6b83: Fix alpha-depth issue in painting\n- #821: Display documentation of completion items.\n- #825: Add config option for cursor blink interval\n- #836: Use new highlighter query for Julia\n- bcd6ff7: Title update\n- c3bcc13: Full screen fix. Changes Lapce icon to work better on common color schemes.\n- #848: Separate plugin panel into two parts, installed and uninstalled.\n- 722c678: Custom title bar on Linux and palette bar\n- #840: Preserve unsaved files when closing, which are restored when opening Lapce again\n- #880: Add the ability to enable, disable, and remove plugins\n- #887: Don't paint tiny tab drags\n- #905: Add command to toggle inlay hints\n- #913: Add daily nightly build\n- #916: Don't show autocomplete after whitespace keys\n- #927: Make completion/palette allow page-up/down keys\n- #895: Add setting to control palette preview editor's width\n- #937: Don't show autocomplete after backspacing over whitespace\n- #948: Add the ability to resize editor/terminal/various splits.\n- #972: Auto update\n- #979: Add elixir file extensions (so that they can be recognized)\n- #807: Add syntax highlighting for WGSL\n- #903: Add more file extensions (so that they can be recognized)\n- #981: Differentiate between stable/nightly/debug. This puts configuration in different folders, depending on your version!\n- #990: Rework titlebar\n- #1010: Add palette commands to open various Lapce directories\n- #1009: Add about dialog\n- 8687b7d: Add the ability to rename a symbol\n- (Bunch of different commits): Added beginnings of plugin-server-protocol, which allows plugins to use LSP features and more.\n\nBug Fixes\n- #781: Give the plugin panel a scroll bar\n- #789: Check LSP capabilities before sending commands. Fixes most of Julia LSP support.\n- #803: Properly check if a line is after the end of the document in line styles\n- #788: Use FS module\n- #833: Fix crashing on semantic tokens going past last line\n- #831: Fix goto ref on single references\n- 1008682: Reload explorer on file change, just in case our watching code didn't activate.\n- #849: (LSP) Specify the workspace folder of the root uri if available.\n- #850: Set the current directory when starting the LSP\n- #869: Read files lossily, in case they are not UTF8.\n- #884: Use UTF16 encoding for communicating with LSPs. This should fix a bunch of bugs with unicode causing RA or the editor to crash.\n- #914: Fix keybinds on Mac with Command+Shift\n- 474bb00: Fix MacOS Option Key\n- #925: Properly open editors when applying text edits via code actions to editors which aren't focused or open.\n- #930: Prevent crash on save with multiple windows\n- #961: Fix calculation of mouse down position in the problem panel\n- #843: Follow system OS paths better\n- fd74be8: Proxy rewrite! This should make lock-ups not happen, and be easier to add onto.\n- 85e41dc: Find box result count should use the current buffer\n- #996: Pass uname as multiple args to fix WSL\n- #1011: Align settings button size to window controls\n- #1017: Don't fail on missing primary monitor\n- Various other bug fixes",
             "ReleaseNotesUrl": "https://github.com/lapce/lapce/releases/tag/v0.2.0",
             "PurchaseUrl": "",
             "InstallationNotes": "",
             "Documentations": null
           },
           "Channel": "",
           "Locales": [
             {
               "PackageLocale": "zh-CN",
               "Publisher": "Lapce",
               "PublisherUrl": "https://lapce.dev/",
               "PublisherSupportUrl": "https://github.com/lapce/lapce/issues",
               "PrivacyUrl": "",
               "Author": "Lapce",
               "PackageName": "Lapce",
               "PackageUrl": "https://lapce.dev/",
               "License": "Apache-2.0",
               "LicenseUrl": "https://github.com/lapce/lapce/blob/master/LICENSE",
               "Copyright": "Copyright (C) 2022",
               "CopyrightUrl": "",
               "ShortDescription": "使用 Rust 编写的闪电般快速和强大的代码编辑器",
               "Description": "",
               "Tags": [
                 "代码",
                 "开发",
                 "编程",
                 "编辑",
                 "编辑器"
               ],
               "Agreements": null,
               "ReleaseNotes": "",
               "ReleaseNotesUrl": "https://github.com/lapce/lapce/releases/tag/v0.2.0",
               "PurchaseUrl": "",
               "InstallationNotes": "",
               "Documentations": null
             }
           ],
           "Installers": [
             {
               "InstallerIdentifier": "",
               "Architecture": "x64",
               "MinimumOSVersion": "10.0.0.0",
               "Platform": null,
               "InstallerType": "wix",
               "Scope": "machine",
               "InstallerUrl": "https://github.com/lapce/lapce/releases/download/v0.2.0/Lapce-windows.msi",
               "InstallerSha256": "8D97AE939B2C859B4F8E674B8F407B0B3CA4039342C3BE0CC58274AAE859C6DB",
               "InstallModes": null,
               "InstallerSwitches": {},
               "ExpectedReturnCodes": null,
               "Dependencies": {},
               "ProductCode": "{B2A52E5A-3AA8-40DC-A0AC-123EF565F54C}",
               "Markets": {},
               "InstallerAbortsTerminal": false,
               "ReleaseDate": "2022-09-02",
               "InstallLocationRequired": false,
               "RequireExplicitUpgrade": false,
               "InstallationMetadata": {}
             }
           ]
         },
         {
           "PackageVersion": "0.1.3",
           "DefaultLocale": {
             "PackageLocale": "en-US",
             "Publisher": "Lapce",
             "PublisherUrl": "https://lapce.dev/",
             "PublisherSupportUrl": "https://github.com/lapce/lapce/issues",
             "PrivacyUrl": "",
             "Author": "Lapce",
             "PackageName": "Lapce",
             "PackageUrl": "https://lapce.dev/",
             "License": "Apache-2.0",
             "LicenseUrl": "https://github.com/lapce/lapce/blob/master/LICENSE",
             "Copyright": "",
             "CopyrightUrl": "",
             "ShortDescription": "Lightning-fast and Powerful Code Editor written in Rust",
             "Description": "",
             "Moniker": "",
             "Tags": [
               "code",
               "coding",
               "develop",
               "development",
               "editing",
               "editor",
               "programming"
             ],
             "Agreements": null,
             "ReleaseNotes": "Features\n- #705/#711: Changed UI\n  - The left activity bar has been moved to above the file explorer, giving more horizontal space by default\n  - The bottom panel now has the terminal/search/problems buttons attached to it, rather than at the very bottom\n  - Panels can now be dragged to the left/bottom/right as you wish\n  - Buttons on tab bar to move between tabs\n- #706: File icons for CSS, HTML, JavaScript, Lua, Python, TypeScript and ZIP files.\n- #666: Add the ability to search for symbols in the workspace in the palette with the # prefix\n- #653: Give the file explorer sidebar a context menu\n  - Create new files/directories, rename them, and move them to the trash\n- #710: Custom titlebar support (Windows)\n- #715: Inlay Hint support\n- Git:\n  - #691: Commit button added to git panel\n  - #708: Workbench Command to discard changes on the current file/workspace, reverting back to the latest git versions\n  - #654: Workbench Command that initializes a git repository in the current folder\n  - #637: Workbench Command to checkout a branch\n- Nightly releases added to Github\n- Languages:\n  - #690: Highlighting for Bash and Zig\n  - #677: Recognize C/C++ file extensions\n  - #674:\n    - Enables CSS highlighting\n    - Bumps tree-sitter\n    - Bumps tree-sitter-python\n  - #664: Highlighting for Hare\n  - #656: Highlighting for several languages: OCaml, Glimmer, Haxe, HCL, and SCSS\n  - #636: Highlighting for Haskell, CodeQL, and Swift\n  - #625: Highlighting for Elm\n  - #762: Highlighting for Julia\n- #549: Language server improvements that make gopls usable\n- #641: Maintain maximised state across restarts\n- #639: Improve the behavior of Home in non-vim mode\n- #634: Add an executable icon on Windows\n- #633: Add an option deciding whether it should format on save\n- #629: Make Ctrl+Space show autocompletion at the cursor\n- #622: Display the number of results and the current result in the find UI\n- #617: Add setting to toggle relative line numbering in modal mode\n- #728: Add option to use LSP from PATH\n- #735: Make Ctrl+C act like Escape in Modal Mode's Normal Mode\n- #744: Properly handle LSP requests for listening to file saving\n- #748: Add setting for hover font family and font size\n- #738: Add commandline arguments\n- #756: Add language name to status bar\n- #702: Add command to change file syntax language\n- a92b3f2: Make status line items clickable\n- #766: Set window minimal size\n\nBug Fixes\n- #701: Verify that the working directory exists or use the home directory for alacritty\n- 7d6b739 Markdown support fixed\n- #670: Make shift-delete and shift-backspace behave the same as delete and backspace respectively.\n- da1d807: Make the distinction between a window tab and normal tab clear\n- #644: Log to stderr. Fix some warnings with regards to druid's layouting.\n- #643: Use file path comparison which normalises drive letters for code actions\n- #616: Fix unicode crash in palette line-search\n- #755: Make the enter key scroll the terminal down\n- #774: Fixes panic on empty active tab\n- Many other fixes\n\nOther\n- #667: Add installation information for various packaging systems for various platforms\n- #645: Cache various steps of the CI workflow to improve the speed\n- #615: Fix clippy warnings; Add clippy to CI; fix a small bug",
             "ReleaseNotesUrl": "https://github.com/lapce/lapce/releases/tag/v0.1.3"
           },
           "Channel": "",
           "Locales": [
             {
               "PackageLocale": "zh-CN",
               "Publisher": "Lapce",
               "PublisherUrl": "https://lapce.dev/",
               "PublisherSupportUrl": "https://github.com/lapce/lapce/issues",
               "PrivacyUrl": "",
               "Author": "Lapce",
               "PackageName": "Lapce",
               "PackageUrl": "https://lapce.dev/",
               "License": "Apache-2.0",
               "LicenseUrl": "https://github.com/lapce/lapce/blob/master/LICENSE",
               "Copyright": "",
               "CopyrightUrl": "",
               "ShortDescription": "使用 Rust 编写的闪电般快速和强大的代码编辑器",
               "Description": "",
               "Tags": [
                 "代码",
                 "开发",
                 "编程",
                 "编辑",
                 "编辑器"
               ],
               "Agreements": null,
               "ReleaseNotes": "",
               "ReleaseNotesUrl": "https://github.com/lapce/lapce/releases/tag/v0.1.3"
             }
           ],
           "Installers": [
             {
               "InstallerIdentifier": "",
               "Architecture": "x64",
               "MinimumOSVersion": "10.0.0.0",
               "Platform": null,
               "InstallerType": "wix",
               "Scope": "machine",
               "InstallerUrl": "https://github.com/lapce/lapce/releases/download/v0.1.3/Lapce-windows.msi",
               "InstallerSha256": "79CCB2169000F787EB4756B66F1617E9194EB9D9B3EA5163A795AA84E287F819",
               "InstallModes": null,
               "InstallerSwitches": {},
               "ExpectedReturnCodes": null,
               "UpgradeBehavior": "install",
               "Dependencies": {
                 "PackageDependencies": [
                   {
                     "PackageIdentifier": "Microsoft.VCRedist.2015+.x64",
                     "MinimumVersion": ""
                   }
                 ]
               },
               "ProductCode": "{598C4A7D-68E8-43AF-A408-5D84ED273FBE}",
               "Markets": {},
               "InstallerAbortsTerminal": false,
               "ReleaseDate": "2022-07-14",
               "InstallLocationRequired": false,
               "RequireExplicitUpgrade": false,
               "UnsupportedOSArchitectures": [],
               "AppsAndFeaturesEntries": []
             }
           ]
         }
       ]
     },
     "RequiredQueryParameters": [],
     "UnsupportedQueryParameters": []
   }
   

   As you can see, Lapce.Lapce 0.1.3 is returned in 1.1.0 API schema (UnsupportedOSArchitectures was not nullable so it is included as an empty array) while Lapce.Lapce 0.2.0 is simultaneously returned as a 1.4.0 API schema (e.g. Documentations is included in the data).

2. Downgrade the manifest 1.2.0 information to fit in the 1.1.0 API schema, possibly losing information such as Documentations, ReturnResponseUrl and UpgradeCodes that aren't representable in the 1.1.0 API
   This loses all the benefit and features enabled by the newer manifest and API schemas, so this approach is problematic.

3. Return the manifest 1.1.0 information in the 1.4.0 API schema format, either omitting (where allowed by the schema) values or setting them to defaults
   This loses compatibility with older winget clients unnecesarily, as they would have been able to understand a 1.1.0 response but are now left in the dark as their theoretically compatible 1.1.0 manifest packages have been returned as 1.4.0 schema API responses for no good reason or benefit. When a future API schema removes fields this becomes equivalent to downgrading the schema as the older manifests cannot be fully represented by the newer API schemas, so this approach is problematic.

Another related note, I can see in the schema that winget could send an APIVersion query parameter with (almost?) every API call. However I have never had this parameter set / sent to me in practice. If the winget CLI would communicate to the server which API schema versions it supports it would be easier to reason about which manifests / packages are compatible with that winget client / that version of the API it supports and return only compatible subsets of packages from the servers "inventory". But, for that to be possible, winget CLI would have to actually set the APIVersion in its requests.

How is a REST source supposed to operate to ensure compatibility between the ManifestVersions of packages it has and the different supported API versions of winget clients?

Thanks!