Proposal: Additional Metadata Attributes

[Redth]

Allow the inclusion of additional metadata properties in package authoring and allow them to be used in search queries.

The motivation for this is to create associations of NuGet packages which bind and/or redistribute platform native libraries on other platforms and package management systems.

[erdembayar]

@joelverhagen @JonDouglas
Fyi, related to nuget.org too.

[joelverhagen]

Would it be possible to create well-known tags for this purpose? We've extension points in the past, based on tag. Here's a prototype I tried on our DEV environment:
https://dev.nugettest.org/packages?q=Tags%3A%22attr_fruit%3Alemon%22

Miraculously, our quotes actually work properly here 😂.

Prior art is "AzureSiteExtension" used for finding Azure site extension packages, before the package type filtering was enabled:
https://www.nuget.org/packages?q=Tags%3A%22AzureSiteExtension%22

[Redth]

Well-known tags could potentially work... I suppose there's not really any more enforced convention with arbitrary metadata key/value pairs... Though it doesn't seem like there would be currently a way to search with multiple tag combinations with "AND" (eg: Tags:"attr_fruit:lemon" Tags:"ArtifactId:NONE" returns results matching just one of the tags, not results only matching both.

[joelverhagen]

Yeah, the AND/OR combination on NuGet.org search is ... not great :)

For non-field-scoped terms foo bar it performs an AND. For multiple field-scoped terms tags:foo tags:bar it performs an OR. For a mixture, at least one of the field-scoped terms per field name must exist in the doc.
https://www.nuget.org/packages?q=owner%3Amicrosoft+owner%3Ajver+tags%3Aentity+tags%3Afoo+design

I think a reasonable step here could be to change the interaction of field-scoped terms to "AND" to unblock this scenario. I think it would be a net win for general usage of field-scoped terms anyway since it would align with the non-field scoped term behavior.

The history here is that we have invested heavily in relevance on non-field scoped queries since they are the 99% case. We have not done the same investment for field-scoped queries or other advanced syntax like + (not supported), - (not supported), " (acts weird).

[nkolev92]

How valuable/important would the combination of atributes/tags be?

How many scenarios would a single attribute/tag solve?

[joelverhagen]

Stuff inside the q (search text) parameter is not spec'd at the protocol level. So different package source implementations will interpret this differently. AND/OR logic, ranking, quote behavior, supported field-scoped terms, etc. These are all package source specific. In general the q property is for search relevance and less about strict package filtering. It's certainly a grey area since it is unspec'd but I think a safer approach is to introduce a new query parameter for these attribute filters.

[joelverhagen]

If someone wants to filter by multiple key-value pairs, is that possible? AND or OR behavior?

[Redth]

Yeah I hadn't realized, there does not seem to be a way to "AND" query terms... This would definitely be helpful, though I guess narrowing search results to a potentially reasonable number (ie: GroupId, or having a concatenated MavenId field) and inspecting the details of the results might be reasonable.

[moljac]

nuget.org search is less concern (for me)

https://www.nuget.org/packages?q=artifact

Huge help would be metadata + API for maintaining 450+ artifacts.

[joelverhagen]

It is possible for NuGet.org or a community member to build their own index based on NuGet.org packages using the V3 catalog. There is a guide on using the V3 catalog here: https://learn.microsoft.com/en-us/nuget/guides/api/query-for-all-published-packages

So the "productionized" version of this map would be to write catalog reader that looks at each published package, checks if there is a cgmanifest.json, then add it to an index. Surface the index on an independent web service. This allows custom projects/views of NuGet.org without the need to block on official service or client support.

[Redth]

Oh interesting, I didn't realize this was available... We don't currently publish the manifest in the packages (I don't think anyway), but looking at this sort of approach might mean we could create our own conventions - the problem is that unless they are 'officially' supported, conventions can be hard to gain traction with.

[joelverhagen]

Just wondering, is this spec suggesting the same model as Maven? I'm wondering if extensions or next steps for the Maven model ever came up. For example: what about values with non-string types allowing range queries ("os_version:10" and querying "os_version>=10") or allowing multiple attributes with the same key but different values.

[Redth]

I'm not sure I fully understand the question here, could you elaborate? It might be nice to support additional types and query operators if that's the basic question? I guess if something is being considered, may as well consider all possible useful cases.

[joelverhagen]

Apologies for the confusion.

From your spec, it sounds like Maven already has a feature like what you're suggesting. Given that, can we learn from any growing pains their ecosystem had? For example, did they originally ship with the string KVP model with unique keys, then run into problems that necessitated a richer data model?

Said another way, if we can learn from another ecosystem's implementation in this area, we can maybe skip some intermediate steps or painful migrations. Or we can know that what we're proposing here is actually enough.

I'm not clued into the Maven ecosystem so I can't provide that perspective.

As a side note, if there are "prior art" design spec/docs about this feature in other ecosystems that would be cool to link in here.

[Redth]

Oh sorry, I'm not really that familiar with how they built up their model.

Having the ability to query on the version as a version number would be useful in an "AND" query scenario, I had just considered the MVP implementation of this not needing to do that since the result set matching only the GroupId and ArtifactId would presumably be small enough to iterate over versions of the results (also if it wasn't clear, different NuGet package versions would potentially have different attr_MavenVersion:1.2.3 values). The one potential gotcha here is that Maven's versioning rules might be a bit different than NuGet (though I think semver is adopted there too).

One other consideration though is the maven version might be used to assert if it satisfies a Maven version range - again, similar to NuGet's version range semantics, but not necessarily identical in rules/implementation. For the binding helpers project/experiment I linked in the proposal, this is part of the process, so in this case the matching GroupId/ArtifactId results would still need to be iterated over, asserting each version's maven range compatibility. Long way to say that there's maybe too many operators to consider for querying by version to make the effort of adding some simple >= particularly useful in value? This is just one example though, and maybe that would be valuable for other scenarios.

[moljac]

Sorry for coming late to the show.

I maintain:

• AndroidX (AX)

  https://github.com/xamarin/AndroidX

• GooglePlayServices, Firebase, MLKit (GPS-FB-MLKit)

  https://github.com/xamarin/GooglePlayServicesComponents

And when I get some air I work on "bindings improvements" which should improve productivity and more.

So, I will express my opinion only for Android (.NET for Android - formerly Xamarin.Android), though
IMO this should be extended to .NET for iOS and maybe other platforms.

"Bindings improvements" include

• discoverability/identification (both Maven and NuGet)

• dependency graphs (both Maven and NuGet)

  Here I work on porting my graph theory library from c++ to c# and thanks to Generic Math it is
  a lot easier nowdays.

  https://devblogs.microsoft.com/dotnet/dotnet-7-generic-math/
  https://devblogs.microsoft.com/dotnet/preview-features-in-net-6-generic-math/

  I need formal methods to build graph/trees, find leaves (all nodes) and detect dependency cycles
  for both Maven and NuGet.

I am already using some of the utilities in our repos for

• building cgmanifest.json

  https://github.com/xamarin/AndroidX/blob/main/cgmanifest.json

  https://github.com/xamarin/GooglePlayServicesComponents/blob/main/cgmanifest.json

• detecting dates when NuGet packages were published

Up to recently we have added Maven fully qualified metadata for artifact in 2 forms:

• artifact=androidx.compose.material:material-ripple
• artifact_versioned=androidx.compose.material:material-ripple:1.0.5

to nuget fields

• Description
• Summary (sometimes)
• Tags

Visible here:

https://api.nuget.org/v3/registration5-gz-semver2/xamarin.androidx.compose.material.ripple/index.json

This was OK and I am able to use server side NuGet protocol (HttpClient + JSON/XML parsing) to increase productivity of maintenance on both of repos.

Last updates .NET for Android team decided to keep this information only in Tags node.

• identifying binaries used (either distributed or downloaded by the package during the build)

• dependency identification

  • type
    • maven
    • native
  • identity
  • version

With this there would be 1:1 mapping from NuGet package (versioned) to Maven/Native package (versioned)
This would help maintainers with

1. keeping track of published (bound Maven or native libraries),

   Getting data for latest nuget package and mapping it to maven fully qualified versioned id
   would ease discoverability what is to be updated.

2. updates and

   see 1.

3. troubleshooting

   Primarily checking dependency graphs

   • for duplicate transitive dependencies (possibly with different versions)

4. security checks (component governance)

5. curation (currated package publishing)

   With lowering the bar for bindings via "bindings improvements" it is to be expected to have
   flood of bindings packages.
   NuGet publishing proces could add step to verify if given Maven/Native artifact is already
   published in some other NuGet package.

• Maven

  • project

    <ItemGroup>
        <PackageAttribute Include="maven.GroupId" Value="androidx.activity" />
        <PackageAttribute Include="maven.ArtifactId" Value="activity" />
        <PackageAttribute Include="maven.VersionId" Value="1.6.0" />
    </ItemGroup>

    NOTE: this could be derived from curernt (and future) .NET for Android (Xamarin.Android)
    BuildActions for binding artifacts (Embedd)

  • nuspec

    <!--
        ... snip
    -->
    <package>
        <metadata>
            <attributes>
            <attribute key="maven.GroupId">androidx.activity</attribute>
            <attribute key="maven.ArtifactId">activity</attribute>
            <attribute key="maven.Version">1.6.0</attribute>
            </attributes>
        </metadata>
    <!--
        ... snip
    -->
    </package>

• Native

  • project (packaging)

    <ItemGroup>
        <AndroidNativeLibrary Include="path/to/libfoo.so">
            <Abi>armeabi</Abi>
        </AndroidNativeLibrary>
    </ItemGroup>

  • nuspec

    <package>
        <metadata>
            <attributes>
            <attribute key="native.LibraryName">libfoo</attribute>
            <attribute key="native.Version">1.6.0</attribute>
            </attributes>
        </metadata>
    </package>

[nkolev92]

Do we have any considerations beyong discoverability?
Maybe something specific to management some of these related packages within your project or is that not a big concern?

[moljac]

Do we have any considerations beyong discoverability?

1:1 mapping of native (maven or native lib) to nuget would make

• security checks easier
• curation (optional) easier

Maybe something specific to management some of these related packages within your project or is that not a big concern?

We do that, but formal/standardized/central method would help, both us and (IMO) nuget team.

[moljac]

Now thinking a bit deeper 1:1 is oversimplification. In cross platform scenario there will be artifact per platform (Android, iOS) and sometimes multiple artifacts per platform.

[ghost]

This PR has been automatically marked as stale because it has no activity for 30 days. It will be closed if no further activity occurs within another 15 days of this comment. If it is closed, you may reopen it anytime when you're ready again, as long as you don't delete the branch.

[ghost]

This PR has been automatically marked as stale because it has no activity for 30 days. It will be closed if no further activity occurs within another 15 days of this comment, unless it has a "Status:Do not auto close" label. If it is closed, you may reopen it anytime when you're ready again, as long as you don't delete the branch.

[ghost]

This PR has been automatically marked as stale because it has no activity for 30 days. It will be closed if no further activity occurs within another 15 days of this comment, unless it has a "Status:Do not auto close" label. If it is closed, you may reopen it anytime when you're ready again, as long as you don't delete the branch.

[donnie-msft]

Hi, we have removed our "proposed" folder, so please move this proposal to the "accepted" folder.
See the update here, and let me know if you have questions/concerns: https://github.com/NuGet/Home/blob/b18b5cc1507df04ea9785f8ba613b1ceb2ad93ea/meta/README.md#what-happens-to-a-proposal
Thanks!

[ghost]

This PR has been automatically marked as stale because it has no activity for 30 days. It will be closed if no further activity occurs within another 15 days of this comment, unless it has a "Status:Do not auto close" label. If it is closed, you may reopen it anytime when you're ready again, as long as you don't delete the branch.