Improving GitHub Issue Management for CNCF projects using LLMs

[rajaskakodkar]

Currently, every GitHub project and specially the ones that come under CNCF use independent processes for issue triage, bot replies and so on. At a broad level, the following patterns arise where project maintainers have to handle chop wood carry water tasks.

• Detecting duplicate issues
• Identifying potential fixes/resolution for the issues which are trivial
• Setting up machinery for bot replies for community engagement

With the use of AI models, running on cloud native infra, the above tasks can be automated using

• Using generative models(LLMs) to provide actionable outputs based on issue description and predictive models to group issues or help in triaging
• Fine tuning of models based on the data of issues/PRs/repository of a specific project
• Providing a well defined user experience for maintainers to set up processes for projects and tweaking, fine tuning LLMs
• Providing required hooks to course correct models in case of false positives and false negatives

This can benefit with:

• Providing bug fixes based on the issue description
• Detecting spam
• Having the ability to interact with an issue/PR to help with debugging, asking for reviews, next steps and so on

The following falls in the purview of this issue:

• Building the machinery of identifying models, fine tuning them and deploying/serving them
• Identifying a project under CNCF org to try out this approach
• Possibly moving the project for developing this tool under https://github.com/cncf-tags/cloud-native-ai

[rajaskakodkar]

cc @raravena80 @cathyhongzhang @zanetworker @ronaldpetty @rootfs

[nikhita]

cc @dynamicwebpaige -- since we talked about this at KubeCon and you had ideas around this

[loganloganlogan]

rajaskakodkar Are you familiar with https://dosu.dev/? It's free for Foundation-backed OSS and does some of what you're suggesting. devstein is the person to talk to.

[devstein]

Thanks for the ping @loganloganlogan!

Hey @rajaskakodkar 👋 at Dosu we are building exactly what you described! Dosu helps with auto-labelling, issue deduplication, spam detection, question answering, and bug triage.

We've already helped resolve thousands of issues across thousands of projects like Apache Superset, Element/Matrix, LlamaIndex, and more.

We worked with @krook to give a special free tier for all CNCF projects (tweet). Would love to chat more it.

[raravena80]

Sounds exciting. Hoping to see this integrated here soon!

[rajaskakodkar]

Thank you for chiming in @loganloganloga and nice to meet you @devstein!

Dosu sounds exciting and perfect for our use case. We would love to work on a setup guide for CNCF Projects starting from say, this repo of tag-runtime.

@devstein can you come hangout with us either in the TAG Runtime meeting (the earliest slot is on July 18th) or the CN AI meeting (the earliest slot is on July 12th) to show us all cool things we can do with Dosu and how to integrate it in this repository.

[rajaskakodkar]

After following up from slack, I have added Dosu presentation to the agenda for the TAG Runtime meeting on 1st Aug.