[Advanced Paste] Add Semantic Kernel opt-in to allow chaining of paste actions #35902

drawbyperpetual · 2024-11-12T21:28:02Z

Summary of the Pull Request

See linked issue for more information.

PR Checklist

Detailed Description of the Pull Request / Additional comments

There's quite a lot of refactoring and code-quality improvements here. A non-exhaustive list:

All transformations now work on and produce a 'virtual' in-memory clipboard representation (DataPackage) rather than on the system clipboard. The system clipboard is only read once per operation and written to only just before pasting (if needed). There are multiple reasons for this:
- Prevents Semantic Kernel intermediate transformations from polluting the clipboard with intermediate data.
- Allows Semantic Kernel tool-calls (and in fact, other transforms too) to easily run on the thread-pool without causing "calling application is not the owner of the data on the clipboard" issues.
- Allows transformations to be unit-tested without manipulating the system clipboard.
- Cleaner and faster as the actual clipboard is touched only once.
More Async within AdvancedPaste (this was also done as part of Additional Actions but there's much more of it now).
Paste actions now have a CanPreview property (and Image to Text has been marked as having it to allow it to be previewed)
lastQuery.json functionality is removed
IFileSystem is used within Advanced Paste to improve testability.

Validation Steps Performed

Tested new functionality
Tested existing Advanced Paste functionality

…/advanced-paste-semantic-kernel

jaimecbernardo · 2024-11-14T08:48:56Z

/azp run

azure-pipelines · 2024-11-14T08:49:09Z

Azure Pipelines successfully started running 1 pipeline(s).

jaimecbernardo · 2024-11-14T08:58:12Z

Regarding the policy, is there any benefit that we would get by adding one given that we already one for online AI models?

Two maybe benefits I am not sure about:

Prevent using gpt-4o and force gpt3-turbo.

Prevent that users create complex queries that costs many credits. (Maybe we can make a max steps limit policy?)

But I am not sure if these are things that require a policy. 🤔

In terms of GPO for the feature, I'm thinking the open AI already has controls to limit credit usage / model usage per token, in case they're using a work API key 🤔 A new GPO seems not necessary at this point 😄

drawbyperpetual · 2024-11-14T12:46:24Z

@htcfreek I've moved the Custom Preview toggle into the Behavior section as you suggested.

jaimecbernardo

Code LGTM.
Testing looks pretty good too. Some errors could be more explicit, like when an image can't be converted through text, but I don't think that's blocking.

Still checking on some verifications, so I'm not approving yet, but will approve after verifications are done. Good work!

htcfreek · 2024-12-07T18:39:50Z

src/settings-ui/Settings.UI/SettingsXAML/Views/AdvancedPaste.xaml

                            IsEnabled="{x:Bind ViewModel.IsOpenAIEnabled, Mode=OneWay}">
-                            <ToggleSwitch IsOn="{x:Bind ViewModel.ShowCustomPreview, Mode=TwoWay}" />
+                            <ToggleSwitch IsOn="{x:Bind ViewModel.IsAdvancedAIEnabled, Mode=TwoWay}" />
                        </tkcontrols:SettingsCard>
                        <InfoBar
                            x:Uid="GPO_SettingIsManaged"


Please move this between "AI enabled" setting and "Advanced AI" setting. Otherwise users can misunderstand for which setting this information is.

Are you referring to the GPO_SettingIsManaged InfoBar?

@htcfreek: Done

htcfreek

From what I can see in the code everything looks good to me.

jaimecbernardo

LGTM! Thank you!

* [Workspaces] implement the move feature (microsoft#35480) * [Workspaces] Add move functionality * spell checker * [Workspaces] Modify Arranger to move apps without launch * moved ipc helper * removed callback * use LauncherStatus in WindowArranger * wait for launching next app * launch in a separate thread and protect by mutexes * update app version in advance * changed canceling launch * increased waiting time * Fix optional parameter load from json * changed arranger waiting time * additional waiting time for Outlook * added app id * ensure ids before launch * set id in editor * minor updates * [Workspaces] Move: Get the nearest window when moving a window * [Workspaces] convert optional boolean to enum to avoid json problems * Handle case when the new Application Property "moveIfExists" does not exist * Re-implementing app-window pairing for moving feature. * spell checker * XAML formatting * Fixing bug: IPC message not arriving * spell checker * Removing app-level-setting for move app. Also fixed compiler errors due styling. * Updating editor window layout * Re-implementing window positioning UI elements * XAML formatting * Code review findings * Code cleanup * Code cleanup * Code cleanup * code cleanup * Code cleanup * Code cleanup * fix Move attribute after launch and snapshot * Extend WindowArranger with PWA functionality to detect different PWA apps. PwaHelper moved to the common library * fix repeat counter in the editor * Code optimization * code cleanup, optimization * fix double-processing window --------- Co-authored-by: Seraphima <[email protected]> Co-authored-by: donlaci <[email protected]> * [KeyboardManager]Fix mapping shift to numpad (microsoft#35890) * Keyboard Manger fix numpad as shift Fixed shift not being released if a numpad key as shift. * Added comments * Fix typo * Fix the numpad unlocked key not working if the locked version is overridden by shift * Fix spelling check. * Revert the VK_CLEAR change. --------- Co-authored-by: Jaime Bernardo <[email protected]> * [Analyzers] Update .editorconfig with rules to relax IDE errors (microsoft#36095) * [Analyzers] Remove duplicate pascal case style from .editorconfig * [Analyzers] Configured severity for individual IDE and CA rules showing as errors in VS Set severity for IDE0005, IDE0008, IDE0016, IDE0018, IDE0019, IDE0021, IDE0022, IDE0023, IDE0025, IDE0027, IDE0028, IDE0029, IDE0031, IDE0032, IDE0034, IDE0036, IDE0039, IDE0042, IDE0044, IDE0045, IDE0046, IDE0047, IDE0057, IDE0051, IDE0052, IDE0054, IDE0055, IDE0056, IDE0057, IDE0059, IDE0060, IDE0061, IDE0063, IDE0071, IDE0073, IDE0074, IDE0075, IDE0077, IDE0078, IDE0083, IDE0090, IDE0100, IDE0130, IDE160, IDE180, IDE0200, IDE0240, IDE0250, IDE0251, IDE0260, IDE0270, IDE0290, IDE0300, IDE0301, IDE0305, IDE1005, IDE1006, CA1859, CA2022, CA2263 * [Analyzers] Fix mismatched analyzer descriptions * [Analyzers] Fix misspelling * Update .editorconfig Made the following style rules `silent` instead of `suggestion`: - Use explicit type instead of 'var' - Use expression body for ... - Use block-scoped namespace * [Analyzers] Set IDE0290 to silent * [Analyzers] Remove IDE1006 configuration from .editorconfig in favor of making exclusions for the few entries * [Analyzers][Indexer] Add IDE1006 suppressions * [Analyzers][Peek] Add IDE1006 suppression * [Analyzers][MWB] Add IDE1006 suppression. * [Analyzers][Plugins] Add IDE1006 suppression * [Analyzers][ImageResizer] Suppress IDE0073 to retain original copyright * [Analyzers] Remove IDE0073 severity change in .editorconfig --------- Co-authored-by: Ani <[email protected]> * [Workspaces] PWA follow-up (microsoft#36217) * [PTRun][Calculator]Allow scientific notation with lowercase 'e' (microsoft#36187) * [Workspaces] Add encoder parameter to bitmap.save() (microsoft#36228) * [Workspaces] Add encoder parameter to bitmap.save() * 1 more call fixed * Move repeated code to the csharp library * [Workspaces] Implement store of app window's size and position (microsoft#36086) * [Workspaces] Implement store of app window's size and position * Modifying the default values to -1. The program will use the original default values for the first run. * [ScreenRuler]Add setting to show the measurement in an extra unit (microsoft#35887) * display ruler: supporting millimeter and other units * Measurement Tool: UI Setting for an extra unit * Update images * spelling * spelling * suit code style * Fix for code review * remove weird file * rename field * [Deps]Update MSTest from 3.5.0 to 3.6.3 (microsoft#36115) * Update MSTest from 3.5.0 to 3.6.3 * Use STA attributes that are now part of MSTest * Adding Jerry to community.md (microsoft#36232) Update COMMUNITY.md * [Workspaces] Arranger: smart timer (microsoft#36096) * [Workspaces] Add move functionality * spell checker * [Workspaces] Modify Arranger to move apps without launch * moved ipc helper * removed callback * use LauncherStatus in WindowArranger * wait for launching next app * launch in a separate thread and protect by mutexes * update app version in advance * changed canceling launch * increased waiting time * Fix optional parameter load from json * changed arranger waiting time * additional waiting time for Outlook * added app id * ensure ids before launch * set id in editor * minor updates * [Workspaces] Move: Get the nearest window when moving a window * [Workspaces] convert optional boolean to enum to avoid json problems * Handle case when the new Application Property "moveIfExists" does not exist * Re-implementing app-window pairing for moving feature. * spell checker * XAML formatting * Fixing bug: IPC message not arriving * spell checker * Removing app-level-setting for move app. Also fixed compiler errors due styling. * Updating editor window layout * Re-implementing window positioning UI elements * XAML formatting * Code review findings * Code cleanup * Code cleanup * Code cleanup * code cleanup * Code cleanup * Code cleanup * [Workspaces] Arranger: Reset wait timer after each successful arrange action * fix merge error --------- Co-authored-by: Seraphima <[email protected]> Co-authored-by: donlaci <[email protected]> * Upgrade to check-spelling v0.0.24 (microsoft#36235) This upgrades to [v0.0.24](https://github.com/check-spelling/check-spelling/releases/tag/v0.0.24). A number of GitHub APIs are being turned off shortly, so you need to upgrade or various uncertain outcomes will occur. There's a new accessibility forbidden pattern: > Do not use `(click) here` links > For more information, see: > * https://www.w3.org/QA/Tips/noClickHere > * https://webaim.org/techniques/hypertext/link_text > * https://granicus.com/blog/why-click-here-links-are-bad/ > * https://heyoka.medium.com/dont-use-click-here-f32f445d1021 ```pl (?i)(?:>|\[)(?:(?:click |)here|link|(?:read |)more)(?:</|\]\() ``` There are some minor bugs that I'm aware of and which I've fixed since this release, but I don't expect to make another release this month. I've added a pair of patterns for includes and pragmas. My argument is that the **compiler** will _generally_ tell you if you've misspelled an include and the **linker** will _generally_ tell you if you misspell a lib. - There's a caveat here: If your include case-insensitively matches the referenced file (but doesn't properly match it), then unless you either use a case-sensitive file system (as opposed to case-preserving) or beg clang to warn, you won't notice when you make this specific mistake -- this matters in that a couple of Windows headers (e.g. Unknwn.h) have particular case and repositories don't tend to consistently/properly write them. * Adjust to community.md, shifting jerry's github user name (microsoft#36242) Update COMMUNITY.md * [AOT compatible] Resolve AOT Build Error in Peek.UI (microsoft#36194) * add partial for aot support * add Microsoft.NET.ILLink.Tasks to packages.props * format * Revert "format" This reverts commit 742d5e2. * add Microsoft.NET.ILLink.Tasks to notice.md * add auto reference * update script to remove the 'Auto-reference line' --------- Co-authored-by: Jaime Bernardo <[email protected]> * Awake Updates - `TILLSON_11272024` (microsoft#36049) * Update with bug fixes for tray icon and support for parent process * Process information enum * Update the docs * Fix spelling * Make sure that PID is used in PT config flow * Logic for checks based on microsoft#34148 * Update with link to PR * Fixes microsoft#34717 * Small cleanup * Proper task segmentation in a function * Cleanup the code * Fix synchronization context issue * Update planning doc * Test disabling caching to see if that manages to pass CI * Cleanup to make sure that we're logging things properly. * Update ci.yml * Disable cache to pass CI * Retry logic * Cleanup * Code cleanup * Fixes microsoft#35848 * Update notes and codename * After third attempt, log error instead of throwing exception * More cleanup to avoid double execution * Add expected word * Safeguards for bad values for timed keep-awake * More updates to make sure I am using uint * Update error message * Update packages * Fix notice and revert CsWinRT upgrade * Codename update * Update expect.txt * Update the struct * Ensuring we're properly awaiting tray initialization * Update to make sure tray reflects the bound process * Cleanup, proper JSON serialization for logs. * Not needed. * Add command validation logic * Moving the initialization logic earlier * Make sure we show the display state in the tooltip * Update tray string * Update src/modules/awake/Awake/Core/Manager.cs Co-authored-by: Jaime Bernardo <[email protected]> * Update src/modules/awake/Awake/Core/Manager.cs Co-authored-by: Jaime Bernardo <[email protected]> * Update src/modules/awake/Awake/Core/Manager.cs Co-authored-by: Jaime Bernardo <[email protected]> * Update src/modules/awake/Awake/Core/Manager.cs Co-authored-by: Jaime Bernardo <[email protected]> * Update logic for icon resets * Update doc * Simplify function for setting mode shell icon * Issues should be properly linked * Minor cleanup * Update timed behavior --------- Co-authored-by: Jaime Bernardo <[email protected]> Co-authored-by: Clint Rutkas <[email protected]> * [Workspaces] detecting right app version (microsoft#36100) * [Workspaces] Add move functionality * spell checker * [Workspaces] Modify Arranger to move apps without launch * moved ipc helper * removed callback * use LauncherStatus in WindowArranger * wait for launching next app * launch in a separate thread and protect by mutexes * update app version in advance * changed canceling launch * increased waiting time * Fix optional parameter load from json * changed arranger waiting time * additional waiting time for Outlook * added app id * ensure ids before launch * set id in editor * minor updates * [Workspaces] Move: Get the nearest window when moving a window * [Workspaces] convert optional boolean to enum to avoid json problems * Handle case when the new Application Property "moveIfExists" does not exist * Re-implementing app-window pairing for moving feature. * spell checker * XAML formatting * Fixing bug: IPC message not arriving * spell checker * Removing app-level-setting for move app. Also fixed compiler errors due styling. * Updating editor window layout * Re-implementing window positioning UI elements * XAML formatting * Code review findings * Code cleanup * Code cleanup * Code cleanup * code cleanup * Code cleanup * Code cleanup * [Workspaces] fix detection of specific version of apps --------- Co-authored-by: Seraphima <[email protected]> Co-authored-by: donlaci <[email protected]> * Move the XamlStyler config to src/ (microsoft#36202) my never ending goal to minimize files in the root dir * [AdvancedPaste]Add Semantic Kernel opt-in to allow chaining of paste actions (microsoft#35902) * [AdvancedPaste] Semantic Kernel support * Changed log-line with potentially sensitive info * Spellcheck issues * Various improvements for Semantic Kernel * Spellcheck issue * Refactored Clipboard routines * Added integration tests for KernelService * Extra telemetry for AdvancedPaste * Added 'Hotkey' suffix to AdvancedPaste_Settings telemetry event * Added IsSavedQuery * Added KernelQueryCache * Refactoring * Added KernelQueryCache to BugReportTool delete list * Added opt-n for Semantic Kernel * Fixed bug with KernelQueryCache * Ability to view last AI chat message on error * Improved kernel query cache * Used System.IO.Abstractions and improved tests * Fixed under-count of token usage * Used Semantic Kernel icon * Cleanup * Add missing EndProject line * Fix dependency version conflicts * Fix NOTICE.md * Correct place of SemanticKernel in NOTICE.md * Unlinked CustomPreview toggle from AI * Added Microsoft.Bcl.AsyncInterfaces dependency to AdvancedPaste * Fixed NOTICE.md order * Moved Custom Preview to behaviour section * Made Image to Text raise error on empty output * Added AIServiceBatchIntegrationTests * Updated AIServiceBatchIntegrationTests * Added prompt moderation * Moved GPO Infobar to better location * [Launcher]Port from WPF-UI to .NET 9 WPF (microsoft#36215) * Initial implementation * Fix fluent style * Fix no endline * Update expect.txt * Fix formatting * Fix light theme looking bad on Windows 10 * fix formatting * test change * Now really fixed W10 * Add a comment * Fix typos * Fix spellcheck errors * Fix spellcheck pattern for websites * Change patterns for spellcheck in the right file * Fix XAML styling * Fix contrast colors on W11 * Fix formatting * Removed emty line * Fix formatting * Added comment to fluentHC file * fix comment * Fix Windows10 again. Adress feedback. * W11 fix chaning from high contrast to normal not having correct background * W10 Fix high contrast not working after switching from light/dark moed * Address feedback * Fix formatting * Second W11 fix chaning from high contrast to normal not having correct background * [UX]Updating New+ and Settings icons (microsoft#36290) * Updated icons * Updating more icons and icos * [Settings][Dashboard] Accessibility fixes (microsoft#36280) * make narrator announce buttons/toggles * add toggles module name * [ci]Sign OpenAI dll that's not signed (microsoft#36299) * Update CODEOWNERS to include gordon, jerry and kayla (microsoft#36308) * Update CODEOWNERS * Update names.txt * Making the powertoys-code-owners team code owners (microsoft#36310) * Update CODEOWNERS * Update names.txt * Update CODEOWNERS * [FZEditor]Fix Create new layout dialog radio buttons IsChecked values (microsoft#36320) * 0.87 changelog (microsoft#36335) * 0.87 changelog * Fix spellcheck * Update README.md Co-authored-by: Heiko <[email protected]> --------- Co-authored-by: Heiko <[email protected]> --------- Co-authored-by: Laszlo Nemeth <[email protected]> Co-authored-by: Seraphima <[email protected]> Co-authored-by: donlaci <[email protected]> Co-authored-by: Ionuț Manța <[email protected]> Co-authored-by: Jaime Bernardo <[email protected]> Co-authored-by: Jeremy Sinclair <[email protected]> Co-authored-by: Ani <[email protected]> Co-authored-by: PesBandi <[email protected]> Co-authored-by: Wenjian Chern <[email protected]> Co-authored-by: Youssef Victor <[email protected]> Co-authored-by: Clint Rutkas <[email protected]> Co-authored-by: Josh Soref <[email protected]> Co-authored-by: leileizhang <[email protected]> Co-authored-by: Den Delimarsky 🔐 <[email protected]> Co-authored-by: Niels Laute <[email protected]> Co-authored-by: Davide Giacometti <[email protected]> Co-authored-by: Heiko <[email protected]>

drawbyperpetual added 30 commits September 11, 2024 21:08

[AdvancedPaste] Additional actions, including Image to text

e79d86d

Merge branch 'main' into dev/ani/advanced-paste-additional-actions

ec0c300

Spellcheck issue

e61460d

Merge branch 'main' into dev/ani/advanced-paste-additional-actions

7dc0f7f

[AdvancedPaste] Paste as file and many other improvements

b953218

Fixed typo

a843152

Fixed typo

35e6375

[AdvancedPaste] Improved paste window menu layout

a25fbe3

[AdvancedPaste] Improved settings window layout

fa6ddbc

Merged main

6f576ee

Merged main

15beab6

[AdvancedPaste] Removed AudioToText for the moment

aee22bc

Code cleanup

f3718c0

Minor fixes

f6d2304

Merge branch 'main' into dev/ani/advanced-paste-additional-actions

ab297d4

Merge branch 'main' into dev/ani/advanced-paste-additional-actions

32cf896

Merge branch 'main' into dev/ani/advanced-paste-additional-actions

7d8a57c

[AdvancedPaste] Semantic Kernel support

737e179

Changed log-line with potentially sensitive info

fe46df5

Spellcheck issues

02e0b24

Merge branch 'main' into dev/ani/advanced-paste-additional-actions

c502479

Merge branch 'dev/ani/advanced-paste-additional-actions' into dev/ani…

3c23130

…/advanced-paste-semantic-kernel

Merge branch 'main' into dev/ani/advanced-paste-additional-actions

b81b686

Merge branch 'dev/ani/advanced-paste-additional-actions' into dev/ani…

623c02f

…/advanced-paste-semantic-kernel

Various improvements for Semantic Kernel

7036e1d

Spellcheck issue

14f0d48

Merge branch 'main' into dev/ani/advanced-paste-additional-actions

76ad851

Merge branch 'dev/ani/advanced-paste-additional-actions' into dev/ani…

9c3f4bc

…/advanced-paste-semantic-kernel

Refactored Clipboard routines

a37673c

Added integration tests for KernelService

c169756

Merged main

6443ddd

crutkas added the Needs-Review This Pull Request awaits the review of a maintainer. label Nov 13, 2024

drawbyperpetual added 2 commits November 13, 2024 22:40

Added Microsoft.Bcl.AsyncInterfaces dependency to AdvancedPaste

bf542e2

Fixed NOTICE.md order

77605b7

Moved Custom Preview to behaviour section

a075402

jaimecbernardo reviewed Nov 20, 2024

View reviewed changes

drawbyperpetual added 6 commits November 20, 2024 16:44

Made Image to Text raise error on empty output

ddbf4a3

Added AIServiceBatchIntegrationTests

3525354

Merged main

85bf360

Updated AIServiceBatchIntegrationTests

5cd4223

Added prompt moderation

4a593e2

Merge branch 'main' into dev/ani/advanced-paste-semantic-kernel

b10d26c

crutkas added the in for .87 label Nov 26, 2024

drawbyperpetual added 3 commits November 27, 2024 19:17

Merged main

b6b503b

Merge branch 'main' into dev/ani/advanced-paste-semantic-kernel

b28ea47

Merged main

58a5854

htcfreek suggested changes Dec 7, 2024

View reviewed changes

htcfreek mentioned this pull request Dec 7, 2024

[Settings] Update GPO infobar icon, improve some pages and fix bugs #33703

Open

11 tasks

drawbyperpetual added 3 commits December 9, 2024 14:15

Merged main

e191935

Merge branch 'main' into dev/ani/advanced-paste-semantic-kernel

cf01fb0

Moved GPO Infobar to better location

8279079

htcfreek approved these changes Dec 11, 2024

View reviewed changes

jaimecbernardo approved these changes Dec 11, 2024

View reviewed changes

jaimecbernardo merged commit bf3474b into main Dec 11, 2024
16 checks passed

drawbyperpetual deleted the dev/ani/advanced-paste-semantic-kernel branch December 13, 2024 02:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Advanced Paste] Add Semantic Kernel opt-in to allow chaining of paste actions #35902

[Advanced Paste] Add Semantic Kernel opt-in to allow chaining of paste actions #35902

drawbyperpetual commented Nov 12, 2024 •

edited

Loading

jaimecbernardo commented Nov 14, 2024

azure-pipelines bot commented Nov 14, 2024

jaimecbernardo commented Nov 14, 2024

drawbyperpetual commented Nov 14, 2024

jaimecbernardo left a comment

htcfreek Dec 7, 2024

drawbyperpetual Dec 8, 2024

htcfreek Dec 8, 2024

drawbyperpetual Dec 11, 2024

htcfreek left a comment

jaimecbernardo left a comment

[Advanced Paste] Add Semantic Kernel opt-in to allow chaining of paste actions #35902

[Advanced Paste] Add Semantic Kernel opt-in to allow chaining of paste actions #35902

Conversation

drawbyperpetual commented Nov 12, 2024 • edited Loading

Summary of the Pull Request

PR Checklist

Detailed Description of the Pull Request / Additional comments

Validation Steps Performed

jaimecbernardo commented Nov 14, 2024

azure-pipelines bot commented Nov 14, 2024

jaimecbernardo commented Nov 14, 2024

drawbyperpetual commented Nov 14, 2024

jaimecbernardo left a comment

Choose a reason for hiding this comment

htcfreek Dec 7, 2024

Choose a reason for hiding this comment

drawbyperpetual Dec 8, 2024

Choose a reason for hiding this comment

htcfreek Dec 8, 2024

Choose a reason for hiding this comment

drawbyperpetual Dec 11, 2024

Choose a reason for hiding this comment

htcfreek left a comment

Choose a reason for hiding this comment

jaimecbernardo left a comment

Choose a reason for hiding this comment

drawbyperpetual commented Nov 12, 2024 •

edited

Loading