Custom DevSkim rules (on yaml files) not showing in VSCode custom tab

[JaneX8]

Describe the bug
I don't manage to get DevSkim custom rules to show up in the Problems Tab in VSCode. I wrote a json file containing:

To Reproduce

[{
    "id": "TestRule",
    "name": "Basic Test Rule",
    "description": "A simple test rule.",
    "applies_to": ["yaml"],
    "tags": ["test"],
    "patterns": [{
        "pattern": "redirects",
        "type": "regex",
        "message": "Detected redirects."
    }]
}]

When I run:

devskim analyze -I "C:\pathToTestFolder" -r "C:\pathToCustomDevSkimRules"

The output shows a match on my custom rule. However when I simply open the yaml file in VSCode nothing is shown in the Problems tab.

I also added this:

    "MS-CST-E.vscode-devskim.rules.customRulesPaths": [
        "C:\\pathToCustomDevSkimRules"
    ],
    "MS-CST-E.vscode-devskim.logging.level": "debug",

Inside C:\Users\username\AppData\Roaming\VSCodium\User\settings.json. Other DevSkim rules do show up in the Problems tab.

Expected behavior
My custom rules to show up in the Problems tab in VSCode.

Versions(please complete the following information):

• OS: Windows 11
• Devskim Version: devskim 1.0.44+10b85ce690

Other

I saw #300 and my file extensions are actually yaml.

[JaneX8]

Ok, I just wasted an hour debugging what was wrong. And despite restarting VSCodium and disabling and reenabling the DevSkim extensions it wasn't enough. What did the trick was

What did fix it in VSCodium/VSCode: I did Ctrl+Shift+P (or Cmd+Shift+P on macOS), then type Reload Window and hit Enter.

[JaneX8]

The question remains: How do I properly and quickly debug my rules? Who lints the linter? Is there a JSONSchema definition for DevSkim rules? Can I have autocomplete/syntax checking on my DevSkim rules in VSCode? I now have to constantly reload the window after every change or run the command line option and find a tracelog of a failed json parse:

Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to Microsoft.DevSkim.FixType. Path: ...

[JaneX8]

Ok, only reloading the whole VSCode window after every change makes it work. This is a bug, it should work automatically.

[gfs]

The DevSkim CLI has a verify command that does some checks for consistency/correctness, but there isn't an entrypoint to that in the VsCode extension at this point.

The VsCode extension uses a Language Server model, so the VsCode front end launches and then talks to a C# language server that is running the C# DevSkim engine. I think what you may be encountering with it working after a reload is that when the language server relaunches it will load the specified rules from disc with your options - but while VsCode is already running any changes to those files will be reflected on disc but are not hot reloaded into the language server.

[JaneX8]

I think that VSCode is the main reason I'm going crazy with DevSkim now because I'm trying to write custom rules and also using verify and analyze CLI and they produce different outputs (as expected) while not showing in VSCode problems tab, even after reloading the window. That's a pity because VSCode is the main reason I want to use DevSkim. I think it makes a very powerful combination if custom rules are easily writable and testable.

Also, it would be nice if we can figure out a way to reload that Language Server once changes are detected in the Custom Rules folders.

Here is an unexplained example that works in CLI but not showing in VSCode.

test.yaml

dfds:
    xyz:
      - kqe: afa
        asf:
          - 1801
    abc:
      - asd: something
        test:
          - "anything1"
          - "XXanything2"
          - "XXanything3"

example_yaml_rule.json

[{
    "name": "NestedRegexExtractorRule",
    "id": "AMLPath.NestedRegexExtractor",
    "description": "Extract regex patterns dynamically from the nested requests section of the YAML file.",
    "severity": "critical",
    "tags": ["test"],
    "patterns": [{
        "pattern": ".*",
        "ymlpaths": [
            "/dfds/abc/*/test/*"
        ],
        "type": "regex",
        "scopes": [
            "code"
        ],
        "modifiers": [
            "i"
        ],
        "confidence": "high"
    }]
}]

devskim analyze -I ".\test.yaml" -r ".\example_yaml_rule.json"

{"$schema":"https://schemastore.azurewebsites.net/schemas/json/sarif-2.1.0-rtm.6.json","version":"2.1.0","runs":[{"tool":{"driver":{"name":"devskim","fullName":"Microsoft DevSkim Command Line Interface","version":"1.0.44+10b85ce690","informationUri":"https://github.com/microsoft/DevSkim/","rules":[{"id":"AMLPath.NestedRegexExtractor","name":"Nestedregexextractorrule","fullDescription":{"text":"NestedRegexExtractorRule: Extract regex patterns dynamically from the nested requests section of the YAML file."},"help":{"text":"Extract regex patterns dynamically from the nested requests section of the YAML file.","markdown":"Visit [https://github.com/Microsoft/DevSkim/blob/main/guidance/](https://github.com/Microsoft/DevSkim/blob/main/guidance/) for guidance on this issue."},"shortDescription":{"text":"Extract regex patterns dynamically from the nested requests section of the YAML file."},"defaultConfiguration":{"level":"error"},"helpUri":"https://github.com/Microsoft/DevSkim/blob/main/guidance/","properties":{"precision":"","problem.severity":"error","DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}}]}},"versionControlProvenance":[{"repositoryUri":"https://github.com/Rebalance3603/scanning-tool.git","revisionId":"2deea1a5840e3e0303253ec1680e9dd980948a3d","branch":"main"}],"results":[{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"charOffset":193,"snippet":{"text":"","rendered":{"markdown":"``"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":11,"startColumn":12,"charOffset":180,"charLength":13,"snippet":{"text":"\"XXanything3\"","rendered":{"text":"\"XXanything3\"","markdown":"`\"XXanything3\"`"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":10,"startColumn":25,"endLine":10,"endColumn":25,"charOffset":166,"snippet":{"text":"","rendered":{"markdown":"``"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":10,"startColumn":12,"endLine":10,"endColumn":25,"charOffset":153,"charLength":13,"snippet":{"text":"\"XXanything2\"","rendered":{"text":"\"XXanything2\"","markdown":"`\"XXanything2\"`"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":9,"startColumn":23,"endLine":9,"endColumn":23,"charOffset":139,"snippet":{"text":"","rendered":{"markdown":"``"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":9,"startColumn":12,"endLine":9,"endColumn":23,"charOffset":128,"charLength":11,"snippet":{"text":"\"anything1\"","rendered":{"text":"\"anything1\"","markdown":"`\"anything1\"`"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}}],"columnKind":"utf16CodeUnits"}]}

Nothing is shown in the Problems tab. Other DevSkim rules work and do show there.

I didn't see anything worth mentioning under the output tab of DevSkim VS Code Client.

[gfs]

Sorry! I understand the frustration. I can definitely note this as a feature request to add hot-reload when custom rules are changed - as a workaround you can cause the language server to reload the rules from disk by changing a setting, you however, then also need to trigger a new analysis on the file - the trigger is scan is event driven from VS Code itself - so that should be if the file is opened or changed, so even if we could hot reload rules, we'd need something to trigger a rescan.

[gfs]

FWIW, the example rule I shared in the other thread does seem to be working in VS Code. Resharing here:

[{
    "name": "YamlPathValidate",
    "id": "YmlPath",
    "tags": ["test"],
    "description": "values under a key named test that start with XX",
    "severity": "critical",
    "patterns": [
      {
        "pattern":"XX\\w*",
        "ymlpaths" : ["/**/test/*"],
        "type": "regex",
        "scopes": [
          "code"
        ],
        "modifiers": [
          "i"
        ],
        "confidence": "high"
      }
    ]
  }]

[JaneX8]

I can confirm that one works indeed.

[JaneX8]

I'm having the suspicion that regex patterns are not behaving the same on CLI and VSCode, or maybe it has to do with the way it is escaped and passed through the language server. The combination of ymlpath with negative lookaheads in regexes for example are working in CLI but not showing in VSC. As do some other patterns like the simple .* that I showed above.

[gfs]

I tested with the rule you were testing above and it seems to be working for me in VS Code as well.

[JaneX8]

The VsCode extension uses a Language Server model, so the VsCode front end launches and then talks to a C# language server that is running the C# DevSkim engine. I think what you may be encountering with it working after a reload is that when the language server relaunches it will load the specified rules from disc with your options - but while VsCode is already running any changes to those files will be reflected on disc but are not hot reloaded into the language server.

Probably this. I just experienced that despite reloading the window, a previous rule kept working and the new one did not yet. I need to find a way to either manually reload that language server, and then preferably get the extension to do hot reloading at some point.

Possibly related:

• Feature Request: Command to reset / reload the language server vscode-cpptools#1152
• Restart language server – a generic solution vscode#76405

[gfs]

Thanks for the references. I believe we are using the language client library mentioned in 76405, I'll try to take a look and see how this could fit in. The simplest solution might be to add an action to the command palate in vscode to trigger a reload.

[JaneX8]

I'm tired of reloading my window every few minutes when I'm developing DevSkim rules and can't hot-reload. I'm diving into this rabbit hole and find several other GH projects for VS code extensions having similar issues with the Problems tab.

Also not being able to programmatically disable and enable the DevSkim extension or restarting it doesn't help either. I tried experimenting force closing the subprocess dotnet.exe and it will be reopened automatically, the problems in the DevSkim Problems-tab disappear but don't come back, I guess an association problem. I'm testing with VSCodium but VSCode should be the same. This is absolute overkill but the main process VSCode contains another process VSCode which contains the dotnet.exe (DevSkim). When I kill that middle process (which I think is the whole extensionhost), reloads the problems tab entirely, with new DevSkim rules.

I'm gonna write a DevSkim custom rules directory watcher that does this on any custom DevSkim rule change and use it when developing and testing DevSkim rules. Note that is is a very quick and dirty workaround for my problem for now. And not a recommended solution. It works for my purpose. Tested with one VSCodium window open, with many extensions where in my case only DevSkim uses dotnet.exe.

I wrote the following small powershell script that I'm going to use in a directory watcher when writing custom DevSkim rules:

$topVSCodium = (Get-Process -Name VSCodium | Sort-Object StartTime | Select-Object -First 1)
$dotnet = (Get-CimInstance Win32_Process | Where-Object { $_.Name -eq "dotnet.exe" })
if ($dotnet) {
    $parent = (Get-CimInstance Win32_Process | Where-Object { $_.ProcessId -eq $dotnet.ParentProcessId -and $_.Name -eq "VSCodium.exe" })
    if ($parent -and $parent.ProcessId -ne $topVSCodium.Id) {
        Stop-Process -Id $parent.ProcessId -Force
        Write-Host "Killed VSCodium parent process of dotnet.exe (PID: $($parent.ProcessId))."
    } else {
        Write-Host "Parent process is the top-level VSCodium.exe; not killing it."
    }
} else {
    Write-Host "No dotnet.exe process found under VSCodium."
}

From sysinternals process explorer, the selected parent of dotnet.exe seems to be the extensionshost process:

I'm not sure if this is something that we could even fix in a VSCode extension, it seems to me it's a VSCode core issue but I hope I'm wrong. Still remains: hot-reloading would make writing customs rules much easier and faster.

[gfs]

I think this unfortunately comes down to a VSCode architecture limitation - even if you restart the extension process the files themselves needs to be processed again to get new issues, which gets triggered by events fired by vscode like file change. The easiest way to trigger this, in my experience, is to add a character to a file you want rescanned and then delete it. I think it is possible to add a manual action to explicitly kick off a scan of all files in the workspace - but this is not implemented in the DevSkim extension, and I'm not sure how trivial it would be to add. I personally develop and debug rules using the command line version of devskim.

[JaneX8]

I tried to find another way to programmatically reload or force a trigger in other ways like adding a file to the workspace, deleting a file to the workspace, editing a temporary file in the workspace, changing timestamps of existing files, none trigger a reload. I couldn't find any other way to achieve this which is why I found the right process and killed it. Not a solution at all, very dirty workaround..

[JaneX8]

@gfs I think you are in a better position to answer microsoft/vscode#235799 (comment). Could you maybe?