Feature Request: Source code highlight and math formula rendering

[xintq]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

1. Highlight the source code in messages.
2. Render the mathematical formulas in messages.
   

Motivation

The following are missing in the default webui:

1. Highlight the source code in messages.
2. Render the mathematical formulas in messages.

Possible Implementation

This can be down by using highlight.js and markdown-it-katex-gpt in the llama-server webui.
I've implemented this by the following steps.

1. Install the packages in webui

npm install highlight.js
npm install markdown-it-katex-gpt

3. Modify main.js like

// Add these two line at the beginning
import 'highlight.js/styles/github-dark-dimmed.min.css'
import 'katex/dist/katex.min.css'
import hljs from 'highlight.js' // https://highlightjs.org
import markdownItKatexGpt from 'markdown-it-katex-gpt'
...
// markdown support
const VueMarkdown = defineComponent(
  (props) => {
    const md = shallowRef(new MarkdownIt({
      breaks: true,
      highlight: function (str, lang) { // Add highlight.js
        if (lang && hljs.getLanguage(lang)) {
          try {
            return '<pre><code class="hljs">' +
                   hljs.highlight(str, { language: lang, ignoreIllegals: true }).value +
                   '</code></pre>';
          } catch (__) {}
        }
        return '<pre><code class="hljs">' + md.value.utils.escapeHtml(str) + '</code></pre>';
      }
    }));
    md.value.use(markdownItKatexGpt, { // Add katex support
      delimiters: [
        { left: '\\[', right: '\\]', display: true },
        { left: '\\(', right: '\\)', display: false },
        { left: '$$', right: '$$', display: false },
      ]
    })

    const origFenchRenderer = md.value.renderer.rules.fence;
    md.value.renderer.rules.fence = (tokens, idx, ...args) => {
      const content = tokens[idx].content;
      const origRendered = origFenchRenderer(tokens, idx, ...args);
      return `<div class="relative my-4">
        <div class="text-right sticky top-4 mb-2 mr-2 h-0">
          <button class="badge btn-mini" onclick="copyStr(${escapeAttr(JSON.stringify(content))})">📋 Copy</button>
        </div>
        ${origRendered}
      </div>`;
    };
    window.copyStr = copyStr;
    const content = computed(() => md.value.render(props.source));
    return () => h("div", { innerHTML: content.value });
  },
  { props: ["source"] }
);

[ngxson]

The proposal seems good. You can add a PR for it if you like

[mirek190]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

1. Highlight the source code in messages.
2. Render the mathematical formulas in messages.
   

Motivation

The following are missing in the default webui:

1. Highlight the source code in messages.
2. Render the mathematical formulas in messages.

Possible Implementation

This can be down by using highlight.js and markdown-it-katex-gpt in the llama-server webui. I've implemented this by the following steps.

1. Install the packages in webui

npm install highlight.js
npm install markdown-it-katex-gpt

3. Modify main.js like

// Add these two line at the beginning
import 'highlight.js/styles/github-dark-dimmed.min.css'
import 'katex/dist/katex.min.css'
import hljs from 'highlight.js' // https://highlightjs.org
import markdownItKatexGpt from 'markdown-it-katex-gpt'
...
// markdown support
const VueMarkdown = defineComponent(
  (props) => {
    const md = shallowRef(new MarkdownIt({
      breaks: true,
      highlight: function (str, lang) { // Add highlight.js
        if (lang && hljs.getLanguage(lang)) {
          try {
            return '<pre><code class="hljs">' +
                   hljs.highlight(str, { language: lang, ignoreIllegals: true }).value +
                   '</code></pre>';
          } catch (__) {}
        }
        return '<pre><code class="hljs">' + md.value.utils.escapeHtml(str) + '</code></pre>';
      }
    }));
    md.value.use(markdownItKatexGpt, { // Add katex support
      delimiters: [
        { left: '\\[', right: '\\]', display: true },
        { left: '\\(', right: '\\)', display: false },
        { left: '$$', right: '$$', display: false },
      ]
    })

    const origFenchRenderer = md.value.renderer.rules.fence;
    md.value.renderer.rules.fence = (tokens, idx, ...args) => {
      const content = tokens[idx].content;
      const origRendered = origFenchRenderer(tokens, idx, ...args);
      return `<div class="relative my-4">
        <div class="text-right sticky top-4 mb-2 mr-2 h-0">
          <button class="badge btn-mini" onclick="copyStr(${escapeAttr(JSON.stringify(content))})">📋 Copy</button>
        </div>
        ${origRendered}
      </div>`;
    };
    window.copyStr = copyStr;
    const content = computed(() => md.value.render(props.source));
    return () => h("div", { innerHTML: content.value });
  },
  { props: ["source"] }
);

Your implementation is working fully offline?

[xintq]

Your implementation is working fully offline?

Yes, as far as I tested. It can work offline. @mirek190

[ngxson]

The katex library is too big for this a nice-to-have feature like this (takes > 1MB from bundled size). We should find other alternatives.

[slaren]

Why is the bundle size a concern here?

[ngxson]

On my build (macos metal), the server without webui is just 1.8MB. So including a 2MB+ frontend doesn't seem right to me.

Also katex is quite a nice-to-have feature. Having a 1.5MB library for the only purpose of rendering latex is quite wasteful. To put it into perspective, all the other part of the UI (daisyui + tailwindcss + vuejs + highlight) combined is just less than 600KB.

[ngxson]

Just to be clear, adding syntax highlighting is quite trivial and only cost 100KB to the bundle, so it will be added. We only have the problem with katex, which has too much JS code and at the same time, also use custom embedded fonts

[slaren]

I understand, but if that's what it takes to render latex, I think we should consider it. I don't think that proper rendering of the model responses is just a nice-to-have feature, I think it is borderline mandatory for an usable UI. Realistically, it may not be possible to find a smaller library, since all latex rendering is going to require custom fonts. Maybe it could be optionally loaded from cdn?

[ngxson]

Yes I was thinking about that too. We can add an option under settings to enable latex rendering, and explicitly state that it will use katex from CDN (not from local)

Another option could be to use gzip compression at it can reduce up the storage size to only 30%. But that requires linking against zlib. WDYT?

[slaren]

I suspect it should be possible to avoid needing to link to zlib if the file is stored pre-compressed by setting the proper http headers. Would that work? From what I understand, zlib is only necessary if we want to compress the output as it is generated.

[ngxson]

Hmm yes that's a good idea. It's possible to pre-compress the file and set appropriate HTTP header as you said. It will break if the client don't have support for gzip deflate, but I think we're fine because no one gonna use internet explorer

[p-e-w]

Note that Brotli is also supported on all browsers except IE, and usually gives about 20% better compression than gzip.

[ngxson]

hmm yeah I do acknowledge brotli, but I also remember some users use this behind a reverse proxy, which may not have support for brotli (i.e. some user may try to decompress the data and encode it back in gzip)

For now gzip is already good enough, we can switch to brotli in the future if we really need.

[p-e-w]

Which reverse proxies don't support Brotli? Brotli is more than 10 years old, and e.g. nginx has supported it since 2016.

[ngxson]

nginx don't have it built-in, need to be installed as a module

[p-e-w]

Ah, I didn't know that. Makes sense to stick to gzip then!