TypeSafe Jev support in term-llm

Dev Log
1

I just landed support for classification in term-llm.

This adds a new primitive to term-llm:

term-llm classify

This is documented at:

Classification with TypeSafe · term-llm docs

The original typesafe docs are at: Introduction - TypeSafe AI

It allows you to classify STDIN, random text or files.

For example:

term-llm classify --file main.go \
   --question "Does the file contain any serious security issues" \ 
   --type choice --option yes --option no \
   --pretty-print

{
  "model": "jev-1.13.0",
  "answers": {
    "result": {
      "type": "choice",
      "choice": "no",
      "confidence": 0.99,
      "probabilities": {
        "no": 1.0,
        "yes": 0.0
      }
    }
  },
  "usage": {
    "input_tokens": 318,
    "output_tokens": 31
  }
}

Jev is a new kind of AI model, it is extremely cheap and extremely fast, however it works extremely different to LLMs. I like to think of it as a classifier, hence the classify top level command.

I also added an experimental:

guardian:
  backend: classify

term-llm, out-of-the-box, ships with a “auto approval” mode which leans on guardian.

This allows you to classify risk of commands using Jev, in local testing this is LIGHTNING fast.

Faster then any other “approve for me” system you have ever seen, however, coming up with the right recipe and evals to ensure similar profile risk to what an LLM provides is still in progress.

My design tried to err on the side of rejecting a bit too much vs approving too many.

Overall I find Jev amazing, I love having first party support in term-llm, and I am sure we will accrue more and more providers for classification. This is just the start of RLCD

Guardian is a perfect fit for the new system, but there are other interesting experiments we could look at. Some ideas could be:

  • An extension that rates “how close a changeset is to being merge ready”
  • Novel compaction techniques
  • Website use. (jev can predict based on user goal what buttons to click)
  • Organisational tools (sort out my folders etc..)

If you are interested in playing with Jev I recommend heading to https://typesafe.ai/ and joining the waitlist.

If you come up with any interesting use cases, let me know.

2 Likes