Testing

Three tools, from cheapest to most real:

	Costs	Shows you
kol prompt	Nothing (no AI model call)	Exactly what the AI is told: your persona, your knowledge, Kol's rules, its tools
kol try	A little of your AI key	One reply, from the files as they are (nothing is saved)
kol test	A little of your AI key per turn	Pass or fail for every check in tests/

And finally, the real thing: kol push, then talk to it by voice in the app.

Try it

kol try                                   # its greeting
kol try "What can I cook with eggs and spinach?"
kol try --session dinner "I'm making pasta tonight"
kol try --session dinner "How much salt?"  # continues the same conversation
kol try --session dinner --reset "Start over"
kol try --as standIn "Is Dana there?"     # answering a call for its owner
kol try --saved "Hi"                      # the version you pushed (with web pages from urls.txt)

Tools run in test mode: when it uses send_request, you see [send_request → …] above the reply, but nothing is sent.

Write tests

A test is a conversation in tests/<name>.json. Each turn is something the person says, and what the reply must (or must not) contain:

{
  "name": "kitchen notes",
  "turns": [
    { "user": "How much salt goes in pasta water?", "expect": { "includesAny": ["10", "ten"] } },
    { "user": "And for a runny egg?", "expect": { "includesAny": ["6", "six"], "maxWords": 60 } }
  ]
}

Checks: includesAny, includesAll, excludes, matches (a regular expression), tool, noTool, maxWords. All are case-insensitive. Full list: Project files.

kol test                 # all of them; exit code 1 if any check fails
kol test --only eggs     # one
kol test --json          # every reply and every failure, for scripts and agents

What to test

• The basics (in every template): it says it's an AI when asked; it doesn't ramble.
• Every fact that matters from your knowledge: prices, hours, quantities.
• Every skill: a request is sent only after the caller confirms (noTool, then tool).
• What it must refuse: medical advice, card numbers, promises.
• Short answers: maxWords on questions that invite a long reply.

AI replies vary from run to run. Prefer includesAny with a few ways of saying the same thing (["6", "six"]) over exact sentences, and run kol test twice before trusting a change.

In CI

Tests need a signed-in account with an AI key. Make a token with kol login on your machine, copy it from ~/.config/kol/credentials.json into a CI secret, and set KOL_TOKEN (and KOL_HOST during the preview):

- run: npm install -g kol-cli
- run: kol validate && kol test
  env:
    KOL_TOKEN: ${{ secrets.KOL_TOKEN }}

The token expires after 90 days; then log in again and update the secret.

Without a Kol account: the mock host

This repository includes a pretend Kol host for trying the CLI offline. Its "AI" just picks the sentences from your persona and knowledge that best match the question, so it's for checking your setup, not your AI's quality:

git clone https://github.com/royeeshemesh/kol-cli.git && cd kol-cli && npm install
npm run mock-host -w cli          # http://127.0.0.1:8787; sign-ins approve themselves
export KOL_HOST=http://127.0.0.1:8787
kol login && kol test --dir examples/cooking-helper