Fine-tuning

Train a Jepela model of your own on your labelled cases. It is used only if it answers cases it never saw better than the standard engine and still listens to memory as well, and only you can use it.

Memory or fine-tuning?

Put facts about people and accounts in memory. Let training cases teach the pattern. A fine-tuned model reads memory like the standard one, so the two work together.

1. Add labelled cases

Your golden set is the training data (see Quality): each case is a situation, your questions, and the right answers.

curl -s $JEPELA_BASE_URL/v1/golden -H "Authorization: Bearer $JEPELA_API_KEY" -H "Content-Type: application/json" -d '{
  "cases": [
    {"id": "t-001", "state": "Where is my order 88213?",
     "questions": {"intent": {"type": "choice", "instructions": "What does the customer want?",
       "criteria": {"order_status": "where an order is", "billing": "charges and refunds", "other": "none of these"}}},
     "expected": {"intent": "order_status"}}
  ]}'

At least 20 cases are needed. More is better: a few hundred real cases from your own work say far more than 20.

A case can name a subject (and values, derive, memory options, as in a request). Jepela then trains on the case exactly as it is answered: with the lines that subject's memory recalls for it (with the same recall a decision makes, the questions' words included unless the case's memory turns question_words off), and the computed facts, placed before the state, and without the options the subject's exclusion rules remove. A case whose expected answer such a rule removes is refused with a 422 that names it: a decision never gives that answer. An array is trained as it is read: cut from its start, and when it is too long to read with its lines, without the oldest elements Jepela leaves out so that the lines are read (see State), as many as the case's method makes it leave out (a choice it matches takes no room). Held-out cases are judged the same way. Cases whose right answer depends on memory teach the model to use it.

2. Train

curl -s $JEPELA_BASE_URL/v1/finetune -H "Authorization: Bearer $JEPELA_API_KEY" -H "Content-Type: application/json" \
  -d '{"base": "english"}'

base is english (default), multilingual or typed-decisions. The answer is the job: job, model (the name your model will have), cases, held_out, state: running. One job runs at a time per account.

A job starts only when it can run. GET /v1/finetune lists in bases the ones the engine serves now; a base it does not serve is refused with a 422 that names the bases it has. A golden case stored before Jepela began refusing something it holds (such as a question's depends_on) is refused with a 422 that names the case; send it again corrected, or delete it. Training shares the GPU that answers decisions: before a job the engine gives back the memory it keeps from answering (it also does that after a quiet minute), and when the GPU still has too little free the job is refused with a 409 that says how much is free; try again in a few minutes.

• One case in five (chosen by its id, so the same cases every time) is held out. Jepela trains on the rest.
• Then it answers the held-out cases twice: with the standard engine and with your new model.
• Two tests, both must pass:

1. Accuracy: on the held-out cases, your model is right more often than the standard engine.
2. Memory: memory still moves its answers at least as much as the standard engine's. Jepela asks a fixed set of paired cases (the same situation with and without one memory line), plus your held-out cases that have memory, both ways. Your model's average effect may be at most 0.05 lower, with at most one fewer strong effect (0.2 or more).

• If either fails, the model is discarded and nothing changes. A model trained only on cases without memory can learn to ignore it; the second test catches that.

3. Follow the job

GET /v1/finetune/<job> (or GET /v1/finetune for all of them): state is running, saving, done or failed (with error, in plain words: the GPU ran out of memory, or the base is not on the engine; anything else says the training failed and that the engine's log has the details). When done, result has won, accuracy_won, memory_kept, base_accuracy, tuned_accuracy, base_memory_effect, tuned_memory_effect, held_out_questions, the log loss of both, and training_seconds; usable is true when your model is in service. cases_with_memory counts the cases trained with memory lines, and warnings lists anything to look at, such as cases that seem to hold personal data.

The console at app.jepela.com has the same under Fine-tuning: your case count, a Train button, your jobs and your models.

4. Use your model

Name it in any request:

{"model": "jepela-ft-acme-1a2b3c4d", "state": "I was charged twice.", "questions": {...}}

GET /v1/models lists it for your account only. Memories, rules, windows and every other option work as with the standard models. Another account naming it gets unknown model.

Delete a model

curl -s $JEPELA_BASE_URL/v1/finetune/delete -H "Authorization: Bearer $JEPELA_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "jepela-ft-acme-1a2b3c4d"}'

Its files are deleted and its name stops working. Delete the model of a job that failed, or whose model was discarded, the same way: the engine keeps every job's training cases, with the memory lines recalled for them, until its model name is deleted. Deleting your account deletes all your models and every job's training cases.

Deleting training cases

When you delete golden cases (POST /v1/golden/delete), every model in use that trained on them is marked: the answer lists them in models_trained_on_deleted_cases, and the job gets a warning. What a model learned from a case cannot be taken out of it. If a case was deleted because a person asked to be forgotten, train again without it and delete the old model.

Good to know

• The engine has to be running to train. Training takes about a minute per hundred cases.
• Training cases with an email address or phone number get a warning: keep such facts in memory.
• Thirteen held-out cases are a small test. Judge a model on your own golden set too: POST /v1/golden/run with {"model": "jepela-ft-..."}.
• Fine-tuning is free during the preview; you pay for the requests your model answers, like any other.