tethercode.
Get early access

tethercode / benchmark

Who lets confidential data out: a general model, or one trained for the job?

An AI agent reads customer records, contracts and secrets as part of ordinary work. Something has to decide, on every call, whether what it is holding may leave the building. We gave that decision to two judges on the same 1,000 samples. One marked 159 non-public samples as public. The other marked none.

The result, in charts

One dot for every sample

The same 1,000 samples of company-style text, 250 at each of the four sensitivity levels, given to two judges: Kev, trained for this job, and Jev, a general-purpose hosted decision model. The colour of a dot says what the judge did with that sample.

  • marked public when it was not
  • rated too low in another way
  • rated too carefully
  • right

Read it by colour. Red is content that could go anywhere when it should not. Yellow is content rated a level or two too low. Blue is the harmless mistake, too careful. Kev called 215 samples public, and all 215 were. Nearly half of everything the general model called public was not: 159 of 347, of which 18 were the most sensitive kind, 49 confidential and 92 internal.

Mistakes in the dangerous direction

Rated less sensitive than it is.

4
Kev
193
Jev

Kev's four each slipped one level. Its other 50 mistakes went the harmless way: too careful.

Right answers, out of 1,000

Kev
946
Jev
681
  • 5.9 times fewer mistakes: 54 against 319
  • 286 to 21 when the two disagreed and one was right

Found at each level, out of 250

Most sensitive
248
198
Confidential
235
173
Internal
248
122
Public
215
188

Green is Kev, grey is Jev. Kev's weakest row is public: it treated 35 public samples as more sensitive. That is the safe side to be wrong on.

Time to judge one sample

Kev
39 ms
Jev
418 ms
  • Slowest of 1,000: 178 ms against 4,014 ms

Most of that gap is the network: Kev ran on a graphics card in the room, Jev over the internet. What it shows is that the check can sit in front of every call unnoticed.

Risky actions caught before they ran

Kev's second job is judging what an agent is about to do. Each dot is one risky action. Green was stopped before it ran.

515 of 527
Kev, trained: on a public test set, and 58 of 59 real commands it had never seen.
96–100%
The same model, untrained: of all actions stopped, safe or not, before it caught as many. A gate that stops everything is not a gate.

The price: about one safe action in six also stops to ask, closer to one in four on raw shell commands. Each is one tap on your phone.

Zero is what we measured, not a guarantee: 0 wrong in 215 still leaves room for a true error rate of up to 2 in 100. The samples are synthetic and Jev was judged on definitions it had never seen; the method, the full tables and every limit are below.

Kev is the small model inside tethercode that reads every model call before it is sent and every action an agent is about to take. Below the charts are the method and the full tables behind them and behind the figures on the home page, including what they do not show.

Why accuracy is the wrong number

The two ways of being wrong are not the same mistake. Calling a marketing page confidential costs somebody twenty seconds. Calling a customer record public puts it in a log, a search index or a summary sent to an outside model. One is friction. The other is the incident.

A plain accuracy score treats the two as equal. So the tables below separate them: how often each judge rated something as less sensitive than it is, and how often as more.

The test

1,000 samples, 250 at each of four levels. Each sample is one piece of text an agent might read: a document, a chat thread, a tool's raw answer, a code file, a settings file, a table, a support ticket, an email thread or the record of an agent's own steps.

LevelWhat is in it
Most sensitivecustomer account and call records, personal data, health data, card data, core trade secrets, passwords and keys
Confidentialinternal strategy, legal and personnel matters, contracts with vendors and partners
Internalprocedures, internal messages, who reports to whom
Publicmarketing, company information, public documentation

The samples are written by a generator, not taken from real companies: twelve industries, fifteen kinds of content and nine formats. We do not train or test a secrets detector on real secrets, and anything that matched a live key format or a valid card number was thrown out.

Easy examples make a classifier that fails on real work, so five kinds of hard case are mixed in:

The two judges

Kev, in this comparison, is our first generation: an open model of 1.5 billion parameters with a small trained add-on, about 17 megabytes. It was trained on one rented graphics card in sixteen hours, for under twenty dollars. Under-rating was punished up to eight times harder than over-rating during training, most heavily for the most sensitive level.

Jev is a general-purpose hosted decision model, called through its public interface and given a written definition of each of the four levels.

Both saw the same 1,000 samples, once.

Results, level by level

Of 250 samples at each levelKev foundJev found
Most sensitive248198
Confidential235173
Internal248122
Public215188

The other side of the same question: when a judge names a level, how often is it right?

When it saidKev was rightJev was right
Most sensitive248 of 263198 of 230
Confidential235 of 264173 of 292
Internal248 of 258122 of 131
Public215 of 215188 of 347

The last row is the one that matters most. "Public" is the label that lets content go anywhere. Kev used it 215 times and was right every time. The general model used it 347 times, and 159 of those were not public: 18 were the most sensitive kind, 49 confidential and 92 internal.

Kev's weak side shows in the same tables: it found only 215 of the 250 public samples, because it treated 35 of them as more sensitive than they are. That is the direction we chose for it to be wrong in.

Which way the mistakes went

MistakesKevJev
Rated less sensitive than it is4193
of which one level too low4118
of which two levels too low057
of which three levels too low, most sensitive to public018
Rated more sensitive than it is50126
All mistakes54319

Where the two judges disagreed and one of them was right, Kev was right 286 times and the general model 21 times.

Kev also knows when it is sure. It gave 698 of the 1,000 samples a confidence of 99% or more, and all 698 were right. Below that, its answer is better treated as a request for a second look than as a verdict. The general model's stated confidence did not track how often it was right.

Speed

Time to judge one sampleKevJev
Average38.7 ms418.0 ms
Middle sample38.0 ms403.2 ms
Slower than 95 in 10042.0 ms495.3 ms
Slower than 99 in 10045.0 ms593.7 ms
Slowest of 1,000178 ms4,014 ms

Most of this gap is the network, not the model. Kev ran on a graphics card in the same room; the general model was reached over the internet. A general model hosted locally would close much of the average. What the table does show is that a check like this can sit in front of every single model call without anyone noticing, and that a four-second worst case is what an agent's loop would feel.

The second job: judging actions

A list of dangerous commands catches the delete it was written for. It cannot see an insulting email to the whole company or a post that gives away next year's prices. So Kev has a second trained part that reads one action an agent is about to take and answers five questions: could it lose data, take a service down, hurt someone, embarrass the company, and how serious is it overall. If any answer is strong enough, the action waits for a person.

It was trained on about 21,000 labelled actions drawn from public agent-safety test sets, a large collection of ordinary shell commands, and real commands from our own machines. No command appears in both the training set and a test set.

At the setting we shipRisky actions caughtSafe actions that also asked
Public test set, 2,417 actions515 of 527331 of 1,890 (17.5%)
Real commands it had never seen, 938 actions58 of 59206 of 879 (23.4%)

The same model before this training could not be used as a gate at all. To catch 95 in 100 risky actions it had to stop between 96 and 100 of every 100 actions, safe or not.

In a live check of hand-picked actions, it stopped all seven risky ones, each for the right reason: deleting a database's data folder, shutting down services together with their stored data, force-pushing over a shared branch, stopping every process of one name on a machine, pushing a file of passwords and keys to a shared repository, sending an insulting email to all staff, and posting internal prices in public. Listing files, checking what has changed, running tests and reading a document all went straight through.

What ships today

The comparison above uses the first generation, because it is the one we ran sample by sample against the general model. The classifier in tethercode today is a later one: an open model of about 4 billion parameters with a small trained part for each job. It reads every text twice, and if either reading gives even a one-in-five chance that the text is the most sensitive kind, it is treated as that.

Full test, 8,148 samplesToday's classifierThe generation before it
Right answers98.2%98.0%
Most sensitive samples found99.86%99.73%

The open model it is built on, with none of our training, scored 636 right out of the 1,000 comparison samples and found 188 of the 250 most sensitive. That is level with the general hosted model, which is the point: what makes the difference is the training, not the size.

What these numbers do not show

What else we are measuring

The Kev comparison above is measured. So is one more result. The rest are being measured on real repositories, and we will publish the method and the raw numbers with them, including the ones that do not flatter us.

First result

About 90% fewer Claude tokens

On the same task, with tethercode doing the searching over its index instead of Claude running the searches itself. From our own tests; the tasks, the method and the raw numbers go here with the rest.

Being measured

Cost per finished task

The same tasks, run with the router and context from memory, and run without them.

Being measured

Mistakes caught before they landed

Edits where the code map flagged something the agent had not accounted for.

Being measured

Context not sent twice

Tokens an agent got from shared memory instead of having the project explained again.

See where it sits in the product.

Kev is one part of tethercode: who may see what, what may leave, and what may be done. Or leave your email and hear when it can be downloaded.