---
translation_of: probability-gates-experiments
lang: en
title: "Probability gate experiments: can Jev's probabilities be used as they are?"
summary: Three experiments with the real Jev test whether its probabilities are self-consistent, whether it can serve as a source of conditional probabilities, and whether composing an answer in steps beats asking directly. It ranks well but pulls values toward the middle, and a small amount of real data is enough to calibrate it; composition by itself does not improve accuracy, and bias in a shared direction passes straight through.
---

The previous post, "[Probability gates: from bits to probabilities](/blog/probability-gates/)", proposed treating Jev as a probability gate and listed several ideas to test with the real Jev. This post records the first results.

The three experiments answer three questions:

1. Are Jev's probabilities self-consistent?
2. Can it serve as a source of conditional probabilities?
3. Is splitting a question into steps and combining them in code more accurate than asking directly?

Jev was called through the Vercel AI Gateway. The three experiments used 161 requests and about 100,000 input tokens, for a total cost of about $0.004. The median server-side time per call was 211 ms, with 95% of requests under 334 ms, consistent with the 70–500 ms the company states.

## Experiment 1: are the probabilities self-consistent?

The data is 40 customer-service messages we wrote ourselves, covering common cases such as shipping, refunds, invoices and faults. For each message we asked about four propositions: A, the customer is expressing dissatisfaction; B, the customer is asking for a refund; C, the issue involves shipping; D, the customer is upset. This experiment needs no labels; it only checks whether Jev's answers obey the rules of probability.

**Asking both ways is mostly consistent.** For A, B and C, asking "yes" and "no" separately gave probabilities whose sum differed from 1 by about 0.04 on average, with a mean close to 0, so there was no visible tendency to answer yes. D was the exception, at 0.14, because of how the question was written: "upset" and "calm" are not strict opposites; "a little annoyed" sits between them. A differential gate needs two questions that are strict complements.

**Compound questions are mostly inconsistent.** For four pairs of propositions we asked separately for P(A), P(B), P(A and B) and P(A or B). In 81% of the 160 pairs at least one rule of probability was violated. The most common violation was P(A or B) being lower than the larger of P(A) and P(B), in 69% of pairs. Asking "A or B" directly gave 0.54 on average; computing A + B − (A and B) from Jev's own marginals and conjunctions gave 0.64. Two other wordings ("either one is enough" and "1 − neither holds") were still 0.10–0.12 too low. So Jev systematically underestimates "or"; it is not a wording problem.

**Questions in the same request do not affect each other.** Asking 16 questions in one request versus one question per request changed the results by 0.006 on average and 0.03 at most.

**The output is almost deterministic.** Asking the same question again gave an identical answer nearly half the time, with an average difference of 0.01. This means Jev's errors are mainly systematic; repeating the same question does not help.

What this means for design:

- A coherence restorer is needed; the violation rate is high.
- Compounds such as "or" should be computed in code from marginals and conditional probabilities, not asked directly.
- Redundancy has to come from different wordings, split evidence or different models, not repeated calls.
- One request can safely ask many questions at once.

## Experiment 2: can it serve as a source of conditional probabilities?

The previous post argued that Bayesian networks never caught on mainly because their conditional probabilities had to be filled in by experts, and that if Jev can supply those numbers cheaply, the approach may become workable again. Testing this needs real frequencies.

The data is UCI Adult, a public extract of the 1994 US census with 32,561 people. I grouped them by sex, education and age band and dropped groups with fewer than 30 people, leaving 33 groups and 18,350 people. The true value for each group is the share earning more than $50,000 a year, ranging from 1.5% to 87.5%. For each group Jev was asked three ways: directly ("pick one person at random; do they earn more than $50,000 a year?"), both ways and combined, and as a share on a 10-level scale.

| Wording | Mean absolute error | Mean bias | Rank correlation | Mean absolute error after calibration |
| --- | --- | --- | --- | --- |
| Ask directly | 0.148 | +0.131 | 0.966 | 0.064 |
| Ask both ways, combined | 0.142 | +0.118 | 0.972 | 0.060 |
| Ask for the share on a 10-level scale | 0.119 | +0.100 | 0.984 | 0.042 |

![33 groups: Jev's estimated share of high earners vs the true share](/blog/assets/probability-gates-experiments/x9-calibration.en.svg)

Three conclusions:

- **The ranking is almost entirely right.** Measured by Spearman correlation, all three wordings are around 0.97: Jev judges well which groups have higher shares and which have lower.
- **The values are pulled toward the middle.** Groups with a true share below 20% were overestimated by 0.16 on average. For example, women aged 25–34 who stopped at 11th grade have a true share of 1.5%; Jev estimated 15.5%. Groups above 50% were slightly underestimated.
- **A little data is enough to calibrate.** A linear calibration in logit space, fitted for each group on the other 32 groups and then used to predict that group, cut the error of the 10-level wording from 0.119 to 0.042.

In other words, Jev gives the right ranking and the rough shape, and the actual values need calibrating with a small amount of real data. This supports two ideas from the previous post, the source of conditional probabilities and the calibrator, but using the numbers as they are, with no data at all, does not hold up.

This experiment has three limits: it covers only one domain; the Adult dataset is well known, so the model may have seen related statistics; and $50,000 is a 1994 figure, so if the model reads it at today's prices, that may be part of why it overestimates.

## Experiment 3: is composing in steps more accurate than asking directly?

This experiment tests building a circuit itself. The target is the share of high earners in 8 groups (sex × four age bands from 25 to 64, 25,655 people), estimated two ways:

- **Ask directly**: one scale question for the group's share.
- **Compose in steps**: a multiple-choice question for the education mix and five scale questions for the share in each education group, then a sum in code: Σ P(education) × P(high income | education).

To see which step the error comes from, I also replaced each of the two gates with the true values, and applied the calibration from experiment 2 to both routes.

![8 groups: mean absolute error by route](/blog/assets/probability-gates-experiments/x10-routes.en.svg)

| Route | Mean absolute error | Mean bias |
| --- | --- | --- |
| Ask directly | 0.099 | +0.099 |
| Compose in steps | 0.093 | +0.088 |
| True education mix + Jev shares by education | 0.089 | +0.087 |
| Jev education mix + true shares by education | 0.023 | −0.005 |
| Ask directly + calibration | 0.047 | −0.047 |
| Compose in steps + calibration | 0.053 | −0.053 |

- **Composing in steps was not more accurate than asking directly**: 0.093 against 0.099.
- **Almost all the error comes from the shares-by-education step.** As in experiment 2, it pulls values toward the middle, overestimating the low-income education groups by 0.12–0.13. All five education groups are off in the same direction, so the errors do not cancel in the sum.
- **The education-mix step has a sizeable error on its own.** It overestimated high school and bachelor's degrees and underestimated "some college" and graduate degrees, with a total variation distance of 0.29 from the true mix. But it barely affects the combined result, because the over- and underestimated groups have similar income shares, so the errors cancel in the sum.
- **Calibration helps both routes**, bringing the error down to about 0.05, but it overcorrects and the bias turns negative. This calibration was fitted on fine-grained cells (sex × education × age) and does not fully carry over to the larger merged groups.

So splitting into steps does not by itself improve accuracy. Whether errors cancel or survive in a composition depends on their direction: errors in different directions cancel, and bias in a shared direction passes straight through. Shared bias can only be handled by calibration, and the calibration is best fitted on data at the same level. This matches the revised view in the previous post: probabilistic circuits are closer to analog circuits and have to be designed to an error budget. With only 8 targets, the conclusion is directional.

## Summary

What holds:

- Jev ranks very well; all three wordings reach a rank correlation of about 0.97 on the 33 census groups.
- A small amount of real data is enough to calibrate the values to a usable level.
- One request can ask a dozen or more questions with little interaction between them; a call takes about 0.2 seconds and costs very little.

What needs correcting:

- Asking compound propositions directly, especially "or", is unreliable; they should be computed in code.
- Jev's output is almost deterministic and its errors are systematic, so redundancy has to come from diversity.
- Composing in steps does not automatically improve accuracy; shared bias needs calibration, and the calibration has to match the level it is used at.

## Next

1. **Error accumulation with depth**: chain judgments so each depends on the previous conclusion, and measure the error at each stage and the correlation between stages.
2. **Forward forecasting**: record Jev's probabilities on public forecasting questions that resolve after its release, and compare them with prediction markets and frontier language models.
3. **Other domains**: repeat experiment 2 with Chinese data closer to business use, to test whether calibration carries over across domains.

---

Note: Jev was called and paid for per use through the Vercel AI Gateway. The customer-service messages were written by us, and the census data comes from the public UCI repository. The author and Tenon AI have no business relationship with TypeSafe AI and have received no payment.