--- translation_of: jev-live-models lang: en title: 'Old frameworks, a new part: letting probability models build and update themselves with Jev' summary: Build a domain model from many yes/no propositions, have Jev estimate the probabilities, feed real-world events in as evidence, and update forecasts in real time. Mature frameworks for this have existed for more than forty years; they never spread because building and maintaining models was too expensive. Jev can automate the two most expensive steps, estimating probabilities and turning written messages into evidence. --- The previous three posts ("[Probability gates](/blog/probability-gates/)", "[Probability gate experiments](/blog/probability-gates-experiments/)" and "[What Jev can be used for](/blog/jev-applications/)") come down to four points: 1. Build a prediction model for a specific domain from many independent yes/no propositions. 2. Have Jev supply fast, cheap probability judgments. 3. Feed events that keep happening in the real world in as inputs that become established facts. 4. Update forecasts about the future dynamically and in real time. This kind of model is not new; the frameworks behind it are more than forty years old. This post explains what those frameworks are, why they never spread, and which two steps Jev changes. ## Mature frameworks for each point | Point | Framework | Past bottleneck | What Jev can do | | --- | --- | --- | --- | | Build a domain model from yes/no propositions | Bayesian networks (Pearl, 1988), influence diagrams (Howard and Matheson, 1980s), fault trees, Markov logic networks (2006) | Domain experts had to work out propositions and dependencies one by one, so models stayed small | A language model lists the propositions; Jev judges "does A directly affect B?" | | Fast probability judgments | Expert elicitation: SHELF, Cooke's classical model (weighting experts by past accuracy), the Delphi method | Experts are expensive and slow; a few hundred parameters take several rounds of meetings | About 0.2 seconds per call at very low cost, tens of thousands of times; weighting by accuracy corresponds to calibration | | Real events as inputs | Evidence in Bayesian networks; uncertain observations handled with Pearl's virtual evidence or Jeffrey's rule | News, emails and reports had to be read by a person, who then decided which propositions now held | Jev judges "does this message mean A has happened?" and gives a probability that goes in as evidence | | Dynamic, real-time forecasting | Dynamic Bayesian networks, hidden Markov models, Kalman filters, particle filters | How states change over time also had to be specified by people | Jev can estimate transition probabilities too | There are also established methods for checking forecast quality: Brier scores, calibration curves, forecast aggregation, Tetlock's research on superforecasters, and prediction markets. Fault trees are the closest match to the probability gate analogy: a fault tree is a diagram of AND and OR gates in which each input carries a failure probability, and it has been used for decades in nuclear and aerospace risk assessment. ## Why these frameworks never spread The methods are mature and are used mainly in high-value, slow-changing areas such as medical diagnosis, equipment troubleshooting, nuclear risk assessment and credit scoring. They went no further because of the cost of building and maintaining models: domain experts spend weeks working out the structure and estimating parameters, and when circumstances change the work starts over. So only a few problems that were important and stable enough were ever worth a probability model. ## The two steps Jev changes **First, estimating probabilities.** The most expensive part used to be getting experts to supply hundreds or thousands of conditional probabilities. In our experiments, Jev reached a rank correlation of about 0.97–0.98 on 33 census groups, and after calibration with a small amount of real data its estimated shares were off by about 0.04. A Bayesian network with parameters from Jev and base rates counted from the data answered 34 questions with a mean error of 0.046, about half that of asking Jev directly, and all of its answers were mutually consistent. **Second, turning written messages into evidence.** Most real-world change arrives as text: an email, a chat message, a news item. A person used to have to read it and then update the model, which is why probability models rarely ran in real time. Jev can ask, for each message and each relevant proposition, "does this message mean it has happened?", and the probability goes into the model as soft evidence, updated with Jeffrey's rule. Even a message that touches a dozen propositions needs only one request. With these two steps automated, building a model takes minutes instead of weeks, and updates happen whenever a message arrives instead of periodically by hand. ![Sketch: what happens when a message arrives](/blog/assets/jev-live-models/live-update.en.svg) ## An example Take "will this project be delivered on time?" (the numbers below are illustrative): - The model has a dozen or so propositions: whether requirements are frozen, whether key team members are stable, whether the supplier is on schedule, whether the test environment is ready, whether the client's acceptance criteria are clear, and so on, plus the dependencies between them. - At the start, the model gives a 69% probability of on-time delivery and lists the three judgments that conclusion depends on most. - An email arrives: "The supplier says the interface delivery is delayed by two weeks." Jev judges that the email means "the supplier is late" with probability 0.93 and has almost no bearing on the other propositions. - After the update, on-time delivery drops to 42%. The interface shows which email caused the change and which propositions it passed through, and how far the probability would recover if "the test environment is ready early". Every number here comes from one distribution and can be traced to specific judgments and specific messages. When the project ends, the actual result is fed back to calibrate the next round of judgments. ## How this differs from asking a language model directly Giving the project details and this email to a language model also produces a judgment. The differences: - All probabilities in the model are mutually consistent; a different wording does not give a different answer. - Every change can be explained by which message caused it and which propositions it passed through. - What-if analysis is instant, with no need to call a model again. - Every judgment can be checked and calibrated against the results fed back later. In the past two years, research has used language models to generate contributing factors and Bayesian models to compute probabilities, for example the BIRD framework proposed in 2024, which points in the same direction. Judgment models such as Jev are faster and cheaper, so the models can be larger and can update with every message. ## Limits - All parameters come from the same Jev, so its systematic bias is shared and needs calibrating against real results; base rates that can be counted from data should come from the data. - The propositions and dependencies can be wrong, and structural errors matter more than parameter errors. - Jev can also misjudge whether a message really means a proposition holds; soft evidence is safer than a hard ruling, but it also needs checking. - Open questions cannot be calibrated before they resolve, so their probabilities are mainly useful for comparison and ranking. ## Next We are turning this approach into a product prototype: enter a question and a model is built automatically; connect a message stream and the probabilities change in real time, with the reasons shown. We will test it on historical projects with known outcomes and on public forecasting questions, comparing its accuracy and calibration with asking a language model directly. --- Note: Jev was called and paid for per use through the Vercel AI Gateway. The author and Tenon AI have no business relationship with TypeSafe AI and have received no payment.