老框架，新部件：用 Jev 让概率模型自己建、自己更新
Old frameworks, a new part: letting probability models build and update themselves with Jev
用大量是否命题建一个领域模型，用 Jev 估计概率，把现实中发生的事件作为证据输入，实时更新对未来的预测。这类模型已有四十多年的成熟框架，没有普及是因为建模和维护太贵。Jev 能把其中两个最贵的环节自动化：估计概率，以及把文字消息转成证据。
Build a domain model from many yes/no propositions, have Jev estimate the probabilities, feed real-world events in as evidence, and update forecasts in real time. Mature frameworks for this have existed for more than forty years; they never spread because building and maintaining models was too expensive. Jev can automate the two most expensive steps, estimating probabilities and turning written messages into evidence.
前三篇文章（《概率门》《概率门实验》《Jev 能用来做什么》）推到最后，可以归结为四点：
- 用大量相互独立的是否命题，构建一个特定领域的预测模型；
- 由 Jev 提供快速、便宜的概率判断；
- 现实中不断发生的事件作为输入，变成已确定的事实；
- 模型据此动态、实时地更新对未来的预测。
这类模型并不是新东西，相关的框架已经有四十多年历史。这篇文章说明这些框架是什么、为什么没有普及，以及 Jev 改变了其中哪两个环节。
四点各自对应的成熟框架
| 四点 | 对应的框架 | 过去的瓶颈 | Jev 能做什么 |
|---|---|---|---|
| 用是否命题构建领域模型 | 贝叶斯网络（Pearl，1988）、影响图（Howard 与 Matheson，1980 年代）、故障树、马尔可夫逻辑网络（2006） | 要领域专家逐条梳理命题和依赖关系，模型做不大 | 大模型列出命题，Jev 判断「A 会直接影响 B 吗」 |
| 快速的概率判断 | 专家概率估计：SHELF、Cooke 的经典模型（按专家过去的准确度加权）、德尔菲法 | 专家贵且慢，几百个参数要开好几轮会 | 每次约 0.2 秒、成本很低，可以问上万次；按准确度加权对应于校准 |
| 现实事件作为输入 | 贝叶斯网络中的证据；不确定的观察用 Pearl 的虚拟证据或 Jeffrey 规则处理 | 新闻、邮件、报告要人读完，再判断哪个命题因此成立 | Jev 判断「这条消息是否说明 A 已经发生」，给出概率，作为证据输入 |
| 动态、实时预测 | 动态贝叶斯网络、隐马尔可夫模型、卡尔曼滤波、粒子滤波 | 状态随时间变化的规律也要人来设定 | 转移概率同样可以由 Jev 估计 |
检验预测质量也有现成方法：Brier 分数、校准曲线、预测聚合，以及 Tetlock 关于超级预测者的研究和预测市场。
其中故障树和概率门的类比最接近：它就是一张由与门、或门组成的图，每个输入带一个故障概率，在核电和航天的风险评估中已经用了几十年。
为什么这些框架没有普及
方法本身是成熟的，主要用在医疗诊断、设备故障排查、核电风险评估、信用评分这类价值高、变化慢的领域。没有走得更远，原因是建模和维护的成本：领域专家要花几周梳理结构、估计参数，情况一变又要重新来过。所以过去只有少数足够重要、足够稳定的问题，才值得建一个概率模型。
Jev 改变的两个环节
第一，估计概率。过去最贵的是让专家给出几百上千个条件概率。在实验里，Jev 在 33 组人口普查数据上的排序相关约为 0.97–0.98，用少量真实数据校准后，估计比例的误差约为 0.04。用 Jev 填参数的贝叶斯网络，加上从数据统计的基础比例，回答 34 个问题的平均误差是 0.046，约为直接问 Jev 的一半，而且所有回答彼此自洽。
第二，把文字消息转成证据。现实中的变化大多以文字出现：一封邮件、一段聊天、一条新闻。过去要人读完再更新模型，所以概率模型很难做到实时。Jev 可以对每条消息、每个相关命题问一次「这条消息是否说明它已经发生」，得到的概率作为软证据输入模型，按 Jeffrey 规则更新。一条消息涉及十几个命题，也只需要一次请求。
这两个环节自动化之后，建一个模型从几周变成几分钟，更新从定期人工变成每条消息进来就更新。
一个例子
以「这个项目能否按期交付」为例（以下数字为示意）：
- 模型里有十几个命题：需求是否已冻结、关键成员是否稳定、供应商是否按时、测试环境是否就绪、客户验收标准是否明确等，以及它们之间的依赖。
- 初始状态下，模型给出「按期交付」的概率是 69%，并列出这个结论最依赖的三个判断。
- 一封邮件进来：「供应商通知，接口交付推迟两周。」Jev 判断这封邮件说明「供应商延期」的概率为 0.93，同时对其他命题几乎没有影响。
- 模型更新后，「按期交付」降到 42%。界面显示变化的原因是哪封邮件，经过了哪几个命题；还显示如果「测试环境提前就绪」，概率能回到多少。
这里的每个数字都来自同一个分布，可以追溯到具体的判断和具体的消息。项目结束后，实际结果回填，用来校准下一次的判断。
和直接问大模型的区别
直接把项目情况和这封邮件交给大模型，也能得到一个判断。区别在于：
- 模型里所有概率彼此自洽，换一种问法不会得到不同的答案；
- 每次变化都能说清是哪条消息、经过哪些命题造成的；
- 可以做假设分析，答案即时得到，不需要重新调用模型；
- 每个判断都能用回填的结果检验和校准。
近两年已有研究用大模型生成影响因素，再用贝叶斯模型计算概率，例如 2024 年提出的 BIRD 框架，方向与此一致。Jev 这类专门做判断的模型更快、更便宜，所以模型可以做得更大，也可以随每条消息更新。
边界
- 所有参数来自同一个 Jev，系统偏差是共有的，需要用真实结果校准；能从数据统计的基础比例，直接用数据。
- 命题和依赖关系可能出错，结构错误比参数误差影响更大。
- 一条消息是否真的说明某个命题成立，Jev 也会判断错；软证据比硬性认定更稳妥，但同样需要检验。
- 开放问题在结果揭晓前无法校准，概率主要用于比较和排序。
下一步
我们正在把这套方法做成一个产品原型：输入一个问题，自动建出模型；接入消息流，概率随消息实时变化，并说明原因。验证方式是在已有结果的历史项目和公开预测题上，比较它和直接询问大模型的准确度与校准度。
说明：Jev 通过 Vercel AI Gateway 按量付费调用。作者和榫卯 Tenon AI 与 TypeSafe AI 没有合作关系，也没有收到任何费用。
The previous three posts ("Probability gates", "Probability gate experiments" and "What Jev can be used for") come down to four points:
- Build a prediction model for a specific domain from many independent yes/no propositions.
- Have Jev supply fast, cheap probability judgments.
- Feed events that keep happening in the real world in as inputs that become established facts.
- Update forecasts about the future dynamically and in real time.
This kind of model is not new; the frameworks behind it are more than forty years old. This post explains what those frameworks are, why they never spread, and which two steps Jev changes.
Mature frameworks for each point
| Point | Framework | Past bottleneck | What Jev can do |
|---|---|---|---|
| Build a domain model from yes/no propositions | Bayesian networks (Pearl, 1988), influence diagrams (Howard and Matheson, 1980s), fault trees, Markov logic networks (2006) | Domain experts had to work out propositions and dependencies one by one, so models stayed small | A language model lists the propositions; Jev judges "does A directly affect B?" |
| Fast probability judgments | Expert elicitation: SHELF, Cooke's classical model (weighting experts by past accuracy), the Delphi method | Experts are expensive and slow; a few hundred parameters take several rounds of meetings | About 0.2 seconds per call at very low cost, tens of thousands of times; weighting by accuracy corresponds to calibration |
| Real events as inputs | Evidence in Bayesian networks; uncertain observations handled with Pearl's virtual evidence or Jeffrey's rule | News, emails and reports had to be read by a person, who then decided which propositions now held | Jev judges "does this message mean A has happened?" and gives a probability that goes in as evidence |
| Dynamic, real-time forecasting | Dynamic Bayesian networks, hidden Markov models, Kalman filters, particle filters | How states change over time also had to be specified by people | Jev can estimate transition probabilities too |
There are also established methods for checking forecast quality: Brier scores, calibration curves, forecast aggregation, Tetlock's research on superforecasters, and prediction markets.
Fault trees are the closest match to the probability gate analogy: a fault tree is a diagram of AND and OR gates in which each input carries a failure probability, and it has been used for decades in nuclear and aerospace risk assessment.
Why these frameworks never spread
The methods are mature and are used mainly in high-value, slow-changing areas such as medical diagnosis, equipment troubleshooting, nuclear risk assessment and credit scoring. They went no further because of the cost of building and maintaining models: domain experts spend weeks working out the structure and estimating parameters, and when circumstances change the work starts over. So only a few problems that were important and stable enough were ever worth a probability model.
The two steps Jev changes
First, estimating probabilities. The most expensive part used to be getting experts to supply hundreds or thousands of conditional probabilities. In our experiments, Jev reached a rank correlation of about 0.97–0.98 on 33 census groups, and after calibration with a small amount of real data its estimated shares were off by about 0.04. A Bayesian network with parameters from Jev and base rates counted from the data answered 34 questions with a mean error of 0.046, about half that of asking Jev directly, and all of its answers were mutually consistent.
Second, turning written messages into evidence. Most real-world change arrives as text: an email, a chat message, a news item. A person used to have to read it and then update the model, which is why probability models rarely ran in real time. Jev can ask, for each message and each relevant proposition, "does this message mean it has happened?", and the probability goes into the model as soft evidence, updated with Jeffrey's rule. Even a message that touches a dozen propositions needs only one request.
With these two steps automated, building a model takes minutes instead of weeks, and updates happen whenever a message arrives instead of periodically by hand.
An example
Take "will this project be delivered on time?" (the numbers below are illustrative):
- The model has a dozen or so propositions: whether requirements are frozen, whether key team members are stable, whether the supplier is on schedule, whether the test environment is ready, whether the client's acceptance criteria are clear, and so on, plus the dependencies between them.
- At the start, the model gives a 69% probability of on-time delivery and lists the three judgments that conclusion depends on most.
- An email arrives: "The supplier says the interface delivery is delayed by two weeks." Jev judges that the email means "the supplier is late" with probability 0.93 and has almost no bearing on the other propositions.
- After the update, on-time delivery drops to 42%. The interface shows which email caused the change and which propositions it passed through, and how far the probability would recover if "the test environment is ready early".
Every number here comes from one distribution and can be traced to specific judgments and specific messages. When the project ends, the actual result is fed back to calibrate the next round of judgments.
How this differs from asking a language model directly
Giving the project details and this email to a language model also produces a judgment. The differences:
- All probabilities in the model are mutually consistent; a different wording does not give a different answer.
- Every change can be explained by which message caused it and which propositions it passed through.
- What-if analysis is instant, with no need to call a model again.
- Every judgment can be checked and calibrated against the results fed back later.
In the past two years, research has used language models to generate contributing factors and Bayesian models to compute probabilities, for example the BIRD framework proposed in 2024, which points in the same direction. Judgment models such as Jev are faster and cheaper, so the models can be larger and can update with every message.
Limits
- All parameters come from the same Jev, so its systematic bias is shared and needs calibrating against real results; base rates that can be counted from data should come from the data.
- The propositions and dependencies can be wrong, and structural errors matter more than parameter errors.
- Jev can also misjudge whether a message really means a proposition holds; soft evidence is safer than a hard ruling, but it also needs checking.
- Open questions cannot be calibrated before they resolve, so their probabilities are mainly useful for comparison and ranking.
Next
We are turning this approach into a product prototype: enter a question and a model is built automatically; connect a message stream and the probabilities change in real time, with the reasons shown. We will test it on historical projects with known outcomes and on public forecasting questions, comparing its accuracy and calibration with asking a language model directly.
Note: Jev was called and paid for per use through the Vercel AI Gateway. The author and Tenon AI have no business relationship with TypeSafe AI and have received no payment.