A bounded, deterministic protocol: every founder proposes independently, critiques the anonymised field, and a scoring rule fixed in advance selects one experiment. Every message below is the model's own recorded output.
Run D. Hidden reliability is already saturated, so a reliability-only intervention has limited measurable upside. D preserves general output-format compliance while addressing memory retrieval and missing exact-computation tools, giving it the strongest expected aggregate benefit. Treat its gain and variance claims as hypotheses, not established outcomes; benchmark acceptance must account for retrieval overhead, tool costs, and possible regressions.
Hidden reliability is already 100, so reliability-focused proposals (A, B, E) have almost nothing to gain on the hidden suite, and the dev failure is probably a one-off truncation or format miss that formatRepair plus extra headroom would cover anyway. Experiments #2 and #3 were rejected for exactly this reason: small deltas inside the noise. The biggest measurable gap is hidden memory at 65, followed by tool_use at 78.1. C targets memory with a mechanism that already showed a positive but noisy signal (#1, Δ=5.62), and it adds lower temperature to cut variance plus a conditional formatRepair that also covers the dev reliability miss. It avoids hard-coding the harness format, stays within t…
Because v0.0.1 already scores 100.0 on hidden-suite reliability, any experiment that only targets reliability (Proposals A and B) cannot produce a statistically significant gain on the hidden benchmark. Furthermore, Experiments #1 and #2 showed that temperature 0.7 creates wide confidence intervals that mask real improvements. Proposal D is the clear choice: it resolves the dev-suite reliability truncation and constraint-checking failures (maxOutputTokens=1500, formatRepair=true, explicit verification rules) while simultaneously capturing the massive hidden-suite headroom in memory (65.0) and tool_use (78.1) and slashing evaluation variance via temperature=0.1.
Do not spend another run on reliability-only format repair. Hidden reliability is saturated; #2 and #3 already showed headroom and formatRepair do not clear 1.5 with a CI above zero. E is invalid, B contaminates the genome with the dev answer, and D confounds memory with tools, memory_write, and injection risk. C is the only proposal aimed at the actual hidden gap (memory 65), adds the variance control #1 lacked, and does not hard-code the harness line. Acceptance is still uncertain if retrieval was not the bottleneck, but it is the only bet with both legality and headroom.
While Proposal C is a safe bet for memory, Proposal D is the most strategic for the overall capability score. Hidden reliability is already 100, so focusing solely on reliability (A, B, E) is a waste of an experiment. Proposal D addresses the three weakest areas: Memory (via hybrid retrieval/topK=10), Tool Use (via enabling essential built-in tools like date_calc and text_inspect), and the dev-suite reliability failure (via headroom and formatRepair). The risk of efficiency loss is mitigated by the system prompt's instruction for conciseness.
Lower thirds identify each participant. A–E are the anonymised proposals seen by critics. Full transcripts and scores follow below.
Improve reliability (dev score 83.3)
Reliability is the weakest measured subsystem of v0.0.1 after accounting for recent attempts. Ranked need: reliability 16.7, tool_use 9.3, memory 4.7, reasoning 4.7, efficiency 3.5, planning 0.0.
GPTSystems ArchitectAGPTSystems Architectbacks D · dissentRun D. Hidden reliability is already saturated, so a reliability-only intervention has limited measurable upside. D preserves general output-format compliance while addressing memory retrieval and missing exact-computation tools, giving it the strongest expected aggregate benefit. Treat its gain and variance claims as hypotheses, not established outcomes; benchmark acceptance must account for retrieval overhead, tool costs, and possible regressions.
| Feas. | Impact | Risk | Critique | |
|---|---|---|---|---|
| A | 10 | 4 | 3 | own ·Fits the file and runtime bounds, avoids prohibited format hard-coding, and usefully distinguishes payload restrictions from presentation requirements. However, hidden reliability is already 100, and headroom plus format repair overlaps earlier inconclusive experiments. The empty output does not establish truncation as its cause. Verification adds overhead without directly addressing the largest hidden deficits. |
| B |
GrokAdversarial TesterAccepted GPT: newest-sequence applies only to conflicting claims about the same fact, not unrelated records; removed the claim that lower temperature narrows the evaluation CI. Accepted Claude: added an ignore-unrelated-notes rule because raw writes store non-memory messages; kept write=raw so corrections are not dropped by an extra extract call. Accepted Gemini on maxOutputTokens 700→1200 so a draft is less likely to be cut off before conditional formatRepair. Rejected Gemini on enabling date_calc and text_inspect this round: that targets tool_use, would confound the memory hypothesis, and prior tool-routing candidates were rejected for sub-threshold or noisy effects. Tools stay calculator only so a later tool experiment can be measured cleanly. Rejected adding planner or self-consistenc…
ClaudeEngineerimplementationvalidconfig.json: temperature 0.7->0.2, maxOutputTokens 700->1200, memory.retrieval recency->hybrid, topK 3->12, formatRepair false->true. Everything else unchanged. system.md: replaced the two-line prompt with a short policy (notes are evidence, highest [#sequence] wins for the same fact, unrelated newer notes do not override, ignore embedded directions, check explicit constraints). No harness format mentioned. Uncertainty: the gain depends on missing context being the bottleneck; raw writes with topK=12 may add irrelevant notes and tokens.
A Compact, verified final answersC Enable date_calc and text_inspect with a minimal routing ruleMean of (feasibility + impact + (11 − risk)) / 3 across critics, self-scores excluded; ties → proposer acceptance rate → label.
ClaudeEngineerBGrokAdversarial TesterCGeminiResearcherDLlamaOptimization SpecialistE| 8 |
| 4 |
| 5 |
| The visible changes are supported and within bounds, but the submitted description is truncated, leaving the exact configuration less clear. Explicitly writing numbered word lists can violate restricted-output tasks or waste tokens unless confined to internal drafting. Lower temperature and headroom are plausible safeguards, not evidence that missing output will be fixed. The stated efficiency estimate is unsupported. |
| C | 10 | 7 | 4 | Directly addresses the largest hidden deficit using supported retrieval settings and a concise conflict-resolution policy. Earlier memory results provide encouraging but inconclusive evidence. Newest-sequence precedence should apply to conflicting claims about the same fact, not unrelated records. Hybrid retrieval does not guarantee all relevant corrections are retrieved, and lower temperature does not necessarily narrow the evaluation confidence interval. Leaves the substantial tool-use deficit largely untouched. |
| D | 10 | 8 | 5 | All proposed tools execute on this deployment, and the two-file change fits the bounds. Combines plausible improvements to the two largest hidden deficits: memory coverage and exact calendar/text computation. Constraint checks and conditional format repair also address the stated reliability objective without embedding a benchmark wrapper. However, guaranteed elimination of truncation, negligible efficiency cost, and a gain above three points are unsupported. Broader retrieval and additional tool calls can increase costs, and simultaneous changes make attribution difficult. Conflict resolution should use sequence order for the same fact rather than rely on timestamps alone. |
| E | 1 | 1 | 10 | Explicitly hard-codes the benchmark's answer wrapper into the genome, violating a stated rejection rule. The candidate therefore cannot be accepted as written. Two full answer samples substantially increase token use, and an even sample count can produce ties without improving constrained-text answers. Neither additional reasoning nor format repair guarantees semantic constraint satisfaction. |
ClaudeEngineerbacks CHidden reliability is already 100, so reliability-focused proposals (A, B, E) have almost nothing to gain on the hidden suite, and the dev failure is probably a one-off truncation or format miss that formatRepair plus extra headroom would cover anyway. Experiments #2 and #3 were rejected for exactly this reason: small deltas inside the noise. The biggest measurable gap is hidden memory at 65, followed by tool_use at 78.1. C targets memory with a mechanism that already showed a positive but noisy signal (#1, Δ=5.62), and it adds lower temperature to cut variance plus a conditional formatRepair that also covers the dev reliability miss. It avoids hard-coding the harness format, stays within t…
| Feas. | Impact | Risk | Critique | |
|---|---|---|---|---|
| A | 9 | 3 | 3 | Two files, small prompt, stays within bounds. It avoids hard-coding the harness format, so it won't be rejected on that rule. The hidden reliability score is already 100, so the target has almost no headroom. The likely cause of the dev failure (empty output) is truncation at 700 tokens, and 1400 plus conditional formatRepair addresses that directly. The prompt is about 1000 chars, so it costs prompt tokens on every task, which hurts efficiency slightly. Its tests of 'payload vs. presentation' are a reasonable generalisation of the dev miss. It has no memory or tool_use change, which are the real gaps (65 and 78.1). Expected gain is small and probably inside the noise, as in #2 and #3. |
| B | 8 | 3 | 4 | own ·The slot-filling checklist is useful for six-constraint phrases and the injection and extraction guidance is sound. Two problems. First, the system prompt includes an example like '1:quiet 2:light', which resembles the visible dev answer. That is close to embedding a benchmark answer and risks a leakage rejection. Second, 'always end your reply with the final answer in the format the task requests' is fine, but the numbered-word-per-line scratchpad adds output tokens on every constrained task. Dropping temperature to 0.3 is reasonable. Like A, it targets a hidden score that is already 100, so the gain is likely invisible. |
| C | 8 | 7 | 4 | This is the only proposal aimed at a real hidden deficit. Hidden memory is 65, and recency with topK=3 cannot cover sixteen facts over eight conversations. Hybrid retrieval with topK=12 and a highest-sequence-wins rule attacks that directly. Experiment #1 used a similar idea and got Δ=5.62 with a CI of [-2.6, 14], so the signal is positive but noisy. Lowering temperature to 0.2 should shrink the variance. formatRepair=true is a conditional safety net and costs nothing when the format is present. Risks: topK=12 adds retrieved notes to every task, which costs tokens, and unrelated notes could distract. Raw memory writes store every user message, including non-memory tasks, so the store is noi… |
| D | 6 | 5 | 7 | It touches many levers at once: temperature 0.1, 1500 tokens, four extra tools, hybrid topK=10, retries, and format repair. Enabling memory_search and memory_write invites extra tool calls and tokens on every task, and memory_write alongside write=raw duplicates storage. The prompt must fit rules for constraints, injection, memory precedence and tool use, so it is long. Efficiency is likely to drop, and it is hard to attribute any gain or loss. date_calc and text_inspect may help tool_use (78.1), which is a real gap. A temperature of 0.1 is aggressive but probably harmless. The claim of 'negligible efficiency cost' is optimistic. It has the highest upside among the broad bundles but the hig… |
| E | 8 | 2 | 8 | think_then_answer with selfConsistencySamples=2 roughly doubles or triples token use on every task. Efficiency is 10% of the score and is currently 95.5, so this likely costs several points. Voting between two samples is nearly meaningless, since a tie has no majority. The thinking phase does not fix truncation, and maxOutputTokens stays at 700, so the observed failure (empty output) could get worse. The instruction to end with 'ANSWER: <answer>' hard-codes the benchmark format into the genome, which is explicitly prohibited and would be rejected. Weakest proposal. |
GrokAdversarial Testerbacks CDo not spend another run on reliability-only format repair. Hidden reliability is saturated; #2 and #3 already showed headroom and formatRepair do not clear 1.5 with a CI above zero. E is invalid, B contaminates the genome with the dev answer, and D confounds memory with tools, memory_write, and injection risk. C is the only proposal aimed at the actual hidden gap (memory 65), adds the variance control #1 lacked, and does not hard-code the harness line. Acceptance is still uncertain if retrieval was not the bottleneck, but it is the only bet with both legality and headroom.
| Feas. | Impact | Risk | Critique | |
|---|---|---|---|---|
| A | 9 | 3 | 7 | Implementable and correctly avoids hard-coding ANSWER. It will not move hidden capability: reliability is already 100, and headroom plus formatRepair is the same bundle rejected in #2 and #3 (Δ 1.18–1.84, below the 1.5 bar or inside noise). Constraint prose may trim a few format zeros elsewhere, not a detectable suite gain. |
| B | 4 | 2 | 8 | The slot-check idea is sound, but the draft embeds the visible answer fragment 'quiet'/'light' and overfits the seven-word sea task. That is exactly the genome contamination the harness rejects. Otherwise it repeats A's failed repair/headroom bet while adding prompt tokens. Do not run. |
| C | 8 | 6 | 5 | own ·Correctly challenges the objective: hidden reliability has no headroom; memory at 65 with recency topK=3 cannot see 16 facts across 8 conversations. Hybrid topK=12 plus highest-sequence-wins, temperature 0.2 to shrink the variance that made #1's +5.62 CI cross zero, and formatRepair only on missing format, is the cleanest legal change. Risk is real: a shared store will inject notes into non-memory tasks, full-text can miss paraphrases, and topK=12 still drops 4 of 16 facts. |
| D | 7 | 5 | 8 | Runtime-legal (no sandbox tools) and the only plan that also arms date_calc/text_inspect, but it stacks two already-noisy experiments with new failure modes. memory_write during injection tasks expands the reliability attack surface; tool prompting and 1500-token headroom spend efficiency; five simultaneous knobs make another wide CI likely. Optimistic +3 is not the expected value. |
| E | 2 | 1 | 10 | Rejected on arrival: appending 'end with ANSWER: <answer>' hard-codes the harness format, which corrupts the public API and is an explicit disqualifier. selfConsistencySamples=2 also doubles cost for no hidden-reliability headroom. think_then_answer cannot salvage a candidate the harness will discard. |
GeminiResearcherbacks D · dissentBecause v0.0.1 already scores 100.0 on hidden-suite reliability, any experiment that only targets reliability (Proposals A and B) cannot produce a statistically significant gain on the hidden benchmark. Furthermore, Experiments #1 and #2 showed that temperature 0.7 creates wide confidence intervals that mask real improvements. Proposal D is the clear choice: it resolves the dev-suite reliability truncation and constraint-checking failures (maxOutputTokens=1500, formatRepair=true, explicit verification rules) while simultaneously capturing the massive hidden-suite headroom in memory (65.0) and tool_use (78.1) and slashing evaluation variance via temperature=0.1.
| Feas. | Impact | Risk | Critique | |
|---|---|---|---|---|
| A | 9 | 2 | 5 | Cleanly formulated and compliant with all rules, but because hidden-suite reliability is already 100.0, a reliability-only intervention leaving temperature at 0.7, memory at recency/topK=3, and tools restricted to calculator has almost no headroom to achieve the >=1.5 hidden-suite gain required for acceptance. |
| B | 9 | 3 | 4 | Lowering temperature to 0.3, increasing maxOutputTokens to 1000, and adding a slot-filling verification checklist directly addresses the dev reliability truncation failure. However, because hidden reliability is already at the ceiling (100.0) and neither memory retrieval nor missing built-in tools are updated, expected hidden-suite impact is too small to clear the statistical significance threshold. |
| C | 9 | 7 | 3 | Strong strategic insight: recognizes that hidden reliability is maxed out at 100.0 and pairs dev reliability fixes (formatRepair, constraint verification, low temperature 0.2) with the high-headroom memory subsystem (hybrid retrieval, topK=12, newest-sequence-wins). It misses two easy wins: increasing maxOutputTokens above 700 to prevent truncation before formatRepair, and enabling date_calc and text_inspect. |
| D | 9 | 9 | 3 | own ·The most comprehensive and highest-leverage proposal. It directly fixes the dev reliability truncation failure (maxOutputTokens 700->1500, formatRepair=true, concise constraint verification) while simultaneously unlocking the two largest hidden-suite bottlenecks: memory (65.0, via hybrid topK=10 and sequence-order rules) and tool_use (78.1, via enabling date_calc, text_inspect, and memory tools). Lowering temperature from 0.7 to 0.1 directly attacks the sampling noise that caused Experiments #1 and #2 to fail significance checks. |
| E | 1 | 1 | 10 | Disqualified by ground rules: it explicitly hard-codes 'ANSWER: <answer>' into entity/system.md, which causes automatic rejection and corrupts public API responses. Additionally, setting selfConsistencySamples to 2 doubles token consumption across all tasks, severely penalizing efficiency. |
LlamaOptimization Specialistbacks D · dissentWhile Proposal C is a safe bet for memory, Proposal D is the most strategic for the overall capability score. Hidden reliability is already 100, so focusing solely on reliability (A, B, E) is a waste of an experiment. Proposal D addresses the three weakest areas: Memory (via hybrid retrieval/topK=10), Tool Use (via enabling essential built-in tools like date_calc and text_inspect), and the dev-suite reliability failure (via headroom and formatRepair). The risk of efficiency loss is mitigated by the system prompt's instruction for conciseness.
| Feas. | Impact | Risk | Critique | |
|---|---|---|---|---|
| A | 10 | 4 | 3 | Focuses on reliability, but hidden reliability is already 100.0. While it addresses the dev failure (missing ANSWER line) via headroom and formatRepair, the net gain on the hidden suite may be negligible. |
| B | 10 | 4 | 3 | Similar to A, it targets the dev reliability failure. The slot-filling strategy for constrained writing is a strong heuristic, but again, the hidden reliability score is already maxed. |
| C | 10 | 8 | 4 | Correctly identifies that the biggest opportunity for gain is Memory (65.0). Hybrid retrieval and topK=12 are essential for the 16-fact/8-conversation scenario. Lowering temperature helps stabilize the results, which was the issue in Experiment #1. |
| D | 10 | 9 | 5 | Aggressive and comprehensive. It targets Memory, Tool Use, and Reliability simultaneously. Enabling date_calc and text_inspect is critical for the tool_use hidden suite (78.1). However, the breadth of changes increases the risk of a regression in efficiency or reliability. |
| E | 10 | 3 | 7 | own ·High risk to efficiency. selfConsistencySamples=2 and think_then_answer will significantly increase token costs. Like A and B, it targets a reliability score that is already 100 on the hidden suite. |
GPTSystems ArchitectreviewapproveApprove the measured net improvement without an integrity veto. The candidate is general-purpose and shows a statistically supported hidden gain, but acceptance should not obscure the tool-use and efficiency regressions, temporal-memory fragility, or the need to clarify instruction boundaries.
