Parse
Validate every submission against the canonical JSON schema.
GANTTBENCHSCHEMA 0.1EARLY-STAGE
A benchmark for schedules that have to work
Can an AI build
a schedule that
survives reality?
GanttBench evaluates machine-readable project schedules—not just plausible timeline pictures. It checks whether tasks, calendars, dependencies, milestones, deadlines, and replans remain structurally valid.
Specimen / C-001
Schedule under examination
StrictPASS
Task / outputW01W02W03W04W05W06
01Requirements
02Data design
03Build
04Validation
05Launch
Current canonical layer
Small enough to inspect. Explicit enough to reproduce.
03system classes defined
09global integrity checks
08requirement types
01Method
Brief in. Evidence out.
An evaluation you can execute, not debate.
A convincing bar chart can still hide impossible dates, missing work, broken dependencies, or a replan that quietly drops constraints.
GanttBench converts each case into explicit predicates and applies them to a canonical schedule. The primary score uses deterministic checks—no embeddings and no LLM judge.
Match
Resolve required tasks one-to-one with stable keys and curated aliases.
Examine
Run global integrity checks and case-specific requirement predicates.
Report
Return scores, failures, explanations, and machine-readable evidence.
Global integrity / 09 checks
The structure gets cross-examined.
Every submission faces the same artifact-level invariants before its case-specific requirements are scored.
- 01Unique task IDsPASS / FAIL
- 02Benchmark calendarPASS / FAIL
- 03Valid referencesPASS / FAIL
- 04Dependency DAGPASS / FAIL
- 05Hierarchy DAGPASS / FAIL
- 06Milestone semanticsPASS / FAIL
- 07Evaluable datesPASS / FAIL
- 08Working-day startsPASS / FAIL
- 09Dependency timingPASS / FAIL
System taxonomy
Capability, separated from tooling.
Three system classes share one canonical contract, so assistance from a scheduler is visible instead of being folded into the model score.
01
modelDirect model
The model translates a project brief directly into the canonical schedule schema.
02
solver_assistedSolver-assisted
A model formulates the constraints; a scheduling engine computes the plan.
03
artifact_agentArtifact agent
An agent creates or edits a native project artifact before an adapter emits the canonical schema.
Status / 2026Today, the repository implements the shared canonical schedule layer. Native-file adapters and broader track runners are planned.
Pass first. Diagnose second.
Measure the plan, not the picture.
Strict success is the headline: every hard check must pass. Partial scores remain visible as diagnostics, never as a substitute for validity.
- Strict success
- Every hard check passes.
- Requirements score
- Case predicates and applicable scope policies.
- Integrity score
- Nine artifact-level invariants.
- Overall score
- A weighted diagnostic summary.
evaluation_report.jsoncanonical / v0.1
{
"strict_success": true,
"score": 1.0,
"requirements_score": 1.0,
"integrity_score": 1.0,
"checks": [
{
"id": "integrity:dependency_timing",
"passed": true,
"severity": "hard",
"evidence": { "errors": [] }
}
]
}Constraint-based acceptance
Valid schedules do not have to look identical.
Cases can evaluate explicit predicates—duration ranges, dependency semantics, deadlines, and preservation—instead of whole-file equality. Different plans can pass when they satisfy the same declared constraints.
Schedule APASS
≠
Schedule BPASS
Available now
A small foundation, built to grow.
- Canonical schedule schema v0.1
- Deterministic evaluator and CLI
- Creation and replan examples
- Automated test suite
On the research agenda
- 01 Resource capacity
- 02 Critical path + float
- 03 Native-file adapters
- 04 Batch evaluation
- 05 Visual rubrics
- 06 Dataset splits + statistics
The benchmark is taking shape.
Build the schedule. Then prove it works.
Explore the evaluator, run the example cases, and help define the next layer of rigorous project-schedule evaluation.
Explore the repository