A PILOT HYPOTHESIS
Evaluate specialist service AI on cases with confirmed outcomes.
A fluent answer is not proof that a system identified the fault. We are exploring original service cases where an expert can verify the actual diagnosis and outcome.
Start with the failure you need to measure.
Pick one equipment family and define what your system should infer from permitted observations. Agree which information is available to the model, what remains a private holdout, and how an expert will score a response. This is a discovery proposal, not an available dataset or performance claim.
A useful case needs more than a ticket title.
- An original, nonpersonal symptom and relevant observations.
- A confirmed fault and evidence of the outcome.
- A qualified review of labels and known uncertainties.
- Clear permission covering collection, evaluation and any later use.
Prevent benchmark leakage.
Keep the answer and evaluation holdout separate from prompt inputs. Avoid near-duplicate cases spanning both partitions. Synthetic examples can explain a schema; they cannot establish performance on real field cases.
Resolve privacy and ownership first.
Customer/site identifiers, OEM manuals and confidential service material do not become saleable simply because they appear in a work order. Business verification is separate from rights and quality review. Current narrative redaction can remove useful expert text; we must confirm the actual schema survives processing before promising delivery.