chisel
llm prompt eval and benchmarking tool
Chisel is a standlone, open-source tool that helps you evaluate LLM prompt effectiveness for a given task. It uses a grid based interface, inspired by Anthropic’s old Workbench tool.
It allows you to setup prompt permutations, gates, and web requests, that build out an executable DAG that allows you to evaluate prompt output at every step of the way. Like Depot, it also leverages CEL forumla expressions for working with JSON request output, giving you robust authoring and validation control.
Users can set up text templates and then thread in calculated data, request output, or previous DAG output as input parameters to templates, allowing the development of complex workflows for running evals.
It was primarily developed as a part of making invoke.world.
