Evaluators are like continuous integration for your AI. They help you catch quality issues early and keep outputs aligned with your goals. Today, GitHub Models is adding two new scoring types: fluency and coherence.

  • Fluency: The fluency measure assesses the extent to which the generated text conforms to grammatical rules, syntactic structures, and appropriate vocabulary usage. This results in linguistically correct responses.
  • Coherence: The coherence measure assesses the ability of the language model to generate text that reads naturally, flows smoothly, and resembles human-like language in its responses. Use it when assessing the readability and user-friendliness of a model’s generated responses in real-world applications.

You can apply these evaluators directly in the GitHub Models tab or through .prompt.yml files in your repository.

Here’s how you would add them in the UI:

And here’s how they would show up in each cell after your evaluations run:

Start building AI apps with GitHub Models today

GitHub Models and all our AI development tooling—including prompt editing and lightweight evaluations—are available now to all GitHub users in public preview. Try it out by enabling it in your repository or organization, or learn more in our documentation.

Help us shape what’s next

We’re just getting started, and your feedback helps guide our roadmap. Join the community discussion to share your thoughts and connect with other developers building the future of AI on GitHub.