We're thrilled to announce Annotation Rubrics, our latest LangSmith feature designed to streamline guided annotation workflows. Whether you're collaborating with subject-matter experts (SMEs) or non-technical teams, creating high-quality ground truth is now simpler.

What's New?

Queue-Level Instructions: Add custom instructions that annotators can see in the sidebar for every run. This ensures clarity and consistency across your evaluation process.
Feedback Tags: Define which feedback keys annotators will see in the sidebar, giving you complete control over how feedback is structured.
Key Descriptions: Add detailed descriptions for each feedback key or category, making your rubrics clear and actionable.

Why annotation rubrics are useful:

• Better ground truth creation: Guided rubrics reduce ambiguity, helping SMEs produce accurate and consistent annotations.

• Enhanced trace revision: Clear feedback structures simplify identifying issues in LLM evaluations, leading to faster and more reliable iterations.

• Effortless collaboration: Non-technical teams can now onboard and annotate with confidence, bridging the gap between technical workflows and domain expertise.

Learn more in our docs: https://docs.smith.langchain.com/evaluation/how_to_guides/annotation_queues#annotation-rubric