Regression testing, hotkeys, and more in LangSmith

• Improved Regression Testing Experience: When making changes to your LLM application, it’s important to understand whether or not behavior has regressed compared to your prior test cases. Changes to the prompt, retrieval strategy, or model choice can have big implications on the responses produced by your application. We’ve updated our Comparison View to make it easier to explore data across multiple experiments:

  • We’ve released more display options so you can customize the granularity of information you want to see. View the columns, datapoints, and charts you find important.

  • Automatically highlight test runs that have increased or decreased on your evaluation metric compared to the baseline, and filter to show only the ones that deviate.

  • An improved side panel view allows you to dive into the details of specific runs that interest you.

Check out our video walkthrough and blog to learn more!

We also have a few new additional updates in LangSmith:

• Hotkeys in Annotation Queue: We’ve introduced hotkeys in Annotation Queues to help you navigate more quickly !  Look for hotkey indicators next to buttons that now support keyboard shortcuts.

• Mustache Support: You now have the option to switch the template language for your prompts in the Playground and Prompt Hub between f-string and Mustache. This gives you more flexibility in how you manage and format your variables.

• Evaluations: Our new video series shows you how LangSmith Experiments can help you add testing coverage, spot regressions, and make informed tradeoffs amongst latency, cost, and quality. We’ve released three more concepts focused on RAG evaluation, showing you how to evaluate response quality: correctness (Video, Docs), hallucinations (Video, Docs), and retrieved document relevance (Video, Docs).

• Azure Marketplace: We’re thrilled to announce that LangSmith is now available in the Azure Marketplace. When you purchase LangSmith through the Azure Marketplace with your Azure credits, you’ll keep data fully contained in your Azure VPC, get ease of deployment, and experience a smoother procurement process. Learn more on our blog and contact sales to start a conversation with one of our experts.