LangSmith’s online evaluator (LLM-as-judge) automatically runs on production traces with a customizable prompt and automation rules. Our new off-the-shelf evaluators give you a custom prompt that can:

• Flag whether retrieved documents are relevant to the user input question (Document Relevance)

• Check for hallucinations in the RAG response (Hallucination)

• Check whether a RAG answer is helpful to address the question, with no ground truth answer required (Answer Helpfulness)

• Flag upon specific criteria, such as for toxicity (Tagging)

Bonus: Watch this video or check out the docs to see how to establish guardrails using LangSmith to detect user inputs with PII or toxic queries.