LLM as a Judge: Score AI Agent Outputs with Claude (2026)
A minimal, framework-free LLM-as-a-judge harness in Python on Claude: rubric design, pointwise and pairwise scoring, and fixes for position, verbosity, and self-preference bias.
Tag
24 posts tagged.
A minimal, framework-free LLM-as-a-judge harness in Python on Claude: rubric design, pointwise and pairwise scoring, and fixes for position, verbosity, and self-preference bias.
You cannot unit-test an agent like a pure function. Build a two-layer pytest harness: deterministic tool-call assertions plus an LLM-as-judge grader, a frozen eval dataset, and a CI gate. Runnable Python, no eval framework required.
Five reusable agentic workflow patterns, built from scratch in Python with runnable code and a start-simple rule.
A runnable 2026 tutorial for turning a local MCP server into a remote one over Streamable HTTP. Serve tools with FastMCP, test the endpoint with curl and MCP Inspector, validate the Origin header, add a bearer token, then connect Claude. Covers the Mcp-Session-Id requirement and the DNS-rebinding gotcha the docs warn about but most walkthroughs skip.
A runnable LangGraph human-in-the-loop tutorial for 2026: pause an agent mid-run with interrupt(), review the proposed action, then approve, edit, or reject it with Command(resume=...). Includes the checkpointer requirement and the re-run-from-the-top gotcha that breaks most tutorials.
A runnable 2026 tutorial: build a write-capable MCP server in Python with FastMCP and the official MCP SDK. Add tools with @mcp.tool(), back them with SQLite, test in the Inspector, and connect it to Claude for Desktop.
A runnable 2026 tutorial: give an AI agent web search in Python with the Claude web_search server tool. Read citations, cap cost with max_uses, filter domains, and go multi turn.
A runnable 2026 tutorial: give a Claude agent long-term memory that survives across sessions using SQLite, embeddings, and cosine retrieval. No LangChain, FAISS, or vector database.
A hands-on LangGraph tutorial for 2026: build the same tool-using AI agent two ways, with the five-line create_agent helper and as an explicit StateGraph you can customize. Fully runnable Python.
The five parts every AI agent has, a runnable Python example in about 40 lines, and an honest guide to the guardrails and framework choices most tutorials skip.
A runnable 2026 guide to custom tools in the Claude Agent SDK: define functions with tool() / @tool, bundle them into an in-process MCP server, register with allowedTools, and gate risky calls per argument with canUseTool. TypeScript and Python.
A runnable Python quickstart: install the Claude Agent SDK, stream a run with query(), then give the agent your own tools with the @tool decorator and ClaudeSDKClient. Builds a weather agent that chains two tools.