Welcome to Datumo Eval

Datumo Eval is an end-to-end evaluation platform that automates quality, reliability, and safety assessments for LLM-based AI services.
From domain-specific query generation to automated scoring, result analysis, and red-teaming, Datumo provides a consistent workflow covering the entire model evaluation lifecycle.

Getting Started with Datumo Eval

New to Datumo Eval? Use the guides below to get up and running quickly.

⏱ Quick StartView Tutorials

Why Datumo Eval?

• Automated evaluation dataset generation
  Upload your documents and automatically generate Queries or Expected Responses, reducing preparation time significantly.

• Built-in and Custom Metrics
  Use pre-built evaluators for Safety, RAG Quality, and Factuality, or design your own domain-specific evaluation metrics.

• Flexible Evaluation Execution
  Datumo supports human review, algorithmic scoring, and LLM-judge–based automatic evaluation within a single platform.

• Claim-Level Factuality & RAG Checker
  Responses are decomposed into granular claims to assess factual accuracy and retrieval quality with precision.

• Automated red-teaming for safety evaluation
  Generate adversarial prompts automatically to test safety-violation risks and perform repeatable, consistent safety assessments using judge prompts.

Feature Overview

📊 Overview

Visualize model performance at a glance across versions, including strengths, weaknesses, and key metrics.

🗂️ Dataset Management

Create evaluation queries that mirror real-world service usage and manage model responses systematically.

Upload CSV/Excel files to build and version your datasets.

🤖 LLM-Judge Evaluation Judgment Evaluation

Automatically score hundreds or thousands of responses using predefined rules and metrics.
Ideal for efficient, repeatable large-scale evaluations.

🧑‍⚖️ Human Evaluation Human Evaluation

Conduct rubric-based qualitative assessments or evaluate model responses interactively through live conversations.

📐 Quantitative Evaluation Quantitative Evaluation

Automatically measure model performance using benchmark datasets and reference-based metrics.

⚙️ Workspace Settings (Models · Metrics · Environment)

Connect models, define custom metrics, and manage access control to operate secure evaluation workflows.

Explore the Docs

Introduction

1. About Datumo Eval
2. Core Concepts
3. Glossary

Workspace & Settings

1. Workspace & Roles
2. Metric Management
3. Model & Agent Management

Evaluation Workflows

1. Datasets
2. Judge Evaluation
3. Human Evaluation
4. Quantitative Evaluation