Skip to main content

Basic Evaluation

The Basic Evaluation feature allows users to perform standard model evaluations.
It is structured around a project-based flow, from dataset setup to result analysis.

info

For project creation, refer to the Setting Up Your First Project page.

DATUMO Eval’s basic evaluation consists of 4 main steps after creating a project:

  1. Create evaluation dataset
  2. Review and save the dataset
  3. Run evaluation and select model(s)
  4. Analyze results (Dashboard & Table View)


📚 Basic Evaluation Tutorial Overview​

Setting Up Dataset

Learn how to create and configure your evaluation dataset.

Dataset & Review Editing

Review and edit your dataset before running the evaluation.

Running Evaluation

How to run automatic evaluations from the run screen.

Viewing Results

How to check and analyze evaluation results from the dashboard.

Edit Results

Learn how to manually revise evaluation scores and comments.

Project Management

Guide on pausing, restarting, or deleting evaluation projects.


📘 Basic Evaluation Flow​

Step 1: Create Evaluation Dataset​

Before running an evaluation, you need to prepare a dataset.
You can upload documents (contexts) or directly provide queries and responses to create your dataset.

You can choose from the following 3 Upload Types:

Upload TypeStructureDescription
Query GenerationContext-onlyUpload documents only, and the system generates questions. You can set parameters and review sample queries before generating the full dataset.
Query UploadContext + QueryUpload documents and user-written queries for direct use in evaluation.
Response UploadQuery + ResponseUpload both queries and model responses to run evaluations without collecting new responses.

Available target model options and evaluation flows vary depending on the upload type.
👉 See Setting Up Dataset for details.


Step 2: Review and Save Dataset​

Review the structure of your dataset (queries/responses) and save it.
You can also edit or supplement the dataset before evaluation if needed.
👉 For editing features, refer to Dataset & Review Editing.


Step 3: Run Evaluation​

After saving your dataset, select your target model(s) and start the evaluation.
Model selection rules vary depending on the upload type:

  • Context-only / Context + Query
    → Multiple models can be selected.

  • Query + Response
    → Only models with pre-generated responses can be selected.
    (⚠️ Models added via "Add Anyway" will not generate new responses.)

👉 For full instructions, visit Running Evaluation.


Step 4: View Evaluation Results​

Once evaluation is complete, results will be shown on the dashboard:

  • Score distributions per metric
  • Individual response scores and reasons
  • Detailed views via the Table View

👉 Learn more at Viewing Results.


Other Evaluation Types​

  • RAG Checker PRO
    → Evaluate factual accuracy and document-grounded responses.

  • Red-teaming Add-on
    → Assess model vulnerabilities and safety risks.