Skip to main content

Using Webhook Evaluators

Sometimes, custom evaluators on Agenta may not be sufficient for your specific use case, or you may wish to host your own evaluators. In such cases, you can create a webhook evaluator to suit your specific needs. Webhook evaluators only need to comply with a predefined API.

info

For the moment, there are limitation on the security measures related to webhook evaluators. There is no authentication and no authorization, and the endpoint must be publicly available.

Accessing the Evaluator Page​

To create a webhook evaluator on Agenta, simply click on the Evaluations button in the sidebar menu, and then select the "Evaluators" tab within the Evaluations page.

Creating an Evaluator​

On the Evaluators tab, click on the "New Evaluator" button at the top right corner of your screen which would open a modal prompting you to provide the following information:

  1. Name: Enter a unique and descriptive name for your webhook evaluator.
  2. Webhook URL: Enter the public URL for your webhook evaluator.

Click on the "Create" button within the modal to confirm and complete the creation of your webhook evaluator.

Webhook API​

The input parameters are as follows:

  1. inputs: A dictionary containing the inputs to the webhook evaluators.
  2. output: The output generated by the LLM app.
  3. correct_answer: The expected correct answer for the LLM app.

The webhook request body has the following signature:

{
    "inputs": { "{input_1}": <String> [,"{input_2}": <String>, ...] }
    "output": <String>,
    "correct_answer": <String>
}

The output parameters are as follows:

  1. score: A float value corresponding to the score of the evaluation. The score should be between 0 and 1, with 0 meaning the evaluation failed, with its lowest score, and 1 meaning the evaluation succeeded, with its highest score.

The webhook response body has the following signature:

{
    "score": <Float>
}

The webhook response status must be 200 for successful evaluations, the webhook response Content-Type header should be set to application/json, and the webhook response body must be a properly-formatted JSON object.