Package com.google.genkit.ai.evaluation

Class Evaluator<O>

java.lang.Object
com.google.genkit.ai.evaluation.Evaluator<O>
Type Parameters:
O - the type of evaluator-specific options
All Implemented Interfaces:
Action<EvalRequest,List<EvalResponse>,Void>, Registerable
public class Evaluator<O> extends Object implements Action<EvalRequest,List<EvalResponse>,Void>
Evaluator represents an evaluation action that assesses the quality of AI outputs.

Evaluators are a core primitive in Genkit that allow you to measure and track the quality of your AI applications. They can be used to:

  • Score outputs based on various criteria (accuracy, relevance, etc.)
  • Compare outputs against reference data
  • Run automated quality checks in CI/CD pipelines
  • Monitor production quality over time

  • Field Details

  • Method Details

    • builder

      public static <O> Evaluator.Builder<O> builder()
      Creates a new Evaluator builder.
      Type Parameters:
      O - the options type
      Returns:
      a new builder

    • define

      public static <O> Evaluator<O> define(Registry registry, String name, String displayName, String definition, EvaluatorFn<O> evaluatorFn)
      Defines a new evaluator and registers it with the registry.
      Type Parameters:
      O - the options type
      Parameters:
      registry - the registry to register with
      name - the evaluator name
      displayName - the display name shown in the UI
      definition - description of what the evaluator measures
      evaluatorFn - the evaluation function
      Returns:
      the created evaluator

    • define

      public static <O> Evaluator<O> define(Registry registry, String name, String displayName, String definition, boolean isBilled, Class<O> optionsClass, EvaluatorFn<O> evaluatorFn)
      Defines a new evaluator with full options and registers it with the registry.
      Type Parameters:
      O - the options type
      Parameters:
      registry - the registry to register with
      name - the evaluator name
      displayName - the display name shown in the UI
      definition - description of what the evaluator measures
      isBilled - whether using this evaluator incurs costs
      optionsClass - the class for evaluator-specific options
      evaluatorFn - the evaluation function
      Returns:
      the created evaluator

    • getName

      public String getName()
      Description copied from interface: Action
      Returns the name of the action.
      Specified by:
      getName in interface Action<EvalRequest,List<EvalResponse>,Void>
      Returns:
      the action name

    • getType

      public ActionType getType()
      Description copied from interface: Action
      Returns the type of the action.
      Specified by:
      getType in interface Action<EvalRequest,List<EvalResponse>,Void>
      Returns:
      the action type

    • getDesc

      public ActionDesc getDesc()
      Description copied from interface: Action
      Returns the descriptor of the action containing metadata, schemas, etc.
      Specified by:
      getDesc in interface Action<EvalRequest,List<EvalResponse>,Void>
      Returns:
      the action descriptor

    • getInfo

      public EvaluatorInfo getInfo()
      Gets the evaluator info.
      Returns:
      the evaluator info

    • run

      public List<EvalResponse> run(ActionContext ctx, EvalRequest input) throws GenkitException
      Description copied from interface: Action
      Runs the action with the given input.
      Specified by:
      run in interface Action<EvalRequest,List<EvalResponse>,Void>
      Parameters:
      ctx - the action context
      input - the input to the action
      Returns:
      the output of the action
      Throws:
      GenkitException - if the action fails

    • run

      public List<EvalResponse> run(ActionContext ctx, EvalRequest input, Consumer<Void> streamCallback) throws GenkitException
      Description copied from interface: Action
      Runs the action with the given input and streaming callback.
      Specified by:
      run in interface Action<EvalRequest,List<EvalResponse>,Void>
      Parameters:
      ctx - the action context
      input - the input to the action
      streamCallback - callback for receiving streaming chunks, may be null
      Returns:
      the output of the action
      Throws:
      GenkitException - if the action fails

    • runJson

      public com.fasterxml.jackson.databind.JsonNode runJson(ActionContext ctx, com.fasterxml.jackson.databind.JsonNode input, Consumer<com.fasterxml.jackson.databind.JsonNode> streamCallback) throws GenkitException
      Description copied from interface: Action
      Runs the action with JSON input and returns JSON output.
      Specified by:
      runJson in interface Action<EvalRequest,List<EvalResponse>,Void>
      Parameters:
      ctx - the action context
      input - the JSON input
      streamCallback - callback for receiving streaming JSON chunks, may be null
      Returns:
      the JSON output
      Throws:
      GenkitException - if the action fails

    • runJsonWithTelemetry

      public ActionRunResult<com.fasterxml.jackson.databind.JsonNode> runJsonWithTelemetry(ActionContext ctx, com.fasterxml.jackson.databind.JsonNode input, Consumer<com.fasterxml.jackson.databind.JsonNode> streamCallback) throws GenkitException
      Description copied from interface: Action
      Runs the action with JSON input and returns the result with telemetry information.
      Specified by:
      runJsonWithTelemetry in interface Action<EvalRequest,List<EvalResponse>,Void>
      Parameters:
      ctx - the action context
      input - the JSON input
      streamCallback - callback for receiving streaming JSON chunks, may be null
      Returns:
      the action result including telemetry data
      Throws:
      GenkitException - if the action fails

    • register

      public void register(Registry registry)
      Description copied from interface: Registerable
      Registers this primitive with the given registry.
      Specified by:
      register in interface Registerable
      Parameters:
      registry - the registry to register with

    • getInputSchema

      public Map<String,Object> getInputSchema()
      Description copied from interface: Action
      Returns the JSON schema for the action's input type.
      Specified by:
      getInputSchema in interface Action<EvalRequest,List<EvalResponse>,Void>
      Returns:
      the input schema as a map, or null if not defined

    • getOutputSchema

      public Map<String,Object> getOutputSchema()
      Description copied from interface: Action
      Returns the JSON schema for the action's output type.
      Specified by:
      getOutputSchema in interface Action<EvalRequest,List<EvalResponse>,Void>
      Returns:
      the output schema as a map, or null if not defined

    • getMetadata

      public Map<String,Object> getMetadata()
      Description copied from interface: Action
      Returns additional metadata for the action.
      Specified by:
      getMetadata in interface Action<EvalRequest,List<EvalResponse>,Void>
      Returns:
      the metadata map