arthur_bench.scoring.qa_quality.QAQualityCorrectness#
- class arthur_bench.scoring.qa_quality.QAQualityCorrectness(llm: BaseChatModel | None = None)#
Given an input question, context string, and model generation, determine if the generation produced a correct answer.
- __init__(llm: BaseChatModel | None = None)#
Methods
__init__([llm])arun(candidate_outputs[, reference_outputs, ...])Async version of run method.
arun_batch(candidate_batch[, ...])Reference batch is not used for this scoring method, QA correctness requires an input_text_batch and context_batch
All possible values returned by the scorer if output type is categorical.
from_dict(config)Load a scorer from a json configuration file.
Whether the scorer is continuous or categorical.
name()Get the name of this Scorer :return: the Scorer name
True if scorer requires reference output to compute score, False otherwise
run(candidate_outputs[, reference_outputs, ...])Score a set of test cases.
run_batch(candidate_batch[, ...])Reference batch is not used for this scoring method, QA correctness requires an input_text_batch and context_batch
to_dict([warn])Provides a json serializable representation of the scorer.
to_metadata()type()Supplies whether a scorer is built-in or custom.
validate_batch(candidate_batch[, ...])