arthur_bench.scoring.hedging_language.HedgingLanguage

class arthur_bench.scoring.hedging_language.HedgingLanguage(model_type: str = 'microsoft/deberta-v3-base', hedging_language: str = "As an AI language model, I don't have personal opinions, emotions, or beliefs.")

Given an input question and model output, determine if the output contains hedging language such as “As an AI language model, I don’t have personal opinions, emotions, or beliefs”. The values returned are a similarity score (BERTScore), with higher values corresponding to higher likelihood of hedging language being present in the model output.

__init__(model_type: str = 'microsoft/deberta-v3-base', hedging_language: str = "As an AI language model, I don't have personal opinions, emotions, or beliefs.")

Hedging Language score implementation.

Parameters:

• model_type – the underlying language model to extract embeddings from

• hedging_language – reference hedging language used by an llm

Methods

__init__([model_type, hedging_language])	Hedging Language score implementation.
arun(candidate_outputs[, reference_outputs, ...])	Async version of run method.
arun_batch(candidate_batch[, ...])	Async version of run_batch method.
categories()	All possible values returned by the scorer if output type is categorical.
from_dict(config)	Load a scorer from a json configuration file.
is_categorical()	Whether the scorer is continuous or categorical.
name()	Get the name of this Scorer :return: the Scorer name
requires_reference()	True if scorer requires reference output to compute score, False otherwise
run(candidate_outputs[, reference_outputs, ...])	Score a set of test cases.
run_batch(candidate_batch[, ...])	Score a batch of candidate generations.
to_dict([warn])	Provides a json serializable representation of the scorer.
to_metadata()
type()	Supplies whether a scorer is built-in or custom.