arthur_bench.scoring.bertscore.BERTScore#

class arthur_bench.scoring.bertscore.BERTScore(model_type='microsoft/deberta-v3-base', precision_weight=0.1)#

Tailored bert score implementation.

https://arxiv.org/abs/1904.09675

__init__(model_type='microsoft/deberta-v3-base', precision_weight=0.1)#

Tailored bert score implementation.

Parameters:

  • model_type – the underlying language model to extract embeddings from

  • precision_weight – the weight given to the precision term in calculating bertscore

Methods

__init__([model_type, precision_weight])

Tailored bert score implementation.

arun(candidate_outputs[, reference_outputs, ...])

Async version of run method.

arun_batch(candidate_batch[, ...])

Async version of run_batch method.

categories()

All possible values returned by the scorer if output type is categorical.

from_dict(config)

Load a scorer from a json configuration file.

is_categorical()

Whether the scorer is continuous or categorical.

name()

Get the name of this Scorer :return: the Scorer name

requires_reference()

True if scorer requires reference output to compute score, False otherwise

run(candidate_outputs[, reference_outputs, ...])

Score a set of test cases.

run_batch(candidate_batch[, ...])

Score a batch of candidate generations.

to_dict([warn])

Provides a json serializable representation of the scorer.

to_metadata()

type()

Supplies whether a scorer is built-in or custom.