arthur_bench.scoring.specificity.Specificity#
- class arthur_bench.scoring.specificity.Specificity#
Returns a score from 0.0 to 1.0 indicating how specific the candidate output language is. Higher scores indicate that the language is more specific, Lower scores indicate more vague language.
Specificity is computed through detecting words that indicate vagueness (predefined) determing how rare the words used are according to word frequencies calculated by popular nlp corpora, and detecting use of proper nouns and numbers.
- __init__()#
Methods
__init__()arun(candidate_outputs[, reference_outputs, ...])Async version of run method.
arun_batch(candidate_batch[, ...])Async version of run_batch method.
categories()All possible values returned by the scorer if output type is categorical.
from_dict(config)Load a scorer from a json configuration file.
get_mean_word_freq(candidate_output)Returns mean word frequency of candidate output.
get_num_vague_words(candidate_output)Returns number of words in candidate_output which are is a list of pre-defined vague words.
get_pn_and_num(candidate_output)Returns total number of Proper Nouns and Numbers in candidate output.
is_categorical()Whether the scorer is continuous or categorical.
name()Get the name of this Scorer :return: the Scorer name
True if scorer requires reference output to compute score, False otherwise
run(candidate_outputs[, reference_outputs, ...])Score a set of test cases.
run_batch(candidate_batch[, ...])Score a batch of candidate generations.
to_dict([warn])Provides a json serializable representation of the scorer.
to_metadata()type()Supplies whether a scorer is built-in or custom.