calculus.scoring package

Submodules

calculus.scoring.kappa module

filename

sppas.src.calculus.scoring.kappa.py

author

Brigitte Bigi

contact

develop@sppas.org

summary

Kappa estimator.

Inter-observer variation can be measured in any situation in which two or more independent observers are evaluating the same thing. The Kappa statistic seems the most commonly used measure of inter-rater agreement in Computational Linguistics.

Kappa is intended to give the reader a quantitative measure of the magnitude of agreement between observers.

class calculus.scoring.kappa.sppasKappa(p=[], q=[])[source]

Bases: object

Inter-observer variation estimation.

The calculation is based on the difference between how much agreement is actually present (“observed” agreement) compared to how much agreement would be expected to be present by chance alone (“expected” agreement).

Imagine a situation in which annotators have to answer Yes or No to 5 questions.

  • Person “P” answered: Yes, No, No, Yes, Yes

  • Person “Q” answered: Yes, No, Yes, Yes, Yes

This results in the following vectors of probabilities:

>>> p = [(1., 0.), (0., 1.), (0., 1.), (1., 0.), (1., 0.)]
>>> q = [(1., 0.), (0., 1.), (1., 0.), (1., 0.), (1., 0.)]

The Cohen’s Kappa is then evaluated as follow:

>>> sppasKappa.check_vector(p)
>>> True
>>> sppasKappa.check_vector(q)
>>> True
>>> kappa = sppasKappa(p, q)
>>> kappa.evaluate()
>>> 0.54545
__init__(p=[], q=[])[source]

Create a sppasKappa instance with two lists of tuples p and q.

>>> p=[(1., 0.), (1., 0.), (0.8, 0.2)]
Parameters

  • p – a vector of tuples of float values

  • q – a vector of tuples of float values

check()[source]

Check if the given p and q vectors are correct to be used.

Returns

bool

static check_vector(v)[source]

Check if the vector is correct to be used.

Parameters

v – a vector of tuples of probabilities.

evaluate()[source]

Estimate the Cohen’s Kappa between two lists of tuples p and q.

The tuple size corresponds to the number of categories, each value is the score assigned to each category for a given sample.

Returns

float value

set_vectors(p, q)[source]

Set the vectors of probabilities to estimate the sppasKappa value.

Parameters

  • p – a vector of tuples of float values

  • q – a vector of tuples of float values

sqm()[source]

Estimate the Euclidian distance between two vectors.

Returns

row, col

sqv()[source]

Estimate the Euclidian distance between two vectors.

Returns

v

calculus.scoring.ubpa module

filename

sppas.src.calculus.scoring.ubpa.py

author

Brigitte Bigi

contact

develop@sppas.org

summary

Estimates the Unit Boundary Positioning Accuracy.

calculus.scoring.ubpa.ubpa(vector, text, fp=<colorama.ansitowin32.StreamWrapper object>, delta_max=0.04, step=0.01)[source]

Estimate the Unit Boundary Positioning Accuracy.

Parameters

  • vector – contains the list of the delta values.

  • text – one of “Duration”, “Position Start”, …

  • fp – a file pointer

  • delta_max – Maximum delta duration to print result (default: 40ms)

  • step – Delta time (default: 10ms)

Returns

(tab_neg, tab_pos) with number of occurrences of each position

Module contents