calculus.scoring package¶
Submodules¶
calculus.scoring.kappa module¶
- filename
sppas.src.calculus.scoring.kappa.py
- author
Brigitte Bigi
- contact
- summary
Kappa estimator.
Inter-observer variation can be measured in any situation in which two or more independent observers are evaluating the same thing. The Kappa statistic seems the most commonly used measure of inter-rater agreement in Computational Linguistics.
Kappa is intended to give the reader a quantitative measure of the magnitude of agreement between observers.
- class calculus.scoring.kappa.sppasKappa(p=[], q=[])[source]¶
Bases:
object
Inter-observer variation estimation.
The calculation is based on the difference between how much agreement is actually present (“observed” agreement) compared to how much agreement would be expected to be present by chance alone (“expected” agreement).
Imagine a situation in which annotators have to answer Yes or No to 5 questions.
Person “P” answered: Yes, No, No, Yes, Yes
Person “Q” answered: Yes, No, Yes, Yes, Yes
This results in the following vectors of probabilities:
>>> p = [(1., 0.), (0., 1.), (0., 1.), (1., 0.), (1., 0.)] >>> q = [(1., 0.), (0., 1.), (1., 0.), (1., 0.), (1., 0.)]
The Cohen’s Kappa is then evaluated as follow:
>>> sppasKappa.check_vector(p) >>> True >>> sppasKappa.check_vector(q) >>> True >>> kappa = sppasKappa(p, q) >>> kappa.evaluate() >>> 0.54545
- __init__(p=[], q=[])[source]¶
Create a sppasKappa instance with two lists of tuples p and q.
>>> p=[(1., 0.), (1., 0.), (0.8, 0.2)]
- Parameters
p – a vector of tuples of float values
q – a vector of tuples of float values
- static check_vector(v)[source]¶
Check if the vector is correct to be used.
- Parameters
v – a vector of tuples of probabilities.
- evaluate()[source]¶
Estimate the Cohen’s Kappa between two lists of tuples p and q.
The tuple size corresponds to the number of categories, each value is the score assigned to each category for a given sample.
- Returns
float value
calculus.scoring.ubpa module¶
- filename
sppas.src.calculus.scoring.ubpa.py
- author
Brigitte Bigi
- contact
- summary
Estimates the Unit Boundary Positioning Accuracy.
- calculus.scoring.ubpa.ubpa(vector, text, fp=<colorama.ansitowin32.StreamWrapper object>, delta_max=0.04, step=0.01)[source]¶
Estimate the Unit Boundary Positioning Accuracy.
- Parameters
vector – contains the list of the delta values.
text – one of “Duration”, “Position Start”, …
fp – a file pointer
delta_max – Maximum delta duration to print result (default: 40ms)
step – Delta time (default: 10ms)
- Returns
(tab_neg, tab_pos) with number of occurrences of each position