Introduction

• In recent years, many annotation software (or tools) have become available for annotation of digital audio-video data.
• For a researcher looking for an annotation software, it is difficult to decide about its usefulness and usability.
• Some are mainly dedicated to Computer Scientists (tools) and some are designed for Linguists (software).

Software selection

• The choise of all annotation software must be done carefully, and before the creation of the corpus.
• It is part of the corpus creation framework.

versus

Software selection: requirements

To decide about usefulness and usability, it is necessary to know all of the followings:

1. about the license,
2. about the ease of use,
3. about the strengths/weaknesses for specific annotation purposes,
4. about the type of data or analysis the tool/software is designed for,
5. about its compatibility with other annotated data.

Finding and evaluating appropriate software (1)

1. About the license

• prefer free and open source software:

Even if you can personally afford to pay for a licence for software you may wish to share your methodology with other students or researchers who cannot afford to buy a license.

Finding and evaluating appropriate software (2)

2. About the ease of use

• software or web-services?
• prefer multi-platform software:
  • different scientific communities tend to use Mac OS, Windows or Unix platforms.
  • multi-platform software makes sharing between such communities much easier.

• GUI or CLI usability: prefer usable software!

If the software requires the help of an engineer each time you need to use it, this will be a serious limitation on your usage.

Finding and evaluating appropriate software (3)

3. About the strengths/weaknesses for specific annotation purposes

• Investigate whether the software has been found to be reliable and is likely to improve the efficiency of workflow, and either accelerate your work or enable you to deal with more extensive data, or both.

Finding and evaluating appropriate software (4)

4. About the type of data or analysis the tool/software is designed for

When annotating corpora at multiple linguistic levels, annotators may use different expert tools for different phenomena or types of annotation. These tools employ different data models and accompanying approaches to visualization, and they produce different output formats. (Chiarcos et al. 2008)

Finding and evaluating appropriate software (5)

5. About its compatibility with other annotated data

• None of the software are interoperable
• Prefer compatible software

• Estimate the availability to import/export data with a minimum loss of information

Automatic vs. Manual

• Manual:
  • Linguistics data are annotated several times by one or several annotators, each one annotates according to his/her knowledge, beliefs and uncertainty.
• Automatic annotation tools/software:
  • add detailed information to language data on the basis of procedures written into the software, without human intervention other than to run the program.
  • sometimes performed by following rules set out by programmers and linguists,
  • most often, annotation programs are at least partly based on machine learning algorithms that are trained using manually annotated examples.

Automatic vs. Manual

Highly reliable manually annotated resources are, naturally, more expensive to construct, rarer and smaller in size than automatically annotated data, but they are essential for the development of automated annotation tools and are necessary whenever the desired annotation procedure either has not yet been automated or cannot be automated. (The Clarin User Guide)

Automatic vs. Manual

• Manual:
  • follow a linguistic theory
  • reliable (at least for the annotator!)
  • fastidious
• Automatic:
  • consistent
  • very fast
  • some rate of error

Automatic vs. Manual

• Before using any automatic annotation tool/software, it is important to consider its error rate and how those errors will affect whatever further purpose annotated corpora are used for.

Brief overview of some software

• In the following, we will briefly present some of the annotation software we already tested and validated to be part of the proposed methodology:

  • Praat
  • Annotation Pro
  • Elan
  • SPPAS

Software requirements

• Most of them are:

  1. under the terms of the GNU Public License, and multi-platform,
  2. ease of use (GUI), with a tutorial and/or documentation,
  3. well-known in their communities, with publications and evaluations.

Praat: the analysis the software is designed for

• Praat is a tool for manually annotating sound files. It provides different visualizations of audio data – waveform or spectrogram display – and, among other, enables pitch contour and formant calculation and visualization.
• Annotations can be created on multiple layers, called tiers.

Praat: the type of data

• The annotation files are in several Praat-specific text formats.
• Interoperability: none!

Praat: screenshot

Annotation Pro: the analysis the software is designed for

free download, Windows only

• Tool for annotation of audio and text files
• You can create many time-aligned annotation layers
• Workspace functionality enables comfortable file management, grouping files, opening previously stored file collections
• Graphical representation of the feature space is an innovative solution that enables using non-categorial features for the annotated recordings or texts
• Supports the design and conducting of Perception Tests

Annotation Pro: the type of data

• The annotation files are in a specific XML format
• Interoperability:
  • can import/export TextGrid, from Praat
  • can import trs, from Transcriber

Annotation Pro: screenshot

Elan: the analysis the software is designed for

• Elan is a tool for the creation of complex annotations in video (and audio) resources.
  
• Annotations can be created on multiple layers, that can be hierarchically interconnected and can correspond to different levels of linguistic analysis.

Elan: the type of data

• The annotation files are in a specific XML format
• Interoperability: Annotation can be imported from and exported to a variety of other formats, including Praat-TextGrid.

Elan: screenshot

SPPAS: the analysis the software is designed for

• SPPAS is a free audio annotation tool that allows you to create, visualize and search annotations for audio data. It is able to produce automatically speech segmentation annotations from a recorded speech sound and its transcription. Some special features are also offered for managing corpora of annotated files.

SPPAS: the type of data

• The annotation files are in a specific XML format
• Interoperability: Annotation can be imported from and exported to a variety of other formats, including:
  • Praat: TextGrid, PitchTier, IntensityTier
  • Elan: eaf
  • Annotation Pro: antx
    
  • Phonedit: mrk
  • HTK: lab, mlf
  • Sclite: ctm, stm
  • subtitles: sub, srt
  • Transcriber: trs (import)
  • Anvil: anvil (import)
  • CSV

SPPAS: dedicated to automatic annotations

• Language-independent algorithms:
  • language-dependent resources
  • easy and fast to add a new language

• Fully-automatic or semi-automatic (with a procedure outcome report)

• Designed to be able to deal with spontaneous speech

• No limit of the corpus size

SPPAS: screenshot

Summary

• Introduction
• Selection of annotation software
• Corpus development methodology
• Momel and INTSINT
• SPPAS
• Time Group Analyzer
• Conclusion and references