Overview

The Bavieca toolkit comprises a set of about 25 command line tools that can be used to build very sophisticated large vocabulary speech recognition systems from scratch. Additionally it offers an Application Programming Interface (API) that exposes speech processing features such as speech recognition, speech activity detection, forced alignment, etc. This API is provided as a C++ library that can be used to create stand-alone applications that exploit Bavieca's speech recognition features.

Phonetic symbol set

The phonetic symbol can contain any set of symbols.

Phonetic rules

Phonetic rules serve to express groupings of phones that share similar properties attending to some criterion (e.g. fricatives, frontvowels, etc). They are used to guide the procedure of clustering context dependent units (see the contextclustering tool) under the assumption that phones that present similar properties are likely to affect the realization of neighboring phones in a similar fashion. During the process of building the decision tree, a cluster of allophones is split into subclusters by testing the applicable phonetic rules and picking the rule that results in a higher likelihood increase.

A very good way to see how to define phonetic rules is by looking at the example below. The character '#' can be used to write comments. Note that each phonetic symbol in the phonetic symbol set must appear in the right-side of at least one rule. However, phonetic symbols typically appear on the right side of many rules.

Pronunciation lexicon

The pronunciation lexicon contains pronunciations for all the words in the vocabulary. A pronunciation for a word is defined by the actual word followed by the sequence of phonemes that express the pronunciation. Alternative pronunciations of a word can be defined by appending '(n)' to the word. This suffix applies once the first pronunciation of the word has been defined (n starts at 2).

Depending on whether it is used for training or recognition, the lexicon file has different uses. For training purposes the lexicon file must contain pronunciations for all words in the master label file, while for recognition, the lexicon file determines the active vocabulary (i.e. those words that the recognizer can potentially recognize).

Below there is an example of a lexicon file containing alternative pronunciations for some words. When creating a pronunciation lexicon the following considerations must be taken into account:

• Symbols between pointy brackets, such as '<SIL>' or '<BREATH>', denote filler symbols, which have a special treatment during recognition. In particular, a special insertion penalty can be specified for them and they are not affected by the language model (do not appear in any n-gram).
• Due to the way the decoding network is built, two words/filler symbols that have the same initial phone cannot have different insertion penalties. Thus, it is recommended that pronunciations for filler symbols do not include phonetic symbols that are used in actual words but phonetic symbols created ad-hoc (such as '_BREATH' or '_UH' in the example below).
• The silence symbol '<SIL>' must always be defined.
• The characters "##" can be used to write comments
• Although it is not a requirement (the pronunciation lexicon is internally reordered when loaded), entries in the lexicon should be ordered alphabetically.

Language model

Currently the Bavieca speech recognition toolkit supports language models in two formats, the ARPA format and binary format. Internally, for efficiency reasons, language models are represented as Finite State Machines (FSMs). Any n-gram orders are supported, however only orders up to fourgram have been tested.

ARPA format

The ARPA format is a simple and human-readable language model format. There are several freely available language modeling toolkits that can be used to build language models in the ARPA format. Two excellent resources are The CMU Statistical Language Modeling (SLM) Toolkit and the MIT Language Modeling Toolkit. Language models built with these toolkits have been successfully used with Bavieca.

Fragments of a trigram language model in the ARPA format are listed below.

Binary format

The binary format allows a much faster loading from disk, usually about an order of magnitude compared to the time it takes to load a language model in the ARPA format. The binary representation is lossless and encodes the language model as a Finite State Machine. In addition, the binary representation uses about half the space on disk. A language model in ARPA format can be converted to binary format using the lmfsm command line tool.

Master Label File

Master Label Files (MLFs) are used to describe a dataset composed of a series of utterances with corresponding transcriptions. MLFs serve as input to tools that initialize acoustic model parameters or accumulate sufficient statistics to reestimate acoustic model parameters.

Each utterance in the MLF is expressed by a line with a relative path to a feature file (containing features extracted from the utterance) followed by a line for each word in the transcription of the utterance. Each transcription must contain at least one word (or symbol). Typically, Bavieca tools produce absolute paths to feature files by appending relative paths in the MLFs to base folders specified as parameters. This is a flexible mechanism to, for example, use different sets of features with a single MLF.

Below there is an example of MLF. Note the use of the character '"' at the beginning and end of the relative paths.

Features

Feature files are binary files containing a series of feature vectors. These files are produced by the param tool according to a series of parameters defined in a feature configuration file. The characteristics of the features, along with their dimensionality, are not specified in the feature file. They are only available through the feature configuration file used to create them.

Alignment file

Alignment files can be produced by a Viterbi aligner or a Forward-Backward aligner. They can be either in binary or text format.

Text format

Binary format

Accumulator

Accumulator files keep sufficient statistics to estimate acoustic model parameters (Gaussian distributions). There exist two types of accumulator files:

• Logical accumulators: this type of accumulators keep sufficient statistics at the logical HMM-state level. A logical HMM-state is an HMM-state in a particular phonetic context. For example, "S-AH-L+UW+T" represents a logical HMM-state for a pentaphone (where L is the central phone and the characters "-" and "+" are used to denote left and right phonetic context respectively). In reality, a logical HMM-state is also characterized for the HMM-state number (HMMs in Bavieca have three states) and the position of the central phone within the word, which can be initial, internal or final. Statistics for logical HMM-states are accumulated everytime a logical HMM-state is observed (i.e. receives some occupation) during the accumulation process. For example the logical accumulator for "S-AH-L+UW+T" will receive some data every time the word ABSOLUTE ("AE B S AH L UW T") is observed in the training data.

  Logical accumulators are used to build context dependent HMM-states using the contextclustering tool. In order to manage data sparsity, this tool clusters a set of logical HMM-states into a physical context-dependent HMM-state.

  When large phonetic contexts are used, the number of different left and right contexts observed for each phone during the accumulation process can be very large, which means that a large number of logical accumulators (and a large storage space) will be needed to keep sufficient statistics.

• Physical accumulators: these types of accumulators keep sufficient statistics at the physical HMM-state level. Physical HMM-states are actual HMM-states that can be used to estimate log-likelihoods during training and recognition, while logical HMM-states are just an abstraction used to describe how context modeling works. Specifically, each physical accumulator keeps sufficient statistics to estimate the mean and covariance of a Gaussian distribution. Physical accumulators are identified by a physical HMM-state identifier and an index within its mixture of Gaussian components. Physical accumulators do not know about context dependency, they can be generated to reestimate the parameters of context-independent monophones or clustered n-phones.

Hypothesis

Hypothesis files contain word-hypothesis generated by recognition or rescoring tools. Bavieca supports two types of hypothesis file: trn and ctm, as defined in the SCLITE scoring toolkit.

Below there is an example of hypothesis file in trn format. There are hypotheses for four utterances (one per line). The trailing code between parentheses is the utterance identifier, which scoring tools use to match pairs hypothesis-reference.

Transcription

Transcription files keep orthographic transcriptions of utterances and are typically used to score recognition hypothesis and compute the word error rate (WER). WER is defined as the number of edit errors resulting from aligning a hypothesis against a reference file (transcription) divided by the number of words in the reference file and it is the most common metric to measure speech recognition accuracy. Transcription files in Bavieca are in the trn format, as defined in the SCLITE scoring toolkit.

Below there is an example of transcription file in the trn format. There are transcriptions for six utterances (one per line). The trailing code between parentheses is the utterance identifier, which scoring tools use to match pairs hypothesis-reference.

Lattice file

Lattice files can be either in binary or text format.

Text format

Decoders and lattice edition tools can output lattices in text format, however this is exclusively an output format and it is only intended for easy visualization of the lattice.

Example 1: Simple lattice in text format. The first four lines contain mandatory lattice properties: the number of edges in the lattice, the number of nodes, the number of time-slices (frames), and the lattice version.

Example 2: Graphical representation of a lattice created from a lattice in text format. Word identities and word boundaries are depicted for each edge in the lattice

Example 3: Segment of a lattice in text format with a number of properties. Each edge is followed by the phone alignment (including HMM-state identifiers) of the word attached to the edge.

• "am" = acoustic model log-likelihood
• "lm" = language model log-likelihood
• "ip" = word insertion penalty
• "fw" = edge forward score
• "bw" = edge backward score
• "pp" = edge posterior probability

Binary format

The param tool is used to extract features from raw audio. Feature vectors can then be used to train acoustic models, compute transforms, etc. The table below summarizes its optional and required command line parameters.

Feature configuration file

This file specifies parameters needed for feature extraction.

The lmfsm tool is used to convert a language model in ARPA format to binary format. Language models in binary format enable fast loading times from disk and use less storage space.

The aligner tool is used to align features against a given sequence of words/symbols using a set of acoustic models. It transforms the given sequence of words along with optional symbols like silence or fillers into a graph of phones, which is then transformed into an optimized graph of HMM-states. Finally it aligns the graph of HMM-states against the feature vectors using the Viterbi algorithm and produces a time alignment of the input lexical units.

The hmminitializer tool is used to create an initial set of acoustic models. Specifically this tool creates a set of single-Gaussian Hidden Markov Models (HMMs) initialized to the global distribution of the data, or to the HMM-state distributions given by an input set of alignment files. Acoustic models created with this tool can be later refined using the mlaccumulator and mlestimator tools.

The mlaccumulator tool accumulates sufficient statistics necessary to estimate acoustic model parameters under the Maximum Likelihood Estimation (MLE) criterion. Statistics are accumulated from the training data by aligning feature vectors extracted from the audio against the transcriptions. For each utterance in the master label file, a graph is constructed from its words considering alternative pronunciations (optional) and optional filler symbols. This graph is then aligned against the feature vectors using the forward-backward algorithm and occupation statistics are dumped into the accumulator file.

The mlestimator tool reestimates the parameters of a given set of acoustic models using a list of accumulator files containing sufficient statistics. Estimation of acoustic model parameters is carried out under the Maximum Likelihood Estimation (MLE) criterion (for discriminative estimation see dtestimator). After the MLE is carried out, Gaussian covariances are floored using the flooring ratio provided.

The mapestimator tool reestimates the parameters of a given set of acoustic models using a list of accumulator files containing sufficient statistics. A Maximum A Posteriori (MAP) adaptation of the acoustic model parameters is performed, which is typically used for adapting a set of well trained acoustic models to a new domain, environment or even to a particular speaker when enough adaptation data is available.

The dtaccumulator tool accumulates sufficient statistics necessary to estimate acoustic model parameters under two alternative discriminative criteria: Maximum Mutual Information (MMI) or boosted Maximum Mutual Information (bMMI). Statistics are accumulated from the training data by aligning feature vectors extracted from the audio against the transcriptions. For each utterance in the master label file, a graph is constructed from its words considering alternative pronunciations (optional) and optional filler symbols. This graph is then aligned against the feature vectors using the forward-backward algorithm and occupation statistics are dumped into the accumulator file.

The dtestimator tool reestimates the parameters of a given set of acoustic models discriminatively. Unlike the mlestimator tool, this tool needs two sets of accumulators, one with numerator statistics and another one with denominator statistics. Numerator and denominator statistics are typically accumulated over a set of lattices using the dtaccumulator tool. After the discriminative estimation is carried out, Gaussian covariances are floored using the flooring ratio provided.

The contextclustering tool refines a given set of context-independent acoustic models by performing context clustering. Context clustering is carried out using logical accumulators obtained from single Gaussian HMMs and decision trees that are either state-specific or global. Decision trees are generated following a standard top-down procedure that iteratively splits the data by applying binary questions using a ML criterion. Questions are asked about the correspondence to phonetic groups (defined by hand-made phonetic rules) and the within-word position (initial, internal and final). The splitting process is governed by two parameters: a minimum occupation count for each leaf and a minimum likelihood increase for each split. Finally, a bottom-up merging process is applied to merge those leaves which, when merged, produce a likelihood decrease below the minimum value used to allow a split.

The gmmeditor tool is used to refine a set of acoustic models by applying mixture splitting and merging to the mixtures of Gaussian distributions. Gaussian splitting allows for a more detailed modeling of the acoustics while Gaussian merging eliminates Gaussian distributions that become unnecesary. Acoustic model refinement through mixture splitting and merging can be performed after each reestimation iteration using the gmmeditor tool, the original set of HMMs and the accumulated statistics. After applying the gmmeditor tool GMMs will typically have a variable number of components depending on the data aligned to the HMM-state, which varies across reestimation iterations.

This tool is used to recognize speech in batch mode, which means that the audio to recognize is completely available beforehand as opposed to live mode, in which the samples of audio are captured and passed to the recognition engine in real time. In order to perform speech recognition in live mode see Bavieca's API.

Dynamic decoder configuration file

This file specifies parameters needed for the dynamicdecoder tool.

Tool used to build a static decoding network in the form of a Weighted Finite State Acceptor (WFSA). Decoding networks built using this tool can be used for recognition with the wfsadecoder tool. WFSA-based decoding is very fast since all sources of information (acoustic models, pronunciation lexicon and language model) are combined and optimized statically before the actual recognition process, which is therefore substantially simplified. For large language model sizes the process of building a WFSA decoding network can be time and memory intensive, which requires large amounts of physical memory. In those cases the use of the dynamicdecoder tool is recommended.

This tool is a Weighted Finite State Acceptor (WFSA) based speech decoder which, in the same way as the dynamicdecoder tool, is used to process input speech and produce recognition hypotheses. In particular, this tool is used to recognize speech in batch mode, which means that the audio to recognize is completely available beforehand as opposed to live mode, in which the samples of audio are captured and passed to the recognition engine in real time. In order to perform speech recognition in live mode see Bavieca's API.

WFSA-decoder configuration file

This file specifies configuration parameters for the wfsadecoder tool.

This command line tool is used to perform Speech Activity Detection (SAD) over features extracted from an audio file. SAD is useful to spot speech segments within an audio stream for further processing. Since speech recognition can be a time consuming process and SAD is usually not, directing the recognition only to those segments of audio where speech is detected is a typical procedure. This implementation of SAD is based on two Hidden Markov Models (HMM) with three states each, one HMM for silence and another HMM for speech. All the HMMs for silence and speech share the same set of Gaussian distributions, which are drawn from the HMM for silence and the HMMs from speech that are found in the set of acoustic models passed as a parameter. Viterbi search is used to find the most likely alignment of features to speech and silence and the resulting segmentation is written to a file.

The accuracy of this tool is very sensitive to the following parameters: '-sil', '-sph' and '-pen', whose values should be determined empirically.

The hldaestimator tool is used to perform Heteroscedastic Linear Discriminant Analysis (HLDA). It consists of estimating a feature transform to decorrelate features and reduce their dimensionality while preserving the most discriminative information. In order to compute the transform a set of full-covariance acoustic models along with physical accumulators is passed as input. Typically full-covariance acoustic models are trained on high dimensional feature vectors (from scratch or doing single pass retraining) and then a HLDA transform is estimated to decorrelate the features and reduce their dimensionality. A common scenario consists of training full-covariance acoustic models on feature vectors with 52 coefficients (static features+Δ+ΔΔ+ΔΔΔ) and then estimating a HLDA transform to reduce the dimensionality to 39 coefficients.

The vtlestimator command line tool performs Maximum Likelihood (ML) based Vocal Tract Length estimation over a set of feature files. Features are extracted for different warp factors (starting with the warp factor specified by '-floor' up to the warp factor specified by '-ceiling', using a step size specified by '-step') and the warp factor that results in the highest overall likelihood is written to the output file. Unvoiced phones and symbols should be excluded from the estimation using the parameter '-fil'.

The regtree tool builds regression trees that can be used to collect statistics for a form of adaptation called Maximum Likelihood Linear Regression (MLLR). The regression tree is built by performing top-down clustering of the means of all Gaussian distributions present in the acoustic models. Regression trees created using this tool are then passed to the mllrestimator tool to perform the actual MLLR adaptation

The mllrestimator tool performs Maximum Likelihood Linear Regression (MLLR) to adapt the Gaussian distributions of a set of acoustic models. This tool receives as input a baseline set of acoustic models and adaptation data (a set of feature files with corresponsing transcriptions, which are typically hand-made transcriptions or recognition hypotheses) and estimates a mean and, optionally, a covariance tranform for each regression-class. The total number of transforms computed is automatically determined based on the adaptation data and the value of the parameters '-occ' and '-gau'. Estimated transforms can be stored into a file for later utilization using the tool hmmx

The fmllrestimator tool performs feature-space Maximum Likelihood Linear Regression (MLLR) to adapt a set of feature vectors. This tool receives as input a baseline set of acoustic models and adaptation data (a set of feature files with corresponsing transcriptions, which are typically hand-made transcriptions or recognition hypotheses) and estimates a transform. The estimated transform is stored into a file for later utilization using the tool paramx.

The paramx tool applies linear and affine transformations to feature vectors. Transformations can be applied over pairs of input and output feature vectors using the parameter '-bat', or, alternatively over a single pair of feature vectors using the parameters '-in' and '-out'.

The hmmx tool applies transformations to acoustic model parameters. Transformations are applied to a set of input acoustic models and adapted acoustic models are stored in a file.

The latticeeditor tool can be used to perform a wide variety of operations over lattices. It can be used for computing the lattice Word Error Rate, computing lattice-based posterior probabilities and confidence scores, aligning lattices against feature vectors and attaching acoustic log-likelihoods and HMM-state information to its edges, compacting a lattice by merging redundant edges and nodes, rescoring lattices according to a given criterion, attaching language model log-likelihoods to edges in the lattice, and adding paths to lattices. Different operations require different combination of input parameters.

Bavieca's Application Programming Interface

Bavieca's API provides an easy way to incorporate speech-recognition capabilities into an application. These capabilities are listed below:

• Stream-based feature extraction: speech features can be extracted from samples of audio as the audio becomes available and fed into the feature extraction process. Feature normalization is done in stream mode and feature vectors (instead of audio samples) are used as input to the various speech processing functions (recognition, speech detection, alignment, etc). The same set of features can be first used for speech activity detection and then for recognition.
• Stream-based speech recognition: while Bavieca's command line speech recognizers (dynamicdecoder and wfsadecoder) enable speech recognition in batch mode, which is ideal for experimentation, they do not provide the means to perform speech recognition over a live stream of audio. Bavieca's API enables speech recognition in live mode.
• Stream-based speech activity detection: speech activity detection is a very common mechanism to reduce the amount of audio fed to the speech recognition system in order to make a better use of computational resources and reduce recognition errors.
• Speech-to-text alignment: time-alignment information of words and phonemes within an utterance can be generated to be used, for example, to highlight words and phonemes while playing back speech, or as input data to perform synthetic lip movement.
• Live-mode MLLR speaker adaptation: Coming soon...

Bavieca's API comprises a relatively reduced set of functions and data structures, all of them are declared in the header file BaviecaAPI.h. Below there is an example showing how to use the API.

Example

The example below shows a very simple way to use Bavieca's API to recognize speech in live-mode. For the sake of simplicity, speech samples in the example are retrieved from a file instead of from the microphone, however they are fed into the recognition process as if they were captured in real time.

	// initialize API
	const char *strFileConfiguration = "configuration.txt";
	BaviecaAPI baviecaAPI(strFileConfiguration);
	if (baviecaAPI.initialize(INIT_SAD|INIT_ALIGNER|INIT_DECODER) == false) {  
		return -1; 
	} 
	
	// load audio samples
	const char *strFileRaw = "audio.raw";
	ifstream is;
	is.open(strFileRaw,ios::binary);
	if (!is.is_open()) {
		cerr << "unable to open the file: \"\"" << strFileRaw; 
	}	
	is.seekg(0, ios::end);
	int iBytes = is.tellg();
	is.seekg(0, ios::beg);	
	int iSamples = (iBytes/2);
	short *sSamples = new short[iSamples];
	is.read((char*)sSamples,iSamples*2); 
	if (is.fail()) {
		cerr << "error reading from stream at position: " << is.tellg();
	}
	is.close();
	
	// begin utterance processing
	baviecaAPI.decBeginUtterance();
	
	// simulate streaming data (1 second chunks)
	int iSamplesChunk = 16000;
	int iSamplesUsed = 0;
	while((iSamplesUsed+iSamplesChunk) < iSamples) {
		
		// extract features from the audio
		int iFeatures = -1;
		float *fFeatures = baviecaAPI.extractFeatures(sSamples+iSamplesUsed,iSamplesChunk,&iFeatures);
		
		baviecaAPI.decProcess(fFeatures,iFeatures);		
		baviecaAPI.free(fFeatures);
		iSamplesUsed += iSamplesChunk;
	}
	delete [] sSamples;
	
	// print recognition hypothesis
	int iWords = -1;
	const char *strFileHypothesisLattice = NULL;
	WordHypothesisI *wordHypothesis = baviecaAPI.decGetHypothesis(&iWords,strFileHypothesisLattice);
	if (wordHypothesis) {
		for(int i=0 ; i < iWords ; ++i) {
			cout << " [" << wordHypothesis[i].iFrameStart << " ";
			cout << wordHypothesis[i].strWord << " ";
			cout << wordHypothesis[i].iFrameEnd << "] ";
		}
		cout << endl;
	}
	
	baviecaAPI.free(wordHypothesis,iWords);
		
	// end utterance processing
	baviecaAPI.decEndUtterance();
	
	baviecaAPI.uninitialize();