Punctuation Restoration for Speech Transcripts using seq2seq Transformers

Authors

  • Aviv Melamud Cresskill High School
  • Alina Duran Mentor, Cresskill High School

DOI:

https://doi.org/10.47611/jsrhs.v10i4.2249

Keywords:

Speech to text, Natural Language Processing, Artificial Intelligence, Machine Learning

Abstract

When creating text transcripts from spoken audio, Automatic Speech Recognition (ASR) systems need to infer appropriate punctuation in order to make the transcription more readable. This task, known as punctuation restoration, is challenging since punctuation is not explicitly stated in speech. Most recent works framed punctuation restoration as a classification task and used pre-trained encoder-based transformers, such as BERT, to perform it. In this work, we present an alternative approach, framing punctuation restoration as a sequence-to-sequence task and using T5, a pretrained encoder-decoder transformer model, as the basis of our implementation. Training our model on IWSLT 2012, a common punctuation restoration benchmark, we find its performance is comparable to state of the art classification-based systems with an F1 score of 80.7 on the test set. Furthermore, we argue that our approach might be more flexible in its ability to adapt to more complex types of outputs, such as predicting more than one punctuation mark in a row.

Downloads

Download data is not yet available.

References or Bibliography

Alexei Baevski and Henry Zhou and Abdelrahman Mohamed and Michael Auli (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. CoRR, abs/2006.11477. https://doi.org/10.21437/interspeech.2021-717

Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., & Dahlgren, N. L. (1993). DARPA TIMIT: https://doi.org/10.6028/nist.ir.4930

Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). Librispeech: An ASR corpus based on public domain audio books. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

https://doi.org/10.1109/icassp.2015.7178964

Tündik, M. Á., Szaszák, G., Gosztolya, G., & Beke, A. (2018). User-centric Evaluation of Automatic Punctuation in ASR Closed Captioning. Interspeech 2018. https://doi.org/10.21437/interspeech.2018-1352

Tilk, O., & Alumäe, T. (2016). Bidirectional Recurrent Neural Network with Attention Mechanism for Punctuation Restoration. Interspeech 2016. https://doi.org/10.21437/interspeech.2016-1517

Tilk, O., & Alumäe, T. (2015). LSTM for punctuation restoration in speech transcripts. Interspeech 2015.

https://doi.org/10.21437/interspeech.2015-240

Attila Nagy and Bence Bial and Judit Ács (2021). Automatic punctuation restoration with BERT models. CoRR, abs/2101.07343.

Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805. https://doi.org/10.18653/v1/N19-1423

Courtland, M., Faulkner, A., & McElvain, G. (2020). Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference. In Proceedings of the 17th International Conference on Spoken Language Translation (pp. 272–279). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.iwslt-1.33

Kolár, J., Svec, J., & Psutka, J. (2004). Automatic punctuation annotation in czech broadcast news speech.

Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. CoRR, abs/1910.10683.

Noam Shazeer and Mitchell Stern (2018). Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. CoRR, abs/1804.04235.

Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, & Blake Hechtman. (2018). Mesh-TensorFlow: Deep Learning for Supercomputers.

Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Jamie Brew (2019). HuggingFace's Transformers: State-of-the-art Natural Language Processing. CoRR, abs/1910.03771.

Gravano, A., Jansche, M., & Bacchiani, M. (2009). Restoring punctuation and capitalization in transcribed speech. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 4741-4744). https://doi.org/10.1109/ICASSP.2009.4960690

Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and Mandar Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke Zettlemoyer and Veselin Stoyanov (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692.

Alam, F. (2020). Punctuation Restoration using Transformer Models for High-and Low-Resource Languages. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020) (pp. 132–142). Association for Computational Linguistics.

https://doi.org/10.18653/v1/2020.wnut-1.18

Stephan Peitz, Markus Freitag, Arne Mauser, & H. Ney (2011). Modeling punctuation prediction as machine translation. IWSLT.

Vāravs, Andris & Salimbajevs, Askars. (2018). Restoring Punctuation and Capitalization Using Transformer Models: 6th International Conference, SLSP 2018, Mons, Belgium, October 15–16, 2018, Proceedings. https://doi.org/10.1007/978-3-030-00810-9_9

Published

11-30-2021

How to Cite

Melamud, A., & Duran, A. (2021). Punctuation Restoration for Speech Transcripts using seq2seq Transformers. Journal of Student Research, 10(4). https://doi.org/10.47611/jsrhs.v10i4.2249

Issue

Section

HS Research Articles