Please use this identifier to cite or link to this item: http://dx.doi.org/10.14279/depositonce-11513
For citation please use:
Main Title: Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods
Author(s): Naderi, Babak
Zequeira Jiménez, Rafael
Hirth, Matthias
Möller, Sebastian
Metzger, Florian
Hoßfeld, Tobias
Type: Article
URI: https://depositonce.tu-berlin.de/handle/11303/12713
http://dx.doi.org/10.14279/depositonce-11513
License: https://creativecommons.org/licenses/by/4.0/
Abstract: Subjective speech quality assessment has traditionally been carried out in laboratory environments under controlled conditions. With the advent of crowdsourcing platforms tasks, which need human intelligence, can be resolved by crowd workers over the Internet. Crowdsourcing also offers a new paradigm for speech quality assessment, promising higher ecological validity of the quality judgments at the expense of potentially lower reliability. This paper compares laboratory-based and crowdsourcing-based speech quality assessments in terms of comparability of results and efficiency. For this purpose, three pairs of listening-only tests have been carried out using three different crowdsourcing platforms and following the ITU-T Recommendation P.808. In each test, listeners judge the overall quality of the speech sample following the Absolute Category Rating procedure. We compare the results of the crowdsourcing approach with the results of standard laboratory tests performed according to the ITU-T Recommendation P.800. Results show that in most cases, both paradigms lead to comparable results. Notable differences are discussed with respect to their sources, and conclusions are drawn that establish practical guidelines for crowdsourcing-based speech quality assessment.
Subject(s): crowdsourcing
reliability
speech quality assessment
validity
listening-only tests
Issue Date: 22-Nov-2020
Date Available: 4-Mar-2021
Language Code: en
DDC Class: 620 Ingenieurwissenschaften und zugeordnete Tätigkeiten
Sponsor/Funder: TU Berlin, Open-Access-Mittel – 2020
Journal Title: Quality and User Experience
Publisher: SpringerNature
Volume: 6
Issue: 1
Article Number: 2
Publisher DOI: 10.1007/s41233-020-00042-1
EISSN: 2366-0147
ISSN: 2366-0139
TU Affiliation(s): Fak. 4 Elektrotechnik und Informatik » Inst. Softwaretechnik und Theoretische Informatik » Quality and Usability Lab
Appears in Collections:Technische Universität Berlin » Publications

Files in This Item:
Naderi_etal_Towards_2020.pdf
Format: Adobe PDF | Size: 2.52 MB
DownloadShow Preview
Thumbnail

Item Export Bar

This item is licensed under a Creative Commons License Creative Commons