Language identification with scarce data: A case study from Peru

Alexandra Espichán-Linares, Arturo Oncevay-Marcos

Producción científica: Informe/libroLibrorevisión exhaustiva

1 Cita (Scopus)


Language identification is an elemental task in natural language processing, where corpus-based methods reign the state-of-the-art results in multi-lingual setups. However, there is a need to extend this application to other scenarios with scarce data and multiple classes to face, analyzing which of the most well-known methods is the best fit. In this way, Peru offers a great challenge as a multi-cultural and linguistic country. Therefore, this study focuses in three steps: (1) to build from scratch a digital annotated corpus for 49 Peruvian indigenous languages and dialects, (2) to fit both standard and deep learning approaches for language identification, and (3) to statistically compare the results obtained. The standard model outperforms the deep learning one as it was expected, with 95.9% in average precision, and both corpus and model will be advantageous inputs for more complex tasks in the future.
Idioma originalEspañol
EstadoPublicada - 1 ene. 2018

Citar esto