No data to crawl? Monolingual corpus creation from PDF files of truly low-resource languages in Peru

Gina Bustamante, Arturo Oncevay, Roberto Zariquiey

Producción científica: Capítulo del libro/informe/acta de congresoContribución a la conferenciarevisión exhaustiva

22 Citas (Scopus)

Resumen

We introduce new monolingual corpora for four indigenous and endangered languages from Peru: Shipibo-konibo, Ashaninka, Yanesha and Yine. Given the total absence of these languages in the web, the extraction and processing of texts from PDF files is relevant in a truly low-resource language scenario. Our procedure for monolingual corpus creation considers language-specific and language-agnostic steps, and focuses on educational PDF files with multilingual sentences, noisy pages and low-structured content. Through an evaluation based on language modelling and character-level perplexity on a subset of manually extracted sentences, we determine that our method allows the creation of clean corpora for the four languages, a key resource for natural language processing tasks nowadays.

Idioma originalInglés
Título de la publicación alojadaLREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings
EditoresNicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
EditorialEuropean Language Resources Association (ELRA)
Páginas2914-2923
Número de páginas10
ISBN (versión digital)9791095546344
EstadoPublicada - 2020
Evento12th International Conference on Language Resources and Evaluation, LREC 2020 - Marseille, Francia
Duración: 11 may. 202016 may. 2020

Serie de la publicación

NombreLREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings

Conferencia

Conferencia12th International Conference on Language Resources and Evaluation, LREC 2020
País/TerritorioFrancia
CiudadMarseille
Período11/05/2016/05/20

Huella

Profundice en los temas de investigación de 'No data to crawl? Monolingual corpus creation from PDF files of truly low-resource languages in Peru'. En conjunto forman una huella única.

Citar esto