Publication details

Efficient Management and Optimization of Very Large Machine Learning Dataset for Question Answering

Investor logo

MEDVEĎ Marek SABOL Radoslav HORÁK Aleš

Year of publication 2020
Type Article in Proceedings
Conference Proceedings of the Fourteenth Workshop on Recent Advances in Slavonic Natural Language Processing, RASLAN 2020
MU Faculty or unit

Faculty of Informatics

Keywords question answering; dataset management; machine learning; optimization
Description Question answering strategies lean almost exclusively on deep neural network computations nowadays. Managing a large set of input data (questions, answers, full documents, metadata) in several forms suitable as the first layer of a selected network architecture can be a non-trivial task. In this paper, we present the details and evaluation of preparing a rich dataset of more than 13 thousand question-answer pairs with more than 6,500 full documents. We show, how a Python-optimized database in a network environment was utilized to offer fast responses based on the 26 GiB database of input data. A global hyperparameter optimization process with controlled running of thousands of evaluation experiments to reach a near-optimum setup of the learning process is also explicated.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.

More info