Publication details

Software Framework for Topic Modelling with Large Corpora



Year of publication 2010
Type Article in Proceedings
Conference Proceedings of LREC 2010 workshop New Challenges for NLP Frameworks
MU Faculty or unit

Faculty of Informatics

Field Computer hardware and software
Keywords document similarity; NLP; software; vector space model; topical modelling; software framework; topical document similarity; Python; IR; LSA; LDA; gensim; DML-CZ
Attached files
Description Large corpora are ubiquitous in today's world and memory quickly becomes the limiting factor in practical applications of the Vector Space Model (VSM). We identify gap in existing VSM implementations, which is their scalability and ease of use. We describe a Natural Language Processing software framework which is based on the idea of document streaming, i.e. processing corpora document after document, in a memory independent fashion. In this framework, we implement several popular algorithms for topical inference, including Latent Semantic Analysis and Latent Dirichlet Allocation, in a way that makes them completely independent of the training corpus size. Particular emphasis is placed on straightforward and intuitive framework design, so that modifications and extensions of the methods and/or their application by interested practitioners are effortless. We demonstrate the usefulness of our approach on a real-world scenario of computing document similarities within an existing digital library DML-CZ.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.

More info