dmlcorpus — gensim v1.0 documentation

### Navigation

- index
- modules |
- next |
- previous |
- gensim v1.0 documentation »
- API Reference »

# `dmlcorpus`

Corpus for the DML-CZ project.

class `gensim.corpora.dmlcorpus.` `DmlConfig`(configId, resultDir, acceptLangs=None)

  DmlConfig contains parameters necessary for the abstraction of a ‘corpus of articles’ (see the DmlCorpus class).

  Articles may come from different sources (=different locations on disk/netword, different file formats etc), so the main purpose of DmlConfig is to keep all sources in one place (= the self.sources attribute).

  Apart from glueing together sources, DmlConfig also decides where to store output files and which articles to accept for the corpus (= an additional filter over the sources).

class `gensim.corpora.dmlcorpus.` `DmlCorpus`

  DmlCorpus implements a collection of articles. It is initialized via a DmlConfig object, which holds information about where to look for the articles and how to process them.

  Apart from being a regular corpus (bag-of-words iterable with a len() method), DmlCorpus has methods for building a dictionary (mapping between words and their ids).

  `buildDictionary`()

    Populate dictionary mapping and statistics.

    This is done by sequentially retrieving the article fulltexts, splitting them into tokens and converting tokens to their ids (creating new ids as necessary).

  classmethod `load`(fname)
    Load a previously saved object from file (also see save).
  `processConfig`(config, shuffle=False)

    Parse the directories specified in the config, looking for suitable articles.

    This updates the self.documents var, which keeps a list of (source id, article uri) 2-tuples. Each tuple is a unique identifier of one article.

    Note that some articles are ignored based on config settings (for example if the article’s language doesn’t match any language specified in the config etc.).

  `save`(fname)
    Save the object to file via pickling (also see load).
  `saveAsText`(normalizeTfidf=False)

    Store the corpus to disk, in a human-readable text format.

    This actually saves multiple files:
      1. Pure document-term co-occurence frequency counts, as a Matrix Market file.
      2. Token to integer mapping, as a text file.
      3. Document to document URI mapping, as a text file.

    The exact filesystem paths and filenames are determined from the config.

### Navigation

- index
- modules |
- next |
- previous |
- gensim v1.0 documentation »
- API Reference »