corpora._mmreader – Read corpus in the Matrix Market format¶
Reader for corpus in the Matrix Market format.
-
class
gensim.corpora._mmreader.MmReader(input, transposed=True)¶ Bases:
objectMatrix market file reader (fast Cython version), used internally in
MmCorpus.Wrap a term-document matrix on disk (in matrix-market format), and present it as an object which supports iteration over the rows (~documents).
-
num_docs¶ Number of documents in the market matrix file.
- Type
int
-
num_terms¶ Number of terms.
- Type
int
-
num_nnz¶ Number of non-zero terms.
- Type
int
Notes
Note that the file is read into memory one document at a time, not the whole matrix at once (unlike e.g. scipy.io.mmread and other implementations). This allows us to process corpora which are larger than the available RAM.
- Parameters
input ({str, file-like object}) – Path to the input file in MM format or a file-like object that supports seek() (e.g. smart_open objects).
transposed (bool, optional) – Do lines represent doc_id, term_id, value, instead of term_id, doc_id, value?
-
docbyoffset(self, offset)¶ Get the document at file offset offset (in bytes).
- Parameters
offset (int) – File offset, in bytes, of the desired document.
- Returns
Document in sparse bag-of-words format.
- Return type
list of (int, str)
-
input¶ object
- Type
input
-
num_docs ‘long long’
- Type
num_docs
-
num_nnz ‘long long’
- Type
num_nnz
-
num_terms ‘long long’
- Type
num_terms
-
skip_headers(self, input_file)¶ Skip file headers that appear before the first document.
- Parameters
input_file (iterable of str) – Iterable taken from file in MM format.
-
transposed¶ ‘bool’
- Type
transposed
-