topic_coherence.text_analysis – Analyzing the texts of a corpus to accumulate statistical information about word occurrences

This module contains classes for analyzing the texts of a corpus to accumulate statistical information about word occurrences.

class gensim.topic_coherence.text_analysis.AccumulatingWorker(input_q, output_q, accumulator, window_size)

Bases: multiprocessing.context.Process

Accumulate stats from texts fed in from queue.

property authkey
close()

Close the Process object.

This method releases resources held by the Process object. It is an error to call this method if the child process is still running.

property daemon

Return whether process is a daemon

property exitcode

Return exit code of process or None if it has yet to stop

property ident

Return identifier (PID) of process or None if it has yet to start

is_alive()

Return whether process is alive

join(timeout=None)

Wait until child process terminates

kill()

Terminate process; sends SIGKILL signal or uses TerminateProcess()

property name
property pid

Return identifier (PID) of process or None if it has yet to start

reply_to_master()
run()

Method to be run in sub-process; can be overridden in sub-class

property sentinel

Return a file descriptor (Unix) or handle (Windows) suitable for waiting for process termination.

start()

Start child process

terminate()

Terminate process; sends SIGTERM signal or uses TerminateProcess()

class gensim.topic_coherence.text_analysis.BaseAnalyzer(relevant_ids)

Bases: object

Base class for corpus and text analyzers.

relevant_ids

Mapping

Type

dict

_vocab_size

Size of vocabulary.

Type

int

id2contiguous

Mapping word_id -> number.

Type

dict

log_every

Interval for logging.

Type

int

_num_docs

Number of documents.

Type

int

Parameters

relevant_ids (dict) – Mapping

Examples

>>> from gensim.topic_coherence import text_analysis
>>> ids = {1: 'fake', 4: 'cats'}
>>> base = text_analysis.BaseAnalyzer(ids)
>>> # should return {1: 'fake', 4: 'cats'} 2 {1: 0, 4: 1} 1000 0
>>> print(base.relevant_ids, base._vocab_size, base.id2contiguous, base.log_every, base._num_docs)
{1: 'fake', 4: 'cats'} 2 {1: 0, 4: 1} 1000 0
__getitem__(word_or_words)
analyze_text(text, doc_num=None)
get_co_occurrences(word_id1, word_id2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word_id)

Return number of docs the word occurs in, once accumulate has been called.

property num_docs
class gensim.topic_coherence.text_analysis.CorpusAccumulator(*args)

Bases: gensim.topic_coherence.text_analysis.InvertedIndexBased

Gather word occurrence stats from a corpus by iterating over its BoW representation.

Parameters

args (dict) – Look at BaseAnalyzer

Examples

>>> from gensim.topic_coherence import text_analysis
>>>
>>> ids = {1: 'fake', 4: 'cats'}
>>> ininb = text_analysis.InvertedIndexBased(ids)
>>>
>>> print(ininb._inverted_index)
[set([]) set([])]
__getitem__(word_or_words)
accumulate(corpus)
analyze_text(text, doc_num=None)

Build an inverted index from a sequence of corpus texts.

get_co_occurrences(word_id1, word_id2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word_id)

Return number of docs the word occurs in, once accumulate has been called.

index_to_dict()
property num_docs
class gensim.topic_coherence.text_analysis.InvertedIndexAccumulator(relevant_ids, dictionary)

Bases: gensim.topic_coherence.text_analysis.WindowedTextsAnalyzer, gensim.topic_coherence.text_analysis.InvertedIndexBased

Build an inverted index from a sequence of corpus texts.

Parameters

  • relevant_ids (set of int) – Relevant id

  • dictionary (Dictionary) – Dictionary instance with mappings for the relevant_ids.

__getitem__(word_or_words)
accumulate(texts, window_size)
analyze_text(window, doc_num=None)
get_co_occurrences(word1, word2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word)

Return number of docs the word occurs in, once accumulate has been called.

index_to_dict()
property num_docs
text_is_relevant(text)

Check if the text has any relevant words.

class gensim.topic_coherence.text_analysis.InvertedIndexBased(*args)

Bases: gensim.topic_coherence.text_analysis.BaseAnalyzer

Analyzer that builds up an inverted index to accumulate stats.

Parameters

args (dict) – Look at BaseAnalyzer

Examples

>>> from gensim.topic_coherence import text_analysis
>>>
>>> ids = {1: 'fake', 4: 'cats'}
>>> ininb = text_analysis.InvertedIndexBased(ids)
>>>
>>> print(ininb._inverted_index)
[set([]) set([])]
__getitem__(word_or_words)
analyze_text(text, doc_num=None)
get_co_occurrences(word_id1, word_id2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word_id)

Return number of docs the word occurs in, once accumulate has been called.

index_to_dict()
property num_docs
class gensim.topic_coherence.text_analysis.ParallelWordOccurrenceAccumulator(processes, *args, **kwargs)

Bases: gensim.topic_coherence.text_analysis.WindowedTextsAnalyzer

Accumulate word occurrences in parallel.

processes

Number of processes to use; must be at least two.

Type

int

args

Should include relevant_ids and dictionary (see __init__).

kwargs

Can include batch_size, which is the number of docs to send to a worker at a time. If not included, it defaults to 64.

Parameters

  • relevant_ids (set of int) – Relevant id

  • dictionary (Dictionary) – Dictionary instance with mappings for the relevant_ids.

__getitem__(word_or_words)
accumulate(texts, window_size)
analyze_text(text, doc_num=None)
get_co_occurrences(word1, word2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word)

Return number of docs the word occurs in, once accumulate has been called.

merge_accumulators(accumulators)

Merge the list of accumulators into a single WordOccurrenceAccumulator with all occurrence and co-occurrence counts, and a num_docs that reflects the total observed by all the individual accumulators.

property num_docs
queue_all_texts(q, texts, window_size)

Sequentially place batches of texts on the given queue until texts is consumed. The texts are filtered so that only those with at least one relevant token are queued.

start_workers(window_size)

Set up an input and output queue and start processes for each worker.

Notes

The input queue is used to transmit batches of documents to the workers. The output queue is used by workers to transmit the WordOccurrenceAccumulator instances.

Parameters

window_size (int) –

Returns

Tuple of (list of workers, input queue, output queue).

Return type

(list of lists)

terminate_workers(input_q, output_q, workers, interrupted=False)

Wait until all workers have transmitted their WordOccurrenceAccumulator instances, then terminate each.

Warning

We do not use join here because it has been shown to have some issues in Python 2.7 (and even in later versions). This method also closes both the input and output queue. If interrupted is False (normal execution), a None value is placed on the input queue for each worker. The workers are looking for this sentinel value and interpret it as a signal to terminate themselves. If interrupted is True, a KeyboardInterrupt occurred. The workers are programmed to recover from this and continue on to transmit their results before terminating. So in this instance, the sentinel values are not queued, but the rest of the execution continues as usual.

text_is_relevant(text)

Check if the text has any relevant words.

yield_batches(texts)

Return a generator over the given texts that yields batches of batch_size texts at a time.

class gensim.topic_coherence.text_analysis.PatchedWordOccurrenceAccumulator(*args)

Bases: gensim.topic_coherence.text_analysis.WordOccurrenceAccumulator

Monkey patched for multiprocessing worker usage, to move some of the logic to the master process.

Parameters

  • relevant_ids (set of int) – Relevant id

  • dictionary (Dictionary) – Dictionary instance with mappings for the relevant_ids.

__getitem__(word_or_words)
accumulate(texts, window_size)
analyze_text(window, doc_num=None)
get_co_occurrences(word1, word2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word)

Return number of docs the word occurs in, once accumulate has been called.

merge(other)
property num_docs
partial_accumulate(texts, window_size)

Meant to be called several times to accumulate partial results.

Notes

The final accumulation should be performed with the accumulate method as opposed to this one. This method does not ensure the co-occurrence matrix is in lil format and does not symmetrize it after accumulation.

text_is_relevant(text)

Check if the text has any relevant words.

class gensim.topic_coherence.text_analysis.UsesDictionary(relevant_ids, dictionary)

Bases: gensim.topic_coherence.text_analysis.BaseAnalyzer

A BaseAnalyzer that uses a Dictionary, hence can translate tokens to counts. The standard BaseAnalyzer can only deal with token ids since it doesn’t have the token2id mapping.

relevant_words

Set of words that occurrences should be accumulated for.

Type

set

dictionary

Dictionary based on text

Type

Dictionary

token2id

Mapping from Dictionary

Type

dict

Parameters

  • relevant_ids (dict) – Mapping

  • dictionary (Dictionary) – Dictionary based on text

Examples

>>> from gensim.topic_coherence import text_analysis
>>> from gensim.corpora.dictionary import Dictionary
>>>
>>> ids = {1: 'foo', 2: 'bar'}
>>> dictionary = Dictionary([['foo', 'bar', 'baz'], ['foo', 'bar', 'bar', 'baz']])
>>> udict = text_analysis.UsesDictionary(ids, dictionary)
>>>
>>> print(udict.relevant_words)
set([u'foo', u'baz'])
__getitem__(word_or_words)
analyze_text(text, doc_num=None)
get_co_occurrences(word1, word2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word)

Return number of docs the word occurs in, once accumulate has been called.

property num_docs
class gensim.topic_coherence.text_analysis.WindowedTextsAnalyzer(relevant_ids, dictionary)

Bases: gensim.topic_coherence.text_analysis.UsesDictionary

Gather some stats about relevant terms of a corpus by iterating over windows of texts.

Parameters

  • relevant_ids (set of int) – Relevant id

  • dictionary (Dictionary) – Dictionary instance with mappings for the relevant_ids.

__getitem__(word_or_words)
accumulate(texts, window_size)
analyze_text(text, doc_num=None)
get_co_occurrences(word1, word2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word)

Return number of docs the word occurs in, once accumulate has been called.

property num_docs
text_is_relevant(text)

Check if the text has any relevant words.

class gensim.topic_coherence.text_analysis.WordOccurrenceAccumulator(*args)

Bases: gensim.topic_coherence.text_analysis.WindowedTextsAnalyzer

Accumulate word occurrences and co-occurrences from a sequence of corpus texts.

Parameters

  • relevant_ids (set of int) – Relevant id

  • dictionary (Dictionary) – Dictionary instance with mappings for the relevant_ids.

__getitem__(word_or_words)
accumulate(texts, window_size)
analyze_text(window, doc_num=None)
get_co_occurrences(word1, word2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word)

Return number of docs the word occurs in, once accumulate has been called.

merge(other)
property num_docs
partial_accumulate(texts, window_size)

Meant to be called several times to accumulate partial results.

Notes

The final accumulation should be performed with the accumulate method as opposed to this one. This method does not ensure the co-occurrence matrix is in lil format and does not symmetrize it after accumulation.

text_is_relevant(text)

Check if the text has any relevant words.

class gensim.topic_coherence.text_analysis.WordVectorsAccumulator(relevant_ids, dictionary, model=None, **model_kwargs)

Bases: gensim.topic_coherence.text_analysis.UsesDictionary

Accumulate context vectors for words using word vector embeddings.

model

If None, a new Word2Vec model is trained on the given text corpus. Otherwise, it should be a pre-trained Word2Vec context vectors.

Type

Word2Vec (KeyedVectors)

model_kwargs

if model is None, these keyword arguments will be passed through to the Word2Vec constructor.

Parameters

  • relevant_ids (dict) – Mapping

  • dictionary (Dictionary) – Dictionary based on text

Examples

>>> from gensim.topic_coherence import text_analysis
>>> from gensim.corpora.dictionary import Dictionary
>>>
>>> ids = {1: 'foo', 2: 'bar'}
>>> dictionary = Dictionary([['foo', 'bar', 'baz'], ['foo', 'bar', 'bar', 'baz']])
>>> udict = text_analysis.UsesDictionary(ids, dictionary)
>>>
>>> print(udict.relevant_words)
set([u'foo', u'baz'])
__getitem__(word_or_words)
accumulate(texts, window_size)
analyze_text(text, doc_num=None)
get_co_occurrences(word1, word2)

Return number of docs the words co-occur in, once accumulate has been called.

get_occurrences(word)

Return number of docs the word occurs in, once accumulate has been called.

ids_similarity(ids1, ids2)
not_in_vocab(words)
property num_docs