torchtext.data.utils

ngrams_iterator

torchtext.data.utils.ngrams_iterator(token_list, ngrams)

Return an iterator that yields the given tokens and their ngrams.

Parameters

• token_list – A list of tokens

• ngrams – the number of ngrams.

Examples

>>> token_list = ['here', 'we', 'are']
>>> list(ngrams_iterator(token_list, 2))
>>> ['here', 'here we', 'we', 'we are', 'are']