It is used to split a stream of text into a series of tokens, where each token is a subsequence of characters in the text. The token produced is then passed through Token Filters that can add, remove or update the tokens. Later, that field is indexed by the resulting token stream.
