0 votes
in Generative AI by
In the following Python code snippet, what is the purpose of the `skip_special_tokens=True` argument?

```python

from transformers import RagTokenizer, RagTokenForGeneration

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-base")

generator = RagTokenForGeneration.from_pretrained("facebook/rag-token-base")

input_ids = tokenizer.encode("What is the capital of France?", return_tensors="pt")

generated = generator.generate(input_ids=input_ids)

print(tokenizer.decode(generated[0], skip_special_tokens=True))

```

a. Skips tokenization of special characters.

b. Skips the generation of special tokens

c. Skips the decoding of special tokens.

1 Answer

0 votes
by
Skips the decoding of special tokens.
...