0 votes
in Generative AI by

Consider the following Python code snippet:

```python

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-base")

retriever = RagRetriever.from_pretrained("facebook/rag-token-base")

generator = RagTokenForGeneration.from_pretrained("facebook/rag-token-base")

query = "What is the capital of France?"

context = "France is a country located in Western Europe. Its capital is Paris."

input_dict = tokenizer.prepare_seq2seq_batch(query, return_tensors="pt")

generated = generator.generate(input_ids=input_dict['input_ids'])

print(tokenizer.decode(generated[0], skip_special_tokens=True))

```

What is the purpose of the `tokenizer.prepare_seq2seq_batch()` method in this code?

a. To prepare the input query for the sequence-to-sequence generation model by tokenizing the query and converting it into the appropriate tensor format.

b. To preprocess the retrieved passages for the RAGRetriever

1 Answer

0 votes
by
To prepare the input query for the sequence-to-sequence generation model by tokenizing the query and converting it into the appropriate tensor format.
...