Contributing A `LabelledRagDataset`
Building a more robust RAG system requires a diversified evaluation suite. That is
why we launched LlamaDatasets in llama-hub. In this page,
we discuss how you can contribute the first kind of LlamaDataset made available
in llama-hub, that is, LabelledRagDataset.
Contributing a LabelledRagDataset involves two high level steps. Generally speaking,
you must create the LabelledRagDataset, save it as a json and submit both this
json file and the source text files to our llama-datasets repository. Additionally, you’ll have to make
a pull request, to upload required metadata of the dataset to our llama-hub repository.
To help make the submission process a lot smoother, we’ve prepared a template
notebook that you can follow to create a LabelledRagDataset from scratch (or
convert a similarly structured question-answering dataset into one) and perform
other required steps to make your submission. Please refer to the “LlamaDataset Submission Template Notebook” linked below.
Contributing Other llama-datasets
Section titled “Contributing Other llama-datasets”The general process for contributing any of our other llama-datasets such as the
LabelledEvaluatorDataset is the same as for the LabelledRagDataset previously
described. Submission templates for these other datasets are coming soon!
Submission Example
Section titled “Submission Example”Read the full submission example Notebook.