---
title: Downloading a LlamaDataset from LlamaHub
 | Developer Documentation
---

You can browse our available benchmark datasets via [llamahub.ai](https://llamahub.ai/). This notebook guide depicts how you can download the dataset and its source text documents. In particular, the `download_llama_dataset` will download the evaluation dataset (i.e., `LabelledRagDataset`) as well as the `Document`’s of the source text files used to build the evaluation dataset in the first place.

Finally, in this notebook, we also demonstrate the end to end workflow of downloading an evaluation dataset, making predictions on it using your own RAG pipeline (query engine) and then evaluating these predictions.

```
%pip install llama-index-llms-openai
```

```
from llama_index.core.llama_dataset import download_llama_dataset


# download and install dependencies
rag_dataset, documents = download_llama_dataset(
    "PaulGrahamEssayDataset", "./paul_graham"
)
```

```
github url: https://raw.githubusercontent.com/nerdai/llama-hub/datasets/llama_hub/llama_datasets/library.json
github url: https://media.githubusercontent.com/media/run-llama/llama_datasets/main/llama_datasets/paul_graham_essay/rag_dataset.json
github url: https://media.githubusercontent.com/media/run-llama/llama_datasets/main/llama_datasets/paul_graham_essay/source.txt
```

```
rag_dataset.to_pandas()[:5]
```

```
.dataframe tbody tr th {
    vertical-align: top;
}


.dataframe thead th {
    text-align: right;
}
```

|   | query                                             | reference\_contexts                                | reference\_answer                                 | reference\_answer\_by | query\_by  |
| - | ------------------------------------------------- | -------------------------------------------------- | ------------------------------------------------- | --------------------- | ---------- |
| 0 | In the essay, the author mentions his early ex... | \[What I Worked On\n\nFebruary 2021\n\nBefore c... | The first computer the author used for program... | ai (gpt-4)            | ai (gpt-4) |
| 1 | The author switched his major from philosophy ... | \[What I Worked On\n\nFebruary 2021\n\nBefore c... | The two specific influences that led the autho... | ai (gpt-4)            | ai (gpt-4) |
| 2 | In the essay, the author discusses his initial... | \[I couldn't have put this into words when I wa... | The two main influences that initially drew th... | ai (gpt-4)            | ai (gpt-4) |
| 3 | The author mentions his shift of interest towa... | \[I couldn't have put this into words when I wa... | The author shifted his interest towards Lisp a... | ai (gpt-4)            | ai (gpt-4) |
| 4 | In the essay, the author mentions his interest... | \[So I looked around to see what I could salvag... | The author in the essay is Paul Graham, who wa... | ai (gpt-4)            | ai (gpt-4) |

With `documents`, you can build your own RAG pipeline, to then predict and perform evaluations to compare against the benchmarks listed in the `DatasetCard` associated with the datasets [llamahub.ai](https://llamahub.ai/).

### Predictions

**NOTE**: The rest of the notebook illustrates how to manually perform predictions and subsequent evaluations for demonstrative purposes. Alternatively you can use the `RagEvaluatorPack` that will take care of predicting and evaluating using a RAG system that you would have provided.

```
from llama_index.core import VectorStoreIndex


# a basic RAG pipeline, uses defaults
index = VectorStoreIndex.from_documents(documents=documents)
query_engine = index.as_query_engine()
```

You can now create predictions and perform evaluation manually or download the `PredictAndEvaluatePack` to do this for you in a single line of code.

```
import nest_asyncio


nest_asyncio.apply()
```

```
# manually
prediction_dataset = await rag_dataset.amake_predictions_with(
    query_engine=query_engine, show_progress=True
)
```

```
100%|███████████████████████████████████████████████████████| 44/44 [00:08<00:00,  4.90it/s]
```

```
prediction_dataset.to_pandas()[:5]
```

```
.dataframe tbody tr th {
    vertical-align: top;
}


.dataframe thead th {
    text-align: right;
}
```

|   | response                                          | contexts                                           |
| - | ------------------------------------------------- | -------------------------------------------------- |
| 0 | The author mentions that the first computer he... | \[What I Worked On\n\nFebruary 2021\n\nBefore c... |
| 1 | The author switched his major from philosophy ... | \[I couldn't have put this into words when I wa... |
| 2 | The author mentions two main influences that i... | \[I couldn't have put this into words when I wa... |
| 3 | The author mentions that he shifted his intere... | \[So I looked around to see what I could salvag... |
| 4 | The author mentions his interest in both compu... | \[What I Worked On\n\nFebruary 2021\n\nBefore c... |

### Evaluation

Now that we have our predictions, we can perform evaluations on two dimensions:

1. The generated response: how well the predicted response matches the reference answer.
2. The retrieved contexts: how well the retrieved contexts for the prediction match the reference contexts.

NOTE: For retrieved contexts, we are unable to use standard retrieval metrics such as `hit rate` and `mean reciprocal rank` due to the fact that doing so requires we have the same index that was used to generate the ground truth data. But, it is not necessary for a `LabelledRagDataset` to be even created by an index. As such, we will use `semantic similarity` between the prediction’s contexts and the reference contexts as a measure of goodness.

```
import tqdm
```

For evaluating the response, we will use the LLM-As-A-Judge pattern. Specifically, we will use `CorrectnessEvaluator`, `FaithfulnessEvaluator` and `RelevancyEvaluator`.

For evaluating the goodness of the retrieved contexts we will use `SemanticSimilarityEvaluator`.

```
# instantiate the gpt-4 judge
from llama_index.llms.openai import OpenAI
from llama_index.core.evaluation import (
    CorrectnessEvaluator,
    FaithfulnessEvaluator,
    RelevancyEvaluator,
    SemanticSimilarityEvaluator,
)


judges = {}


judges["correctness"] = CorrectnessEvaluator(
    llm=OpenAI(temperature=0, model="gpt-4"),
)


judges["relevancy"] = RelevancyEvaluator(
    llm=OpenAI(temperature=0, model="gpt-4"),
)


judges["faithfulness"] = FaithfulnessEvaluator(
    llm=OpenAI(temperature=0, model="gpt-4"),
)


judges["semantic_similarity"] = SemanticSimilarityEvaluator()
```

Loop through the (`labelled_example`, `prediction`) pais and perform the evaluations on each of them individually.

```
evals = {
    "correctness": [],
    "relevancy": [],
    "faithfulness": [],
    "context_similarity": [],
}


for example, prediction in tqdm.tqdm(
    zip(rag_dataset.examples, prediction_dataset.predictions)
):
    correctness_result = judges["correctness"].evaluate(
        query=example.query,
        response=prediction.response,
        reference=example.reference_answer,
    )


    relevancy_result = judges["relevancy"].evaluate(
        query=example.query,
        response=prediction.response,
        contexts=prediction.contexts,
    )


    faithfulness_result = judges["faithfulness"].evaluate(
        query=example.query,
        response=prediction.response,
        contexts=prediction.contexts,
    )


    semantic_similarity_result = judges["semantic_similarity"].evaluate(
        query=example.query,
        response="\n".join(prediction.contexts),
        reference="\n".join(example.reference_contexts),
    )


    evals["correctness"].append(correctness_result)
    evals["relevancy"].append(relevancy_result)
    evals["faithfulness"].append(faithfulness_result)
    evals["context_similarity"].append(semantic_similarity_result)
```

```
44it [07:15,  9.90s/it]
```

```
import json


# saving evaluations
evaluations_objects = {
    "context_similarity": [e.dict() for e in evals["context_similarity"]],
    "correctness": [e.dict() for e in evals["correctness"]],
    "faithfulness": [e.dict() for e in evals["faithfulness"]],
    "relevancy": [e.dict() for e in evals["relevancy"]],
}


with open("evaluations.json", "w") as json_file:
    json.dump(evaluations_objects, json_file)
```

Now, we can use our notebook utility functions to view these evaluations.

```
import pandas as pd
from llama_index.core.evaluation.notebook_utils import get_eval_results_df


deep_eval_df, mean_correctness_df = get_eval_results_df(
    ["base_rag"] * len(evals["correctness"]),
    evals["correctness"],
    metric="correctness",
)
deep_eval_df, mean_relevancy_df = get_eval_results_df(
    ["base_rag"] * len(evals["relevancy"]),
    evals["relevancy"],
    metric="relevancy",
)
_, mean_faithfulness_df = get_eval_results_df(
    ["base_rag"] * len(evals["faithfulness"]),
    evals["faithfulness"],
    metric="faithfulness",
)
_, mean_context_similarity_df = get_eval_results_df(
    ["base_rag"] * len(evals["context_similarity"]),
    evals["context_similarity"],
    metric="context_similarity",
)


mean_scores_df = pd.concat(
    [
        mean_correctness_df.reset_index(),
        mean_relevancy_df.reset_index(),
        mean_faithfulness_df.reset_index(),
        mean_context_similarity_df.reset_index(),
    ],
    axis=0,
    ignore_index=True,
)
mean_scores_df = mean_scores_df.set_index("index")
mean_scores_df.index = mean_scores_df.index.set_names(["metrics"])
```

```
mean_scores_df
```

```
.dataframe tbody tr th {
    vertical-align: top;
}


.dataframe thead th {
    text-align: right;
}
```

| rag                              | base\_rag |
| -------------------------------- | --------- |
| metrics                          |           |
| mean\_correctness\_score         | 4.238636  |
| mean\_relevancy\_score           | 0.977273  |
| mean\_faithfulness\_score        | 0.977273  |
| mean\_context\_similarity\_score | 0.933568  |

On this toy example, we see that the basic RAG pipeline performs quite well against the evaluation benchmark (`rag_dataset`)! For completeness, to perform the above steps instead by using the `RagEvaluatorPack`, use the code provided below:

```
from llama_index.core.llama_pack import download_llama_pack


RagEvaluatorPack = download_llama_pack("RagEvaluatorPack", "./pack")
rag_evaluator = RagEvaluatorPack(
    query_engine=query_engine, rag_dataset=rag_dataset, show_progress=True
)


############################################################################
# NOTE: If have a lower tier subscription for OpenAI API like Usage Tier 1 #
# then you'll need to use different batch_size and sleep_time_in_seconds.  #
# For Usage Tier 1, settings that seemed to work well were batch_size=5,   #
# and sleep_time_in_seconds=15 (as of December 2023.)                      #
############################################################################


benchmark_df = await rag_evaluator_pack.arun(
    batch_size=20,  # batches the number of openai api calls to make
    sleep_time_in_seconds=1,  # seconds to sleep before making an api call
)
```