---
title: Refine with Structured Answer Filtering
 | LlamaIndex OSS Documentation
---

When using our Refine response synthesizer for response synthesis, it’s crucial to filter out non-answers. An issue often encountered is the propagation of a single unhelpful response like “I don’t have the answer”, which can persist throughout the synthesis process and lead to a final answer of the same nature. This can occur even when there are actual answers present in other, more relevant sections.

These unhelpful responses can be filtered out by setting `structured_answer_filtering` to `True`. It is set to `False` by default since this currently only works best if you are using an OpenAI model that supports function calling.

If you’re opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

```
%pip install llama-index-llms-openai
```

```
!pip install llama-index
```

## Load Data

```
texts = [
    "The president in the year 2040 is John Cena.",
    "The president in the year 2050 is Florence Pugh.",
    'The president in the year 2060 is Dwayne "The Rock" Johnson.',
]
```

## Summarize

```
import os


os.environ["OPENAI_API_KEY"] = "sk-..."
```

```
from llama_index.llms.openai import OpenAI


llm = OpenAI(model="gpt-3.5-turbo-0613")
```

```
from llama_index.core import get_response_synthesizer


summarizer = get_response_synthesizer(
    response_mode="refine", llm=llm, verbose=True
)
```

```
response = summarizer.get_response("who is president in the year 2050?", texts)
```

```
> Refine context: The president in the year 2050 is Florence Pugh.
> Refine context: The president in the year 2060 is Dwayne "The R...
```

### Failed Result

As you can see, we weren’t able to get the correct answer from the input `texts` strings since the initial “I don’t know” answer propogated through till the end of the response synthesis.

```
print(response)
```

```
I'm sorry, but I don't have access to information about the future.
```

Now we’ll try again with `structured_answer_filtering=True`

```
from llama_index.core import get_response_synthesizer


summarizer = get_response_synthesizer(
    response_mode="refine",
    llm=llm,
    verbose=True,
    structured_answer_filtering=True,
)
```

```
response = summarizer.get_response("who is president in the year 2050?", texts)
```

```
Function call: StructuredRefineResponse with args: {
  "answer": "It is not possible to determine who the president is in the year 2050 based on the given context information.",
  "query_satisfied": false
}
> Refine context: The president in the year 2050 is Florence Pugh.
Function call: StructuredRefineResponse with args: {
  "answer": "Florence Pugh",
  "query_satisfied": true
}
> Refine context: The president in the year 2060 is Dwayne "The R...
Function call: StructuredRefineResponse with args: {
  "answer": "Florence Pugh",
  "query_satisfied": false
}
```

### Successful Result

As you can see, we were able to determine the correct answer from the given context by filtering the `texts` strings for the ones that actually contained the answer to our question.

```
print(response)
```

```
Florence Pugh
```

## Non Function-calling LLMs

You may want to make use of this filtering functionality with an LLM that doesn’t offer a function calling API.

In that case, the `Refine` module will automatically switch to using a structured output `Program` that doesn’t rely on an external function calling API.

```
# we'll stick with OpenAI but use an older model that does not support function calling
instruct_llm = OpenAI(model="gpt-3.5-turbo-instruct")
```

```
from llama_index.core import get_response_synthesizer


summarizer = get_response_synthesizer(
    response_mode="refine",
    llm=instruct_llm,
    verbose=True,
    structured_answer_filtering=True,
)
```

```
response = summarizer.get_response("who is president in the year 2050?", texts)
print(response)
```

```
Florence Pugh
```

### `CompactAndRefine`

Since `CompactAndRefine` is built on top of `Refine`, this response mode also supports structured answer filtering.

```
from llama_index.core import get_response_synthesizer


summarizer = get_response_synthesizer(
    response_mode="compact",
    llm=instruct_llm,
    verbose=True,
    structured_answer_filtering=True,
)
```

```
response = summarizer.get_response("who is president in the year 2050?", texts)
print(response)
```

```
Florence Pugh
```