Migration Guide: Index v1 to v2

You only need this page if you have existing Index v1 pipelines. If you are starting fresh, use the getting started guide.

Index v1 (pipelines, data sources, data sinks) is deprecated. Index v2 uses a simpler model: you put files in a directory, create an index over that directory, and query it with the retrieval and chat APIs.

This guide covers what maps over, what changed, what v2 doesn’t support, and how to move an existing pipeline.

Before you migrate

• Index v2 needs a Starter, Pro, or Enterprise plan.
• Python SDK: pip install "llama-cloud>=2.8". TypeScript SDK: npm install @llamaindex/llama-cloud.
• All Index v2 SDK methods live under client.beta.*: client.beta.directories, client.beta.indexes, client.beta.retrieval, and client.beta.chat.
• v1 and v2 don’t share storage. Migrating means adding your files to v2, which parses, chunks, and embeds them again, so plan for the parse credits.

Concept mapping

Index v1	Index v2	Notes
Pipeline	Index	One index per source directory.
Data source (connector or file upload)	Directory	Upload files into a directory, or sync them into it from a connected folder (SharePoint, Google Drive, S3).
Data sink (Pinecone, Qdrant, Postgres, …)	Managed vector store, or a custom export	v2 writes to a vector store LlamaCloud manages. To keep the data in your own database, export the parsed output and load it yourself. See Replace data sinks with a custom export.
sync_interval on a data source	POST /api/v1/indexes/{index_id}/sync	You trigger syncs. See Sync an index on a schedule.
transform_config / embedding_config	None	LlamaCloud manages chunking and embedding; neither is configurable.
llama_parse_parameters	Saved Parse configuration passed in products	Defaults to the cost_effective tier.
Pipeline file custom_metadata	Directory file metadata	Filterable at query time with custom_filters.
pipelines.retrieve	retrieval.retrieve	Hybrid search, with reranking on by default.
Retrieval modes files_via_metadata / files_via_content	retrieval.find / grep / read	File operations inside an index.
Pipeline chat	chat sessions	Streaming chat agent, up to 10 indexes per session.

Feature parity

Legend: ✅ supported · 🔄 supported, works differently · ❌ not supported

Ingestion feature parity

Capability	v1	v2
File upload via API	✅	✅ client.beta.directories.files.upload / .add
SharePoint	✅	🔄 connected folder, set up in the web app
Google Drive	✅	🔄 connected folder, set up in the web app
Amazon S3	✅	🔄 connected folder, set up in the web app
OneDrive	✅	❌
Box	✅	❌
Confluence	✅	❌
Jira	✅	❌
Azure Blob Storage	✅	❌
Insert raw text documents (pipelines.documents.create)	✅	❌ upload files instead
Per-file custom metadata	✅	✅ metadata on directory files

For a source v2 doesn’t support, pull the files with your own code (for example an open-source LlamaIndex reader or the provider’s SDK) and upload them to the directory. Running that script on a schedule does the job a v1 connector did.

Processing feature parity

Capability	v1	v2
Parse settings	✅ llama_parse_parameters	🔄 saved Parse configuration referenced from products
Configurable chunking	✅ transform_config	❌ fixed: sentence-based, 1024 tokens per chunk, 200-token overlap
Choice of embedding model / bring your own key	✅ OpenAI, Azure, Bedrock, Cohere, Gemini, HuggingFace	❌ managed by LlamaCloud
Change detection on sync	✅	✅ unchanged files are skipped; a change to metadata alone doesn’t parse the file again

Storage feature parity

Capability	v1	v2
Managed vector store	✅	✅ default
Your own vector database as a data sink	✅ Pinecone, Qdrant, Milvus, Postgres, MongoDB Atlas, Azure AI Search, AstraDB	❌ use a custom export

Retrieval feature parity

Capability	v1	v2
Dense (vector) search	✅	✅
Hybrid (vector + keyword)	✅	✅ default, with tunable weights
Reranking	✅	✅ on by default
Metadata filters	✅ MetadataFilters	🔄 custom_filters (eq, ne, gt, lt, gte, lte, in, nin)
Filter by file	✅	✅ static_filters on parsed_directory_file_id
Find, grep, and read files in an index	❌	✅ new
Page screenshots returned with results	✅ retrieve_page_screenshot_nodes	🔄 create the index with store_attachments=["screenshots"]. Each result lists the screenshots of the pages it covers, and you fetch them from the attachments endpoint. Search itself is text-only.
Figure-level image retrieval (retrieve_page_figure_nodes)	✅	❌ store_attachments=["items"] stores structured items with bounding boxes, but figures aren’t returned as separate results
Composite retrieval (several indexes, one call)	✅ client.retrievers	❌ query each index and merge the results, or use a chat session over several indexes
Auto-routed retrieval mode	✅	❌
Chat agent	🔄 basic	✅ streaming agent with sessions, up to 10 indexes; it can look at page screenshots when the index stores them
LlamaCloudIndex framework integration	✅	❌ call the retrieval API directly
Chunk-level listing (documents/{id}/chunks)	✅	❌

Endpoint mapping

Index v2 REST paths under /api/v1/indexes, /api/v1/retrieval, and /api/v1/chat have no beta segment, but the SDK methods for them all live under client.beta.

v1 route	v2 route	v2 SDK method
POST /api/v1/data-sources	POST /api/v1/beta/directories	client.beta.directories.create
POST /api/v1/pipelines/{id}/files	POST /api/v1/beta/directories/{id}/files (existing file_id) or .../files/upload (multipart)	client.beta.directories.files.add / .upload
GET /api/v1/pipelines/{id}/files	GET /api/v1/beta/directories/{id}/files	client.beta.directories.files.list
DELETE /api/v1/pipelines/{id}/files/{file_id}	DELETE /api/v1/beta/directories/{id}/files/{file_id}	client.beta.directories.files.delete
POST /api/v1/data-sinks	none	use a custom export
PUT /api/v1/pipelines (create / upsert)	POST /api/v1/indexes	client.beta.indexes.create
GET /api/v1/pipelines	GET /api/v1/indexes	client.beta.indexes.list
GET /api/v1/pipelines/{id} and /status	GET /api/v1/indexes/{id}	client.beta.indexes.get (status in metadata["status"])
DELETE /api/v1/pipelines/{id}	DELETE /api/v1/indexes/{id}	client.beta.indexes.delete
POST /api/v1/pipelines/{id}/sync	POST /api/v1/indexes/{id}/sync	client.beta.indexes.sync
POST /api/v1/pipelines/{id}/retrieve	POST /api/v1/retrieval/retrieve (index_id in the body)	client.beta.retrieval.retrieve
retrieve_page_screenshot_nodes on pipelines/{id}/retrieve	GET /api/v1/beta/attachments/{attachment_name}?source_id=... (presigned URL) and GET /api/v1/beta/attachments?source_id=... (list)	none yet; call the REST endpoint
none	POST /api/v1/retrieval/files/find / grep / read	client.beta.retrieval.find / grep / read
POST /api/v1/pipelines/{id}/chat	POST /api/v1/chat, then POST /api/v1/chat/{session_id}/messages/stream (SSE)	client.beta.chat.create / .stream
POST /api/v1/retrievers/{id}/retrieve	none	not supported

Deleting an index removes its sync and export configuration. The source directory and its files stay.

Migrate a pipeline step by step

The examples below use the async Python SDK. The TypeScript SDK has the same method names and takes an options object.

Step 1: Create a directory

from llama_cloud import AsyncLlamaCloud

client = AsyncLlamaCloud()  # reads LLAMA_CLOUD_API_KEY

directory = await client.beta.directories.create(
    name="product-docs",
    description="Migrated from an Index v1 pipeline",
)

SharePoint, Google Drive, or S3 source: create a connected folder in the LlamaCloud web app instead, from the Files page or from Connect a folder when creating an index. LlamaCloud syncs the folder’s contents into a directory for you. The public API can’t set up connected folders yet.

Step 2: Add files to the directory

Upload files directly, attaching the metadata you set as custom_metadata in v1:

from pathlib import Path

for path in Path("./docs").glob("*.pdf"):
    await client.beta.directories.files.upload(
        directory.id,
        upload_file=path,
        display_name=path.name,
        unique_id=str(path),  # stable ID, so a re-upload replaces the file instead of duplicating it
        metadata='{"department": "legal", "year": 2026}',  # JSON string on upload
    )

Or add a file you already uploaded with client.files.create:

file_obj = await client.files.create(file=open("report.pdf", "rb"), purpose="user_data")
await client.beta.directories.files.add(
    directory.id,
    file_id=file_obj.id,
    metadata={"department": "legal"},
)

Step 3: Save a Parse configuration (optional)

An index uses the Parse cost_effective tier by default. To use other settings in place of v1’s llama_parse_parameters, save a Parse configuration:

parse_config = await client.configurations.create(
    name="Index parse config",
    parameters={"product_type": "parse_v2", "version": "latest", "tier": "agentic"},
)

Step 4: Create the index

index = await client.beta.indexes.create(
    source_directory_id=directory.id,
    name="product-docs",
    # Omit `products` to use the default Parse configuration.
    products=[{"product_type": "parse", "product_config_id": parse_config.id}],
    # Store per-page screenshots, the v2 replacement for v1 page screenshot nodes.
    # Add "items" to also store structured items with bounding boxes.
    store_attachments=["screenshots"],
)

Set store_attachments when you create the index. An index can’t be updated after creation, so adding screenshots later means creating a new index over the same directory.

Creating an index starts its first sync, which parses, chunks, embeds, and stores every file in the directory.

Step 5: Wait for the index to be ready

import asyncio

while True:
    idx = await client.beta.indexes.get(index.id)
    status = idx.metadata.get("status")
    if status == "ready":
        break
    if status == "failed":
        raise RuntimeError(idx.metadata.get("error_message"))
    await asyncio.sleep(10)

Only ready and failed are stable status values. Treat any other value as still in progress.

Step 6: Update retrieval calls

A v1 retrieval call:

results = await client.pipelines.retrieve(
    pipeline_id=PIPELINE_ID,
    query="What is the refund policy?",
    dense_similarity_top_k=10,
    sparse_similarity_top_k=10,
    alpha=0.5,
    enable_reranking=True,
    rerank_top_n=5,
    search_filters={"filters": [{"key": "department", "value": "legal", "operator": "=="}]},
)
for node in results.retrieval_nodes:
    print(node.score, node.node.text)

The same call in v2:

results = await client.beta.retrieval.retrieve(
    index_id=index.id,
    query="What is the refund policy?",
    top_k=5,
    num_candidates=100,
    vector_pipeline_weight=0.5,
    full_text_pipeline_weight=0.5,
    rerank={"enabled": True, "top_n": 5},
    custom_filters={"department": {"operator": "eq", "value": "legal"}},
)
for r in results.results:
    print(r.score, r.rerank_score, r.content)
    print(r.static_fields.parsed_directory_file_id, r.static_fields.page_range_start)

v1 retrieval parameters map to v2 like this:

v1	v2
pipeline_id	index_id
dense_similarity_top_k / sparse_similarity_top_k	num_candidates (candidate pool for each search) and top_k (results returned, max 500)
alpha	vector_pipeline_weight / full_text_pipeline_weight (defaults 0.5 / 0.5)
enable_reranking	rerank.enabled (default true)
rerank_top_n	rerank.top_n
dense_similarity_cutoff	score_threshold
search_filters ({"filters": [...]})	custom_filters, keyed by metadata field
filter on a file ID	static_filters on parsed_directory_file_id
retrieve_page_screenshot_nodes=True	store_attachments=["screenshots"] at index creation, then results.results[i].static_fields.attachments
results.retrieval_nodes[i].node.text	results.results[i].content

See Retrieval for every v2 parameter.

Step 7: Fetch page screenshots for retrieval results

When the index stores screenshots, each result’s static_fields.attachments lists one entry for every page the chunk covers, from page_range_start to page_range_end:

{"type": "screenshot", "attachment_name": "screenshots/page_7.jpg", "source_id": "dfl-..."}

GET /api/v1/beta/attachments/{attachment_name}?source_id={source_id} returns a presigned URL for the image. The SDKs don’t wrap this endpoint yet, so call it over HTTP:

import os
import httpx

API_BASE = "https://api.cloud.llamaindex.ai"  # EU: https://api.cloud.eu.llamaindex.ai
headers = {"Authorization": f"Bearer {os.environ['LLAMA_CLOUD_API_KEY']}"}

async with httpx.AsyncClient(headers=headers) as http:
    for r in results.results:
        for att in r.static_fields.attachments or []:
            if att.type != "screenshot":
                continue
            resp = await http.get(
                f"{API_BASE}/api/v1/beta/attachments/{att.attachment_name}",
                params={"source_id": att.source_id},
            )
            resp.raise_for_status()
            image_url = resp.json()["url"]  # presigned URL to the page JPEG

GET /api/v1/beta/attachments?source_id=... lists every attachment stored for a file.

Step 8: Replace file-level retrieval modes

The v1 files_via_metadata and files_via_content modes map to v2 file operations:

found = await client.beta.retrieval.find(index_id=index.id, file_name_contains="quarterly")
file_id = found.items[0].file_id

matches = await client.beta.retrieval.grep(
    index_id=index.id, file_id=file_id, pattern="revenue|profit", context_chars=100,
)
text = await client.beta.retrieval.read(
    index_id=index.id, file_id=file_id, offset=0, max_length=5000,
)

Step 9: Validate and cut over

Run the same evaluation queries against the v1 pipeline and the v2 index before you switch traffic. v2 chunks and embeds differently, so scores and ranking will change. Delete the v1 pipeline once you’re happy with the v2 results.

Replace data sinks with a custom export

Index v2 doesn’t support external data sinks: Pinecone, Qdrant, Milvus, Postgres, MongoDB Atlas, Azure AI Search, and AstraDB can’t be attached to an index. To keep the data in your own database, use a custom export. LlamaCloud still manages the directory, parsing, and sync; you chunk, embed, and write to your database.

1. Create a parse-only index with vector_target="DISABLED":

   index = await client.beta.indexes.create(
       source_directory_id=directory.id,
       vector_target="DISABLED",
   )

2. After each sync, download the parsed output from the index’s output directory. The source directory holds your original files; the index writes one parsed JSON payload per source file into the directory named by output_directory_id:

   import httpx
   
   idx = await client.beta.indexes.get(index.id)
   
   async with httpx.AsyncClient() as http:
       async for f in client.beta.directories.files.list(idx.output_directory_id):
           if f.file_id is None:
               continue
           presigned = await client.files.get(f.file_id)
           pages = (await http.get(presigned.url)).json()["parse"]["pages"]
           # f.id           -> parsed_directory_file_id, stable across syncs
           # f.metadata     -> the source file's metadata
           # f.updated_at   -> changes when the file is parsed again or its metadata changes

3. Chunk, embed, and write to your store:

   • Key records on (f.id, idx.export_config_id). Delete a file’s existing records before inserting its new chunks, so a file that’s parsed again replaces its old chunks instead of duplicating them.
   • Export only what changed: pass your last export time as updated_at_on_or_after when listing the output directory.
   • Delete records whose parsed_directory_file_id no longer appears in the output directory. A sync removes the output file when its source file is deleted, but LlamaCloud can’t clean up your database for you.

The index-v2-data-sinks repo has runnable exporters for MongoDB Atlas, PostgreSQL + pgvector, Qdrant, Pinecone, Turbopuffer, and Azure AI Search. They’re built on a shared Exporter protocol (export, delete_file, delete_files, list_snapshots); copy the one closest to your stack.

With vector_target="DISABLED" there is no managed vector store, so retrieval.*, chat, and the file operations have nothing to query. You serve retrieval from your own database. See Custom vector stores for the full walkthrough.

Sync an index on a schedule

v1 synced data sources on a schedule set by sync_interval (6, 12, or 24 hours). Index v2 has no built-in schedule: an index syncs once when you create it, and after that whenever you call sync.

What an index sync does

1. Compares the source directory with what the index last processed.
2. Parses new and changed files, skipping unchanged ones by their content fingerprint and source_modified_at. A change to metadata alone updates the metadata without parsing the file again.
3. Chunks, embeds, and stores the changed files, and deletes the chunks of files removed from the directory.

Changes to the directory (uploads, deletions, connected-folder updates) only reach the index on the next sync.

Trigger an index sync safely

Only one sync runs per index at a time: a second call while one is running returns 409 Conflict, and calling sync in rapid succession returns 429 Too Many Requests. Read last_synced_at before you trigger the sync, so the previous sync’s ready status isn’t mistaken for the new one:

import asyncio
from llama_cloud import ConflictError, RateLimitError


async def sync_and_wait(client, index_id, timeout_s=3600):
    before = (await client.beta.indexes.get(index_id)).last_synced_at
    try:
        await client.beta.indexes.sync(index_id)
    except ConflictError:
        pass  # a sync is already running; wait for it instead
    except RateLimitError:
        await asyncio.sleep(60)
        await client.beta.indexes.sync(index_id)

    delay, waited = 2, 0
    while waited < timeout_s:
        idx = await client.beta.indexes.get(index_id)
        status = idx.metadata.get("status")
        if status == "failed":
            raise RuntimeError(idx.metadata.get("error_message"))
        if status == "ready" and idx.last_synced_at != before:
            return idx
        await asyncio.sleep(delay)
        waited += delay
        delay = min(delay * 2, 30)
    raise TimeoutError(f"index {index_id} did not finish syncing")

See Syncing for the sync endpoint in every SDK.

Replace sync_interval with a scheduled job

Run sync_and_wait on a schedule you control, such as cron, a Kubernetes CronJob, Airflow, or a scheduled GitHub Actions workflow. A daily or 12-hourly schedule matches what most v1 pipelines used.

• Connected folders (SharePoint, Google Drive, S3): LlamaCloud refreshes a connected folder from its source periodically, but that refresh doesn’t sync the index. Keep the scheduled indexes.sync call so new and changed files reach the index.
• Sources v2 doesn’t support: have the same scheduled job pull changed files from the source, upload them to the directory with the same unique_id so they replace the old version, delete files that are gone, and then call sync.

Migration checklist

• List your v1 pipelines with their data sources, data sinks, and sync intervals.
• Check each data source against the ingestion feature parity table. For anything other than SharePoint, Google Drive, S3, or file upload, plan an upload script.
• For each pipeline that writes to your own vector database, plan a custom export.
• If you used page screenshot retrieval, create the v2 index with store_attachments=["screenshots"]. It can’t be added later.
• If you relied on composite retrieval, figure-level image retrieval, custom chunking, or a specific embedding model, confirm the v2 behavior works for you before migrating.
• Create directories and add files, carrying over custom_metadata.
• Create indexes, with a saved Parse configuration if needed, and wait for ready.
• Port retrieval calls using the parameter mapping.
• Schedule indexes.sync calls to replace sync_interval.
• Compare retrieval quality between v1 and v2 on your evaluation queries.
• Switch traffic to v2, then delete the v1 pipelines.