--- title: Migration Guide: Index v1 to v2 | Developer Documentation description: Migrate LlamaCloud Index v1 pipelines, data sources, and data sinks to Index v2 directories and indexes, with feature parity, endpoint and parameter mappings, and scheduled sync. --- > **You only need this page if you have existing Index v1 pipelines.** If you are starting fresh, use the [getting started guide](../getting_started). Index v1 (pipelines, data sources, data sinks) is deprecated. Index v2 uses a simpler model: you put files in a **directory**, create an **index** over that directory, and query it with the **retrieval** and **chat** APIs. This guide covers what maps over, what changed, what v2 doesn’t support, and how to move an existing pipeline. ## Before you migrate - Index v2 needs a Starter, Pro, or Enterprise plan. - Python SDK: `pip install "llama-cloud>=2.8"`. TypeScript SDK: `npm install @llamaindex/llama-cloud`. - All Index v2 SDK methods live under `client.beta.*`: `client.beta.directories`, `client.beta.indexes`, `client.beta.retrieval`, and `client.beta.chat`. - v1 and v2 don’t share storage. Migrating means adding your files to v2, which parses, chunks, and embeds them again, so plan for the parse credits. ## Concept mapping | Index v1 | Index v2 | Notes | | ---------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Pipeline | Index | One index per source directory. | | Data source (connector or file upload) | Directory | Upload files into a directory, or sync them into it from a connected folder (SharePoint, Google Drive, S3). | | Data sink (Pinecone, Qdrant, Postgres, …) | Managed vector store, or a custom export | v2 writes to a vector store LlamaCloud manages. To keep the data in your own database, export the parsed output and load it yourself. See [Replace data sinks with a custom export](#replace-data-sinks-with-a-custom-export). | | `sync_interval` on a data source | `POST /api/v1/indexes/{index_id}/sync` | You trigger syncs. See [Sync an index on a schedule](#sync-an-index-on-a-schedule). | | `transform_config` / `embedding_config` | None | LlamaCloud manages chunking and embedding; neither is configurable. | | `llama_parse_parameters` | Saved Parse configuration passed in `products` | Defaults to the `cost_effective` tier. | | Pipeline file `custom_metadata` | Directory file `metadata` | Filterable at query time with `custom_filters`. | | `pipelines.retrieve` | `retrieval.retrieve` | Hybrid search, with reranking on by default. | | Retrieval modes `files_via_metadata` / `files_via_content` | `retrieval.find` / `grep` / `read` | [File operations](../file_operations) inside an index. | | Pipeline chat | `chat` sessions | [Streaming chat agent](../chat), up to 10 indexes per session. | ## Feature parity Legend: ✅ supported · 🔄 supported, works differently · ❌ not supported ### Ingestion feature parity | Capability | v1 | v2 | | -------------------------------------------------------- | -- | ------------------------------------------------- | | File upload via API | ✅ | ✅ `client.beta.directories.files.upload` / `.add` | | SharePoint | ✅ | 🔄 connected folder, set up in the web app | | Google Drive | ✅ | 🔄 connected folder, set up in the web app | | Amazon S3 | ✅ | 🔄 connected folder, set up in the web app | | OneDrive | ✅ | ❌ | | Box | ✅ | ❌ | | Confluence | ✅ | ❌ | | Jira | ✅ | ❌ | | Azure Blob Storage | ✅ | ❌ | | Insert raw text documents (`pipelines.documents.create`) | ✅ | ❌ upload files instead | | Per-file custom metadata | ✅ | ✅ `metadata` on directory files | For a source v2 doesn’t support, pull the files with your own code (for example an open-source LlamaIndex reader or the provider’s SDK) and upload them to the directory. Running that script on a schedule does the job a v1 connector did. ### Processing feature parity | Capability | v1 | v2 | | ---------------------------------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------- | | Parse settings | ✅ `llama_parse_parameters` | 🔄 saved Parse configuration referenced from `products` | | Configurable chunking | ✅ `transform_config` | ❌ fixed: sentence-based, 1024 tokens per chunk, 200-token overlap | | Choice of embedding model / bring your own key | ✅ OpenAI, Azure, Bedrock, Cohere, Gemini, HuggingFace | ❌ managed by LlamaCloud | | Change detection on sync | ✅ | ✅ unchanged files are skipped; a change to metadata alone doesn’t parse the file again | ### Storage feature parity | Capability | v1 | v2 | | --------------------------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------- | | Managed vector store | ✅ | ✅ default | | Your own vector database as a data sink | ✅ Pinecone, Qdrant, Milvus, Postgres, MongoDB Atlas, Azure AI Search, AstraDB | ❌ use a [custom export](#replace-data-sinks-with-a-custom-export) | ### Retrieval feature parity | Capability | v1 | v2 | | ----------------------------------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Dense (vector) search | ✅ | ✅ | | Hybrid (vector + keyword) | ✅ | ✅ default, with tunable weights | | Reranking | ✅ | ✅ on by default | | Metadata filters | ✅ `MetadataFilters` | 🔄 `custom_filters` (`eq`, `ne`, `gt`, `lt`, `gte`, `lte`, `in`, `nin`) | | Filter by file | ✅ | ✅ `static_filters` on `parsed_directory_file_id` | | Find, grep, and read files in an index | ❌ | ✅ new | | Page screenshots returned with results | ✅ `retrieve_page_screenshot_nodes` | 🔄 create the index with `store_attachments=["screenshots"]`. Each result lists the screenshots of the pages it covers, and you fetch them from the attachments endpoint. Search itself is text-only. | | Figure-level image retrieval (`retrieve_page_figure_nodes`) | ✅ | ❌ `store_attachments=["items"]` stores structured items with bounding boxes, but figures aren’t returned as separate results | | Composite retrieval (several indexes, one call) | ✅ `client.retrievers` | ❌ query each index and merge the results, or use a chat session over several indexes | | Auto-routed retrieval mode | ✅ | ❌ | | Chat agent | 🔄 basic | ✅ streaming agent with sessions, up to 10 indexes; it can look at page screenshots when the index stores them | | `LlamaCloudIndex` framework integration | ✅ | ❌ call the retrieval API directly | | Chunk-level listing (`documents/{id}/chunks`) | ✅ | ❌ | ## Endpoint mapping Index v2 REST paths under `/api/v1/indexes`, `/api/v1/retrieval`, and `/api/v1/chat` have no `beta` segment, but the SDK methods for them all live under `client.beta`. | v1 route | v2 route | v2 SDK method | | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | `POST /api/v1/data-sources` | `POST /api/v1/beta/directories` | `client.beta.directories.create` | | `POST /api/v1/pipelines/{id}/files` | `POST /api/v1/beta/directories/{id}/files` (existing `file_id`) or `.../files/upload` (multipart) | `client.beta.directories.files.add` / `.upload` | | `GET /api/v1/pipelines/{id}/files` | `GET /api/v1/beta/directories/{id}/files` | `client.beta.directories.files.list` | | `DELETE /api/v1/pipelines/{id}/files/{file_id}` | `DELETE /api/v1/beta/directories/{id}/files/{file_id}` | `client.beta.directories.files.delete` | | `POST /api/v1/data-sinks` | none | use a custom export | | `PUT /api/v1/pipelines` (create / upsert) | `POST /api/v1/indexes` | `client.beta.indexes.create` | | `GET /api/v1/pipelines` | `GET /api/v1/indexes` | `client.beta.indexes.list` | | `GET /api/v1/pipelines/{id}` and `/status` | `GET /api/v1/indexes/{id}` | `client.beta.indexes.get` (status in `metadata["status"]`) | | `DELETE /api/v1/pipelines/{id}` | `DELETE /api/v1/indexes/{id}` | `client.beta.indexes.delete` | | `POST /api/v1/pipelines/{id}/sync` | `POST /api/v1/indexes/{id}/sync` | `client.beta.indexes.sync` | | `POST /api/v1/pipelines/{id}/retrieve` | `POST /api/v1/retrieval/retrieve` (`index_id` in the body) | `client.beta.retrieval.retrieve` | | `retrieve_page_screenshot_nodes` on `pipelines/{id}/retrieve` | `GET /api/v1/beta/attachments/{attachment_name}?source_id=...` (presigned URL) and `GET /api/v1/beta/attachments?source_id=...` (list) | none yet; call the REST endpoint | | none | `POST /api/v1/retrieval/files/find` / `grep` / `read` | `client.beta.retrieval.find` / `grep` / `read` | | `POST /api/v1/pipelines/{id}/chat` | `POST /api/v1/chat`, then `POST /api/v1/chat/{session_id}/messages/stream` (SSE) | `client.beta.chat.create` / `.stream` | | `POST /api/v1/retrievers/{id}/retrieve` | none | not supported | Deleting an index removes its sync and export configuration. The source directory and its files stay. ## Migrate a pipeline step by step The examples below use the async Python SDK. The TypeScript SDK has the same method names and takes an options object. ### Step 1: Create a directory ``` from llama_cloud import AsyncLlamaCloud client = AsyncLlamaCloud() # reads LLAMA_CLOUD_API_KEY directory = await client.beta.directories.create( name="product-docs", description="Migrated from an Index v1 pipeline", ) ``` **SharePoint, Google Drive, or S3 source:** create a connected folder in the LlamaCloud web app instead, from the **Files** page or from **Connect a folder** when creating an index. LlamaCloud syncs the folder’s contents into a directory for you. The public API can’t set up connected folders yet. ### Step 2: Add files to the directory Upload files directly, attaching the metadata you set as `custom_metadata` in v1: ``` from pathlib import Path for path in Path("./docs").glob("*.pdf"): await client.beta.directories.files.upload( directory.id, upload_file=path, display_name=path.name, unique_id=str(path), # stable ID, so a re-upload replaces the file instead of duplicating it metadata='{"department": "legal", "year": 2026}', # JSON string on upload ) ``` Or add a file you already uploaded with `client.files.create`: ``` file_obj = await client.files.create(file=open("report.pdf", "rb"), purpose="user_data") await client.beta.directories.files.add( directory.id, file_id=file_obj.id, metadata={"department": "legal"}, ) ``` ### Step 3: Save a Parse configuration (optional) An index uses the Parse `cost_effective` tier by default. To use other settings in place of v1’s `llama_parse_parameters`, save a Parse configuration: ``` parse_config = await client.configurations.create( name="Index parse config", parameters={"product_type": "parse_v2", "version": "latest", "tier": "agentic"}, ) ``` ### Step 4: Create the index ``` index = await client.beta.indexes.create( source_directory_id=directory.id, name="product-docs", # Omit `products` to use the default Parse configuration. products=[{"product_type": "parse", "product_config_id": parse_config.id}], # Store per-page screenshots, the v2 replacement for v1 page screenshot nodes. # Add "items" to also store structured items with bounding boxes. store_attachments=["screenshots"], ) ``` Set `store_attachments` when you create the index. An index can’t be updated after creation, so adding screenshots later means creating a new index over the same directory. Creating an index starts its first sync, which parses, chunks, embeds, and stores every file in the directory. ### Step 5: Wait for the index to be ready ``` import asyncio while True: idx = await client.beta.indexes.get(index.id) status = idx.metadata.get("status") if status == "ready": break if status == "failed": raise RuntimeError(idx.metadata.get("error_message")) await asyncio.sleep(10) ``` Only `ready` and `failed` are stable status values. Treat any other value as still in progress. ### Step 6: Update retrieval calls A v1 retrieval call: ``` results = await client.pipelines.retrieve( pipeline_id=PIPELINE_ID, query="What is the refund policy?", dense_similarity_top_k=10, sparse_similarity_top_k=10, alpha=0.5, enable_reranking=True, rerank_top_n=5, search_filters={"filters": [{"key": "department", "value": "legal", "operator": "=="}]}, ) for node in results.retrieval_nodes: print(node.score, node.node.text) ``` The same call in v2: ``` results = await client.beta.retrieval.retrieve( index_id=index.id, query="What is the refund policy?", top_k=5, num_candidates=100, vector_pipeline_weight=0.5, full_text_pipeline_weight=0.5, rerank={"enabled": True, "top_n": 5}, custom_filters={"department": {"operator": "eq", "value": "legal"}}, ) for r in results.results: print(r.score, r.rerank_score, r.content) print(r.static_fields.parsed_directory_file_id, r.static_fields.page_range_start) ``` v1 retrieval parameters map to v2 like this: | v1 | v2 | | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `pipeline_id` | `index_id` | | `dense_similarity_top_k` / `sparse_similarity_top_k` | `num_candidates` (candidate pool for each search) and `top_k` (results returned, max 500) | | `alpha` | `vector_pipeline_weight` / `full_text_pipeline_weight` (defaults 0.5 / 0.5) | | `enable_reranking` | `rerank.enabled` (default `true`) | | `rerank_top_n` | `rerank.top_n` | | `dense_similarity_cutoff` | `score_threshold` | | `search_filters` (`{"filters": [...]}`) | `custom_filters`, keyed by metadata field | | filter on a file ID | `static_filters` on `parsed_directory_file_id` | | `retrieve_page_screenshot_nodes=True` | `store_attachments=["screenshots"]` at index creation, then `results.results[i].static_fields.attachments` | | `results.retrieval_nodes[i].node.text` | `results.results[i].content` | See [Retrieval](../retrieval) for every v2 parameter. ### Step 7: Fetch page screenshots for retrieval results When the index stores screenshots, each result’s `static_fields.attachments` lists one entry for every page the chunk covers, from `page_range_start` to `page_range_end`: ``` {"type": "screenshot", "attachment_name": "screenshots/page_7.jpg", "source_id": "dfl-..."} ``` `GET /api/v1/beta/attachments/{attachment_name}?source_id={source_id}` returns a presigned URL for the image. The SDKs don’t wrap this endpoint yet, so call it over HTTP: ``` import os import httpx API_BASE = "https://api.cloud.llamaindex.ai" # EU: https://api.cloud.eu.llamaindex.ai headers = {"Authorization": f"Bearer {os.environ['LLAMA_CLOUD_API_KEY']}"} async with httpx.AsyncClient(headers=headers) as http: for r in results.results: for att in r.static_fields.attachments or []: if att.type != "screenshot": continue resp = await http.get( f"{API_BASE}/api/v1/beta/attachments/{att.attachment_name}", params={"source_id": att.source_id}, ) resp.raise_for_status() image_url = resp.json()["url"] # presigned URL to the page JPEG ``` `GET /api/v1/beta/attachments?source_id=...` lists every attachment stored for a file. ### Step 8: Replace file-level retrieval modes The v1 `files_via_metadata` and `files_via_content` modes map to v2 [file operations](../file_operations): ``` found = await client.beta.retrieval.find(index_id=index.id, file_name_contains="quarterly") file_id = found.items[0].file_id matches = await client.beta.retrieval.grep( index_id=index.id, file_id=file_id, pattern="revenue|profit", context_chars=100, ) text = await client.beta.retrieval.read( index_id=index.id, file_id=file_id, offset=0, max_length=5000, ) ``` ### Step 9: Validate and cut over Run the same evaluation queries against the v1 pipeline and the v2 index before you switch traffic. v2 chunks and embeds differently, so scores and ranking will change. Delete the v1 pipeline once you’re happy with the v2 results. ## Replace data sinks with a custom export Index v2 doesn’t support external data sinks: Pinecone, Qdrant, Milvus, Postgres, MongoDB Atlas, Azure AI Search, and AstraDB can’t be attached to an index. To keep the data in your own database, use a custom export. LlamaCloud still manages the directory, parsing, and sync; you chunk, embed, and write to your database. 1. Create a parse-only index with `vector_target="DISABLED"`: ``` index = await client.beta.indexes.create( source_directory_id=directory.id, vector_target="DISABLED", ) ``` 2. After each sync, download the parsed output from the index’s **output directory**. The source directory holds your original files; the index writes one parsed JSON payload per source file into the directory named by `output_directory_id`: ``` import httpx idx = await client.beta.indexes.get(index.id) async with httpx.AsyncClient() as http: async for f in client.beta.directories.files.list(idx.output_directory_id): if f.file_id is None: continue presigned = await client.files.get(f.file_id) pages = (await http.get(presigned.url)).json()["parse"]["pages"] # f.id -> parsed_directory_file_id, stable across syncs # f.metadata -> the source file's metadata # f.updated_at -> changes when the file is parsed again or its metadata changes ``` 3. Chunk, embed, and write to your store: - Key records on `(f.id, idx.export_config_id)`. Delete a file’s existing records before inserting its new chunks, so a file that’s parsed again replaces its old chunks instead of duplicating them. - Export only what changed: pass your last export time as `updated_at_on_or_after` when listing the output directory. - Delete records whose `parsed_directory_file_id` no longer appears in the output directory. A sync removes the output file when its source file is deleted, but LlamaCloud can’t clean up your database for you. The [`index-v2-data-sinks`](https://github.com/run-llama/index-v2-data-sinks) repo has runnable exporters for MongoDB Atlas, PostgreSQL + pgvector, Qdrant, Pinecone, Turbopuffer, and Azure AI Search. They’re built on a shared `Exporter` protocol (`export`, `delete_file`, `delete_files`, `list_snapshots`); copy the one closest to your stack. With `vector_target="DISABLED"` there is no managed vector store, so `retrieval.*`, `chat`, and the file operations have nothing to query. You serve retrieval from your own database. See [Custom vector stores](../custom_vector_stores) for the full walkthrough. ## Sync an index on a schedule v1 synced data sources on a schedule set by `sync_interval` (6, 12, or 24 hours). Index v2 has no built-in schedule: an index syncs once when you create it, and after that whenever you call `sync`. ### What an index sync does 1. Compares the source directory with what the index last processed. 2. Parses new and changed files, skipping unchanged ones by their content fingerprint and `source_modified_at`. A change to metadata alone updates the metadata without parsing the file again. 3. Chunks, embeds, and stores the changed files, and deletes the chunks of files removed from the directory. Changes to the directory (uploads, deletions, connected-folder updates) only reach the index on the next sync. ### Trigger an index sync safely Only one sync runs per index at a time: a second call while one is running returns `409 Conflict`, and calling `sync` in rapid succession returns `429 Too Many Requests`. Read `last_synced_at` before you trigger the sync, so the previous sync’s `ready` status isn’t mistaken for the new one: ``` import asyncio from llama_cloud import ConflictError, RateLimitError async def sync_and_wait(client, index_id, timeout_s=3600): before = (await client.beta.indexes.get(index_id)).last_synced_at try: await client.beta.indexes.sync(index_id) except ConflictError: pass # a sync is already running; wait for it instead except RateLimitError: await asyncio.sleep(60) await client.beta.indexes.sync(index_id) delay, waited = 2, 0 while waited < timeout_s: idx = await client.beta.indexes.get(index_id) status = idx.metadata.get("status") if status == "failed": raise RuntimeError(idx.metadata.get("error_message")) if status == "ready" and idx.last_synced_at != before: return idx await asyncio.sleep(delay) waited += delay delay = min(delay * 2, 30) raise TimeoutError(f"index {index_id} did not finish syncing") ``` See [Syncing](../syncing) for the sync endpoint in every SDK. ### Replace sync\_interval with a scheduled job Run `sync_and_wait` on a schedule you control, such as cron, a Kubernetes CronJob, Airflow, or a scheduled GitHub Actions workflow. A daily or 12-hourly schedule matches what most v1 pipelines used. - **Connected folders (SharePoint, Google Drive, S3):** LlamaCloud refreshes a connected folder from its source periodically, but that refresh doesn’t sync the index. Keep the scheduled `indexes.sync` call so new and changed files reach the index. - **Sources v2 doesn’t support:** have the same scheduled job pull changed files from the source, upload them to the directory with the same `unique_id` so they replace the old version, delete files that are gone, and then call `sync`. ## Migration checklist - [ ] List your v1 pipelines with their data sources, data sinks, and sync intervals. - [ ] Check each data source against the [ingestion feature parity](#ingestion-feature-parity) table. For anything other than SharePoint, Google Drive, S3, or file upload, plan an upload script. - [ ] For each pipeline that writes to your own vector database, plan a [custom export](#replace-data-sinks-with-a-custom-export). - [ ] If you used page screenshot retrieval, create the v2 index with `store_attachments=["screenshots"]`. It can’t be added later. - [ ] If you relied on composite retrieval, figure-level image retrieval, custom chunking, or a specific embedding model, confirm the v2 behavior works for you before migrating. - [ ] Create directories and add files, carrying over `custom_metadata`. - [ ] Create indexes, with a saved Parse configuration if needed, and wait for `ready`. - [ ] Port retrieval calls using the [parameter mapping](#step-6-update-retrieval-calls). - [ ] Schedule `indexes.sync` calls to replace `sync_interval`. - [ ] Compare retrieval quality between v1 and v2 on your evaluation queries. - [ ] Switch traffic to v2, then delete the v1 pipelines.