Skip to content
Developer Documentation
Guide
Connectors
Set Up a Connector with the API

Set Up a Connector with the API

Connect an S3 bucket to a LlamaCloud project entirely over the REST API, sync it into a folder, and build an index or run a batch over that folder, with no web UI steps.

Everything the Files → Connections page does is also available over the REST API. This page walks through the whole path for an S3 bucket, from access key to a running index or batch:

  1. Create a connection with the bucket’s access key.
  2. Subscribe the connection to a bucket and prefix.
  3. Create a folder (a directory, in API terms) and route the subscription into it.
  4. Start the first sync.
  5. Build an index over the folder, or run a batch over it.

Every request authenticates with your API key as a bearer token. The examples read it from LLAMA_CLOUD_API_KEY. For the EU region, use https://api.cloud.eu.llamaindex.ai as the base URL instead.

Create an S3 connection with the API

Section titled “Create an S3 connection with the API”

POST /api/v2/connectors stores the access key pair and checks it with AWS before saving. The IAM user needs s3:ListBucket and s3:GetObject on the bucket, as described in Create an IAM user for the S3 connector.

import os


import httpx


client = httpx.Client(
    base_url="https://api.cloud.llamaindex.ai",
    headers={"Authorization": f"Bearer {os.environ['LLAMA_CLOUD_API_KEY']}"},
    timeout=60,
)


connector = client.post(
    "/api/v2/connectors",
    json={
        "name": "Invoices bucket",
        "integration": "s3",
        "config": {
            "source_type": "s3",
            "auth_credentials": {
                "auth_type": "service_account",
                "aws_access_id": os.environ["AWS_ACCESS_KEY_ID"],
                "aws_access_secret": os.environ["AWS_SECRET_ACCESS_KEY"],
                # For an S3-compatible store, add its base URL:
                # "s3_endpoint_url": "https://s3-compatible.example.com",
            },
        },
    },
)
connector.raise_for_status()
connector_id = connector.json()["id"]
print("Connector:", connector_id)

The snippets below build on this one — paste them into one file in order.

The response never includes the secret. What can come back instead of 201:

StatusMeaning
409This project already has a connection for the same AWS identity. Reuse it and add another subscription instead.
422AWS rejected the key pair, or a required field is missing. The detail says which.
503AWS couldn’t be reached. Retry the same request.

Subscribe an S3 connection to a bucket

Section titled “Subscribe an S3 connection to a bucket”

A subscription names what to sync: one bucket, optionally narrowed to a prefix. LlamaCloud lists the bucket with the connection’s key before creating it, so a misspelled bucket or a missing s3:ListBucket permission fails here rather than on the first sync.

subscription = client.post(
    f"/api/v2/connectors/{connector_id}/subscriptions",
    json={
        "scope_config": {
            "source_type": "s3",
            "bucket": "acme-documents",
            "prefix": "invoices/2024/",  # omit to sync the whole bucket
        },
    },
)
subscription.raise_for_status()
subscription_id = subscription.json()["id"]
print("Subscription:", subscription_id)

A 404 means the bucket doesn’t exist, a 403 means the key can’t list it, and a 409 means this connection already has a subscription for that bucket and prefix.

Route an S3 subscription into a folder

Section titled “Route an S3 subscription into a folder”

Files land in a directory, and two calls wire the subscription to one:

  • POST /api/v1/beta/directories with connector_subscription_id creates the directory and marks it as synced from that subscription.
  • POST /api/v1/beta/directories/ingest-configs routes the subscription’s files into the directory. Without it the directory exists but stays empty.
directory = client.post(
    "/api/v1/beta/directories",
    json={
        "name": "invoices-2024",
        "connector_subscription_id": subscription_id,
    },
)
directory.raise_for_status()
directory_id = directory.json()["id"]


ingest_config = client.post(
    "/api/v1/beta/directories/ingest-configs",
    json={
        "target_directory_id": directory_id,
        "source": {"type": "connector_subscription", "id": subscription_id},
    },
)
ingest_config.raise_for_status()
print("Directory:", directory_id)

By default the ingest config mirrors deletions: an object deleted from the bucket is removed from the directory on the next sync. To keep files after they’re deleted at the source, send "deletion_mode": "append_only".

A subscription can feed only one directory. If either call fails, delete what the earlier steps created before retrying, or the subscription is left syncing into nothing and blocks a new one for the same bucket and prefix.

Start and monitor an S3 sync

Section titled “Start and monitor an S3 sync”

A new subscription syncs on its own about once a day. To start the first sync now, request one. The call returns as soon as the sync is queued.

import time


client.post(
    f"/api/v2/connectors/{connector_id}/subscriptions/{subscription_id}/sync"
).raise_for_status()


# Wait for the first sync to finish before indexing or batching the directory.
while True:
    subscriptions = client.get(
        f"/api/v2/connectors/{connector_id}/subscriptions"
    ).json()["items"]
    current = next(s for s in subscriptions if s["id"] == subscription_id)
    if current["status"] in ("invalid_auth", "invalid_scope") or current["last_error"]:
        raise RuntimeError(f"Sync failed: {current['status']} {current['last_error']}")
    if current["last_sync_at"] and current["status"] != "running":
        break
    time.sleep(10)


print("Synced at", current["last_sync_at"])

A subscription’s status is one of:

StatusMeaning
idleNot syncing right now. Check last_sync_at and last_error for how the last run went.
runningA sync is in progress.
invalid_authThe access key stopped working. Syncing is paused until the connection is replaced.
invalid_scopeThe bucket or prefix is no longer readable with the key. Syncing is paused.

To see the synced files, list the directory with GET /api/v1/beta/directories/{directory_id}/files.

Index an S3-synced folder

Section titled “Index an S3-synced folder”

An index built from the directory parses and embeds whatever the connector has synced into it. Set sync_frequency to daily so the index picks up the connector’s changes without you calling sync; the default, manual, only re-syncs when you call POST /api/v1/indexes/{index_id}/sync.

index = client.post(
    "/api/v1/indexes",
    json={
        "name": "invoices-2024",
        "source_directory_id": directory_id,
        "sync_frequency": "daily",
    },
)
index.raise_for_status()
print("Index:", index.json()["id"])

Retrieval, status polling, and vector store options work the same as for any other index. See Index getting started and Syncing.

Run a batch over an S3-synced folder

Section titled “Run a batch over an S3-synced folder”

A batch runs one job type, such as a Parse or Extract job, over the files in the directory. Wait for the first sync to finish before creating it, or the batch can start on a directory that’s still empty.

batch = client.post(
    "/api/v2/batches",
    json={
        "source_directory_id": directory_id,
        "config": {
            "job": {
                "type": "parse_v2",
                "configuration_id": "cfg-PARSE_AGENTIC",
            },
        },
    },
)
batch.raise_for_status()
print("Batch:", batch.json()["id"], batch.json()["status"])

Monitoring a batch and reading its results is covered in Batches.

Stop syncing or disconnect with the API

Section titled “Stop syncing or disconnect with the API”
RequestEffect on synced files
DELETE /api/v2/connectors/{connector_id}/subscriptions/{subscription_id}Kept. The directory stops getting changes.
DELETE /api/v2/connectors/{connector_id}Kept. Every subscription on the connection is deleted and its directories stop syncing.
DELETE /api/v1/beta/directories/{directory_id}Deleted, along with the directory and its subscription.

Moving from an S3 data source to an S3 connector

Section titled “Moving from an S3 data source to an S3 connector”

If you set up S3 with the older data source API, the fields carry over:

Data source (CloudS3DataSource)Connector
aws_access_id, aws_access_secret, s3_endpoint_urlconfig.auth_credentials on POST /api/v2/connectors
bucket, prefixscope_config on POST /api/v2/connectors/{connector_id}/subscriptions
Attaching the data source to a pipelineAn index or batch with source_directory_id set to the synced directory

The connection holds the credentials once, so one access key can feed subscriptions for any number of buckets, and each synced directory can back an index, a batch, or both.

Note for AI agents: this documentation is built for programmatic access. - Overview of all docs: https://developers.llamaindex.ai/llms.txt - Any page is available as raw Markdown by appending index.md to its URL — e.g. https://developers.llamaindex.ai/llamaparse/parse/getting_started/index.md - Agent-friendly REST search APIs live under https://developers.llamaindex.ai/api/ — search (BM25 full-text), grep (regex), read (fetch a page), and list (browse the doc tree). See https://developers.llamaindex.ai/llms.txt for parameters. - A hosted documentation MCP server is available at https://developers.llamaindex.ai/mcp. If you support MCP, you can ask the user to install it for browsing these docs directly (an alternative to the REST API). Setup: https://developers.llamaindex.ai/for-agents/mcp/ - Other LlamaIndex tooling for agents — the LlamaParse Platform MCP server, agent skills and plugins, and the n8n node — is mapped at https://developers.llamaindex.ai/for-agents/