--- title: Set Up a Connector with the API | Developer Documentation description: Connect an S3 bucket to a LlamaCloud project entirely over the REST API, sync it into a folder, and build an index or run a batch over that folder, with no web UI steps. --- Everything the **Files → Connections** page does is also available over the REST API. This page walks through the whole path for an S3 bucket, from access key to a running index or batch: 1. Create a connection with the bucket’s access key. 2. Subscribe the connection to a bucket and prefix. 3. Create a folder (a directory, in API terms) and route the subscription into it. 4. Start the first sync. 5. Build an index over the folder, or run a batch over it. Every request authenticates with your API key as a bearer token. The examples read it from `LLAMA_CLOUD_API_KEY`. For the EU region, use `https://api.cloud.eu.llamaindex.ai` as the base URL instead. Note The Python SDK doesn’t wrap the connection and subscription endpoints yet, so the Python examples call the REST API directly with `httpx`. Indexes and batches have SDK methods too, covered in [Index getting started](/llamaparse/cloud-index-v2/getting_started/index.md) and [Batches](/llamaparse/batches/getting_started/index.md). ## Create an S3 connection with the API `POST /api/v2/connectors` stores the access key pair and checks it with AWS before saving. The IAM user needs `s3:ListBucket` and `s3:GetObject` on the bucket, as described in [Create an IAM user for the S3 connector](/llamaparse/connectors/s3/#create-an-iam-user-for-the-s3-connector/index.md). - [Python](#tab-panel-230) - [cURL](#tab-panel-231) ``` import os import httpx client = httpx.Client( base_url="https://api.cloud.llamaindex.ai", headers={"Authorization": f"Bearer {os.environ['LLAMA_CLOUD_API_KEY']}"}, timeout=60, ) connector = client.post( "/api/v2/connectors", json={ "name": "Invoices bucket", "integration": "s3", "config": { "source_type": "s3", "auth_credentials": { "auth_type": "service_account", "aws_access_id": os.environ["AWS_ACCESS_KEY_ID"], "aws_access_secret": os.environ["AWS_SECRET_ACCESS_KEY"], # For an S3-compatible store, add its base URL: # "s3_endpoint_url": "https://s3-compatible.example.com", }, }, }, ) connector.raise_for_status() connector_id = connector.json()["id"] print("Connector:", connector_id) ``` The snippets below build on this one — paste them into one file in order. Terminal window ``` curl -sS -X POST "https://api.cloud.llamaindex.ai/api/v2/connectors" \ -H "Authorization: Bearer $LLAMA_CLOUD_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Invoices bucket", "integration": "s3", "config": { "source_type": "s3", "auth_credentials": { "auth_type": "service_account", "aws_access_id": "'"$AWS_ACCESS_KEY_ID"'", "aws_access_secret": "'"$AWS_SECRET_ACCESS_KEY"'" } } }' ``` Save the `id` from the response as `CONNECTOR_ID`. The response never includes the secret. What can come back instead of `201`: | Status | Meaning | | ------ | --------------------------------------------------------------------------------------------------------------- | | `409` | This project already has a connection for the same AWS identity. Reuse it and add another subscription instead. | | `422` | AWS rejected the key pair, or a required field is missing. The `detail` says which. | | `503` | AWS couldn’t be reached. Retry the same request. | ## Subscribe an S3 connection to a bucket A subscription names what to sync: one bucket, optionally narrowed to a prefix. LlamaCloud lists the bucket with the connection’s key before creating it, so a misspelled bucket or a missing `s3:ListBucket` permission fails here rather than on the first sync. - [Python](#tab-panel-232) - [cURL](#tab-panel-233) ``` subscription = client.post( f"/api/v2/connectors/{connector_id}/subscriptions", json={ "scope_config": { "source_type": "s3", "bucket": "acme-documents", "prefix": "invoices/2024/", # omit to sync the whole bucket }, }, ) subscription.raise_for_status() subscription_id = subscription.json()["id"] print("Subscription:", subscription_id) ``` Terminal window ``` curl -sS -X POST "https://api.cloud.llamaindex.ai/api/v2/connectors/$CONNECTOR_ID/subscriptions" \ -H "Authorization: Bearer $LLAMA_CLOUD_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "scope_config": { "source_type": "s3", "bucket": "acme-documents", "prefix": "invoices/2024/" } }' ``` Save the `id` from the response as `SUBSCRIPTION_ID`. A `404` means the bucket doesn’t exist, a `403` means the key can’t list it, and a `409` means this connection already has a subscription for that bucket and prefix. ## Route an S3 subscription into a folder Files land in a directory, and two calls wire the subscription to one: - `POST /api/v1/beta/directories` with `connector_subscription_id` creates the directory and marks it as synced from that subscription. - `POST /api/v1/beta/directories/ingest-configs` routes the subscription’s files into the directory. Without it the directory exists but stays empty. * [Python](#tab-panel-234) * [cURL](#tab-panel-235) ``` directory = client.post( "/api/v1/beta/directories", json={ "name": "invoices-2024", "connector_subscription_id": subscription_id, }, ) directory.raise_for_status() directory_id = directory.json()["id"] ingest_config = client.post( "/api/v1/beta/directories/ingest-configs", json={ "target_directory_id": directory_id, "source": {"type": "connector_subscription", "id": subscription_id}, }, ) ingest_config.raise_for_status() print("Directory:", directory_id) ``` Terminal window ``` curl -sS -X POST "https://api.cloud.llamaindex.ai/api/v1/beta/directories" \ -H "Authorization: Bearer $LLAMA_CLOUD_API_KEY" \ -H "Content-Type: application/json" \ -d '{"name": "invoices-2024", "connector_subscription_id": "'"$SUBSCRIPTION_ID"'"}' # Save the directory's id as DIRECTORY_ID, then: curl -sS -X POST "https://api.cloud.llamaindex.ai/api/v1/beta/directories/ingest-configs" \ -H "Authorization: Bearer $LLAMA_CLOUD_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "target_directory_id": "'"$DIRECTORY_ID"'", "source": {"type": "connector_subscription", "id": "'"$SUBSCRIPTION_ID"'"} }' ``` By default the ingest config mirrors deletions: an object deleted from the bucket is removed from the directory on the next sync. To keep files after they’re deleted at the source, send `"deletion_mode": "append_only"`. A subscription can feed only one directory. If either call fails, delete what the earlier steps created before retrying, or the subscription is left syncing into nothing and blocks a new one for the same bucket and prefix. ## Start and monitor an S3 sync A new subscription syncs on its own about once a day. To start the first sync now, request one. The call returns as soon as the sync is queued. - [Python](#tab-panel-236) - [cURL](#tab-panel-237) ``` import time client.post( f"/api/v2/connectors/{connector_id}/subscriptions/{subscription_id}/sync" ).raise_for_status() # Wait for the first sync to finish before indexing or batching the directory. while True: subscriptions = client.get( f"/api/v2/connectors/{connector_id}/subscriptions" ).json()["items"] current = next(s for s in subscriptions if s["id"] == subscription_id) if current["status"] in ("invalid_auth", "invalid_scope") or current["last_error"]: raise RuntimeError(f"Sync failed: {current['status']} {current['last_error']}") if current["last_sync_at"] and current["status"] != "running": break time.sleep(10) print("Synced at", current["last_sync_at"]) ``` Terminal window ``` curl -sS -X POST \ "https://api.cloud.llamaindex.ai/api/v2/connectors/$CONNECTOR_ID/subscriptions/$SUBSCRIPTION_ID/sync" \ -H "Authorization: Bearer $LLAMA_CLOUD_API_KEY" # Check progress: status, last_sync_at, last_error curl -sS "https://api.cloud.llamaindex.ai/api/v2/connectors/$CONNECTOR_ID/subscriptions" \ -H "Authorization: Bearer $LLAMA_CLOUD_API_KEY" \ | jq --arg id "$SUBSCRIPTION_ID" '.items[] | select(.id == $id) | {status, last_sync_at, last_error}' ``` A subscription’s `status` is one of: | Status | Meaning | | --------------- | --------------------------------------------------------------------------------------- | | `idle` | Not syncing right now. Check `last_sync_at` and `last_error` for how the last run went. | | `running` | A sync is in progress. | | `invalid_auth` | The access key stopped working. Syncing is paused until the connection is replaced. | | `invalid_scope` | The bucket or prefix is no longer readable with the key. Syncing is paused. | To see the synced files, list the directory with `GET /api/v1/beta/directories/{directory_id}/files`. ## Index an S3-synced folder An index built from the directory parses and embeds whatever the connector has synced into it. Set `sync_frequency` to `daily` so the index picks up the connector’s changes without you calling sync; the default, `manual`, only re-syncs when you call `POST /api/v1/indexes/{index_id}/sync`. - [Python](#tab-panel-238) - [cURL](#tab-panel-239) ``` index = client.post( "/api/v1/indexes", json={ "name": "invoices-2024", "source_directory_id": directory_id, "sync_frequency": "daily", }, ) index.raise_for_status() print("Index:", index.json()["id"]) ``` Terminal window ``` curl -sS -X POST "https://api.cloud.llamaindex.ai/api/v1/indexes" \ -H "Authorization: Bearer $LLAMA_CLOUD_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "invoices-2024", "source_directory_id": "'"$DIRECTORY_ID"'", "sync_frequency": "daily" }' ``` Retrieval, status polling, and vector store options work the same as for any other index. See [Index getting started](/llamaparse/cloud-index-v2/getting_started/index.md) and [Syncing](/llamaparse/cloud-index-v2/syncing/index.md). ## Run a batch over an S3-synced folder A batch runs one job type, such as a Parse or Extract job, over the files in the directory. Wait for the first sync to finish before creating it, or the batch can start on a directory that’s still empty. - [Python](#tab-panel-240) - [cURL](#tab-panel-241) ``` batch = client.post( "/api/v2/batches", json={ "source_directory_id": directory_id, "config": { "job": { "type": "parse_v2", "configuration_id": "cfg-PARSE_AGENTIC", }, }, }, ) batch.raise_for_status() print("Batch:", batch.json()["id"], batch.json()["status"]) ``` Terminal window ``` curl -sS -X POST "https://api.cloud.llamaindex.ai/api/v2/batches" \ -H "Authorization: Bearer $LLAMA_CLOUD_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "source_directory_id": "'"$DIRECTORY_ID"'", "config": {"job": {"type": "parse_v2", "configuration_id": "cfg-PARSE_AGENTIC"}} }' ``` Monitoring a batch and reading its results is covered in [Batches](/llamaparse/batches/getting_started/index.md). ## Stop syncing or disconnect with the API | Request | Effect on synced files | | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `DELETE /api/v2/connectors/{connector_id}/subscriptions/{subscription_id}` | Kept. The directory stops getting changes. | | `DELETE /api/v2/connectors/{connector_id}` | Kept. Every subscription on the connection is deleted and its directories stop syncing. | | `DELETE /api/v1/beta/directories/{directory_id}` | Deleted, along with the directory and its subscription. | ## Moving from an S3 data source to an S3 connector If you set up S3 with the older [data source API](/llamaparse/deprecated/cloud-index/integrations/data_sources/s3/index.md), the fields carry over: | Data source (`CloudS3DataSource`) | Connector | | ------------------------------------------------------- | ------------------------------------------------------------------------ | | `aws_access_id`, `aws_access_secret`, `s3_endpoint_url` | `config.auth_credentials` on `POST /api/v2/connectors` | | `bucket`, `prefix` | `scope_config` on `POST /api/v2/connectors/{connector_id}/subscriptions` | | Attaching the data source to a pipeline | An index or batch with `source_directory_id` set to the synced directory | The connection holds the credentials once, so one access key can feed subscriptions for any number of buckets, and each synced directory can back an index, a batch, or both.