--- title: File Storage | Developer Documentation description: Configure blob storage for self-hosted LlamaCloud: required buckets, which buckets to expire for parse retention and which to keep, AWS S3 via IRSA or credentials, bucket-name overrides, and S3Proxy for Azure Blob, MinIO and other S3-compatible providers. --- ## Self-Hosting Documentation Access This section requires a password to access. Interested in self-hosting? [Contact sales](https://www.llamaindex.ai/contact) to learn more. Password: Access Documentation Self-Hosting Documentation Access Granted Logout File storage is an integral part of LlamaCloud. Without it, many key features would not be possible. This page walks through how to configure file storage for your deployment — which buckets you need to create and for non-AWS deployments, how to configure the S3 Proxy to interact with them. ## Requirements - A valid blob storage service. We recommend the following: - [Amazon S3](https://aws.amazon.com/s3/) - [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - [Google Cloud Storage](https://cloud.google.com/storage) - Because LlamaCloud heavily relies on file storage, you will need to create the following buckets: - `llama-platform-parsed-documents` - `llama-platform-etl` - `llama-platform-external-components` - `llama-platform-file-parsing` - `llama-platform-raw-files` - `llama-cloud-parse-output` - `llama-platform-file-screenshots` - `llama-platform-extract-output` (for `LlamaExtract`) Caution Amazon S3 bucket names are **globally unique**, so the default names above are almost certainly already taken and `aws s3 mb` will fail. In practice you must pick your own names (e.g. prefix them with your org or deployment name) and map them with the [Overriding Default Bucket Names](#overriding-default-bucket-names) section below. On Azure Blob and GCS the names only need to be unique within your account, but a prefix is still recommended. ## Retention and Bucket Lifecycle **LlamaCloud never deletes parse or extract output itself.** There is no delete endpoint for parse results, so object expiry is the only deletion path. A deployment with no lifecycle rules keeps every parsed document, extraction, and intermediate artifact indefinitely, which is both a storage-cost problem and a data-retention one. Configure the rules below to bring your deployment in line with the retention the managed service enforces. ### The parse cache: the rule most deployments are missing Parse writes every result into a cache so that re-parsing the same file is free within the cache window (see [Cache Control](/llamaparse/parse/v1/features/cache_options/index.md)). Nothing removes those objects, so the cache grows for the life of the deployment until you expire them. The cache key is derived from the file’s content together with the parse options that affect output, so the same document parsed with different settings is stored as a separate entry. A deployment therefore accumulates roughly one entry per distinct file-and-settings combination, which is what makes the total grow without bound rather than plateau. The cache lives under a **`cache/` prefix** in the bucket you set as `parseFileOutput` (which defaults to the same bucket as `parsedFileParsing`, `llama-platform-file-parsing`): ``` s3:///cache// # Parse v1 entries s3:///cache//v2/ # Parse v2 entries ``` Both API versions cache to the same prefix, so **one rule on `cache/` covers v1 and v2**. Set it to 2 days to match the documented 48-hour cache window: ``` { "Rules": [{ "ID": "parse-cache-expiry", "Status": "Enabled", "Filter": { "Prefix": "cache/" }, "Expiration": { "Days": 2 }, "NoncurrentVersionExpiration": { "NoncurrentDays": 2 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } }] } ``` Caution **On a versioned bucket, an `Expiration` rule on its own reclaims nothing.** S3 does not remove a versioned object when it expires. It writes a delete marker and keeps the previous version as non-current, so the bucket keeps growing while listings appear to shrink. Only `NoncurrentVersionExpiration` frees that space, which is why both rules are in the policy above. Cache contents are reproducible by re-parsing, so consider suspending versioning on this bucket altogether. ### Other buckets that hold transient output | Bucket key | Holds | Expire? | | --------------------------------------- | ----------------------------------------------- | ------------------------------- | | `parseOutput` | Parse job output served through the results API | Yes (2 days) | | `extractOutput` | LlamaExtract output | Yes (2 days) | | `parseFileUpload` / `parsedFileParsing` | Parse job input and working data | Yes, but read the warning below | Caution **Check the bucket’s other prefixes before applying a bucket-wide rule.** `parsedFileParsing` holds more than parse working data. Spreadsheet region data lives under `spreadsheet-regions/`, is removed by the platform when its source is deleted, and is meant to last as long as that source does. A blanket expiration on the bucket deletes it early. Scope your rules by prefix (`cache/`, and your job-data prefix) rather than applying one rule to the whole bucket, or list the bucket first and confirm which prefixes it actually holds. ### Apply to every bucket Regardless of the above, two rules are worth setting everywhere: - **Abort incomplete multipart uploads** after \~7 days. Failed large-file uploads otherwise leave parts that are billed but invisible in listings. - **Expire non-current versions** after \~14 days on any bucket with versioning enabled. Every overwrite and delete otherwise leaves the previous version behind forever; on a busy deployment this dominates bucket size. * [AWS S3 (CLI)](#tab-panel-1127) * [Azure Blob / GCS](#tab-panel-1128) Terminal window ``` # 1. The parse cache. Prefix-scoped, so nothing else in the bucket is touched. aws s3api put-bucket-lifecycle-configuration \ --bucket \ --lifecycle-configuration '{ "Rules": [{ "ID": "parse-cache-expiry", "Status": "Enabled", "Filter": { "Prefix": "cache/" }, "Expiration": { "Days": 2 }, "NoncurrentVersionExpiration": { "NoncurrentDays": 2 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } }] }' # 2. Dedicated transient buckets (parseOutput, extractOutput). Safe bucket-wide. aws s3api put-bucket-lifecycle-configuration \ --bucket \ --lifecycle-configuration '{ "Rules": [{ "ID": "parse-output-expiry", "Status": "Enabled", "Filter": {}, "Expiration": { "Days": 2 }, "NoncurrentVersionExpiration": { "NoncurrentDays": 2 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } }] }' # Confirm what is actually in effect, and what the cache prefix currently holds: aws s3api get-bucket-lifecycle-configuration --bucket aws s3 ls --summarize --human-readable --recursive \ s3:///cache/ | tail -2 # Durable buckets: no object expiration, just version and upload hygiene. aws s3api put-bucket-lifecycle-configuration \ --bucket \ --lifecycle-configuration '{ "Rules": [{ "ID": "raw-files-hygiene", "Status": "Enabled", "Filter": {}, "Transition": { "Days": 0, "StorageClass": "INTELLIGENT_TIERING" }, "NoncurrentVersionExpiration": { "NoncurrentDays": 14 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } }] }' ``` The same split applies; only the mechanism differs. - **Azure Blob**: a [lifecycle management policy](https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview) with `delete: { daysAfterModificationGreaterThan: 2 }`, scoped with `prefixMatch: ["/cache/"]` for the parse cache, plus `deleteAfterDaysSinceCreationGreaterThan` under `version` where blob versioning is on. - **GCS**: an [Object Lifecycle Management](https://cloud.google.com/storage/docs/lifecycle) rule with action `Delete` and condition `age: 2`, scoped with `matchesPrefix: ["cache/"]` for the parse cache, plus `numNewerVersions` for versioned buckets. Apply expiry to the parse cache prefix and the transient buckets only. Leave the durable buckets to version and upload hygiene. Caution Verify which bucket each key points at before applying any rule. If you pointed several keys at a single shared bucket, a 2-day expiration there will also delete your raw uploads and Index documents. Either give the transient keys their own bucket, or scope the rule with a prefix filter. ## Connecting to AWS S3 Below are two ways to configure a connection to AWS S3: ### (Recommended) IAM Role for Service Accounts We recommend that users create a new IAM Role and Policy for LlamaCloud. You can then attach the role ARN as a service account annotation. ``` // Example IAM Policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:*"], // this is not secure "Resource": [ "arn:aws:s3:::llama-platform-parsed-documents", "arn:aws:s3:::llama-platform-parsed-documents/*", ... ] } ] } ``` After creating something similar to the above policy, set `serviceAccountAnnotations` on `backend`, `jobsService`, `temporalWorkloads.llamaParse`, and each configured `temporalWorkloads.workers` entry with the EKS annotation. ``` # Example for the backend service account. Repeat for each of the services listed above. backend: serviceAccountAnnotations: eks.amazonaws.com/role-arn: arn:aws:iam:::role/ ``` The IAM Role must also **trust your cluster’s OIDC provider**, scoped to the LlamaCloud service accounts in your release namespace. Get your cluster’s OIDC issuer with: Terminal window ``` aws eks describe-cluster --name \ --query 'cluster.identity.oidc.issuer' --output text # e.g. https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE0123456789 ``` Then attach a trust policy like the following to the role (replace the account id, region, OIDC id, and set the namespace to your release namespace): ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com" }, "StringLike": { "oidc.eks..amazonaws.com/id/:sub": "system:serviceaccount::*" } } } ] } ``` For more information, feel free to refer to the [official AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) about this topic. ### AWS Credentials Create a user with a policy attached for the aforementioned s3 buckets. Afterwards, you can configure the platform to use the aws credentials of that user by setting the following values in your `values.yaml` file: ``` config: storageBuckets: provider: "aws" s3proxy: enabled: true containerPort: 8080 config: JCLOUDS_PROVIDER: "aws-s3" JCLOUDS_IDENTITY: JCLOUDS_CREDENTIAL: JCLOUDS_REGION: # e.g. "us-east-1" JCLOUDS_ENDPOINT: "https://s3..amazonaws.com" ``` ## Overriding Default Bucket Names We allow users to override the default bucket names in the `values.yaml` file. ``` config: storageBuckets: parsedDocuments: "" parsedEtl: "" parsedExternalComponents: "" parsedFileParsing: "" parsedRawFile: "" parseOutput: "" parsedFileScreenshot: "" extractOutput: "" parseFileUpload: "" parseFileOutput: "" ``` Note There are ten keys but only eight distinct buckets are required — `parseFileUpload` and `parseFileOutput` default to the same bucket as `parsedFileParsing` (`llama-platform-file-parsing`). You can point all three at one bucket. The remaining keys map one-to-one to the buckets listed under [Requirements](#requirements). ## Advanced S3 Configuration For deployments using KMS server-side encryption or custom S3-compatible backends, you may need to configure the S3 signature version. By default, the platform defers to botocore’s automatic region-based resolution. If your buckets use AWS KMS encryption and presigned URLs fail with a signature version error, set `S3_SIGNATURE_VERSION` to `s3v4`: ``` config: storageBuckets: signatureVersion: "s3v4" ``` Note This is typically only needed when using KMS-encrypted S3 buckets in deployments where `AWS_REGION` is not set in the pod environment. IRSA provides credentials but does not always set the region, which can cause botocore to generate presigned URLs with the wrong signature version. Alternatively, you can set `AWS_REGION` via `extraEnvVariables`: ``` config: storageBuckets: extraEnvVariables: AWS_REGION: "us-east-1" ``` ## Connecting to Azure Blob Storage or Other Providers with S3Proxy LlamaCloud was first developed on AWS, which means that we started by natively supporting S3. However, to make a self-hosted solution possible, we need a way for the platform to interact with other providers. We leverage the open-source project [S3Proxy](https://github.com/gaul/s3proxy) to translate the S3 API requests into requests to other storage providers. A containerized deployment of S3Proxy is supported out of the box in our helm charts. S3Proxy is required whenever LlamaCloud talks to a **non-AWS** provider (Azure Blob, GCS, MinIO, etc.), and for the AWS **access-key** path shown above. When it is enabled, S3Proxy is deployed as a sidecar on several of the LlamaCloud pods. If you are on AWS and using the **(Recommended) IRSA** path, you do **not** need S3Proxy — the platform talks to S3 natively using the service-account role, so you can leave `config.storageBuckets.s3proxy.enabled` unset (`false`). The following is an example for how to connect your LlamaCloud deployment to Azure Blob Storage. For more examples of connecting to different providers, please refer to the project’s [Examples](https://github.com/gaul/s3proxy/wiki/Storage-backend-examples) page. - [Azure Blob Storage with S3 Proxy](#tab-panel-1129) - [MinIO / S3-compatible with S3 Proxy](#tab-panel-1130) ``` config: storageBuckets: provider: "azure" s3proxy: enabled: true containerPort: 8080 config: S3PROXY_ENDPOINT: "http://0.0.0.0:80" S3PROXY_AUTHORIZATION: "none" S3PROXY_IGNORE_UNKNOWN_HEADERS: "true" S3PROXY_CORS_ALLOW_ORIGINS: "*" JCLOUDS_PROVIDER: "azureblob" JCLOUDS_REGION: "eastus" # Change to your region JCLOUDS_AZUREBLOB_AUTH: "azureKey" JCLOUDS_IDENTITY: "fill-out" # Change to your storage account name JCLOUDS_CREDENTIAL: "fill-out" # Change to your storage account key JCLOUDS_ENDPOINT: "fill-out" # Change to your storage account endpoint ``` Use this for MinIO, Ceph RADOS Gateway, and other S3-compatible object stores, including a local evaluation deployment where you run MinIO in the same cluster. `config.storageBuckets.provider` accepts only `aws`, `gcp`, or `azure`; there is no S3-compatible value. Leave it as `aws`, which only sets the `CLOUD_PROVIDER` environment variable and does **not** control whether S3Proxy runs. S3Proxy is what actually reaches your store, and it is gated solely on `s3proxy.enabled`. ``` config: storageBuckets: provider: "aws" # sets CLOUD_PROVIDER only; S3Proxy does the talking s3proxy: enabled: true containerPort: 8080 config: JCLOUDS_PROVIDER: "s3" JCLOUDS_IDENTITY: "" JCLOUDS_CREDENTIAL: "" JCLOUDS_REGION: "us-east-1" # In-cluster Service address of your store, reachable from the LlamaCloud pods JCLOUDS_ENDPOINT: "http://minio:9000" ``` Create the eight buckets listed under [Requirements](#requirements) in your store before installing the chart, because S3Proxy does not create them on demand. Note S3Proxy runs as a sidecar and is addressed at `localhost`, so this endpoint is resolved from inside each pod. Presigned URLs handed to a browser are generated against the same endpoint, which means a store that is only reachable inside the cluster will not serve browser-side downloads. For anything beyond local evaluation, expose the store on an address that both the pods and your users’ browsers can reach.