File Storage

Self-Hosting Documentation Access

This section requires a password to access. Interested in self-hosting? Contact sales to learn more.

Password:

Access Documentation

Self-Hosting Documentation Access Granted Logout

File storage is an integral part of LlamaCloud. Without it, many key features would not be possible. This page walks through how to configure file storage for your deployment — which buckets you need to create and for non-AWS deployments, how to configure the S3 Proxy to interact with them.

Requirements

• A valid blob storage service. We recommend the following:
  • Amazon S3
  • Azure Blob Storage
  • Google Cloud Storage
• Because LlamaCloud heavily relies on file storage, you will need to create the following buckets:
  • llama-platform-parsed-documents
  • llama-platform-etl
  • llama-platform-external-components
  • llama-platform-file-parsing
  • llama-platform-raw-files
  • llama-cloud-parse-output
  • llama-platform-file-screenshots
  • llama-platform-extract-output (for LlamaExtract)

Caution

Amazon S3 bucket names are globally unique, so the default names above are almost certainly already taken and aws s3 mb will fail. In practice you must pick your own names (e.g. prefix them with your org or deployment name) and map them with the Overriding Default Bucket Names section below. On Azure Blob and GCS the names only need to be unique within your account, but a prefix is still recommended.

Retention and Bucket Lifecycle

LlamaCloud never deletes parse or extract output itself. There is no delete endpoint for parse results, so object expiry is the only deletion path. A deployment with no lifecycle rules keeps every parsed document, extraction, and intermediate artifact indefinitely, which is both a storage-cost problem and a data-retention one. Configure the rules below to bring your deployment in line with the retention the managed service enforces.

The parse cache: the rule most deployments are missing

Parse writes every result into a cache so that re-parsing the same file is free within the cache window (see Cache Control). Nothing removes those objects, so the cache grows for the life of the deployment until you expire them.

The cache key is derived from the file’s content together with the parse options that affect output, so the same document parsed with different settings is stored as a separate entry. A deployment therefore accumulates roughly one entry per distinct file-and-settings combination, which is what makes the total grow without bound rather than plateau.

The cache lives under a cache/ prefix in the bucket you set as parseFileOutput (which defaults to the same bucket as parsedFileParsing, llama-platform-file-parsing):

s3://<parseFileOutput-bucket>/cache/<cache-key>/         # Parse v1 entries
s3://<parseFileOutput-bucket>/cache/<cache-key>/v2/      # Parse v2 entries

Both API versions cache to the same prefix, so one rule on cache/ covers v1 and v2. Set it to 2 days to match the documented 48-hour cache window:

{
  "Rules": [{
    "ID": "parse-cache-expiry",
    "Status": "Enabled",
    "Filter": { "Prefix": "cache/" },
    "Expiration": { "Days": 2 },
    "NoncurrentVersionExpiration": { "NoncurrentDays": 2 },
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
  }]
}

Caution

On a versioned bucket, an Expiration rule on its own reclaims nothing. S3 does not remove a versioned object when it expires. It writes a delete marker and keeps the previous version as non-current, so the bucket keeps growing while listings appear to shrink. Only NoncurrentVersionExpiration frees that space, which is why both rules are in the policy above. Cache contents are reproducible by re-parsing, so consider suspending versioning on this bucket altogether.

Other buckets that hold transient output

Bucket key	Holds	Expire?
parseOutput	Parse job output served through the results API	Yes (2 days)
extractOutput	LlamaExtract output	Yes (2 days)
parseFileUpload / parsedFileParsing	Parse job input and working data	Yes, but read the warning below

Caution

Check the bucket’s other prefixes before applying a bucket-wide rule. parsedFileParsing holds more than parse working data. LlamaCloud Sheets stores region data under spreadsheet-regions/, which is deleted by the application when a sheet is deleted and is meant to live as long as the sheet does. A blanket expiration on that bucket removes it. If you use Sheets, scope your rules by prefix (cache/, and your job-data prefix) instead of applying one rule to the whole bucket.

Apply to every bucket

Regardless of the above, two rules are worth setting everywhere:

• Abort incomplete multipart uploads after ~7 days. Failed large-file uploads otherwise leave parts that are billed but invisible in listings.
• Expire non-current versions after ~14 days on any bucket with versioning enabled. Every overwrite and delete otherwise leaves the previous version behind forever; on a busy deployment this dominates bucket size.

AWS S3 (CLI)

# 1. The parse cache. Prefix-scoped, so nothing else in the bucket is touched.
aws s3api put-bucket-lifecycle-configuration \
  --bucket <YOUR-PARSE-FILE-OUTPUT-BUCKET> \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "parse-cache-expiry",
      "Status": "Enabled",
      "Filter": { "Prefix": "cache/" },
      "Expiration": { "Days": 2 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 2 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }]
  }'

# 2. Dedicated transient buckets (parseOutput, extractOutput). Safe bucket-wide.
aws s3api put-bucket-lifecycle-configuration \
  --bucket <YOUR-PARSE-OUTPUT-BUCKET> \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "parse-output-expiry",
      "Status": "Enabled",
      "Filter": {},
      "Expiration": { "Days": 2 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 2 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }]
  }'

# Confirm what is actually in effect, and what the cache prefix currently holds:
aws s3api get-bucket-lifecycle-configuration --bucket <YOUR-PARSE-FILE-OUTPUT-BUCKET>
aws s3 ls --summarize --human-readable --recursive \
  s3://<YOUR-PARSE-FILE-OUTPUT-BUCKET>/cache/ | tail -2

# Durable buckets: no object expiration, just version and upload hygiene.
aws s3api put-bucket-lifecycle-configuration \
  --bucket <YOUR-RAW-FILES-BUCKET> \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "raw-files-hygiene",
      "Status": "Enabled",
      "Filter": {},
      "Transition": { "Days": 0, "StorageClass": "INTELLIGENT_TIERING" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 14 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }]
  }'

Azure Blob / GCS

The same split applies; only the mechanism differs.

• Azure Blob: a lifecycle management policy with delete: { daysAfterModificationGreaterThan: 2 }, scoped with prefixMatch: ["<container>/cache/"] for the parse cache, plus deleteAfterDaysSinceCreationGreaterThan under version where blob versioning is on.
• GCS: an Object Lifecycle Management rule with action Delete and condition age: 2, scoped with matchesPrefix: ["cache/"] for the parse cache, plus numNewerVersions for versioned buckets.

Apply expiry to the parse cache prefix and the transient buckets only. Leave the durable buckets to version and upload hygiene.

Caution

Verify which bucket each key points at before applying any rule. If you pointed several keys at a single shared bucket, a 2-day expiration there will also delete your raw uploads and Index documents. Either give the transient keys their own bucket, or scope the rule with a prefix filter.

Connecting to AWS S3

Below are two ways to configure a connection to AWS S3:

(Recommended) IAM Role for Service Accounts

We recommend that users create a new IAM Role and Policy for LlamaCloud. You can then attach the role ARN as a service account annotation.

// Example IAM Policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:*"], // this is not secure
      "Resource": [
        "arn:aws:s3:::llama-platform-parsed-documents",
        "arn:aws:s3:::llama-platform-parsed-documents/*",
        ...
      ]
    }
  ]
}

After creating something similar to the above policy, update the backend, jobsService, jobsWorker, and llamaParse service accounts with the EKS annotation.

# Example for the backend service account. Repeat for each of the services listed above.
backend:
  serviceAccountAnnotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<role-name>

The IAM Role must also trust your cluster’s OIDC provider, scoped to the LlamaCloud service accounts in your release namespace. Get your cluster’s OIDC issuer with:

aws eks describe-cluster --name <cluster> \
  --query 'cluster.identity.oidc.issuer' --output text
# e.g. https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE0123456789

Then attach a trust policy like the following to the role (replace the account id, region, OIDC id, and set the namespace to your release namespace):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<OIDC-ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<region>.amazonaws.com/id/<OIDC-ID>:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "oidc.eks.<region>.amazonaws.com/id/<OIDC-ID>:sub": "system:serviceaccount:<namespace>:*"
        }
      }
    }
  ]
}

For more information, feel free to refer to the official AWS documentation about this topic.

AWS Credentials

Create a user with a policy attached for the aforementioned s3 buckets. Afterwards, you can configure the platform to use the aws credentials of that user by setting the following values in your values.yaml file:

config:
  storageBuckets:
    provider: "aws"
    s3proxy:
      enabled: true
      containerPort: 8080
      config:
        JCLOUDS_PROVIDER: "aws-s3"
        JCLOUDS_IDENTITY: <AWS-ACCESS-KEY>
        JCLOUDS_CREDENTIAL: <AWS-SECRET-KEY>
        JCLOUDS_REGION: <AWS-REGION>  # e.g. "us-east-1"
        JCLOUDS_ENDPOINT: "https://s3.<AWS-REGION>.amazonaws.com"

Overriding Default Bucket Names

We allow users to override the default bucket names in the values.yaml file.

  config:
    storageBuckets:
      parsedDocuments: "<your-bucket-name>"
      parsedEtl: "<your-bucket-name>"
      parsedExternalComponents: "<your-bucket-name>"
      parsedFileParsing: "<your-bucket-name>"
      parsedRawFile: "<your-bucket-name>"
      parseOutput: "<your-bucket-name>"
      parsedFileScreenshot: "<your-bucket-name>"
      extractOutput: "<your-bucket-name>"
      parseFileUpload: "<your-bucket-name>"
      parseFileOutput: "<your-bucket-name>"

Note

There are ten keys but only eight distinct buckets are required — parseFileUpload and parseFileOutput default to the same bucket as parsedFileParsing (llama-platform-file-parsing). You can point all three at one bucket. The remaining keys map one-to-one to the buckets listed under Requirements.

Advanced S3 Configuration

For deployments using KMS server-side encryption or custom S3-compatible backends, you may need to configure the S3 signature version. By default, the platform defers to botocore’s automatic region-based resolution. If your buckets use AWS KMS encryption and presigned URLs fail with a signature version error, set S3_SIGNATURE_VERSION to s3v4:

config:
  storageBuckets:
    signatureVersion: "s3v4"

Note

This is typically only needed when using KMS-encrypted S3 buckets in deployments where AWS_REGION is not set in the pod environment. IRSA provides credentials but does not always set the region, which can cause botocore to generate presigned URLs with the wrong signature version.

Alternatively, you can set AWS_REGION via extraEnvVariables:

config:
  storageBuckets:
    extraEnvVariables:
      AWS_REGION: "us-east-1"

Connecting to Azure Blob Storage or Other Providers with S3Proxy

LlamaCloud was first developed on AWS, which means that we started by natively supporting S3. However, to make a self-hosted solution possible, we need a way for the platform to interact with other providers.

We leverage the open-source project S3Proxy to translate the S3 API requests into requests to other storage providers. A containerized deployment of S3Proxy is supported out of the box in our helm charts.

S3Proxy is required whenever LlamaCloud talks to a non-AWS provider (Azure Blob, GCS, MinIO, etc.), and for the AWS access-key path shown above. When it is enabled, S3Proxy is deployed as a sidecar on several of the LlamaCloud pods.

If you are on AWS and using the (Recommended) IRSA path, you do not need S3Proxy — the platform talks to S3 natively using the service-account role, so you can leave config.storageBuckets.s3proxy.enabled unset (false).

The following is an example for how to connect your LlamaCloud deployment to Azure Blob Storage. For more examples of connecting to different providers, please refer to the project’s Examples page.

Azure Blob Storage with S3 Proxy

config:
  storageBuckets:
    provider: "azure"
    s3proxy:
      enabled: true
      containerPort: 8080
      config:
        S3PROXY_ENDPOINT: "http://0.0.0.0:80"
        S3PROXY_AUTHORIZATION: "none"
        S3PROXY_IGNORE_UNKNOWN_HEADERS: "true"
        S3PROXY_CORS_ALLOW_ORIGINS: "*"
        JCLOUDS_PROVIDER: "azureblob"
        JCLOUDS_REGION: "eastus" # Change to your region
        JCLOUDS_AZUREBLOB_AUTH: "azureKey"
        JCLOUDS_IDENTITY: "fill-out" # Change to your storage account name
        JCLOUDS_CREDENTIAL: "fill-out" # Change to your storage account key
        JCLOUDS_ENDPOINT: "fill-out" # Change to your storage account endpoint

MinIO / S3-compatible with S3 Proxy

Use this for MinIO, Ceph RADOS Gateway, and other S3-compatible object stores, including a local evaluation deployment where you run MinIO in the same cluster.

config.storageBuckets.provider accepts only aws, gcp, or azure; there is no S3-compatible value. Leave it as aws, which only sets the CLOUD_PROVIDER environment variable and does not control whether S3Proxy runs. S3Proxy is what actually reaches your store, and it is gated solely on s3proxy.enabled.

config:
  storageBuckets:
    provider: "aws" # sets CLOUD_PROVIDER only; S3Proxy does the talking
    s3proxy:
      enabled: true
      containerPort: 8080
      config:
        JCLOUDS_PROVIDER: "s3"
        JCLOUDS_IDENTITY: "<ACCESS-KEY>"
        JCLOUDS_CREDENTIAL: "<SECRET-KEY>"
        JCLOUDS_REGION: "us-east-1"
        # In-cluster Service address of your store, reachable from the LlamaCloud pods
        JCLOUDS_ENDPOINT: "http://minio:9000"

Create the eight buckets listed under Requirements in your store before installing the chart, because S3Proxy does not create them on demand.

Note

S3Proxy runs as a sidecar and is addressed at localhost, so this endpoint is resolved from inside each pod. Presigned URLs handed to a browser are generated against the same endpoint, which means a store that is only reachable inside the cluster will not serve browser-side downloads. For anything beyond local evaluation, expose the store on an address that both the pods and your users’ browsers can reach.