Migrating off RabbitMQ

Self-Hosting Documentation Access

This section requires a password to access. Interested in self-hosting? Contact sales to learn more.

Password:

Access Documentation

Self-Hosting Documentation Access Granted Logout

AMQP transport has been removed from LlamaCloud. All jobs run through Temporal. Remove rabbitmq, jobsWorker, and llamaParse from your values before upgrading; the chart schema rejects these retired keys. The chart no longer mounts RabbitMQ credentials. AMQP_DISABLED is no longer used.

For most deployments the move is smaller than it looks. Parse, Extract, and Index all run on Temporal already, and on current releases they do so by default rather than because you configured it. This includes the legacy Pipelines surface (Index v1).

Note

On older releases, AMQP was the default transport for the v1 Parse API and Temporal was opt-in. Current releases always dispatch v1 Parse jobs through Temporal. The v1 Parse API remains available, but AMQP is no longer selectable for new Parse jobs.

Extract v1 always dispatches through Temporal. The v1 API and job-record storage settings are unchanged. Before upgrading from AMQP, verify the capacity and autoscaling of temporal-jobs-worker, which runs Extract with the chart’s default consolidated profile. Scalers targeting the old worker or reading RabbitMQ queue depth do not scale this deployment.

Job transport

Temporal is enabled in every supported configuration, and is the default transport:

Feature	Transport	Needs RabbitMQ
Parse (v2)	Temporal	No
Parse (v1)	Temporal	No
Extract	Temporal	No
Index directory sync, export, retrieval, chat	Temporal	No
Legacy Pipelines (Index v1)	Temporal	No

“Legacy Pipelines” means the older pipeline-centric ingestion API: creating and upserting pipelines, pipeline sync, pipeline data-source and file updates, metadata updates, and managed document ingestion. The recommended replacement is Index v2, which syncs a source directory to a vector store over Temporal. Migrating to Index v2 is not required to remove RabbitMQ: existing Index v1 pipelines also run through Temporal without changing the v1 API or migrating their data.

Note

Parse v1 is dispatched as a Temporal workflow, not an AMQP publish. Staying on the v1 Parse API does not hold up this migration. It does still write job records to MongoDB, so keep MongoDB (see below).

Before you start

• MongoDB stays. This migration removes RabbitMQ only. Index v1 still requires MongoDB for its job records. Changing transport does not migrate those records. Dropping both stores is the separate minimal profile, which disables the legacy pipeline surface.
• Upgrade through version 0.10.0 first. Verify Temporal processing and drain all queued and in-flight AMQP jobs before upgrading beyond it.
• Remove the retired Parse transport override. V1_PARSE_VIA_TEMPORAL_PERCENTAGE has been removed. Delete it from your values or environment; setting it to 0 no longer routes Parse jobs to AMQP. Re-enabling RabbitMQ alone does not restore that path either.
• Check Parse worker capacity. New Parse jobs run on llamacloud-temporal-parse. Before upgrading from AMQP, update autoscalers that target the old worker or read RabbitMQ queue depth, including OCR scalers using that signal.
• Replace queue-status autoscalers. /api/queue-statusz returns 410 Gone after this upgrade. It does not report Temporal backlog or health; follow the autoscaling guide before upgrading.

Step 1: Prepare the Temporal workers and drain AMQP

Legacy Pipelines (Index v1) can stay on the v1 API. Before removing RabbitMQ:

• Upgrade through version 0.10.0, which dispatches Index v1 through Temporal unconditionally. Confirm that no queued or in-flight AMQP jobs remain before upgrading further. Remove the retired INDEX_V1_*_TEMPORAL_PERCENTAGE overrides; setting them below 100 no longer selects AMQP.
• Verify the capacity and autoscaling of temporal-jobs-worker, which runs Index v1 with the chart’s default consolidated profile. File processing also uses llamacloud-temporal-parse. Existing HPAs or ScaledObjects targeting the AMQP workers do not scale these deployments. Replace scaling signals based on RabbitMQ queue depth, including any OCR scaler using that signal.
• Submit a pipeline sync through your existing v1 client and confirm it completes on the Temporal workers before upgrading further.

Step 2: Upgrade and remove obsolete configuration

After completing step 1, remove the rabbitmq: and jobsWorker: blocks and any AMQP_DISABLED environment override from your values, then upgrade the chart and application images together. No RabbitMQ credentials or connection details are required.

The chart no longer creates llamacloud-parse. Move any required settings from llamaParse to temporalWorkloads.llamaParse, then remove the old block before upgrading. Remove custom autoscalers and monitors targeting the retired deployment.

Step 3: Verify

Check that the Temporal worker deployments are available:

kubectl get deployment llamacloud-temporal-parse temporal-jobs-worker -n llamacloud

Submit a Parse job and wait for SUCCESS. If you use Extract or Index, submit an extraction or trigger a pipeline sync and confirm it completes too.

Step 4: Remove RabbitMQ

Only after step 3 passes. If you installed it in-cluster with the Bitnami chart:

helm uninstall rabbitmq --namespace llamacloud

If you use a managed broker, delete it in your cloud provider’s console. The upgraded application no longer connects to it.

What returns an error afterwards

Legacy Pipelines no longer return 410 Gone just because RabbitMQ is disabled. Index v1 still returns 410 if v1 is disabled or its required MongoDB store is absent:

{
  "detail": "Pipelines is unavailable on this deployment: it requires MongoDB, which is disabled (MONGO_DISABLED=true). Use the v2 API instead."
}

Rolling back

Re-enabling RabbitMQ does not move new Parse v1, Extract v1, or Index v1 jobs back to AMQP. Returning to AMQP requires an older application release with its consumers, a compatible chart, and the original broker. Restore the application and chart versions together; restoring the broker alone does not restore job processing.