Google Vertex AI Setup

Self-Hosting Documentation Access

This section requires a password to access. Interested in self-hosting? Contact sales to learn more.

Password:

Access Documentation

Self-Hosting Documentation Access Granted Logout

LlamaCloud supports Google Vertex AI for enterprise-grade access to Google’s AI models through Google Cloud Platform. Vertex AI provides advanced features like private endpoints, enhanced security, and deep GCP integration compared to the direct Gemini API.

Prerequisites

• A valid Google Cloud Platform account
• Google Cloud project with Vertex AI API enabled
• Service account with appropriate IAM permissions
• Access and quota for supported models:
  • Gemini 3.1 Pro
  • Gemini 3.0 Flash
  • Gemini 2.5 Flash
  • Gemini 2.5 Pro
  • Gemini 2.5 Flash Lite
  • Gemini 2.0 Flash
  • Gemini 2.0 Flash Lite

Environment Variables

The Google Vertex AI integration uses the following environment variables:

• GOOGLE_VERTEX_AI_ENABLED - Set to “true” to enable Google Vertex AI (required)
• GOOGLE_VERTEX_AI_PROJECT_ID - Google Cloud project ID (required)
• GOOGLE_VERTEX_AI_LOCATION - Google Cloud location/region (optional, defaults to us-central1)
• GOOGLE_VERTEX_AI_CREDENTIALS_JSON - Service account credentials JSON string (required)
• GOOGLE_VERTEX_AI_BASE_URL - Override the Vertex AI endpoint (optional; defaults to https://{location}-aiplatform.googleapis.com)

Configuration

Follow these steps to configure Google Vertex AI integration:

Step 1: Setup Google Cloud Project

1. Create or use an existing Google Cloud project
2. Enable the Vertex AI API:

   gcloud services enable aiplatform.googleapis.com

3. Ensure billing is enabled for the project

Step 2: Create Service Account

Create a service account with the required IAM permissions:

Required IAM Permissions

Your service account needs the following permissions:

• aiplatform.endpoints.predict
• aiplatform.endpoints.explain
• ml.models.predict (for legacy model versions)

Option 1: Use Predefined Role

gcloud iam service-accounts create llamacloud-vertex \
    --description="Service account for LlamaCloud Vertex AI" \
    --display-name="LlamaCloud Vertex AI"

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="serviceAccount:llamacloud-vertex@PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/aiplatform.user"

Option 2: Custom Role

Create a custom role with minimal permissions:

{
  "title": "LlamaCloud Vertex AI User",
  "description": "Custom role for LlamaCloud Vertex AI access",
  "stage": "GA",
  "includedPermissions": [
    "aiplatform.endpoints.predict",
    "aiplatform.endpoints.explain",
    "ml.models.predict"
  ]
}

Step 3: Generate Service Account Key

Generate a JSON key for the service account:

gcloud iam service-accounts keys create llamacloud-vertex-key.json \
    --iam-account=llamacloud-vertex@PROJECT_ID.iam.gserviceaccount.com

Step 4: Create Kubernetes Secret

Create a secret with your Vertex AI credentials:

apiVersion: v1
kind: Secret
metadata:
  name: vertex-ai-credentials
type: Opaque
stringData:
  GOOGLE_VERTEX_AI_ENABLED: "true"
  GOOGLE_VERTEX_AI_PROJECT_ID: "your-project-id"
  GOOGLE_VERTEX_AI_LOCATION: "us-central1"
  GOOGLE_VERTEX_AI_CREDENTIALS_JSON: |
    {
      "type": "service_account",
      "project_id": "your-project-id",
      "private_key_id": "key-id",
      "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
      "client_email": "llamacloud-vertex@your-project-id.iam.gserviceaccount.com",
      "client_id": "client-id",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/llamacloud-vertex%40your-project-id.iam.gserviceaccount.com"
    }

Apply the secret:

kubectl apply -f vertex-ai-secret.yaml

Step 5: Configure Helm Values

Reference the secret in your Helm configuration:

# External Secret (recommended)
config:
  llms:
    googleVertexAi:
      secret: "vertex-ai-credentials"
      baseUrl: "https://us-central1-aiplatform.googleapis.com"  # Optional — override endpoint

# or direct configuration (not recommended for production)
config:
  llms:
    googleVertexAi:
      projectId: "your-project-id"
      location: "us-central1"
      credentialsJson: |
        {
          "type": "service_account",
          ...
        }
      baseUrl: "https://us-central1-aiplatform.googleapis.com"  # Optional — override endpoint

The optional baseUrl / GOOGLE_VERTEX_AI_BASE_URL override sends Vertex AI traffic through a proxy or gateway (e.g. a self-hosted LLM gateway) instead of Google Cloud’s regional endpoint. Leave unset to use the default regional endpoint derived from location.

Verification

After configuration, verify your Google Vertex AI integration:

1. Verify in Admin UI: Check available Google models in LlamaCloud admin interface

2. Test functionality: Upload a document to confirm Vertex AI models are working

Troubleshooting

Common Issues

Service Account Permissions

Error: Permission denied

Solution:

• Verify your service account has the required IAM permissions
• Ensure aiplatform.endpoints.predict permission is granted
• Check that the service account is active and not disabled

Project Configuration

Error: Project not found or Vertex AI API not enabled

Solution:

• Ensure Vertex AI API is enabled in your Google Cloud project
• Verify the project ID is correct in your configuration
• Check that billing is enabled for the project

Credentials Issues

Error: Could not load credentials

Solution:

• Verify the service account JSON is properly formatted
• Ensure all required fields are present in the credentials JSON
• Check that the private key is properly escaped in the secret
• Verify the service account key hasn’t been deleted or expired

Region Restrictions

Error: Model not available in region

Solution:

• Check model availability in your specified region
• Try a different region (e.g., us-central1, us-east1)
• Some models may not be available in all regions

Debug Steps

1. Test Vertex AI authentication:

   gcloud auth activate-service-account --key-file=vertex-key.json
   gcloud ai models list --region=us-central1

2. Verify secret mounting:

   kubectl get secret vertex-ai-credentials -o yaml
   kubectl describe pod <pod-name> | grep -A 20 Environment

3. Check network connectivity: Ensure your cluster can reach aiplatform.googleapis.com

4. Test API call:

   curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-pro:predict" \
     -d '{"instances": [{"content": "Hello"}]}'