API Reference

Pipelines

Data Sources

Copy Markdown

Open in Claude

Open in ChatGPT

Open in Cursor

---

Copy Markdown

View as Markdown

Update Pipeline Data Source

Deprecated

PipelineDataSource Pipelines.DataSources.Update(DataSourceUpdateParamsparameters, CancellationTokencancellationToken = default)

PUT/api/v1/pipelines/{pipeline_id}/data-sources/{data_source_id}

Update the configuration of a data source in a pipeline.

ParametersExpand Collapse

DataSourceUpdateParams parameters

required string pipelineID

Path param

formatuuid

required string dataSourceID

Path param

formatuuid

Double? syncInterval

Body param: The interval at which the data source should be synced.

ReturnsExpand Collapse

class PipelineDataSource:

Schema for a data source in a pipeline.

required string ID

Unique identifier

formatuuid

required Component Component

Component that implements the data source

One of the following:

IReadOnlyDictionary<string, JsonElement>

class CloudS3DataSource:

required string Bucket

The name of the S3 bucket to read from.

string? AwsAccessID

The AWS access ID to use for authentication.

string? AwsAccessSecret

The AWS access secret to use for authentication.

formatpassword

string ClassName

string? Prefix

The prefix of the S3 objects to read from.

string? RegexPattern

The regex pattern to filter S3 objects. Must be a valid regex pattern.

string? S3EndpointUrl

The S3 endpoint URL to use for authentication.

Boolean SupportsAccessControl

class CloudAzStorageBlobDataSource:

required string AccountUrl

The Azure Storage Blob account URL to use for authentication.

required string ContainerName

The name of the Azure Storage Blob container to read from.

string? AccountKey

The Azure Storage Blob account key to use for authentication.

formatpassword

string? AccountName

The Azure Storage Blob account name to use for authentication.

string? Blob

The blob name to read from.

string ClassName

string? ClientID

The Azure AD client ID to use for authentication.

string? ClientSecret

The Azure AD client secret to use for authentication.

formatpassword

string? Prefix

The prefix of the Azure Storage Blob objects to read from.

Boolean SupportsAccessControl

string? TenantID

The Azure AD tenant ID to use for authentication.

class CloudGoogleDriveDataSource:

required string FolderID

The ID of the Google Drive folder to read from.

string ClassName

string? FolderName

Human-readable name of the selected folder, for display.

IReadOnlyDictionary<string, string>? ServiceAccountKey

A dictionary containing secret values

Boolean SupportsAccessControl

class CloudOneDriveDataSource:

required string ClientID

The client ID to use for authentication.

required string ClientSecret

The client secret to use for authentication.

formatpassword

required string TenantID

The tenant ID to use for authentication.

required string UserPrincipalName

The user principal name to use for authentication.

string ClassName

string? FolderID

The ID of the OneDrive folder to read from.

string? FolderPath

The path of the OneDrive folder to read from.

IReadOnlyList<string>? RequiredExts

The list of required file extensions.

SupportsAccessControl SupportsAccessControl

class CloudSharepointDataSource:

required string ClientID

The client ID to use for authentication.

required string ClientSecret

The client secret to use for authentication.

formatpassword

required string TenantID

The tenant ID to use for authentication.

string ClassName

string? DriveName

The name of the Sharepoint drive to read from.

IReadOnlyList<string>? ExcludePathPatterns

List of regex patterns for file paths to exclude. Files whose paths (including filename) match any pattern will be excluded. Example: [‘/temp/’, ‘/backup/’, ‘.git/’, ‘.tmp$’, ’^~’]

string? FolderID

The ID of the Sharepoint folder to read from.

string? FolderPath

The path of the Sharepoint folder to read from.

Boolean GetPermissions

Whether to get permissions for the sharepoint site.

IReadOnlyList<string>? IncludePathPatterns

List of regex patterns for file paths to include. Full paths (including filename) must match at least one pattern to be included. Example: [‘/reports/’, ‘/docs/..pdf$’, ‘^Report..pdf$’]

IReadOnlyList<string>? RequiredExts

The list of required file extensions.

string? SiteID

The ID of the SharePoint site to download from.

string? SiteName

The name of the SharePoint site to download from.

SupportsAccessControl SupportsAccessControl

class CloudSlackDataSource:

required string SlackToken

Slack Bot Token.

formatpassword

string? ChannelIds

Slack Channel.

string? ChannelPatterns

Slack Channel name pattern.

string ClassName

string? EarliestDate

Earliest date.

Double? EarliestDateTimestamp

Earliest date timestamp.

string? LatestDate

Latest date.

Double? LatestDateTimestamp

Latest date timestamp.

Boolean SupportsAccessControl

class CloudNotionPageDataSource:

required string IntegrationToken

The integration token to use for authentication.

formatpassword

string ClassName

string? DatabaseIds

The Notion Database Id to read content from.

string? PageIds

The Page ID’s of the Notion to read from.

Boolean SupportsAccessControl

class CloudConfluenceDataSource:

required string AuthenticationMechanism

Type of Authentication for connecting to Confluence APIs.

required string ServerUrl

The server URL of the Confluence instance.

string? ApiToken

The API token to use for authentication.

formatpassword

string ClassName

string? Cql

The CQL query to use for fetching pages.

FailureHandlingConfig FailureHandling

Configuration for handling failures during processing. Key-value object controlling failure handling behaviors.

Example: { “skip_list_failures”: true }

Currently supports:

• skip_list_failures: Skip failed batches/lists and continue processing

Boolean SkipListFailures

Whether to skip failed batches/lists and continue processing

Boolean IndexRestrictedPages

Whether to index restricted pages.

Boolean KeepMarkdownFormat

Whether to keep the markdown format.

string? Label

The label to use for fetching pages.

string? PageIds

The page IDs of the Confluence to read from.

string? SpaceKey

The space key to read from.

Boolean SupportsAccessControl

Boolean SyncPermissions

Whether to fetch space-level permissions (allowed users/groups) and attach them to document metadata for access control. Disable for Confluence Server/Data Center versions whose permission APIs are unavailable (e.g. the JSON-RPC API removed in Data Center 9.2.6+), which otherwise surface as 401 errors during sync.

string? UserName

The username to use for authentication.

class CloudJiraDataSource:

Cloud Jira Data Source integrating JiraReader.

required string AuthenticationMechanism

Type of Authentication for connecting to Jira APIs.

required string Query

JQL (Jira Query Language) query to search.

string? ApiToken

The API/ Access Token used for Basic, PAT and OAuth2 authentication.

formatpassword

string ClassName

string? CloudID

The cloud ID, used in case of OAuth2.

string? Email

The email address to use for authentication.

string? ServerUrl

The server url for Jira Cloud.

Boolean SupportsAccessControl

class CloudJiraDataSourceV2:

Cloud Jira Data Source integrating JiraReaderV2.

required string AuthenticationMechanism

Type of Authentication for connecting to Jira APIs.

required string Query

JQL (Jira Query Language) query to search.

required string ServerUrl

The server url for Jira Cloud.

string? ApiToken

The API Access Token used for Basic, PAT and OAuth2 authentication.

formatpassword

ApiVersion ApiVersion

Jira REST API version to use (2 or 3). 3 supports Atlassian Document Format (ADF).

One of the following:

"2"2

"3"3

string ClassName

string? CloudID

The cloud ID, used in case of OAuth2.

string? Email

The email address to use for authentication.

string? Expand

Fields to expand in the response.

IReadOnlyList<string>? Fields

List of fields to retrieve from Jira. If None, retrieves all fields.

Boolean GetPermissions

Whether to fetch project role permissions and issue-level security

Long? RequestsPerMinute

Rate limit for Jira API requests per minute.

Boolean SupportsAccessControl

class CloudBoxDataSource:

required AuthenticationMechanism AuthenticationMechanism

The type of authentication to use (Developer Token or CCG)

One of the following:

"ccg"Ccg

"developer_token"DeveloperToken

string ClassName

string? ClientID

Box API key used for identifying the application the user is authenticating with

string? ClientSecret

Box API secret used for making auth requests.

formatpassword

string? DeveloperToken

Developer token for authentication if authentication_mechanism is ‘developer_token’.

formatpassword

string? EnterpriseID

Box Enterprise ID, if provided authenticates as service.

string? FolderID

The ID of the Box folder to read from.

Boolean SupportsAccessControl

string? UserID

Box User ID, if provided authenticates as user.

required string DataSourceID

The ID of the data source.

formatuuid

required DateTimeOffset LastSyncedAt

The last time the data source was automatically synced.

formatdate-time

required string Name

The name of the data source.

required string PipelineID

The ID of the pipeline.

formatuuid

required string ProjectID

required SourceType SourceType

One of the following:

"AZURE_STORAGE_BLOB"AzureStorageBlob

"BOX"Box

"CONFLUENCE"Confluence

"GOOGLE_DRIVE"GoogleDrive

"JIRA"Jira

"JIRA_V2"JiraV2

"MICROSOFT_ONEDRIVE"MicrosoftOnedrive

"MICROSOFT_SHAREPOINT"MicrosoftSharepoint

"NOTION_PAGE"NotionPage

"S3"S3

"SLACK"Slack

DateTimeOffset? CreatedAt

Creation datetime

formatdate-time

IReadOnlyDictionary<string, CustomMetadata?>? CustomMetadata

Custom metadata that will be present on all data loaded from the data source

One of the following:

IReadOnlyDictionary<string, JsonElement>

IReadOnlyList<JsonElement>

string

Double

Boolean

Status? Status

The status of the data source in the pipeline.

One of the following:

"CANCELLED"Cancelled

"ERROR"Error

"IN_PROGRESS"InProgress

"NOT_STARTED"NotStarted

"SUCCESS"Success

DateTimeOffset? StatusUpdatedAt

The last time the status was updated.

formatdate-time

Double? SyncInterval

The interval at which the data source should be synced.

string? SyncScheduleSetBy

The id of the user who set the sync schedule.

DateTimeOffset? UpdatedAt

Update datetime

formatdate-time

DataSourceReaderVersionMetadata? VersionMetadata

Version metadata for the data source

ReaderVersion? ReaderVersion

The version of the reader to use for this data source.

One of the following:

"1.0"1_0

"2.0"2_0

"2.1"2_1

Update Pipeline Data Source

C#

HTTP

TypeScript

Python

Go

Java

C#

CLI Tool

DataSourceUpdateParams parameters = new()
{
    PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
    DataSourceID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
};

var pipelineDataSource = await client.Pipelines.DataSources.Update(parameters);

Console.WriteLine(pipelineDataSource);

200 example

{
  "id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
  "component": {
    "foo": "bar"
  },
  "data_source_id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
  "last_synced_at": "2019-12-27T18:11:19.117Z",
  "name": "name",
  "pipeline_id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
  "project_id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
  "source_type": "AZURE_STORAGE_BLOB",
  "created_at": "2019-12-27T18:11:19.117Z",
  "custom_metadata": {
    "foo": {
      "foo": "bar"
    }
  },
  "status": "CANCELLED",
  "status_updated_at": "2019-12-27T18:11:19.117Z",
  "sync_interval": 0,
  "sync_schedule_set_by": "sync_schedule_set_by",
  "updated_at": "2019-12-27T18:11:19.117Z",
  "version_metadata": {
    "reader_version": "1.0"
  }
}

Returns Examples

200 example

{
  "id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
  "component": {
    "foo": "bar"
  },
  "data_source_id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
  "last_synced_at": "2019-12-27T18:11:19.117Z",
  "name": "name",
  "pipeline_id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
  "project_id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
  "source_type": "AZURE_STORAGE_BLOB",
  "created_at": "2019-12-27T18:11:19.117Z",
  "custom_metadata": {
    "foo": {
      "foo": "bar"
    }
  },
  "status": "CANCELLED",
  "status_updated_at": "2019-12-27T18:11:19.117Z",
  "sync_interval": 0,
  "sync_schedule_set_by": "sync_schedule_set_by",
  "updated_at": "2019-12-27T18:11:19.117Z",
  "version_metadata": {
    "reader_version": "1.0"
  }
}