Data Sources

List Pipeline Data Sources

client.Pipelines.DataSources.GetDataSources(ctx, pipelineID) (*[]PipelineDataSource, error)

GET/api/v1/pipelines/{pipeline_id}/data-sources

Add Data Sources To Pipeline

client.Pipelines.DataSources.UpdateDataSources(ctx, pipelineID, body) (*[]PipelineDataSource, error)

PUT/api/v1/pipelines/{pipeline_id}/data-sources

Update Pipeline Data Source

client.Pipelines.DataSources.Update(ctx, dataSourceID, params) (*PipelineDataSource, error)

PUT/api/v1/pipelines/{pipeline_id}/data-sources/{data_source_id}

Get Pipeline Data Source Status

client.Pipelines.DataSources.GetStatus(ctx, dataSourceID, query) (*ManagedIngestionStatusResponse, error)

GET/api/v1/pipelines/{pipeline_id}/data-sources/{data_source_id}/status

Sync Pipeline Data Source

client.Pipelines.DataSources.Sync(ctx, dataSourceID, params) (*Pipeline, error)

POST/api/v1/pipelines/{pipeline_id}/data-sources/{data_source_id}/sync

ModelsExpand Collapse

type PipelineDataSource struct{…}

Schema for a data source in a pipeline.

ID string

Unique identifier

formatuuid

Component PipelineDataSourceComponentUnion

Component that implements the data source

One of the following:

type PipelineDataSourceComponentMap map[string, any]

type CloudS3DataSource struct{…}

Bucket string

The name of the S3 bucket to read from.

AwsAccessID stringoptional

The AWS access ID to use for authentication.

AwsAccessSecret stringoptional

The AWS access secret to use for authentication.

formatpassword

ClassName stringoptional

Prefix stringoptional

The prefix of the S3 objects to read from.

RegexPattern stringoptional

The regex pattern to filter S3 objects. Must be a valid regex pattern.

S3EndpointURL stringoptional

The S3 endpoint URL to use for authentication.

SupportsAccessControl booloptional

type CloudAzStorageBlobDataSource struct{…}

AccountURL string

The Azure Storage Blob account URL to use for authentication.

ContainerName string

The name of the Azure Storage Blob container to read from.

AccountKey stringoptional

The Azure Storage Blob account key to use for authentication.

formatpassword

AccountName stringoptional

The Azure Storage Blob account name to use for authentication.

Blob stringoptional

The blob name to read from.

ClassName stringoptional

ClientID stringoptional

The Azure AD client ID to use for authentication.

ClientSecret stringoptional

The Azure AD client secret to use for authentication.

formatpassword

Prefix stringoptional

The prefix of the Azure Storage Blob objects to read from.

SupportsAccessControl booloptional

TenantID stringoptional

The Azure AD tenant ID to use for authentication.

type CloudGoogleDriveDataSource struct{…}

FolderID string

The ID of the Google Drive folder to read from.

ClassName stringoptional

ServiceAccountKey map[string, string]optional

A dictionary containing secret values

SupportsAccessControl booloptional

type CloudOneDriveDataSource struct{…}

ClientID string

The client ID to use for authentication.

ClientSecret string

The client secret to use for authentication.

formatpassword

TenantID string

The tenant ID to use for authentication.

UserPrincipalName string

The user principal name to use for authentication.

ClassName stringoptional

FolderID stringoptional

The ID of the OneDrive folder to read from.

FolderPath stringoptional

The path of the OneDrive folder to read from.

RequiredExts []stringoptional

The list of required file extensions.

SupportsAccessControl booloptional

type CloudSharepointDataSource struct{…}

ClientID string

The client ID to use for authentication.

ClientSecret string

The client secret to use for authentication.

formatpassword

TenantID string

The tenant ID to use for authentication.

ClassName stringoptional

DriveName stringoptional

The name of the Sharepoint drive to read from.

ExcludePathPatterns []stringoptional

List of regex patterns for file paths to exclude. Files whose paths (including filename) match any pattern will be excluded. Example: [‘/temp/’, ‘/backup/’, ‘.git/’, ‘.tmp$’, ’^~’]

FolderID stringoptional

The ID of the Sharepoint folder to read from.

FolderPath stringoptional

The path of the Sharepoint folder to read from.

GetPermissions booloptional

Whether to get permissions for the sharepoint site.

IncludePathPatterns []stringoptional

List of regex patterns for file paths to include. Full paths (including filename) must match at least one pattern to be included. Example: [‘/reports/’, ‘/docs/..pdf$’, ‘^Report..pdf$’]

RequiredExts []stringoptional

The list of required file extensions.

SiteID stringoptional

The ID of the SharePoint site to download from.

SiteName stringoptional

The name of the SharePoint site to download from.

SupportsAccessControl booloptional

type CloudSlackDataSource struct{…}

SlackToken string

Slack Bot Token.

formatpassword

ChannelIDs stringoptional

Slack Channel.

ChannelPatterns stringoptional

Slack Channel name pattern.

ClassName stringoptional

EarliestDate stringoptional

Earliest date.

EarliestDateTimestamp float64optional

Earliest date timestamp.

LatestDate stringoptional

Latest date.

LatestDateTimestamp float64optional

Latest date timestamp.

SupportsAccessControl booloptional

type CloudNotionPageDataSource struct{…}

IntegrationToken string

The integration token to use for authentication.

formatpassword

ClassName stringoptional

DatabaseIDs stringoptional

The Notion Database Id to read content from.

PageIDs stringoptional

The Page ID’s of the Notion to read from.

SupportsAccessControl booloptional

type CloudConfluenceDataSource struct{…}

AuthenticationMechanism string

Type of Authentication for connecting to Confluence APIs.

ServerURL string

The server URL of the Confluence instance.

APIToken stringoptional

The API token to use for authentication.

formatpassword

ClassName stringoptional

Cql stringoptional

The CQL query to use for fetching pages.

FailureHandling FailureHandlingConfigoptional

Configuration for handling failures during processing. Key-value object controlling failure handling behaviors.

Example: { “skip_list_failures”: true }

Currently supports:

skip_list_failures: Skip failed batches/lists and continue processing

SkipListFailures booloptional

Whether to skip failed batches/lists and continue processing

IndexRestrictedPages booloptional

Whether to index restricted pages.

KeepMarkdownFormat booloptional

Whether to keep the markdown format.

Label stringoptional

The label to use for fetching pages.

PageIDs stringoptional

The page IDs of the Confluence to read from.

SpaceKey stringoptional

The space key to read from.

SupportsAccessControl booloptional

UserName stringoptional

The username to use for authentication.

type CloudJiraDataSource struct{…}

Cloud Jira Data Source integrating JiraReader.

AuthenticationMechanism string

Type of Authentication for connecting to Jira APIs.

Query string

JQL (Jira Query Language) query to search.

APIToken stringoptional

The API/ Access Token used for Basic, PAT and OAuth2 authentication.

formatpassword

ClassName stringoptional

CloudID stringoptional

The cloud ID, used in case of OAuth2.

Email stringoptional

The email address to use for authentication.

ServerURL stringoptional

The server url for Jira Cloud.

SupportsAccessControl booloptional

type CloudJiraDataSourceV2 struct{…}

Cloud Jira Data Source integrating JiraReaderV2.

AuthenticationMechanism string

Type of Authentication for connecting to Jira APIs.

Query string

JQL (Jira Query Language) query to search.

ServerURL string

The server url for Jira Cloud.

APIToken stringoptional

The API Access Token used for Basic, PAT and OAuth2 authentication.

formatpassword

APIVersion CloudJiraDataSourceV2APIVersionoptional

Jira REST API version to use (2 or 3). 3 supports Atlassian Document Format (ADF).

One of the following:

const CloudJiraDataSourceV2APIVersion2 CloudJiraDataSourceV2APIVersion = "2"

const CloudJiraDataSourceV2APIVersion3 CloudJiraDataSourceV2APIVersion = "3"

ClassName stringoptional

CloudID stringoptional

The cloud ID, used in case of OAuth2.

Email stringoptional

The email address to use for authentication.

Expand stringoptional

Fields to expand in the response.

Fields []stringoptional

List of fields to retrieve from Jira. If None, retrieves all fields.

GetPermissions booloptional

Whether to fetch project role permissions and issue-level security

RequestsPerMinute int64optional

Rate limit for Jira API requests per minute.

SupportsAccessControl booloptional

type CloudBoxDataSource struct{…}

AuthenticationMechanism CloudBoxDataSourceAuthenticationMechanism

The type of authentication to use (Developer Token or CCG)

One of the following:

const CloudBoxDataSourceAuthenticationMechanismDeveloperToken CloudBoxDataSourceAuthenticationMechanism = "developer_token"

const CloudBoxDataSourceAuthenticationMechanismCcg CloudBoxDataSourceAuthenticationMechanism = "ccg"

ClassName stringoptional

ClientID stringoptional

Box API key used for identifying the application the user is authenticating with

ClientSecret stringoptional

Box API secret used for making auth requests.

formatpassword

DeveloperToken stringoptional

Developer token for authentication if authentication_mechanism is ‘developer_token’.

formatpassword

EnterpriseID stringoptional

Box Enterprise ID, if provided authenticates as service.

FolderID stringoptional

The ID of the Box folder to read from.

SupportsAccessControl booloptional

UserID stringoptional

Box User ID, if provided authenticates as user.

DataSourceID string

The ID of the data source.

formatuuid

LastSyncedAt Time

The last time the data source was automatically synced.

formatdate-time

Name string

The name of the data source.

PipelineID string

The ID of the pipeline.

formatuuid

ProjectID string

SourceType PipelineDataSourceSourceType

One of the following:

const PipelineDataSourceSourceTypeS3 PipelineDataSourceSourceType = "S3"

const PipelineDataSourceSourceTypeAzureStorageBlob PipelineDataSourceSourceType = "AZURE_STORAGE_BLOB"

const PipelineDataSourceSourceTypeGoogleDrive PipelineDataSourceSourceType = "GOOGLE_DRIVE"

const PipelineDataSourceSourceTypeMicrosoftOnedrive PipelineDataSourceSourceType = "MICROSOFT_ONEDRIVE"

const PipelineDataSourceSourceTypeMicrosoftSharepoint PipelineDataSourceSourceType = "MICROSOFT_SHAREPOINT"

const PipelineDataSourceSourceTypeSlack PipelineDataSourceSourceType = "SLACK"

const PipelineDataSourceSourceTypeNotionPage PipelineDataSourceSourceType = "NOTION_PAGE"

const PipelineDataSourceSourceTypeConfluence PipelineDataSourceSourceType = "CONFLUENCE"

const PipelineDataSourceSourceTypeJira PipelineDataSourceSourceType = "JIRA"

const PipelineDataSourceSourceTypeJiraV2 PipelineDataSourceSourceType = "JIRA_V2"

const PipelineDataSourceSourceTypeBox PipelineDataSourceSourceType = "BOX"

CreatedAt Timeoptional

Creation datetime

formatdate-time

CustomMetadata map[string, PipelineDataSourceCustomMetadataUnion]optional

Custom metadata that will be present on all data loaded from the data source

One of the following:

type PipelineDataSourceCustomMetadataMap map[string, any]

type PipelineDataSourceCustomMetadataArray []any

string

float64

bool

Status PipelineDataSourceStatusoptional

The status of the data source in the pipeline.

One of the following:

const PipelineDataSourceStatusNotStarted PipelineDataSourceStatus = "NOT_STARTED"

const PipelineDataSourceStatusInProgress PipelineDataSourceStatus = "IN_PROGRESS"

const PipelineDataSourceStatusSuccess PipelineDataSourceStatus = "SUCCESS"

const PipelineDataSourceStatusError PipelineDataSourceStatus = "ERROR"

const PipelineDataSourceStatusCancelled PipelineDataSourceStatus = "CANCELLED"

StatusUpdatedAt Timeoptional

The last time the status was updated.

formatdate-time

SyncInterval float64optional

The interval at which the data source should be synced.

SyncScheduleSetBy stringoptional

The id of the user who set the sync schedule.

UpdatedAt Timeoptional

Update datetime

formatdate-time

VersionMetadata DataSourceReaderVersionMetadataoptional

Version metadata for the data source

ReaderVersion DataSourceReaderVersionMetadataReaderVersionoptional

The version of the reader to use for this data source.

One of the following:

const DataSourceReaderVersionMetadataReaderVersion1_0 DataSourceReaderVersionMetadataReaderVersion = "1.0"

const DataSourceReaderVersionMetadataReaderVersion2_0 DataSourceReaderVersionMetadataReaderVersion = "2.0"

const DataSourceReaderVersionMetadataReaderVersion2_1 DataSourceReaderVersionMetadataReaderVersion = "2.1"