# Documents ## Create Batch Pipeline Documents `IReadOnlyList Pipelines.Documents.Create(DocumentCreateParamsparameters, CancellationTokencancellationToken = default)` **post** `/api/v1/pipelines/{pipeline_id}/documents` Batch create documents for a pipeline. ### Parameters - `DocumentCreateParams parameters` - `required string pipelineID` - `required IReadOnlyList body` - `required IReadOnlyDictionary Metadata` - `required string Text` - `string? ID` - `IReadOnlyList ExcludedEmbedMetadataKeys` - `IReadOnlyList ExcludedLlmMetadataKeys` - `IReadOnlyList? PagePositions` indices in the CloudDocument.text where a new page begins. e.g. Second page starts at index specified by page_positions[1]. ### Example ```csharp DocumentCreateParams parameters = new() { PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", Body = [ new() { Metadata = new Dictionary() { { "foo", JsonSerializer.SerializeToElement("bar") } }, Text = "text", ID = "id", ExcludedEmbedMetadataKeys = [ "string" ], ExcludedLlmMetadataKeys = [ "string" ], PagePositions = [ 0 ], }, ], }; var cloudDocuments = await client.Pipelines.Documents.Create(parameters); Console.WriteLine(cloudDocuments); ``` #### Response ```json [ { "id": "id", "metadata": { "foo": "bar" }, "text": "text", "excluded_embed_metadata_keys": [ "string" ], "excluded_llm_metadata_keys": [ "string" ], "page_positions": [ 0 ], "status_metadata": { "foo": "bar" } } ] ``` ## Paginated List Pipeline Documents `DocumentListPageResponse Pipelines.Documents.List(DocumentListParamsparameters, CancellationTokencancellationToken = default)` **get** `/api/v1/pipelines/{pipeline_id}/documents/paginated` Return a list of documents for a pipeline. ### Parameters - `DocumentListParams parameters` - `required string pipelineID` - `string? fileID` - `Long limit` - `Boolean? onlyApiDataSourceDocuments` - `Boolean? onlyDirectUpload` - `Long skip` - `StatusRefreshPolicy statusRefreshPolicy` - `"cached"Cached` - `"ttl"Ttl` ### Returns - `class DocumentListPageResponse:` - `required IReadOnlyList Documents` The documents to list - `required string ID` - `required IReadOnlyDictionary Metadata` - `required string Text` - `IReadOnlyList ExcludedEmbedMetadataKeys` - `IReadOnlyList ExcludedLlmMetadataKeys` - `IReadOnlyList? PagePositions` indices in the CloudDocument.text where a new page begins. e.g. Second page starts at index specified by page_positions[1]. - `IReadOnlyDictionary? StatusMetadata` - `required Long Limit` The limit of the documents - `required Long Offset` The offset of the documents - `required Long TotalCount` The total number of documents ### Example ```csharp DocumentListParams parameters = new() { PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e" }; var page = await client.Pipelines.Documents.List(parameters); await foreach (var item in page.Paginate()) { Console.WriteLine(item); } ``` #### Response ```json { "documents": [ { "id": "id", "metadata": { "foo": "bar" }, "text": "text", "excluded_embed_metadata_keys": [ "string" ], "excluded_llm_metadata_keys": [ "string" ], "page_positions": [ 0 ], "status_metadata": { "foo": "bar" } } ], "limit": 0, "offset": 0, "total_count": 0 } ``` ## Get Pipeline Document `CloudDocument Pipelines.Documents.Get(DocumentGetParamsparameters, CancellationTokencancellationToken = default)` **get** `/api/v1/pipelines/{pipeline_id}/documents/{document_id}` Return a single document for a pipeline. ### Parameters - `DocumentGetParams parameters` - `required string pipelineID` - `required string documentID` ### Returns - `class CloudDocument:` Cloud document stored in S3. - `required string ID` - `required IReadOnlyDictionary Metadata` - `required string Text` - `IReadOnlyList ExcludedEmbedMetadataKeys` - `IReadOnlyList ExcludedLlmMetadataKeys` - `IReadOnlyList? PagePositions` indices in the CloudDocument.text where a new page begins. e.g. Second page starts at index specified by page_positions[1]. - `IReadOnlyDictionary? StatusMetadata` ### Example ```csharp DocumentGetParams parameters = new() { PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", DocumentID = "document_id", }; var cloudDocument = await client.Pipelines.Documents.Get(parameters); Console.WriteLine(cloudDocument); ``` #### Response ```json { "id": "id", "metadata": { "foo": "bar" }, "text": "text", "excluded_embed_metadata_keys": [ "string" ], "excluded_llm_metadata_keys": [ "string" ], "page_positions": [ 0 ], "status_metadata": { "foo": "bar" } } ``` ## Delete Pipeline Document `Pipelines.Documents.Delete(DocumentDeleteParamsparameters, CancellationTokencancellationToken = default)` **delete** `/api/v1/pipelines/{pipeline_id}/documents/{document_id}` Delete a document from a pipeline; runs async (vectors first, then MongoDB record). ### Parameters - `DocumentDeleteParams parameters` - `required string pipelineID` - `required string documentID` ### Example ```csharp DocumentDeleteParams parameters = new() { PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", DocumentID = "document_id", }; await client.Pipelines.Documents.Delete(parameters); ``` ## Get Pipeline Document Status `ManagedIngestionStatusResponse Pipelines.Documents.GetStatus(DocumentGetStatusParamsparameters, CancellationTokencancellationToken = default)` **get** `/api/v1/pipelines/{pipeline_id}/documents/{document_id}/status` Return a single document for a pipeline. ### Parameters - `DocumentGetStatusParams parameters` - `required string pipelineID` - `required string documentID` ### Returns - `class ManagedIngestionStatusResponse:` - `required Status Status` Status of the ingestion. - `"CANCELLED"Cancelled` - `"ERROR"Error` - `"IN_PROGRESS"InProgress` - `"NOT_STARTED"NotStarted` - `"PARTIAL_SUCCESS"PartialSuccess` - `"SUCCESS"Success` - `DateTimeOffset? DeploymentDate` Date of the deployment. - `DateTimeOffset? EffectiveAt` When the status is effective - `IReadOnlyList? Error` List of errors that occurred during ingestion. - `required string JobID` ID of the job that failed. - `required string Message` List of errors that occurred during ingestion. - `required Step Step` Name of the job that failed. - `"DATA_SOURCE"DataSource` - `"FILE_UPDATER"FileUpdater` - `"INGESTION"Ingestion` - `"MANAGED_INGESTION"ManagedIngestion` - `"METADATA_UPDATE"MetadataUpdate` - `"PARSE"Parse` - `"TRANSFORM"Transform` - `string? JobID` ID of the latest job. ### Example ```csharp DocumentGetStatusParams parameters = new() { PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", DocumentID = "document_id", }; var managedIngestionStatusResponse = await client.Pipelines.Documents.GetStatus(parameters); Console.WriteLine(managedIngestionStatusResponse); ``` #### Response ```json { "status": "CANCELLED", "deployment_date": "2019-12-27T18:11:19.117Z", "effective_at": "2019-12-27T18:11:19.117Z", "error": [ { "job_id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", "message": "message", "step": "DATA_SOURCE" } ], "job_id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e" } ``` ## Sync Pipeline Document `JsonElement Pipelines.Documents.Sync(DocumentSyncParamsparameters, CancellationTokencancellationToken = default)` **post** `/api/v1/pipelines/{pipeline_id}/documents/{document_id}/sync` Sync a specific document for a pipeline. ### Parameters - `DocumentSyncParams parameters` - `required string pipelineID` - `required string documentID` ### Example ```csharp DocumentSyncParams parameters = new() { PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", DocumentID = "document_id", }; var response = await client.Pipelines.Documents.Sync(parameters); Console.WriteLine(response); ``` #### Response ```json {} ``` ## List Pipeline Document Chunks `IReadOnlyList Pipelines.Documents.GetChunks(DocumentGetChunksParamsparameters, CancellationTokencancellationToken = default)` **get** `/api/v1/pipelines/{pipeline_id}/documents/{document_id}/chunks` Return a list of chunks for a pipeline document. ### Parameters - `DocumentGetChunksParams parameters` - `required string pipelineID` - `required string documentID` ### Example ```csharp DocumentGetChunksParams parameters = new() { PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", DocumentID = "document_id", }; var textNodes = await client.Pipelines.Documents.GetChunks(parameters); Console.WriteLine(textNodes); ``` #### Response ```json [ { "class_name": "class_name", "embedding": [ 0 ], "end_char_idx": 0, "excluded_embed_metadata_keys": [ "string" ], "excluded_llm_metadata_keys": [ "string" ], "extra_info": { "foo": "bar" }, "id_": "id_", "metadata_seperator": "metadata_seperator", "metadata_template": "metadata_template", "mimetype": "mimetype", "relationships": { "foo": { "node_id": "node_id", "class_name": "class_name", "hash": "hash", "metadata": { "foo": "bar" }, "node_type": "1" } }, "start_char_idx": 0, "text": "text", "text_template": "text_template" } ] ``` ## Upsert Batch Pipeline Documents `IReadOnlyList Pipelines.Documents.Upsert(DocumentUpsertParamsparameters, CancellationTokencancellationToken = default)` **put** `/api/v1/pipelines/{pipeline_id}/documents` Batch create or update a document for a pipeline. ### Parameters - `DocumentUpsertParams parameters` - `required string pipelineID` - `required IReadOnlyList body` - `required IReadOnlyDictionary Metadata` - `required string Text` - `string? ID` - `IReadOnlyList ExcludedEmbedMetadataKeys` - `IReadOnlyList ExcludedLlmMetadataKeys` - `IReadOnlyList? PagePositions` indices in the CloudDocument.text where a new page begins. e.g. Second page starts at index specified by page_positions[1]. ### Example ```csharp DocumentUpsertParams parameters = new() { PipelineID = "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", Body = [ new() { Metadata = new Dictionary() { { "foo", JsonSerializer.SerializeToElement("bar") } }, Text = "text", ID = "id", ExcludedEmbedMetadataKeys = [ "string" ], ExcludedLlmMetadataKeys = [ "string" ], PagePositions = [ 0 ], }, ], }; var cloudDocuments = await client.Pipelines.Documents.Upsert(parameters); Console.WriteLine(cloudDocuments); ``` #### Response ```json [ { "id": "id", "metadata": { "foo": "bar" }, "text": "text", "excluded_embed_metadata_keys": [ "string" ], "excluded_llm_metadata_keys": [ "string" ], "page_positions": [ 0 ], "status_metadata": { "foo": "bar" } } ] ``` ## Domain Types ### Cloud Document - `class CloudDocument:` Cloud document stored in S3. - `required string ID` - `required IReadOnlyDictionary Metadata` - `required string Text` - `IReadOnlyList ExcludedEmbedMetadataKeys` - `IReadOnlyList ExcludedLlmMetadataKeys` - `IReadOnlyList? PagePositions` indices in the CloudDocument.text where a new page begins. e.g. Second page starts at index specified by page_positions[1]. - `IReadOnlyDictionary? StatusMetadata` ### Cloud Document Create - `class CloudDocumentCreate:` Create a new cloud document. - `required IReadOnlyDictionary Metadata` - `required string Text` - `string? ID` - `IReadOnlyList ExcludedEmbedMetadataKeys` - `IReadOnlyList ExcludedLlmMetadataKeys` - `IReadOnlyList? PagePositions` indices in the CloudDocument.text where a new page begins. e.g. Second page starts at index specified by page_positions[1]. ### Text Node - `class TextNode:` Provided for backward compatibility. - `string ClassName` - `IReadOnlyList? Embedding` Embedding of the node. - `Long? EndCharIdx` End char index of the node. - `IReadOnlyList ExcludedEmbedMetadataKeys` Metadata keys that are excluded from text for the embed model. - `IReadOnlyList ExcludedLlmMetadataKeys` Metadata keys that are excluded from text for the LLM. - `IReadOnlyDictionary ExtraInfo` A flat dictionary of metadata fields - `string ID` Unique ID of the node. - `string MetadataSeperator` Separator between metadata fields when converting to string. - `string MetadataTemplate` Template for how metadata is formatted, with {key} and {value} placeholders. - `string Mimetype` MIME type of the node content. - `IReadOnlyDictionary Relationships` A mapping of relationships to other node information. - `class RelatedNodeInfo:` - `required string NodeID` - `string ClassName` - `string? Hash` - `IReadOnlyDictionary Metadata` - `NodeType? NodeType` - `"1"1` - `"2"2` - `"3"3` - `"4"4` - `"5"5` - `IReadOnlyList` - `required string NodeID` - `string ClassName` - `string? Hash` - `IReadOnlyDictionary Metadata` - `NodeType? NodeType` - `"1"1` - `"2"2` - `"3"3` - `"4"4` - `"5"5` - `Long? StartCharIdx` Start char index of the node. - `string Text` Text content of the node. - `string TextTemplate` Template for how text is formatted, with {content} and {metadata_str} placeholders.