4.2. Sink node

4.2.1. Description

Icon: image93

  • Number of inputs: 1.

  • Number of outputs: 0.

Definition

Sink nodes let you send rows to a storage system.

Configuration

You must specify the data sink image94. If necessary, you can create a new data sink (remote file repository, database,, etc.) image95.

Specific settings are suggested for some types of sink (files in particular) image96.

Note

When a data sink is altered, the changes are applied automatically as soon as the sink configuration has been validated.

Example

image97

This example shows the configuration for a sink node.

The node lets you retrieve data processed in the flow in a csv file.

4.2.2. Save mode

Save mode image98 lets you decide how a flow is to behave if a sink already exists:

  • Overwrite :

    overwrite an existing table or file.

  • Append :

    add records to the end of an existing table or file.

  • Create :

    create a new table or file. An error will be flagged if the table (or file) already exists.

  • Insert or replace a tab :

    This mode exists for Excel sheets. This option allows you to insert an additional tab into an existing Excel file to add the rows sent to these types of targets. If the Excel file does not exist, it will be created. If a tab with the same name already exists in the Excel file, it will be overwritten.

  • Overwrite (keep the target table if it exists)

    Available for databases, with the following option (Truncate). The table will be deleted, recreated, and then filled with the data.

  • Overwrite (keep the target table if it exists)

    Available for databases, with the following option (Truncate). The table will be deleted, recreated, and then filled with the data.

  • Update :

    Available for the Snowflake connector. Allows updating data in the CRM. For more information, see the relevant section.

Hint

You can therefore create flows with multiple sinks by writing to different tabs in the same Excel file. To do this, all sinks linked to that Excel file MUST use the Insert a tab backup mode. If this is not the case (i.e. if any sink linked to the Excel file uses the Overwrite or Create backup mode), a configuration error will be triggered, preventing the running of the flow.

4.2.3. Record limit

This is the number of records image99 that can be sent to the sink. The number MUST be positive.

If no number is specified, this means that all the records are to be sent.

Warning

The limit on the number of records to be written will be taken into account after sampling.

The limit on record numbers will be taken into account during production (this does not apply to sink nodes).

4.2.4. Sampling (%)

Use this option if you do not want to send all, or the first N, records to the sink. Unlike a source node, a sink nœud uses this option during production.

This is a randomized sample of records image100 expressed as a percentage of the total size of the dataset that is to be sent to the sink.. On performance grounds, the percentage is an estimate.

Value 100 = no sampling. This means you either want to send all the records (no specified limit) or the first N records (max limit N)

4.2.5. Alerts ( Notifications )

You can configure notifications for each sink to tell you when records written to a sink validate an activation setting. To configure a notification, see Setting a notification in a flow.

4.2.6. Operation of the Add (Append) mode based on target types.

The Add mode (see save modes) behaves differently depending on the target type. This is noticeable if the data structure does not match that of the existing target as it appears at the time of the flow execution.

The main cases to report are:

  • Writing to a database table

    If the data to be written has fewer columns than the existing target at the time of execution, these missing columns will be filled with null for the added rows. If there are additional columns compared to the existing target at the time of execution, the flow will encounter an error during execution.

  • Writing to a Parquet file

    If the data to be written has fewer columns than the existing target at the time of execution, those missing columns will be filled with null for the added rows. If there are additional columns compared to the existing target at the time of execution, these columns will be added to the Parquet file and filled with null for the previous rows.

  • writing to a CSV or Excel file

    The data will be appended to the previous ones without verification against the structure of the existing columns. It is essential for the flow creator to maintain the column order to preserve the integrity of the file.