> ## Documentation Index
> Fetch the complete documentation index at: https://docs.allyy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Data

> Documentation for setting up and managing data sources

Integrating data into the system is key to making your machine learning models work effectively. This process involves setting up a **data source** that can be automated once configured. All operations occur at the **dataset level**, so understanding how to create and manage datasets and data sources is crucial.

***

## **Overview of the Process**

Here’s an outline of the steps involved in setting up a data source:

**1. Create a Dataset**

This is the container where your data sources will live. Think of it as a project-level container that can hold multiple data sources.

This image shows how to create a new dataset in the system. Datasets allow you to organize your data sources for different business cases.

Dataset creation

**2. Add a Data Source**

Within the dataset, create multiple data sources to load the necessary data.

This image displays the various data source connection options such as CSV, JSON, Google BigQuery, and Microsoft SQL Server.

Add data source screen

**3. Select the Source**

Choose the data source you want to integrate from the list of supported sources:

* CSV
* JSON
* Google BigQuery
* Google Cloud Storage
* Microsoft SQL Server
* SFTP
* Agillic
* Active Campaign

Data source selection options

**4. Enter Credentials**

Input the credentials required to connect to the data source. This might involve entering project IDs, API keys, or authentication tokens, depending on the source.

This image shows examples of credential input for Google BigQuery and Microsoft SQL Server.

Enter credentials

**5. Preview Data**

Once connected, preview the data to ensure the source is correctly connected and the data format is as expected.

In this image, you can see a preview of the data fields (e.g., ContactID, Gender, Birthdate) before confirming the connection.

Data preview

**6. Create Data Mapping**

After previewing, map the data fields to specific entities in the **allyy data structure**. Mapping allows the system to understand the relationship between fields in your data and allyy's internal structure (Contacts, Responses, Offers, etc.).

Read more about [data mapping here](#data_mapping).

This image shows the mapping process, where fields like ContactID and OfferResponse are mapped.

Data mapping screen

**7. Save the Mapping**

Once the mapping is saved, the data source can be accessed or synchronized at any time.

**8. Data Synchronization or Workflow**

For **batch data**, click the **synchronize** button to pull in data periodically or on-demand.

For **streaming data**, use the **start streaming** or **stop streaming** buttons to manage real-time data ingestion.

**9. View Details and History**

You can inspect the data source for details such as source information, last sync status, errors, and history of uploads.

***

## **Streaming vs Batch**:

> [card] Batch Data
>
> Synchronization happens manually or on a schedule. To pull in data, click the **synchronize** button or schedule it via a workflow.

> [card] Streaming Data
>
> Data flows in real time. Use the **start/stop streaming** buttons to manage continuous data flow. Streaming data cannot be scheduled.

***

## **Conclusion**

Setting up data sources allows for seamless integration of external data into the system.
By following this process, you can configure a data source once and reuse it indefinitely, either by manually pulling in data or using real-time streaming.
Ensuring that the data is correctly mapped to the **Allyy data structure** is crucial for leveraging it effectively in models and predictions.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.