Introduction

DataFlowMapper is a lightweight yet powerful data transformation tool designed to bridge the gap between expensive enterprise ETL solutions and simple data cleaners. It's built for implementation teams and data migration specialists working with CSV, Excel, and JSON files.

Our platform enables you to create repeatable transformations with an intuitive interface, powerful logic builder, and AI-assisted mapping capabilities.

Getting Started

Get up and running with DataFlowMapper quickly. Learn how to run your first data transformation.

Account Setup

Getting started with DataFlowMapper is quick and straightforward. Simply create an account to access all features immediately.

1. Visit the DataFlowMapper sign-up page
2. Enter your email address and create a password
3. Verify your email address
4. Log in to access your dashboard

Quick Start Guide

This guide will help you complete your first data transformation in minutes:

For a more detailed walkthrough, continue to the Basic Workflow section below.

Basic Workflow

The standard DataFlowMapper workflow consists of three main steps:

Transformation Engine

The core of DataFlowMapper is its powerful transformation engine that processes your data according to defined mappings and logic rules.

Supported File Formats

DataFlowMapper supports a wide range of file formats including CSV (with any delimiter), Excel spreadsheets, and JSON. You can specify header rows, concatenate files, and work with nested data structures. After uploading files, you can remove them by clicking the X next to the file name under Selected Files section.

JSON Path Syntax

When working with JSON files, DataFlowMapper uses a consistent path syntax to reference fields within nested structures. Understanding this syntax is essential for accurate mapping and transformation.

{
            "company": "Acme Inc.",
            "employees": [
              {
                "name": "John",
                "skills": ["JavaScript", "Python"],
                "projects": [
                  { "name": "Website", "priority": "High" },
                  { "name": "Mobile App", "priority": "Medium" }
                ]
              },
              {
                "name": "Jane",
                "skills": ["Java", "C++", "SQL"],
                "projects": [
                  { "name": "Database", "priority": "Critical" }
                ]
              }
            ]
          }

File Settings Panel

The file settings panel can be opened by clicking the gear below the source file input. Here you can specify header rows, whether you will concatenate files, delimiter, as well as the record level for JSON transformations.

Data Sources

Users can choose from multiple data sources, including local files, APIs, and databases. Databases and APIs can be configured for direct data import/export. To configure sources and destinations, click the desired data source, click the gear icon, and fill in the relevant fields.

Transformation Workflow

The typical workflow involves uploading a source file, creating or uploading a mapping file, transforming the data, and then viewing the output and any validations. You can save mappings for future use to create repeatable transformations.

Data Viewer

After transforming data, you will see a stat summary of the transformation. You can download the log file to view any warnings or errors with the transformation. From here you can choose to push data to API or DB destination. Clicking View Transformed Data will open the Data Viewer. The Data Viewer presents you with the output data in a flattened tabular format. You can filter, sort, and search through the data to validate the transformation results. The Data Viewer also allows you to download the transformed data and validations to Excel for further analysis.

Mapping Editor

The Mapping Editor is where you define how source data fields map to destination fields. You can create one-to-one mappings or add custom logic for complex transformations.

Field Mapping

• Create direct mappings between source and destination fields by selecting which field to map from dropdown or drag and drop from the sidebar.
• For complex scenarios, you can add custom logic using our Logic Builder by clicking Add Custom Logic
• You can also choose to add validations by clicking Add Validation.
• Filters can be created by clicking Add Pre-Processing Filter or Add Post-Processing Filter. This will open the Logic Builder where you can define the filter logic. The filter logic is applied to the source data before or after the transformation, respectively.

Header Management

Import headers from template files, create custom headers, or modify existing ones. DataFlowMapper makes it easy to align your data with the required output format. Upon mapping file creation you have the option to create headers manually by adding each field, or import headers from a template. Importing takes the first row from CSV and Excel files. For JSON, it takes the first record and flattens the data to extract field names.

Import Validations

Validations can be imported from an existing mapping file. Click the Import Validations button and upload a mapping file that has destination fields that match the current mapping. The system will automatically import the validations from the uploaded mapping file.

Logic Builder

The Logic Builder provides a visual interface for creating complex transformation logic without writing code. Drag and drop fields, variables, and functions to build your transformation rules.

Each destination field has its own separate logic that processes data row by row. The Logic Builder maintains isolated state between fields, ensuring clean and predictable transformations.

Visual Logic Interface

Our intuitive no-code interface lets you build complex transformations by dragging and dropping components. Connect fields, functions, and operations to create your transformation logic.

To define what value should be returned from the logic, set the 'result' equal to the value you want. A typical workflow in the Logic Builder would be:

1. Declare an empty variable to capture the logic you want to set the field to.
2. Create any functions needed to manipulate the data.
3. Create an if then statement and set the variable's value based on your business rules.
4. Set your variable in the Return Results tab.
5. Test and confirm output is correct by inputting a row number to sample from.
6. Save Logic

The Logic Builder is organized into multiple tabs, each serving a specific purpose:

Python Editor

For advanced users, we provide a full IDE experience with Python editor where you can write custom transformation code directly. This gives you full control over complex transformations.

# Example Python transformation
def transform_row(row):
    # Access source fields
    first_name = row['FirstName']
    last_name = row['LastName']
    
    # Return transformed value
    return f"{last_name}, {first_name}"
    
  # Return the result
  result = transform_row(row)

Filter Logic

To filter data, you can use the If/Then tab in the Logic Builder to create conditions that determine which rows should be included or excluded from the output. Setting result to False will filter the record out. By default, anything unspecified will default to True and be unfiltered.

Validation Logic

To set validations, you can use the If/Then tab in the Logic Builder to create conditions that determine which criteria should pass or fail a record in the output data. Setting result to False will denote failure. Upon selecting False, a button will appear to add a reason. This reason will be shown when reviewing the output data for the given failed record. By default, anything unspecified will default to True and pass validation.

Functions

DataFlowMapper includes a comprehensive library of built-in functions for string manipulation, mathematical operations, date handling, type conversion, and utility operations.

All functions can be dynamically constructed using the Logic Builder with zero coding. Transformation logic built with the Logic Builder will parse back into the UI state and allow you to edit the logic visually.

Functions are specified by a user input name and pass 'row' as a parameter. User defined names can&apost be duplicates or overlap with reserved words or names of function definitions below. To parse Python code back into the UI state, it must match the expected definition. See below for examples of function generation.

# Example function structure with generated Python code
            # UPPER - Converts text to uppercase
            def myUpperFunction(row):
                return str(row['field1']).upper()
            # CONCAT - Joins text values together
            def myConcatFunction(row):
                return ''.join([str(x) for x in [row['first_name'], row['last_name']]])

AI Co-Pilot

DataFlowMapper's AI Co-Pilot features help automate and simplify the mapping and transformation process with intelligent suggestions and automation.

Mapping Suggestions

The AI can analyze your source and destination fields to suggest logical mappings. You can review and approve these suggestions to speed up the mapping process. The AI will provide rationale and confidence rating for each mapping.

AI Logic Assist

Describe your transformation requirements in plain English, and the AI will generate the appropriate logic in our visual builder. This feature makes complex transformations accessible to non-technical users. This can be used for transformations and validations.

Map All Feature

The Map All feature allows you to describe your entire transformation requirement in plain language. The AI will analyze your needs and create a complete mapping file, determining which fields need direct mapping and which require custom logic as well as validations. The AI will provide details, rationale, and confidence rating for each mapping.

// Example AI prompt for Map All
"Map customer data from our CRM export to the billing system format. 
Combine first and last name into a full_name field. 
Convert dates to YYYY-MM-DD format.
Calculate total_spent as the sum of all transaction amounts."