Data Connectors - Trendteller

Overview

The Data Connectors component provides custom Airbyte connectors that extract data from 9 different e-commerce platforms, enabling Trendteller to aggregate data from 11 brands into a unified analytics platform.

Technology Stack

Airbyte CDK

Connector Development Kit for building integrations

TypeScript

Type-safe connector development

Lerna

Monorepo management for multiple connectors

Jest

Comprehensive testing framework

Connector Architecture

Monorepo Structure

The connectors are organized in a Lerna monorepo:

airbyte-connectors/
├── sources/
│   ├── source-bling/         # Bling ERP connector
│   ├── source-bling-v3/      # Bling API v3
│   ├── source-vnda/          # VNDA platform
│   ├── source-vnda-moda/     # VNDA fashion variant
│   ├── source-shoppub/       # Shoppub marketplace
│   ├── source-tiny/          # Tiny ERP
│   ├── source-tiny-v3/       # Tiny API v3
│   ├── source-microvix/      # Microvix retail
│   ├── source-braavo/        # Braavo e-commerce
│   ├── source-totvs-moda/    # Totvs fashion ERP
│   ├── source-varejo-online/ # Varejo Online
│   ├── source-google-shopping/ # Google Shopping
│   └── ...
├── destinations/
│   ├── destination-crm360/   # CRM360 integration
│   └── destination-shops/    # Shops destination
└── shared/
    └── common-utils/         # Shared utilities

Source Connectors

Available Sources

ERP Systems
E-commerce Platforms
Other Integrations

Bling (source-bling, source-bling-v3)

Orders, products, customers, inventory
Invoices, payments, shipping
Incremental sync support

Tiny (source-tiny, source-tiny-v3)

Complete ERP data extraction
Multi-entity support
Real-time inventory updates

Totvs Moda (source-totvs-moda)

Fashion industry specific
Size/color variant handling
Collection management

VNDA (source-vnda, source-vnda-moda)

Orders and products
Customer data and analytics
Webhook integration

Shoppub (source-shoppub)

Marketplace integration
Multi-channel orders
Fulfillment tracking

Braavo (source-braavo)

E-commerce platform connector
Order and catalog sync

Microvix (source-microvix)

Retail management system
POS integration
Inventory synchronization

Google Shopping (source-google-shopping)

Product feed data
Shopping ads performance
Competitive pricing

Varejo Online (source-varejo-online)

Online retail platform
Order management

Sync Modes

Full Refresh

When to use: Small datasets, no incremental support

Replaces all existing data
Ensures consistency
Higher resource usage

Example: Product catalogs, brand configurations

Incremental Sync

When to use: Large datasets with timestamp fields

Only syncs new/modified records
Uses cursor field (e.g., updated_at)
Efficient and fast

Example: Orders, customer updates, inventory changes

Change Data Capture (CDC)

When to use: Real-time requirements

Tracks all changes at source
Minimal latency
Requires source support

Example: Real-time order processing

Destination Connectors

Available Destinations

CRM360

destination-crm360Syncs customer and order data to CRM360 for marketing campaigns and customer engagement.

Shops

destination-shopsPushes consolidated product and inventory data to Shops platform for multi-channel selling.

Connector Development

Creating a New Connector

Generate Connector Scaffold

cd airbyte-connectors
lerna create source-new-platform sources/

Implement Source Interface

import { AirbyteSourceRunner } from '@airbyte/cdk'

export class SourceNewPlatform {
  async spec() {
    // Define configuration schema
  }

  async check(config) {
    // Test connection
  }

  async discover(config) {
    // Return available streams
  }

  async read(config, catalog, state) {
    // Extract data
  }
}

Add Stream Definitions

const streams = {
  orders: {
    name: 'orders',
    json_schema: OrderSchema,
    supported_sync_modes: ['full_refresh', 'incremental'],
    source_defined_cursor: true,
    default_cursor_field: ['updated_at']
  }
}

Write Tests

describe('SourceNewPlatform', () => {
  it('should connect successfully', async () => {
    const result = await source.check(config)
    expect(result.status).toBe('SUCCEEDED')
  })
})

Stream Configuration

Each stream defines:

Schema: JSON schema for data validation
Sync modes: Supported synchronization methods
Cursor field: Field for incremental sync
Primary key: Unique identifier(s)
Partitioning: How data is divided for sync

Testing Strategy

Test Levels

Unit Tests
Integration Tests
Acceptance Tests

Test individual functions and utilities:

API request formatting
Response parsing
Data transformation
Error handling

npm run test:unit -- source-bling

Test against real APIs (sandbox environment):

Connection verification
Stream discovery
Data extraction
Incremental sync logic

npm run test:integration -- source-bling

Airbyte standard tests for certification:

Spec validation
Connection check
Discovery validation
Full and incremental sync

npm run test:acceptance -- source-bling

Test Configuration

# acceptance-test-config.yml
connector_image: airbyte/source-bling:dev
tests:
  spec:
    - spec_path: "source_bling/spec.json"
  connection:
    - config_path: "secrets/config.json"
  discovery:
    - config_path: "secrets/config.json"
  basic_read:
    - config_path: "secrets/config.json"
      configured_catalog_path: "integration_tests/configured_catalog.json"

Error Handling

Retry Strategies

Transient Errors

Strategy: Exponential backoff with jitter

Network timeouts
Rate limit errors (429)
Server errors (5xx)

const retry = exponentialBackoff({
  maxRetries: 5,
  initialDelay: 1000,
  maxDelay: 30000
})

Permanent Errors

Strategy: Fail fast and log

Authentication failures (401)
Invalid configuration (400)
Resource not found (404)

Require user intervention to resolve.

Logging

Comprehensive logging for debugging:

Request/response logging (sanitized)
Sync progress and statistics
Error details with context
Performance metrics

Performance Optimization

Batching

Fetch multiple records per API call to reduce overhead

Pagination

Efficiently handle large datasets with cursor-based pagination

Parallel Streams

Sync independent streams concurrently

Caching

Cache API responses for repeated requests

Deployment

Building Connectors

# Build all connectors
lerna run build

# Build specific connector
lerna run build --scope=source-bling

Docker Images

Connectors are packaged as Docker images:

FROM airbyte/integration-base:latest

COPY source-bling /airbyte/integration_code
RUN npm install

ENTRYPOINT ["node", "/airbyte/integration_code/main.js"]

Airbyte Cloud

Connectors can be deployed to:

Airbyte Cloud: Hosted Airbyte service
Self-hosted: On-premise Airbyte instance
Kubernetes: Scalable container orchestration

Monitoring

Sync Metrics

Track connector performance:

Sync duration: Time to complete sync
Records synced: Number of records extracted
Data volume: Bytes transferred
Error rate: Failed syncs percentage

Health Checks

Automated monitoring:

Connection health (daily checks)
API quota usage
Sync schedule adherence
Data freshness alerts

Next Steps

Architecture

Understand connector architecture

Development Guide

Build custom connectors

Testing Guide

Test and validate connectors

Source Connectors

Explore all available sources

​Overview

​Technology Stack

Airbyte CDK

TypeScript

Lerna

Jest

​Connector Architecture

​Monorepo Structure

​Source Connectors

​Available Sources

​Sync Modes

​Destination Connectors

​Available Destinations

CRM360

Shops

​Connector Development

​Creating a New Connector

​Stream Configuration

​Testing Strategy

​Test Levels

​Test Configuration

​Error Handling

​Retry Strategies

​Logging

​Performance Optimization

Batching

Pagination

Parallel Streams

Caching

​Deployment

​Building Connectors

​Docker Images

​Airbyte Cloud

​Monitoring

​Sync Metrics

​Health Checks

​Next Steps

Architecture

Development Guide

Testing Guide

Source Connectors

Overview

Technology Stack

Connector Architecture

Monorepo Structure

Source Connectors

Available Sources

Sync Modes

Destination Connectors

Available Destinations

Connector Development

Creating a New Connector

Stream Configuration

Testing Strategy

Test Levels

Test Configuration

Error Handling

Retry Strategies

Logging

Performance Optimization

Deployment

Building Connectors

Docker Images

Airbyte Cloud

Monitoring

Sync Metrics

Health Checks

Next Steps