> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trendteller.com.br/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Connectors

> Airbyte integrations for e-commerce platforms

## Overview

The Data Connectors component provides custom Airbyte connectors that extract data from 9 different e-commerce platforms, enabling Trendteller to aggregate data from 11 brands into a unified analytics platform.

## Technology Stack

<CardGroup cols={2}>
  <Card title="Airbyte CDK" icon="plug">
    Connector Development Kit for building integrations
  </Card>

  <Card title="TypeScript" icon="code">
    Type-safe connector development
  </Card>

  <Card title="Lerna" icon="cubes">
    Monorepo management for multiple connectors
  </Card>

  <Card title="Jest" icon="vial">
    Comprehensive testing framework
  </Card>
</CardGroup>

## Connector Architecture

### Monorepo Structure

The connectors are organized in a Lerna monorepo:

```
airbyte-connectors/
├── sources/
│   ├── source-bling/         # Bling ERP connector
│   ├── source-bling-v3/      # Bling API v3
│   ├── source-vnda/          # VNDA platform
│   ├── source-vnda-moda/     # VNDA fashion variant
│   ├── source-shoppub/       # Shoppub marketplace
│   ├── source-tiny/          # Tiny ERP
│   ├── source-tiny-v3/       # Tiny API v3
│   ├── source-microvix/      # Microvix retail
│   ├── source-braavo/        # Braavo e-commerce
│   ├── source-totvs-moda/    # Totvs fashion ERP
│   ├── source-varejo-online/ # Varejo Online
│   ├── source-google-shopping/ # Google Shopping
│   └── ...
├── destinations/
│   ├── destination-crm360/   # CRM360 integration
│   └── destination-shops/    # Shops destination
└── shared/
    └── common-utils/         # Shared utilities
```

## Source Connectors

### Available Sources

<Tabs>
  <Tab title="ERP Systems">
    **Bling** (`source-bling`, `source-bling-v3`)

    * Orders, products, customers, inventory
    * Invoices, payments, shipping
    * Incremental sync support

    **Tiny** (`source-tiny`, `source-tiny-v3`)

    * Complete ERP data extraction
    * Multi-entity support
    * Real-time inventory updates

    **Totvs Moda** (`source-totvs-moda`)

    * Fashion industry specific
    * Size/color variant handling
    * Collection management
  </Tab>

  <Tab title="E-commerce Platforms">
    **VNDA** (`source-vnda`, `source-vnda-moda`)

    * Orders and products
    * Customer data and analytics
    * Webhook integration

    **Shoppub** (`source-shoppub`)

    * Marketplace integration
    * Multi-channel orders
    * Fulfillment tracking

    **Braavo** (`source-braavo`)

    * E-commerce platform connector
    * Order and catalog sync
  </Tab>

  <Tab title="Other Integrations">
    **Microvix** (`source-microvix`)

    * Retail management system
    * POS integration
    * Inventory synchronization

    **Google Shopping** (`source-google-shopping`)

    * Product feed data
    * Shopping ads performance
    * Competitive pricing

    **Varejo Online** (`source-varejo-online`)

    * Online retail platform
    * Order management
  </Tab>
</Tabs>

### Sync Modes

<AccordionGroup>
  <Accordion title="Full Refresh">
    **When to use**: Small datasets, no incremental support

    * Replaces all existing data
    * Ensures consistency
    * Higher resource usage

    **Example**: Product catalogs, brand configurations
  </Accordion>

  <Accordion title="Incremental Sync">
    **When to use**: Large datasets with timestamp fields

    * Only syncs new/modified records
    * Uses cursor field (e.g., `updated_at`)
    * Efficient and fast

    **Example**: Orders, customer updates, inventory changes
  </Accordion>

  <Accordion title="Change Data Capture (CDC)">
    **When to use**: Real-time requirements

    * Tracks all changes at source
    * Minimal latency
    * Requires source support

    **Example**: Real-time order processing
  </Accordion>
</AccordionGroup>

## Destination Connectors

### Available Destinations

<CardGroup cols={2}>
  <Card title="CRM360" icon="address-book">
    **destination-crm360**

    Syncs customer and order data to CRM360 for marketing campaigns and customer engagement.
  </Card>

  <Card title="Shops" icon="store">
    **destination-shops**

    Pushes consolidated product and inventory data to Shops platform for multi-channel selling.
  </Card>
</CardGroup>

## Connector Development

### Creating a New Connector

<Steps>
  <Step title="Generate Connector Scaffold">
    ```bash theme={null}
    cd airbyte-connectors
    lerna create source-new-platform sources/
    ```
  </Step>

  <Step title="Implement Source Interface">
    ```typescript theme={null}
    import { AirbyteSourceRunner } from '@airbyte/cdk'

    export class SourceNewPlatform {
      async spec() {
        // Define configuration schema
      }

      async check(config) {
        // Test connection
      }

      async discover(config) {
        // Return available streams
      }

      async read(config, catalog, state) {
        // Extract data
      }
    }
    ```
  </Step>

  <Step title="Add Stream Definitions">
    ```typescript theme={null}
    const streams = {
      orders: {
        name: 'orders',
        json_schema: OrderSchema,
        supported_sync_modes: ['full_refresh', 'incremental'],
        source_defined_cursor: true,
        default_cursor_field: ['updated_at']
      }
    }
    ```
  </Step>

  <Step title="Write Tests">
    ```typescript theme={null}
    describe('SourceNewPlatform', () => {
      it('should connect successfully', async () => {
        const result = await source.check(config)
        expect(result.status).toBe('SUCCEEDED')
      })
    })
    ```
  </Step>
</Steps>

### Stream Configuration

Each stream defines:

* **Schema**: JSON schema for data validation
* **Sync modes**: Supported synchronization methods
* **Cursor field**: Field for incremental sync
* **Primary key**: Unique identifier(s)
* **Partitioning**: How data is divided for sync

## Testing Strategy

### Test Levels

<Tabs>
  <Tab title="Unit Tests">
    Test individual functions and utilities:

    * API request formatting
    * Response parsing
    * Data transformation
    * Error handling

    ```bash theme={null}
    npm run test:unit -- source-bling
    ```
  </Tab>

  <Tab title="Integration Tests">
    Test against real APIs (sandbox environment):

    * Connection verification
    * Stream discovery
    * Data extraction
    * Incremental sync logic

    ```bash theme={null}
    npm run test:integration -- source-bling
    ```
  </Tab>

  <Tab title="Acceptance Tests">
    Airbyte standard tests for certification:

    * Spec validation
    * Connection check
    * Discovery validation
    * Full and incremental sync

    ```bash theme={null}
    npm run test:acceptance -- source-bling
    ```
  </Tab>
</Tabs>

### Test Configuration

```yaml theme={null}
# acceptance-test-config.yml
connector_image: airbyte/source-bling:dev
tests:
  spec:
    - spec_path: "source_bling/spec.json"
  connection:
    - config_path: "secrets/config.json"
  discovery:
    - config_path: "secrets/config.json"
  basic_read:
    - config_path: "secrets/config.json"
      configured_catalog_path: "integration_tests/configured_catalog.json"
```

## Error Handling

### Retry Strategies

<AccordionGroup>
  <Accordion title="Transient Errors">
    **Strategy**: Exponential backoff with jitter

    * Network timeouts
    * Rate limit errors (429)
    * Server errors (5xx)

    ```typescript theme={null}
    const retry = exponentialBackoff({
      maxRetries: 5,
      initialDelay: 1000,
      maxDelay: 30000
    })
    ```
  </Accordion>

  <Accordion title="Permanent Errors">
    **Strategy**: Fail fast and log

    * Authentication failures (401)
    * Invalid configuration (400)
    * Resource not found (404)

    Require user intervention to resolve.
  </Accordion>
</AccordionGroup>

### Logging

Comprehensive logging for debugging:

* Request/response logging (sanitized)
* Sync progress and statistics
* Error details with context
* Performance metrics

## Performance Optimization

<CardGroup cols={2}>
  <Card title="Batching" icon="layer-group">
    Fetch multiple records per API call to reduce overhead
  </Card>

  <Card title="Pagination" icon="list">
    Efficiently handle large datasets with cursor-based pagination
  </Card>

  <Card title="Parallel Streams" icon="arrows-split-up-and-left">
    Sync independent streams concurrently
  </Card>

  <Card title="Caching" icon="database">
    Cache API responses for repeated requests
  </Card>
</CardGroup>

## Deployment

### Building Connectors

```bash theme={null}
# Build all connectors
lerna run build

# Build specific connector
lerna run build --scope=source-bling
```

### Docker Images

Connectors are packaged as Docker images:

```dockerfile theme={null}
FROM airbyte/integration-base:latest

COPY source-bling /airbyte/integration_code
RUN npm install

ENTRYPOINT ["node", "/airbyte/integration_code/main.js"]
```

### Airbyte Cloud

Connectors can be deployed to:

* **Airbyte Cloud**: Hosted Airbyte service
* **Self-hosted**: On-premise Airbyte instance
* **Kubernetes**: Scalable container orchestration

## Monitoring

### Sync Metrics

Track connector performance:

* **Sync duration**: Time to complete sync
* **Records synced**: Number of records extracted
* **Data volume**: Bytes transferred
* **Error rate**: Failed syncs percentage

### Health Checks

Automated monitoring:

* Connection health (daily checks)
* API quota usage
* Sync schedule adherence
* Data freshness alerts

## Next Steps

<CardGroup cols={2}>
  <Card title="Architecture" icon="sitemap" href="/architecture/connectors/architecture">
    Understand connector architecture
  </Card>

  <Card title="Development Guide" icon="code" href="/development/connectors/guide">
    Build custom connectors
  </Card>

  <Card title="Testing Guide" icon="vial" href="/development/connectors/testing">
    Test and validate connectors
  </Card>

  <Card title="Source Connectors" icon="plug" href="/architecture/connectors/source-connectors">
    Explore all available sources
  </Card>
</CardGroup>