> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trendteller.com.br/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Workflows

> Kestra automation scripts for AI insights and forecasting

## Overview

The AI Workflows component leverages Kestra orchestration to run automated scripts that generate insights, forecasts, and intelligence using AI models from OpenAI and Google Gemini.

## Technology Stack

<CardGroup cols={2}>
  <Card title="Node.js" icon="node">
    JavaScript runtime for script execution
  </Card>

  <Card title="OpenAI" icon="brain">
    GPT models for text generation and analysis
  </Card>

  <Card title="Google Gemini" icon="sparkles">
    Advanced AI for multi-modal insights
  </Card>

  <Card title="Puppeteer" icon="chrome">
    Web automation and crawling
  </Card>
</CardGroup>

## Key Capabilities

### AI-Powered Insights

<Tabs>
  <Tab title="Sales Forecasting">
    Generate predictions for future sales using historical data and AI models:

    * **Time Series Analysis**: Trend identification and seasonality
    * **Multi-variate Predictions**: Factor in marketing, inventory, pricing
    * **Confidence Intervals**: Probabilistic forecasts with uncertainty
    * **Brand-specific Models**: Customized for each brand's patterns

    **Output**: Daily/weekly/monthly forecast reports
  </Tab>

  <Tab title="Trend Detection">
    Identify emerging trends and patterns in e-commerce data:

    * **Product Trends**: Rising/declining products and categories
    * **Customer Behavior**: Shifts in purchasing patterns
    * **Market Dynamics**: Competitive landscape changes
    * **Anomaly Detection**: Unusual sales patterns or inventory issues

    **Output**: Weekly trend reports with AI commentary
  </Tab>

  <Tab title="Automated Reporting">
    Generate natural language reports from data:

    * **Executive Summaries**: High-level KPIs with narrative
    * **Performance Analysis**: Deep dives into metrics
    * **Exception Reports**: Automated alerts for critical issues
    * **Comparative Analysis**: Brand-to-brand benchmarking

    **Output**: Daily/weekly automated email reports
  </Tab>
</Tabs>

### Web Crawling & Intelligence

<AccordionGroup>
  <Accordion title="Competitive Pricing">
    Monitor competitor prices across the web:

    * Automated price scraping from competitor sites
    * Price change detection and alerting
    * Historical price tracking
    * Margin impact analysis

    **Frequency**: Daily crawls for key products
  </Accordion>

  <Accordion title="Market Availability">
    Track product availability across channels:

    * Stock status monitoring
    * Out-of-stock duration tracking
    * Multi-marketplace availability
    * Fulfillment speed comparison

    **Use Case**: Identify stock-out opportunities
  </Accordion>

  <Accordion title="Review Analysis">
    Aggregate and analyze customer reviews:

    * Sentiment analysis across platforms
    * Feature extraction from reviews
    * Competitive review comparison
    * Product improvement insights

    **AI Models**: GPT-4 for sentiment, Gemini for summarization
  </Accordion>
</AccordionGroup>

## Script Architecture

### Kestra Workflows

Scripts are organized as Kestra workflows:

```yaml theme={null}
id: daily-sales-forecast
namespace: trendteller.insights

tasks:
  - id: fetch-data
    type: io.kestra.plugin.scripts.node.Script
    script: |
      const data = await fetchSalesData()
      return { data }

  - id: generate-forecast
    type: io.kestra.plugin.scripts.node.Script
    script: |
      const forecast = await openai.forecast(data)
      return { forecast }

  - id: save-results
    type: io.kestra.plugin.gcp.bigquery.Query
    sql: INSERT INTO forecasts ...
```

### Script Organization

```
kestra-scripts/
├── insights/
│   ├── sales-forecast.js      # Sales predictions
│   ├── trend-analysis.js      # Trend detection
│   └── anomaly-detection.js   # Outlier identification
├── crawlers/
│   ├── price-monitor.js       # Competitive pricing
│   ├── availability-check.js  # Stock monitoring
│   └── review-scraper.js      # Customer reviews
├── reports/
│   ├── daily-summary.js       # Daily reports
│   ├── weekly-analysis.js     # Weekly deep dive
│   └── executive-dashboard.js # Executive KPIs
└── shared/
    ├── bigquery-client.js     # BigQuery utilities
    ├── ai-helpers.js          # AI model wrappers
    └── email-sender.js        # Report distribution
```

## AI Integration

### OpenAI Integration

<Steps>
  <Step title="Initialize Client">
    ```javascript theme={null}
    import OpenAI from 'openai'

    const openai = new OpenAI({
      apiKey: process.env.OPENAI_API_KEY
    })
    ```
  </Step>

  <Step title="Generate Insights">
    ```javascript theme={null}
    const completion = await openai.chat.completions.create({
      model: "gpt-4-turbo",
      messages: [
        {
          role: "system",
          content: "You are a data analyst specializing in e-commerce."
        },
        {
          role: "user",
          content: `Analyze this sales data and provide insights: ${data}`
        }
      ]
    })
    ```
  </Step>

  <Step title="Process Results">
    ```javascript theme={null}
    const insights = completion.choices[0].message.content
    await saveToBigQuery(insights)
    await sendEmailReport(insights)
    ```
  </Step>
</Steps>

### Google Gemini Integration

```javascript theme={null}
import { GoogleGenerativeAI } from '@google/generative-ai'

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY)
const model = genAI.getGenerativeModel({ model: "gemini-pro" })

// Generate multi-modal insights
const result = await model.generateContent([
  "Analyze this sales trend chart and provide insights",
  { inlineData: { data: chartImageBase64, mimeType: "image/png" } }
])
```

<Tip>
  Gemini's multi-modal capabilities allow analysis of charts, images, and text together for richer insights.
</Tip>

## Data Access

### BigQuery Integration

Scripts access BigQuery for historical data:

```javascript theme={null}
import { BigQuery } from '@google-cloud/bigquery'

const bigquery = new BigQuery({
  projectId: 'togo-425319',
  keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS
})

// Query sales data
const query = `
  SELECT date, brand_id, total_revenue
  FROM \`togo-425319.gold.daily_sales\`
  WHERE date >= DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY)
  ORDER BY date
`

const [rows] = await bigquery.query(query)
```

### PostgreSQL Access

For operational data stored in Postgres:

```javascript theme={null}
import { Client } from 'pg'

const client = new Client({
  host: process.env.PG_HOST,
  database: 'trendteller',
  user: process.env.PG_USER,
  password: process.env.PG_PASSWORD
})

await client.connect()
const result = await client.query('SELECT * FROM users WHERE active = true')
```

## Web Crawling

### Puppeteer Automation

```javascript theme={null}
import puppeteer from 'puppeteer'

const browser = await puppeteer.launch({ headless: true })
const page = await browser.newPage()

// Navigate to competitor site
await page.goto('https://competitor.com/product/12345')

// Extract price
const price = await page.$eval('.product-price', el => el.textContent)

// Take screenshot for verification
await page.screenshot({ path: `screenshots/${productId}.png` })

await browser.close()
```

<Warning>
  Always respect robots.txt and implement rate limiting to avoid overloading target sites.
</Warning>

### Anti-Bot Evasion

```javascript theme={null}
// Use stealth plugin to avoid detection
import puppeteer from 'puppeteer-extra'
import StealthPlugin from 'puppeteer-extra-plugin-stealth'

puppeteer.use(StealthPlugin())

// Random delays to mimic human behavior
await page.waitForTimeout(Math.random() * 2000 + 1000)

// Rotate user agents
const userAgents = [/* ... */]
await page.setUserAgent(userAgents[Math.floor(Math.random() * userAgents.length)])
```

## Scheduling & Orchestration

### Kestra Schedules

<Tabs>
  <Tab title="Daily Jobs">
    Run every day at specific times:

    ```yaml theme={null}
    triggers:
      - id: daily
        type: io.kestra.core.models.triggers.types.Schedule
        cron: "0 6 * * *"  # 6 AM daily
    ```

    * Sales forecasts (6 AM)
    * Daily summary reports (7 AM)
    * Price monitoring (8 AM)
  </Tab>

  <Tab title="Weekly Jobs">
    Run once per week:

    ```yaml theme={null}
    triggers:
      - id: weekly
        type: io.kestra.core.models.triggers.types.Schedule
        cron: "0 8 * * 1"  # Monday 8 AM
    ```

    * Weekly performance analysis
    * Trend reports
    * Competitive intelligence summary
  </Tab>

  <Tab title="On-Demand">
    Triggered manually or via API:

    ```yaml theme={null}
    triggers:
      - id: manual
        type: io.kestra.core.models.triggers.types.Webhook
    ```

    * Custom date range forecasts
    * Ad-hoc competitor analysis
    * Emergency reporting
  </Tab>
</Tabs>

## Output & Distribution

### Report Generation

AI-generated reports are distributed via:

<CardGroup cols={2}>
  <Card title="Email" icon="envelope">
    Automated email delivery using SendGrid or AWS SES
  </Card>

  <Card title="Dashboard" icon="chart-line">
    Reports saved to BigQuery and displayed in frontend
  </Card>

  <Card title="Slack" icon="slack">
    Critical alerts and summaries posted to Slack
  </Card>

  <Card title="API" icon="code">
    Insights exposed via REST API for integrations
  </Card>
</CardGroup>

### Storage

* **Reports**: Stored in BigQuery `insights` dataset
* **Forecasts**: Saved to `forecasts` table with confidence intervals
* **Screenshots**: Uploaded to Google Cloud Storage
* **Logs**: Kestra execution logs retained for 90 days

## Monitoring & Alerting

### Execution Monitoring

Track workflow execution:

* **Success rate**: Percentage of successful runs
* **Duration**: Average and P95 execution time
* **Resource usage**: CPU, memory, API credits
* **Error rates**: Failed tasks and reasons

### Alerts

Automated alerts for:

* Workflow failures or timeouts
* API quota exceeded (OpenAI, Gemini)
* Anomalous insights detected
* Price changes exceeding thresholds

## Cost Management

### AI API Costs

<Info>
  Monthly AI API spending is monitored and budgeted:

  * **OpenAI**: \~\$500/month (GPT-4 Turbo)
  * **Google Gemini**: \~\$200/month (Gemini Pro)
  * **Total**: \~\$700/month for AI services
</Info>

### Optimization Strategies

* Use GPT-3.5 for simple tasks
* Batch API requests when possible
* Cache AI responses for repeated queries
* Implement token limits per request

## Next Steps

<CardGroup cols={2}>
  <Card title="Architecture" icon="sitemap" href="/architecture/workflows/architecture">
    Explore workflow architecture
  </Card>

  <Card title="AI Services" icon="brain" href="/architecture/workflows/ai-services">
    Learn about AI model integration
  </Card>

  <Card title="Development Guide" icon="code" href="/development/scripts/guide">
    Build custom automation scripts
  </Card>

  <Card title="Forecasting Engine" icon="chart-line" href="/architecture/workflows/forecasting-engine">
    Understand forecasting algorithms
  </Card>
</CardGroup>