System Architecture
Trendteller is designed as a modern, cloud-native multi-part application that processes e-commerce data at scale. The architecture follows industry best practices for data pipelines, API design, and frontend development.High-Level Architecture
Architectural Principles
Separation of Concerns
Each component has a single, well-defined responsibility
Cloud-Native
Built for cloud deployment with serverless and managed services
API-First
All data access through well-defined GraphQL API
Type Safety
TypeScript throughout for compile-time error prevention
Component Breakdown
Data Ingestion Layer
Technology: Airbyte CDK + TypeScript Purpose: Extract data from e-commerce platforms- Custom connectors for 9 platforms
- Incremental and full-refresh sync modes
- Error handling and retry logic
- Rate limiting and API quota management
Storage Layer
Technology: Google BigQuery Purpose: Scalable data warehouse for all analytics data- Serverless, auto-scaling compute
- Columnar storage for fast queries
- Partitioning and clustering for optimization
- Multi-region replication for disaster recovery
Transformation Layer
Technology: Dataform + SQLX Purpose: Transform raw data into analytics-ready tables- Medallion architecture (Bronze/Silver/Gold)
- Git-based version control for SQL
- Dependency management and orchestration
- Data quality assertions
API Layer
Technology: Hasura GraphQL Engine Purpose: Unified API for all data access- Auto-generated GraphQL schema from database
- Role-based access control
- Query optimization and caching
- Real-time subscriptions
Application Layer
Technology: Nuxt 3 + Vue 3 + TypeScript Purpose: User-facing analytics dashboard- Server-side rendering for performance
- Type-safe GraphQL with Genql
- Component-based architecture
- Progressive web app (PWA) capabilities
Automation Layer
Technology: Kestra + Node.js Purpose: AI-powered insights and automation- Scheduled and event-driven workflows
- OpenAI and Gemini AI integration
- Web crawling with Puppeteer
- Automated reporting and alerting
Design Patterns
Medallion Architecture
Three-layer data architecture for data quality and performance:1
Bronze Layer
Raw data as ingested, no transformations
2
Silver Layer
Cleansed, standardized, and validated data
3
Gold Layer
Business-ready aggregations and analytics
Event-Driven Architecture
Kestra workflows react to events:- New data arrivals trigger transformations
- Data quality failures trigger alerts
- Threshold breaches trigger AI analysis
Microservices Pattern
Loosely coupled services:- Each component can scale independently
- Failures are isolated to individual services
- Technology choices optimized per service
Security Architecture
Authentication
Authentication
- Auth0 for user authentication
- OAuth 2.0 / OpenID Connect
- Social login support (Google, Microsoft)
- Multi-factor authentication (MFA)
Authorization
Authorization
Data Security
Data Security
- Encryption at rest (BigQuery, Postgres)
- Encryption in transit (TLS 1.3)
- API keys stored in Google Secret Manager
- Regular security audits and penetration testing
Compliance
Compliance
- LGPD compliance (Brazilian data protection)
- GDPR-ready data handling
- Data retention policies
- Audit logging for all data access
Scalability
Horizontal Scaling
Components scale horizontally as needed:| Component | Scaling Method | Limits |
|---|---|---|
| BigQuery | Auto-scaling | Virtually unlimited |
| Hasura | Kubernetes HPA | 10+ instances |
| Frontend (Netlify) | Edge distribution | Global CDN |
| Airbyte | Docker Compose | Per-connector scaling |
| Kestra | Kubernetes | Workflow parallelism |
Performance Optimization
Query Optimization
- BigQuery clustering and partitioning
- Hasura query caching
- Frontend data prefetching
- API response compression
Cost Optimization
- BigQuery query cost monitoring
- Incremental data processing
- Materialized views for common queries
- API rate limiting
Disaster Recovery
Backup Strategy
- BigQuery: Automatic snapshots (7-day retention)
- PostgreSQL: Daily backups to Google Cloud Storage
- Code: Git repositories with protected branches
- Configuration: Infrastructure as Code (Terraform)
Recovery Time Objectives (RTO)
| Component | RTO | RPO |
|---|---|---|
| Frontend | < 5 minutes | 0 (stateless) |
| API (Hasura) | < 15 minutes | < 5 minutes |
| BigQuery Data | < 1 hour | < 1 day |
| PostgreSQL | < 30 minutes | < 1 hour |
Monitoring & Observability
- Application Monitoring
- Infrastructure Monitoring
- Business Monitoring
- Frontend: Netlify Analytics + Sentry
- API: Hasura Cloud monitoring
- Workflows: Kestra execution logs
Technology Decisions
Why TypeScript?
- Strong typing prevents runtime errors
- Excellent IDE support and refactoring
- Growing ecosystem and community
- Seamless JavaScript interop
Why BigQuery?
- Serverless (no infrastructure management)
- Cost-effective for analytics workloads
- Fast query performance at scale
- Native integration with Google Cloud
Why Hasura?
- Auto-generates GraphQL from database schema
- Real-time capabilities out of the box
- Fine-grained permissions engine
- Performance optimization built-in
Why Nuxt 3?
- Server-side rendering for SEO and performance
- Auto-imports and file-based routing
- Strong TypeScript support
- Large ecosystem of modules