Data Connections

Connect DataCrate to external databases, APIs, and storage systems to sync and manage test data.

info

Coming Soon This feature is currently in development.

Supported Connections

Databases

Database	Type	Features
PostgreSQL	Relational	Full CRUD, schemas
MySQL	Relational	Full CRUD, views
SQL Server	Relational	Full CRUD, stored procs
MongoDB	Document	Collections, aggregation
Redis	Key-Value	Cache, session data
Elasticsearch	Search	Full-text search

Cloud Storage

Service	Features
AWS S3	File storage, versioning
Azure Blob	Container management
Google Cloud Storage	Bucket operations
MinIO	S3-compatible

APIs

Type	Authentication
REST	API Key, OAuth, JWT
GraphQL	Bearer token
SOAP	WS-Security

Creating Connections

Database Connection

Connection:
  name: Production Clone
  type: postgresql
  host: db.example.com
  port: 5432
  database: test_db
  username: ${DB_USER}
  password: ${DB_PASSWORD}
  ssl: required
  pool_size: 10

API Connection

Connection:
  name: User Service
  type: rest_api
  base_url: https://api.example.com
  auth:
    type: oauth2
    client_id: ${CLIENT_ID}
    client_secret: ${CLIENT_SECRET}
  headers:
    X-API-Version: '2.0'

Cloud Storage Connection

Connection:
  name: Test Data Bucket
  type: aws_s3
  region: us-east-1
  bucket: test-data-bucket
  credentials:
    access_key: ${AWS_ACCESS_KEY}
    secret_key: ${AWS_SECRET_KEY}

Connection Operations

Sync Data

Direction	Description
Import	Pull data from source
Export	Push data to target
Two-Way	Bidirectional sync

Sync Configuration

Sync:
  source: production_db
  target: test_dataset
  table: users

  options:
    mode: incremental # full, incremental
    key: updated_at
    batch_size: 1000

  transform:
    - mask: [email, phone]
    - anonymize: [name]
    - exclude: [password, ssn]

Query Execution

Run queries against connected databases:

-- Read data
SELECT * FROM users WHERE status = 'active' LIMIT 100

-- Parameterized queries
SELECT * FROM orders
WHERE user_id = :userId
AND created_at > :startDate

Data Masking

Masking Rules

Protect sensitive data during sync:

Field Type	Masking Method
Email	john@*** .com
Phone	--1234
SSN	*--6789
Credit Card	**-**-****-1234
Name	J*** D**

Configuration

Masking:
  email:
    method: partial
    show_first: 3
    show_domain: true

  phone:
    method: partial
    show_last: 4

  ssn:
    method: hash
    algorithm: sha256

  name:
    method: fake
    generator: faker.name()

Scheduling

Sync Schedules

Frequency	Use Case
Real-time	Critical test data
Hourly	Frequent updates
Daily	Regular refresh
Weekly	Static datasets
On-demand	Manual trigger

Schedule Configuration

Schedule:
  name: Nightly User Sync
  connection: production_db
  cron: '0 2 * * *' # 2 AM daily

  sync:
    tables: [users, profiles]
    mode: incremental

  notifications:
    on_success: slack
    on_failure: email

Security

Credentials Management

Storage	Security Level
Vault	Enterprise secrets
Env Variables	Environment-based
Encrypted Config	File-based secrets

Access Control

Connection:
  name: Production Data

  permissions:
    admin: full
    developer: read, sync
    tester: read

  restrictions:
    tables:
      - users: read_only
      - payments: masked
      - audit_logs: deny

Audit Logging

Track all connection activities:

• Connection attempts
• Queries executed
• Data synced
• Errors and failures

Monitoring

Connection Health

Metric	Description
Status	Connected/Disconnected
Latency	Response time
Throughput	Records per second
Errors	Failed operations

Alerts

Configure alerts for:

• Connection failures
• Sync errors
• Performance degradation
• Security events

Best Practices

Security

• Use environment variables for credentials
• Enable SSL/TLS connections
• Implement least-privilege access
• Audit connection usage

Performance

• Use connection pooling
• Optimize query performance
• Schedule syncs during off-hours
• Monitor resource usage

Maintenance

• Regularly test connections
• Update credentials before expiry
• Document connection purposes
• Review access permissions