Operations

Webhooks, OpenTelemetry, status monitoring, config management, diagnostics.

Doctor

Run stockyard doctor before first start to verify your environment.

$ stockyard doctor

  ✓ Go runtime         go1.22.5
  ✓ Config             port=4200
  ✓ Data directory     ./stockyard-data (writable)
  ✓ SQLite database    connected (76 modules)
  ✓ Port 4200          available
  ✓ OpenAI API key     set
  ✓ Anthropic API key  set
  ⚠ Gemini API key     not set
  ✓ Ollama             detected (localhost:11434)
  ⚠ License key        not set (free tier)
  ✓ Disk space         writable

  9 passed · 2 warnings · 0 failures

Use stockyard doctor --json for machine-readable output in CI pipelines.

Webhooks

Register HTTP endpoints to receive events from Stockyard: alerts firing, cost thresholds, trust policy violations, and more.

# Register a Slack webhook
curl -X POST localhost:4200/api/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://hooks.slack.com/services/YOUR/SLACK/URL",
    "secret": "my-signing-secret",
    "events": "alert.fired,cost.threshold,trust.violation"
  }'

# List registered webhooks
curl localhost:4200/api/webhooks

# Send a test event
curl -X POST localhost:4200/api/webhooks/test

# Delete a webhook
curl -X DELETE localhost:4200/api/webhooks/1

Event Types

HMAC Signatures

If you set a secret, every delivery includes an X-Stockyard-Signature header with sha256=HMAC of the JSON payload. Verify this on your server to confirm the webhook came from Stockyard.

Auto-disable

Webhooks that fail 10 consecutive deliveries are automatically disabled. Check fail_count in the list response.

OpenTelemetry Export

Export every proxy request as an OTLP trace span. Works with Jaeger, Grafana Tempo, Datadog, Honeycomb, or any OTLP-compatible backend.

# Set the collector endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Optional: service name (default: "stockyard")
export OTEL_SERVICE_NAME=my-stockyard

# Optional: auth headers
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer my-token"

# Start Stockyard — traces flow automatically
stockyard

Span Attributes

Spans are batched (100 per batch, 5s flush interval) and flushed gracefully on shutdown.

Status Monitoring

Real-time system health at GET /api/status:

curl localhost:4200/api/status | jq .
{
  "status": "healthy",
  "uptime": "2d 5h 30m",
  "total_requests": 12453,
  "error_rate": 0.002,
  "avg_latency_ms": 245.3,
  "memory": { "alloc_mb": 8.2, "sys_mb": 24.1 },
  "goroutines": 12,
  "components": {
    "database": { "status": "healthy", "detail": "1.2ms" },
    "modules": { "status": "healthy", "detail": "76 registered" },
    "observe": { "status": "healthy", "detail": "12,453 traces" },
    "trust": { "status": "healthy", "detail": "12,453 events" }
  }
}

The web status page at /status/ auto-refreshes every 30 seconds.

Config Export & Import

Snapshot your entire Stockyard configuration and restore it on another instance.

# Export current config
curl localhost:4200/api/config/export > stockyard-config.json

# Preview changes before applying
curl -X POST localhost:4200/api/config/diff \
  -d @stockyard-config.json

# Import on another instance
curl -X POST localhost:4200/api/config/import \
  -d @stockyard-config.json

The export includes module states, webhooks, and trust policies. Provider API keys are never included in exports.

API Key Rotation

Rotate API keys without downtime — the old key is immediately revoked and a new one generated:

# Rotate your own key
curl -X POST localhost:4200/api/auth/me/keys/1/rotate \
  -H "Authorization: Bearer sk-sy-current-key"
# Returns: { "status": "rotated", "new_key": { "key": "sk-sy-new-...", ... } }

# Admin: rotate any user's key
curl -X POST localhost:4200/api/auth/users/1/keys/1/rotate

Config Export & Import

Export the full system configuration as a snapshot:

curl http://localhost:4200/api/config/export \
  -H "Authorization: Bearer sy_admin_..." \
  -o stockyard-config.json

Import a config snapshot to a different instance:

curl -X POST http://localhost:4200/api/config/import \
  -H "Authorization: Bearer sy_admin_..." \
  -H "Content-Type: application/json" \
  -d @stockyard-config.json

Config Diff

Before importing, diff the snapshot against the current state:

curl -X POST http://localhost:4200/api/config/diff \
  -H "Authorization: Bearer sy_admin_..." \
  -H "Content-Type: application/json" \
  -d @stockyard-config.json

{
  "changes": [
    {"path": "modules.cachelayer.enabled", "current": false, "incoming": true},
    {"path": "modules.costcap.config.daily_limit_usd", "current": 10, "incoming": 50}
  ],
  "additions": 2,
  "removals": 0,
  "modifications": 2
}

Webhooks

Register webhooks to receive real-time notifications:

curl -X POST http://localhost:4200/api/webhooks \
  -H "Authorization: Bearer sy_admin_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://hooks.slack.com/services/...",
    "events": ["alert.triggered", "cost.threshold", "module.toggled"],
    "secret": "whsec_..."
  }'

Test your webhooks:

curl -X POST http://localhost:4200/api/webhooks/test \
  -H "Authorization: Bearer sy_admin_..."

Monitoring

Health endpoint for load balancers and uptime monitors:

curl http://localhost:4200/health
# {"status":"ok","uptime":"48h12m","version":"1.0.0"}

Detailed system status with component health:

curl http://localhost:4200/api/status \
  -H "Authorization: Bearer sy_admin_..."

{
  "status": "healthy",
  "uptime": "48h12m",
  "version": "1.0.0",
  "components": {
    "proxy": "ok",
    "observe": "ok",
    "trust": "ok",
    "studio": "ok",
    "forge": "ok",
    "exchange": "ok"
  },
  "database_size_mb": 42.5,
  "total_requests": 15240
}

Railway Deployment

Stockyard runs on Railway with zero configuration. Set environment variables in the Railway dashboard:

OPENAI_API_KEY=sk-...
STOCKYARD_ADMIN_KEY=sy_admin_...
PORT=4200

Backup & Recovery

The SQLite database contains all traces, costs, audit logs, and configs. Back it up regularly:

# Export config snapshot (modules, webhooks, policies)
curl http://localhost:4200/api/config/export \
  -H "Authorization: Bearer sy_admin_..." > backup.json

# Copy the SQLite database for full backup
cp /data/stockyard.db /backups/stockyard-$(date +%Y%m%d).db