Mandatum

Appearance

Analytics ​

Track costs, usage, and performance with Mandatum's built-in analytics.

Dashboard Overview ​

The analytics dashboard provides insights into:

  • Cost tracking: Token usage and estimated costs
  • Usage patterns: Requests per day/week/month
  • Performance metrics: Latency and response times
  • Error rates: Success vs. failure rates
  • Top prompts: Most-used prompts and versions

Analytics Dashboard

Cost Tracking ​

Overview ​

View total costs across all prompts:

python
from mandatum import Mandatum

client = Mandatum(api_key="your-api-key")

# Get cost summary
costs = client.analytics.get_costs(
    start_date="2025-01-01",
    end_date="2025-01-31"
)

print(f"Total cost: ${costs.total_usd:.2f}")
print(f"Total tokens: {costs.total_tokens:,}")

By Prompt ​

Break down costs by prompt:

python
# Cost per prompt
prompt_costs = client.analytics.get_costs(
    group_by="prompt_name",
    start_date="2025-01-01",
    end_date="2025-01-31"
)

for prompt in prompt_costs:
    print(f"{prompt.name}: ${prompt.cost_usd:.2f}")

By User ​

Track costs per user:

python
# Cost per user (requires metadata with user_id)
user_costs = client.analytics.get_costs(
    group_by="metadata.user_id",
    start_date="2025-01-01",
    end_date="2025-01-31"
)

for user in user_costs:
    print(f"User {user.user_id}: ${user.cost_usd:.2f}")

By Model ​

Compare costs across models:

python
# Cost per model
model_costs = client.analytics.get_costs(
    group_by="model",
    start_date="2025-01-01",
    end_date="2025-01-31"
)

for model in model_costs:
    print(f"{model.name}: ${model.cost_usd:.2f} ({model.request_count} requests)")

View cost trends over time:

python
# Daily cost breakdown
daily_costs = client.analytics.get_costs(
    group_by="date",
    granularity="day",
    start_date="2025-01-01",
    end_date="2025-01-31"
)

for day in daily_costs:
    print(f"{day.date}: ${day.cost_usd:.2f}")

Usage Analytics ​

Request Volume ​

Track request volume:

python
# Total requests
usage = client.analytics.get_usage(
    start_date="2025-01-01",
    end_date="2025-01-31"
)

print(f"Total requests: {usage.total_requests:,}")
print(f"Successful: {usage.success_count:,} ({usage.success_rate:.1%})")
print(f"Failed: {usage.error_count:,} ({usage.error_rate:.1%})")

Top Prompts ​

Find most-used prompts:

python
# Top 10 prompts by usage
top_prompts = client.analytics.get_usage(
    group_by="prompt_name",
    order_by="request_count",
    limit=10
)

for i, prompt in enumerate(top_prompts, 1):
    print(f"{i}. {prompt.name}: {prompt.request_count:,} requests")

Usage Patterns ​

Analyze usage by hour/day:

python
# Hourly usage pattern
hourly_usage = client.analytics.get_usage(
    group_by="hour_of_day",
    start_date="2025-01-01",
    end_date="2025-01-31"
)

for hour in hourly_usage:
    print(f"{hour.hour}:00 - {hour.request_count} requests")

Performance Metrics ​

Latency ​

Track request latency:

python
# Latency stats
latency = client.analytics.get_latency(
    start_date="2025-01-01",
    end_date="2025-01-31"
)

print(f"Average: {latency.avg_ms:.0f}ms")
print(f"P50: {latency.p50_ms:.0f}ms")
print(f"P95: {latency.p95_ms:.0f}ms")
print(f"P99: {latency.p99_ms:.0f}ms")

By Prompt ​

Compare performance across prompts:

python
# Latency per prompt
prompt_latency = client.analytics.get_latency(
    group_by="prompt_name"
)

for prompt in sorted(prompt_latency, key=lambda p: p.avg_ms, reverse=True):
    print(f"{prompt.name}: {prompt.avg_ms:.0f}ms avg, {prompt.p95_ms:.0f}ms p95")

Integration Performance ​

Track integration code execution time:

python
# Integration execution time
integration_perf = client.analytics.get_integration_performance(
    prompt_name="customer-classifier"
)

print(f"Pre-process: {integration_perf.pre_process_avg_ms:.0f}ms avg")
print(f"Post-process: {integration_perf.post_process_avg_ms:.0f}ms avg")

Error Tracking ​

Error Rates ​

Monitor error rates:

python
# Error rate over time
errors = client.analytics.get_errors(
    group_by="date",
    granularity="day",
    start_date="2025-01-01",
    end_date="2025-01-31"
)

for day in errors:
    print(f"{day.date}: {day.error_count} errors ({day.error_rate:.1%})")

Error Types ​

Break down errors by type:

python
# Error types
error_types = client.analytics.get_errors(
    group_by="error_type"
)

for error in error_types:
    print(f"{error.type}: {error.count} occurrences")

Most Failing Prompts ​

Find problematic prompts:

python
# Prompts with highest error rates
failing_prompts = client.analytics.get_errors(
    group_by="prompt_name",
    order_by="error_rate",
    limit=10
)

for prompt in failing_prompts:
    print(f"{prompt.name}: {prompt.error_rate:.1%} error rate")

Custom Reports ​

Build Custom Reports ​

Combine multiple metrics:

python
# Custom report
report = client.analytics.build_report(
    metrics=["cost", "usage", "latency", "errors"],
    group_by="prompt_name",
    filters={
        "metadata.environment": "production"
    },
    start_date="2025-01-01",
    end_date="2025-01-31"
)

for prompt in report:
    print(f"{prompt.name}:")
    print(f"  Requests: {prompt.request_count:,}")
    print(f"  Cost: ${prompt.cost_usd:.2f}")
    print(f"  Avg latency: {prompt.avg_latency_ms:.0f}ms")
    print(f"  Error rate: {prompt.error_rate:.1%}")

Scheduled Reports ​

Email reports automatically:

python
# Create scheduled report
scheduled_report = client.analytics.create_scheduled_report(
    name="Weekly Production Report",
    metrics=["cost", "usage", "latency", "errors"],
    frequency="weekly",
    recipients=["team@company.com"],
    filters={
        "metadata.environment": "production"
    }
)

Alerts ​

Cost Alerts ​

Get notified when costs exceed thresholds:

python
# Create cost alert
alert = client.analytics.create_alert(
    name="High Daily Cost",
    metric="cost",
    threshold=100.00,  # $100/day
    period="1d",
    notification_channels=["email", "slack"]
)

Error Rate Alerts ​

Monitor error rates:

python
# Create error rate alert
alert = client.analytics.create_alert(
    name="High Error Rate",
    metric="error_rate",
    threshold=0.05,  # 5% error rate
    period="1h",
    filters={
        "prompt_name": "customer-classifier"
    },
    notification_channels=["email", "pagerduty"]
)

Latency Alerts ​

Track performance degradation:

python
# Create latency alert
alert = client.analytics.create_alert(
    name="Slow Responses",
    metric="latency_p95",
    threshold=5000,  # 5 seconds
    period="5m",
    notification_channels=["slack"]
)

Visualization ​

Charts ​

View charts in the dashboard:

  • Cost over time: Line chart showing daily/weekly/monthly costs
  • Usage trends: Bar chart of request volume
  • Latency distribution: Histogram of response times
  • Top prompts: Bar chart of most-used prompts
  • Error rates: Line chart of error trends

Analytics Charts

Export Data ​

Export analytics data for external visualization:

python
# Export to CSV
client.analytics.export(
    format="csv",
    output_file="analytics_jan_2025.csv",
    metrics=["cost", "usage", "latency"],
    start_date="2025-01-01",
    end_date="2025-01-31"
)

# Use with pandas
import pandas as pd

df = pd.read_csv("analytics_jan_2025.csv")
df.plot(x="date", y="cost_usd")

Integrations ​

Datadog ​

Send metrics to Datadog:

python
# Enable Datadog integration
client.integrations.enable_datadog(
    api_key="your-datadog-api-key",
    metrics=[
        "mandatum.requests.count",
        "mandatum.requests.latency",
        "mandatum.requests.cost",
        "mandatum.requests.errors"
    ]
)

Grafana ​

Query Mandatum metrics in Grafana:

python
# Enable Grafana integration
client.integrations.enable_grafana(
    prometheus_endpoint="https://prometheus.your-company.com"
)

Custom Webhooks ​

Send analytics data to custom endpoints:

python
# Create analytics webhook
webhook = client.webhooks.create(
    url="https://your-app.com/analytics",
    events=["analytics.daily_summary"],
    schedule="0 0 * * *"  # Daily at midnight
)

Best Practices ​

Tag Everything ​

Use consistent metadata for better analytics:

python
response = client.prompts.run(
    prompt_name="classifier",
    input_variables={...},
    metadata={
        "environment": "production",
        "user_id": "user_123",
        "team": "customer-support",
        "version": "v2.1.0"
    }
)

Set Budgets ​

Configure cost budgets:

python
# Set monthly budget
client.settings.set_budget(
    amount=1000.00,  # $1000/month
    period="monthly",
    alert_thresholds=[0.5, 0.8, 1.0]  # Alert at 50%, 80%, 100%
)

Track trends over time:

python
# Compare month-over-month
current_month = client.analytics.get_costs(
    start_date="2025-02-01",
    end_date="2025-02-28"
)

previous_month = client.analytics.get_costs(
    start_date="2025-01-01",
    end_date="2025-01-31"
)

growth = (current_month.total_usd - previous_month.total_usd) / previous_month.total_usd
print(f"Month-over-month growth: {growth:.1%}")

Optimize Costs ​

Identify optimization opportunities:

python
# Find expensive prompts
expensive_prompts = client.analytics.get_costs(
    group_by="prompt_name",
    order_by="cost_usd",
    limit=10
)

# Check if cheaper models could work
for prompt in expensive_prompts:
    print(f"{prompt.name}:")
    print(f"  Current model: {prompt.model}")
    print(f"  Cost: ${prompt.cost_usd:.2f}")
    print(f"  Avg tokens: {prompt.avg_tokens:.0f}")
    print(f"  Consider: gpt-3.5-turbo for simpler tasks")

API Reference ​

Full API reference for analytics:

Next Steps ​