Mandatum

Appearance

Request Logging ​

Mandatum automatically logs all LLM requests with full traceability and searchability.

What Gets Logged? ​

Every prompt execution is logged with:

  • Request details: Prompt name, version, input variables
  • Response data: LLM output, model used, provider
  • Timing: Latency, execution time for integrations
  • Costs: Token usage and estimated cost
  • Metadata: Custom metadata you attach
  • Context: User ID, session ID, source

Viewing Logs ​

Dashboard ​

Navigate to Logs in the sidebar to view all requests:

Request Logs Dashboard

Filter Options ​

Filter logs by:

  • Prompt name: See all requests for a specific prompt
  • Date range: Last hour, day, week, month, or custom
  • Status: Success, error, timeout
  • Metadata: Filter by user ID, session ID, etc.
  • Model: Filter by LLM provider or model
  • Cost: Filter by token usage or cost range

Search logs by content:

Search for: "billing issue"

This searches across:

  • Input variables
  • LLM outputs
  • Metadata values

Request Details ​

Click any request to view full details:

Overview ​

json
{
  "id": "req_abc123",
  "prompt_name": "customer-classifier",
  "version": 3,
  "status": "success",
  "created_at": "2025-01-15T10:30:00Z"
}

Input ​

json
{
  "message": "I need help with my billing",
  "user_id": "user_123"
}

Output ​

json
{
  "category": "billing",
  "confidence": 0.95
}

Metadata ​

json
{
  "user_id": "user_123",
  "session_id": "session_456",
  "source": "web_app",
  "environment": "production"
}

Performance ​

json
{
  "total_latency_ms": 1234,
  "pre_process_ms": 50,
  "llm_call_ms": 1100,
  "post_process_ms": 84
}

Costs ​

json
{
  "prompt_tokens": 150,
  "completion_tokens": 50,
  "total_tokens": 200,
  "estimated_cost_usd": 0.004
}

Adding Metadata ​

Attach metadata to track requests:

Python ​

python
from mandatum import Mandatum

client = Mandatum(api_key="your-api-key")

response = client.prompts.run(
    prompt_name="customer-classifier",
    input_variables={"message": "I need help"},
    metadata={
        "user_id": "user_123",
        "session_id": "session_456",
        "source": "web_app",
        "environment": "production",
        "feature_flag": "new_classifier_v2"
    }
)

TypeScript ​

Coming Soon

A TypeScript/JavaScript SDK is in development. For now, use the REST API directly.

Learn more about metadata best practices.

Querying Logs ​

Via API ​

Get Recent Logs ​

python
logs = client.logs.list(
    limit=100,
    offset=0,
    start_date="2025-01-01",
    end_date="2025-01-31"
)

for log in logs:
    print(f"{log.created_at}: {log.prompt_name} - {log.status}")

Filter by Prompt ​

python
logs = client.logs.list(
    prompt_name="customer-classifier",
    limit=50
)

Filter by Metadata ​

python
logs = client.logs.list(
    metadata={
        "user_id": "user_123",
        "environment": "production"
    }
)

Search Content ​

python
logs = client.logs.search(
    query="billing issue",
    limit=20
)

Via REST API ​

bash
# Get recent logs
curl -X GET "https://mandatum-api.gavelinivar.com/api/v1/logs?limit=100" \
  -H "Authorization: Bearer YOUR_API_KEY"

# Filter by prompt
curl -X GET "https://mandatum-api.gavelinivar.com/api/v1/logs?prompt_name=customer-classifier" \
  -H "Authorization: Bearer YOUR_API_KEY"

# Search
curl -X GET "https://mandatum-api.gavelinivar.com/api/v1/logs/search?q=billing" \
  -H "Authorization: Bearer YOUR_API_KEY"

Exporting Logs ​

CSV Export ​

Export logs to CSV for analysis:

python
# Export logs
client.logs.export(
    format="csv",
    output_file="logs_jan_2025.csv",
    start_date="2025-01-01",
    end_date="2025-01-31"
)

JSON Export ​

Export logs as JSON:

python
# Export logs
client.logs.export(
    format="json",
    output_file="logs_jan_2025.json",
    filters={
        "prompt_name": "customer-classifier",
        "status": "success"
    }
)

Scheduled Exports ​

Set up automatic exports:

  1. Navigate to Settings → Exports
  2. Click New Scheduled Export
  3. Configure:
    • Format (CSV or JSON)
    • Frequency (daily, weekly, monthly)
    • Destination (email, GCS, S3)
  4. Click Create

Retention ​

Log retention varies by plan:

PlanRetention Period
Free7 days
Pro90 days
EnterpriseCustom (up to unlimited)

TIP

Export logs before they expire if you need long-term storage.

Privacy & Compliance ​

Data Redaction ​

Automatically redact sensitive data from logs:

python
# Configure redaction rules
client.settings.update_redaction_rules([
    {
        "pattern": r"\b\d{16}\b",  # Credit card numbers
        "replacement": "[REDACTED_CC]"
    },
    {
        "pattern": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",  # Emails
        "replacement": "[REDACTED_EMAIL]"
    }
])

GDPR Compliance ​

Delete user data for GDPR compliance:

python
# Delete all logs for a user
client.logs.delete_by_metadata(
    metadata={"user_id": "user_123"}
)

Debugging ​

Error Logs ​

View only failed requests:

python
error_logs = client.logs.list(
    status="error",
    limit=50
)

for log in error_logs:
    print(f"Error: {log.error_message}")
    print(f"Prompt: {log.prompt_name}")
    print(f"Input: {log.input_variables}")

Integration Errors ​

Debug integration code failures:

python
integration_errors = client.logs.list(
    prompt_name="customer-classifier",
    has_integration_error=True
)

for log in integration_errors:
    print(f"Pre-process error: {log.pre_process_error}")
    print(f"Post-process error: {log.post_process_error}")

Slow Requests ​

Find slow requests:

python
slow_logs = client.logs.list(
    min_latency_ms=5000,  # Slower than 5 seconds
    limit=20
)

for log in slow_logs:
    print(f"Latency: {log.total_latency_ms}ms")
    print(f"Breakdown: pre={log.pre_process_ms}ms, llm={log.llm_call_ms}ms, post={log.post_process_ms}ms")

Webhooks ​

Get notified when certain events occur:

python
# Create webhook for errors
webhook = client.webhooks.create(
    url="https://your-app.com/webhooks/mandatum-errors",
    events=["request.error"],
    filters={
        "prompt_name": "customer-classifier"
    }
)

Learn more about webhooks.

Best Practices ​

Metadata Strategy ​

Use consistent metadata keys:

python
# Good: consistent naming
metadata = {
    "user_id": "user_123",
    "session_id": "session_456",
    "environment": "production",
    "version": "v2.1.0"
}

# Bad: inconsistent naming
metadata = {
    "userId": "user_123",
    "session": "session_456",
    "env": "prod",
    "ver": "2.1.0"
}

Sampling ​

For high-volume applications, consider sampling:

python
import random

# Log 10% of requests
if random.random() < 0.1:
    response = client.prompts.run(
        prompt_name="high-volume-classifier",
        input_variables={...},
        metadata={"sampled": True}
    )
else:
    # Use direct LLM call without logging
    response = call_llm_directly(...)

Monitoring ​

Set up alerts for:

  • High error rates (>5%)
  • Slow requests (>5 seconds p95)
  • High costs (>$X per hour)
  • Unusual patterns (e.g., spike in requests)

Next Steps ​