Skip to content

Getting Started

Prerequisites

  • Python 3.9+
  • valid Prodloop API key
  • local audio file (.mp3, .wav, .webm, etc.) for call evaluation
  • litellm provider credentials when running local user_orchestrated bot simulations

Install

pip install prodloop-observability-sdk

First Request

from prodloop import ProdloopClient, EvaluationParameter

client = ProdloopClient(api_key="sk_live_...")

response = client.evaluate_call(
    audio_file_path="sample_call.mp3",
    parameters=[
        EvaluationParameter.E2E_RESPONSE_TIME,
        EvaluationParameter.HALLUCINATION,
    ],
    thresholds={"e2e_response_time_max_ms": 800},
)

print(response)

For prompt-aware checks, pass the bot prompt used during the call as input_prompt:

response = client.evaluate_call(
    audio_file_path="sample_call.mp3",
    parameters=[
        EvaluationParameter.SECTION_SEQUENCING,
        EvaluationParameter.INTERNAL_JARGON_LEAKAGE,
    ],
    input_prompt="The production prompt used by the bot during this call...",
)

Prompt-aware results return passed as "true", "false", or "N/A". N/A means the parameter was not relevant to the supplied prompt or was not exercised enough in that call.

For extraction validation use:

response = client.evaluate_call(
    audio_file_path="sample_call.mp3",
    parameters=[EvaluationParameter.EXTRACTION_VARIABLES],
    extraction_schema={"customer_name": "string"},
    bot_captured_variables={"customer_name": "ram"},
)

First Prompt Simulation

from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins

client = ProdloopClient(api_key="sk_live_...")

start = client.simulate_prompt(
    simulation_mode=SimulationMode.SELF_SIMULATION,
    prompt="You are a concise support bot. Do not invent policy details.",
    parameters=[EvaluationParameter.HALLUCINATION],
    bot_llm=plugins.LiteLLM(model="vertex_ai/gemini-2.5-pro"),
    max_turns=10,
    adaptive_max_conversations=50,
)

print(start["chat_id"])

self_simulation returns a chat_id immediately. Poll with client.get_simulation(chat_id) until status is completed or failed. In adaptive simulations, max_turns is the turns-per-conversation limit and adaptive_max_conversations is the conversation exploration limit.