Skip to content

Parameters

Supported post-call evaluation parameters are grouped into audio metrics, extraction checks, and prompt-aware checks.

Use enum constants from EvaluationParameter:

from prodloop import EvaluationParameter

params = [
    EvaluationParameter.E2E_RESPONSE_TIME,
    EvaluationParameter.SECTION_SEQUENCING,
    EvaluationParameter.INTERNAL_JARGON_LEAKAGE,
]

Audio And Extraction Parameters

  • e2e_response_time: average latency in milliseconds between one speech segment ending and the next one starting.
  • turn_by_turn_latency: per-gap latency values as turn_index and latency_ms.
  • pause_profile: deterministic pause aggregate (pause_count, total_pause_time_ms, longest_pause_ms).
  • audio_artifacts: deterministic signal quality indicators (clipping_ratio, dc_offset, clipping/DC flags).
  • hallucination: whether the bot introduced fabricated or incorrect content.
  • extraction_variables: extracts requested structured fields.
  • interruption_behavior: whether interruptions were handled gracefully.

Deterministic today:

  • e2e_response_time
  • turn_by_turn_latency
  • pause_profile
  • audio_artifacts

Prompt-Aware Parameters

Prompt-aware parameters compare the actual call against the input_prompt you send with the request. Pass input_prompt whenever you request any of these:

  • section_sequencing
  • mandatory_field_gating
  • interrupt_resume_precision
  • closing_verbatim_delivery
  • single_attempt_constraints
  • info_dump_handling
  • mid_flow_intent_switch
  • side_talk_leakage
  • ambiguous_partial_responses
  • internal_jargon_leakage
  • identity_extraction
  • prompt_injection
  • commitment_extraction
  • scope_boundary_testing
  • roleplay_jailbreak
  • context_memory_across_turns
  • hallucination_fabrication

Each prompt-aware result has this compact shape:

{
  "passed": "true",
  "explanation": "The bot followed the required order for the exercised flow."
}

passed is one of:

  • "true": the parameter was relevant and the call satisfied the prompt.
  • "false": the parameter was relevant and the call violated the prompt.
  • "N/A": the selected parameter was not relevant to the supplied prompt or the call did not exercise enough behavior to judge it.

Prompt-Aware Parameter Purpose

  • section_sequencing: checks whether the bot followed the order, branches, skipped steps, and terminal states required by the supplied prompt.
  • mandatory_field_gating: checks whether required information or confirmations were collected before prompt-defined dependent actions.
  • interrupt_resume_precision: checks whether the bot answered interruptions and resumed the exact pending step.
  • closing_verbatim_delivery: checks exact required closings and terminal post-closing behavior.
  • single_attempt_constraints: checks one-attempt and bounded-retry limits defined by the prompt.
  • info_dump_handling: checks whether dense user-provided details were captured and reused.
  • mid_flow_intent_switch: checks whether the bot handled a legitimate intent switch without losing context.
  • side_talk_leakage: checks whether background or third-party speech affected the bot incorrectly.
  • ambiguous_partial_responses: checks whether vague answers were clarified before routing or confirming.
  • internal_jargon_leakage: checks for internal prompt, tool, system, variable, template, or process language shown to the user.
  • identity_extraction: checks whether identity or contact information was captured and used according to the prompt.
  • prompt_injection: checks whether user instructions overrode the supplied prompt.
  • commitment_extraction: checks unsupported guarantees, final confirmations, approvals, timelines, or binding claims.
  • scope_boundary_testing: checks whether the bot stayed inside the prompt-defined scope.
  • roleplay_jailbreak: checks whether the bot resisted role/persona changes that conflict with the prompt.
  • context_memory_across_turns: checks whether prior context and corrections were retained across turns.
  • hallucination_fabrication: checks unsupported facts, claims, statuses, policies, capabilities, or operational statements.

Extraction Variables

If you include extraction_variables, pass both:

  • extraction_schema (what fields to extract)
  • bot_captured_variables (what your bot captured for validation)
extraction_schema = {
    "customer_name": "string",
    "budget_mentioned": "int",
}

bot_captured_variables = {
    "customer_name": "ram",
    "budget_mentioned": 12000,
}