ElevenLabs Conversational AI

The ElevenLabs deploy provider pushes versioned prompts and tools from Ellaworks into ElevenLabs Conversational AI (Convai) agents. Ellaworks creates and updates the agent, syncs its webhook, client and MCP tools, can import agents you already built in ElevenLabs, and checks deployed agents for drift.

ElevenLabs Account Setup

Ellaworks talks to the ElevenLabs API with a single API key, sent as the xi-api-key header on every request.

1. Sign in to ElevenLabs and open Settings > API Keys.
2. Create a key for Ellaworks and copy it.
3. Make sure the key’s workspace is the one where your agents should live. Everything Ellaworks does (listing agents, importing, deploying, verifying) uses the key configured on the selected environment.

Connection Setup in Ellaworks

1. Add ElevenLabs to an Environment

1. Open Environments and edit the environment you want to deploy to.
2. Under Providers, choose ElevenLabs from Select provider and click Add.
3. In the Secret field, pick an existing organization secret that holds your API key, or click Add Secret to create one inline.

The key is stored as an encrypted organization secret and referenced from the environment as ${SECRET_NAME}. It is never saved in plain text on the environment.

[!NOTE] If ElevenLabs is configured on the Global environment, other environments inherit those credentials automatically. Add ElevenLabs to a specific environment only when it needs its own key.

2. Agent Configuration

When you create or edit an ElevenLabs agent, the configuration editor exposes the ElevenLabs agent settings. The main fields are:

• Agent Name: The agent’s name in ElevenLabs.
• Agent Prompt (required): The system prompt. Reference a prompt from your registry (e.g. system-prompt@^1.0.0); Ellaworks resolves it at deploy time.
• First Message: The agent’s opening line. This can also reference a registry prompt.
• Voice ID (required): The ElevenLabs voice to use.
• Language: e.g. en.
• Model: The LLM, e.g. gpt-4o-mini.
• Tool References: The Ellaworks tools to attach to the agent.

Advanced settings are also available, including speech recognition keyterms, turn-taking (eagerness, timeouts, hang up after silence, filler messages), interruptions, background voice filtering, language presets, data collection, evaluation criteria, guardrails, authentication and workflows.

[!IMPORTANT] Validation rules:

• Voicemail detection requires a non-empty voicemail message.
• Expressive mode requires the Eleven v3 Conversational TTS model.
• TTS speed must be between 0.7 and 1.2.

Secrets in setup webhook headers

Request headers on the conversation-initiation (setup) webhook can reference an organization secret, for example Bearer ${API_KEY}. On deploy, Ellaworks resolves the secret, stores it as an ElevenLabs workspace secret, and sends only the secret’s ID in the agent configuration. The workspace secret is updated on every deploy, so rotating the organization secret takes effect on the next deploy.

Working with Tools

ElevenLabs supports four tool types in Ellaworks:

Built-in tools

The seven ElevenLabs built-ins are configured directly on the agent: End call, Transfer to number, Transfer to agent, Voicemail detection, Keypad tones, Language detection and Skip turn.

Tool redeploys

Ellaworks records the ElevenLabs ID and a hash of each deployed tool per environment. On redeploy, an unchanged tool is reused as-is, a changed tool is updated in place, and a tool that was deleted in ElevenLabs is recreated.

Importing Existing Agents

ElevenLabs agents can be imported into Ellaworks in bulk:

1. Go to Import Agents and choose ElevenLabs as the provider.
2. Choose a target environment. Import uses that environment’s ElevenLabs API key, so the Global environment can’t be used.
3. Choose a target prompt registry. Each imported agent creates one prompt artifact there (named <agent-key>-imported-system) from the agent’s inline system prompt.
4. Select the remote agents and click Start import.

Tools referenced by an imported agent are fetched and imported as separate Ellaworks tool records. Secret-like fields are stripped from imported tools, with a warning, so re-enter those values (ideally as secrets) before redeploying.

Deploying, Updating and Removing

• Deploy: Ellaworks deploys the agent’s tools first, then creates the ElevenLabs agent with the resolved prompts and tool IDs.
• Update: Redeploying sends the full agent configuration as an update to the existing ElevenLabs agent.
• Prompt-only updates: When only prompts change, Ellaworks reads the live agent and patches just the Agent Prompt and First Message, leaving other settings untouched. Other prompt fields are reported as skipped.
• Remove: Removing a deployment deletes the agent in ElevenLabs.

Rate-limited (429) and server error (5xx) responses are retried up to three times, honoring the Retry-After header. Create requests are retried only on 429, so a lost response never creates a duplicate agent.

Drift Detection

When Drift Detection is enabled on the environment, Ellaworks verifies deployed agents against the live ElevenLabs agent daily and reports any differences. Only one environment per organization can have drift detection enabled.

• Compared: name, tags, conversation_config, platform_settings and workflow, including attached tool IDs and knowledge-base IDs. Edits made in the ElevenLabs dashboard to the prompt, voice, language, LLM settings, tools, platform settings or workflow show up as drift.
• Ignored: Fields ElevenLabs owns, such as the agent ID, created/updated timestamps, archived, access_info, usage_stats and response_mocks.
• Server defaults: Only fields you authored are compared, so defaults ElevenLabs fills in don’t cause false drift.

Troubleshooting & Typical Errors

Deployment Verification

1. Click Deploy in Ellaworks.
2. Open the ElevenLabs dashboard and find your agent under Conversational AI.
3. Confirm the prompt, first message and voice match, and that your tools and MCP servers are attached.