Users & access Value lists AI Data sources

AI settings

Steer how the AI answers — without code changes. Every change is a draft until it passes evaluation.

Your AI settings. Each role sees only its own level; admins manage global settings.

Editor view. Your personal preferences and the Playground. AI notes for a client live in that client's Settings. Global instructions and prompts are managed by admins.

Viewer view. Only your personal preferences: how answers look for you. Global and client settings are managed by admins and account leads.

Only your personal preferences. Global and client settings are managed by admins and account leads.

Editing scope GlobalClient overrideMy preferences Narrower scope wins, only for the fields it sets (REQ-AISET-4) Live v6 · since Sep 30 Draft v7 · unpublished · edited by Dmytro P. 10 min ago

Global instructions — CRO playbook

Applied to every AI feature, after locked guardrails (REQ-AISET-1)

changed in draft v7

You are the CRO analyst assistant at Absolute Web, a Shopify CRO agency.
You help PMs and strategists understand A/B test results and find ideas from our own test library.

## How to answer
- Lead with the answer in 1–2 sentences, then the evidence.
- Cite every test you rely on as [T-xxxx]. If no test supports a claim, say
  "general CRO practice, not from our data".
- Quote only numbers present in the context. Never compute statistics yourself.
  If confidence is missing, say so.
- Prefer decision segments (All / Desktop / Mobile). Mention insight segments
  (new vs returning, source, country) only as supporting evidence.

## Our decision rules
- A segment wins only if the primary KPI has ≥ 95% probability to beat control
  AND no guardrail KPI drops.
- Rollouts are device-level only: All devices, Desktop only, Mobile only.
- Underpowered segment → "not enough data", never "no effect".

## Similar businesses
Match first on vertical, business model, AOV band; then traffic band and mobile share.

## Ideas  (NEW in v7)
Every idea must name: page, element, lever, the source tests it is based on,
and why it should transfer to this client (audience, model, AOV).

## Glossary
1st Sub CR = first-time subscription orders ÷ visitors.  RPV = revenue ÷ visitors.
P2BC = probability to beat control.

Markdown. ~420 tokens added to every call. Compare with live v6 →

Default response settings

Features inherit these unless they override (REQ-AISET-3)

Answer languageMatch the user's question

LengthShortStandardDetailed

FormatBulletsProseTable

Citation styleInline [T-xxxx] + sources list

Creativity (temperature)Low · 0.2

Max sources per answer12

Min record quality0.5 (hide weaker records)

Legacy deltas-only recordsInclude, labeled as legacy

Locked guardrails

🔒 code

Always applied first. No prompt or setting can switch them off (REQ-AISET-11).

Only records the user may see; others anonymized
Tests excluded from library never leave their client
Cite the records used
Propose changes as a diff — never save without approval
Statistics come from the stats engine, never from the model
Anonymized sources use opaque IDs (gap A14)

Per-test chat v4 Global chat v7 draft Idea generation v3 Learning draft v5 Test summary v2 Auto-tagging v6 Sheets normalization v2 Duplicate detection v1

Global chat

Searches the library and answers questions across clients the user can access (REQ-AI-2)

Live v6Draft v7

Variables: {{playbook}}{{sources}}{{user_question}} {{allowed_clients}}{{client_profile}}{{test_card}}{{examples}}{{today}}

Response settings for this feature

Blank = inherit global default

ModelClaude Sonnet 5

Creativityinherit · 0.2

LengthDetailed

Formatinherit · Bullets

Max sources20

Min record qualityinherit · 0.5

Examples included3 most relevant

RetrievalFilters + full-text + semantic

My preferences

Only for you · applied last, presentation only

Answer languageRussian

LengthShortStandardDetailed

FormatBulletsProseTable

Audience of the answerInternal (CRO terms)

Default client contextNone — ask me

Show sourcesInline + list

Personal note to the assistant

Style only. It can't change decision rules, permissions or what data you see.

What each role can set

Level	Who
Global instructions, prompts, examples, evaluation, budget	Admin
Client overrides	Admin: all · Editor: assigned clients
My preferences	Everyone, for themselves
Locked guardrails	Nobody (code)

Order: guardrails → global → client → mine. Narrower level wins only for fields it sets.

Feature All Status Active

Approved answers the AI should imitate — our house style and reasoning (REQ-AISET-6). Mark any learning, analysis or idea as an example from its record, or add here.

Feature	Example	From	Approved by	Used in answers (30d)
Learning draft	Default-to-subscribe learning Segment-specific, names the mechanism and the next test	CAR-001	Ivan M.	41
Learning draft	Flat result written as a useful learning Explains why "flat" still informs the roadmap	CAR-003	Ivan M.	27
Global chat	"What PDP subscription tests won?" Grouped by outcome, cites 6 tests, states confidence	Chat, Oct 2	Anna K.	18
Idea generation	Bundle default idea for a supplements client Explains transferability: same model, close AOV	Idea #212	Dmytro P.	12
Per-test chat	"Roll out to all devices or mobile?" Uses decision rule + guardrails, refuses to over-claim desktop	CAR-007	Ivan M.	9
Auto-tagging	Sticky ATC → CTA + Friction reduction	FNT-022	Marta S.	64

Golden questions

search 22 · analysis 18 · ideas 14 · permissions 10

Permission leaks

must be 0 to publish

Hallucinated numbers

must be 0 to publish

Last run

Draft v7 vs Live v6

8 min ago · 3 min · $2.10

✓

Publish gate passed. No leaks, no hallucinated numbers. Source recall +6 pts. Cost per answer +10%.

Metrics

Metric	Live v6	Draft v7	Change
Source recall @10	82%	88%	+6 pts
Citation correctness	95%	97%	+2 pts
Ideas with transfer reason	41%	93%	+52 pts
Hallucinated numbers	0	0	—
Permission leaks	0	0	—
Avg cost / answer	$0.031	$0.034	+10%
Latency p50	4.2 s	4.6 s	+0.4 s
Human rating (1–5)	4.1	rate 10 answers →

Changed answers

11 of 64 differ

"Ideas for Greenroot's PDP"better

v7 explains why each idea transfers (same model, close AOV)

"Buy box tests that won on mobile"better

Found 2 more matching tests (recall)

"Did trust badges ever work at checkout?"worse

v7 answer longer; missed the Inconclusive CAR-006 caveat

Viewer asks about Carlson testssame

Anonymized in both — opaque IDs only

See all 11 →

FeatureGlobal chat

Run asViewer · not assigned to Carlson LabsMe

Client contextGreenroot Botanicals

CompareLive v6 ↔ Draft v7

Live v6

4.1 s · $0.029 · 9 sources

Three PDP subscription tests won at supplement brands:

Subscribe preselected — mobile 1st Sub CR +28% T-8F21C
Savings badge on subscription option — +16% T-2A90D
Subscribe & save preselected — mobile +19% T-77B1E

Ideas for Greenroot: preselect subscription; add a savings badge.

Draft v7

4.7 s · $0.035 · 11 sources

3 wins, 1 flat among 4 PDP subscription tests at supplement brands; the effect is mobile-driven.

Win · Subscribe preselected — mobile +28% 1st Sub CR, desktop underpowered T-8F21C
Win · Savings badge — +16% all devices T-2A90D
Win · Subscribe & save preselected — mobile +19% T-77B1E
Flat · Toggle → radio cards (Pet brand, not supplements) T-41C07

Ideas for Greenroot (Subscription + Bundles, $50–100 AOV, 66% mobile):

PDP · Subscription widget · Value framing — preselect subscription on mobile first. Transfers: same model and AOV band. T-8F21C T-77B1E
PDP · Subscription widget · Clarity — show the saving in $ next to the option. T-2A90D

Anonymized sources show opaque IDs (T-…) because "Run as" user is not assigned to those clients.

Answers rated (30d)

212

of 1,480 answers

Positive

82%

+5 pts vs previous 30d

Proposals accepted as-is

64%

edited 27% · rejected 9%

New to review

negative ratings

Date	Feature	User	Rating	Reason	Question / proposal	Version
Oct 12	Global chat	Marta S.	👎	Missed a test	"Cart upsell tests for home brands" — didn't include OAK-014	v6
Oct 11	Learning draft	Ivan M.	edited	Too generic	CAR-011 learning rewritten by editor (diff 64%)	v5
Oct 11	Auto-tagging	Anna K.	edited	Wrong lever	Value framing → Clarity on CAR-011	v6
Oct 10	Per-test chat	Anna K.	👎	Over-claimed	"Roll out to all devices" while desktop was underpowered	v4
Oct 9	Idea generation	Dmytro P.	👍	—	3 ideas for Greenroot saved to backlog	v3

Version history

every change is versioned (REQ-AISET-7)

Version	What changed	Author	Eval	Status
v7	Playbook: "Ideas" rules · Global chat: grouped by outcome	Dmytro P. · Oct 12	passed	Draft
v6	Min record quality 0.4 → 0.5	Ivan M. · Sep 30	passed	Live
v5	Learning draft: add "next test" line	Ivan M. · Sep 12	passed	Archived
v4	Client override: Solace Skin NDA	Dmytro P. · Aug 28	passed	Archived
v3	Tried temperature 0.7 for ideas	Anna K. · Aug 20	failed · 2 hallucinated numbers	Rejected

Usage & cost — October

REQ-AISET-12

Budget $300 / month · alert at 80%$142 used (47%)

Feature	Calls	Cost	p50
Global chat	1,020	$61.80	4.2 s
Per-test chat	640	$38.40	3.6 s
Idea generation	96	$17.30	7.9 s
Learning draft	58	$6.10	5.1 s
Auto-tagging	410	$2.90	1.2 s
Embeddings	2,300	$0.40	—
Evaluation runs	14	$15.10	—

Who can edit

Manage

Admins and users with the AI editor permission. Everyone else read-only. All changes in the audit log (REQ-AISET-13).

Dmytro P. · AdminIvan M. · AI editorAnna K. · AI editor