Sharon Martin

Case Study 2: AI Reading

  • Case Study 2: AI Reading

Optimization Test: AI Psychic Reading for New Visitors

Create a non-intrusive, interactive side message pop-up that engages New Visitors while on their journey.

Project Overview

Hypothesis: Offering a free reading with AI Psychic to new visitors on higher funnel pages will engage users that otherwise would have left the site and lead them to converting customers.

My Role: Web Optimization Team Researched & UI/UX Designer, UI/UX Lead on Cross Functional Development Team

Teams: Web Optimization Team, Product strategy, Sentience (AI Psychic creator), Cross Functional Development Team consisting of: UI/UX designer, Front-end developer, Product manager, Back-end Developer, Data base administrator, Quality assurance specialist

Timeframe: 4-6 Sprints to implement, iterations to follow based on initial learning from test results

Description: Server side test. Create a popup that engages New Visitors on specific locations of their journey. Use only one AI Psychic for the experiment. One pop-up per session, pop up can be minimized and travel to other pages. If visitor leaves site and returns in a new session, visitor will receive new pop up. Visitor must enter name, email address & must consent to terms & conditions to proceed with free reading. Email address stored in data base for future lead generation opportunities.

 

Discovery

Can we utilize existing functionality i.e., Chat interface? Direct Message interface? Psychic Cloud AI app? Can we reuse Advisor promo pop up from Test 137? Do we need to start from scratch – find/compare examples of sites that utilize side message pop-ups.

After research & discussion, the Web CX team received direction from business to use the existing chat interface. Mock ups utilizing the chat interface were presented. Modifications were reviewed & noted, new mock ups were created and then… Pivot. New direction from business. Start from scratch. Sometimes it’s just the way things go…

Requirements & Details

User experience requirements: 

  1. Original, variation 1 – pop-up open, variation 2 – pop-up minimized
  2. User visits /articles, /horoscopes, /our-psychics
  3. User can minimize or close pop-up
  4. User must submit name, email & consent to T&C’s
  5. AI Psychic initiates conversation
  6. Unlimited character count for user’s messages
  7. User can continue conversation or end chat 
  8. End chat: message reinforcing chat has ended, present with optional thumbs up/down feedback & “Close window”
  9. Show feedback selected & “Close window”
  10. Show validation errors 
  11. Responsive design; smallest screen 320px

Technical requirements: 

  1. Sentience to provide AI Psychic image
  2. Pop up appears after 20 seconds page view in /articles, /horoscopes sections
  3. Pop-up appears after 2nd visit within single session /our-psychics section
  4. Terms & Conditions: expands & collapses within the pop-up, user scrolls to bottom of text and activates check box, box must be checked in order to continue
  5. No time limit or character length limit on reading
  6. Present feedback if user doesn’t respond to AI Psychic within 2 minutes
  7. Convo completed, feedback appears – if user doesn’t reply to feedback within 1 min, we automatically close pop up
  8. During reading, user can minimize chat window & convo stays, after 2 minutes of inactivity, chat window closes
  9. Connection issue – display message

Server Side Experiment Mock Ups

Technical Requirements Mock Ups

Experiment Results & Learning

57,062 Visitors

Audience set for new visitors to the website on all devices.

25 days

Keeping an eye on engagement, the test was limited to 25 days.

15 Free Readings

An extremely low number of readings = more learning!

Metric Original Variation 1 Variation 2
Visitors
18,756
18,926
19,380
AI Psychic Appears
N/A
32.17%
30.02%
Clicks on Chat Now
N/A
5
4
Clicks on Unlock Free Insights
N/A
3
3
Enroll Step 1
1.84%
1.83%
1.88%
Enroll Confirmation
.19%
.18%
.21%

Based on a combination of data analysis & intuitive insights, the test was re-ran with the following:

Revised hypothesis: Offering a free AI Psychic reading to all visitors on higher funnel pages will enable users to get answers to their questions right away.

Removed barriers: It appeared that forcing the user to provide a name and email address had a negative impact. Removing the form fields changed the vibe, leaving the visitor with the opportunity to explore a fun, interactive experience.

Status: Although engagement was beginning to gain momentum, the experiment was stopped prior to statistical significance.

My Notes:  The initial directive for the experiment felt invasive from a user experience perspective. I believe that we could have encouraged engagement by leaving out the request for personal information.

From a development perspective it was complex & challenging, required a significant amount of cross team collaboration & clarification outside of the scrum process. For me as the UIUX Lead, updating the Figma doc was crucial because it was the reference document for all involved.

On a positive note it was exciting to experiment with the AI Psychic. What kind of answers were we going to get? Would there be actionable advice? How in-depth could “she” get within a topic? Would it feel like you were talking to a person? The answers are yes! The AI Psychic continued to learn and adapt emotionally to the type of question, incorporating empathy & spiritual connections.

*For all intents & purposes we used toaster and pop-up interchangeably 🙂