What is more insulting is that eHarmony has done the same thing to you that they have done to multiple users; they suddenly find a huge crop of “matches” for you immediately before your free subscription runs out. It is a transparent marketing tactic, and an annoying one. They could have trickled those matches out with their batch e-mails rather than trying to “hook” you at the last minute with quantity.

eHarmony also doesn’t let certain people sign up at all if you don’t answer the questions in the inital profile to their satisfaction; I was one of them. I believe my mistake was being overwhelmingly negative (I call it pragmatic) on the initial questionnaire. If I really cared though and I wasn’t just doing it as a social experiment, I would have answered the questions according to what they wanted to hear. Do you want dishonesty in your mate? Don’t choose eHarmony.

I would like to comment though that I sincerely doubt that you experienced A/B testing. The subject line of eHarmony’s are the most important part of the emails that they send you: they are the grabber that either catches your attention (and gets you to their site) or not. In the first examples you showed, I agree that they all look very similar. I believe that the eHarmony admins realized the importance of those subject lines and decided to do an overhall of them to make them more attention grabbing. There would be no reason to change the contents of the email as again, it is the subject line that grabs your attention. From my perspective, there is no reason to believe that they are doing any specific testing of how you respond: I think they simply decided to make a slight update to their email system with the hopes that it will be more successful (ie with the hopes that the subject lines are more attention-grabbing).

Which isn’t very interesting. Really, it’s not.

I have heard eharmony does the same thing.

okcupid does a lot better, in my opinion, by extracting a fair of the “who” you are. Far more interesting and relevant.

The only thing I liked about it was that it did focus more on personality by making you fill the about you sections properly.

But seriously, to me it seemed like a glossier version of email notifications from any classified service.

Excellent post, really enjoyed reading it and the follow-up comments. I feel like I learned something!

More please on UX testing :)

1. My theory is that each subject line is a separate variable, based on the experience that the inside contents of the email are all the same (they were). I imagine eHarmony’s team is poised each morning to see which emails all the “Chrissies” out there open. The result is that it’s a terribly executed test.

2. I measured these emails and this technique for 3 months, concluding on January 4, 2010, when I noticed that suddenly the subject lines got even nuttier, prompting this post. I would have thought that after 3 months (or another reasonable period of time to test) that the update to the strategy would have been to select one line over the other, not to launch a new campaign of even battier-sounding diverse subject lines. The strategy didn’t refine itself from last year’s execution.

3. “There is no test.” Woah! In a world where even crazier things happen, this could be absolutely true. That eHarmony is simply outputting based on whimsy and not measurement, which is really, really stupid.

Jason, you have me really thinking this through even more, and you articulated Drew’s points perfectly. Thank you! Checking out http://www.zaaz.com now:)

Best from Brooklyn,
CB

1. Each subject line is a separate variable in what is probably the world’s most poorly executed A/B/n test (this is your theory, I think, and a very reasonable one). In this case, the eHarmony folks would be looking to see which specific subject lines got more people to open/click through to a profile view. If this is the test, they are not taking into account the disruption and noise from sending different variables to the same person that you mentioned, and will have an epic amount of meaningless data on their hands.

2. You are in an A/B test that is tracking multiple subject lines vs. a consistent subject line. Someone hypothesized that variety will inspire more interest (i.e., that the disruption you felt is actually a positive) and will result in increased engagement. In this world, there is another Chrissie out there who is getting the same emails all with the same (consistent) subject line, and you are being compared against here. This could actually result in good learnings for eHarmony (especially if everyone is reacting to the variety as negatively as you are).

3. There is no test. You are just getting random subject lines because some guy thought that variety was a good idea, and the email program is being measured globally rather than by subject line. This, of course, would be really, really stupid, but it happens.

Awesome post. Thanks!

Jason

Completely agree. Match.com sends a daily digest around noon with the photos of all of your latest matches. I’m far more likely to open a single email and click-through to a profile. I almost never clicked through on an eHarmony profiles for a few reasons:

1.) no photos, which is strange and feels dishonest or like eHarmony is embarrassed of it’s users, Ok Cupid wrote a great article “Your Looks & Online Dating” http://ow.ly/Vbqd

2.) had to open 4-8 per day (which if I wasn’t studying them, would have led to immediate batch deletes)

3.) the core experience took me to one profile, and it wasn’t easy to navigate and match up the new matches with the emails in my inbox, so the experience was not succinct.

Ciao!
CB

So again great point building on Drew’s observation – but why isn’t this A/B testing?

I didn’t explain that inside each of these messages (for the 3 month duration) the content and design of all of these emails is completely the same (exception the name of the person and his age). Nothing changes, not the color of the buttons, template, copy, etc.

The only difference is the subject line.

I’ve worked with multivariate testing for years (including subject line A/B), so I’m curious as to why you wouldn’t classify this as A/B testing – is it because they’re batch sent, which then just makes the campaign badly designed?

Maybe I’m wrong:) Please help!

:)CB

If they sent one email with the same information (there is after all a body to an email, not just a subject) then the emails would be between 60% to 90% more effective, since they’d all stand a chance at getting read.

You make a great point. A/B testing can be delivered in any number of ways, divided between multiple users or based on a single user’s experience. In this case, I consider this A/B testing, because it’s clearly subject line testing. The content inside the messages is the same (exception the person’s name), but it’s eHarmony trying to learn what headlines compel users to open more frequently.

What bugs me is what they do with that information? Do they fine tune based on individual users, so that I might wake up to a row of the same subject lines? Or, do they compile that data and use it to make community-wide decisions?

In my experience with this series of emails, delivered over a 3 month subscription period, I never noted a change in receiving more of one kind v. the other. (and, I did measure this)

A/B testing (even subject line testing) doesn’t stop at the user’s action – it goes on into how the company uses the data to create a better, more compelling, overly awesome experience that doesn’t result in a grumpy (me) user:)

Cheers!
CB

They probably DO test which headlines get the best response. But they are counting on the VARIETY to actually cause an increase in responsiveness all on its own, regardless of which headlines are better than others.