if you're split testing responsive vs.

non, and looking at clicks or opens, you're doing it wrong #emaildesign — Elliot Ross (@iamelliot) July 10, 2014 I watched a webinar yesterday where the presenter was strongly advocating doing a split test between a Responsive email vs.

a Non-Responsive one (ideally the same email just sans-media queries) and I was reminded of a rant I half-wrote on my flight back from the Litmus Conference last year.

There's still too many swear words in that, but nevertheless I still think this approach demonstrates a lack in understanding in the benefits of going responsive.

Responsive email design is an experience thing.

It's about making the experience for a user better when they're using a mobile device, and that's a very difficult thing to quantify using numbers.

Any difference in opens, or clicks, can be down to so many things, that they're at best only a vague indicator.

Even if you look at conversions, there's so many factors at play that it isn't a reliable statistic.The true measure of things like responsive design, is to gauge the user's comprehension of the message and how happy they are with the brand as a result — but whilst that's perhaps the ultimate aim in marketing, that's incredibly hard to put a number on.Incidentally, towards the end of my flight on that trip home, the flight attendants handed out free choc-ices to everyone.

That wasn't part of the in-flight meal — that was long gone — so what was the return on investment on that? could you split test that and see who re-books another flight? No you can't.

There's no reliable correlation there whatsoever.

But I've still flown with Virgin Atlantic three times since then.

More on this Why should a business go responsive? Is responsive email design really worth it? WhichTestWon:

Responsive vs. Non Responsive Image: Ivan Colic / Noun Project