Bernie Malinoff of Element54 presented “Sexy Questions, Dangerous Results?” at The Market Research Event (TMRE) in October. Bernie said that “Do It Yourself” survey software is “in a space race: widgets and ROI are pulling ahead of best practices.” He described research from two studies his firm conducted contrasting Web 1.0 surveys (traditional HTML forms) with Web 2.0 surveys (multimedia data-collection widgets). For both surveys, the questionnaire was the same; only the presentation differed.

Unfortunately, “sexy” surveys do not fundamentally make surveys more enjoyable. In a survey of 100 respondents, randomly divided between 1.0 and 2.0 survey formats, respondents reported no statistically significant differences in:

• Enjoyment of the survey
• Desire to repeat the survey, or
• Ease of use of the survey.

In fact, the novelty of these new formats may actually make surveys harder to use compared to basic HTML forms. In a separate study of 2000 respondents, again evenly divided between 1.0 and 2.0 formats, respondents needed more time to complete the Web 2.0 survey; for instance, rich-media buttons took 20% longer to use than traditional radio buttons for a 1-to-10 scale.

Worse, the data was inconsistent. For three different questions, Bernie reported a statistically significant 8% to 36% variance in results collected with a Flash slider compared to traditional radio buttons.

Before we enthusiastically embrace Web 2.0 formats, we need to practice what Bernie calls “responsible innovation”. Advances in survey interfaces have great potential, but only when they produce valid and reliable data.

For more on Bernie’s research, check out his blog at Element-54.com.