CharityVillage® Research: How to Conduct Your Own Member Survey

Resources & Library

Path: Main Street : Resources & Library : Research Articles : Feature Article

How to Conduct Your Own Member Survey
By Paulette Vinette, CAE
November 21, 2005

Your organization's leadership supports your plan to survey members; regrettably your financial position does not allow you to pay an expert consultant to help you. This article provides ideas for you to consider as you set out to conduct your own member survey.

First, two pieces of advice:

A. Begin with the end in mind.

These simple yet wise words were made famous by management guru Steven Covey (who wrote 7 Habits of Highly Successful People). Before constructing your survey tool, write a list of the specific information you want to generate from your survey and how you intend to use it. This document will serve as a checklist to evaluate your end product before you use it.

It is always a good idea to test a survey instrument on a representative sample of intended respondents; keep their responses - they are usable. It is so easy to not recognize how two people will interpret the same words so differently. By trying out your survey you might find out how to tighten up the questions before it goes to everyone.

B. Use the KISS principle.

You've heard about the KISS principle? Keep it simple silly (okay, some people remember the last s to stand for stupid). Resist convoluting your questions with anything other than short, very clear questions that will generate comparable answers. How to do that?

1. What is the best way to glean the information you decided you need? The answer to that question depends on the number of people that you determine you need to participate. Theresa Kane, President of the Piper Group (a marketing company), uses a web-based tool to establish confidence intervals - see for yourself: www.macorr.com/ss_calculator.htm. The smaller the sample size, the larger the percentage of people you need to survey to achieve a high confidence interval.
Example: If you want to achieve a 95% +/- representative sample and you have 1,000 members, you would need to survey 278; if you have 100 members you would need to survey 80.
Each organization should rationalize their representative sample and expect no more than a 25% response rate. These days 35% is great; 10% - 15% is average in my experience. Representative presumes that all types of members are included in your sample (small/medium/large; contribution level categories; proficiency levels; whatever represents the breadth of members your organization represents). So now that you have decided the number of people you want to survey...

2. What type of survey instrument should you use?

The following are options for you to choose from:
a. In-person interviews
In-person surveys are very resource intensive; they require a lot of your time (time is money). However, you can learn from body language signs if you know how to read them, and a skilled interviewer can follow the respondent's conversation and glean additional information.

b. Focus groups
Focus groups appear simple and are quite complex. Unless you are trained to conduct them, you might want to leave the interviewing of a group of people all at once to the pros.

c. Telephone surveys
Telephone surveys can be every effective if you:

Send the questions out days in advance;
Open the call by restating the intention of the questions before proceeding with the interview;
Stick to the amount of time you pledged to restrict the call to (15 minutes is optimum);
Avoid having opinions about the respondent's comments;
Don't expect the respondent to pay for long distance charges; and
Always express your appreciation at the end of the call.

d. Event surveys
If members have an opportunity to sit down and complete a questionnaire during an event, it could be a great opportunity to ask them to evaluate their experience. Just a short, simple multiple choice tool works best. Don't expect a great response if you decide to ask them questions that require pensive answers.

e. Mail surveys
Mail survey used to be the tool of choice, that is, until online surveys came along. If you have reasons to use a mail survey, remember that:

The time to complete the survey should not exceed 10 minutes if you hope for a good response rate, and you should tell respondents upfront how long the survey should take them.
The logic shared in this article should be considered as you frame your mail out/in questionnaire.
You will probably increase your response rate if you provide a self-addressed enveloped - postage paid. Alternatively, offer a fax-back option (remember to include your fax number, preferably toll-free).

f. E-mail surveys
Some organizations like to e-mail the survey questionnaire and ask respondents to fax or mail back their responses. The e-mail method saves cost for the sender. Some attach an spreadsheet-type questionnaire that can be returned as an attachment to an e-mail.

g. Online surveys
Online surveys are proving very effective. You e-mail respondents a link and with one click they are in the survey tool. The one-click rule is very important when providing any kind of link in your e-mail communiqués. Constance Wrigley-Thomas, President Director of How, Essentient Association Management, says the majority of responses usually come in within three or four days of sending out the link by e-mail. She also suggests not using "N/A" as a multiple choice answer; instead use "Other" and invite the respondent to explain that choice.

There are a number of online survey services that are very affordable. Constance likes Survey Monkey (a 12-month subscription can cost as little as $250). Other products include websurveyor.com, zoomerang.com, questionpro.com, surveyclub.com, greenfield.com, advancesurvey.com, and more.

These online tools also allow you to follow up with only those who have not completed the survey, and can protect confidentiality if required. Plus, some allow you to transfer the aggregate results into spreadsheet files and from there you can create attractive and effective graphs and charts.

Online surveys can accommodate open-ended questions, however, remember how time consuming and challenging it will be for the person who has to read those individual responses and interpret them into a "survey finding". One way around this is to ask the respondent if they would allow someone to follow up on a question with a telephone interview. Then provide space for them to write in their phone number.
3. What type of questions should you ask?

The best way to get comparable responses is to ask members to select from a list of answers.
Example: Which of the following member services do you value most?
- Information (website, publications, etc.)
- Networking events
- Discounts, benefits, products
This works best if:

You only want the respondent to select one answer
Respondents can select more than one answer because you don't need to know which answers are more important than another.
If you want respondents to rank the order of importance of a list of answers, you need to ensure that the method used to compile the answers can capture the information correctly. Some online survey programs end up adding the numbers and leaving you with averages.

If you want to ask members to use scores in their answers, use a range from 1 - 5 (you can increase the range number, but always use an "odd" number). Clearly define what describes a 5-rating, a mid-point rating, and the lowest rating to help clarify questions respondents will have upfront.

Asking members to express their views in their own words is a complex way to approach a survey (unless, of course, the respondent is talking to you in person).

If you want to probe into certain areas, it is advisable to use a "skip ahead" prompt to allow those who are not engaged in the question to avert the supporting questions.

Remember to always indicate a survey response deadline. If your results are disappointing, you may be tempted to notify members that the deadline has been extended. Do you really want everyone to know your survey is being ignored? Perhaps you should switch to a telephone survey approach with the same bank of questions reformulated into three broad telephone interview questions.

4. How to use your survey findings?

If you want your survey findings to be memorable, distill the information into three or four key points. People can absorb that quantity of information.

If your culture requires a detailed report on how you chose the survey recipients and details of the methodology you used, it might make your report more readable if you support your written text with charts and graphs. Whenever you produce a long report, it is advisable to summarize the results in an Executive Summary at the beginning of your report.

Those who took the time to complete your survey should be interested in knowing the results, so it is smart to offer the information to them. In this information-saturated world, it is best to summarize the findings in a punchy communiqué for all.

5. So who needs a consultant?

A visit to dictionary.com offers the following definitions:

Survey: A gathering of a sample of data or opinions considered to be representative of a whole.

Research: Scholarly or scientific investigation or inquiry.

If it is important to you to have quantifiable, reliable data it may be advisable for you to use a consultant with expertise in the kind of information mining you need.

This article is by no means exhaustive on the ways you can approach conducting your own survey. Rather, it intends to offer thoughts and tips from someone "who has been there".

Paulette Vinette, CAE is President of Solution Studio Inc. - consulting firm that focuses on helping not-for-profit organizations. www.solutionstudioinc.com