Content uploaded by Thomas J. Johnson
Author content
All content in this area was uploaded by Thomas J. Johnson on Jun 23, 2017
Content may be subject to copyright.
SOCIALSCIENCECOMPUTER REVIEW
Kaye,Johnson/ IMPROVINGONLINE SURVEYS Research Methodology:
Taming the Cyber Frontier
Techniques for Improving Online Surveys
BARBARA K. KAYE
Valdosta State University
THOMAS J. JOHNSON
Southern Illinois University
TheWorldWide Webandothernewelectronictechnologiesmightsoonbecomeprimesurvey vehicles
dueto convenient,verifiable,low-costdeliveryandreturnsystemsaswellas easy access and feedback
mechanisms. However, along with the benefits of this new technology come new experiences and les-
sons to be learned and shared by researchers. The authors created and posted a survey on the Web to
examine the uses of the Web for political information. The purpose of the article, however, is not to
reportonthepoliticalsurvey’sfindingsbutrathertoaddressissuesconcerningonlineresearch,discuss
the implications of an online environment for traditional survey techniques, share Web survey experi-
ences from an ex post facto perspective, and present recommendations for future online research, spe-
cifically in the areas of Web survey design, sampling, data collection and responses, and publicity.
Keywords: online survey, online data collection, Internet research, survey research,
online questionnaire design
INTRODUCTION
The authors created and posted a survey on the World Wide Web to examine the uses of
the Web for political information. The purpose of this article is not to report on the political
survey’s findings but rather to address issues concerning online research, discuss the impli-
cationsof an online environmentfortraditionalsurveytechniques, share Websurveyexperi-
ences from an ex post facto perspective, and present recommendations for future online
research.
Recently, researchers have been examining the Internet as a mass medium (December,
1996; Lock, 1995; Morris & Ogan, 1995; Newhagen & Rafaeli, 1996). Although it has
existedsincethe1970s, the Internet only hasreceivedseriousstudyduringthepast fewyears
as easy-to-navigate Web browsers have attracted a “critical mass” of users. Rogers (1986)
states, “A critical mass of adopters of an interactive communication technology [is] neces-
sary for the utility of the new idea to be sufficient for an individual to adopt” (p. 120).
Estimates of the number of people who use the Internet vary widely. Recent data claim
that the number of online users in the United States reaches between 52 million and 62 mil-
323
AUTHORS’ NOTE: The authors thank Jeffrey Gunther, lead Internet/groupware analyst at Southern Illinois
Healthcare, for his help in putting our survey online.
Social Science Computer Review, Vol. 17 No. 3, Fall 1999 323-337
© 1999 Sage Publications, Inc.
lion (“Florida Is Among,” 1998; “Latest Intelliquest Survey,” 1998). Other studies, whose
estimates are not quite so high, report between 26.4 million and 45.0 million Internet users
(“About One in Four,” 1996; “American Internet User Survey,” 1997; “eOverview Report,”
1998;“45Million,”1998;“GVU’s Seventh WWW User Survey,” 1997; Hoffman, Kalsbeek,
& Novak, 1996a; McGarvey, 1996; Taylor, 1997). An additional 9 million users have tried
the Internet but have not used it for at least 1 year and, therefore, are not considered regular
users by some researchers (American Internet User Survey, 1997).
Beforeanymediumcanbe considered a mass medium, acriticalmassofadopters must be
reached. Generally, critical mass is achieved when about 16% of the population has adopted
aninnovation(Markus,1990). In the caseofmass media, however,50millionusers seems to
bethe milestone (Neufeld, 1997;“Why Internet Advertising,”1997).The rate of radioadop-
tion crawled along for 38 years before hitting the magic 50 million users, whereas television
tookonly13years and cable television only 10 years to achieve this goal. Internet adoption is
racing along at a pace that guarantees 50 million regular users by the year 2002, only 8 years
after its emergence as a consumer medium (Neufeld, 1997; “Why Internet Advertising,”
1997). Depending on the various estimates, the Internet, and specifically the Web (its most
widely used resource), might already have reached critical mass.
In addition, early adopters of a new technology might differ from later adopters in many
characteristics (Rogers, 1986). Previous research claims that early adopters of the Internet
tend to be White males with high socioeconomic status, but more recent studies suggest a
demographic shift as the Internet becomes more mainstream (“eOverview Report,” 1998;
“45 Million,” 1998; “GVU’s Ninth User Survey,” 1998; Hoffman, Kalsbeek, & Novak,
1996b; “Latest Intelliquest Survey,” 1998).
Because of the Internet’s exponential growth, its impact on traditional means of commu-
nication, its dynamic nature, and its potential for reaching large and diverse segments of the
population, it is gaining interest from academia and industry researchers. Many are inter-
ested in learning more about the multitude of Web users and of Web uses, strengthening the
Web’s appeal as a survey tool. The Web and other new electronic technologies soon might
become prime survey vehicles due to “fast access, reliable, verifiable delivery, and ease of
feedback”(James, Wotring,& Forrest, 1995, p.47). However,alongwith the benefitsofthis
new technology come new experiences and lessons to be learned and shared by researchers.
The Web is still in the early stages of development, similar to television during the 1950s
(Bimber, 1998). Early adopters are creating new uses for the Web, and future refinements
will expand its capabilities.
SURVEY RESEARCH AND THE WORLD WIDE WEB
Many researchers are beginning to post surveys on the Web. For this study, an informal
Yahoo! inquiry using the search term “survey” yielded more than 2,000 online surveys in 59
subjectcategories.Thesurveyscovertopicssuchasfamilycomputing,motherhood employ-
ment, bicycling to work, home shopping, and Web advertising.
Although a large number of people access the Web, they still account for less than one
third (13%-31%) of the overall U.S. population (“45 Million,” 1998; “About One in Four,”
1996; “American Internet User Survey,” 1997; “eOverview Report,” 1998; “Florida Is
Among,”1998;“GVU’s Seventh WWW User Survey,”1997;Hoffman,Kalsbeek,&Novak,
1996a;“LatestIntelliquest Survey,”1998; McGarvey,1996;Taylor,1997).Researcherswho
areattemptingtoreachWebusersthroughtraditionalmeans,suchasmailingsandtelephone,
arelikelytoface high nonresponse rates becausemanypeoplestilldo not access the Internet.
A random sample of the general population is likely to yield a disproportionate number of
324 SOCIAL SCIENCE COMPUTER REVIEW
nonusers who are not qualified to respond to questions concerning Web use. An alternative
and effective way in which to reach an audience of Web users is to post an online survey.
However, online surveys do not readily conform to traditional research methods.
Sampling
Byitsnature,theInternetposesauniquesetofproblemsinguaranteeingarandomsample
of respondents. Unlike telephone and mail surveys in which samples can be produced
through census lists random digit dialing, the Web has no central registry of users. In addi-
tion, e-mail addresses are so varied that they are virtually impossible to construct randomly,
making Internet sampling very complex.
Recent efforts have been made to ensure randomness and representativeness in the elec-
tronic environment. Swoboda, Muhlberger, Weitkunat, and Schneeweib (1997) randomly
selected e-mail addresses from 200 newsgroups available through the news server at the
Technical University of Munich. James et al. (1995) e-mailed surveys to a stratified random
sampleof proprietary bulletinboard users. Anderson and Gansneder (1995) obtained a sam-
pling frame from a list of Cleveland Free Net users, and from this list, they generated a ran-
dom sample using the SPSS (Statistical Package for the Social Sciences) sample command.
Penkoff,Colman,andKatzman(1996) constructed their stratified random samplefromindi-
viduals whose electronic addresses were obtained from various Usenet newsgroups.
Swoboda et al. (1997) preceded their questionnaire with a short introductory message.
James et al. (1995) and Anderson and Gansneder (1995) first e-mailed their samples cover
letters of request and then e-mailed the surveys to only those who agreed to participate,
whereas Penkoff et al. (1996) electronically sent their questionnaire unsolicited and without
advance warning. Within a matter of hours after e-mailing the surveys, Penkoff et al. were
“flamed” with critical and hostile comments to the degree that the study was aborted. The
researchers immediately revised their plan and e-mailed potential participants in advance
requesting their permission to e-mail them the survey.
As Swoboda et al. (1997), James et al. (1995), Anderson and Gansneder (1995), and
Penkoff et al. (1996) have demonstrated, a sampling frame can be obtained from Usenet
newsgroups, listservs, chat forums, and other Internet outlets where e-mail addresses are
posted. Then, a selected sample can be e-mailed in advance and asked to complete a ques-
tionnaire. However, surveys sent by e-mail are relatively simple to complete and return,
whereasWeb-basedsurveysrequirerespondentsto access theirWebbrowsers,openthe sur-
vey site, and then complete and return the survey. The more burdensome the task, the lower
the response rate (Babbie, 1990). People are beginning to use newer e-mail software and
Web browser-based e-mail that transform URLs into direct links to Web sites. Potential
respondents who receive e-mail requests to fill out a surveys might be more inclined to par-
ticipate if the survey is only a mouse click away.
Inastudy conducted by BertotandMcClure (1996), respondents were giventheoptionof
completinganelectronic surveyratherthan the printed version. A stratified sample oflibrary
personnel was mailed the survey, which contained a bright orange-colored insert informing
recipientsofthesurvey’sURL.Slightlymorethanhalf(52.3%)ofthe respondents described
the online version as “very easy to complete,” and just under one third (32.6%) thought that
the electronic version was “very time-efficient.” Also, 6 of 23 participants who responded to
thefollow-upquestionnairementioned that they completed boththeprint and electronic ver-
sions and found little difference in time or effort between the two surveys (Bertot &
McClure, 1996).
Kaye, Johnson / IMPROVING ONLINE SURVEYS 325
Bertot and McClure (1996) present a sampling method in which a traditional model of
stratificationisfirstused to select asampling frame and thenis adapted to anelectronic envi-
ronment by mailing the survey’s URL to the selected sample. However, once a survey is
posted to the Web, anyone who stumbles across it can fill it out. To block such access, Bertot
andMcClure (1996) required onlinerespondents to enter asurveyidentificationnumberand
other identifying data, ensuring that only the selected sample could complete the question-
naire. Establishing a password method of entry to a survey site also might prevent respon-
dents from completing the survey more than once.
Nonprobability Sampling
In situations where probability sampling is not feasible, nonprobability sampling is com-
monly used (Babbie, 1990), and there are many instances in which it is the preferred method
(Babbie, 1990; Fowler, 1993; Wimmer & Dominick, 1991). Because there is no mechanism
for random sampling the population of Web users, nonprobability sampling is more appro-
priatewhenpostinganonlinesurvey.Cybersurveys are commonly defined as volunteer sam-
ples because respondents are self-selected (Wu & Weaver, 1997). However, steps can be
takentomake online polls approachpurposivesampling.TheWebisconducivetopurposive
samplinginthatsubsetswithinthelargerpopulationofWebuserscanbeidentified and solic-
ited through announcements and connecting links posted on key online sites and topic-
relateddiscussiongroups(Bimber,1996;Johnson&Kaye,1998).Purposivesampling,how-
ever, generatesresults that are notrepresentativeofthe larger population. Still, carefuluseof
this sampling technique can produce samples that may be representative of a specific subset
of the population (Babbie, 1990; Wimmer & Dominick, 1991).
Beforeonline surveys can trulybe considered purposive sampling, otherchallenges must
be overcome. Response rate is one criterion of the representativeness of the sample respon-
dents (Babbie, 1990). Although nonresponse (e.g., mail surveys that are thrown away) is a
problem with most surveys, the overall response rate can be calculated. However, the
response rate of a Web-posted survey cannot be calculated because there is no way in which
toknow how many individuals might haveseenthe surveyor its linksbutdeclined to partici-
pate. Only the number of completed surveys is known and not the number of refusals. A
counterthatkeepstrackofthenumberoftimesthatasitehasbeenaccessedmaybeoflimited
use in calculating response rate. The number of returned surveys divided by the number of
times the survey was accessed might give researchers an estimate of the response rate. How-
ever, counters typically do not keep track of the number of unduplicated visitors. For exam-
ple, 10 “hits” could mean that each of 10 different individuals was exposed to the survey
once, or it could mean that each of 2 visitors accessed the survey 5 times. Response rate esti-
mates derived from counters are questionable at best.
Also, individuals who choose to complete a cybersurvey might differ from those who
choose not to participate. Voluntary participants might be more interested, informed, and
concerned about the survey topic, and typically they hold viewpoints that are stronger and
more extreme than those of other individuals (Asher, 1995; Rosenthal & Rosnow, 1969; Wu
& Weaver, 1997).
Researchers from the Georgia Institute of Technology’s Graphic, Visualization, and
Usability Center (GVU) have posted nine surveys online, (“GVU’s WWW User Surveys,”
1998), and Bimber (1996) conducted a Web survey examining political attitudes. Both used
nonprobability sampling techniques. Whereas Bimber sought a convenience sample of
respondents who access politically based Web sites, the GVU solicited the general popula-
326 SOCIAL SCIENCE COMPUTER REVIEW
tionof Webusers.To solicitrespondents,Bimber posted the survey’s URL onpolitically ori-
entedWebsites,Usenet newsgroups, and listservsto attract politically interested Webusers,
thereby purposely avoiding advertising to the general Web population (Bimber, 1996;
Bimber,personalcommunication, January 24, 1997).The GVU conducteda more extensive
publicity campaign that included placing rotating banners on search engines and other high-
exposure sites, as well as on neutral-content sites, and posting announcements on Internet-
related newsgroups and in traditional nonelectronic media. As an additional incentive, the
GVU offered several cash rewards ranging between $100 and $250 to those who completed
any of their nine surveys. The publicity efforts seemed to have been worthwhile; Bimber
received6,000completedresponsesduringthe 6 months online,andtheGVU’sninthsurvey
recorded more than 12,000 completions after 6 weeks online (Bimber, 1996; “GVU’s Ninth
WWW User Survey,” 1998).
A very tangled Web is being woven, and although it is witnessing a tremendous increase
in the numbers of users and sites, there are no data describing how many different sites are
visitedbyeach individual.Studies report that television viewers tend to establish a repertoire
of channels they watch most frequently (Ferguson, 1992; Heeter, D’Allessio, Greenberg, &
McVoy,1988).Thus,itispossiblethatWebusersalsomighthavearepertoireoffavoritesites
fromwhich they rarely stray.Unlikean environmentof35 to 60 televisionchannels,theWeb
containstens of thousands of“channels.”Thus,publicizingasingle surveyonline might be a
formidable task.
METHOD: POLITICAL SURVEY
Posting the Survey on the Web
Theauthors placed asurveyon the Webto examine the extentto which the Web and other
media sources affect voting behaviors as well as the relationship between use of the Web for
political information and feelings of alienation and political interest. The Internet, analysts
argue,canreconnectcitizenstothe political system. WebuserscanaccesspoliticalWebsites
and can send and receive messages directly from candidates or their campaigns. Usenet
newsgroups,electronicbulletinboards,andchatforums create a virtual community inwhich
citizensfromall overthe world can voice their opinions onissuesandcandidates (Johnson &
Kaye,1998). This studyalso allowed us to exploretheutility,merits,and drawbacks of post-
ing a survey on the Web.
Web Survey Design
We first designed the survey using the traditional paper-and-pencil format and then
adapted it to the Internet. The standard Internet protocols for the development of this project
included hypertext markup language (HTML) and an additional common gateway interface
(CGI)scriptwritten in Perl. These toolsgaveusthebasicframeworknecessaryfordesigning
an interactive online survey. The survey consists of 20 questions, most using a Likert-type
scale. The survey was developed in such a way that participants could enter a response by
clicking on a select a response drop-down box and then choosing one of the answer options.
Adrop-downboxsavesspaceand clutter by concealingtheresponses until the boxisclicked
on. Open-ended responses were simply typed in highlighted boxed areas. Completed sur-
veys were e-mailed to the researchers by clicking on the send survey button, after which
Kaye, Johnson / IMPROVING ONLINE SURVEYS 327
another Web page immediately would appear thanking the respondents for their participa-
tion and verifying that their surveys had been returned.
Originally, the paper survey was six pages in length and could quickly be completed by
simply circling the appropriate responses. However, to fill out the cyber version, each drop-
downboxforeach question had to beclickedandthenthe response selected; therefore, when
displayed on Netscape Navigator, the survey took longer to complete than was anticipated.
Realizing this, we shortened the survey’s length by deleting several questions that we felt
wereancillaryto the overall project. Although we didnotconductformal tests measuring the
length of time to complete the survey, it took each of us longer to fill out the online version
than to fill out the paper version.
When designing a cybersurvey, various technical issues arise. Different Web browsers
have varying display capabilities. For example, one long line of text on one browser might
wrap into two lines on another browser, causing a survey to lose its formatting. Display of
fontsizes and styles,as well asimage alignment, placement,sizes, and color,mayvaryfrom
browser to browser, which also might cause design problems (Weinman, 1996). In addition,
large graphics (GIFs) might slow downloading time. Although 28.8-Kbps (kilobits per sec-
ond) and higher modems are commonly used today, many individuals still have slower Web
access (Hafner & Croal, 1996).
With these considerations in mind, we designed our survey with a neutral background
color (light blue) and two small GIFs: a donkey and an elephant. We attempted to keep lines
of text and questions as short as possible, and we designed the overall survey such that if
wrapping did occur, the formatting would be only minimally disturbed.
Sampling
ThesurveywaspostedontheWeb fromOctober23throughNovember20,1996.Thesur-
vey was online for 2 weeks before and 2 weeks after the presidential election, when political
Web sites and news sites were expected to be most active. For example, on Election Night,
CNN Interactive had 50 million requests for information (“Slowdown of the World Wide
Web,” 1996).
We sought to draw conclusions from the population of Web users who are interested in
politicsand who turnto the Webfor political information. Reachingthis subset of Web users
is formidable because only 6% of Americans used the Internet for political information dur-
ing the 1996 presidential campaign (“New Media,” 1996). Thus, placing the survey online;
linking it to politically oriented Web pages; and posting the URL with political listservs,
Usenet newsgroups, and chat forums was appropriate to reach this convenience sample.
The respondents to the political survey were demographically similar to those of other
Web studies. Like other surveys, this one suggests that the Internet is dominated by young
White males with high education levels and high socioeconomic status. The average age of
respondents to this survey was 31.2 years, and the gender ratio was 75.5% male and 24.5%
female. The American Internet User Survey (conducted by telephone) and the GVU Ninth
WWW User Survey (conducted online) both found that the average Internet user is a male
(64.1% and 61.3%, respectively) in his 30s (35.2 and 35.5 years of age).
In addition, 6 out of 10 of our respondents had a college degree or higher, slightly under
half (45.5%) reported an annual income between $25,001 and $65,000, and nearly 9 out of
10 (88.3%) were White. About half (50.1%) of the respondents in the GVU Ninth User Sur-
vey had a college education or higher, and the average income was $52,000. In addition, the
clear majority of respondents in both the GVU poll and the American Internet User Survey
identified themselves as White. Finally, our finding that the typical Internet user spends an
328 SOCIAL SCIENCE COMPUTER REVIEW
average of 13.2 hours per week on the Internet compares favorably with a recent study that
put the number at 13.6 hours (“What’s Your Daily Dose?,” 1997).
Publicity
AlistofpoliticallybasedWebsiteswascompiledfromInternetarticlesintraditionalpub-
licationsandfromonline searches. Once the survey was online, Webmastersof19politically
oriented Web sites were e-mailed requesting that links be established from their pages to the
politicalsurvey;of these, 9 agreed1and e-mailed confirmations. The NewYork Timesonthe
Web andMSNBC sites also createdlinksto the survey.(AnMSNBCreporter solicited us for
an interview about the survey, which was granted in return for a link from MSNBC’s online
site.) In addition, two Webmasters e-mailed us, asking permission to link the survey to their
political pages; we agreed.2
Web searches yielded hundreds of politically oriented listservs, Usenet newsgroups, and
chat forums. The TILE.NET/LIST Web site,3which indexes listservs, was used to find those
dealing with politics and the Internet. Messages also were sent to two journalism listservs
andtoour university’sInternetclublist,yieldingatotalof11 listservs.4Lists of names of dis-
cussion forums and Usenet groups were obtained from the Vote Smart Web page.5An Alta-
Vista Usenet search using keywords such as “1996 election,” “Clinton,” “Dole,” “politics,”
and “Internet” yielded additional sources. When announcements were posted to these dis-
cussiongroups,thesurvey’scompleteURLwasincludedsothatinterestedindividualscould
click on it to access the survey.
Carewas taken to contact groups representing a wide variety ofpolitical ideologies when
selecting from among the various listservs, newsgroups, and chat forums. Thus, we sent
announcements to diverse groups such as alt.newt.grinch.grinch.grinch, alt.politics.green,
and alt.fan.rush-limbaugh. Although studies indicate a demographic mainstreaming among
Web users, the Internet still may be dominated by White males, so ethnic-oriented and alter-
native lifestyle Usenet newsgroups, such as soc.culture.African.American and alt.poli-
tics.homosexuality, also were selected.
Announcementsweresent to 33 Usenet newsgroups,610listservs,and6chatforums7on a
rotating basis throughout the 4 weeks of data collection. After a few weeks, one discussion
group felt that it was being “spammed” by the repeated announcements of the political sur-
vey. The punishment for spamming usually consists of being “flamed” with hundreds of
replies; however, we were barely “singed” when we received one hostile comment and were
restricted from accessing that Usenet newsgroup.
Our cybersurvey also was listed with most of the major search engines such as Yahoo!,
AltaVista, and Web Crawler. Listings were conducted through Submit It,8which is an online
site that registers URLs with many of the major search engines with one entry rather than
contacting each one separately.
The political survey’s final question asked respondents for the name of the Web site, list-
serv, Usenet group, or other means from which they found out about the survey. From the
responses, we were able to gauge which means of publicizing the survey was most effective.
In this case, just under one third (32.1%) discovered the survey through announcements
posted to chat forums, especially those that serve both as forums and as Web sites. Slightly
under one in four (22.7%) respondents clicked to the survey via hot links from other Web
sites. Usenet newsgroups and listservs each drew about one sixth of the respondents. The
remaining respondents (14.2%) reported discovering the survey through other sources such
as search engines, e-mail, and word of mouth.
Kaye, Johnson / IMPROVING ONLINE SURVEYS 329
Data Collection and Responses
After 4 weeks online, 306 completed surveys were returned. Of these, 165 were returned
priorto the 1996 presidential election and 141 during the 2 weeks followingthe election. On
receipt of each completed survey, e-mail software listed the date and time the questionnaire
was received and the sender’s Internet server address. In addition, the survey’s first question
askedrespondentstoentertheir e-mail address; of the 306 respondents,276complied.These
Internet server and e-mail addresses were used to delete duplicated surveys and to track the
number of surveys received each day.
It should be noted that surveys are not returned in the same format as displayed on
NetscapeNavigator. Thesurveysweredesignedtoreturnviae-mailwitheachquestionlisted
inorder followed by itsresponse (text and numericalvalue).Receivingresponses as text and
as the corresponding numerical value facilitated data entry into SPSS.
Thetotal number ofindividualswho saw the survey or itsURL but declined to participate
is unknown. Therefore, the number of refusals cannot be calculated. Follow-up requests to
complete an e-mail questionnaire can be sent to a sample to increase the response rate (Kit-
tleson, 1997). However, there is no way in which to follow up with surfers who might have
comeacross a Websurveybutdeclinedto participate. An additionalproblemof doing online
surveys is that the researchers might not always receive all of the surveys that are returned.
Forexample,withthreeofthepolitical surveystheelectronic transmissions were interrupted
so that they were only partially returned. In an attempt to correct the problem, we contacted
one respondent asking him to resend the survey. However, again the survey came back only
partially intact. Although we consulted with several Internet experts, none could identify or
solve the problem.
DISCUSSION: RECOMMENDATIONS FOR WEB SURVEYS
The Web’s rapid growth rate and demographic shift toward a more mainstream audience
makesitanappealingsurveytool.Thepastfewyearshavewitnessedaproliferationofonline
surveys that challenge traditional survey methods. Before the Web can become a viable sur-
vey instrument, a number of methodological hurdles must be overcome. Some of the major
concerns of online surveying include sampling, response rate, generalizability, response
duplication, and survey design. However, recent studies have shown that if certain proce-
dures are followed carefully, the Web can be considered a useful survey instrument for tar-
geted online populations.
Web Survey Design Considerations: Discussion
When designing a cybersurvey, various technical issues arise concerning downloading
speed as well as text and graphic display. Bearing these considerations, we designed a basic
text survey with only two small GIFs and one background color. We only had access to
NetscapeNavigator,sowedidnothavetheopportunitytoviewthesurvey onotherbrowsers.
However, we were not informed of any browser-related design flaws.
Our response rate might have been affected by the survey’s length. Unfortunately, we
havenoway of knowing whether anyoneabandoned the survey because itwas too long, tire-
some, or difficult to complete. As with traditional paper surveys, online questionnaires must
be easy to complete, short, and concise.
330 SOCIAL SCIENCE COMPUTER REVIEW
Web Survey Design Considerations: Recommendations
1. The survey should be as short as possible for quick completion and to minimize excessive
scrolling.
2. Simple designs with sparse use of graphics save downloading time.
3. Drop-down boxes save space and clutter by avoiding repeating responses.
4. Instructions should be clearly stated.
5. Responding to questions should be easy and intuitive.
6. Pretests should be conducted to measure length of time and ease of completion.
7. A check of the survey using different browsers will uncover any browser-based design flaws.
Sampling: Discussion
The difficulties of online sampling stem from the lack of a central registry of Web users.
Without a sense of the online population, constructing a sampling frame and reaching a spe-
cific sample is problematic. Some online surveys seek respondents from the overall popula-
tionof Webusers but then attemptto generalize the results as public opinion,whereas others
generalize just to Web users. More sophisticated techniques move toward a more purposive
method.
The political cybersurvey we conducted, as well as other Internet research (Anderson &
Gansneder, 1995; Bimber, 1996; James et al., 1995; Penkoff et al., 1996; Swoboda et al,
1997), has adapted traditional sampling techniques to an online environment. For e-mail-
distributed surveys, Swoboda et al. (1997), James et al. (1995), Anderson and Gansneder
(1995), and Penkoff et al. (1996) constructed stratified random samples from newsgroups,
bulletin boards, Free-Nets, and Usenet addresses, respectively. Seeking a specific subset of
politically interested Web users, we moved toward purposive sampling by posting links on
politically oriented Web pages and by announcing the survey with politically based Usenet
newsgroups, listservs, and chat forums (see also Bimber, 1996). Although these techniques
will not allow generalizations to the overall public or even to the population of Web users,
they may be representative of the specified targeted audience.
In addition, sampling methods are confounded by unrestricted access to online surveys.
In general, once a survey is posted, it can be filled out by anyone who happens to find it. For
example, our survey could have been completed by Web users who came across the site by
chanceand who mighthavehad little interest inpolitics. Toovercomethislimitation, a pass-
word method should be devised where only individuals from a selected sample can access
the Web survey.
Another obstacle in reaching a selected sample frame is that it is virtually impossible to
knowhowmanyWebsitesarelinked to the surveyandfor what period oftime. For example,
our survey’s URL was circulated among various discussion-type outlets and on several Web
pages that we had not contacted. Apparently, Webmasters who were aware of the political
survey took it upon themselves to link the survey to their pages, and word of the survey
spread among various discussion groups. Even with a daily search, there is no way in which
to know exactly how many sites might have linked to the survey or how long the URL
remained on certain Web sites. And with Usenet newsgroups, listservs, and chat forums,
unlessparticipants notice thesurveyannouncement the day on which it isposted, it mightbe
overlooked.
Also, the World Wide Web is indeed worldwide. As one Scandinavian pointed out to us,
people living anywhere on the planet can answer a survey posted on the Web even if they are
not part of the intended audience. The Internet population, then, is very inclusive, so care
Kaye, Johnson / IMPROVING ONLINE SURVEYS 331
must be taken in constructing surveys to take this into account. Perhaps the intended audi-
ence needs to be very clearly identified in the survey’s introduction.
Ourexperience, and theexperiencesof others, suggeststhat there aresome methodologi-
calproblems(e.g.,lackofcontrol overrespondentsandoverthenumberandtypeofsitesthat
linktothesurvey)thatstillneedtobeovercomebeforeonlineresultscanbegeneralized even
tothe population of Web users.However, bycarefully defining a population,restricting Web
links, and posting notices only to discussion-type groups likely to be accessed by the target
audience,researchersaremorelikelytoreacharepresentative,albeitself-selected,sampleof
their intended audience.
Sampling: Recommendations
1. Toincrease representativeness, definesamplesassubsetsofWeb users based onspecificcharac-
teristics.
2. Solicitrespondentsbylinkingthesurveyfromkey online sites and bypostingannouncementsto
discussion-type groups that are likely to be used by the targeted population. Or, select a sam-
plingframefrome-mailaddressespostedonkeyUsenetnewsgroups, listservs, and chat forums.
E-mail the selected sample a request to complete the questionnaire along with an identification
number and password required for accessing the Web-based survey.
3. TheWorld WideWebistruly worldwide, and individualsfromanycountrycancompleteaques-
tionnaire.Thus,clearlystatetheintendedaudience of respondents inthesurvey’s introduction.
Publicity: Discussion
After our survey was online for 1 day, we realized that merely posting links to it from
politicallyoriented Websiteswasnotenough to create awarenessortopersuade our targeted
sample to complete the survey. Other means of publicity were needed including posting
announcements on Usenet newsgroups, listservs, and chat forums. Once the initial round of
announcements was made and a surge of completed surveys was received, the responses
ebbed. Out of necessity, publicizing the survey became a daily occurrence throughout the
4-week period. Unlike the GVU surveys of 1994-1998, it was not possible for us to offer
financial incentive or to pay for banner ads or traditional advertising.
Although it is possible to estimate the number of Web sites and the number of Web users,
thenumberof differentsites that each user accesses isunknown.Eachindividual’srepertoire
ofWebsitesvisitedon a regularbasisis unique, andthe number of sitesmight be limited toa
smallset, making itdifficultto know on whichWebsitesto publicize a cybersurveytomaxi-
mize response. A combination of financial incentive, online and traditional advertising, and
online public relations and marketing efforts might be needed to attain a sizable number of
responses.
Publicity: Recommendations
1. Devise a method to systematically publicize the survey daily through various means. In other
words,donotspamafewdiscussiongroupswhileignoringothers.To reduce bias, create aware-
ness from a wide variety of Internet outlets.
2. Listthesurveywithasmanyofthemajorsearchenginesaspossible.Web sites such as Submit It
facilitatethisprocessbysendinglistingsto many search engineswithjustoneentry. After listing
the survey, try locating it by using different search strategies and terms.
3. Whensending announcements about the survey,writethe entire URL in the message.In Usenet
newsgroup postings and in some e-mail tranmissions, the URL becomes a clickable link.
4. Take care not to get flamed. Do not oversell the survey; just announce it.
332 SOCIAL SCIENCE COMPUTER REVIEW
5. Follow up confirmations of survey links to gauge how long the URL is posted and whether it is
visible on the page. Remember that if it is difficult for the researchers to find the survey’s URL,
then others probably will overlook it as well.
6. Asking respondents how they found out about the survey is an excellent way in which to gauge
the most effective sites and discussion outlets.
7. Placing banner advertisements on selected sites might increase the number of completions.
8. Offer incentives for completing the survey. The incentives can be as simple as promising the
respondents the results or as alluring as GVU’s lottery system (which rewards winning respon-
dents with cash). The types of incentives offered clearly depend on the researchers’budgets.
9. A combination of financial incentives, online and traditional advertising, and public relations
and marketing efforts might be needed.
Data Collection and Responses: Discussion
Cybersurveys should be easy to return; our respondents merely had to click on the send
survey button. We designed the survey so that, on return, each question would be listed on a
separate line followed by the written response and the corresponding numerical value to
facilitate data entry into SPSS. Although there are other online data collection sites9where
responsesarecountedandsimplepercentages calculated immediately after surveysarecom-
pleted, we opted not to use this method because our data require more sophisticated statisti-
cal analyses.
Once our political survey was online, linked with other politically oriented sites, and
listed with the major search engines, we waited for the onslaught of completed question-
naires.However, theycame in at averyslow trickle. Wewere prepared for—and expected—
thousandsofresponses, and we were disappointed withthelowrateofreturn. At the time our
surveywasposted, the Web wasestimated to contain 16 million to 50 million pages of infor-
mation(Caruso, 1996; Croal& Stone, 1996), sothe odds ofsomeone coming across oursur-
vey that was linked to only a few pages was highly improbable. Even links on more popular
sites with thousands of visitors a day did not guarantee completions. Thus, we immediately
began our daily announcements to Usenet newsgroups, listservs, and chat forums, dramati-
cally increasing the number of responses.
We attemptedtopost a consistent numberofdaily announcements in a systematicfashion
to discussion-type outlets. We exercised care, however, so that we were not overzealous in
our efforts to reach the groups’ discussants and inadvertently spam them with too many
announcements.
Online surveys can be particularly vulnerable to duplication. With unlimited access, an
individual might feel compelled to complete a survey a number of times, or a respondent
might inadvertently return the survey more than once or do so purposely just to make sure it
wasindeedtransmitted. Tominimizeredundantcompletions, the first question onthe survey
asked respondents to enter their e-mail addresses. As a double check, the Internet protocol
address was listed at the top of each returned survey. A separate list consisting of both the
e-mail and the Internet protocol addresses was compiled and checked for duplication, and
repeated surveys were discarded. In addition, after the send survey button was clicked,
anotherWebpageautomaticallycame up containing a“thankyou” message andverification
that the survey was successfully returned, further decreasing the likelihood of respondents
resendingthesurvey. Althoughwedidnotkeeptrack of the number of duplicatedsurveyswe
discarded, we did notice that in nearly every instance, multiple surveys were sent within 1 or
2 minutes of each other. The quick retransmission rate indicates that, despite the thank you
message, respondents were unsure of whether the first electronic mailings were successful,
so they sent the survey again.
Kaye, Johnson / IMPROVING ONLINE SURVEYS 333
Online surveys’response rates also might be influenced by a general mistrust of online
surveys and a reluctance to share opinions and preferences in a nontraditional environment
(Swoboda, et al., 1997). A low response rate also might be attributed to the nontangible
nature of the Internet. Individuals might intend to fill out an online questionnaire at their lei-
sure, bookmarking the survey’s URL for later use, but then forget all about it. By compari-
son,traditionalpapersurveysoftenareleftonthepotentialrespondent’sdesk,whereitspres-
ence serves as a constant reminder (Kittleson, 1997).
Inaddition,unliketraditionaldatacollectionmethodsthat usually are free to respondents,
some Web users might have to pay for their time online and, therefore, could be unwilling to
spend valuable minutes filling out a survey, further hampering the number of responses
(James et al., 1996). Not only does this influence the response rate, it also might introduce
additional bias based on income.
Cybersurveys also limit nonresponse calculations. It is not possible for us to know how
many online individuals declined to participate in our survey. The number of people who
came across our site but decided not to complete our survey is unknown. Currently, the tech-
nological means to gather those data do not exist.
Data Collection and Responses: Recommendations
1. Ask for respondents’ e-mail addresses to check for duplication. If the e-mail addresses are not
given, then keep track of the senders’Internet protocol addresses.
2. Surveys should be easy to return with just the click of a mouse button. A thank you or other type
of verification page should come up on the sender’s screen on returning the survey so that the
respondent is not left wondering whether the survey was indeed transmitted.
3. When the survey returns as an e-mail message, it should be designed so that it returns with each
question listed on one line followed by its response and a corresponding numerical value. This
makesiteasyforresearchersto eyethedata and facilitates coding the surveysandenteringthem
into a statistical software program.
THE FUTURE OF ONLINE SURVEYS
Traditional sampling methods need to be reformulated and new ones constructed to
accommodate the Web’s unique features before it can be considered a viable survey instru-
ment.Asan increasing number ofpeoplediscovertheWebandbecomefrequentandcompe-
tentusers,theWebwillbecomemorerepresentativeofthegeneralpopulationand,therefore,
survey results will be more reliable and generalizable.
Many reputable marketing and research firms are surveying Web users to examine topics
such as online shopping habits, hours spent online, most frequently accessed Web sites, atti-
tudes toward the Web, and other general knowledge about the Web and its users. Jupiter
Communications, FIND/SVP, Odyssey Communications, Nua Ltd., GVU, Intelliquest, ZD
Market Intelligence, Forrester Research, Business Wire, Louis Harris and Associates, and
Mediamark Research are just a few firms and organizations that have conducted surveys
about the Web. Whereas many of these companies reach their respondents through tradi-
tional means, some (e.g., GVU, Business Wire) have posted surveys online.
Witheasyaccess,immediatefeedback, low cost, and verifiabledelivery,it is possible that
Web-based questionnaires might some day vie with traditional survey vehicles. But before
thiscanhappen,techniquesfordrawingrandomandrepresentativesamplesmustbeinplace.
A type of clearinghouse or online “yellow pages” containing listings of e-mail addresses of
Web users would boost online sampling by providing a base of potential respondents.
Whereas most proprietary services do not release their clients’ online addresses, others do
334 SOCIAL SCIENCE COMPUTER REVIEW
(e.g., universities list faculty and student e-mail addresses in the university telephone books
for sampling of those particular populations). Randomly constructing telephone numbers is
a commonly used sampling technique, and perhaps one day new software will allow a ran-
dom construction of e-mail addresses, although this seems an impossibility at this point in
time. In addition, methods of counting unduplicated site visitors would aid in calculating
response rates and would increase survey reliability.
Cybersurveyscanbe excellent vehicles forreaching a targetedgroup of Webusers. How-
ever, untilthecyberenvironmentismademoreconducivetoonlinestudies, researchers must
take certain limitations into account when designing online surveys and when interpreting
the results.
NOTES
1. Comeback.com: http://www.comeback.com/; Election Nexus: http://dodo.crown.net/
~mpg/election/96.html; MSNBC: http://www.msnbc.com/news/default.asp/; New York
Times on the Web: http://www.nytimes.com/; NewPolitics: http://www.newpolitics.com/;
Politics.com:http://www.politics.com/;PoliticsNow:http://politicsusa.com/;Resourcesin
Political Science (West Virginia University): http://www.polisci.wvu.edu/polycy; The Daily Egyp-
tian: http://www.dailyegyptian.com/; The Jefferson Project: http://www.voxpop.org/
jefferson/
2. Whitehouse96: http://www.niu.edu/newsplace/whitehouse.html
3. TILE.NET/LIST: http://www.tile.net/tile/listserv/index.html
4. Listservs: ACTIV-L, CARR-L, COMSERV, DEMOCRATS96, JOURNET, NCTETALK, POLCOMM,
POLITICS, PSRT-L, PUBLIC RELATIONS, SIU.EDU
5. Vote Smart: http://www.vote-smart.org/other/participate.html
6. Usenet newsgroups: alt.conspiracy, alt.fang-gordon-liddy, alt.politics, alt.politics.bush, alt.politics.clinton,
alt.politics.correct, alt.politics.datahighway, alt.politics.democrat, alt.politics.economics, alt.politics.elections, alt.
politics.green, alt.politics.homosexual, alt.politics.libertarians, alt.politics.marrou, alt.politics.misc, alt.politics.
newt.grinch.grinch.grinch, alt.politics.org, alt.politics.perot, alt.politics.radical.left, alt.politics.usa.congress, alt.
politics.usa.constitution, alt.politics.youth, alt.rush-limbaugh, alt.total.internet, bit.listserv.politics, soc.culture.
african.american, soc.culture.asian.american, talk.politics.crypto, soc.culture.mexican.american, soc.politics,
talk.politics, talk.politics.libertarian, talk.politics.theory
7. Chat forums: Election96, George Magazine, Opinion Page, Politics Now, TownHall, Utne Cafe
8. Submit It: http://www.submit-it.com/
9. The Electronic TownHall: http://www.electronic-townhall.com/
REFERENCES
About one in four adults has access. (1996). 1996 National Omnibus Survey. University of Maryland Survey
Research. [Online]. Available: http://www.bsos.umd.edu/src/
American Internet User Survey. (1997). FIND/SVP: Emerging Technologies Research Group. [Online]. Available:
http://www.etrg.findsvp.com/internet/findf.html [January 7]
Anderson,S.E.,&Gansneder, B. M.(1995).Usingelectronicmail surveys andcomputer-monitoreddataforstudy-
ing computer-mediated communication. Social Science Computer Review,13, 34-47.
Asher,H.(1995).Pollingandthepublic:What every citizen shouldknow(3rded.).Washington,DC:Congressional
Quarterly.
Babbie, E. (1990) Survey research methods. Belmont, CA: Wadsworth.
Bertot, J. C., & McClure, C. R. (1996). Electronic surveys: Methodological implications for using the World Wide
Web to collect survey data. In S. Hardin (Ed.), Global complexity: Information, chaos, and control (pp. 173-
192). Medford, MA: Information Today.
Bimber, B. (1996). Government and Politics on the Net Project. University of California, Santa Barbara. [Online].
Available: http://www.polsci.ucsb.edu/~bimber/research/
Bimber, B. (1998). The Internet and political transformation: Populism, community and accelerated pluralism.
Polity,31, 133-60
Caruso, D. (1996, January 29). Technology. The New York Times,p.C3.
Kaye, Johnson / IMPROVING ONLINE SURVEYS 335
Croal, N., & Stone, B. (1996, May 27). Cyberscope: More sites. Newsweek,p.10.
December, J. (1996). Units of analysis for Internet communication. Journal of Communication,46(1), 14-38.
EOverviewreportindicates: Only 37millioncurrentlyonline in U.S.(1998,July8). Business Wire[news release].
Ferguson, D. A. (1992). Channel repertoire in the presence of remote control devices, VCRs, and cable television.
Journal of Broadcasting and Electronic Media,36(1), 83-91.
Floridaisamongfastest-growingInternet states: California leads intotalnumberofInternetusers. (1998, March 3).
CommerceNet and Neilsen Research homepage. [Online]. Available: http://www.commerce.
net/news/press/030398_1.html [October 29]
45 million U.S. PCs access the Internet regularly. (1998, May). ZDMarket Intelligence homepage. [Online]. Avail-
able: http://www.zdintelligence.com/news/ [October 29]
Fowler, F. J., Jr. (1993). Survey research methods. Newbury Park, CA: Sage.
GVU’s ninth WWW user survey. (1998). Georgia Institute of Technology’s Graphic, Visualization, and Usability
Center.[Online].Available:http://www.cc.gatech.edu/gvu/user_surveys/survey-1998-
04/
GVU’sseventh WWW user survey. (1997). Georgia InstituteofTechnology’sGraphic,Visualization,and Usability
Center.[Online].Available:http://www.cc.gatech.edu/gvu/user_surveys/survey_1997/
Hafner, K., & Croal, N. (1996, March 4). Getting up to speed. Newsweek, pp. 46-47.
Heeter, C., D’Allessio, D., Greenberg, B., & McVoy, D. D. (1988). Cableviewing behaviors: An electronic assess-
ment. In C. Heeter & B. Greenberg (Eds.), Cableviewing (pp. 51-63). Norwood, NJ: Ablex.
Hoffman,D.L.,Kalsbeek,W. D.,&Novak,T. P.(1996a). Internet use in the United States:1995baselineestimates
and preliminary market segments. [Online]. Available: http://www.2000.ogsm.vanderbilt.
edu/baseline/1995.internet.estimates.html
Hoffman, D. L., Kalsbeek, W. D., & Novak, T. P. (1996b). Internet and Web use in the United States: Baselines for
commercial development. [Online]. Available: http://www.2000.ogsm.vanderbilt.edu/
papers/Internet_demos_july9_1996.html
James, M. L., Wotring, C. E., & Forrest, E. J. (1995). An exploratory study of the perceived benefits of electronic
bulletin board use and their impact on other communication activities. Journal of Broadcasting and Electronic
Media,39(1), 30-50.
Johnson, T. J., & Kaye, B. K. (1998). The Internet: Vehicle for engagement or a haven for the disaffected? In T. J.
Johnson,C.E.Hays,& S. P.Hays (Eds.), Engagingthepublic:Howthe government and mediacanreinvigorate
American democracy (pp. 123-135). Lanham, MD: Rowman & Littlefield.
Kittleson, M. J. (1997). Determining effective follow-up on e-mail surveys. American Journal of Health Behavior,
21, 193-196.
Latest Intelliquest survey reports 62 million American adults access Internet/online services. (1998, February 5).
Intelliquest homepage. [Online]. Available: http://www.intelliquest.com/press/ [October 29]
Lock,J.J.(1995). The Internetasamass medium: Themediaenterthe world ofcyberspace.Feedback,36(4),7-10.
Markus, M. L. (1990). Toward a “critical mass” theory of interactive media. In J. Fulk & C. Steinfield (Eds.),
Organizations and communication technology (pp. 194- 218). Newbury Park, CA: Sage.
McGarvey, J. (1996, January). Latest Net study: 9.5 million active surfers. Interactive Week,p.9.
Morris, M., & Ogan, C. (1995). The Internet as mass medium. Journal of Communication,46(1), 39-50.
Neufeld, E. (1997, May 5). Where are audiences going? MediaWeek, pp. S22-S29.
New media. (1996). The Media & Campaign ’96 (Briefing No. 4). New York: Media Studies Center.
Newhagen,J.E.,& Rafaeli,S.(1996).Whycommunication researchers shouldstudytheInternet: A dialogue.Jour-
nal of Communication,46(1), 4-13.
Penkoff, D. W., Colman, R. W., & Katzman, S. L. (1996, May). From paper-and-pencil to screen-and-keyboard:
Toward a methodologyforsurveyresearchontheInternet. Paper presentedattheannualconference of theInter-
national Communication Association, Chicago.
Rogers, E. M. (1986). Communication technology. New York: Free Press.
Rosenthal, R., & Rosnow, R. L. (1969). Artifact in behavioral research. New York: Academic Press.
SlowdownofWorldWideWebmaylead to bigger“backbone.”(1996, November8).St.Louis PostDispatch,p.A8.
Swoboda,W. J., Muhlberger,N.,Weitkunat, R.,&Schneeweib,S.(1997). Internet surveysbydirectmailing.Social
Science Computer Review,15, 242-255.
Taylor, C. (1997, July 5). Net use adds to decline in TV use: Radio stable. Billboard,p.85.
Weinman, L. (1996). Designing Web graphics. Indianapolis, IN: New Riders.
What’s your daily dose? (1997, November). PC Magazine,p.9.
Why Internet advertising. (1997, May 5). MediaWeek, pp. S8-S13.
Wimmer, R. D., & Dominick, J. R. (1991). Mass media research. Belmont, CA: Wadsworth.
336 SOCIAL SCIENCE COMPUTER REVIEW
Wu, W., & Weaver, D. (1997, Fall). Online democracy or online demagoguery: Public opinion “polls” on the Inter-
net. Harvard International Journal of Press/Politics,2, 71-86.
Barbara K. Kaye (Ph.D., Florida State University, 1994) is an assistant professor in the Department of
Communication Arts at Valdosta State University. She is coauthor of the textbook, The World Wide Web: A
MassCommunicationPerspective.Herresearch interestsincludetheWeb and mediaeffects.Herpublished
researchhas appeared in Journalism &MassCommunicationQuarterly,JournalofBroadcastingandElec-
tronic Media,Journal of Promotion Management, and New Jersey Journal of Communication. She may be
contacted by e-mail at
bkaye@valdosta.edu
.
Thomas J. Johnson (Ph.D., University of Washington, 1989) is an associate professor in the School of Jour-
nalism at Southern Illinois University at Carbondale. His fields of specialization are public opinion and
political communication research, particularly media coverage of presidential elections. He has published
onebook,TheRehabilitation of RichardNixon:TheMedia’s EffectonCollective Memory. Heiscoeditorof
Engaging the Public: How Government and the Media Can Reinvigorate American Democracy, which was
written in conjunction with the Simon Public Policy Institute.
Kaye, Johnson / IMPROVING ONLINE SURVEYS 337
