REPORTING AND ANALYZING DATA FROM A MULTI-REGION
RESEARCH SURVEY
Related Application
This application claims priority to U.S. Provisional Application number 60/164,585, entitled "System and Method for Obtaining and Collating Survey Information in Real Time for Multiple Languages and Multiple Character Encodings," filed on November 10, 1999, which is hereby fully incorporated by reference. Field of the Invention
The present invention relates to the field of data processing. More particularly, the present invention relates to reporting and analyzing data from a multi-region research survey. Background
With increased globalization of commerce, market researchers increasingly have to conduct market research that involves multiple regions of the globe. Creating and conducting such a survey, and collecting, analyzing, and reporting data from such a survey present a number of challenges that otherwise do not exist in the case of single region market research studies. That is, multi-region surveys necessarily involve diverse panelists who may speak different languages, have cultural differences, different price point sensitivity, and so forth.
Historically, market researchers would develop and implement a survey in one region, like the United States, and then work with local counterparts in the various regions to have the survey translated and conducted. Under the prior art, it was often difficult to coordinate the creation of the survey so as to ensure consistency in the collected data for consolidation, analysis, an reporting.
Thus, an improved approach to the creation and administration of a multi-region survey, as well as data collection, analysis, and reporting from such a survey is desired.
SUMMARY OF THE INVENTION
A research service includes a number of functions for reporting and analyzing data from a multi-region research survey. The service provides an
analysis interface to a research client to receive client input using a regional communication mode corresponding to the research client. Based on the client input, the service analyzes data from an on-line survey of a panelist. The online survey uses a second, distinct regional communication mode corresponding to the panelist. Analysis results are provided to the client in the analysis interface, again using the regional communication mode corresponding to the research client. In various embodiments, the analysis service similarly provides for analysis of survey data from multiple panelists using multiple regional communication modes.
BRIEF DESCRIPTION OF THE DRAWINGS
Examples of the present invention are illustrated in the accompanying drawings. The accompanying drawings, however, do not limit the scope of the present invention. Similar references in the drawings indicate similar elements.
Figure 1 illustrates one embodiment of the present invention.
Figure 2 demonstrates one embodiment of the present invention.
Figure 3 illustrates one embodiment of an identification window.
Figure 4 illustrates one embodiment of a survey list.
Figure 5 illustrates one embodiment of a survey summary.
Figure 6 illustrates one embodiment of an analysis report for a particular response variable.
Figure 7 illustrates one embodiment of an analysis report for demographic information.
Figure 8 illustrates one embodiment of an analysis report for semantic differential questions.
Figure 9 illustrates one embodiment of a three dimensional analysis report.
Figure 10 illustrates one embodiment of a price point analysis report.
Figure 11 illustrates one embodiment of a computer system to perform various operations of the present invention.
Figure 12 illustrates one embodiment of a machine readable medium to store instructions to implement various operations of the present invention.
Figure 13 illustrates one embodiment of a text coding analysis report.
Figure 14 illustrates one embodiment of a configuration window tor a text coding analysis report.
DETAILED DESCRIPTION
In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, those skilled in the art will understand that the present invention may be practiced without these specific details, that the present invention is not limited to the depicted embodiments, and that the present invention may be practiced in a variety of alternate embodiments. In other instances, well known methods, procedures, components, and circuits have not been described in detail.
Parts of the description will be presented using terminology commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. Also, parts of the description will be presented in terms of the execution of operations. As well understood by those skilled in the art, these operations often take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, and otherwise manipulated through, for instance, electrical components.
Various operations will be described as multiple discrete steps performed in turn in a manner that is helpful in understanding the present invention. However, the order of description should not be construed as to imply that these operations are necessarily performed in the order they are presented, or even order dependent. Lastly, repeated usage of the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may.
Overview
Generally, a survey collects better data when it is administered in the panelist's native mode of communication including, for instance, the panelist's native language, currency, and cultural customs. In which case, to administer a survey across multiple regions, the best data can be obtained by translating the survey into the various modes of communication native to panelists in the various regions. As discussed above however, coordinating multiple versions of a survey presents a number of challenges for a researcher.
Creating a multi-region survey is the subject of co-pending U.S. Patent Application <number to be inserted>, entitled "Multi-Region Market Research Study Creation" (Docket # P003). The administration of a multi-region survey is the subject of co-pending U.S. Applications <number to be inserted>, entitled "Multi-Region Market Research Study Processing" (Docket # P004), <number to be inserted>, entitled "Content Delivery In A Preferred Language For A Large Number Of Languages" (Docket # P002), and <number to be inserted>, entitled "Language Sensitive Electronic Mail Generation And Associated Applications" (Docket # P006). Each application has been filed contemporaneously, and has common inventorship, with the present invention. Except for any circular incorporation, the applications are hereby fully incorporated by reference.
The present invention is drawn to analyzing survey data created and collected by the processes of these related applications. The present invention advantageously allows a researcher to access, analyze, and report on multi- region survey data using his or her native mode of communication without any need whatsoever to understand the diverse modes of communication used by the panelists.
Figure 1 illustrates one embodiment of the present invention. A multi- region research service provider 102 is connected to multi-region research clients 104 and multi-region survey panelists 106 through a networking fabric 108. Research clients use service provider 102 to conduct on-line, multi-region surveys of the diverse pool of survey panelists 106. Surveys can focus on virtually any concept including, for instance, market research for a product or service, marketability of a political candidate or issue, sociological studies, health studies, and the like. Surveys can also take virtually any form including, for instance, text-based, graphics-based, audio-based, or multimedia interactions (called "response variables") designed to illicit responses.
As described in the incorporated applications referred to above, research clients create a survey using one particular mode of communication, such as the United States version of the English language and U.S. dollars, and specify multiple regions in which the survey is to be administered. The service provider 102 interacts with translation services (not shown) to translate
the response variables into the various modes of communication used in the specified survey regions. The multi-region survey is then administered to the diverse set of panelists 106 on-line through networking fabric 108.
The service provider 102 includes a multi-region survey reporting and analysis service 110. Analysis service 110 includes analysis interface 114 for interacting with research clients 104 and has access to survey data 112. Survey data 112 may include data representing the surveys themselves as well as data representing any responses that have already been collected.
Typically, a researcher wants to analyze survey data 112 to determine some objective statistical issue. For instance, the researcher may want to know what percentage of a pool of panelists are interested in a particular product or service and the researcher may want to rank their level of interest within a predefined range. To obtain survey data that can be statistically analyzed, surveys are often created using various forms of multiple choice questions rather than free form answers. Since the service provider 102 uniformly administers surveys, whether a question and set of possible answers is presented to a panelist in German, Japanese, Arabic, or some other language, the service provider 102 can uniformly record that the panelist selected a particular answer from the set of possible answers. In which case, the analysis service 110 can perform statistical analysis on the uniformly recorded data without regard to the mode of communication used to administer individual questions.
Furthermore, under various circumstances, a researcher may be interested to see the responses from a particular panelists. Again, since the service provider 102 uniformly administers surveys, whether a question and set of possible answers is presented to a panelist in German, Japanese, Arabic, or some other language, the analysis service 110 can display the questions in one particular language and match-up the uniformly recorded answers. The same approach can be applied to currency conversions, measurement conversions, and the like.
In other words, according to the teachings of the present invention, the analysis service 110 takes advantage of the uniformly administered survey data provided by the multi-region research service provider 102 to allow a
research client 104 using a single mode of communication to analyze multi- region survey data. In various embodiments, analysis can be performed in real-time on centralized data collected from multiple regions using multiple modes of communication. By collecting data from multiple time zones around the world, data processing loads for large surveys, or multiple simultaneous surveys, are likely to be distributed or balanced across a 24 hour operating clock. By supporting multiple modes of communication, a researcher can select his or her native mode of communication from a number of supported modes and analyze data from surveys created and/or administered in different modes of communication. Furthermore, any number of data mining techniques and statistical analysis can be performed. Various examples of the analysis service 110 are described below in more detail.
Except for the novel aspects of the research service provider 102, the various equipment used by the service provider 102, the research clients 104, the survey panelists 106, and the networking fabric 108 are intended to represent a broad range of these elements known in the art. Examples of computing and networking equipment suitable for practicing the present invention include, but are not limited to, various palm-sized, notebook sized, or desktop computers available from, for instance, IBM of Armonk, New York, servers available from Sun Microsystems of Mountain View, California, and routers/switches available from CISCO Systems of San Jose, California.
Method
In general, the analysis service provides the analysis interface to a research client, receives client input through the analysis interface in one mode of communication, analyzes data from the multi-region, on-line survey based on the client input, and provides analysis results through the analysis interface. The analysis interface could be, for instance, provided as a web page on-line or as an installed software application on the client's machine.
Figure 2 demonstrates one embodiment of the present invention including a number of implementation specific details. Other embodiments may not include all of the illustrated elements, may include additional elements, and/or may perform various elements in a different order. The illustrated process is basically several iterations of the general process. That is, the
illustrated process repeatedly provides some form of analysis interface, receives client input, analyzes survey data based on the input, and provides results through the analysis interface.
In the illustrated embodiment, the process begins by prompting the research client in his or her native mode of communication to provide identification and authentication at block 210. This is the first analysis interface. As discussed in the incorporated patent applications, any number of approaches can be used by the client to indicate his or her native mode of communication. For instance, a one-click pick list could be used to select from a list of supported modes of communication. Identification and authentication could be a login and password, an encryption key, biometric data, or the like. The analysis service then waits to receive client input at block 215.
When input is received, the analysis service analyzes survey data based on the input at block 220. For instances, the analysis service filters survey data to identify any surveys that the client is authorized to access. In other words, survey data can include more than just the questions and answers. Survey data may also include supporting information such as who created the survey, who has authority to access the survey, what mode of communication was used to create the survey, purpose statements for the survey, and the like.
If one or more surveys are found, the analysis service displays summary information for the surveys in a list. Summary information may include a title or number for the survey or some other identifying information, as well as the number of panelists in the survey, regions of administration for the survey, and just about anything else that a client might want to know. Listed surveys may have already completed, may be currently in progress, or may be waiting to start. The type of information to be displayed in the summary may depend on the stage the survey is currently in and may depend on the availability of data. That is, if a survey has not started, no response data will be available for statistical analysis. In certain embodiments, the client can configure the fields in the summary list to display information of particular interest to the client.
Of course, the surveys can include data in diverse modes of communication. As discussed above, the analysis service takes advantage of
the uniformly administered data provided by the multi-region research survey provider to display information in the client's mode of communication.
The summary list also includes at least one mechanism to select and act upon one or more of the surveys. Any number of mechanisms can be used. A number of examples are illustrated with respect to Figures 3 through 10 below, and may include, for instance, hyper-links, check boxes and window buttons, pop-up menus, drop-down menus, single-click icons, double-click icons, pick lists, and a window or a series of windows including various configuration options.
At block 225, if no surveys are listed, the process returns to block 210 to prompt the client again. In alternate embodiments, various security measures could be used to prevent an un-authorized user from repeatedly trying input combinations until he or she gains access. For instance, the analysis interface could lockup after a certain number of failed attempts.
If one or more surveys are listed, the process waits for client input at block 230. When client input is received, if the client input indicates that the client wishes to exit at block 235, the process returns to block 210. If the client does not wish to exit, the process performs another iteration of analysis on the survey data at block 240. The analysis is based on the client input and can include any number of data mining and/or statistical analysis operations including, for instance, data filtering such as identifying a particular group of panelists who gave a particular response to a question or identifying related data such as other surveys in which a particular panelist participated, data calculations such as calculating an average response to a particular question, generating mechanisms to configure further analysis such as links to demographic information about particular panelists identified by filtering the survey data, and displaying an analysis report such as a table, graph, or chart.
After analyzing data in block 240, the process returns to block 230 to wait for client input. The process will continue to go through iteration after iteration of blocks 230, 235, and 240, until the client exits, repeatedly analyzing data, presenting analysis results in the analysis interface including at least one mechanism for further analysis, and waiting for further client input. Using this iterative process, the client can move up and down through various levels of
detailed analysis of the survey data, viewing analysis reports in a wide variety of formats.
Examples
Figures 3 through 10, 13, and 14 illustrate a number of embodiments of analysis interfaces, input mechanisms, and analysis reports. Any number of alternate embodiments can be used to provide an analysis interface, receive client input, and provide analysis results.
Figure 3 illustrates one embodiment of an identification window 300. The illustrated window may be the first step to access survey data using the analysis service of the present invention. In the illustrated embodiment, the client is prompted to supply a login and password. The "continue" button submits the supplied information. As discussed above, any number of alternate approaches can be used to identify and authenticate a client.
Figure 4 illustrates one embodiment of a survey list window 400. Assuming, for instance, that the client successfully gained access to the analysis service through the interface of Figure 3, the client is presented with a list of summary information for surveys to which the client has access. A client may have access only to surveys that the client created, or the client may have been granted access to surveys, or parts of surveys, created by others. Various forms of security could be used to limit a client's access to certain data. For instance, a client may not be given access to specific names and addresses of panelists, or other personal or proprietary information.
In the illustrated embodiment, the summary information includes a survey identification number, a study type, the number of panelists, the geographic regions for administering the survey, and a concept identifier and status for each survey. The underlined entries in each summary illustrate one example of an input mechanism. That is, each underlined entry is a hyper-link to related or supporting data. For instance, if a client wants to know more about the make-up of a particular survey, he or she may click on the survey identification number to, for instance, see a title of the survey, who created it, the questions presented, etc. Clicking on the concept identification number may lead to information on the purpose for the survey and what information the creator hopes to glean from it. Clicking on the countries or cities entries may
lead to more detailed information about the target pool of panelists for the survey, including demographic information, modes of communication used for different groups of panelists, and the like. In alternate embodiments, any number of input mechanisms, such as those discussed above, could be used in place of hyperlinks to lead to more detailed information from a summary page. Various browser functions could also be used to exit the analysis service, return to a previous screen, etc.
In the illustrated embodiment, the first two surveys are either complete or in progress because results are available for analysis as indicated by the hyper-link "Analyze Results." The third survey has not yet been initiated or has not yet received any results. For the surveys that have received results, the "Analyze Results" hyperlink can lead to more detailed information about those results, including some form of overall survey summary.
Figure 5 illustrates one embodiment of a survey summary window 500. The survey summary includes a list of question identifiers, number of responses to each question, average responses, and a check box for each question to select one or more questions for further analysis. An average response can be calculated by assigning a weighted valued to each possible response, adding the weighted values from all the received responses, and dividing the total by the maximum possible weighted value.
In one embodiment, the data can be updated in real time. For instance, were the client is analyzing data on-line, the analysis service can update the numbers whenever new responses are received by incrementing the response count and recalculating the average response. Furthermore, the centralized database allows real-time analysis of data from multiple regions collected using multiple modes of communication.
The exemplary input mechanism in the illustrated embodiment is the combination of the check boxes and the "Continue Analysis" button. The client can select one or more questions for further analysis by checking the boxes and pressing the button. Any number of alternate embodiments may be used, of course, such as making the question identifiers hyper-links and browser controls to return to previous pages.
Figure 6 illustrates one embodiment of an analysis report window 600 providing further details about a question presented, for instance, in the overall survey summary of Figure 5. The window includes the wording of the actual question, the number of responses received, the list of possible answers, and the percentage of responses received for each possible answer. The percentages are also illustrated using bar graphs. In alternate embodiments, a pie chart, histogram, linear or area function, or the like could be used to display the distribution of responses. If the client had selected more than one check box in the summary window of Figure 5, multiple reports such as the one illustrated in Figure 6 could be generated in a single window or in multiple windows. The data could also be updated in real time as new responses are received.
The input mechanisms in Figure 6 include hyper-links leading from each of the possible responses. For instance, selecting one of the responses in Figure 6 could lead to the analysis report window 700 shown in Figure 7, which details demographic information about panelists who selected the corresponding response. Demographic information in the illustrated embodiment includes gender, marital status, educational level, age, employment status, and household composition. In alternate embodiments, any number of demographic factors could be used.
Figure 7 also illustrates another example of an input mechanism, menu/tool bar 710. From the bar 710, a client can select a variety of analysis reports and report formats including graphics, related questions, related surveys, information on individual panelists, and a survey specification. In one embodiment, a user may select a field from the window by, for instance, highlighting it, and then select a menu option related to that field. In an alternate embodiment, a pop-up window of menu options may pop-up when the client passes a cursor over the field in the window. In yet another alternate embodiment, fields may be hyper-links so that, for instance, a client can select the "male" hyper-link and view an analysis report of responses from all male panelists. Any number of alternate input mechanisms can be used, including browser commands to return to previous windows.
Figure 8 illustrates one embodiment of an analysis report window 800 for a particular type of response variable, semantic differential questions. In the illustrated embodiment, semantic differential questions are listed with weighted average responses for each. Any number of input mechanisms can be used to delve into more or different details.
Figure 9 illustrates one embodiment of a particular type of analysis report. The table in window 900 includes three axes, a horizontal axis 905, a vertical axis 910, and a Z axis 920. Using various forms of configuration windows, a user can specify various filter criteria for each axis to identify particular data. In the illustrated embodiment, the horizontal axis filter criteria is education level so that each column corresponds to a different level of education. The vertical axis filter criteria is income so that each row corresponds to a different level of income. The Z axis filter criteria is age so that each table in the Z axis corresponds to a different age category. The data that is being filtered is intent to buy. The each cell in each table is a weighted average response to the intent to buy question for respondents of a particular education, income, and age category.
Figure 10 illustrates one embodiment of another type of analysis report. In window 1000, subcategories of panelists were presented with different price points for a product and asked about their intent to buy. The table includes the number of responses, illustrates the distribution of responses in graphical form, and specifies the average weighted response for each subcategory.
The illustrated embodiment presents a good example of analyzing multi- region data. That is, the price points would have been presented to the panelists in their respective native currencies. The table, however, reports the price points using the native currency and language of the researcher.
Skipping to Figure 13, Figure 13 illustrates one embodiment on an analysis report that can be used to assign codes to open-ended questions. The illustrated embodiment includes two windows. In the window on the right, a client as configured the report to list open-ended responses to a particular question in English. As discussed in the related applications, translation services can be used to present results in any number of supported modes of
communication. The question is displayed at the top of the window and the first five responses are listed below.
Next to each response is a code field. The client can select from codes listed in the window on the left. That is, after reading a particular response, the client can assign a code to the response for statistical analysis purposes. For instance, the first response indicates that the panelist is interested in wireless networking because the panelist is generally interested in cutting edge technology and wireless networking is new to the panelist. Therefore, the client codes the response with the "New" code, indicating that the panelist was interested in wireless networking because it is new.
In the window on the left, each listed code has next to it a number of configuration buttons. The configuration buttons can be used to add new codes if, for instance, the client begins to see a number of similar responses and wants to create a new code for those types of responses, to delete codes if, for instance, no responses receive a particular code number and the client wants to eliminate the code to shorten the list of possibilities, and to configure various aspects of a code.
For instance, as shown in Figure 14, a code figuration window can be used to specify text to identify a particular code number, as well as key words typically found in responses that receive this particular code number. The key words can be used to perform an automatic coding of responses. For instance, if a client selects the "Autocode" button in Figure 13, the analysis service can scan open-ended responses for key words and automatically assign codes accordingly. Once the responses are coded, any kind of data mining or statistical analysis can be performed based on the codes.
Example Computer System
Figure 11 illustrates one embodiment of a hardware system intended to represent a broad category of computer systems such as personal computers, workstations, and/or embedded systems. In the illustrated embodiment, the hardware system includes processor 1110 coupled to high speed bus 1105, which is coupled to input/output (I/O) bus 1115 through bus bridge 1130. Temporary memory 1120 is coupled to bus 1105. Permanent memory 1140 is coupled to bus 1115. I/O device(s) 1150 is also coupled to bus 1115. I/O
device(s) 1150 may include a display device, a keyboard, one or more external network interfaces, etc.
Certain embodiments may include additional components, may not require all of the above components, or may combine one or more components. For instance, temporary memory 1120 may be on-chip with processor 1110. Alternately, permanent memory 1140 may be eliminated and temporary memory 1120 may be replaced with an electrically erasable programmable read only memory (EEPROM), wherein software routines are executed in place from the EEPROM. Some implementations may employ a single bus, to which all of the components are coupled, or one or more additional buses and bus bridges to which various additional components can be coupled. Those skilled in the art will be familiar with a variety of alternate internal networks including, for instance, an internal network based on a high speed system bus with a memory controller hub and an I/O controller hub. Additional components may include additional processors, a CD ROM drive, additional memories, and other peripheral components known in the art.
In one embodiment, the various elements of the present invention described above are each implemented using one or more computers such as the hardware system of Figure 11. Where more than one computer is used, the systems can be coupled to communicate over an external network, such as a local area network (LAN), an internet protocol (IP) network, etc. In one embodiment, the present invention is implemented as software routines executed by one or more execution units within the computer(s). For a given computer, the software routines can be stored on a storage device, such as permanent memory 1140.
Alternately, as shown in Figure 12, the software routines can be machine executable instructions 1210 stored using any machine readable storage medium 1220, such as a diskette, CD-ROM, magnetic tape, digital video or versatile disk (DVD), laser disk, ROM, Flash memory, etc. The series of instructions need not be stored locally, and could be received from a remote storage device, such as a server on a network, a CD ROM device, a floppy disk, etc., through, for instance, I/O device(s) 1150 of Figure 11.
From whatever source, the instructions may be copied from the storage device into temporary memory 1120 and then accessed and executed by processor 1110. In one implementation, these software routines are written in the C programming language. It is to be appreciated, however, that these routines may be implemented in any of a wide variety of programming languages.
In alternate embodiments, the present invention is implemented in discrete hardware or firmware. For example, one or more application specific integrated circuits (ASICs) could be programmed with one or more of the above described functions of the present invention. In another example, one or more functions of the present invention could be implemented in one or more ASICs on additional circuit boards and the circuit boards could be inserted into the computer(s) described above. In another example, field programmable gate arrays (FPGAs) or static programmable gate arrays (SPGA) could be used to implement one or more functions of the present invention. In yet another example, a combination of hardware and software could be used to implement one or more functions of the present invention.
Thus, reporting and analyzing data from a multi-region survey is described. Whereas many alterations and modifications of the present invention will be comprehended by a person skilled in the art after having read the foregoing description, it is to be understood that the particular embodiments shown and described by way of illustration are in no way intended to be considered limiting. Therefore, references to details of particular embodiments are not intended to limit the scope of the claims.