by
Karla Hahn
College of Library and Information Services
University of Maryland
College Park, MD 20742
CLIS TECHNICAL REPORT NO. 97-03
January 1997
This report is a modified version of the paper that was submitted to Dr. Delia Neuman
as an independent study in Fall 1996
Copyright © 1997
by Karla Hahn
The College of Library and Information Services, University of Maryland, College Park,
supports the production and distribution of its Technical Report Series to disseminate research findings in a timely manner.
The author(s) of each report is(are) solely responsible for its content, editorial style, and all other substantive aspects.
INTRODUCTION AND RESEARCH QUESTIONS
Appendices
About the Author
LIST
OF FIGURES AND TABLES
The author would like to thank Dr. Delia Neuman for her guidance throughout the course of the project. In addition, Dr. Marilyn White provided many helpful comments in the course of bringing this work to its present form.
This project would not have been possible without the support
of many staff at the William H. Welch Medical Library. Several
staff members provided their time and also their thoughtful consideration
of the questions under investigation. This work would not have
been possible without their generosity.
Many organizations are currently experimenting with providing help services via e-mail. This investigation used content analysis of service logs and interviews with staff and users of a successful service to identify participants models of ideal exchanges and the impacts of the communication medium on service provision. Participants' models of ideal exchanges include single concise question/response pairs and extended dialogue. Staff are more likely to consider dialogue as typical, while users seem to almost exclusively consider the minimum exchange as normal. Content analysis of service logs suggests that most exchanges are simple question/answer pairs and that users tend to make explicit requests for instructions, explanations, brief informational answers to specific questions, or staff action on behalf of the user. Interviewees identified a range of both benefits and limitations conferred by the use of e-mail as the main medium of communication. While e-mail offers real benefits in terms of increased service access, convenience, and staff efficiency, it also makes service provision subject to time lags and loss of communication richness. One of the main problems participants face is the frequent occurrence of incomplete information. Users tend to either underspecify their request or omit needed information while staff often respond incompletely to queries. Evidence was also found to support the idea that users may feel freer to express negative attitudes and emotions via e-mail as opposed to face-to-face communication. This research suggests that the use of e-mail powerfully influences both the nature of a help service and its perception by users.
Asking questions and seeking answers are common human activities. Interactive communication media of all sorts can be used to support them, and library staff have enlisted new media enthusiastically as they have appeared. Electronic mail (e-mail) is no exception. Several authors have reported on either the potential of the medium or the effects of implementation of e-mail-based reference services. (Abels 1996; Abels and Liebscher 1994; Hodges 1989; Still and Campbell 1993; Tomer 1994; Whitaker 1989) However the literature to date suggests that e-mail-based reference services are not widely used by patrons for questions requiring negotiation. Some libraries report use for interlibrary loan, online searching, and photocopy requests, suggesting that perhaps question-answering services of limited scope could be useful.
This limited success is somewhat surprising given the flexibility the medium of e-mail offers. Sproull and Kiesler (1986) characterized the medium as asynchronous, fast, and text-only, although providing reduced social context cues. In a recent review Garton and Wellman (1995) similarly characterized e-mail as asynchronous and providing rapid transmission of text and added that it supports single and multiple communication rounds and allows easy storage and manipulation of messages.
Because there is some suggestion that question-asking behaviors are particularly suited to the medium, research into this aspect of e-mail communication seems particularly promising. In a study surveying 458 e-mail users at six government and corporate facilities, Rice (1993) reported that exchanging information and asking questions were the top two (of ten) preferred uses of e-mail. This finding indicates that e-mail users are willing to use electronic mail to ask questions generally, raising the question of why there seems to be little interest in using library systems in particular to seek the answers to questions.
This fairly long period of use, the service's high activity level, and the fact that it is available only through e-mail mean that the service offers a special opportunity to examine the effects of using e-mail to provide question-answering services.
These questions both developed out of and were addressed using two complementary data-gathering techniques: analysis of logs of outgoing messages generated by the service and interviews with service staff and service users.
A qualitative approach to answering the questions was chosen both because of the exploratory nature of the study and because of the small number of staff involved with the service. Since only six staff were responsible for providing responses, statistical hypothesis testing was precluded. Qualitative approaches are useful for contextual or descriptive research in which the basic questions relate to "What's going on here?" (Weingand 1993). Neuman (1989) noted the particular usefulness of applying qualitative methods to the study of users' interactions with interactive media. This project's focus on human interactions with goals of exploring the nature of those interactions and the participants' perceptions suggests that qualitative techniques are appropriate.
Jackson (1995) reports that to date three methodologies have commonly been used to study electronic communication: a case study approach, usually comparing an organization before and after implementation of the new medium (a diffusion-of-innovation approach); controlled experiments comparing various media; and, more recently, content-analyses of messages. The latter approach was used in this study. The logs of outgoing messages provide a view of the scope of the service, levels of usage, and characteristics of actual exchanges. Because in this help service, as a matter of policy, outgoing messages incorporate the original message (hereafter referred to as the query) into the reply, these messages provide the opportunity to document these characteristics as they apply both to requesters and respondents. A total of 265 outgoing messages from a single month provided the basis for content analysis of messages. Methodology details are discussed in the appropriate section.
Interviews of all staff and five service users complement the message analysis and allow the researcher to obtain the perceptions of the participants in the service and validate findings from the content analysis of the service logs. The use of complementary approaches enables the use of triangulation to enhance the validity of the findings in terms of their credibility, consistency, and confirmability ( Fidel 1994; Grover and Glazier 1985; Guba and Lincoln 1982). Because of the two-stage data-collection process, the methodology and results for each phase are discussed separately. The results of the service log analysis are discussed prior to the interview methodology and results, since these results shaped the later analysis.
After the broad subject had been determined and a general methodological approach had been selected, the next step was the development of foreshadowing questions. This was done prior to the first phase of data analysis, the content analysis of the January 1996 logs of the service. The initial foreshadowing questions are presented in the top half of Figure 1. Generally, the first group was used to frame the content analysis of the logs. The second set (Figure 1, bottom half) was envisioned to frame the interviews. The second set, along with the results of the content analysis of the service logs, aided in the development of interview guide questions.
Data from incoming and outgoing messages were obtained from service logs for the month of January 1996. This period of analysis provided 265 help service responses. This level of activity is fairly typical for the service during non-peak times. A month's requests were expected to provide sufficient numbers of communication events for meaningful analysis covering a moderate time period. The service is most heavily used in the summer and early fall when the campus experiences annual influxes of new faculty, students, and residents. The mid-year period was expected to provide a balance of requests without a disproportionate number of first time-users, as would be expected from a period chosen in the summer or fall. The service log analysis was expected to provide likely candidates for later user interviews, and the investigator hoped to identify as many long-term users of the service as possible. The sampling period was also selected to occur as close to the interview phase of the project as possible.
Service policy is to respond to all requests and include the initiating message's text in the response. Thus, by examining outgoing requests, all incoming requests should be captured. Because data were collected by analysis of the log of outgoing messages, it is possible that some requests were not captured, although all responses certainly were. In a few cases the initial request was not captured for various reasons. Also, some messages sent by staff were generated for reasons other than response to an incoming request. Some messages were part of an ongoing dialogue but did not include the text of all previous
Communication process in the service
How do users frame queries when using the medium of e-mail based help service?
How do they ask their questions?
What information do they provide (if any)?
What type of service do they want? An explanation, instructions, advice, etc.?
What other things appear in their request messages?
Does true dialogue develop?
Under what circumstances?
How do staff frame their responses?
Do they respond to the user's query accurately?
What amount of service do they provide relative to the user's request?
For instance, do they tend to provide more information than is explicitly requested (perhaps sensing that users rarely express their true needs explicitly)?
Do they provide less information than requested?
Only what is called for?
What circumstances influence the above? Personality, request framing, type of problem, etc.
Does the type of service or information provided match what is asked for? Again, what controls this?
How do the participants in this service perceive
each other? Is there a sense of personality constructed by the
invisible partner and perceived by the correspondent?
Service effectiveness
How effective do the participants in the service find it?
What do the participants think are the benefits of using this communication mode?
What do the participants think are the limitations of using this communication mode?
Under what circumstances do users prefer this service
mode to others that are available?
What problems stem from the use of e-mail for this service?
How do staff deal with users' omission of needed information (if they perceive it)?
How do users deal with staff's omission of desired information (if they perceive it)?
How do users feel about receiving more information than they requested?
discussion. Because of circumstance, the analysis of the January 1996 service logs yielded 1967 question-response pairs from the 265 outgoing messages from that month.
Initially the first 100 outgoing messages were examined considering the foreshadowing questions. This examination resulted in a set of initial coding categories describing characteristics of the user query and the staff response. All the messages were then analyzed using the coding categories; the first 100 were re-analyzed. This technique is commonly referred to as the constant comparative method of analysis (Glaser and Strauss, 1967). Mellon (1990) explains that, with this method, incidents within a category are constantly compared to each other during the coding process. This comparison allows the researcher to develop ideas about the relationships between category members. No additional categories emerged during the coding of the additional exchanges from January. The coding sheet appears in Appendix A.
| Message Type | Number | Percent |
| Response to first-time query | 197 | 74.3 |
| Response without original query | 13 | 4.9 |
| Not a query response | 22 | 8.3 |
| Part of an ongoing dialogue | 33 | 12.5 |
| Total | 265 | 100.0 |
As Table 1 shows, during the month of January 265 outgoing messages were sent from the Helpline account. Of these, 197 (74 percent) include a response to an initiating query (captured in the logs through the reply function) and thus act as a question/answer with no further dialogue anticipated. Thirteen (5 percent) outgoing messages are responses that do not include an initial query. In some instances a message in this group reflects a response to a telephone request. Twenty-two (8 percent) outgoing messages were sent for various purposes other than as responses to user queries, for instance, as test messages or forwarded messages. Thirty-three messages (12 percent) were responses to responses to previous questions from Helpline staff and thus go beyond the single question/answer pair that provides the focus for this analysis.
In the service, messages are answered by six Helpline staff with varying levels and areas of expertise. Some Helpline staff answer general or basic questions exclusively; others answer only questions covering a particular subject area, for instance, Macintosh communication applications, Windows-based communication applications, network administration, or UNIX administration. In January the number answered by an individual staff member ranged from 11 to 78. The service manager also answered two queries. Most queries are actual requests for assistance of some sort.
Messages were analyzed to check for the presence of explicit questions, both in the initial user query and in the staff member's response. Of the 197 initiating queries, 151 (77 percent) include an explicit question. While most users ask explicit questions (165 out of 230 interchanges, 72 percent), in many instances (36, 16 percent) users are able to express needs simply by describing problems. Also, some users report what they think to be system problems and seem not necessarily to expect responses (although one was given in each instance).
| Assistance | Explicit Request | Implicit Request | Total |
| Information | 69 | 3 | 72 |
| Instructions | 70 | 20 | 90 |
| Explanations | 39 | 7 | 46 |
| Service | 21 | 11 | 32 |
| Total | 199 | 41 | |
Each initiating query was analyzed to determine what category of response was desired: simple information, instructions, explanation, or staff action. Table 2 presents the results of this analysis. Informational requests are essentially questions that require the provision of non-instructional, non-explanatory information: "What telecommunications software do you recommend?" Instructional requests are requests for a sequence of actions the user could take to attain some objective or resolve a problem: "What are the commands to decode a Binhex(Mac) [sic] file sent to me by someone?" Explanation requests occur when library users describe some sort of system behavior which they believe is unusual or diagnostic and then request an explanation. This type of request often includes an additional implicit or explicit request for either instructions or service: "Why can't I post a notice on 'XYZ.forsale'?" Service requests involve requests for the staff to perform some action on the user's behalf: "Please check and let me know the status asap [sic]." Requests for staff to provide instructions by some means other than return e-mail were classified as service requests rather than as instruction requests.
During the month of January, 199 (75 percent of all messages) explicit requests for assistance were received from library users. During the same period, 41 (15 percent of all messages) fairly clear implicit requests were also received. Certainly the judgment of the nature of an implicit request is subjective. In many cases, however, the library user is quite clear about indicating what he or she wants without explicitly phrasing a question. The following message provides a good example:
| Type of Assistance | Number | Percent* |
| Information | 69 | 37 |
| Instructions | 72 | 41 |
| Explanations | 55 | 35 |
| Service | 33 | 23 |
| Total | 209 |
*N = 165 messages with explicit requests. Percentages total more than 100 because library users requested multiple types of assistance in a message. For instance, many users requested both instructions and explanations.
I received a rather large message which I attempted to save and export and in the process, I think I fouled up my account. Now I receive a message that my disc space is full every time I log on. I don't know UNIX well enough to know how to delete the messages I have saved; I suppose this would solve the problem; Also, I have another problem that may or may not be related: I can't use my addressbook.
Thanks for the help...
Clearly this user is hoping for instructions on how to delete messages. It is less clear whether the person is hoping the staff member will explain the address book problem, provide instructions so the user could resolve it himself, or resolve the problem using system administrator functions. This example is coded only as an implicit request for instructions, since it is unclear exactly what type of request is implicit in the description of the second problem. It should be clear from this decision that the investigator was fairly conservative in interpreting this type of situation, with the result that the total number of implicit questions is an undercount. The investigator believes the attempt to classify implicit requests is worthwhile, since this is frequently done by staff members. The frequency of courtesy statements (please, thanks, etc.) was also noted. Library users include courteous remarks in most (53 percent) initiating queries.
| Exact Match | Broader Response | Lesser Response | Not Applicable | Total | |
| Initial Explicit Question | 79
(53%) | 21 (14%) | 47 (32%) |
3 (2%) | 150 |
| Initial Implicit Question | 18 (39%) | 18 (39%) | 10 (22%) |
0 (0%) | 46 |
The investigator judged the match between the level of service provided and the level of service requested. Initiating message/response pairs were rated as providing only the level of response requested by the library user, providing a broader response than requested, providing less of a response than requested, or as inappropriate to the initiating query. The results are presented here since the data clearly indicate a perception that staff frequently provide either more or less of a response than users appear to be requesting, either explicitly or implicitly. Table 4 shows a breakdown of the results for messages including implicit questions and for those including explicit questions. The proportions in each category for implicit and explicit queries do seem to differ. Staff tend disproportionately to provide a broader response (more information) than requested when the query is implicit. This response seems to be a natural reaction to vagueness in an initial query.
Messages lacking sufficient information to provide a specific answer were also noted. Omission of needed information is relatively rare, occurring in only seven percent of the initiating requests. Messages were also classified based on inclusion of environmental information. Environmental information was defined as information about the computer platform, applications, connections, etc., that influence the answer provided. Of the initiating requests, 63 (32 percent) include environmental information.
The type of information requested was also matched against the type of information provided. Table 5 displays these results. It should be noted that it is possible for a user to request an explanation and for an explanation to be included in the response, but the response may still be judged as providing a lower level of service than specifically requested.
Instructions are the most common type of response provided and are requested about as frequently as information. Users who request information or instructions are equally likely to receive the type of response requested. These types of requests usually result in responses in kind. Requests for explanations or service are much less likely to receive the type of response requested. In fact, explanations are provided in slightly fewer than half the cases when they are requested, while service is provided in less than one-third of the cases in which it is explicitly requested. Unelicited instructions are the most common substitution for requested information, although unelicited information responses are also quite common. Staff provide unelicited explanations and services with approximately equal frequency. Interestingly, unrequested service is provided more frequently than requested service.
| ||
| Requested | ||
| by Users | Yes | No |
| Yes | 46
(66%*) | 23
(33%†) |
| No | 23
(33%)‡ | |
| Total Provided | 69 | |
| Yes | 42
(58%) | 30
(42%) |
| No | 28
(40%) | |
| Total Provided | 70 | |
| Yes | 24
(44%) | 31
(56%) |
| No | 15
(38%) | |
| Total Provided | 39 | |
| Yes | 8
(24%) | 25
(76%) |
| No | 13
(62%) | |
| Total Provided | 21 | |
| Note: N = 164 messages analyzed.
*Percentage of requests of matching type. †Percentage of requests not matched by type. ‡Percentage of responses that were unsolicited. | ||
The log analysis suggests that users generally make four types of requests: for information (an answer), for instructions (information on how to do something), for an explanation (why something happened), or for service (request for staff to perform some action on the user's behalf). While the scope of the service in this case is quite limited, it could be that these categories can be broadly applied to other e-mail based service systems. Each type of response can require different resources and skills on the part of the library and require varying levels of user involvement in the user's resolution of a problem.
In addition, quite often the query does not include an explicit question, although often the user's implicit request may be clear. In cases in which a query is ambiguous, the staff member is forced to cope in some way, such as by asking a question by return e-mail, interpreting the query, or perhaps by using some other methodology. It is also not uncommon for the user simply to describe a problem and leave it to the discretion of the staff member to frame an appropriate response.
Users include a variety of information besides requests in their queries: courtesies, problem descriptions, error messages, environmental information, and information about past events relating to the problem. At this point it is not clear how the inclusion of these types of information might affect the exchange; however, future analysis could show some patterns.
How Staff Frame Responses
Staff quite frequently do not provide users with the type of response requested. While this finding could be related to the nature of the communication medium, it is clear from the content of some of these exchanges that this behavior can be quite appropriate on the part of staff members. Users sometimes request services that were not available, instructions for activities that are either impossible or undesirable, or explanations for problems rather than instructions or some other step toward resolving the problems. Participant perceptions of this situation and their satisfaction in situations where there is a discrepancy were collected in the interview portion of the study.
Staff also seem sometimes to provide either more or less information than the user requests. This result is particularly likely in situations in which the request is implicit rather than explicit. Some questions raised include: Why do staff provide incomplete responses? How do users respond to this problem? It seems possible that providing more information than requested could be an attempt to avoid future problems, but it could also be an attempt to address needs that staff perceive as implicit in the initial request. Again, participants' perceptions on this situation were targeted during subsequent interviews.
Development of Dialogue
Somewhat surprisingly, extended dialogue is relatively rare for the service. Most of the messages could be easily grouped into question/answer pairs. However, some staff members seem more likely than others to become involved in extended dialogues. Nothing in the logs themselves clearly explains why this might be, although providing an incomplete response may be one factor. Some staff might also see dialogue as a more appropriate tool or behavior than others.
Although the log analysis provides some information in relation to the foreshadowing questions, many of them are answered only incompletely while additional questions have emerged. The enlarged set of research questions used to shape the interviews are included in the second part of Figure 1.
Staff were contacted by telephone and invited to participate in the study with their supervisor's permission. Between March 29 and April 25, 1996, all six staff active in the help service in January were interviewed. The interviews ranged in duration from approximately 20 to 40 minutes and followed the questions outlined in the Appendix C. Staff members reported experience with the service ranging from 3 months to 2 ½ years. Staff reported providing expertise on such topics as telecommunications, UNIX, e-mail, file transfer, and system administration.
Service Users
Potential interviewees were selected to provide a group of representative respondents. This purposive sample was also assembled to include as broad a representation of users of the service as possible. Potential interviewees represented males and females; all four of the schools on campus, and each of the three main categories of users (faculty, staff, and students). Several repeat users of the service were identified through the content analysis of the January message logs; a list of 14 potential interviewees was developed by supplementing the initial group of repeat users by scanning February logs. Potential interviewees had used the service at least twice in the January/February period. Users were invited to participate in the study through e-mail and five volunteered through return e-mail. Interviews of users began on March 29, 1996 and were completed April 15. User interviews ranged from roughly 20 to 30 minutes and followed the interview guide presented in Appendix D.
The users interviewed were four females and one male. Two affiliates of one school were interviewed, along with one affiliate each from the other two schools and the university hospital. Two users were doctoral students, one was an undergraduate, one was a faculty member, and the other was a staff person. Thus, while the number of users interviewed was fairly small, the sample was about as diverse as could be accomplished with this number of people.
During interviews all users reported multiple uses of the service over time periods ranging from several months to 2 ½ years. Reasons mentioned by users for contacting the service included Netscape problems, e-mail problems, telecommunication problems, system failure, account management, printing, UNIX, and resource location.
Interview guide questions were developed for both staff and users after the content analysis of the January 1996 logs. The interview guide questions were designed to collect several types of information: background on the individuals being interviewed, such as the extent of their experience with the service, areas of expertise for staff, and affiliation and types of questions asked for users; perceived benefits and limitations of the medium; coping strategies for situations in which the information provided by a dialogue partner is incomplete, and patterns of information exchange in such situations; desirable qualities of the exchange and idealized qualities of the service; and assessments of the service's success (users and staff) and the information's usefulness and specificity (users). Analysis of the service logs suggested that dialogue partners frequently failed to provide complete information in their messages. Interviewees were asked to verify this assessment and to describe any patterns they had observed and any coping strategies they had used. Users were also queried about the benefits and limitations they perceived relating to the use of electronic mail, since this information is not inferable from service logs. Likewise, desirable qualities of an interchange were also elicited.
Each interview was transcribed within three days of the interview. Several initial coding categories were developed from the foreshadowing questions. Transcripts were analyzed according to the constant comparative technique (Glaser and Strauss, 1967) with the aim of developing initial groupings of category responses and identifying emergent categories. Several categories emerged from this initial analysis that were not explicit in the foreshadowing questions or the questions evolving from the content analysis: success criteria and communication virtues. The transcripts were then analyzed again with the complete list of categories, and category response groupings and counts were made. During the second analysis no new categories emerged; however, a small number of new response groupings were added and a few grouping names were modified.
Coding categories and examples of responses grouped within categories are presented in Appendix E. In some instances both staff and user responses are reported for a category, while other categories include only staff responses or user responses. The number of interviewees reporting a response was counted and the results appear in various tables in the following section. Figure 2 summarizes the main coding categories.
Perceptions of the nature of prototypical service exchanges
User benefits of using e-mail for providing help service
Institutional benefits of using e-mail for providing help service
Limitations of the medium in relation to the service
Staff response to insufficient information in query
User response to incomplete staff response
Staff responses to unclear user need
Explanations for omission of information
Correlates suggested relating to need for negotiation
Service success criteria
Desirable qualities of e-mail communications
The interview responses address perceptions of the impact of the communication medium in several ways. Main findings related to participants' mental models of the communication process and perceptions of the benefits and limitations conferred by the e-mail medium.
Staff and users described several competing internal models for ideal e-mail communication in the setting of help-seeking. Two common models emerge: one of extended dialogue similar to a face-to-face conversation and a more limited direct match of question statement with appropriate response. Figure 3 presents a schematic description of these two models. Staff seem more likely than users to consider an extended dialogue beyond a single message-response pair as normal. A staff member described a the situation in this way: "There are going to be lots of complicated things which will take several back and forth e-mails or even phone calls." By contrast, a user indicated "The ideal [service] exchange would be: describe the problem to them and then receive a timely response, say within half a day, of someone attempting to give you clear and simple instructions in response..."
Staff and users also vary in whether they talk about the initial
query as a problem description or a question. Users tend to speak
of their queries as questions seeking answers rather than descriptions
of problems requiring resolutions. Staff members seem more likely
to view the service as problem solving rather than question answering,
although both views are well represented among staff, and some
individuals used both terms. This variety may reflect broader
exposure to the service on staff members' part, since they see
hundreds of queries a month while users see only the small number
of queries they generate themselves.
Staff's and users' perceptions of benefits both for the user and for the institution generally were quite similar (Table 6). Comprehensive service hours and general convenience were mentioned by the most respondents as user benefits, although frequently users also reported benefits of communicating in writing. One user noted that "...for something like what do I do with Trumpet Winsock [telecommunications software], I had it written down and I could type it out. So it wasn't just somebody walking me through on the phone and me taking notes simultaneously..." Efficiency gains are the most widely perceived institutional benefits: both users and the institution providing the service benefit from greater efficiency, the documentation of exchanges, and the use of written communication. However, users and the institution were also perceived to receive some individual benefits: users were the main beneficiaries of the comprehensive service hours and increased convenience, while the service-providing institution was the main beneficiary of service-management benefits and benefits of congruence with other services.
| User Benefits | ||
| Comprehensive service hours | ||
| General convenience | ||
| Benefits of written communication | ||
| Greater efficiency | ||
| Documentation creation | ||
| Increased service scope | ||
| Institution Benefits | ||
| Greater efficiency | ||
| Benefits of written documentation | ||
| Service management | ||
| Increased service scope | ||
| Documentation creation | ||
| Congruence with other services | ||
| Note: N of staff = 6; N of users = 5. | ||
Interestingly, while staff and users saw very similar benefits from the use of e-mail, staff were aware of a wider range of limitations to using electronic mail as the service-delivery medium (Table 7). Perceived limitations fall into three categories: limitations that are shared with at least one other medium, limitations that are shared by other media but may be worse when e-mail is used, and limitations arising from the specific implementation of e-mail for the service. Only staff reported limitations falling within the latter category. System vulnerability was the most widely perceived limitation by both staff and users and was often mentioned first in remarks like "If something is wrong with your e-mail system, then you're in trouble" (from a user) or "It's hard to rely on [the service] for the e-mail service, if the e-mail service isn't working..." (from a staff member). This concern for reliability was probably enhanced by a recent series of system failures following a long period of system stability. Staff and users also emphasized the penalties imposed by e-mail on extended dialogue. One user even spoke as if return dialogue were simply precluded: "I also think, obviously, if you want to ask a follow-up question, you can't do that. To the extent that what you get on e-mail is exhaustive and complete, it's very helpful; to the extent that it doesn't explain what you need to know, it's a little frustrating." Only staff members emphasized problems stemming from the ability to ignore queries and problems relating to dealing with negative emotions. Again, because staff deal with a much larger volume of mail relating to the service, they may be expected to have a broader perspective on problems generated by the medium.
Both users and staff verified the observation from the log analysis that messages often provide incomplete information. All staff agreed that users often omit some information needed to provide a complete response. Similarly, users universally agreed that staff frequently do not provide complete responses to their queries while estimates of how often incomplete response occurs ranged between 20 percent and 66 percent of the time.
| Limitations | ||
| Shared with other media: | ||
| Dealing with negative emotions | ||
| "Phone tag" situations | ||
| Difficulty in defining service | ||
| Exacerbated by e-mail: | ||
| System vulnerability | ||
| High dialogue penalties | ||
| Ability of staff to ignore requests | ||
| Perception of automatic response | ||
| Inability to divide messages with multiple questions | ||
| Long message creation time | ||
| Created by local implementation: | ||
| Lack of sorting capability | ||
| Escalating demands as volume increases | ||
| Note: N of staff = 6; N of users = 5. | ||
Missing information arose from different sources, depending on whether the message author was a user or a staff member. Staff felt that users tend to omit information because they are inexperienced or unable to follow instructions. The main source of problems in this area was user omission of information on the computing environment. Users attributed staff omission to careless reading of queries, confusion caused by multiple questions in a single message, or poor presentation on their own part in creating the initial query. One user somewhat sympathetically suggested that staff may provide incomplete responses because
they're not paying attention ... to the information that I am providing. But part of that could be my fault. You know how you do. You get on and start in the middle of the story instead of at the beginning. And you're assuming they know everything about your situation, but of course they can't.
Users also reported staff inability to accurately assess the user's expertise as a cause for provision of incomplete information. Both users and staff reported problems relating to a dialogue partner forgetting earlier events relevant to the exchange.
Both staff and users tended to deal with the problem in similar ways, usually by dialogue by return mail or by switching to another medium. Staff changed media by calling the user. Users changed the communication medium by contacting a service site either in person or by phone. Staff also reported implementing a template as part of an automatic response to incoming queries. The template specifically addresses an area which staff consider frequently problematic, omitted information on system parameters or connection mode. This template has the effect of encouraging users to consider whether an initial query provides complete information in this area and to resubmit the query, if necessary. One staff member, because of system administrator privileges, had the unique option of inspecting the user's account to supplement omitted information directly.
Staff and users apparently also simply did without the missing information, at least in some situations. Staff may provide multiple answers to cover the range of possible contingencies. Users may either solve the problem independently or simply ignore it.
Finally, in the process of answering questions about benefits, limitations, and common problems, staff and users mentioned a number of variables related both to the success of the service itself and to the success of particular exchanges. Success criteria proved somewhat difficult to elicit from users, while staff seemed able to discuss this area relatively easily. Several users commented that they did not think they could assess the service based only on their own experience. Nevertheless, it is possible to paint a fairly detailed picture of an ideal service based on the interviews. Desired characteristics include utility, timeliness, reliability, and ease of use. A well-managed service would offer alternative information delivery methods and would experience high user satisfaction, a low cost/benefit ratio, and heavy use.
Even more interesting than the qualities of an ideal service were a set of what have been labeled "communication virtues" which emerges from the interviews generally and from staff's responses to a question regarding how they had changed the way they answered questions over time. Probably because they were responding to a question that related directly to this concept, staff provide a wider range of virtues while users mentioned many of the same qualities spontaneously. Ideal characteristics of individual query responses are thoroughness, matching of terminology with a user's vocabulary, empathy, timeliness, conciseness, consistency, and simplicity (see Table 8). Staff reported these characteristics in comments like
I think since I started, I'm... maybe more thorough and [have] a little more understanding of where they're coming from and the types of problems they're having.
If the user has very low skill - low computer skill - then you just answer them as simply as possible without causing any confusion. And if you're dealing with a user with good skills then you have to give them more detail and try to satisfy them based on the level the users have.
| Communication Virtue | Staff | Users | |
| Empathy | 2 | 4 | |
| Timeliness | 3 | 2 | |
| Thoroughness | 4 | ||
| Terminology matching | 2 | 2 | |
| Consistency | 1 | 2 | |
| Conciseness | 2 | ||
| Dialogue when appropriate | 1 | ||
| Simplicity | 1 | ||
| Note: N of staff = 6; N of users = 5. | |||
It is interesting that the virtue mentioned by the most staff is thoroughness, since the analyses of both the service logs and the user interviews revealed this to be a fairly common failing of both staff and user correspondence.
Many of the findings in this study agree with earlier work, particularly research on e-mail based reference services, reference work generally, and work on the effects of decreasing social context cues.
Several findings of Abels (1996) were validated in this study. Both her study and this one report frequent difficulties relating to underspecified queries and incomplete staff responses. Abels found that during dialogue questions were frequently left unanswered and that this created problems for participants. The content analysis of service logs shows that questions occasionally are ignored. Several users suggested this spontaneously as a reason for staff's failing to provide all requested information. Abels also reported that the development of an automatic template enhanced service quality by encouraging more complete specification of the initial request. In 1995 the service studied here had similarly implemented a template in response to what staff perceived as common problems in underspecifying queries in relation to computing environment. While a before-and-after study would more adequately describe the impact of the template, certainly the amount of underspecification observed in the January logs was fairly low (around 7 percent).
The help seeking investigated here can also be related to work investigating reference desk interactions. Prototypically, question answering in the reference setting occurs in the context of the reference interview. The interview can be considered a dialogue with the aim either of refining the initial question or developing a query that adequately reflects an underlying user need (Lynch 1978; White 1989). In the case of the more focused question answering service studied here, extended dialogue beyond a single question-answer pair was quite rare, while the user interviews suggest it is generally undesirable, at least from the user's viewpoint. Dialogue penalties were one of the most frequently mentioned limitations of the medium. At least one user interviewed preferred to do without an answer rather than extend the dialogue with staff. Several users reported being quite frustrated in situations which required extended dialogue even if they ultimately were able to resolve their problems. At the same time, staff reported situations where they felt either that they did not have enough information to answer the question or that the user's real need might be different from the one expressed in the initial query. Extended dialogue through either e-mail or some other medium was the most frequent response, but clearly the risk of decreased user satisfaction is high. This tension has no easy resolution as long as the communication medium is e-mail. How to implement an e-mail interview remains an open question.
Turning to research focusing on e-mail communication generally, Sproull and Kiesler (1991; 1986) have reported that e-mail provides decreased social context cues, which can make users both more open about expressing negative emotions and more willing to report negative behaviors. Sproull and Kiesler's research suggests that greater expression of negative emotions and behaviors may be a more common problem with e-mail than face-to-face or telephone communication. Clearly both staff and users of the service recognize that the reduction in what Sproull and Kiesler call social context cues affect their interactions in ways that could both enhance the interaction and inhibit it. Many staff and users used the word "impersonal" in describing e-mail as a communication medium. Both groups saw this characteristic as having both useful impacts and negative aspects. Several staff commented on the difficulty of dealing with negative emotions expressed in e-mail messages. Conversely, one user described the nature of the medium as muting the impact of negative emotions, making use of the medium a benefit for staff by providing easier handling of negative emotions. Some users thought that staff may hide behind the medium or, conversely, that a greater sense of anonymity might make it easier for people to admit problems and ask for help. One user explained: "...for some people it's just less embarrassing to ask via e-mail than it would be to actually say to someone, 'I don't know how to do this." Staff expressed concern that users disliked the lack of direct human contact, stating that a limitation of the medium was that users "perceive it as being impersonal," and some users did express this feeling. However, staff also feel that users seem more likely to use abusive language than they would in more immediate communication settings.
Although many of the findings of the research supported previous work, some findings are unique. First, the staff and users seem to have somewhat incompatible models of the help service. Users particularly seem reluctant to become involved in extended dialogues via e-mail during help-seeking and prefer a model of question submission followed by a single, complete response. Staff recognize the advantages of this type of exchange and also are attracted to this model, often guessing at user needs to provide it. However, they are more likely to see the medium as supporting an ongoing dialogue in the process of resolving complex problems.
Perhaps because of their widespread acceptance of the model of a single question/answer pair for service provision, users tend to pose explicit questions requiring fairly specific answers. This high degree of query specification may explain why requests for staff to provide service are relatively rare in comparison to requests for instructions, explanations, or informational answers. However, although more often than not, questions are explicit and clear, missing information is a common problem. Clearly, some limitations are imposed both by the nature of providing a help service and the character of the communication medium. These limit service providers' ability to complete help-provision within the confines of a single query/response pair.
This investigation found that the use of e-mail created limitations through time lags, loss of message richness, and system vulnerability. The loss of message richness shows both in the characterization of the medium as impersonal and in the difficulties staff experience in judging user competencies. This latter problem may be the most significant in the case of the service studied here. Both staff and users attribute many of the errors of omission that inhibit the service to the difficulty of communicating user expertise in the course of an e-mail message.
While these problems are not surprising, this work details benefits of the medium which counterbalance general problems in service provision. The medium greatly enhances user access to the service and in many cases increases the convenience of the service. The asynchronous nature of the medium also enhances the efficiency of service staff. Even the time lags that users and staff frequently lament are admitted to have a hidden advantage in allowing time for reflection. The requirement of written communication can force clarity on the communicators. The ease of automating documentation of exchanges is also a very real benefits to all participants in the service.
This study suggests that the use of e-mail powerfully influences both the nature of a help service and its perception by users. Although the technology was used by many people in the setting studied, it was still a relatively new innovation. This early developmental state is indicated by frequent comments from staff and users that the service needs to be more clearly defined. As time passes, users' expectation that the computer-supported service function as a completely automated service may ameliorate, and dialogue may become both more frequent and more acceptable. Whatever drawbacks or frustrations participants experience, the high use of the service suggests that further development will continue.
Abels, Eileen G. 1996. The e-mail reference interview. RQ 35(3):345-58.
Abels, Eileen G., and Peter Liebscher. 1994. A new challenge for intermediary-client communication: The electronic network. Reference Librarian 41/42:185-96.
Fidel, Raya. 1993. Qualitative methods in information retrieval research. Library and Information Science Research 15 (3):219-247.
________. 1994. User-centered indexing. Journal of the American Society for Information Science 45 (8):572-76.
Garton, Laura, and Barry Wellman. 1995. Social impacts of electronic mail in organizations: A review of the research literature. Communication Yearbook 18: 434-453.
Glaser, B. G., and A. L. Strauss. 1967. The discovery of grounded theory: Strategies for qualitative research. New York: Aldine.
Grover, Robert, and Jack Glazier. 1985. Implications for application of qualitative methods to library and information science. Library and Information Science Research 7 (3):247-260.
Guba, Egon G. 1981. Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Communication and Technology Journal 29 (2): 5-91.
Guba, Egon G., and Yvonna S. Lincoln. 1982. Epistemological and methodological bases of naturalistic inquiry. Educational Communication and Technology Journal 30 (4):233-252.
Hodges, Pauline R. 1989. Reference in the age of automation: changes in reference service at Chemical Abstracts Service Library. Special Libraries 80 (4):251-7.
Jackson, Michele H. 1995. The meaning of "communication technology": The technology-context scheme. Communication Yearbook 19:229-267.
Katz, William A. 1992. Introduction to reference work: volume II reference services and reference processes. 6th ed. New York: McGraw-Hill.
Lynch, Mary Jo. 1978. Reference interviews in public libraries. Library Quarterly 48 (2): 119-42.
Mellon, Constance Ann. 1990. Naturalistic inquiry for library science: Methods and applications for research, evaluation, and teaching. New York: Greenwood Press.
Morse, Janice M. 1992. Designing funded qualitative research. In Handbook of qualitative research, edited by Y. S. Lincoln and N. L. Denzin. Thousand Oaks, CA: Sage.
Neuman, Delia. 1989. Naturalistic inquiry and computer-based instruction: Rationale, procedures and potential. Educational Technology Research and Development 37 (3):39-51.
Rice, Ronald E. 1993. Media appropriateness: Using social presence theory to compare traditional and new organizational media. Human Communication Research 19 (4):451-484.
Sproull, Lee S., and Sara Kiesler. 1991. Computers, networks and work. Scientific American 265(3):116-127.
________. 1986. Reducing social context cues: Electronic mail in organizational communication. Management Science 32 (11):1492-1512.
Still, Julie, and Frank Campbell. 1993. Librarian in a box: The use of electronic mail for reference. Reference Services Review 21 (1):15-18.
Tomer, Christinger. 1994. MIME and electronic reference services. Reference Librarian 41/42: 347-73.
Weingand, Darlene E. 1993. Grounded theory and qualitative methodology. IFLA Journal 19 (1):17-26.
Whitaker, Becki. 1989. Electronic mail in the library: A perspective. Library Trends 37 (3):357-65.
White, Marilyn Domas. 1989. Different approaches to the reference interview. Reference Librarian 25/26:631-46.
A | ID | |
| B | Sender type (0=patron, 1=staff) | 0 1 |
| C | Response type (0=response including original query, 1=response w/out original query, 2=not a response to a query, 3=query was response in an ongoing dialogue) | 0 1 2 3 |
| D | Related responses (identified by id) | |
| E | From (0=anonymous, 1=staff2, 2=staff4, 3=staff1, 4=staff5, 5=staff3, 6=supervisor, 7=staff6) | 0 1 2 3 4 5 6 7 |
| F | Return question in response? (0=no, 1=yes) Did staff ask for clarification? | 0 1 |
| G | Standardized instructions in response(0=no, 1=yes) | 0 1 |
| H | Referral? (0=no, 1=yes) was the question referred via email to another person or site? | 0 1 |
| I | Explanation in response (0=no, 1=yes) | 0 1 |
| J | Instructions in response (0=no, 1=yes) | 0 1 |
| K | Service report in response (0=no, 1=yes) | 0 1 |
| L | Alternate instructions in response (0=no, 1=yes) | 0 1 |
| M | Response query match (0= all info explicitly requested provided, no more, 1=more info provided than explicitly requested, 2=less info provided than explicitly requested, 3=info provided seems unrelated to info requested) | 0 1 2 3 |
| N | Explicit request included (0=no, 1=yes) | 0 1 |
| O | Request for service (0=no, 1=yes) | 0 1 |
| P | Request for explanation (0=no, 1=yes) | 0 1 |
| Q | Request for instructions (0=no, 1=yes) | 0 1 |
| R | Environmental info included in request (0=no, 1=yes) | 0 1 |
| S | Problem description included in request (0=no, 1=yes) | 0 1 |
| T | Error message reported (0=no, 1=yes) | 0 1 |
| U | Complaint included in request (0=no, 1=yes) | 0 1 |
| V | Time frame included (0=no, 1=yes) | 0 1 |
| W | History included (0=no, 1=yes) | 0 1 |
| X | Courtesies (0=no, 1=yes) | 0 1 |
| Y | Request for information (0=no, 1=yes) | 0 1 |
| Z | System problem reported (0=no, 1=yes) | 0 1 |
| AA | Courtesy in response (0=no, 1=yes) | 0 1 |
| AB | Information in response (0=no, 1=yes) | 0 1 |
| AC | Unelicited but related info in response (0=no, 1=yes) | 0 1 |
| AD | Insufficient info in query to provide complete response (0=no, 1=yes) | 0 1 |
Please describe your experiences with Helpline. If you use Helpline very frequently, describe perhaps a few of your most recent experiences.
What are limitations that you see to this service, particularly due to the nature of the communication medium?
Some analysis of the logs of outgoing messages show that staff sometimes don't provide all of the information users ask for. Does your experience support this observation?
Have you had to go either to Helpline or some other source for the missing information?
(In terms of how much information you feel you need when you ask questions,) What proportion of the time do you feel that staff usually provide all the information you wanted? Too little information? More information than you needed?
If you feel that staff sometimes provide more information than you ask for, what proportion of the time do you find the extra information helpful? Not helpful?
How successful do you think this service is?
Have you recommended the service to someone else? What did you say?
What would you like to change about this service if you could?
What other sources do you think you could use (assuming they were available at the time) to answer the types of questions you send to Helpline?
What questions should I have asked that I haven't?
Note: (U) indicates summary of a user response, (S) indicates summary of a staff member response.
Category: The prototypical exchange (S and U)
Single exchange models: Questions
Single exchange models: Problems
Dialogue models: Questions
Dialogue models: Problems
Category: User benefits (S and U)
Efficiency - efficient use of user time:
Continuous service access:
Convenience:
Service scope:
Documentation (is created)
Benefits of written exchange:
Category: Institution benefits (S and U)
Efficiency:
Service scope:
Service management benefits:
Documentation:
Benefits of written communication:
Congruence:
Category: Service limitations (S and U)
Limitations of e-mail that are shared by some other media but may be influenced by the medium
Limitations that are unique to e-mail
Limitations caused by the XYZ Library's particular implementation of e-mail for this service
Category: Staff response to insufficient information (S)
Multiple answers
Dialogue via return e-mail
Dialogue via another medium
System template
Direct system inspection
Wait
Category: User coping with incomplete response(U)
Dialogue via return e-mail:
Dialogue via another medium
Alternate source
Solve problem independently
Do without information:
Category: Patterns of user omitted information (S)
User characteristics:
Inexperience of user
Inability to follow instructions
Environment:
History:
Category: Patterns of staff omitted information (U)
Careless reading
Poor presentation in query
History
Lack of understanding of user expertise
Multiple questions inquery
Lack of expertise
Category: Coping strategy for need for negotiation of the user's need (S)
Dialogue via return e-mail
Guess
Change medium
Category: Patterns in need for need negotiation (S)
Inexperience
Vagueness
Courtesy
Category: Success criteria (S and U)
Heavy use
Staff development
User development
User satisfaction
Cost/benefit
Range of service provided:
Effective management
Timeliness
Management problems:
Inconsistency
Incompleteness
Incomplete range of service provided
Ease of use
Reliability
Utility
Category: E-mail virtues (S and U)
Thoroughness:
Terminology matching:
Empathy
Conciseness
Dialogue:
Timeliness
Category: Service idealization
More staff
Alternative information delivery
Better management
Timeliness
User training
Completed Feb.2 , 1996
Completed March 22
Completed Feb. 26 - March 11
Completed Feb. 26 - March 11
March 11-29
March 29-April 19
Completed Feb. 26
February 29-March 11
Further content analysis and coding. Development of Interview questions based on analysis.
Transcription, content analysis and coding.
End of semester - May 13
Karla Hahn is a doctoral student in the College of Library and Information Services at the University of Maryland, College Park, MD. In 1986, she received her B.S. degree from Wittenberg University with a major in biology and geology; in 1988, she was awarded an M.S. degree in evolutionary biology from the University of Chicago. She received her master's degree in library science from Syracuse University in 1990. As a professional librarian, she has worked in the medical libraries at Johns Hopkins University and the University of Michigan. She has presented several papers at national and regional meetings of the Medical Library Association and has published articles in the Bulletin of the Medical Library Association and the Handbook of Medical Library Practice (Chicago: Medical Library Association, 1995).
In her doctoral program, her areas of interest are scientific and technical communication, including medical informatics and electronic publishing, and information storage and retrieval, including search strategy. She is the recipient of a U. S. Office of Education Training Fellowship for 1996-1997. In addition, she currently serves as a graduate assistant for the online information access course in the College and as a research assistant on a project funded by Disclosure, Inc. to identify user-based criteria for World Wide Web pages. Her e-mail address is khahn@oriole.umd.edu.