Speakers:
- LR: Lee Rainie, Pew Internet & American Life Project
- JT: Dr. Joe Turow, University of Pennsylvania
- PG: Peter Goldschmidt, Health Improvement Institute
- TL: Dr. Thomas W. Langfitt, The College of Physicians of Philadelphia
- GD: Guy D’Andrea, URAC
- NS: New Speaker
Note: This is an edited transcript of the proceedings.
LR: I’m Lee Rainie. I’m the director of the Pew Internet & American Life Project. For my money, this is the highest news-generating session that will be conducted here. You’re going to learn more stuff that’s more noteworthy here than at any other place, so you’re discerning folks to begin with.
Let me quite quickly talk about three things that we know about health Web sites, the way people use them and they don’t necessarily — as a matter of fact, they don’t at all fit together. They cut against each other.
The first thing we know is that a lot of people use health Web sites. We did a piece of research in December that suggested that the number that Don Kimelman cited at the beginning of his presentation is actually low-balling. If you ask people about specific kinds of health information — insurance information, mental health information, doctor information, hospital information, sexual health, diet, nutrition, things like that — you end up getting about 80 percent of U.S. Internet users have gone online to get health information. They go online to diagnose or find out what’s wrong with them; to get second opinions about what their doctor has told them about what they have and how to treat it; to check out doctors that they might be going to; to check out hospitals they might be going to; to go to support groups. There are all kinds of uses of health Web sites. So we know that that’s happening.
The second thing is that there’s enormous blow back against that, or concern about that which centers in a couple of places. It centers in the medical establishment. It centers in academic circles, in research circles. And in some sense, it centers in press circles. There’s a lot of crummy information online. Any quack can set up a perfectly good Web site, and if it’s designed well and navigates well, as Dr. Nielsen was telling us, it probably will seem credible.
So there’s a lot of concern about bad information, outdated information, inappropriate information online that these people who are using health sites might be accessing.
And the third thing we know, though, at least in the earliest rounds of research, is that despite the fact that there’s lots of bad information, there isn’t yet much evidence of people harming themselves. There have been a couple of meta-analyses of a lot of studies that have been done and they have not been able to document more than a handful of cases of true, serious harm being done to people. Obviously, the potential is there, but we don’t yet know the scope of the problem that in some sense we are meant to address in this panel today.
It’s a terrific panel that actually will be making news on several fronts. You’ve already heard a little preview of the information that Peter Goldschmidt is going to be giving us about his analysis of the variety of standards that are used at health Web sites. The Health Improvement Institute has been looking at this material longer than anyone else and caring about it longer than anyone else. Its work preceded much of the good scholarship that has subsequently emerged and what you will hear today will add on to that.
We’re going to start with Joe Turow from the University of Pennsylvania, who has also done research about the portrayal of medical uses of the Web in popular and medical media. There are interesting differences and gaps among them. Then we’re going to hear from Dr. Tom Langfitt, who is doing the most innovative thing I have heard of related to helping people assess the quality of information. He is running a project that is centered in the Philadelphia region and actually setting up a process where people who are doing health searches can have a librarian’s or medical expert’s help as they’re doing it, so they don’t end up in those bad places.
And then you’ll be hearing from Guy D’Andrea, who has set up a very interesting and important accreditation process that has now established itself as one of the premier accrediting tools for medical Web sites online.
So let me turn it over to Joe and you can hear the first bit of news that you’re going to be privileged to hear.
JT: Well, thank you. And we have to go fast, because we have about 10 minutes, is that right?
LR: Yes.
JT: What I wanted to do is talk about the aim of a study that I’ve done with a number of our graduate students at Penn. The aim was to explore the extent to which, and the way in which, a range of media discussed issues of Web site quality about health and what to do about health on the Web.
And this is what we studied, just to give you a sense of this. We looked at two academic meta-analyses of research on health Web sites. Research that looked at lots of academic studies about Web sites to see how accurate they are and the quality of the sites. We looked at one very large one and we extended that very large one. We also looked at trade and popular press discussions from August 2002 through January 2003, essentially six months of the following: Five medical journals — and they included New England Journal, Lancet, JAMA, British Medical Journal, Annals of Internal Medicine; 16 health industry trade publications, going from American Medical News — I won’t read them all — Journal of Nursing, Drug Topics, Managed Healthcare, Health Forum Journal, covering essentially physicians, hospital administration, managed care organizations, general health industry stuff; five major daily newspapers: The Wall Street Journal, New York Times, Washington Post, New York Daily News, Houston Chronicle; 14 consumer magazines: Modern Maturity, Better Homes & Gardens, Good Housekeeping, Family Circle, Time, Newsweek, Prevention, Cosmopolitan, U.S. News & World Report, Shape, Men’s Health, Self, Fitness, GQ. We looked at CNET and ZDNet; and we looked at transcripts from ABC, CBS, CNN, Fox and NBC.
So six months of this. I won’t go into the detail, in great detail, because we don’t have time, but if anybody’s interested, we did, using Lexus/Nexus, we did some very, I think, strong, incredible searching using terms “health and web and Web site,” but also looking at topics. So, for example, we would get an article about diabetes, because the way Lexus/Nexus works, you can also look under the category of health. So and even if the word ‘health’ wasn’t in the article, it would pull a diabetes thing up.
We looked at it in a qualitative way. This was not a quantitative content analysis. We wanted to explore. But we did have six broad categories of information. We looked at the Web sites that were referenced in the pieces, the audience, notions about the audience, the audience’s knowledge of the Web. That is, what the pieces said about that, what they said about the Web site’s effect on the audience, the health industry’s use of the Web and what they said about the effects of Web sites on the medical community. In other words, we were interested in the world of the Web as portrayed by these medical and popular periodicals, as they related to health.
We found what I would call three key disconnects. That the more you look at the stuff, the more confusing it gets. And it’s really very interesting, the confusion on top of confusion. One confusion we found, or at least a disconnect, was between academic medical writing and the popular press regarding Web sites as an issue of public concern about quality. The academic medical writings that we saw in the meta-analyses — and I have some sheets there — were very, very concerned about Web site quality and accuracy. Over and over again. What’s interesting about that is, however, if you look at this table that’s coming from a report that we’re going to issue not too long from now, the notions of quality and accuracy and other criteria that were used in these studies were all over the map. So even in the academic research, the authors found from zero to 99 percent on the same topic of, say, accuracy, given the sites they used, given the different operationalizations of the terms they used. So, the focus in the academic medical writings was on these issues of what I would call clinical accuracy or quality, but it was all over the map in terms of the answers they got. Generally, however, the finding was that the Web is not terribly credible and one has to be careful. But the focus in the academic community was on the accuracy issue.
In the popular press, very little focus was specifically on issues of quality, and I’ll go into that in a bit.
Another disconnect was between the medical literature and emerging research about the way that people search for medical information — what they expect and what accuracy and quality mean. We can go into this in more detail afterwards, but the findings by the people at Lee Rainie’s group and some others suggest, too, that people simply don’t look at clinical accuracy. When they go on the Web, they may do many different things, by themselves, with other people. They’re interested in notions of accuracy that may be very different from the notions that we saw in the medical journals. They believe oftentimes that they get what they want. Whether it’s true or not is not clear from the research yet.
But there’s a real disconnect between what emerging research shows about what people think about their searching and their success and what they see on the Web compared to what the medical research literature says is true about the Web sites.
The third disconnect is between the popular press and emerging research about the way that people search for medical information and what they expect and what accuracy and quality mean. So not only did the medical press and the research and surveys have this disconnect, but the popular press also has its disconnect between what — they don’t talk about how it is that people search and best practices. And so while survey researchers are beginning to emphasize the importance of thinking about this, how it is that people understand what’s true on the Web, what they expect to get from Web sites in terms of quality and accuracy, the general media don’t cover this at all.
And let me quickly go through, because of time, some related findings. Across the media we studied, the digital divide is typically not presented as an issue. Neither is Web literacy. There’s hardly any discussion — it was hard — what was interesting about it is we had five people working on this. Four of them were working on popular materials. And they came to very converging conclusions about what they found.
In popular media, health Web sites are mentioned mostly in passing, to quote information or to send people to information. The Web is overwhelmingly mentioned positively as a way to search for health information. The assumption is that the audience has the Web and knows how to search even medical professional sites. Prevention magazine would send people to medical professional sites and tell them to use an online dictionary.
Few health sites are noted with any regularity. Most of those are government sites or dot.org sites, which raises a whole interesting issue, because dot.org has a certain sense of quality, though anybody can get dot.org today. Systematic critiques of Web sites are rare. The critiques that we found, interestingly, AOL had a person show up on CBS and on CNN critiquing Web sites — basically, just flattering them.
There were at most two mentions of what Web practices ought to be. Seals of approval programs are almost never mentioned. That was astonishing to us. So, just to finalize some of these things — again, we can go into a little more detail later — the disconnects we have seen are confusing and challenging. Confusing because, the more we looked at this, the more we weren’t even sure what the academic literature was finding. And it’s really interesting to see how that disconnects from the popular literature. The medical journal survey disconnect that I talked about portrays how little we know about what really comprises a quality Web site. If people think they’re successful searching, as a lot of people say they are, does that really mean they are? Should we be worried? What is the balance of concern here between clinical accuracy and search accuracy that we should talk about?
The popular media are removed from both matters. They’re not interested in best practices for searching. And they don’t talk a heck of a lot about what we would call clinical accuracy and, typically, don’t address the quality issue at all when it comes to health.
Researchers concerned with clinical accuracy in Web sites, we suggest, must start grappling with consumer search strategies in thinking about quality. Quality and accuracy are more than individual issues — they’re interpersonal. It’s much more complicated than is it true from a clinician’s standpoint.
Popular media ought to be encouraged to bring up issues related to clinical accuracy in Web sites, true. And to best practices in consumer search strategies, which they don’t. And more generally, how to be critical users of health information, because that’s what we found to be hardly ever discussed in the popular media.
Thank you.
PG: Well, I’m delighted to be here today. I know that time is of the essence, so I will be brief and hopefully, there’ll be time for questions and we’ll be able to elaborate on some of the very interesting information that we will hear today.
What I’d like to do is just briefly to introduce you to the Health Improvement Institute and then go through some of the findings that we have from our study, which is a fairly comprehensive one and one I’m going to present today are just some of the findings from the first stage.
The Health Improvement Institute was founded in 1990 as a charitable organization and we’re best known for our awards programs. We have for the last 11 years given awards for excellence in health communication. We do Web sites from 1997 and we’ve done public service announcements from the beginning. We also hold forums and workshops and one of the workshops we held in 1997 was on the quality of health information on the Internet. That was the beginning of our programs to look at this area.
What I’d like to do today is provide you with an oversight of Phase One. It consists of a number of objectives. One, we searched for previous evaluations of Web sites. And secondly, we searched for criteria sets, guidelines that would be helpful to consumers in evaluating the quality of Web sites.
I think it’s very important to re-emphasize a point and that is there is a tremendous amount of confusion out there in every aspect of this subject. One of the things that is a core element is the quality of health information generally. We know that’s a problem for healthcare practitioners, never mind consumers. We’re not going to address the fundamental issue: Is information out there about health of any value? Because that affects the quality of medical care itself.
What we’re looking at here is the quality of health information from the point of view of a consumer, and we’re not going to look at the impact of using the Web sites as a healthcare intervention. For example, there is some evidence that people may be helped by Web sites in terms of helping to adhere to diets and so on and so forth. We’re not going to look at the impact of them. But we’re going to look at the Web sites themselves.
We searched the World Wide Web using many search engines and many terms and we found a number of interesting articles and so on. Part of our search was also of Medline, which as you know is the National Library of Medicine’s online service. And that index is 3,000 medical journals. We found one evaluation of the quality of the health Web site and I’ll describe that in a minute. We found 151 criteria sets.
Now, many of these evaluations were very specialized, dealt with links and what guidelines are for that. But we found 20 that were actually on health Web sites themselves. And what we did in order to analyze them was to develop from first principles a set of meta criteria which we used then to classify the various guidelines that we found.
The one study we found dealt with Web sites on asthma. What they’d done was look at the HONcode — which many of you may know is one of the earliest sets of principles — very simple, had a limited number of principles. But they found in their study that only 16 percent of the sites that they looked at actually met the principles. And more concerning was that, of the sites that had signed up to adhere to these principles, only 38 percent actually did so. So what that means is that someone may see that HON logo on there, which is not policed but rather just something that people volunteer to subscribe to. But they found that didn’t actually adhere to the principles. So that throws into question some issues about the value of seals of approval.
What we did with respect to the 22 criteria sets is that we classified them according to our meta criteria. And we had 77 meta criteria that spanned these nine domains. There were a total of 826 elements within these 22 sets. They fell out in terms of frequency on these things. As you can tell, people are very concerned about the currency and validity of the content of Web sites, for the most part; about a quarter of them addressed that. The other issue was the accessibility.
One of the things when we’re looking at consumer friendliness is consumers need help to evaluate Web sites. One of the criteria you see very often is the navigability of the site. Well, a consumer can tell for him- or herself whether it’s navigable. I mean, if they find it difficult, it is difficult. So that’s one difference between a consumer-oriented issue and a trade issue. That is, how do we help Webmasters improve the quality of their sites — navigability is a very important issue — so we can help them? But from a consumer point of view, it’s very obvious whether it’s navigable or not.
What’s not so obvious to a consumer, for example, are some of the hidden things. It’s very hard for them to judge the quality of the content itself. It’s very hard for practitioners to do that. Another criteria we found, for example, was reading level is very important. We know that from other studies — I’m not sure the Web is written for people that are college level. But a consumer will know for him- or herself whether or not he understands what he reads, in the sense if I find it difficult or I find it easy. But where it becomes important is in the privacy notices and other issues like that. They can be very difficult to read and a consumer may be misled about what’s happening to the information that’s being collected about them when they land on that site. And that would be under our ninth domain here, the safeguards. But it’s very interesting to us that none of the criteria at all addressed the issue of quality assurance, quality improvement and feedback from consumers.
The next thing we did was take a different cut, an evaluative cut as opposed to a descriptive cut. For that, we developed 22 criteria in these six domains as to how user-friendly are these criteria sets themselves as tools to evaluate Web sites? And the bottom line here is we found no tool that met all the evaluation criteria. None of them was consumer friendly. There was only one tool that actually said that it was developed for consumers and it was so complicated that not even professionals could use it.
The other thing that was interesting to us is that none of them have really been tested. That’s very important in terms of the reliability with which these tools evaluate sites. We found, for example, under domain four, that many of the criteria sets had no operationalization at all. It’s very difficult to understand how to use it if you don’t have a well-articulated criteria and you know it’s unreliable when you try to apply it in practice.
We found a variety of information and, from that, we were able to discern a few things that are well-known that people, individual consumers should look for. Obviously, the Web site should be easy to use, but you can judge that for yourself. The things to look for: the sponsorship, the purpose, the audience for which the Web site was constructed, the currency of the information and the source and credentials of the information presented. If you can’t find these things, then you have to be very concerned.
But the bottom line is, as we’ve heard before, it’s a matter of educating consumers to be good users of information and to be somewhat skeptical. So it really is a question of educating consumers to use the tools that are available, and to use their own insights that they may have, and be critical and not believe everything that they read.
In terms of our future plans, we expect to continue to evaluate these Web sites. We are in the process of developing and testing some tools to assess Web sites, and we’ll be inviting everyone here to participate in that who wants to. Then, ultimately, we’ll be rating selected Web sites to see the extent to which they conform to what we would believe are consumer-oriented criteria for evaluating health Web sites.
Thank you. If there are any questions, I’d be pleased to answer them at the right time.
TL: The College of Physicians that I represent today is an honorific institution founded by Benjamin Rush College in 1787. It was the first medical society in the United States and a birthplace of American medicine.
A long-standing mission of the College has been to serve the health of the public and especially the people of greater Philadelphia in unique and useful ways. Toward that end, in 1995, the College launched a Community Health Information Center based on the Internet. The Center occupies a 500-square foot room near the entrance to the College building that is open to the public seven days a week. It’s lined with pamphlets, brochures and texts. It also contains computers staffed by one or more librarians trained to help patrons of the Center access health and medical information on the Internet. The College also is home to one of the foremost comprehensive historical medical libraries in the nation.
In 1999, the Centers for Disease Control learned of the operation and funded a formal evaluation that was highly laudatory. The CDC then commenced funding of the Community Health Information Center, which continues to the present time.
In the summer of 2002, the College completed an intensive two-year strategic planning process that reached four cardinal conclusions. First, the American healthcare system is badly broken, manifested by a long list of familiar shortcomings. The number of uninsured, the demands of managed care, malpractice, medical errors and a return to double-digit inflation lead the list.
We also concluded, however, that the most worrisome fault line within the healthcare system today is steadily deteriorating relationships between the provider and the consumer, between the physician and the patient. At its core are misunderstanding and mistrust by both parties and generated by a host of challenges to the relationship that did not exist a generation ago.
The third conclusion declared that, within a commercially-driven and largely unforgiving healthcare system, the best place to expend the energies of the College was in a plan to try to restore understanding and trust between doctor and patient. Clearly, that effort must commence with more effective education of patients and the nature of their diseases and disorders and the rationale of their treatments, accompanied by a genuine desire on the part of physicians to participate in that process.
Diagnose and adios is a recent chilling and, on the whole I believe, unfair characterization of that current relationship. Nevertheless, it is stated with increasing fervor by many health consumers.
So patient education and persuasion by physicians dedicated to the purposes are a key to success.
Our final conclusion reasoned that the most powerful medium available to educate the public and thereby to begin restoration of that mutual understanding and trust is the Internet. And I should note from the Pew Internet & American Life Project surveys that about two-thirds of health information seekers go online at the time of a diagnosis, either for themselves or for family and friends.
In the summer of 2003, after a long preparation, the College will launch a regional Community Health Information System for the five million citizens of greater Philadelphia. It is titled Philly Health Info, and with the Web address of phillyhealthinfo.org. It contains a host of elements and in the brief time available, I will describe two of its most salient components.
First, is a Web portal that in the fullness of time we intend to be accessible, comprehendible, comprehensive, accurate, timely and unbiased. Needless to say, that is a tall order to fill, those six criteria, but that is the long-term objective.
The portal opens with a health newspaper or periodical that will include among its many, many features essays and commentaries and question-and-answer sessions with our most distinguished physician fellows of the College.
Second is the core Web content built upon the eight-year experience with the College’s Community Health Information Center.
A third is a comprehensive directory of health and medical services and resources for greater Philadelphia, the most notable regional feature of Philly Health Info.
And, last is development of a new intermediary health information caregiver, titled An Interactive Online Assistant, the IOA. These people will be volunteer retired physicians, nurses, librarians, they’ll be students and other Internet literate people who will be trained in a formal curriculum at the Drexel University Institute for Healthcare Informatics, that’s ideal for the purpose: To help Internet searchers access the information they desire. The IOAs will be on call at their home computers for specified periods to assist patrons who are stumbling around cyberspace, trying to find a way.
Also this summer, we will launch an ambitious and, to our knowledge, unique project in Northeast Philadelphia, to reach across the digital divide that that one-half — and I emphasize one-half, 48 percent — of the adult American population who in the last national literacy survey could not access information above the 10th grade level. One half of the population. At the Center, it has been determined that one must have the equivalent of a 10th grade education in order to access and use most healthcare information on the Internet.
So at the center of the pilot project that encompasses a population of some 200,000 people are the Northeast branch of the Free Library of Philadelphia and Community Health Center Number 10 of Philadelphia Department of Health. They are the busiest facilities of their kinds in the city and they are physically attached to each other.
So let’s say Mrs. Rodriguez, a 65-year-old widow with an 8th-grade education, is diagnosed with insulin-dependent diabetes in that extraordinarily busy Community Health Center Number 10. Her physician has 7 to 10 minutes to explain her disease and provide her with a pamphlet or two about diabetes, including the methods she must learn to inject herself with insulin. Chances are, she comprehends little or nothing from that encounter. She does not understand and, therefore, she cannot adhere to her doctor’s recommendations for treatment of her diabetes.
So, in effect, the doctor has accomplished little or nothing and that is the weakest link in the American healthcare system today.
Come July of this year, in addition to her Rx for insulin, she will receive an Ix, an information prescription, from a physician or nurse, that she will carry next door to the library branch. Waiting for her at a computer kiosk with a health information sign attached to it, will be Mrs. Ryan. Mrs. Ryan is a retired librarian and Northeast Philadelphia resident, who also was trained at the Drexel Institute to be an interactive personal assistant, an IPA, to be distinguished from the interactive online assistant, the IOA.
Furthermore, Mrs. Ryan received training not only in how to access health and medical information on the Internet, she was taught ways to reach out to Mrs. Rodriguez to understand and cope with the cultural language and literacy barriers that likely separate them. This is an essential component of the Project, because English is now a second language for 40 percent of the population of Northeast Philadelphia.
To help [inaudible], Council of Southeastern Pennsylvania is dedicated to managing literacy, language and cultural barriers among our citizens and nothing has challenged them more than framing that dialogue between Mrs. Rodriguez and Mrs. Ryan and incorporating it into the curriculum for the IPAs.
All of this bears the mark of a good beginning, we believe. But if we wait for those most vulnerable and least accessible citizens on the other side of the divide to come to the library branch for the health information they want and need, we likely will wait forever. So we have under development an extensive Northwest community collaborative designed to transport the model into the community, into the neighborhood and, ultimately, into the individual household. The collaborative now embraces some two dozen members, including the American Association of Retired People, the Philadelphia Corporation on Aging, the Catholic Archdiocese, the [inaudible], and the Newman Center for Jewish Life.
Also under development is a healthcare collaborative that commences with the outreach programs from the five hospitals that serve most of the citizens of Northeast Philadelphia. In practice, we intend these collaboratives — one emanating from the community, the other from healthcare providers — to interact with each other in order to access all citizens of Northeast Philadelphia who wish to avail themselves of Philly Health Info.
So the long-term objective of the Project is to saturate that region of Philadelphia with health computer terminals, situated in convenient community sites and staffed by volunteer IPAs and by family members and friends who function as IPAs. The achievement of that latter goal — engagement of the family, including the grandchildren — will transport health education of the public into the home through the medium of a Web site they can comprehend.
But we must continually work both sides of the equation — the system and the consumer, the physician and the patient. And do so symmetrically. After all, the powers of the Internet should be just as revealing for the provider as for the consumer.
In the overall scheme of things, the singular advantages for the College of Physicians of Philadelphia are our distinguished history and unblemished track record, as far as we know, and, perhaps most importantly, we have nothing for sale. Thank you.
GD: Thanks, Lee. What I’m going to be talking about with you this morning is a couple of things that URAC is working on both internally, in terms of our health Web site accreditation program — which relates to a large extent to some comments our earlier speakers made — and then to a larger project that we’re working on in collaboration with Consumer Reports WebWatch, to look at some of the connectivity and health search issues that have been a theme, I think, for all the speakers and really are one of the key issues that have to be resolved to fully maximize and realize the benefit of the health Internet.
By way of background, URAC is a private, non-profit organization. We’ve been around now for about 13 years. We promote healthcare quality through accreditation. That’s our mission. We really have done most of our work historically in the managed care sector. As many of you are aware, there’s a very rich tradition in healthcare generally of promoting quality through accreditation programs: hospitals, through the Joint Commission; HMOs, through NCQA; and managed care organizations, to a large extent, through URAC. There’s a whole litany of different accreditation organizations.
While I don’t think any of these organizations, and certainly not URAC, claims to be the be-all and end-all of quality, we do think we’re an important contributor to quality, especially in a field that evolves so rapidly that it often evolves far beyond the ability of regulators, even where they have the authority, to fully understand and then implement oversight that is effective and appropriate for the current state of the art. If the healthcare sector generally has evolved rapidly, that statement is doubly true for the health Internet, where things change very rapidly.
So, several years ago, URAC identified a need for an accreditation program for health Web sites. Working with our governance structure, which includes the American Medical Association, the American Hospital Association and a whole litany of healthcare stakeholder groups — consumers, provides, regulators — we developed this program. I think one important distinction that I want to make — and Peter raised a very important issue from our perspective in his presentation — one thing I want to point out is that from our perspective, accreditation by definition includes a third-party evaluation. And we think that that’s what distinguishes us from some of the other seal programs that may offer value, but which have the issue of basically allowing sites to self-certify. For all of our accreditation programs, including the health Web site accreditation program, we actually go in and verify clients with our standards and then re-verify that compliance on an annual basis, as well as fielding consumer complaints. So that’s an important distinction.
But, basically, what are we trying to accomplish with our accreditation program for health Web sites? Accreditation is essentially shorthand for what consumers might do or might wish to do if they had the time and the training and the expertise. Think about when you buy an electrical appliance. You might go through the testing process yourself and do all the things to test its safety, but ultimately, somebody else has done that. There’s a UL seal there and that at least gives you some comfort of that product. You still may be dissatisfied with that product. I mean, it may not meet your specific needs as a consumer, but at least you know that it meets certain standards for safety and quality and that sort of thing. And that’s essentially what accreditation is. It doesn’t guarantee 100 percent excellent experience with anything, health Web sites included. But it does, from our perspective, provide a shorthand, this seal of approval, where a consumer can look at that and say, “Okay, I know that this site has been reviewed, I know that they meet certain standards and I can look at those standards.” Although I think Peter might suggest, and we’re certainly suggesting, that the standards themselves aren’t that easy for consumers to digest.
So that’s essentially what the purpose of our Web site accreditation program is. This program was launched in December of 2001, and the standards were launched a little bit earlier than that. So far we’ve accredited about 50 sites, which is not that large a number. Although that includes some very large sites, like WebMD and Intelihealth and a number of health plan sites. For example, we’ve accredited the Web sites of health plans that provide services to about 40 million Americans. So we think this at least is getting out there and is potentially having some benefit.
I know time is short, so I’m not going to go into a lot of details about our standards. I would encourage anybody who’s more interested in this to either see me after and give me your card or you can go to our Web site at urac.org where we provide a lot of information about the program, as well as the standards themselves.
But the issue I think I want to spend more time on this morning is an issue that Joe mentioned in his analysis of media coverage of health Web sites and the point that seals of approval are rarely, if ever, mentioned. Which is obviously a concern for us. I mean, we go through the process and Web sites go through the process and the trouble of accreditation. And the idea is that the seal has a benefit, that users are more likely to trust them and therefore use them. But if consumers don’t know about it, for starters, that’s obviously going to be a barrier for creating value around this seal of approval.
Although I think — and I don’t know if there’s any research on this — I don’t know that even if seals of approval were mentioned frequently in media, I’m not sure how much of an impact that would actually have on usage. Because the way that most consumers get the health information is not by looking for seals of approval — even if they’re aware of them, they get to them through search engines. Lee’s group has done some, I think, really amazing work around how consumers get to health information. The way they get to health information is through search engines. A large majority of them don’t go to a site because it was recommended to them. What’s really troubling from my perspective is a large majority of them don’t then consult with a healthcare provider about the information they saw. So, literally, they’re sort of flying solo and flying blind, to a certain extent, typing in a term into a search engine and then, based on the results, getting information that they may never talk to anybody about. I think the project that Dr. Langfitt has described is directly aimed at that sort of problem, but of course there’s a lot of work to be done and it’s still geographically narrow.
So the real question then becomes, well, how do you — there are Web sites out there that are credible, I think we can all agree on that — how do we help consumers find them? We’ve done some work to help them identify them once they get there, but if they can’t get there in the first place, that’s really the threshold issue. And so that’s the second part of my presentation to talk briefly about a project that we’re working on with Consumers Union.
This is a project that’s funded by the Robert Wood Johnson Foundation and the Department of Health & Human Services. And basically what we’re trying to do is look at health search engines: What is known currently about how consumers search for health information online? What is technically possible in terms of filters or other mechanisms that could be built into search engines? What’s the business case for search engines to do any of these things, to help consumers find better information?
But the hope would be that at least we would have a plan of action or a model whereby a consumer could go to a search engine, just as they do now, and type in a search term, just as they do now. But they’d get results that are ranked — at least in some ways — on the quality criteria, on the credibility criteria that Consumer Reports WebWatch puts out, that URAC puts out, that other folks have put out, so that in the normal course of their activities, consumers would achieve better results.
So, as I mentioned, this project has been funded and we’re just now getting underway. The first state of the project is a white paper which will be released, I believe, in mid-June. And that is essentially a literature review to understand what is currently known about consumer search behavior online for health information specifically. It certainly will include a lot of the Pew work. There was a study that came out of the California Healthcare Foundation a couple of years ago about search engines. And just the whole range of research that’s out there.
And then the second part, and I think what will be the very exciting part for us, is a summit, not unlike this one, but focusing specifically on the issue of health search. That’s going to be on June 23rd and 24th in San Francisco. We just, I think we’re just signing the hotel contract today and we’ll be putting out a press release about this. But I certainly would encourage all of you to let us know if you’d like to participate in that. And the purpose of that summit is to bring the multiple stakeholders together. Consumer representatives, the search engines themselves, healthcare providers. Talk about what are the issues and then where can we go from here? What are the gaps in the research? Is there additional work that needs to be done so that we can better understand this issue? What are the business obstacles to getting search engines to perhaps build health search filters, or to rank their health search results based upon some other factors beyond the factors that are currently used for health search engines?
What are some of the things that make searching for health information online different than searching for other information online? We believe that it is, but right now, the methodologies that search engines use are essentially the same across the various types of searches.
So this project we really see as the first step in what will be an ongoing effort to foster debate, but then also foster change. If we can foster that change, we believe that it’s one of the built-in ways that the Internet can help consumers improve their health outcomes through access to better health information.
So I’ll stop there. I think that leaves, hopefully, some time for questions.
LR: We do have a bunch of questions.
Q: Hi. I’m Marty Abrams from the Center for Information Policy Leadership. I found all four of the presentations incredibly interesting and useful. I especially enjoyed the one from the doctor in Philadelphia [Dr. Langfitt] because it goes to the concept of process reengineering around the way individuals interact with their diseases, which the Internet needs to be part of and is not a replacement for process reengineering.
I think there’s a paradigm that needs to overarch all of the projects that you were talking about and there’s one that comes from an old lesson from, for those of us who are Jewish, just had reinforced with Passover. That is the concept of the four sons…[Tape change; discussion resumed in progress.]…that you have to answer the questions differently based on who’s answering the question. It gives you the concept of layering of communication based on the recipient of the information. Since you can’t see it, it goes to the concept of layering of information from simple to complex. Because the other thing that we know is that healthcare, that in most areas, consumers have limited time to invest and will not invest large amounts of time in learning and they want snippets in this very busy world.
But healthcare is different. Healthcare is different because it’s very personal, it’s very important. It’s very — it goes to survival. So I think, as you think about all of those projects, I think we need to go back to this concept of: How is it that people learn? The fact that consumers are different and we need to have methodologies of evaluating search engines and certifying search engines that goes to this concept of: Can they accommodate the need to go from simple to very complex?
The third point is that health information changes very rapidly. What was absolutely the appropriate way to do something two years ago is absolutely inappropriate way to do it today and that we go through a transition stage where all the medical profession learns that last year’s knowledge is wrong and this year’s knowledge is good.
So that’s the third thing that I think we have to function in. And the fourth is the interactivity. The fourth thing goes to this concept of process reengineering, because individuals are an equilibrium system. One treatment affects all of the other systems in a person. Anyone’s who’s dealt with a relative or friend who’s had cancer treatment understands this concept of treatment as equilibrium management. And I think that goes to the fourth piece of complexity, that there needs to be a linkage in this education that allows people to understand that it’s not a linear system and it’s complex.
But I think all of the presentations were great and just offered some suggestions about thinking about it in a slightly different way.
TL: Thank you very much for your comments. You know, when medical students arrive for the first year, one of the first things they are taught to do is how to take a history. And they also learn, before long, that it’s as much art as science. That you have to pay very careful attention and, depending upon how you ask the questions, you get different responses. And I think probably the greatest errors are still made in the practice of medicine by not taking a good history. And I’m afraid that’s been aided and abetted by modern technology. The thing is, go get an MRI or get a CAT scan or what have you.
So this business of relating, the doctor and the patient relating to each other is enormously complex. If you then take that and put that into the Internet, you’re compounding the issues enormously.
The second thing is, if you look at some absolutely classical work done particularly by David Eddy in the 1980s, even the most distinguished specialists in the country do not agree among themselves on how to treat even the most common conditions for which they have responsibility. If you look at the distribution curve, what you’d like to see is a sharp type, bell-shaped, sharp bell, right? Except it’s flat — it’s almost flat. So, when we talk about [vetting], and we talk about credibility, all those sorts of things, what I’m suggesting is that there’s a limit. I’m not sure where that limit is, but it’s not very far out there.
Q: Talking about things connecting, I’m really struck by having to say something that I feel like I’ve been saying for a long time, but the Web is only another medium. So it does not surprise me at all that when you look at consumer articles, you don’t find articles about dealing with seals. There’s a lot of things you don’t find in magazines and on television programs, in radio. The Web is just reflecting that now, because it’s newer. But I think it’s important to remember that these concepts that we’re looking at for the Web are the same as when I wrote for women’s magazines, when I look at writing for physicians, and I look at on the Web, and I look at on TV.
So there is an artificial separation sometimes, and a focus that says, “Now, how come this is?” Well, because it’s not there in any other way. So why would it be there? Here? I think the critical thing is the education process. If consumers aren’t critical when they’re reading something or they’re watching it on television, or they’re hearing it on the radio, why would we expect that they’re going to be critical to see it on a Web page?
Conversely, if they are, then those skills are going to be brought to bear on all the media. And I think that what I would like to see come out of your [unintelligible] summit is that understanding that the Web particularly now is one more medium of information, so how – you’re dealing with something that’s much larger than that, we’re just dealing with it through this particular method.
GD: I would — can I just say one thing — I agree with everything you said. The one thing I’ve seen — I think it’s from some of the Pew stuff, but also elsewhere — some people think that they are appropriately critical. And it’s a very interesting phenomena. And it may have nothing to do with what is, quote, unquote, “really going on”–
Q: But Web developers thinking that their sites are —
GD: Yeah, right, so it compounds the problem in terms of: What does it mean to be appropriately critical?
TL: Let me just say — I agree with everything you said as well. But the difference, I think, in terms of the Internet, is the immediacy of the medium and the fact that it can potentially involve everyone. So I think that it is an extension of everything else, but it is a fundamental change in the way that people can and will access information.
Q: I think there’s a change in how you get information, but I’m not sure there’s a change in how you receive and process the information. You may get more and more of it faster, but how are you looking at it?
TL: Right, but that — exactly, but what you just said is very important. You get more of it faster. People have to be able to cope more readily than they ever have with that particular interaction. So I think that’s the thing that we’re trying to address here or one of the things we’re trying to address.
NS: If I could, I think health may enjoy somewhat special status in this range of different media, because much of the information people get on the Web now, they got from other sources before the news. They might go to a CNN.com, but they used to read the paper. I think the information that people are now getting on the Web for health conditions is information that they might not have gotten at all, because it was so tightly controlled or so hard to access. So I think that, you know, when we see how many people are using the Internet for health, I think that is a new phenomenon in terms of information that’s out there.
So I agree with you that many of the issues cross over, but I think health amplifies some of these things on the Internet.
LR: We’ll do one, two, three.
Q: I wondered how you feel about search engines like Google, who are setting trends on displaying results based on link popularity. And how important is it to have independent accreditation of your health information, if users are finding the information useful? And Google does a great job of finding information that the Web community finds useful.
JT: I think, last week I did a search on SARS [severe acute respiratory syndrome] on a number of different search engines, and I thought Google did very well. I think they probably do very well in a lot of areas. But, in health, one issue that is raised by their methodology of link popularity, is that link popularity creates an inherent preference for information that’s been out there longer, because more people have had a chance to link to it. Often, because there are so many consumer-run health Web sites, where somebody perhaps has a condition, they set up the site and then they move on. But the site stays there. Those links also stay there, so there’s a resistance to change in some ways. So link popularity may end up favoring older information. Given some of the comments and, certainly, what we all know, that health information changes — new research comes out, guidelines are updated — having some way to build currency of the information into the search methodology through whatever form would, it seems to me, improve that.
And again, I think in health it’s especially an important issue.
TL: Let me just elaborate on that. I think that the popularity issue is very different from how useful consumers find the site. There’s no relationship necessarily between one and the other.
There was an idea some years ago that the best medical articles are cited more often. We actually did a study and showed that it isn’t true.
LR: In the green? And could we just have one response, because we’re running a little late.
Q: I was wondering, is there a particular taxonomy that is being used within the health search engines?
NS: Taxonomy?
Q: Within the news industry, there’s the IPTC codes which all the news providers use so that all the stories get to all the different right places. The Library of Congress has developed their own taxonomy. What I’d like to suggest is that the health search engines develop taxonomy [unintelligible] and also [unintelligible].
NS: My answer to that question is I don’t know.
LR: And lots of places are health sites that wouldn’t necessarily they’re in the business exclusively of health information.
Q: Well, that’s why [unintelligible] start using health search engines, you develop your own taxonomy.
TL: You mean the search engine does?
Q: Yeah. And, again, one for consumers, one for the doctors.
LR: Okay. And there’s one–?
Q: Hi, I run a Web site called Lab Tests Online, gives basic information on lab tests, a hopefully unaddressed area on the Internet.
I followed URAC’s development and I understand there’s a fee attached for accreditation. This site is supported by 12 professional societies from the clinical lab community. If I have an extra $5,000, I’m going to start spending that on translating the site into Spanish rather than spending it on a URAC accreditation. I’m just wondering if there’s some way to communicate to the user community out there that, because a site has a URAC seal of approval, in the interest of full disclosure, this site was able to afford the fee as well as meet the criteria and there are other sites — you know.
GD: Right. I mean, we struggle with that issue throughout the process and continue to struggle with it. I mean, our challenge is, as Peter suggests, the HON seal is very, I think, inexpensive for the Health On the Net Foundation to administer, because they don’t actually verify compliance. So the verification process is labor intensive, unfortunately.
Q: Understood. Just the idea, once URAC — and I’ll give you the benefit of the doubt here that URAC is going to become, get to be a household seal, if that should happen, it’s going to imply that if we don’t have that seal, we’re not credible.
GD: Well, I think — I guess what I would say is we recognize that. If it ever comes to pass that we are a household name, I think our ability to provide access to the process will be greatly enhanced. I mean, right now we have to be self-supporting and it’s a small program and there’s just an economic barrier for us.
But I think, we’ve talked about trying to develop essentially scholarship programs, other funding sources to allow sites to come in. But in this economy, obviously, there’s not a lot of that funding lying around. So we recognize it as an issue and I’m certainly happy to continue to try to work towards a resolution.
LR: We’re right at the end of our time and maybe we would turn to Tom for one last word and idea.
TL: Well, just a proposal. For someone to put together a task force to ask the question what is the best we can do right now. Given the limits of our knowledge and experience and resources, what’s the best we can do? In effect, maybe stop tilting at windmills and talking past each other to say what’s the best we can do? That would have to be a very diverse task force. It would have to include experts like my colleagues here. Have to — the providers, the consumers, the insurers. And say what’s the best we could do?
I don’t know what would come of it, but I think it might be a very interesting exercise. At least it would set some kind of realistic limit and get people to agree in general this is about where we are. Now, in the future, hopefully, things will be better. But right now, this is the best that we can do. I cannot emphasize enough, when we talk about vetting Web sites, to the get the very best information you can get on how you manage this or that disorder — I tell you folks, you can’t get the docs to agree.
LR: Thank you. Let’s thank the panelists again for coming.
END OF PANEL