Speakers:
- EP: Eugenie Prime, Hewlett-Packard
- DL: Doug Leeds, Overture
- MC: Matt Cutts, Google
- SS: Sandy Schlosser, ConsumerReports.org
- LM: Leslie Marable, Consumer Reports WebWatch
- NS: New Speaker
Note: This is an edited transcript of the proceedings.
EP: You know, we have all been frustrated at some time or other using the Web. We have used these search engines and have seen the disparity between getting zero hits or getting a billion hits. And it reminds me of what James Dimon, who’s the CEO of Bank One once said about the Internet. He said the Internet basically stinks. It is slow and it is tedious and it’s like walking into a library where all the books are on the floor.
Well, as a librarian, I’d like to differ. Because two of the key things that separates a library from the Internet is the idea of selection and the idea of context. And what search engines are trying to do on the Internet is to actually provide some form of selection and provide some sort of context, albeit after the fact.
Tim Berners-Lee, who created the Web, in an interview some years ago, talking about trust and credibility on the Web, gave this example. He said, “If a foreigner came to New York and looked at a newspaper for the first time and saw The New York Times and the Daily News, they could immediately tell the difference. Because there are certain cues that they know and understand that tell them that one paper has more credibility perhaps than the other.”
So, he said, maybe what we need on the Web is an “Oh, yeah” button. Something you can press and say “Oh, yeah?” And many times you’ve been there where you want to say, “Oh, yeah? Who says?” And the truth is, you can’t tell who says.
So we have some intriguing people, I think. We’re very fortunate to have here — and I’ll just give you a quick bio on each of these people and give you an idea of the process we’re going to use for this discussion.
We will first hear from Doug Leeds. Doug is Senior Director of Business Affairs for Overture Services. At Overture, Doug leads the company’s trust and community efforts. So it will be interesting to find out what that means for Overture and what it focuses — it tells me on the integrity of Overture’s products and services and the suitability of the content in Overture’s search sites. Prior to joining Overture, Doug held a senior management position at Vodaphone.
He’s going to be followed by Matt Cutts from Google. Matt is a software engineer at Google. He joined Google in January 2000, and so he’s been there long enough for us to ask him some serious questions. He works in a quality team to improve search — the right person to have on this panel.
And from Consumer Reports WebWatch, we have two people. We have Sandy Schlosser, who is the new media project manager for ConsumerReports.org, which is the largest publication-based subscription site with over one million subscribers. Sandy works on information architecture, usability and search engine marketing.
And we have Leslie Marable. She’s responsible for project managing Consumer Reports WebWatch research reports, a position that she has held since 2001. And she has also co-authored Consumer Reports WebWatch’s last two reports on credibility. Prior to Consumer Reports WebWatch, she was the research editor for media business site Inside.com, and a magazine reporter for Time, Inc. publications Moneyand Entertainment Weekly.
So we will hear from each of these people in turn and then we will have Q&A with the members of the panel, and we hope that we have time to entertain some questions from the audience.
So, Doug?
DL: Thank you very much, Eugenie. And I want to thank the Consumers Union and Consumer Reports WebWatch for putting on this event, because it’s an excellent forum to discuss some issues that — especially within a search context — I feel very passionate about, I’m very interested in, as Eugenie now knows.
Overture Services is the leader and the pioneer in paid search, commercial search. Many of the search results that appear under the title “Sponsored Listings” on the sites that people use, like Yahoo and MSN, are search results provided by Overture Services. These are what we’ve been calling paid search results. We have been in this business, as I said, we pioneered, we started this business at GoTo.com several years ago. And since that time, we’ve taken a focus on providing great commercial results, very high quality results. And the way we define this is through a measure, not just the fact that they’re paying, which is a precursor to be included in our results, but it’s not the way we define relevance. We have an editorial team of over a hundred people who look at the listings and their relevance to the search terms and to the Web sites and determine relevance that way.
We’re also very concerned in quality with precision, which is that you have the right mix of sites, not just relevant sites, but the ones you expect to see. And then, lastly, suitability. That’s become my role at Overture, which is making sure that the Web sites that are included in our search listings are appropriate in terms of content. So we’ve always made an effort from the very beginning to screen for appropriate content — child pornography and prostitution and automatic weapons and the like you will not find in our search results. And this effort is ongoing, to make sure that what we provide is the highest quality to both end users and to our affiliate partners, the Yahoos and the MSNs of the world.
We are now, as you may know, moving into the “algorithmic” search market, as well. We recently closed our acquisition of the FAST Web search properties, with Alltheweb.com being their premier site. And then we will be closing this month our acquisition of AltaVista. So we are moving heavily into an area that Matt knows all about. Matt?
MC: Great. I am just delighted to be here. When I got the invitation, I was, like, well, this is just a mom and apple pie kind of issue for Google, so I’m really excited to be here and talk to people.
I’m a software engineer at Google, so when I’m not doing things like this, I’m writing code. So I really appreciate the chance to come here, as opposed to writing code. But what that means is I’m happy to answer any kind of questions. I’ll take a stab at business issues, but I can also go down to why particular things are coded or labeled certain ways.
Google is definitely a company with a mission. We’ve been a company with a mission since we started. In fact, this is our mission statement: “We want to organize the world’s information and make it universally accessible and usable.” And every Googler should be able to cite that back to you if you ask them, “What’s your job as a company?” We don’t define ourselves in terms of maximizing shareholder value. We haven’t had to worry about those aspects of things. We basically want to return the best, most relevant results first. And that’s been a very fun thing to work on, because you really get to feel like you’re making a difference in the world.
This is something that everybody knows about, but something that most people don’t realize is that we actually consider honesty a feature at Google. So the FTC last year, for example, sent out letters saying this is what search engines should disclose — which things are paid, which things are not paid. And we were already in compliance with all of that. And it’s kind of interesting, we actually think of honesty as a competitive advantage. We think of it as something that our users build their trust upon.
Just to give a very quick example, when someone types something like “cell phone” into Google, we want them to know which results are editorial results in which money can have no influence whatsoever, and which results are paid listings, advertisements, sponsored links, whatever you want to call it. So, for example, anything in white — the news, all of the search results, things like that — that’s the result of the Quality Team, where we try to return the very best Web sites that we can. Anything that’s marked with a different color or that has a border around it or has “sponsored links” beside it is something that’s advertising. So we try our very hardest to make sure that users understand what the distinction is. Because for some searches, there’s some small or — not the majority, but some percentage of searches which are commercial. Say, 30, 40 percent, where somebody’s looking to buy a product. And in those sorts of searches, the advertisements can be just as useful as the search results. In fact, we often order the search results by the click-through percentage.
So it’s kind of interesting, whenever you take a look at search engines, the vast majority of the articles that compare side-by-side search engines concentrate on the top line up here. They look at things like depth, freshness, relevancy, features. If you’re going to see a side-by-side, you’ll say, “Oh, this search engine is this big.” Or there’s how often a page has to be up on the Web before a search engine will find it. Those are the typical benchmarks that people use when they compare search engines.
But I would actually posit, I would claim that the foundation, the things on which these other features are listed above, are at least as important. Because if you don’t build the trust of your end user, if you don’t have honest search results, clear marking of sponsored listings, if you don’t value the user’s privacy — in effect, if you don’t try to build a relationship of trust, then users won’t come back to you as often.
So I’m really looking forward to all the questions on the Q&A and the panel, but I just wanted to basically express what Google’s philosophy is about how we search the Web and try to build trust.
SS: I think I’m next, I’m Sandy Schlosser. I’m from Consumer Reportsand Consumers Union. I actually started off in the library and then moved on to work on the Web site. One of the things that came up as we were beginning to work with, obviously, a Web [unintelligible] is how to interact and work with the search engines. And so, as the librarian, I was the one who did the initial research and then became project manager for what we started out calling our Search Engine Optimization Project. And just because the names of the marketplace terms have changed, it’s now referred to as Search Engine Marketing.
Consumer Reports, obviously, because we are who we are and our parent organization is, obviously, Consumers Union, we are very much — as soon as we started approaching the whole issue of search engines, we were looking at some of the things that we were just talking about: Honesty, integrity. How to do things in interactive search engines in a way that continues to promote the mission of creating a fair and honest marketplace for consumers and how to participate in it, because we have a Web product that we want people to find and we want to be able to do that in a way that doesn’t conflict at all with who we are. And so that was a lot of what my initial research was about. Issues that came up very quickly were pay-per-click, paid inclusion, search engine optimization, and how to tailor your Web pages so that your users will come in and immediately understand who you are, what topics you represent and where they can find the information that they’re searching for.
And, oddly enough, I found search engines were very interested in the same thing — who you are, what topics you’re representing, what key words best apply to the pages — so that they can serve up good results.
Most of what we do for Consumerreports.org is make sure that our pages are very, very clear, both for the search engines and our users. So that when you hit any of our different pages that are being spidered right now, you will immediately know what our main topic areas are and what the best key words in. So when a person is actually typing in something, they’ll come to the page and go, “Oh, this is where I want to be.” Or they can tell very quickly, “This isn’t where I want to be,” and find the most relevant place.
We also addressed a little bit the idea of pay for submission and paid inclusion. Which has been a little bit of a dicey topic in the industry. We have found that, as where pay for inclusion is something that is just allowing a page to be spidered more often and not affecting the relevancy whatsoever, that that is something we feel pretty comfortable in participating in. But anytime that there’s a question about that suddenly being a hidden ad or something like that, we have a problem with that. So we monitor that situation very, very closely.
As far as pay-per-click has been concerned, I personally, as a project manager, and the team as a whole have been studying the industry and found that right now we’re not participating in pay-per-click, especially because we find that there are some questions in people’s minds. The users are not always being able to distinguish that this is an ad as opposed to search results. And we just want to make sure that we stay behind our mission, this is a fair and honest marketplace and that’s who we are. And, until we are satisfied as a team that participating in a pay-per-click plan is not going to in any way show the users that this is a search result as opposed to an advertisement, we’ve steered away from that.
It’s my job as the project manager to be monitoring the industry. I get about a million e-mails, it feels like, in my inbox every day, just on this particular topic, and it’s something that is very dear to my heart, and Overture and Google, obviously, seem to be in my headlines an awful lot.
LM: Okay, hi, I’m Leslie Marable with Consumer Reports WebWatch. Thank you so much for coming to our first national summit. We’re glad to see you.
I’m going to discuss briefly the findings or key highlights of our new study, an anthropological study relating to search engines.
Just to recap, Consumer Reports WebWatch is grant funded, so obviously our research is funded by these grants versus advertising. We accept no advertising dollars. All of our research is free; we charge no fees. And just for clarity, since we are talking about search engines, Consumerwebwatch.org does not participate in either paid inclusion or paid submission marketing plans, nor does it participate in paid search, as we felt that there might be a conflict of interest. We wanted to put that up-front.
For those of you who are not familiar with a lot of these search engine terms, Sandy touched on it already, but I’ll just briefly whiz through. Essentially, paid inclusion or paid — I’m sorry, I need to go back. Paid search, rather, is essentially a way for advertisers or marketers to have their Web sites or pages listed more prominently in the search results. You may hear a number of terms — pay-per-click, pay for performance — it’s generally the same thing. It does affect your relevance or your ranking.
Secondly, there’s paid inclusion, which, as Sandy mentioned, is a way for you to have your Web site either indexed more frequently or spidered more frequently or deeply. However, it does not affect relevancy. And that’s an important distinction.
I wanted to recap quickly the research that Consumer Reports WebWatch has done relating to search engines. One of the first things that we learned in our first study by Princeton Survey Research Associates is that 60 percent of the consumers we polled — and we polled 1,500 Web-savvy consumers last year — 60 percent had no idea that some search engines charged fees in exchange for prominent placement of search results.
We also, in our findings from a study that we did in conjunction with Stanford University in which we found that about half the consumers in that study judged the credibility of a Web site in part on visual or superficial aspects of the site.
When you look at search engines specifically, consumers tended to judge the credibility of a search engine moreso on visual or superficial aspects than overall — than the average, say.
We also learned from the same Stanford study that less than 4 percent, a very small percentage, made any kind of comments as they assessed search engine sites that related to potential information bias. Just for comparison’s sake, on average, we got 11.6 percent information bias-related comments. And when you look at news sites, where people are really familiar [with the concept of information bias], we got 30 percent, a good third. Just for perspective.
As far as focusing on search engines, there are a number of reasons, but I’m going to zone in on the salient issues. For many consumers, search engines are perceived as a searching station, a way, a point of entry, particularly when you don’t know where to go, where to go next, where to start. Then you have the paid search factor which, to some degree, will nudge a consumer one way or another, based on who’s more highly ranked or not. And even well-informed or savvy Web users are finding it difficult to discern what’s paid, what isn’t, is it relevant, is it pure.
There’s the issue of consumer detriment. There are people who are making decisions, particularly if it’s finance or health-related, that if they use the information based on the site that they got that was highly or prominently ranked, which could in fact be inaccurate and, as a result, they may lose money, it might adversely affect their health, etc.
Just to make it very clear, Consumer Reports WebWatch has a very liberal definition of a search engine. Some people are very specific — that’s not a directory, this is pure search, this is pay-per-click. As far as we’re concerned, we’re interested in any site, any major site that has the opportunity to navigate or point thousands or hundreds of thousands of users to a destination. So that’s our definition.
After polling people via telephone and then also having them do a Web-based survey, we needed to figure out a way to study consumers’ attitudes and their behavior about search engines and paid search. And we decided that the best way to go is to do an anthropological approach. And the way to think about it quickly is think of Margaret Mead. And instead of looking at primitive cultures in New Guinea, she’s now looking on the Web and search engines and how people use it.
The merits of ethnographic or anthropological research, there are several. The first is you get to see them [consumers] participate in their natural surroundings, whether it be their computer at home, at work or at the office. It’s not artificial, like a lab and it’s not as artificial as a focus group, etc. You also have the advantage of real time exchange. You have a researcher ethnographer there, watching [a participant’s] behavior, and there’s an interchange between the two.
Real behavior and not opinion. The researcher can observe what people actually do versus what they say they do. And there could be dichotomy. They say they do one thing, but the researcher can actually notice, “Well, that’s not really true. I’m observing something differently.”
The why and not just the what. Ethnographic research gets at why people say — why people do what they say, rather. There’s also the issue of extended time. You could spend several hours with them versus one hour, a few minutes on the telephone. And there’s an interchange.
We’ve commissioned a firm in Baltimore called Context-Based Research; it’s a research group, they’re an anthropological and consulting firm there. And the report, or study, was led by Robbie Blinkoff, who’s the managing partner and the lead anthropologist. What makes this study unique is that it’s the first study, to my knowledge, done in the public sector. The search engines have done usability-related studies on their own sites, but of course it’s proprietary, so the average consumer has no access to that and it’s generally not released.
The questions that we hope to answer from this study. Building on the Princeton Survey statistics, we wanted to kind of assess are consumers still unaware of how paid search works and how it might influence rankings? Do consumers discern or tell the difference between a paid listing or what is considered pure algorithmic search? Do consumers notice and understand search links and all disclosure of pages that search engines have? And we wanted to gauge, now that it’s been roughly a year later from the FTC letters, how well do consumers navigate these disclosures? And then we of course wanted to know: Do consumers care? Once they know, do they care? Does their behavior change?
What this study doesn’t cover, just for clarity, after much going back and forth, we decided not to focus on paid inclusion, for the reasons that Sandy’s mentioned as well. It technically, or typically, does not taint the relevancy of the results, so we didn’t want to include that. And then, secondly, we were concerned about confusing the participants. Okay, we have paid search, paid listings and oh, and then here’s another thing, this odd thing called paid inclusion. So we decided it would just cause too much confusion.
Secondly, we did not make any attempt to assess whether the search results that the participants received were accurate or relevant — that wasn’t our job. It was whether or not they could discern the results and their behavior for or against it.
We have here the 15 sites that we selected for this study. It shouldn’t be a surprise. For most of them, they’re the leaders across the board. And I’ll quickly whip through why we selected them. Some of them are quite obvious.
We went through the top 30 most-trafficked sites based on quarterly traffic. We tried to capture the last online shopping market, the last quarter of 2002. And we based that on ComScore Media Metrix and Nielsen//NetRatings data. We also tried to monitor which search engines logged the most search queries per average visitor per month or per week. We also looked at data from Websidestory.com in which they did analysis on which search engines provided the most referrals — online shopping referrals to e-commerce sites.
And, lastly, we tried to take the list [of search companies] that the Federal Trade Commission began, based on the complaint that was submitted in 2001. Then in 2002, the FTC responded. And they, shall we say, that in 2001, the initial complaint mentioned eight companies, but the FTC felt, well, actually, I think a lot of these people should be getting letters for clear and better disclosure, so they raised the bar to 14 search companies.
So, to make it a level playing field, what Consumer Reports WebWatch did is said, Okay, we’ll include one search property from each of these 14 parent companies. We didn’t think it was fair to double-dip, you know, hit the site twice or hit the parent company twice, that wasn’t fair. Even though we did try to hit one property — oh, let me just say that it was the most-trafficked property among the parent company. Even though we tried diligently to just hit one property per parent company, of course, in the search engine industry, you blink and something happens. And there was merger and acquisition activity, thanks to Overture. As Doug mentioned, [the acquisitions of] AltaVista and Alltheweb were announced around the time mid-study. And then, even though Inktomi is more paid submission, it might change things. Yahoo announced the acquisition of Inktomi, so things kind of shifted.
Just to give you a quick recap of what the participants are all about. We had a total of 17 participants. We had four test cities in the United States. And, after much debate, we decided to go with lower profile, medium to large metropolitan areas, and I’ll explain why. Many cities typically dip into the huge markets — like New York, Boston, San Francisco. And we felt we didn’t want to go there. We wanted smaller markets, areas in which they’re not tapped and we wanted to get more of a varied voice of offline and online consumer attitudes and behavior. So thus, as you can see, we picked Kansas City, Phoenix, Providence and Raleigh-Durham, North Carolina. And, generally, it was the four regions: east, west, north, south, albeit crudely. There were four anthropologists in each city and they interview ed each participant at least twice. The study was in the field for the month of March.
A quick sketch of the participants as a group. There were eight men, nine women. And as you can see, we tried very diligently to select people across the board in age range. We even found two people, seniors, so to speak, with broadband — but we’ll get into that later — who took the test for us. It was very difficult, but we did locate them.
You’ll also see that we tried to get variance in households: Kids, no kids, college students, types of professions. And, just for perspective, the minimum online experience was five years online; the average was eight.
Let me tell you how we recruited [participants]. Before any participant joined the so-called panel that we had, they went through kind of a hurdle process before they were even admitted. The first thing was, we didn’t want any insiders or experts, so anybody who was a librarian — sorry, Sandy — journalists, IT tech person, etc., immediately, if they happened to mention that was their profession, we excluded them. They had to not know about pay for placement and we asked them an open-ended question and, based on their response, we could figure out whether or not they knew [about pay-for-placement]. We also did not want people who only surf the Web and chat rooms and send e-mail. We wanted people who searched frequently. So we had some kind of parameter, where they had to search online at least three days a week, that the average time was roughly 30 minutes to an hour. And that the search sessions lasted 20 minutes or more, more or less.
The big bar that we had to cross also was getting broadband users. We didn’t want any dial-up users. And although there was internal debate early in the project, we ultimately did not want any participant to be hampered by technological issues — they can’t download a page, it takes forever to get the query, etc. We wanted their experience to be as natural as possible, so ultimately we decided we only want high speed [users].
Q: Wouldn’t that skew the results towards high-end users? Or in some way skew the results?
LM: Generally, yes. In our participants’ grouping, we got a mix, we got high and low. I would agree with you generally, however — well, yes and no. I think generally it would, but I think we also were very particular about who we selected. So if you’re doing two thousand people, you might make sure that you get people from different groupings or at different levels. But, for us, we looked at different factors and having broadband was kind of the control, that was the only way we could control. We couldn’t control what an ISP [Internet Service Provider] did. We couldn’t control if they had any blockages or other problems with their dial-up. So, if that answers your question.
I’m going to go through the methodology very quickly. It’s kind of detailed and layered, but I’ll try to explain it as quickly and clearly as possible. Each ethnographer visited the participant twice. First, they had a contextual interview in which they talked, they gathered more information about the participants, their Web searching patterns and general online usage. They then had to do, with the participant, what we called five planned searches. And the participant — or, I should say, the ethnographer gave to the participant five pre-assigned search engine sites to assess. And we did this because we didn’t want the participant to only search sites that they knew. We wanted them to experience all kinds of sites and to gauge their pay-for-placement reaction, so to speak.
We wanted them to do two searches per search engine and we had them do a commercial search and an informational search. And by commercial, it was generally, we want you to purchase something online. You don’t necessarily have to close the transaction. Some people did, but we wanted them to focus on that. On the informational search, we had them focus specifically on either health, finance or travel. And we did this because we knew that consumer detriment would be highest in these areas. But they only got information. Say, I want to know about Acapulco for a travel trip. Health, I want to know about diabetes, something like that. And these were vested searches in that the participant got to pick the subject matter of their choice and the search terms used. So they would be more dedicated, spend more time searching.
Next, the ethnographer went through a sort of flow chart, to kind of earmark [participant] decisions, kind of meta steps. I did this, I did that, I did this, then I did that.
After that, the last thing the ethnographer did was what we called the enlightenment period. And that’s when they talked about pay for placement. And what we had the participant do is go through preselected disclosure links from each of the search engine sites. And that would include About Search links, how we do results or disclosure pages. So, for the example of AOL.com, a participant would go to the link called About AOL Search and they would read the whole thing and they would learn. And in there, they might learn that Google supplies results to AOL Search and then they reclick to Google Ad Words and then they would continue. In some cases, they could have gone through five or six links, as in the case of Infospace, because they have so many inter-deals, you know, it’s going to take a while. But they had to read it all and they kind of got it.
After that enlightenment period, they then had to do, over the course of several days, 10 more searches in the post-enlightenment mode. Like, a-ha, now that you’ve been told about it, we want you to do 10 more searches and record your attitudes and your behaviors. They were not forced to go back to the same sites. Most people did, but they could also add in other sites that were not included in the study, for example, HotBot or Excite.
Then we have visit number two, where the ethnographer went back to the participant, went over their attitudes and behavioral changes, quick flow chart, went over a quick browse with the participant, just to make sure they understood the process. And then, lastly, we asked the participants to propose guidelines for search engine disclosure, as well as tips for consumers if they were to go back online using search sites.
Key findings. Let’s go through this very quickly so we have time for Q&A. Essentially, all participants were surprised when they learned about pay per placement. Many had emotional reactions. One, from Vivien, who’s a Brown University sophomore: “Oh, I’m such a sucker; my fantasy world is wrecked.” So, for her to go through that process — she’s smart, she’s Web savvy, she thinks she’s got it. And then she feels like, wow, I’m such a sucker. So there was a sense of being let down, of not having it delivered.
As far as behavior, one field note mentioned Jack, who’s a graphic artist in Kansas City, that he, as a result of learning about pay for placement, he now scrolls down more pages. He looks through more options before he makes a decision.
We also found that participants said that paid search disclosures as they are now are often too hard to find and the information that is available is often not clearly written and is clearly written for an advertiser, not a consumer. And, for example, Dennis from Providence, who’s 40, said, “Users should not have to dig into layers of information to find disclosure on whether someone is paying to be listed on a search results page.” We got a lot of comments like this, but this was probably the most salient one.
As far as the whole term of “sponsored,” many participants said the term “sponsored” when you look at paid search listings and similar terms, like “feature,” etc., are confusing and potentially misleading to the consumer. Larry, who’s an empty-nester from Providence, said, “A sponsor is someone who gives money to support programs.” So many people looked at it as a charity thing or a PBS — some people mentioned, yes, like a charity, a charitable trust or something like that. They did not see that on search engines as meaning the same thing. So there’s a disconnect with consumers.
Participants were comfortable with pay per placement, as Matt mentioned, in the commercial sphere. So when they’re going to buy something, they don’t care if there’s pay for placement, because that might help them save time and whittle it down quickly. However, it bothered them when they were looking for information, because they were concerned about bias and getting the incorrect information. So as you can see, Claude mentioned, essentially, “It doesn’t bother me for commerce.” However, Sarah, from Raleigh-Durham, had a problem when she looked at health information. It was wrong and it really upset her.
Participants didn’t notice or comprehend the so-called inter-company deals in the search engine industry. And they didn’t understand how these deals might affect their results or their searching. And, for example, Vivien from Brown University again, the student, said she was surprised to see that Iwon was powered by Google. So they don’t quite understand that — “What is Google doing on Iwon? I don’t get that.” Again, there’s a disconnect.
And then also participants didn’t realize that more results doesn’t necessarily equal better results. And you could see from Dennis — oh, let me just go back. Sometimes some participants over-reacted. Instead of saying, Okay, paid search exists, they then said, Oh, everything is paid search, I can’t trust anything. And Dennis, for example, said, “Well, I assumed that this sponsored stuff on the right-hand side is an ad, but I don’t know.” So now they’re questioning themselves. Whereas Penny, from Phoenix, her strategy is, “I just skip anything that’s a sponsored link now.” So they just go to the extreme in some cases.
Lastly, some of the participants said that they predict that as consumers learn about paid search and how it works, that they’ll be more savvy about it in their searching strategy. And a wrap-up of the report will be coming out in June 2003. And then, lastly, I’ll also be putting out a report that’s related, that will talk about the business models of search engines, how they work, who owns whom, whose having a deal with whom as of that moment, so consumers will understand, oh, this is a business, this isn’t just an online repository of things where I can get good stuff. And that’s it.
EP: Thank you, Leslie. The issue of pay for search has come up a lot. And, interestingly, looking at a white paper that looked at search engines, the question — this is a statement that was made and I would like to get everybody’s response to this. It says, “Because pay for placement delivers traffic, the era of bias result sets has officially begun.” Is that true? Would you agree with that statement?
DL: Well, I don’t think it’s a surprise to anyone in this room that I don’t agree with that statement. But let me tell you why I don’t agree with that statement, which is more relevant. Payment in our model and in many — well, certainly in our model, let me just talk about Overture — in our model, payment does not equate to relevance. So it’s an important distinction. Payment allows for us to review a site and have it be potentially [in] the results. But relevance is a completely different determination. And, in fact, many times, I would submit, is much more relevant than algorithmic results, because we have a lot more steps in place to look at this. We have a team of a hundred editors who review the listings. You don’t get that kind of review on algorithmic results. We have a cost-per-click model, which you discussed, Leslie, which means that the advertisers are only paying if they’re getting good traffic, they’re only paying if they’re getting good clicks, so there’s built into the model is an idea of relevance.
We have something called a “click index” at Overture, which means if you’re not getting click-through, if the listings aren’t getting clicked on, they’ll be removed from the database for that search term. So there’s feedback with respect to users, if this isn’t relevant, then it doesn’t stay in the listings.
And then we have one of our most powerful relevance check, which is the other advertisers. If anytime a listing shows up for something that isn’t relevant to that — because we have a whole set of editorial guidelines that require relevance — we find out immediately from our advertisers who call in or send email to our bad listings department and tell us right away, this isn’t relevant to this, this listing is not relevant to this search term. And we immediately act upon that and take it down as we know, when we find out about it.
So all these filters, all these levels are ways that we provide, I think, a much greater relevancy in some types of results than you would get from algorithmic. And none of what I just talked about has anything to do with payment. Payment has to do with ranking, but not with relevancy.
So I don’t think there’s a bias there. I think you’re getting much better quality results for certain types of searches.
MC: I can take a quick stab at that from the Google philosophy as well. I think the answer is a little complex. I think for many, many sites, the era of biased search results has begun — a lot of people that don’t necessarily put users’ interests at their first priority. And I’m not talking about anybody in particular, I’m talking about sort of very small sites that people have never heard of, they sort of stumble across them and they realize maybe these search results are paid for, but they wouldn’t know, because they’re not clearly marked. I think that the lead players in the industry do try to mark their advertisements as well as they can.
I think that another thing related specifically to Google is present both sets of results — present commercial results, present editorial results. And try to make sure that the user is fully informed so that they have the ability to make that decision themselves. Within our commercial results, we also have additional checks. For example, it’s not just how much you pay that determines where you show up in the listings. We also sort by click-through. So if an ad is very helpful, people will click on it more often and that ad will show up higher in the search rankings. So even within the commercial search results, I do think that it’s possible to allow more relevance to come back. But I think it’s also extremely important that you present both perspectives to people, both commercial and editorial, and that you give them a clear way of choosing between those.
SS: I think there’s been a lot of good points brought out, but one of the things that has concerned me in pay-per-click is people who don’t have a big budget, who might have a fabulous site out there that’s very, very relevant and just don’t have the money to participate with the big players in pay-per-click. And that’s one of the reasons why I feel that, yes, you’re going to end up with some bias there, because the money is not there for everybody to participate in some of these plans. And so I think the importance of having the editorial results very clearly marked is that these people who might have a wonderful site and can’t afford the advertising dollars, they do have a chance of being pulled up high in the rankings.
LM: To add to what Sandy said, I agree exactly, that’s one argument I would have. But also, from the consumer perspective, research has shown that consumers assign relevancy or ranking as more important, more relevant. For example, Amanda Watlington, from iProspect, her research, as well as Amanda Spink, I believe, from Penn State University, have done research. She did, the Penn State professor, did research on Excite.com data. And, basically, search engines know this, that people are going to click at the top, because they think it’s scientific, more relevant and that’s why they have pay-per-click to begin with. And that’s why people are fighting to get the top, or at least the top five spots, because they know that typically consumers are not going to go past the fourth or fifth page. So there lies the problem. What about the people who are on the sixth-plus page?
EP: Doug, you wanted to respond, before we entertain questions from the audience?
DL: Oh, yeah, I’ll try to be very brief. I just — this distinction we’re making between editorial listings and paid listings is one that confuses me. Because, for example, our listings, we have a team of a hundred editors I mentioned that review the listings; this is an editorial function. Whereas, many of the algorithmic pages are simply crawled and indexed using a mathematical formula, which I think of as something different than editorial.
So not that — the difference is only in that the methodologies provide different levels of quality for different types of searches. And that having a human read something and make a determination of relevance, in some instances may be much better, in some instances worse than having a computer do this. But it’s I think somewhat confusing to describe one as editorial and the other as something other than editorial.
MC: And if I could just clarify very briefly before we go on — I’m sort of borrowing the nomenclature from newspapers, where there’s sort of a church versus state. And so editorial results are those which are written by reporters, and money is not supposed to enter into them. And so when I say editorial, I don’t mean a team of editors has looked at these pages, because it’s the exact opposite. We try to come up with robust, scaleable algorithms. But what I should say is money is not involved versus money is involved.
EP: Okay. Let’s just take these first two questions. Yes, sir?
Q: Two questions. One is: Did your research rank the search engines on the basis of which were the, I don’t know, most honest or most clear-cut and which were the worst offenders, most deceptive?
LM: No, that would be really difficult for us to do.
EP: Leslie, could you repeat the question first before you answer it?
LM: Okay, you were asking did Consumer Reports WebWatch — which search engines were more relevant and more accurate? That was the question?
Q: Or more deceptive.
LM: No, we did not. We were most concerned about how consumers perceived [them] and their attitudes, but for us to figure out what’s deceptive and what isn’t, what’s relevant and what isn’t would be very difficult to do.
Q: Okay, then let me rephrase it. Did you rank the search engines according to what the consumers you tested liked best, liked worst?
LM: Well, we only had 17 participants, so we couldn’t rank anything. But what we might do — again, we just got the data and are going through it now, but what we’ll probably do is, say Participant X said this about this searching site; however, these participants didn’t like that as it pertained to pay-per-click. But we can’t rank it, we only had 17 participants.
Q: Okay, another very quick question —
EP: Before you ask another question, let’s take a question from there — yes, thank you?
Q: Just to put a little context on this, I’ve been working with some standard [unintelligible] on best practices to help identify ways of improving the quality search. Which actually, some are orthogonal to the current discussion, but relevant in the following sense, a topic we’ve addressed in part is how to get the right information to the right person at the right time.
And the sources of information are not infinite, but very, very large. [unintelligible] number Google comes to mind, but we’ll put that aside.
And it strikes me that one of the key differences between paid inclusion and the automation is the breadth of sources that can actually be embraced and maybe to some degree, the issue that Leslie brought up, whether it’s a commercial search or whether it’s an informational search. I work at a university currently. I can’t envision a university ever paying — well, I could see them paying to have their catalogs put in there. But there certainly, like the professors’ or the students’ pages in there, there’s no funding for that. Which means it’s only going to come in what I call informational and yet that may be a critical source, the Stanford study, for example, may only be available in the non-paid space. So I guess I’m interested in the panel’s feedback as to where the boundaries are between paid inclusion, the breadth of search, what that means for the kinds of searches people might be pursuing.
MC: I can take a quick stab at that and just say, yeah, I completely agree, there are billions of pages on the Web. And the vast majority of them are probably not commercial. And so, in particular, Google’s advertising works by matching the key word or the key word phrase that people type in. So we never show what are known as “run of site” ads, which are completely unrelated to what the user typed into the search box. If someone types in “sneakers,” we’re only going to show ads for sneakers. Now, I think it is important to clearly put that in a place where it’s marked as advertising. But I think that there’s definitely value to being able to access a comprehensive copy of the Web, because that’s where most of the information lies.
DL: I would agree with that, but I would also say that there are challenges and opportunities to be reached in trying to determine whether something is commercial or informational or how commercial, what the user’s intent is and then what results to give them. So one example we use internally is the word “jaguar.” When someone searches on the word “jaguar,” we don’t necessarily know if they mean a sports team, a sports car or an animal. It could mean any of those things. And so getting smarter about what the user means is a way to create a better search. And sometimes, would someone who is offering information on a jaguar animal be able to bid in a marketplace or show up in a marketplace against people who are selling Jaguar cars? Probably not. But if we were able to understand better what you’re looking for, then we could probably deaggregate those marketplaces and allow for lower cost, more informational type searches to become more accessible. That’s a challenge that we’re all trying to meet, I think.
EP: Yes, there are a number of questions, I will entertain them, but let me just make a statement and draw another question. Because since this whole issue is related to the issue of trust. And implicit, without being overly stated, implicit in our assumptions and the questions we’re asking and the statements that are being made is the issue of trust. So what is trust on the Web? B.J. Fogg describes credibility as being a perceived quality that has two dimensions, a dimension of trustworthiness and excellence.
And then, what he says about trust is a question of a sense of fairness and lack of bias. So what can search engines do to provide appropriate trust for the consumer? So does the issue of placement define that or does it make it more difficult?
So, having said that, we’ll entertain some more questions. Okay.
Q: Yes, Matt, this is kind of a question for you, although, really, I think it goes across the whole search industry. The results from Consumer Reports WebWatch pointed out something through their research that I’ve heard [unintelligible], which is the term “sponsorship” is misunderstood and yet here you said again, “Oh, our ads are clearly marked.” Well, clearly, from the research, no, they’re not clearly marked. They’re marked with a term that can be construed as misleading, therefore not to be trusted entirely.
Now, I realize all the other sites use some kind of term, “sponsor” this, “sponsor” that. It seems like it’s the veil that the search engine industry has decided to hide behind to label sponsorship. What’s your reaction, if nothing else, to the research results that say, are you going to go back to your company and say, “Gee, maybe we ought to figure a better term”?
MC: I’m certainly going to raise this issue, yes. Let me just tackle that very briefly. We do our own user studies. We actually have a user interface lab where we have the one-way mirror and we watch people. I can honestly say this is the first time I’ve heard someone say that “sponsored link” is unclear. I’ve certainly [heard] people inside Google say, “You know, let’s try to make that ‘featured listing’.” And we’re like, no, no, we want to make that more explicit.
But, I mean, that’s the sort of struggle that you face every day. I can tell you, for example, back in January, we darkened our ads slightly, to make them stand out more from the background of the page. We are constantly doing experiments. We’ve done experiments in Japan where we do very bright yellow ads. If you find on the chat rooms there’s lots of examples where we experiment with our ads as much as we can to try to find good ways of marking them so that people know things.
It’s certainly an action item for me to go back and say, “Here’s one thing that we could do better.” And then we’ll test that and verify that ourselves and see what we see.
Q: I think I called it a veil, because the common term — I’m curious why you feel the industry doesn’t just simply say “advertisements.” As in newspapers, where for years they’ve said “paid ad” on top. Why doesn’t it simply use that word “ad” that everyone knows?
MC: Honestly, I had never had the connotation of “sponsored” immediately leads to PBS. But I’m seeing — but we actually, I actually thought it was pretty clear before today, but I’ll certainly go back and raise that as an issue.
EP: Okay, I feel like a traffic cop here. The person just behind the gentleman, yes?
Q: Sure. The question I think Leslie raised raises it well. It’s really, if I’m hearing right, it’s really a matter of real estate. The user — and, again, I’ve seen this in my own research — the user will not go beyond a certain point. And it’s how to use that real estate well. If you block the whole, if you will, above the fold, to borrow Matt’s use of newspaper terminology, with paid material, it may be — and I’m not saying it is — your research suggests that perhaps the user could find this as distasteful in some way.
It’s really, back into what Jakob [Nielsen] was saying: It’s really a user interface issue. It’s making sure that the user can perceive and everybody’s needs are met. And that’s my — because there is the user’s behavior of not going beyond a certain point. Once we accept that and move forward, it becomes a different issue.
Q: Can I be a naysayer about the word “sponsor?”
EP: Yes.
Q: I mean, I’m a child of the television era and I’m sure there’s probably no one in the room who is from the radio era. Who we’ve always heard the words, “And now for a word from our sponsor.” No one thought they were a benefactor in any way. So if you can’t translate that into the Web, the sponsor is going to be an advertiser. It’s a nice, euphemistic way of saying advertiser. But I think most people know that a sponsor is basically someone who’s paid money for something.
DL: I just might point out — not that I’m at all critical of your study, because I think it provides a valuable insight into certain aspect of people using the Web — but you’ve selected, it seemed like, people who specifically did not know what paid placement. So the sample was specifically targeted to people who didn’t know what this was. So if you knew what sponsored listings were, then you were out of the sample.
LM: I think we actually — it was people who didn’t know that the practice was so prevalent in search [engines].
DL: Okay, I don’t know, but I —
LM: To clarify, we had an open-ended question and it was something to the effect of, “Can you explain how a typical search engine works?” And most people said, “I don’t know.” So you’re in. Some people said, “Well, I think it’s based on key word.” But no one said, “There’s paid search, there are paid listings.” It was mostly “I don’t know.” And just to double emphasize again, these are people who are smart. Most of them are college educated. They’re broadband users, so they’re not tech-adverse. These are people who are smart and yet they didn’t get it, so what about those people who are still dialing up? They’re probably going to have problems as well.
EP: Okay, the gentleman in the yellow dotted tie.
Q: I have a question about news.google.com, which fascinates me. At one point in my career, I was an editor and that’s the complete repudiation of my entire professional history. So, fascinating. What that does, incidentally, is it automatically, without the intervention of humans, generates — it searches 4,000 or something sites and generates the results. Now, I notice you don’t have ads on that, or you don’t have ads on it yet. Are you thinking of having ads and do people really want to be placed next to something from the North Korean news agency or sources they might not perceive as credible?
MC: That’s a great — so the question is, Google News searches over 4,000 sources and what’s the philosophy on showing ads or monetizing that in some way? So the first point I’d make is I don’t think that Google News is at all a repudiation of editors. I think it can only exist because 4,000 editors cull what’s really interesting and we look at the output of those sites and we can say, okay, these three editors all put up this story at the same time and it’s only been an hour, and so this is a really interesting, relevant story. So Google News could not function without the editorial capacity of all those Web sites.
The simple answer about monetization is we don’t worry about that when we put out a project. We’re not worried about how we’ll make money off of it. We have a Google Labs project in which people just put things that are fun, things that they think are neat. And we sort of see how many people use it, what sort of response rate we get. So this was literally one research engineer who said, “You know, I could probably come up with a good algorithm to spot breaking news.” And he pushed it a little ways and then people said, “This is cool, let’s pick it up and run with it.” And so it came out in that way.
I don’t think we’re at all worried about the monetization aspect. If at some point we charge money for it, we’d have to be very clear and make sure the ads were clearly marked. But I don’t think there are any plans at this time. It’s interesting to contrast Consumer Reports, which is not advertiser-supported at all, it’s all subscription, but that’s much harder to do on the Web, because people are much less willing to give up their personal information.
So I think at some point it’d be great if we could offer news searches for subscription or something like that. But I don’t see us putting ads on Google News anytime soon.
Q: Just one quick follow-up —
EP: Could I—? Since we have such a limited time and, okay?
Q: Yes, I have a question for Overture. Are there any guidelines that you have for your partner sites about how they have to label their ads? It seems there’s no consistency across, from site to site.
DL: So, everyone is aware of the FTC’s guidance on this stuff, that the staff opinions, with respect to our partners. And what we do is provide them with the search results. We don’t pretend to know what their users are thinking. We know what our users are thinking and we label our results on Overture.com in a way that’s designed to meet their needs. Now, their needs are very particular, most of the people who are searching on Overture.com are advertisers who are searching to find out where they have to bid in order to place themselves in certain listings. So we don’t have the access or the studies on what the users of a Yahoo or an MSN are thinking, but we feel fairly comfortable letting them do that part of their business.
EP: Question down there?
Q: Yes. I have a two, it’s kind of a two-part thing. I want to describe something and I want to ask both Overture and Google. Number one, is this technically feasible? And, number two, is it feasible from a business standpoint?
What if the user got results and what first comes up is what you currently have, but then you have a set of screens that allow you to screen all of the results that are edus, all of the results that — along with whether there’s been payment or not? So that the user can use the set of screens to verify the source and what might be behind the motives behind the ranking.
EP: Before you respond, that is a good question, because links to one of the other [unintelligible], because we make the assumption that — I mean, there is something wrong with pay for placement. And then we assume that the algorithmic method is consequently sacred and good. And yet the algorithmic method gives us marketplace determination. Which could be used in a commercial situation, but in a health situation or a legal situation, it may not be and this is where your issue about using edu or where you — there’s a difference between marketplace determination and expertise based on the question and the context of the question. So could you relate those two?
MC: Definitely. The quick answer is, you can always do site.edu or site.gov in your key words and that will restrict your search only to that domain, only educational institutions or to government sites. That’s a very good way to find information.
Google also has what’s known as the Google Web APIs, which allows you to do up to a thousand searches per day and basically say as long as it’s for non-commercial use, you can rig up your own special search which only searches educational institutions or things like that. I think that Google is the only search engine that does not allow any sort of pay for inclusion, because we haven’t found a way to make sure that that’s completely conflict-free. And so there’s no — you wouldn’t have to screen out where money was involved, because there’s no money involved in those search results.
I do think that there’s definitely a need to educate the users more about power tools like site colon, a lot of people don’t realize you can use the minus sign to subtract a term from your search results. Or double quotes to say, “I want to match only this phrase.”
So I do think that it’s incredibly important to have those sorts of features and also to get the word out about them.
DL: I would agree with all of that. I think we take what users do and try and learn from that. And so we are very cognizant of the way a search session progresses. So if you search for something and then you search, you refine that search in another way and you refine that search again and you’re looking for something that’s more relevant, and we are constantly — I’m sure you guys are, too — looking for ways to use that information to try and better tailor results to the first search. But there’s only so much we can do at the first site, which is that we’re just now, Overture’s just now getting into the algorithmic business. And we haven’t even closed the AltaVista deal, so it’s a little premature for me to comment on this stuff. But I’ve seen, as a consumer, and maybe even on Google, things like “See more pages like this.” Or stuff like that. So you can build upon the search results to find more relevant sites based on the ones you found in the first search. And those types of things are ways to hone in on what you’re talking about, I think.