ABSTRACT
This paper uncovers a new phenomenon in web search that
we call domain bias — a user’s propensity to believe that
a page is more relevant just because it comes from a par-
ticular domain. We provide evidence of the existence of
domain bias in click activity as well as in human judgments
via a comprehensive collection of experiments. We begin
by studying the difference between domains that a search
engine surfaces and that users click. Surprisingly, we find
that despite changes in the overall distribution of surfaced
domains, there has not been a comparable shift in the dis-
tribution of clicked domains. Users seem to have learned the
landscape of the internet and their click behavior has thus
become more predictable over time. Next, we run a blind
domain test, akin to a Pepsi/Coke taste test, to determine
whether domains can shift a user’s opinion of which page
is more relevant. We find that domains can actually flip a
user’s preference about 25% of the time. Finally, we demon-
strate the existence of systematic domain preferences, even
after factoring out confounding issues such as position bias
and relevance, two factors that have been used extensively in
past work to explain user behavior. The existence of domain
bias has numerous consequences including, for example, the
importance of discounting click activity from reputable do-
mains.
