Library and Information Service >
A Methodological Survey on the Study of User Relevance Judgment in Context of Cognitive Turn
Received date: 2014-07-09
Revised date: 2014-08-14
Online published: 2014-09-20
The paper constructs a set of sample documents with 55 representative research documents from 82 published during 1967 to 2013. A survey on these sample documents shows that there are many common and regular ideas and operations among methodology thoughts and specific research methods, such methodological consensus form a reference methodological framework which 1) takes its core from three principles including situation dependency of user relevance judgment, cognitive main-factors of relevance judgment and operating research in real situation, 2) covers vertically the layers of methodology philosophy, research strategy and research planning, and 3) horizontally touches some key procedures of making a research plan such as sampling, data collecting and analyzing. The paper concludes that the cognitive view in the domain of information seeking and retrieval (ISR) is the main driver for the forming, developing and further evolving the framework. By the viewpoint the paper forecasts the future development of the framework.
Wang Jian , Wang Zhiqiang , Liu Qian , Cui Yunpeng , Zhao Hua , Wang Jian , Man Rui . A Methodological Survey on the Study of User Relevance Judgment in Context of Cognitive Turn[J]. Library and Information Service, 2014 , 58(18) : 66 -76 . DOI: 10.13266/j.issn.0252-3116.2014.18.010
[1] Ingwersen P, Jarvelin K. The turn: Integration of information seeking and retrieval in context(the information retrieval series)[M]. New York: Springer-Verlag, Inc., 2005.
[2] Robertson S E, Hancock-Beaulieu M M.On the evaluation of IR systems[J]. Information Processing & Management, 1992, 28(4): 457-466.
[3] Cuadra C A, Katter R V. Opening the black box of “relevance”[J]. Journal of Documentation, 1967, 23(4): 291-303.
[4] Rees A M, Schultz D G. A field experimental approach to the study of relevance assessments in relation to document searching[R]. Cleveland: Case Western Reserve University,1967.
[5] Schamber L. Users' criteria for evaluation in multimedia information seeking and use situations[D]. New York: Syracuse University, 1991.
[6] Park T K.The nature of relevance in information retrieval: An empirical study[J]. The Library Quarterly, 1993,63(3):318-351.
[7] Cuadra C A.Experimental studies of relevance judgments. Final report. Volume 1:Project summary[M].Santa Monica: System Development Corporation, 1967.
[8] Bateman J A. Modeling changes in end-user relevance criteria: An information seeking study[D]. Texas: University of North Texas, 1998.
[9] Vakkari P, Hakala N. Changes in relevance criteria and problem stages in task performance[J]. Journal of Documentation, 2000, 56(5): 540-562.
[10] Eisenberg M, Barry C. Order effects: A study of the possible influence of presentation order on user judgments of document relevance[J]. Journal of the American Society for Information Science, 1988, 39(5): 293-300.
[11] Janes J W, McKinney R. Relevance judgments of actual users and secondary judges: A comparative study[J]. The Library Quarterly, 1992, 62(2): 150-168.
[12] Janes J W. Other people's judgments:A comparison of users' and others' judgments of document relevance, topicality, and utility[J]. Journal of the American Society for Information Science, 1994, 45(3): 160-171.
[13] Spink A, Greisdorf H, Bateman J. Examining different regions of relevance: From highly relevant to not relevant[C]//Proceedings of the ASIS Annual Meeting. Pittsburgh: American Society for Information Science and Technology, 1998: 3-12.
[14] Hirsh S G. Children's relevance criteria and information seeking on electronic resources[J]. Journal of the American Society for Information Science, 1999, 50(14): 1265-1283.
[15] Cool C, Belkin N, Frieder O, et al. Characteristics of text affecting relevance judgments[C]//Proceedings of the 14th National Online Meeting. New York: Learned Information Inc., 1993: 77-81.
[16] Janes J W. Relevance judgments and the incremental presentation of document representations[J]. Information Processing & Management, 1991, 27(6): 629-646.
[17] Maglaughlin K L, Sonnenwald D H. User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments[J]. Journal of the American Society for Information Science and Technology, 2002, 53(5): 327-342.
[18] Sedghi S, Sanderson M, Clough P. A study on the relevance criteria for medical images[J]. Pattern Recognition Letters, 2008, 29(15): 2046-2057.
[19] Savolainen R, Kari J. User-defined relevance criteria in Web searching[J]. Journal of Documentation, 2006, 62(6): 685-707.
[20] Taylor A R, Cool C, Belkin N J,et al. Relationships between categories of relevance criteria and stage in task completion[J]. Information Processing & Management, 2007, 43(4): 1071- 1084.
[21] Taylor A. Relevance criterion choices in relation to search progress[D]. New Brunswick: Rutgers University, 2009.
[22] Taylor A, Zhang Xiangmin, Amadio W J. Examination of relevance criteria choices and the information search process[J]. Journal of Documentation, 2009, 65(5): 719-744.
[23] Papaeconomou C, Zijlema A F, Ingwersen P. Searchers' relevance judgments and criteria in evaluating Web pages in a learning style perspective[C]//Proceedings of the Second International Symposium on Information Interaction in Context. Texas: ACM, 2008: 123-132.
[24] Burton V T, Chadwick S A. Investigating the practices of student researchers: Patterns of use and criteria for use of Internet and library sources[J]. Computers and Composition, 2000, 17(3): 309-328.
[25] Hamid R A, Thom J A. Criteria that have an effect on users while making image relevance judgments[C]//Fifteenth Australasian Document Computing Symposium. Melbourne:RMIT University, 2010: 1-8.
[26] Balatsoukas P, Ruthven I. What eyes can tell about the use of relevance criteria during predictive relevance judgment?[C]//Proceedings of the Third Symposium on Information Interaction in Context. Texas: ACM, 2010: 389-394.
[27] Balatsoukas P, Ruthven I. An eye-tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine[J]. Journal of the American Society for Information Science and Technology, 2012, 63(9): 1728-1746.
[28] Balatsoukas P, Ruthven I. The use of relevance criteria during predictive judgment: An eye tracking approach[J]. Proceedings of the American Society for Information Science and Technology, 2010, 47(1): 1-10.
[29] Inskip C, MacFarlane A, Rafferty P. Creative professional users' musical relevance criteria[J]. Journal of Information Science, 2010, 36(4): 517-529.
[30] Savolainen R. Source preference criteria in the context of everyday projects: Relevance judgments made by prospective home buyers[J]. Journal of Documentation, 2010, 66(1): 70-92.
[31] Laplante A. Users' relevance criteria in music retrieval in everyday life: An exploratory study[C]//Proceedings of the 11th International Society for Music Information Retrieval Conference. Utrecht: ISMIR, 2010: 601-606.
[32] Taylor A. User relevance criteria choices and the information search process[J]. Information Processing & Management, 2012, 48(1): 136-153.
[33] Taylor A. Examination of work task and criteria choices for the relevance judgment process[J]. Journal of Documentation, 2013, 69(4): 523-544.
[34] Janes J W. The binary nature of continuous relevance judgments: A study of users' perceptions[J]. Journal of the American Society for Information Science, 1991, 42(10): 754-756.
[35] Wang Peiling. Contextualizing user relevance criteria:A meta-ethnographic approach to user-centered relevance studies[C]//Proceedings of the Third Symposium on Information Interaction in Context. Texas: ACM, 2010: 293-298.
[36] Wang Peiling. A cognitive model of document selection of real users of information retrieval systems[M]. College Park: University of Maryland at College Park, 1994.
[37] Tang Rong, Solomon P. Toward an understanding of the dynamics of relevance judgment: An analysis of one person's search behavior[J]. Information Processing & Management, 1998, 34(2): 237-256.
[38] Wang Peiling, Soergel D. A cognitive model of document use during a research project. Study I. Document selection[J]. Journal of the American Society for Information Science, 1998, 49(2): 115-133.
[39] Wang Peiling, Domas White M. A cognitive model of document use during a research project. Study ll. Decisions at the reading and citing stages[J]. Journal of the American Society for Information Science, 1999, 50(2): 98-114.
[40] Howard D L. Pertinence as reflected in personal constructs[J]. Journal of the American Society for Information Science, 1994, 45(3): 172-185.
[41] Tang Rong, Solomon P. Use of relevance criteria across stages of document evaluation: On the complementarity of experimental and naturalistic studies[J]. Journal of the American Society for information Science and Technology, 2001, 52(8): 676-685.
[42] Kim S, Oh J S, Oh S. Best-answer selection criteria in a social Q&A site from the user-oriented relevance perspective[J]. Proceedings of the American Society for Information Science and Technology, 2007, 44(1): 1-15.
[43] 成颖. 信息检索相关性判据及应用研究[D]. 南京: 南京大学, 2011.
[44] Barry C L. The identification of user criteria of relevance and document characteristics: Beyond the topical approach to information retrieval[D]. New York: Syracuse University, 1993.
[45] Park T K. The nature of relevance in information retrieval: An empirical study[J]. The Library Quarterly, 1993, 63(3): 318-351.
[46] Meng Yang, Marchionini G. Exploring users' video relevance criteria-A pilot study[C]//Proceedings of the American Society for Information Science and Technology. Providence: American Society for Information Science and Technology, 2004: 229-238.
[47] Barry C L. User-defined relevance criteria:An exploratory study[J]. Journal of American Society and Information Science, 1994, 45(3): 149-159.
[48] Nilan M S, Peek R P, Snyder H W. A methodology for tapping user evaluation behaviors: An exploration of users' strategy, source and information evaluating[C]//Proceedings of the 51st Annual Meeting of the American Society for Information Science. Atlanta: American Society for Information Science and Technology,1988:152-159.
[49] Crystal A, Greenberg J. Relevance criteria identified by health information users during Web searches[J]. Journal of the American Society for information Science and Technology, 2006, 57(10): 1368-1382.
[50] Lawley K N, Soergel D, Huang Xiaoli. Relevance criteria used by teachers in selecting oral history materials[J]. Proceedings of the American Society for Information Science and Technology, 2005, 42(1): 421-448.
[51] Westbrook L. Faculty relevance criteria: Internalized user needs[J]. Library Trends, 2001, 50(2): 197-206.
[52] Park T K. Toward a theory of user-based relevance: A call for a new paradigm of inquiry[J]. Journal of the American Society for Information Science, 1994, 45(3): 135-141.
[53] Schamber L, Bateman J. User criteria in relevance evaluation: Toward development of a measurement scale[C]//Proceedings of the Annual Meeting-American Society for Information Science. Baltirnore: American Society for Information Science and Technology, 1996: 218-225.
[54] Tombros A, Ruthven I, Jose J M. Searchers' criteria for assessing Web pages[C]//Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval. Texas: ACM, 2003: 385-386.
[55] 于春, 彭爱东, 王波,等. 信息用户对信息检索相关性判断的因素分析[J]. 图书情报工作, 2009，53(3): 103-107.
[56] Barnes M D, Penrod C, Neiger B L,et al. Measuring the relevance of evaluation criteria among health information seekers on the Internet[J]. Journal of Health Psychology, 2003, 8(1): 71-82.
[57] Goodrum A, Pope R, Godo E, et al. Newsblog relevance: Applying relevance criteria to news-related blogs[J]. Proceedings of the American Society for Information Science and Technology, 2010, 47(1): 1-2.
[58] 童迎, 李鹏, 夏慧. 合作信息查寻与检索相关性判据的探索性因子分析[J]. 图书情报工作, 2013, 57(19): 30-36.
[59] Schamber L. Time-line interviews and inductive content analysis:Their effectiveness for exploring cognitive behaviors[J]. Journal of the American Society for Information Science, 2000, 51(8): 734-744.
[60] Case D O. Looking for information: A survey of research on information seeking, needs and behavior[M]. Bingley: Emerald Group Publishing, 2012.
/
| 〈 |
|
〉 |
