Evaluating Recommender Systems.ppt

menderSystemsAmyriadoftechniqueshasbeenproposed,butWhichoneisthebestinagivenapplicationdomain?parativeanalysisbasedonanoptimalitycriterion?Researchquestionsare:uracy,usersatisfaction,responsetime,serendipity,onlineconversion,ramp-upefforts,….Docustomerslike/mendeditems?Docustomersbuyitemstheyotherwisewouldhavenot?mendationafterpurchase?EmpiricalresearchCharacterizingdimensions:Whoisthesubjectthatisinthefocusofresearch?Whatresearchmethodsareapplied?Inwhichsettingdoestheresearchtakeplace?SubjectOnlinecustomers,students,puters,…ResearchmethodExperiments,quasi-experiments,non-experimentalresearchSettingLab,real-worldscenariosEvaluationsettingsLabstudiesExpresslycreatedforthepurposeofthestudyExtraneousvariablescanbecontrolledmoreeasybyselectingstudyparticipantsButdoubtsmayexistaboutparticipantsmotivatedbymoneyorprizesParticipantsshouldbehaveastheywouldinareal-worldenviromentFieldstudiesConductedinanpreexistingreal--experimental(observational)researchmethodsExperiment(test,trial):"Anexperimentisastudyinwhichatleastonevariableismanipulatedandunitsarerandomlyassignedtodifferentlevelsorcategoriesofmanipulatedvariable(s)."Units:users,historicsessions,…Manipulatedvariable: typeofRS,mendeditems, explanationstrategies…Categoriesofmanipulatedvariable(s):content-basedRS,collaborativeRSExperimentdesignsEvaluationininformationretrieval(IR)HistoricalCranfieldcollection(late1950s)1,398journalarticleabstracts225queriesExhaustiverelevancejudgements(over300K)mendeditemsMetrics:mendationisviewedasinformationretrievaltask:Retrieve(mend)allitemswhicharepredictedtobe“good”.Precision:ameasureofexactness,:pleteness,,recalldecreasesasaresult(orviceversa)parisonpurpo