Data-DrivenBatchScheduling.ppt

Data-,MadisonMay31,essful,widely-deployedtechnologyFrequentlyusedacrossthesciencesAswellasinvideoproduction,documentprocessing,datamining,financialservices,ingproblematicProblemwithEnvironmentalTransparencyEmergingtrendsDatasetsaregettinglarger[Roselli00,Zadok04,******@CERN]Datagrowthoutpacingincreaseinprocessingability[Gray03]Morewide-arearesourcesareavailable[Foster01,Thain04]essfultechnologybeyonddesign[Wall02]BatchschedulersareCPU-centricIgnoredatadependenciesDatamovementhappensasaside-pute-intensiveworkloadsexecutinglocallyHorriblefordata-intensiveworkloadsexecutingremotelyHowtorundata-intensiveworkloadsremotely?ApproachestoRemoteExecutionDirectI/O:essSimpleandreliableLargethroughputhitPrestagingData:ManipulatingenvironmentAllowsremoteexecutionwithoutperformancehitHighuserburdenFaulttolerance,restartsaredifficultDistributedFileSystems:AFS,NFS,-monFundamentalProblemLayersuponCPU-centricschedulingDataplanningexternaltothescheduler“Secondclasscitizeninbatchscheduling”Data-awareschedulingpoliciesneededKnowledgeofI/ObehaviorofbatchworkloadsSupportfromthefilesystemData-drivenbatchschedulerOutlineIntroProfilingdata-intensivebatchworkloadsFilesystemsupportData-drivenschedulingConclusionsandcontributionsSelectingWorkloadstoProfileGeneralcharacteristicsofbatchworkloads?SelectarepresentativesuiteofworkloadsBLAST biologyIBIS ecologyCMS physicsHartree-Fock chemistryNautilus moleculardynamicsAMANDA ******@home astronomyKeyObservation:Batch-plexjobanddatadependenciesThreetypesofdataanddatasharingVerticalrelationships:esinputtochildHorizontalrelationships:BatchesUsersoftensubmitmultiplepipelinesMaybereadsharingacrosssiblingsBatch-essIBIS,CMScloseto100%,HF~80%NoisolatedpipelineisresourceintensiveMaxmemoryis500MBMaxdiskbandwidthis8MB/sAmdahlandGraybalancesskewedDrasticallyoverprovisionedintermsofI/Obandwidthandmemorycapacity