16_与大象共舞——持续交付高质量的产品

DanceswithElephantsBuildHighQualityProductwithContinuousDeliveryDANGZhengfaJuly,2011FreeWheel©FreeWheel2011 ThePremiumVideoEcoSystem22AdNetworksSERVINGTHEVIDEOREVOLUTION.AllKeyDistributorsConnectedTV’s,devices,platformsConfidential2 FreeWheelClientsConfidential3 Challenges•Technic–CommontoallInternetcompanies•UI:sophisticatedworkflow,Enormousvalidations•AdServing:Highperformance,highavailability•PlayerSDK:Diversifiedplatforms(flash,IOS,HTML5,android,Silverlight…)•Business–SpecialchallengesforFreeWheel•Giantclientswithhugeimpacts•Certificationrequired:SAS70,MRC,IAB,andetc•StrictSLAandhighpenalty•Slowpaceclients,butquickreactionrequiredConfidential4 Confidential5 DanceswithElephantsConfidential6 1.MindYourStep:DefineFeatures2.ContinuousIntegration:BuildSoftware3.ManagedDeployment4.ContinuousTestinginProductionConfidential7 1.MindYourStep:FeaturesDatabaseConfidential8 1.MindYourStep:ScrumCodeFreezingFreeWheelCalendar1stWeek2ndWeek3rdWeek4thWeekLastWeekV#1SprintTimeline(5weeks)ReqclarificationTechDesign&ImplementationIntegrationTestingTestingonstagingTechdesignStagingReleaseIC1:Design/IC2:IC3:CodeImplementationreviewtookplaceTestcasesFreezingQualityAssuranceorhavebeenreviewbookedC2-1.EngSoftC2-2.EngHardC3.PMFinishC5.StagingRel.C6.Prod.Rel.C1.PMRankCommitmentscommitmentsDetailsdesignInfrastructurerank1stWeek2ndWeek3rdWeek4thWeekLastWeekPrioritizationDebatingandBeginreqdesignElaboratereqCP0ReqTechDesignIntegrationTestingonEstimationLockdowntheanddiscussionClarificationImplementationTestingonStagingitemsTechdesignStagingReleasetoProd.EngplanningImplementationIC0:EngplandoneQualityAssuranceV#2SprintTimeline(5weeks)OverlappedfourweeksNOTE:30%ENGresourcesreservedforrefactoring&emergency!Confidential 1.MindYourStep•LimitedResources•Don’tbeafraidtoshareyourplanswithclients•Pre-releasenoes,releasenote•KeepSchedule•Manageclients’expectation•Thingscan’tbeperfectatonce•BetaConfidential10 2.ContinuousIntegration•CentralizedIntegrationServer•Unifiedtools/libraries•Codingstyle,syntaxcheck•Triggeredbycheck-in,orperiodically•Integratedwithautomatedtesting•EmailNotification•AutomatedTesting•Frameworks•Units/E2E•Quick/FullMode•TestCasesPrioritizationConfidential11 2.ContinuousIntegration:AdServerAdServerRequestResponsePlayerSimulatorLuaCallbacksResponseOutputBaseRunDataRegressionResultLogConfidential12 3.Deployment:Staging•StagingEnvironmentSimplified,fullyOperationalConfidential13 3.Deployment:Staging•Deploymentrehearsalbeforeproductionrelease•Identifybugsin“real”environment•StagingEnvironment•Miniproductionusingvirtualizationtech•Realtraffic:•Replay•Production-levelmonitoring•ItDOEShelp!•AdServercrashedbecauseofspecialcookiesConfidential14 3.Deployment:PhaseRollout•Upgradeappsinmultiplephases•Predefinedgroups,datacenters•Pros•Cons•Lessrisky•Complexity•Naturalfallback•CompatibilitytestConfidential15 3.Deployment:Automation•DeploymentAutomation•Staging✘PhaseRollout✘Releases=?•OperationInfrastructureProject•Validationautomation•Deployment,insteadoffeatures•Quickvalidation•IntuitiveresultsConfidential16 4.ContinuousTestinginProductionSystemmonitors•Disk/CPU/IO/Network•AppAlerts/ErrorsExternalmonitors•GomezTheyarenotenough!Realusersimulationinproduction:•Dedicatedclientpages•Browsersimulatortogeneratedesignedevents•Player
AdServer
Reporting•CheckfinalreportingresultsConfidential17 4.ContinuousTestinginProductionConfidential18 Confidential19 Q&AEmail:zfdang@freewheel.tvConfidential 关注我们：t.baidu-tech.com资料下载和详细介绍：infoq.com/cn/zones/baidu-salon“畅想•交流•争鸣•聚会”是百度技术沙龙的宗旨。百度技术沙龙是由百度与InfoQ中文站定期组织的线下技术交流活动。目的是让中高端技术人员有一个相对自由的思想交流和交友沟通的的平台。主要分讲师分享和OpenSpace两个关键环节，每期只关注一个焦点话题。讲师分享和现场Q&A让大家了解百度和其他知名网站技术支持的先进实践经验，OpenSpace环节是百度技术沙龙主题的升华和展开，提供一个自由交流的平台。针对当期主题，参与者人人都可以发起话题，展开讨论。InfoQ策划·组织·实施关注我们：weibo.com/infoqchina