NASA/SP-2011-3422Version 1.0November 2011NASARisk ManagementHandbookNASA/SP-2011-3422Version 1.0NASARisk ManagementHandbookNational Aeronautics and Space AdministrationNASA HeadquartersWashington, D.C. 20546November 2011iTo request print or electronic copies or provide comments, contact theOffice of Safety and Mission Assurance.Electronic copies are also available fromNASA Center for AeroSpace Information7115 Standard DriveHanover, MD 21076-1320athttp://ntrs.nasa.goviiNASA STI Program in ProfileSince its founding, NASA has been dedicatedto the advancement of aeronautics and spacescience. The NASA scientific and technicalinformation (STI) program plays a key part inhelping NASA maintain this important role.CONFERENCE PUBLICATION. Collectedpapers from scientific and technicalconferences, symposia, seminars, or othermeetings sponsored or co-sponsoredby NASA.The NASA STI program operates under theauspices of the Agency Chief InformationOfficer. It collects, organizes, provides forarchiving, and disseminates NASAs STI.The NASA STI program provides access tothe NASA Aeronautics and Space Databaseand its public interface, the NASA TechnicalReport Server, thus providing one of thelargest collections of aeronautical and spacescience STI in the world. Results arepublished in both non-NASA channels and byNASA in the NASA STI Report Series,which includes the following report types:SPECIAL PUBLICATION. Scientific,technical, or historical information fromNASA programs, projects, and missions,often concerned with subjects havingsubstantial public interest.TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientificand technical material pertinent toNASAs mission.Specialized services also include creatingcustom thesauri, building customizeddatabases, and organizing and publishingresearch results.TECHNICAL PUBLICATION. Reports ofcompleted research or a major significantphase of research that present the results ofNASA Programs and include extensive dataor theoretical analysis. Includes compilationsof significant scientific and technical data andinformation deemed to be of continuingreference value. NASA counterpart of peerreviewed formal professional papers but hasless stringent limitations on manuscript lengthand extent of graphic presentations.For more information about the NASA STIprogram, see the following:Access the NASA STI program home pageat http://www.sti.nasa.govE-mail your question via the Internet [email protected] your question to the NASA STI HelpDesk at 443-757-5803TECHNICAL MEMORANDUM. Scientificand technical findings that are preliminary orof specialized interest, e.g., quick releasereports, working papers, and bibliographiesthat contain minimal annotation. Does notcontain extensive analysis.Phone the NASA STI Help Desk at443-757-5802Write to:NASA STI Help DeskNASA Center for AeroSpace Information7115 Standard DriveHanover, MD 21076-1320CONTRACTOR REPORT. Scientific andtechnical findings by NASA-sponsoredcontractors and grantees.iiiivACKNOWLEDGMENTSThe project manager and the authors express their gratitude to NASA Office of Safety andMission Assurance (OSMA) management (Mr. Bryan OConnor, former Chief of OSMA; Mr.Terrence Wilcutt, Chief of OSMA; Mr. Wilson Harkins, Deputy Chief of OSMA; and Mr.Thomas Whitmeyer, Director of Mission Support Division) for their support and encouragementin developing this document. The development effort leading to this document was conducted instages, and was supported by the individuals listed alphabetically below, who each broughtunique experience and insights to the development.1AUTHORS:Dr. Homayoon Dezfuli(Project Manager)NASA System Safety Technical Fellow, NASAHeadquartersDr. Allan BenjaminMr. Christopher EverettMr. Gaspare MaggioDr. Michael StamatelatosInformation Systems LaboratoriesInformation Systems LaboratoriesInformation Systems LaboratoriesDirector of Safety and Assurance Requirements Division,NASA HeadquartersIdaho National LaboratoryDr. Robert YoungbloodCONTRIBUTING AUTHORS:Dr. Sergio GuarroDr. Peter RutledgeMr. James SherrardDr. Curtis SmithDr. Rodney WilliamsThe Aerospace CorporationQuality Assurance & Risk Management ServicesInformation Systems LaboratoriesIdaho National LaboratoryInformation Systems LaboratoriesREVIEWERS:This development benefited from review comments provided on the initial draft by manyindividuals. The authors wish to specifically thank the following individuals:Dr. Robert AbelsonDr. Timothy BarthMr. John ChioriniMr. Chester EverlineMr. Louis FussellDr. Frank GroenMr. David LengyelDr. Robert MulvihillMs. Sylvia Plants1NASA Jet Propulsion LaboratoryNASA Kennedy Space CenterCenter for Systems ManagementJet Propulsion LaboratoryFutron CorporationNASA HeadquartersNASA HeadquartersQuality Assurance & Risk Management ServicesScience Applications International CorporationAffiliations are as of the time of contribution to the development effort.vMr. William PowellMr. David PyeDr. James RoseDr. Fayssal SafieDr. Nathan SiuDr. Clayton SmithMs. Sharon ThomasMs. Ellen StigbergDr. William VeselyMr. Tracy WrigleyDr. Thomas ZangNASA Marshall Space Flight CenterPerot SystemsJet Propulsion LaboratoryNASA Marshall Space Flight CenterU. S. Nuclear Regulatory CommissionApplied Physics LaboratoryNASA Johnson Space CenterNASA HeadquartersNASA HeadquartersBastion Technologies, Inc.NASA Langley Research CenterviTABLE OF CONTENTSTable of Contents ………………………………………………………………………………….. viiList of Figures …………………………………………………………………………………………xiList of Tables ……………………………………………………………………………………….. xiiiPreface ………………………………………………………………………………………………….. xv1 INTRODUCTION ……………………………………………………………………………….11.11.21.31.41.4.11.4.21.51.5.11.5.21.5.31.5.41.61.6.11.6.21.6.31.6.41.6.51.6.6Purpose ……………………………………………………………………………………………… 1Scope and Depth …………………………………………………………………………………. 1Background ……………………………………………………………………………………….. 3Applicability of Risk Management…………………………………………………………. 4When is RIDM Invoked? ……………………………………………………………………… 5When is CRM Applied? ……………………………………………………………………….. 6Overview of the RIDM Process …………………………………………………………….. 6Part 1, Identification of Alternatives ………………………………………………………. 9Part 2, Risk Analysis of Alternatives …………………………………………………….. 10Part 3, Risk-Informed Alternative Selection …………………………………………… 12Avoiding Decision Traps ……………………………………………………………………. 14Overview of the CRM Process …………………………………………………………….. 15Step 1, Identify …………………………………………………………………………………. 16Step 2, Analyze …………………………………………………………………………………. 16Step 3, Plan………………………………………………………………………………………. 19Step 4, Track ……………………………………………………………………………………. 19Step 5, Control ………………………………………………………………………………….. 20Communicate and Document ………………………………………………………………. 202 RIDM PROCESS INTERFACES ……………………………………………………….. 212.12.22.32.3.12.3.22.4Negotiating Objectives across Organizational Unit Boundaries …………………. 22Preparing a Preliminary Risk Management Plan …………………………………….. 22Coordination of RIDM and CRM …………………………………………………………. 22Initializing the CRM Risks Using the Risk Analysis of the SelectedAlternative ……………………………………………………………………………………….. 23Rebaselining of Performance Requirements …………………………………………… 24Maintaining the RIDM Process ……………………………………………………………. 273 THE RIDM PROCESS ………………………………………………………………………. 293.13.1.13.1.1.13.1.1.23.1.23.1.2.13.1.2.2Part 1 Identification of Alternatives……………………………………………………. 30Step 1 Understand Stakeholder Expectations and Derive PerformanceMeasures …………………………………………………………………………………………. 30Understand Stakeholder Expectations …………………………………………………… 30Derive Performance Measures …………………………………………………………….. 32Step 2 – Compile Feasible Alternatives …………………………………………………. 43Compiling an Initial Set of Alternatives ………………………………………………… 43Structuring Possible Alternatives (e.g., Trade Trees) ……………………………….. 43vii3.23.2.13.2.1.13.2.1.23.2.1.33.2.1.43.2.1.53.2.23.2.2.13.2.2.23.2.2.33.2.2.43.2.2.53.2.2.63.2.2.73.2.2.83.33.3.13.3.1.13.3.1.23.3.23.3.2.13.3.2.23.3.2.33.3.2.43.3.2.53.3.2.63.3.2.73.3.2.83.3.2.9Part 2 Risk Analysis of Alternatives …………………………………………………… 46Step 3 Set the Framework and Choose the Analysis Methodologies ………… 46Structuring the Analysis Process ………………………………………………………….. 48Configuration Control ………………………………………………………………………… 49Implementing Various Levels of Model Rigor in Selecting Risk AnalysisMethods…………………………………………………………………………………………… 49Implementing a Graded Approach in Quantifying Individual Scenarios ……… 53Use of Existing Analyses ……………………………………………………………………. 53Step 4 Conduct the Risk Analysis and Document the Results …………………. 53Probabilistic Modeling of Performance …………………………………………………. 54Use of Qualitative Information in RIDM ……………………………………………….. 57Risk Analysis Support of Robust Decision Making …………………………………. 59Sequential Analysis and Downselection ………………………………………………… 60Model Uncertainty and Sensitivity Studies …………………………………………….. 62Analysis Outputs ………………………………………………………………………………. 64Assessing the Credibility of the Risk Analysis Results …………………………….. 65The Technical Basis for Deliberation ……………………………………………………. 66Part 3 Risk-Informed Alternative Selection …………………………………………. 70Step 5 Develop Risk-Normalized Performance Commitments ………………… 71Establishing Risk Tolerances on the Performance Measures …………………….. 73Ordering the Performance Measures …………………………………………………….. 75Step 6 Deliberate, Select an Alternative, and Document the DecisionRationale …………………………………………………………………………………………. 78Convening a Deliberation Forum …………………………………………………………. 78Identify Contending Alternatives …………………………………………………………. 79Additional Uncertainty Considerations …………………………………………………. 80Other Considerations …………………………………………………………………………. 82Deliberation Is Iterative ……………………………………………………………………… 83Communicating the Contending Alternatives to the Decision Maker ………….. 83Alternative Selection Is Iterative ………………………………………………………….. 85Selecting a Decision Alternative ………………………………………………………….. 86Documenting the Decision Rationale ……………………………………………………. 864 THE CRM PROCESS ……………………………………………………………………….. 894.14.1.14.1.24.1.2.14.1.2.24.1.34.1.3.14.1.44.24.2.14.2.1.14.2.1.24.2.1.3Initializing the CRM Process ………………………………………………………………. 93Development of Risk Management Plan ……………………………………………….. 93Inputs to CRM ………………………………………………………………………………….. 94Inputs from the RIDM Process …………………………………………………………….. 94Inputs from Systems Engineering ………………………………………………………… 95Risk Tolerance Targets at Projected Milestones ……………………………………… 96Alternate Performance Margin Targets at Projected Milestones ………………… 97Developing Initial Risk Taxonomies …………………………………………………… 100The CRM Identify Step …………………………………………………………………….. 100The Structure of an Individual Risk…………………………………………………….. 101The Risk Statement ………………………………………………………………………….. 101Validating an Individual Risk ……………………………………………………………. 103Taxonomic Categorization of Individual Risks……………………………………… 106viii4.2.1.44.2.24.2.34.34.3.14.3.24.3.2.14.3.2.24.3.2.34.3.2.44.3.2.54.3.2.64.3.34.3.3.14.3.3.24.3.3.34.44.4.14.4.1.14.4.1.24.4.24.4.2.14.4.2.24.4.34.54.64.74.7.14.7.24.84.8.14.8.24.8.3The Narrative Description …………………………………………………………………. 112Sources of Risk Identification ……………………………………………………………. 113Risk Advocacy and Ownership ………………………………………………………….. 116Analyze Step ………………………………………………………………………………….. 118Introduction to Graded Analysis and the Use of Risk Scenario Diagrams inCRM……………………………………………………………………………………………… 120Quick Look Analyze Step …………………………………………………………………. 122Likelihood and Severity Ranking ……………………………………………………….. 126Uncertainty Ranking ………………………………………………………………………… 134Timeframe Ranking …………………………………………………………………………. 137Near-Term (Tactical) Criticality Ranking…………………………………………….. 138Long-Term (Strategic) Criticality Ranking …………………………………………… 141Relationship Between Criticality Rankings and the Risk Matrix………………. 143Graded Approach Analyze Step …………………………………………………………. 146Developing Risk Scenario Diagrams (RSDs) ……………………………………….. 146Updating the Performance Risk Models and Calculating Performance Risk . 155Determining the Risk Drivers ……………………………………………………………. 156The CRM Plan Step …………………………………………………………………………. 160Generating Risk Response Alternatives……………………………………………….. 163Generating Risk Response Options …………………………………………………….. 163Combining Risk Response Options to produce a set of Candidate RiskResponse Alternatives ………………………………………………………………………. 172Risk Analysis of Risk Response Alternatives ……………………………………….. 173Integrating the Candidate Risk Response Alternatives into the Risk Analysis…………………………………………………………………………………………………….. 173Conducting the Risk Analysis and Documenting the Results …………………… 177Deliberation and Selection of a Risk Response……………………………………… 179The CRM Track Step ……………………………………………………………………….. 183The CRM Control Step …………………………………………………………………….. 185Communicate and Document …………………………………………………………….. 187Communication within CRM …………………………………………………………….. 187Documentation within CRM ……………………………………………………………… 188Applicability of Project-Centered CRM Processes to Other Risk Domains … 188Institutional Risks ……………………………………………………………………………. 189Enterprise Risks ………………………………………………………………………………. 189Agency-Wide Strategic Risks ……………………………………………………………. 1905 REFERENCES ……………………………………………………………………………….. 193APPENDIX A: Acronyms and Abbreviations …………………………………………. 197APPENDIX B: Definitions ……………………………………………………………………. 201APPENDIX C: Content Guide for the Technical Basis for Deliberation …… 205APPENDIX D: Content Guide for the Risk-Informed Selection Report ……. 207APPENDIX E: Selected NASA Examples of RIDM Process Elements………. 209ixAPPENDIX F: Practical Aspects of the Risk Management Plan ……………… 225APPENDIX G: Hypothetical Individual Risks Used for the Planetary ScienceExample in the CRM Development ……………………………………………………….. 229xLIST OF FIGURESFigure 1. Systems Engineering Engine …………………………………………………………………………….2Figure 2. Risk Management as the Interaction of Risk-Informed Decision Making andContinuous Risk Management ………………………………………………………………………….3Figure 3. Flowdown of Performance Requirements (Illustrative) ………………………………………….5Figure 4. The RIDM Process ………………………………………………………………………………………….7Figure 5. Functional Roles and Information Flow in RIDM (Notional) ………………………………….8Figure 6. Uncertainty of Forecasted Outcomes Due to Uncertainty of Analyzed Conditions …… 10Figure 7. The CRM Process ………………………………………………………………………………………… 15Figure 8. Integration of Individual Risks to Produce Performance Risks ……………………………… 18Figure 9. Coordination of RIDM and CRM within the NASA Hierarchy (Illustrative) …………… 21Figure 10. RIDM Input to CRM Initialization ………………………………………………………………… 23Figure 11. Rebaselining of Performance Requirements ……………………………………………………. 25Figure 12. Scope of Potentially Affected Organizations Given Rebaselining ……………………….. 26Figure 13. RIDM Process Steps……………………………………………………………………………………. 29Figure 14. RIDM Process Flowchart: Part 1, Identification of Alternatives ………………………….. 31Figure 15. Notional Objectives Hierarchy ……………………………………………………………………… 34Figure 16. Fundamental vs. Means Objectives ………………………………………………………………… 37Figure 17. Types of Performance Measures ……………………………………………………………………. 39Figure 18. The Relationship between Performance Objectives and Performance Measures …….. 40Figure 19. Example Launch Vehicle Trade Tree from ESAS …………………………………………….. 44Figure 20. RIDM Process Part 2, Risk Analysis of Alternatives …………………………………………. 46Figure 21. Risk Analysis Framework (Alternative Specific) ……………………………………………… 48Figure 22. Analysis Methodology Guidance Chart ………………………………………………………….. 50Figure 23. Risk Analysis Using a Monte Carlo Sampling Procedure…………………………………… 56Figure 24. Uncertain Performance Parameters Leading to Performance Measure Histograms …. 58Figure 25. Robustness and Uncertainty………………………………………………………………………….. 60Figure 26. Downselection of Alternatives ………………………………………………………………………. 61Figure 27. Conceptualization of the Formulation of Modeling Uncertainty ………………………….. 63Figure 28. Notional Depiction of Decision Sensitivity to Input Parameters ………………………….. 64Figure 29. Analysis Level Matrix …………………………………………………………………………………. 66Figure 30. Notional Imposed Constraints Risk Matrix ……………………………………………………… 67Figure 31. Notional Band Aid Chart for Performance Measure X ………………………………………. 68Figure 32. Comparison of Uncertainty Distributions ………………………………………………………… 68Figure 33. RIDM Process Part 3, Risk-Informed Alternative Selection ……………………………….. 70Figure 34. Establishing Performance Commitments ………………………………………………………… 72Figure 35. Performance Commitments and Risk Tolerances for Three Alternatives ………………. 76Figure 36. An Example Uncertainty Consideration: The Potential for High Performance ………. 81Figure 37. Notional Performance Commitment Chart ………………………………………………………. 84Figure 38. Notional Risk List for Alternative X ………………………………………………………………. 85Figure 39. The CRM Process ………………………………………………………………………………………. 91Figure 40. CRM Process Flow Diagram ………………………………………………………………………… 92Figure 41. Decreasing Uncertainty and Risk over Time ……………………………………………………. 96Figure 42. Notional Risk Burn-Down Schedules for Several Performance Requirements ………. 98xiFigure 43. Notional Margin Burn-Down (Risk Relaxation) Schedules for Several PerformanceMargins ……………………………………………………………………………………………………. 99Figure 44. The CRM Identify Step ……………………………………………………………………………… 101Figure 45. Generating and Validating an Individual Risk………………………………………………… 104Figure 46. Example Condition/Departure Taxonomy …………………………………………………………Week FiveNASA Weekly Risk paperPaper limit 3 pages1) In page 31 it shows an excellent flowchart of the RIDM process. What is the keyarea in this process? What are the weaknesses in this process and how could theybe improved?2) On page 57 they mention the use of qualitative measures in the RIDM. What aresome of the challenges using these types of measures?3) In page 73 they mention the use of risk tolerances in performance measures. Whatare some striking comments that are made in this process?4) Page 79 they reference considering alternatives. Please describe this process andhow it may cause more confusion.