Independent Review: 2013 Department of National Defence Annual Update on Next Generation Fighter Capability Life Cycle Costs

Final Report - August 5th, 2013

Table of Contents

• List of Acronyms
• 1. Executive Summary
• 2. Background
  • 2.1 Context
  • 2.2 Objective and Scope
    • 2.2.1 Limitation of Scope
  • 2.3 Report’s Limitations
  • 2.4 Key Framework Principles
  • 2.5 Approach
• 3. Independent Review Observations
  • 3.1 NGFC LCC Planning
  • 3.2 Boundaries and Assumptions
    • 3.2.1 Cost Boundaries
    • 3.2.2 Ground Rules and Assumptions
  • 3.3 Develop Model
    • 3.3.1 Cost Breakdown Structure
    • 3.3.2 Model
    • 3.3.3 Cost Methods
  • 3.4 Data, Populate and Document Model
    • 3.4.1 Data Collection and Normalization
    • 3.4.2 Baseline Estimate
  • 3.5 Review, Analyze and Update
    • 3.5.1 Sensitivity Analysis
    • 3.5.2 Risk and Uncertainty Analysis
    • 3.5.3 Document Results
    • 3.5.4 Model Documentation
    • 3.5.5 LCC Assurance
  • 3.6 Interpret and Report Results
  • 3.7 People and Organization
    • 3.7.1 NGFC Estimator Team
    • 3.7.2 Cost Assurance Role
  • 3.8 Comparative Review
• 4. Conclusion
• 5. Appendix A | Recommendations from 2012 Independent Review
• 6. Appendix B | List of Documentation Reviewed
• 7. Appendix C | Summary of Suggested SSRB Spreadsheet Standards to Implement

List of Acronyms

• CBS Cost Breakdown Structure
• D Cost S Directorate of Costing Services
• DMS Diminishing Manufacturing Sources
• DND Department of National Defence
• DoD Department of Defence (United States)
• DRMIS Defence Resource Management Information System
• Framework The Life Cycle Cost Framework
• JPO Joint Program Office
• JSF Joint Strike Fighter
• LCC Life Cycle Cost
• Model Life Cycle Cost Model
• MOU Memoranda of Understanding
• NGFC Next Generation Fighter Capability
• NFPS National Fighter Procurement Secretariat
• PWGSC Public Works and Government Services Canada
• TBS Treasury Board Secretariat
• URF Unit Recurring Flyaway

1. Executive Summary

Raymond Chabot Grant Thornton Consulting Inc. (RCGT) has been contracted by the National Fighter Procurement Secretariat (NFPS) to conduct an independent review of the 2013 Department of National Defence (DND) Annual Update on Next Generation Fighter Capability (NGFC) life cycle costs (LCC) to verify the assumptions and cost estimates, including their alignment to the LCC Framework (“the Framework”) developed in 2012. ¹ This report presents the observations and recommendations resulting from the 2013 Independent Review.

The focus of the Independent Review was to assess whether the Framework was appropriately applied. This process included assessing whether the cost estimates and all changes to the underlying assumptions, validated in the 2012 Annual Update ², were calculated and presented in a manner consistent with the Framework.  The review did not assess the appropriateness or accuracy of source data relied upon in the LCC estimation.

The Independent Review involved assessing DND’s LCC processes, procedures and documentation against the Framework. The review process included a:

• Review of the estimates in the 2013 Annual Update;
• Review of the Model used to calculate the 2013 estimates; and
• Comparative review of the 2013 LCC estimation approach to the 2012 LCC estimation approach.

Our Independent Review of DND’s application of the Framework did not reveal any deviations from the Framework’s principles that would result in any material changes to the overall LCC estimate.

The Framework includes key principles that focus on the planning, sustainability and the continuous improvement of DND’s cost estimating processes and tools. Based on the review conducted, we have provided a set of recommendations for DND and other relevant stakeholders that are aimed at supporting a more sustainable approach to costing and ensuring the consistent application of the methodology and principles from year-to-year. The recommendations have been grouped into three (3) categories related to:

• Documentation of processes, procedures and information;
• Improvements to the Model or analytical methods; and
• Risk management enhancement.

The recommendations have been summarized in the following table:

The overall assessment of the NGFC LCC process is that DND has continued to improve and refine its processes and methods as the NGFC Project continues to evolve. There are further improvements that can be made in terms of process documentation and formalization, but these are largely reflective of the maturity level of the NGFC LCC process, recognizing that the NGFC project remains in the options analysis phase. In the interim period between the 2013 Annual Update and the 2014 Annual Update, DND should remain focused on addressing the remaining recommendations from the 2012 Independent Review (See Section 4 and Appendix A) along with the recommendations provided by the 2013 Independent Review.

2. Background

2.1 Context

On April 3rd, 2012, the Auditor General of Canada presented his 2012 Spring Report to Parliament, identifying concerns with the way key information relating to Canada’s fighter capability was being developed and presented to Canadians. The report recommended that the F-35 fighter capability cost estimate be refined to include the full life cycle cost and that the estimate be made public. The Government accepted the Auditor General's recommendation and launched a Seven-Point Plan ³ in response.

The Government, via the NFPS (formerly known as the F-35 Secretariat) within Public Works and Government Services Canada (PWGSC), is committed to ensuring that due diligence is performed throughout the implementation of the Seven-Point Plan.

As part of the Seven-Point Plan, the Treasury Board Secretariat (TBS) commissioned an independent review ⁴, which included the development of a NGFC LCC Framework ⁵. This Framework, completed in November 2012, was designed to provide DND with clear direction for the consistent estimation and reporting of NGFC LCC.

Another element of the Seven-Point Plan is that:

“The Department of National Defence, through the F-35 Secretariat, will provide annual updates to Parliament. These updates will be tabled within a maximum of 60 days from receipt of annual costing forecasts from the Joint Strike Fighter program office, beginning in 2012.”

DND’s first annual update ⁶ on NGFC LCC was completed in December 2012. An independent review of the 2012 annual update provided eight (8) recommendations for improvements to the NGFC LCC estimation approach (See Appendix A). ⁷

RCGT has been contracted by the NFPS to conduct an independent review of the 2013 Annual Update on NGFC LCC to verify the assumptions and cost estimates, including their alignment to the Framework. This report presents the observations and recommendations resulting from the 2013 Independent Review.

2.2 Objective and Scope

The focus of the independent review was to review whether the Framework was appropriately applied..

This independent review includes assessing whether the cost estimates and all changes to the underlying assumptions validated in the 2012 Annual Update are calculated and presented in a manner consistent with the NGFC LCC Framework released by the NFPS in December 2012.

2.2.1 Limitation of Scope

While RCGT assessed whether data sources were appropriately documented and validated by DND, our review did not assess the appropriateness or accuracy of source data relied upon in the LCC estimation. For example, the costing information provided by the Joint Program Office (JPO) was not subject to validation (i.e., RCGT accepted the figures as provided and did not validate the accuracy of the costing assumptions underlying the information).

DND is also currently conducting a thorough evaluation of all available options to replace the CF-18 fleet. RCGT did not consider this options analysis as part of the scope of the Independent Review.

2.3 Report’s Limitations

The purpose of this report is to inform decision-making within DND, NFPS and TBS and to support DND’s 2013 NGFC LCC Annual Update to Parliament. No party should act on the contents of this report without conducting further analysis. DND, TBS and the Government of Canada are responsible for decisions made relating to the NGFC Program. RCGT will not assume liability for the reliance on this report by any third parties. Final benefits and costs realized from implementing a plan to acquire a fleet of F-35 aircraft will be based on future events and government decisions, which may lead to material variances from the estimates included in this report.

The review consisted of specific activities agreed to by DND and the NFPS. The review assessed the alignment of the NGFC LCC estimate to the Framework, relying on information provided by DND. RCGT did not conduct an audit of the estimate and, therefore, does not provide assurance or express an opinion on cost estimates. Furthermore, the inability to review the Statement of Operational Requirement (SOR) limits our ability to conclude on whether the Model’s Cost Breakdown Structure (CBS) includes all project capability requirements.

Readers of this report should consider the document in its entirety. Selection of, or reliance on, certain elements of the report may result in misinterpretation of information provided. RCGT will not accept liability for such interpretations.

RCGT reserves the right, but will be under no obligation, to review and/or revise all findings, calculations and recommendations referred to herein, if we consider it necessary in light of further information that becomes known to us after the date of this report.

2.4 Key Framework Principles

The Framework sets out seven key principles to guide DND in achieving the estimate outcomes required by multiple decision makers and stakeholders. The review criteria used by RCGT during the independent review are aligned to the following principles:

• NGFC LCC Planning | Develop a plan to ensure the NGFC LCC model meets the needs of all prospective users and aligns with the LCC Framework.
• Boundaries and Assumptions | Well-defined and agreed-upon boundaries are established. Key ground rules and assumptions are understood and agreed. Project documentation is readily available and forms the basis for costs.
• Develop Model | A Cost Breakdown Structure (CBS) is developed representing the total Program. The model developed is in line with leading practices and supporting CBS and the range of decisions anticipated. Appropriate cost estimation methods are selected for each cost element.
• Data, Populate and Document Model | Data is collected and normalized. The baseline estimate is developed and internal validation of model and results is conducted.
• Review, Analyze and Update | Undertake sensitivity, risk and uncertainty analyses and develop risk-adjusted cost estimates. Results are established and documented. Independent cost assurance activities are undertaken and necessary adjustments are made to the NGFC LCC Model.
• Interpret and Report Results | Purpose-focused reports are developed for decision makers and stakeholders in accordance with prescribed guidelines.
• People and Organization | the NGFC estimator team is drawn from a professional costing organization, supported by standard tools, techniques and methods. The Cost Assurance role is integrated into the process with appropriate policies to ensure a non-advocacy approach.

2.5 Approach

The first step of the review was to understand the Framework, the 2012 NGFC Independent Review and 2012 NGFC Annual Update, as well as any supporting documentation initially provided by DND. The RCGT team also attended a workshop organized by the Costing Team to walk through the 2013 LCC Model.

As information was made available by DND, RCGT reviewed the finalized Model, the draft 2013 Annual Update and related supporting documentation (listed in Appendix B) to:

• Examine cost data, assumptions and analyses to assess whether they were consistent with the principles outlined in the Framework;
• Examine cost data, assumptions and analyses for general appropriateness, reasonableness and accuracy;
• Examine whether cost data were developed using the most recent and up-to-date costing information provided by the JPO and other source data providers; and
• Analyse and summarize results.

The review of the estimates considered whether:

Each individual component of the estimate was traceable back to appropriate source documentation, aligned to project capability requirements and assumptions, and an appropriate calculation method was chosen;

• Cost estimates and related assumptions are documented, communicated and consistently applied;
• The estimates were derived from project capability requirements and a detailed CBS, appropriate for the stage of the NGFC project;
• All cost elements included in the estimates are aligned to the purpose and capability requirements as identified in DND’s NGFC Project Charter, CBS and ground rules and assumptions, and are neither omitted, nor double counted;
• Underlying data has been correctly normalized/adjusted for the technical baseline cost and for inflation using appropriate guidance; and
• The normalization/adjustments and time phasing of the cost estimate are logical, accurate and consistently applied.

The review of the Model included:

• A detailed review of all cost data inputted into the model by reconciling all inputs to supporting documentation;
• A sampling of all formulae within the model to verify that they are consistently applied within the model and the identification of potential errors;
• A high-level review of the Model against spreadsheet best practices as determined by the Spreadsheet Standards Review Board (SSRB) ⁸.

In addition, a comparative review of the 2013 LCC estimation approach to the 2012 LCC estimation approach was conducted, which included:

• Reviewing ground rules and assumptions data to identify any changes in global assumptions between 2012 and 2013 Annual Updates; and
• Reviewing the variance analysis provided by DND to determine if any significant variances were present from year-to-year.

3. Independent Review Observations

The independent review of the estimates and Model included an assessment of the scope, assumptions and calculations underlying the estimates. The independent review criteria were developed in 2012 based on relevant TBS policies, other relevant Government of Canada instruments and applicable leading practices and principles identified in the Framework. To facilitate year-over-year comparability, RCGT has applied the same review criteria used in the 2012 Independent Review, with some minor revisions. Notably, the “People and Organization” Framework principle has been added as a new criterion for this 2013 Independent Review.

The key observations associated with each review criterion are summarized below:

In addition to the Framework, our review included a comparative analysis of the estimation approach used in 2013 to the approach used in the previous year. This is discussed further in Section 3.8.

The table below summarizes DND’s 2013 estimates based on the application of the Framework. Actual costs will vary from the estimates over time, and these variances may be material.

3.1 NGFC LCC Planning

Principle: Develop a plan to ensure the NGFC LCC model meets the needs of all prospective users and aligns with the LCC Framework.

Planning is an important step to ensure the Costing Team successfully achieves their tasks and work effectively toward objectives. Per the Framework, we expected the LCC Plan to outline key elements such as scope, purpose, schedule, data, costing methods and quality assurance. The Costing Team provided RCGT with an LCC plan ⁹ that included high-level background, methodology and costing estimate results. It also included the purpose of the Model, high level inputs and outputs, the CBS and costing methodology at the cost element level for each LCC phase.

The LCC Plan currently does not include a master schedule with resource requirements and milestones; however, we did observe evidence of DND work plan documents ¹⁰ which include tasks related to the NGFC LCC, as well as PWGSC documentation outlining the 2013 Annual Update schedule, responsibilities and milestones. ¹¹

Overall, the LCC Plan is a well-developed document that acts as a preliminary findings report, explaining work completed and summarizing cost estimates, including key assumptions and cost details. 

Currently, configuration of planning documentation is formally controlled through the processes and procedures of DND’s D Cost S.

3.2 Boundaries and Assumptions

Principle: Well-defined and agreed-upon boundaries are established. Key ground rules and assumptions are understood and agreed. Project documentation is readily available and forms the basis for costs.

3.2.1 Cost Boundaries

The LCC Framework states that cost boundaries should include all costs from initiation through asset disposal at the Program level. The Framework defines Program level costs as costs  “related to any group of resources and activities, and their related direct outputs, pursuing an objective or a set of objectives. A program may include various projects at various times”  ¹².

Through the review of the LCC Plan and Model, we compared the purpose of the Model and the Estimate with the expectations established in the Framework. We observed that the Model presents costs down to the Program level, from the Project Development stage through to the Disposal phase of the NGFC project. For costing purposes, July 2010 is used as the project start date, the year 2021 as the start of the Initial Operational Capability ¹³ and the year 2025 as the start of the Full Operation Capability ¹⁴. The Model assumes a lifespan of 30 years for each aircraft following its delivery ¹⁵.

Our review of the cost boundaries did not identify discrepancies in DND’s application of the Framework.

3.2.2 Ground Rules and Assumptions

Because the project is still at an early stage, the Framework states that  “it is necessary to use a series of assumptions that constrain elements of the project in a meaningful way in order to allow the development of an LCC. These assumptions form a key element of the NGFC LCC as they define the basis on which the estimates are being developed”. The Costing Team should therefore maintain a separate document recording all assumptions and maintain relevant supporting information. Assumptions and any subsequent changes should continue to be reviewed and approved by the appropriate stakeholders.

Ground rules and assumptions have been documented through the NGFC Project Charter, the LCC Plan supporting appendices, the Annual Update and separate documents dedicated to ground rules and assumptions. RCGT observed that these four (4) documents did not present the same details and versions of assumptions. An email ¹⁶ from the Costing Team explained that two assumptions had changed and were reflected in the Annual Update and LCC Plan. DND subsequently provided documentation indicating that representatives from both the PMO NGFC and Director of Air Requirements validated the ground rules and assumptions. ¹⁷ There was no quantitative impact to the estimate related to the ground rules and assumptions, as the same ground rules and assumptions are reflected in the 2012 and 2013 Annual Updates. While DND does include information on its key assumptions within the LCC plan document, the Framework ¹⁸ indicates that ground rules and assumptions be  “A separate document […] maintained by the Cost Estimator that records all of these assumptions”. In an effort to better align with the Boundaries and Assumptions Principle in the Framework, we recommend that DND should maintain a dedicated and separate ground rules and assumptions document, containing all current and approved ground rules and assumptions.

Recommendation D1: Although ground rules and assumptions are included and well-documented through the Model and LCC Plan, in an effort to better align with the Boundaries and Assumptions Principle in the Framework, DND should maintain a dedicated and separate ground rules and assumptions document, containing all current and approved ground rules and assumptions.

3.3 Develop Model

Principle: A Cost Breakdown Structure (CBS) is developed representing the total Program. The model developed is in line with leading practices and supporting CBS and the range of decisions anticipated. Appropriate cost estimation methods are selected for each cost element.

3.3.1 Cost Breakdown Structure

The Framework requires that  “the Cost Breakdown Structure (CBS) provides a logical and complete breakdown of the NGFC Program” ¹⁹. Per the Framework, we expected the CBS in the 2013 LCC Plan and Annual Update to be aligned to the elements defined in the SOR. We were not able to conduct this test, as DND was not able to provide the SOR because the document has been set aside and therefore not available. ²⁰ In the absence of adequate documentation to directly compare against the CBS, RCGT compared the 2013 CBS with last year’s Annual Update to identify any gaps or changes. We identified the addition of two cost categories to the CBS since 2012, which were added as a result of changes to the source data by the JPO. ²¹ These changes are appropriately reflected in the final 2013 cost estimates and Annual Update.

Without reviewing the SOR, RCGT cannot conclude on whether the Model’s CBS includes all project capability requirements, should a revised SOR become available in future years, the Model should be assessed against it to ensure compliance with the Framework.

Until such time as a new SOR document is available there will continue to be a limitation on the reviewer’s ability to assess compliance against this Framework principle (See Section 2.3 Report’s Limitations).

3.3.2 Model

According to the Framework,  “the Model is the tool that captures all inputs, undertakes the necessary calculations to provide outputs suitable for consideration”. ²² It should be: Accurate; Comprehensive; Replicable; Auditable; Traceable; Flexible; Credible; and Timely.

In addition, the Framework indicates that the Model should be fully documented for any reader to understand what inputs are used and how they are manipulated to create outputs. The Model itself should also be documented in a way that the user knows the purpose of each worksheet, the inputs and outputs, and the identity of the Model owner, including contact information. Finally, all modifications to the model should be documented.

RCGT’s review of the Model included a detailed review of model inputs to identify any potential copy/paste or keying errors for data provided by JPO or from other DND sources, as well as a review of model formulae for consistency and reasonability.

Overall the Model is well-organized, with specific LCC elements logically ordered from assumption data to outputs. We have noted some opportunities for improvement to the Model that do not have any impact on the LCC estimate itself. We have summarized these findings along two categories:

• Spreadsheet Improvements – potential improvements to the model to help ensure it complies with best practices; and
• Documentation Improvements – areas where documentation supporting the model can be developed or improved.

3.3.2.1 Spreadsheet Improvements

The model has been developed in line with the principles of SSRB best practices. As part of the review, we identified some additional SSRB standards and conventions that could be incorporated into future iterations of the model. These improvements do not have any impacts to the LCC Estimate. A summary of suggested SSRB standards and conventions to implement is included in Appendix C.

3.3.2.2 Documentation Improvements

Through the review of the Model, we were able to confirm that the Costing Team created an introductory worksheet with general notes and explanations on how the Model and each worksheet are laid out. Most worksheets, with a few exceptions, contain a description of the source data, purpose of the worksheet, the methodology behind the calculations and a link to an index worksheet listing and linking all worksheets in the Model. This index includes a short description for each worksheet.

• We noted that there are minor variances between the data received from JPO for the Programming Lab (Reprogramming Lab) and what is employed in the model. The rationale for these variances are based upon Joint Strike Fighter (JSF) partner discussions prior to the 2012 Annual Update that led to the discovery that certain elements of the Programming Lab estimates from the JPO sustainment cost team were no longer applicable to Canada, and alternative data from a JPO technical lead was adopted. ²³ At present, Canada is anticipated to be involved in a JSF Partner Programming Lab with both Australia and the United Kingdom, which has initiated a design review process that will lead to refinement of Programming Lab costs later in 2013. ²⁴ The total observed variance between the standard JPO estimate and the adjusted Canadian specific estimate is approximately $0.5M and is immaterial to the overall LCC Estimate. However, the Model should increase the level of details contained in its notes explaining the fact that the Model employs JPO data with slight modifications and the rationale. Improved model documentation, particularly in terms of clearly identifying source documentation and model data flow mapping, would further improve the traceability of the Model.
• In addition, there is currently no Model configuration management plan, which is a specific requirement of the Framework. ²⁵ A configuration management plan helps to ensure that the manner in which costs are estimated through the Model remains consistent year-over-year and that any change is well considered and executed. Model documentation is further discussed in section 3.5.4.

Recommendation I1: DND should continue to build on its existing improvements to the Model and supporting documentation by developing a Model configuration management plan that is aligned with best practicesand incorporating additional incremental improvements and simplifications to the Model that further improve sustainability, flexibility, traceability and auditability.

3.3.3 Cost Methods

The Framework indicates that appropriate cost estimation techniques should be selected for each cost element included within the model. The appropriateness of the cost method is largely determined by the accuracy and adequacy of available information related to the given cost element, both of which are expected to improve as the project moves from the Development to Acquisition and then Operational stages.

There are a number of different costing methods that can be applied to different costing elements within the Model; these methods are discussed briefly below ²⁶:

• Engineering cost method – Estimation of a cost element by examining products component by component. This can also be characterized as a 'Bottom up’ Approach;
• Analogous cost method – Estimation based on experience with the same or similar products or technology;
• Parametric cost method – Uses significant parameters and variables to develop cost estimating relationships; and
• Extrapolation from actuals – Uses actual contract and project performance to estimate costs at completion including estimate of actual learning against projected learning curves and earned value management approaches.

The estimation of costs related to Development, such as the PMO, are derived from a combination of parametric data, obtained from a Memoranda of Understanding (MOU), and historical actual cost information related to the PMO from DND’s financial systems. There are some other costs related to initial source data and secondary studies which are derived using a parametric costing method.

The cost method applied for the Acquisition is primarily based on engineering cost, which is obtained via a component cost breakdown for the aircraft provided in the data generated by JPO. As with the JPO data used in the 2012 Annual Update, the JPO data used in the 2013 Annual Update continues to combine historical actual costs of production for F-35 units with forecasted production efficiencies through learning and economies of scale. ²⁷

With regard to Sustainment cost, the Model has continued to rely upon the JPO estimates as its primary source data. Sustainment costs are determined through use of parametric cost methods using the key drivers of number of aircraft and estimated flying hours to generate an estimate. It is anticipated that, at some point, JPO Sustainment cost estimates may include the integration of actual observed sustainment costs for F-35 aircraft in the US, but as of the writing of this report, this information had not been captured or integrated.

Operating costs estimates within the model are generated from DND actual historical financial system information. At this point in the analysis, given that no F-35s are in operation, DND has opted to use an analogous approach to estimation using cost data related to existing CF-18 support units and bases. This approach will be the most appropriate until additional information can be obtained through either operation of an F-35 in Canada, or via other JSF partner countries.

The Operating cost of the fuel and lubricants expended is primarily driven by the buy profile and the steady state estimated flying hours. While this approach is appropriate, there is an opportunity to better align fuel cost estimates more directly to forecasted flying hours. Given that estimated annual flying hours are provided for JPO’s development of Sustainment costs and that in future year flying hour estimates/forecasts may be revised, estimated annual flying hours could be used for estimating fuel and lubricant consumption. This could assist in proactively positioning the model to adapt to any future changes in flying hour estimates should they occur. The estimated impact of revising the fuel cost methodology would be an increase in the fuel cost estimate of approximately $40M over the life of the fleet, which represents an approximate 0.1% increase in the unadjusted LCC Estimate of $40.7B. This is not deemed to be a material impact to the LCC estimate.

Finally, the cost estimation of Disposal cost for the F-35 has been based on a parametric/analogous method, as it is derived from an estimate of Disposal Costs related to the CF-18 fleet. When disposal of these aircraft is initiated DND will then be able to update the disposal estimates accordingly.Based on the review conducted, the cost methods currently employed are considered to be appropriate, but there may be an opportunity to better align fuel and lubricant costs estimates with flying hours estimates.

3.4 Data, Populate and Document Model

Principle: Data is collected and normalized. The baseline estimate is developed and internal validation of model and results is conducted.

3.4.1 Data Collection and Normalization

The Framework stipulates that DND should collect all data elements from appropriate sources and normalize / adjust these elements as appropriate within the Model. Inherent in this process is the review and analysis of key cost factors and drivers, in order to establish reasonability and assess sensitivity and risk related to those estimates.

As part of our review, we identified and traced the source of data within the Model and analyzed the method of normalization. These have been organized along each major Model cost element and discussed below.

3.4.1.1 Treatment of Indices

There are two key indices required within the Model:

1. The Inflation Rate – costs, whether they be in US$ or CAN$, must be escalated in order to account for the forecasted inflation rates, which converts a “Current Year” estimate to a “Budget Year” estimate. Inflation information is derived from two sources:
   1. The DND economic model ²⁸, which is developed within DND to estimate defence sector focused inflation figures across the department; and
   2. The US Economic model ²⁹ which is provided via the JPO.
2. Foreign Currency Exchange – the foreign exchange rate is based on the Consensus Economic Inc. report ³⁰ which is published on a monthly basis. The Model uses the May 2013 Average Annual rate.

The Model uses the most recent forecast available at the time that JPO was projected to provide its data.

Based on our assessment of the data provided, the indices used are both current, consistent with methods used for the 2012 Annual Update and appear to be reasonable.

3.4.1.2 Development

The cost data for Development incorporates a number of cost items, including PMO costs for personnel, travel and other administrative costs, along with costs related to development of initial source data and secondary studies related to the project. The bulk of these costs are estimated based on a MOU, but also include some actual historical costs related to the PMO.

All material development cost information was able to be traced to its source.

3.4.1.3 Acquisition

There are numerous cost items included in the Acquisition portion of the LCC estimate, including:

• The purchase of the aircraft itself (the Unit Recurring Flyaway (URF) cost);
• Costs related to ammunition initial spare parts inventory;
• Aircraft modifications;
• Infrastructure and sustainment set-up for the aircraft;
• Initial training;
• Reprogramming lab costs;
• PMO Acquisition costs; and
• Other acquisition costs.

Approximately 93% of the estimated Acquisition costs of $8.6B are driven by JPO provided data. This data has not been validated by the independent review team beyond ensuring that the data is appropriately entered into the model and whether the data provided reasonably estimates the expected Acquisition costs related to an F-35.

The JPO cost estimates are provided in 2012 year US$, meaning that they are adjusted for both inflation and foreign exchange in order to be translated into DND ‘Budget Year’ dollars. Infrastructure and other costs that are driven by DND sourced cost information (e.g. initial training and reprogramming lab costs) are adjusted for inflationary increases.

Acquisition cost estimates also integrate the use of the payment schedule which is described below:

• Long Lead (5%) - Due 1 year before order
• Full Funding (30%) - Due in year of order
• Payment (45%) - Due 1 year after order
• Delivery (20%) - Due 2 years after order

The data collected and input in the model is consistent with the data received from both JPO and internal DND sources.

3.4.1.4 Sustainment

Sustainment cost data is sourced directly from JPO, and therefore is adjusted for both inflationary and foreign exchange impacts. Based on the review, Sustainment data has been collected and entered from JPO correctly.

3.4.1.5 Operating

The majority of Operating cost information is derived from actual historical results of DND’s CF-18 program. Information on personnel and most consumables is extracted from the Defence Resource Management Information System (DRMIS) and populated into the model. Other consumable items, such as ammunition and lubricants are based on DRMIS historical data and then pro-rated to the anticipated acquisition schedule for the F-35 fleet. Fuel usage in particular is estimated based upon:

• The steady state assumed flying hour estimate;
• The JPO estimated fuel burn rate for the F-35;
• The weighted average standing offer price for aviation fuel at the Cold Lake and Bagotville bases; and
• The number of F-35 aircraft in operation in a given year.

As all information for Operating costs are in Canadian dollars, the only normalization required for the data is to apply annual inflation to the estimate, as forecasted in the DND Economic Model.

3.4.1.6 Disposal

Disposal costs have been updated since last year’s Annual Update, which relied upon a US Government Accountability Office report. Disposal estimates are now based on a DND-developed estimate for the disposal of the CF-18 fleet. ³¹ This information is provided in Canadian dollars. Therefore, the data is normalized for inflation using Canadian Economic Model rates.

3.4.1.7 Summary of Data Collection and Normalization

Based on the review of the Model and documentation provided there is sufficient evidence to support the tracing of data collection and all data has been normalized. As per the Framework, all cost elements that have a significant impact on the overall estimates are identified and related data is collected from a reliable source.

3.4.2 Baseline Estimate

The LCC Framework outlines key expectations regarding the completeness of a baseline estimate that is derived from up-to-date and normalized cost information (as per Section 3.4.1) and utilized appropriate cost methods to create the estimate (as per Section 3.3.3). As part of the review, we have analyzed the components of each major cost element to review whether costs are appropriately estimated and that source data is appropriately documented. The following sub-sections identify the major sub-components of each cost element.

3.4.2.1 Development Cost

Development costs currently comprise approximately $527M (1.3%) of the unadjusted LCC estimate of $40.7B. The cost estimate for Development includes costs related to:

• The PMO; and
• Contributions under the JSF Production, Sustainment and Follow-on Development MOU.

The estimated costs of Development include expenses already incurred related to Canada’s role in the JSF program. Remaining development costs include remaining development PMO and JSF MOU costs, as well as initial and secondary studies on the aircraft. 

Development costs for the most part are well-documented and traceable to their source information. However, documentation related to estimated costs related to Secondary Studies could be improved in future Model versions.

3.4.2.2 Acquisition Cost

Acquisition costs currently account for approximately $8.6B (21.4%) of the unadjusted LCC estimate of $40.7B. The main sub-components of the Acquisition cost estimate include:

• The unit recurring flyaway (URF) cost;
• Initial spare parts, training and ammunition;
• Infrastructure;
• Sustainment set-up and ancillary equipment;
• Acquisition PMO;
• Diminishing manufacturing sources and concurrency modifications; and
• Other potential acquisition costs.

Overall, costs related to acquisition have increased from $8.4B to $8.6B, an increase of approximately 3.1% year over year. This section will briefly discuss the subcomponents of Acquisition, while the risks and contingencies will be explored further in Section 3.5.2.

The following subsections provide some additional details on individual Acquisition cost elements.

URF Cost

The URF cost is essentially the estimated unit price of an F-35 as estimated by the JPO. The 2013 URF cost estimates reflects JPO data from the latest actual production costs for the aircraft. ³² The current weighted average cost of an aircraft is $88.5M USD. Once adjusted for inflation and foreign exchange forecasts, the total estimated URF cost is approximately $6.2B or 72% of the total estimated Acquisition costs of $8.6B.

The URF cost categories and means of data collection have not changed from the 2012 Annual Update and still incorporate the five (5) major components of the aircraft:

• Airframe;
• Vehicle systems;
• Mission systems;
• Propulsion system; and
• Engineering change orders.

These cost elements are comprehensive of anticipated URF cost as indicated by current JPO production cost information. ³³

Initial Spare Parts, Training and Ammunition

As part of the acquisition process, there is a need to obtain initial complements of spare parts and ammunition to cover the training and initial operational period. Likewise, pilots and maintenance personnel must receive training to learn how to operate and maintain an F-35. Overall, these costs are estimated to be approximately $411M, or 4.8% of total estimated acquisition costs of $8.6B.

It was noted that there appears to be fuel and lubricant costs included in two different areas of the Model, Acquisition (specifically in the training costs) and Operating, estimated at $83.1M and $79.7M  respectively. This is potentially double counting the fuel and lubricant costs for the 2017-18 to 2020-21 periods. This would lead to a maximum estimated reduction of $79.7M to $83.1M from the total LCC estimate, which is not considered to be a material impact.

The source data for cost estimates related to the ammunition costs for the fleet state that they include a foreign exchange and a contingency factor. ³⁴ Other areas of the Model (such as infrastructure), remove all built in contingency factors in the source data in order to ensure that the Model itself is the only source of contingency modifiers. We would recommend that the initial ammunition calculation follow this price and be based upon the unadjusted estimate figure, which could result in an potential estimated reduction of up to $9.5M from the total LCC estimate. This does not represent a material reduction in the LCC estimate.

As with the fuel and lubricant costs, we noted that costs related to ammunition appear in both the Operating and Acquisition (specifically in the training costs) sections. Based on information provided by DND, the rationale for this is that there are costs related to the initial ammunition purchase for the F-35, and subsequent costs related to Operating ammunition needs. Based on our observation of the ammunition cost calculation, there does not appear to be an adjustment for initial ammunition load out purchased with the aircraft. Therefore, there is some double-counting of ammunition costs during the acquisition period; however the full extent of this double-counting is unclear due to the uncertainty around the transition from training to operations for acquired aircraft. We recommend DND consider making an adjustment to the Operating ammunition costs during the acquisition period to reflect the initial ammunition purchases. Such an adjustment could reduce the LCC estimate by up to $51.3M. This does not represent a material reduction in the LCC estimate.Based on the items to be addressed regarding ammunition, fuel and lubricants, there would be a cumulative impact of reducing the LCC estimate by a maximum of $140.5M to $143.9M, which is not considered to be a material reduction in cost (less than 0.35% of the unadjusted LCC Estimate). It should be noted that any double counting present actually increases the conservativeness of the estimate.

Recommendation I2: While instances of double counting in fuel, lubricant and ammunition estimates are not deemed to be material (i.e. less than 1% of the LCC estimate value), and result from limits of the source data, DND should consider modifications to their cost estimating process in order to mitigate the risk of double counting in the future.

Infrastructure

The costs related to infrastructure are related to 23 construction and renovation projects needed to support an F-35 fleet. These costs are developed in a separate estimation template, whose outputs are then fed into the Model. Currently, the costs related to the infrastructure estimate are at an early stage of development and should be considered ‘rough order of magnitude’. ³⁵ Overall infrastructure costs are estimated to be approximately $244M, or about 2.8% of total estimated Acquisition costs of $8.6B.

The cost estimate for infrastructure lays out comprehensive ground rules and assumptions that have been revised for the 2013 Annual Update. These include construction and renovation costs per square metre and other items such as design fees, travel and furniture costs. The original estimate provided to the DND Costing Team also includes a provision for contingency, which is not included with the Model in order to prevent duplication of contingency amounts.

Sustainment Set-up and Ancillary Equipment

Sustainment set-up and ancillary equipment mainly consists of costs related to training devices and support equipment for the F-35, along with the labour costs needed to install and set-up the equipment. These costs are currently estimated to be approximately $1.1B, representing the largest share of the estimated $8.6B in Acquisition costs after URF cost (12.4%).

Costs related to sustainment set-up are inclusive of all the major components highlighted in JPO’s sustainment estimate. ³⁶

Acquisition PMO

Similar to Development PMO costs, Acquisition PMO costs are primarily related to personnel, overhead and travel costs for the PMO office during the Acquisition phase. These costs are forecasted to be approximately $123M or 1.4% of total estimated Acquisition costs of $8.6B. Based on our review, these costs appear to be comprehensive of expected Acquisition PMO costs.

Diminishing Manufacturing Sources and Concurrency Modifications

Diminishing Manufacturing Sources (DMS) and Concurrency Modifications are new Acquisition sub-cost categories as a result of JPO additions to cost information provided. DMS costs relate to the costs associated with losing a source of supply for parts or materials needed in the development, production or post-production of the F-35. Concurrency Modifications are costs associated with modifications to the F-35 resulting from design changes in the Acquisition phase. ³⁷ Overall, these costs represent approximately $94M, or 1.1% of total estimated Acquisition costs of $8.6B.

The inclusion of DMS and Concurrency Modification costs represents an incremental improvement to the Model and makes the Acquisition cost estimate more comprehensive. As the JSF program continues to mature, it is anticipated the DND will add such cost elements as they are developed.

Other Potential Acquisition Costs

As part of the 2012 Independent Review, information was provided on Other Potential Acquisition Costs related to the F-35, including assumptions and cost treatment of:

• The drag chute;
• Air-to-air refueling;
• Weapons; and
• NORAD costs.

The 2012 Independent Review determined that none of the above costs elements should be included in the Model as it would not be appropriate. ³⁸ Based on discussions with DND, none of the assumptions regarding Other Potential Acquisition Costs have changed since the 2012 Annual Update; therefore no modifications to the Model are needed in this regard. ³⁹

3.4.2.3 Sustainment Costs

Sustainment costs are related to costs for sustaining the F-35 over its expected useful life of 30 years. Overall, Sustainment represents approximately 28.4% of the total LCC estimate. The cost components of Sustainment are described in the table below:

The costs included in the Sustainment estimate are comprehensive of what is included in JPO sustainment cost information.

3.4.2.4 Operating Costs

The Operating phase includes costs related to consumable items for the aircraft, such as fuel and ammunition, as well as costs related to personnel salaries and benefits, base repairs and ongoing training for pilots and maintainers. Overall, the estimated Operating costs are approximately $19.9B, comprising roughly 48.8% of the unadjusted LCC estimate of $40.7B. Within the Model, Operating costs are broken into two (2) sub-categories:

• Personnel costs, which include direct and support personnel salaries and benefits costs at the Cold Lake and Bagotville bases; and
• Operating costs, which include costs of fuel, unit level consumption and base support costs.

Given that Operating cost estimates are based on actual costs related to the CF-18 program, the costs appear to be comprehensive of all costs related to the operations and maintenance of a fighter aircraft fleet.

3.4.2.5 Disposal Costs

Disposal costs in the model reflect the costs to disarm and safely dispose of aircraft that are removed from service. Currently the costs related to Disposal are estimated to be around $129M, or 0.3% of the unadjusted LCC estimate of $40.7B. The current disposal estimate includes costs for:

• Initial salvage work to remove useful components from the aircraft;
• De-militarization of the aircraft; and
• Other miscellaneous disposal costs.

The CF-18 Fleet disposal estimate recognizes that there is potential revenue that could be obtained from salvaged equipment, but given the level of uncertainty around multiple variables related to aircraft disposal, such as demand for salvage components for the CF-18 in future (and likewise the F-35), geopolitical restrictions, and so forth, these revenues have not been estimated. ⁴² Should more reliable information on aircraft disposal revenues become available in the future, we would expect that DND would adjust their Disposal estimates accordingly.

3.4.2.6 Attrition Costs

Attrition costs are related to the cost of replacing aircraft that are lost during operations. These costs are not treated as part of the full program LCC estimate, but are instead stated on a separate line to respect the assumption that the Government will need to make a decision regarding the replacement of aircraft. ⁴³ The approach to attrition costs in the model is consistent with the 2012 Annual Update, the attrition estimate continues to use the adjusted weighted average cost of an F-35 by the midpoint (9) of the estimated number of replacement aircraft required (7 to 11). Given the current level of information available to the Costing Team, the current approach remains valid.

3.5 Review, Analyze and Update

Principle: Undertake sensitivity, risk and uncertainty analyses and develop risk-adjusted cost estimates. Results are established and documented. Independent cost assurance activities are undertaken and necessary adjustments are made to the NGFC LCC Model.

3.5.1 Sensitivity Analysis

The Framework states that sensitivity analysis should be undertaken and that the results be well documented and communicated. Sensitivity analysis can be leveraged to quantitatively analyse the identified risk factors related to the full program LCC Estimate presented.

As part of the Model, DND has included sensitivity analysis calculations based upon DND and Defense Research Developmental Canada subject matter experts discussions. The Costing Team included details on the methodology behind its sensitivity analysis within its LCC Plan, which includes additional documentation on its risk analysis and contingency. 44 Currently the sensitivity analysis within the model analyzes the potential variability in the Acquisition, Sustainment and Operating estimates due to a number of risk factors, including:

• Foreign exchange;
• Inflation;
• The learning and production curve of Lockheed Martin;
• Changes in the total number of aircraft ordered by all JSF partners;
• Shifting the buy profile by one (1) year; and
• Number of annual flying hours.

Based upon our review of the model and supporting documentation, there appears to be a good depth of risk factors considered in the sensitivity analysis, however, the current Model sensitivity analysis could be expanded to show the full range of impacts related to any potential reductions in the purchase of the aircraft as the result of any increase to the baseline Acquisition estimate beyond the ceiling of $9.0B. Any reduction in the purchase of aircraft would have a corresponding reduction in Sustainment, Operating and Disposal costs. Currently, DND has not received sufficient information from JPO to estimate the impacts of a reduction in the number of aircraft purchased by Canada. In future years, we recommend that DND work with JPO to obtain the information needed in order to estimate the impact of a change in the number of aircraft purchased by Canada in order to quantify the costs/savings.

Recommendation I3: Although the sensitivity analysis conducted as part of the 2013 LCC estimate considered a wide array of risk factors, which is consistent with the Framework and deemed to be comprehensive given the data that was available to DND at the time, DND should consider adding additional sensitivity analysis scenarios in future estimates to quantify the cumulative impact of changing the number of aircraft purchased.

3.5.2 Risk and Uncertainty Analysis

Decision makers need to be informed of cost risks and uncertainties relevant to the cost estimates. The Framework sets the appropriate confidence levels range between 50% and 90%. Since the NGFC Project Charter states that the overall risk assessment for the project is “High”, we expected to observe that DND had documented and conducted a risk and uncertainty analysis as well as calculated a contingency budget to address identified risks and uncertainties. This would include stakeholder engagement and updates to the project risk log.

Through the review of the LCC Plan and supporting risk methodology documents, we were able to confirm that a standardized risk management process is in place. A risk and uncertainty analysis was completed to assess:

• The volatility of:
  • Inflation;
  • Canadian dollar vis-à-vis the US dollar; and
  • The flying hour rates.
• The fuel price;
• Potential production delays or reduced availability of aircrafts;
• Whether or not the Production Sustainment and Follow-on Development MOU will be renewed; and
• Whether or not Lockheed Martin realizes estimated production efficiency improvements.

In June 2013, a risk assessment was conducted with the participation of stakeholders’ subject matter experts drawn from across DND, including the Royal Canadian Air Force, the Project Managment Office, Chief Financial Officer Staff, and other relevant parties.

We were provided with a centralized DND risk log that includes elements related to cost risk. ⁴⁵ Risks are also listed in the LCC Plan, which is updated annually. The following sub-sections outline the contingencies that have been built into the Model by DND to address identified risks.

3.5.2.1 Acquisition Contingency

The approach to estimating the contingency on Acquisition is consistent with the approach used in the 2012 Annual Update.

An analysis of the current DND estimate of Acquisition contingency, based on the DND analysis of risks, should be $1.6B. ⁴⁶ DND's current contingency provision of $342 million is not in line with this estimate, and therefore a shortfall in the Acquisition contingency persists. The baseline Acquisition estimate has increased from $8.4B to $8.6B year over year (an increase of 3.1%) and the Acquisition contingency amount has decreased from $602M to $342M year over year (a decrease of 43.2%) in order to keep the total Acquisition estimate in line with the ceiling amount of $9.0B. ⁴⁷ The reduction in the Acquisition contingency amount used in the risk adjusted LCC estimate means that the realization of the maximum impact of a single risk factor, such as foreign exchange risk, could potentially necessitate a reduction in the number of aircraft purchased.

In light of the fact that there is no current SOR document, we cannot comment whether or not a reduction in the number of aircraft purchased would bring the fleet size below DND’s requirements.

While the majority of risk factors cannot be reasonably mitigated by DND, we understand that some efforts have been made to explore options in terms of foreign exchange risk management. We recommend that these efforts continue in order to mitigate as many Acquisition risk factors as possible.

Recommendation R1: While risk and uncertainty analysis conducted as part of the 2013 Annual Update is consistent with the Framework’s requirements, DND should continue to evaluate options to further improve the robustness of its risk mitigation strategies. Specifically, DND should continue exploring options to mitigate foreign exchange risk.

3.5.2.2 Sustainment Contingency

JPO Sustainment cost estimates are based on a parametric costing methodology meaning there is still a significant level of uncertainty and risk related to the estimates. Based on unadjusted Sustainment costs of approximately $11.6B and a contingency estimate of approximately $3.5B, the total risk adjusted Sustainment cost estimate of $15.1B. This equates to a contingency of roughly 30%, which is commensurate with a ‘rough order of magnitude’ (ROM) estimate as per DND risk analysis report. ⁴⁸ This is reflective of increased uncertainty related to the JPO provided Sustainment data. This is borne out by evidence provided indicating that the US Department of Defense (DoD) has elected not to revise its Sustainment cost estimates until a secondary review of the JPO data has been performed. ⁴⁹

We would expect that once the secondary review has been completed, DND will continue to maintain a Sustainment cost position consistent with the DoD.

3.5.2.3 Operating Contingency

Currently, Operating cost estimates do not have a contingency amount included in the full program LCC. Given that the current Operating cost estimates are based on the costs related to the CF-18 program, which operates more aircraft than are forecasted for acquisition it is possible that it might be feasible to save on Operating costs through potential reductions in personnel requirements and other related costs.

3.5.3 Document Results

Estimate results are documented in two ways: through the Model, and in the Annual Update. The Model documentation is discussed in Section 3.5.4 below. The review of the Annual Update is included in Section 3.6.

3.5.4 Model Documentation

To follow leading practices, documentation of the Model should be a continuous process undertaken at every stage of the LCC estimation. The Framework also lists two minimum criteria:

•  “Document the model such that another cost analyst unfamiliar with the program could recreate it quickly and produce the same result; and”
•  “Create an executive summary that provides sufficient explanation for a non-expert cost modeller to understand the costs and underlying assumptions.”

The Framework specifies that documentation should be routinely examined by other members of DND to confirm that it is appropriately updated and fit for task. RCGT reviewed a draft quality assurance report ⁵⁰ dated April 30, 2013. We were also provided with correspondence outlining the methodology to be undertaken regarding the internal review of the Model ⁵¹. In general, we found the quality assurance methodology description provided to be of reasonable quality.

During the familiarization phase and while reviewing the Model, RCGT observed the Model contained some documentation that would allow a reviewer with limited knowledge of the Model to gain an understanding of the inputs, outputs and mechanics behind the Model. As discussed in Section 3.3.2.2, there are potential areas of improvement with regard to model documentation, but we anticipate that these can be addressed through the continuous improvement processes applied to the Model. Enhancing Model documentation, its assumptions and formulae will not only facilitate internal quality assurance and the Independent Review, but will also facilitate the maintenance of the Model’s ‘history’ and provide a roadmap for any new members of the Costing Team.

The LCC Plan developed by DND currently serves as an executive summary document to allow non-expert cost modellers to understand the costs and underlying assumptions.

3.5.5 LCC Assurance

In addition to conducting an internal review of the Model, the Framework requires that it be independently reviewed prior to any major milestone.  “The primary purpose is to challenge the existing LCC estimate to ensure it is robust and reliable, taking into account the current life cycle of the project and knowledge of the system under investigation”  [1].

The exercise conducted by RCGT and the subsequent results presented in this report constitute an independent review as defined by the Framework. This independent review is performed on an annual basis and aligns with practices in other JSF Partner nations.

There are no recommendations related to the conduct of the Model LCC assurance.

3.6 Interpret and Report Results

Principle: Purpose-focused reports are developed for decision makers and stakeholders in accordance with prescribed guidelines.

The Framework defines the report structure requirements as follows:

“[…] an LCC analysis report structure would bring out the key issues related to the costs presented in a concise, factual and easily understood manner. It includes details pertinent to the decision at hand, including the cost confidence level, risks and uncertainties, summaries of analysis such as sensitivity, risk and affordability analysis, recommendations and conclusions. The report does not assume the reader has a detailed understanding of LCC principles and careful attention should be paid to the expression of uncertainty.”

Per the Framework, cost estimates should be prepared at a minimum of 50% confidence. It is also recommended that baseline and contingency costs be presented against a range of confidence levels that would provide reference points for the decision maker.

The 2013 Annual Update[1] prepared by DND and supported by the Model introduces adequately the subject and decision at hand. It defines and details project phases, cost categories, assumptions and estimates. Cost risks and uncertainties, as well as contingencies, are calculated and presented by range (incremental cost decrease/increase), which follows best practices. Our review indicated that all confidence levels used within the risk analysis met or exceeded the minimum Framework requirements.

Overall, the report appears complete and offers proper support to decision-makers, and we have no recommendations relating to the report structure.

3.7 People and Organization

Principle: The NGFC estimator team is drawn from a professional costing organization, supported by standard tools, techniques and methods. The Cost Assurance role is integrated into the process with appropriate policies to ensure a non-advocacy approach.

3.7.1 NGFC Estimator Team

The LCC framework identifies people and the organization as a foundational component to the development of robust and reliable costs. During our review, we expected to observe that cost modellers are drawn from a centralized cost organization, and that quality assurance activities related to the Model and estimates were undertaken within DND, employees preparing the LCC model are suitably experienced and the team is multidisciplinary.

Through documentation review, we were able to confirm that DND has organizationally endorsed and standardized LCC tools/templates tailored to the specific program, such as the costing risk management framework. DND’s D Cost S has centralized activities and efforts related to building and managing the Model for the NGFC LCC. This Costing Team is composed of financial analysts with good knowledge and experience with financial and cost accounting, including planning and budgeting to develop cost estimates.

3.7.2 Cost Assurance Role

According to the Framework and leading practices, the cost assurance team members should be independent and therefore not involved with costing estimates.  “To successfully manage these two functions (Estimation and Assurance), formal policy and organizational arrangements are established to guide LCC Estimation and Assurance activities and an NGFC LCC plan would capture the planned LCC approach to LCC Estimation and Assurance for this project” [1]. As the cost assurance role is currently a process external to DND, via the Independent Review, DND’s role in cost assurance revolves mainly around internal quality assurance activities related to the Model and estimates.

The Cost Plan[2] provided by DND mentions how cost estimates are prepared and various quality assurance activities are performed to validate the Model before the Independent Review. RCGT reviewed a quality assurance report[3] of the Model dated April 30, 2013. We were also provided with correspondence describing the methodology to be undertaken regarding the quality assurance of the Model[4]. In general, we found the documentation provided to be of reasonable quality, and would suggest that DND consider formalizing the internal review process by developing policies or organizational guidelines clearly defining quality assurance activities to ensure consistency of these procedures from year to year.

Recommendation D2: In 2013, DND effectively conducted quality assurance on the Model and estimates. However, as DND strives to assume greater responsibility for LCC assurance in future years, DND should consider formalizing quality assurance activities through guidelines or policies to ensure roles are clear and structured.

3.8 Comparative Review

As part of the 2012 Independent Review, it was recommended that DND refine and simplify the Model to better align with the some Framework principles. During the cost Model overview session provided by DND, it became apparent that the Model used to generate LCC estimates for the 2012 Annual Update had been significantly modified. In order to maintain a level of prudence in the review process, RCGT conducted additional work to identify whether changes in the Model methodology resulted in any material changes in the LCC estimate from year to year.

DND prepared a variance analysis as part of the 2013 Annual Update to identify sources of variance related to the Model redesign. The variance sources that were identified and quantified by DND were:

• Foreign exchange rate; (applicable only to Development, Acquisition and Sustainment Costs)
• Inflation;
• Source data; and
• Formula corrections (applicable only to Operating Costs).

The variance analysis was conducted through using the 2013 Model and reverting back to 2012 data for each factor outlined above by inputting 2012 data in the 2013 (new) Model. The resulting cumulative variance was compared with 2012 total estimates to calculate the Delta, which represents the variance attributed to Model changes to address 2012 Independent Review Report recommendations and other improvements.

Between the 2012 and 2013 Annual Updates, the contingency calculation methodology was also modified[1]:

This variance can largely be attributed to the Sustainment contingency budget, following an increase in the Sustainment cost contingency rate to 30% in order to reflect the uncertainty in the JPO estimate. Once JPO Sustainment figures are independently reviewed (scheduled for Fall 2013), DND may need to adjust the contingency figure accordingly.

While the review team did not verify each individual variance analysis scenario, we did verify that the 2012 and 2013 Annual Update figures used in the variance analysis were aligned with their respective documents.  Based on the summary of variances provided by DND, the 2012 and 2013 Annual Updates appear to be comparable and there have not been any material changes to the Model mechanics and methodology that would impact the LCC Estimate. DND has also documented amendments made to the Model along with impacts to individual cost elements within the Model itself and has included this information within the LCC Plan.[1]

4. Conclusion

As part of the 2012 Independent Review, eight (8) recommendations were provided for improvements to the LCC, which are listed in Appendix A of this report. DND has made progress toward addressing a majority of these recommendations. Given that DND has only had approximately six (6) months since the 2012 Annual Update, DND has made good progress with regard to the 2012 Independent Review recommendations and is well positioned to address the remaining recommendations.

The 2013 Independent Review has provided six (6) recommendations regarding areas related to documentation updates and modifications to help better align the LCC process with the Framework, incremental improvements to the Model and analytical methods and potential enhancements to the existing risk analysis/management efforts. These recommendations largely reflect forward looking opportunities for improvement.

Our Independent Review of DND’s application of the Framework did not reveal any deviations from the Framework’s principles that would result in any material changes to the overall LCC estimate.

With regard to the 2013 Independent Review, the overall assessment of the NGFC LCC process is that DND has continued to improve and refine its processes and methods as the project continues to evolve. In the interim period between the 2013 Annual Update and the 2014 Annual Update, DND should remain focused on addressing any remaining recommendations from the 2012 Independent Review, along with the recommendations provided within this report.

5. Appendix A | Recommendations from 2012 Independent Review

1. It is recommended that DND formalize and document the life cycle costing plan in accordance with Framework guidance.
2. It is recommended that DND clarify documented assumptions with respect to yearly flying rate and fleet size and review and update the key assumptions and the Life Cycle Cost Estimate on a regular basis and that agreed changes are reflected in the Life Cycle Cost Estimate in a timely manner.
3. It is recommended that DND continue to review and update the Cost Breakdown Structure and the Ground Rules and Assumptions document to help ensure that the Cost Breakdown Structure and Life Cycle Cost Estimate include all capability requirements.
4. It is recommended that DND refine and simplify the comprehensive financial model so that it better meets the Framework principles of flexibility, traceability, and ease of sensitivity analysis.
5. It is recommended that the Government of Canada investigate mechanisms to more proactively manage foreign exchange risk for the NGFC Program due to the potential significant impact of FOREX on the Estimate.
6. It is recommended that DND normalize and adjust all CF-18 Operating Costs to further refine the estimation of F-35 Operating Costs.
7. It is recommended that DND conduct further analysis, and communicate key assumptions, in regards to the effective use of the remaining aircraft life at the end of 30 years.
8. It is recommended that DND allocate an appropriate level of contingency to Acquisition cost, to reflect the remaining acquisition risks and desired level of cost certainty.

6. Appendix B | List of Documentation Reviewed

Consensus Economics Inc. (May 2013). Foreign Exchange Consensus Forecast.

Defence R&D Canada, DND. (May 2012). Forecasting National Procurement Costs for the Joint Strike Fighter (U).

Defence R&D Canada, DND. (November 2011). The Unit Recurring Flyaway Cost of a Canadian Joint Strike Fighter (U).

Department of National Defence. (2012). Economic Model 2012-2013.

Department of National Defence. (2013). Next Generation Fighter Capability Annual Update 2013, Draft WME-410294 Ver 10B.

Department of National Defence. (2013). NGFC Cost Model.

Department of National Defence and Defence Research and Development Canada. (2013). Risk Analysis and Contingency for the Next Generation Fighter Capability 2013.

Department of National Defence. (April 18, 2013). Draft Work Plan v3.

Department of National Defence. (April 23, 2012). Project Charter Next Generation Fighter Capability, Version 2.

Department of National Defence. (n.d.). CF188 Fleet Long Term Disposal Cost Estimates, RDIMS #1049050.

Department of National Defence. (December 2012). Next Generation Fighter Capability Annual Update.

Department of National Defence. (July 11, 2013). Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability - F35 Cost Model.

Department of National Defence. (July 22, 2013). NGFC Summary Risk.

Department of National Defence. (July 3, 2013). NGFC LCC Estimate: F35 Source Data, Background & Analysis Documents.

Department of National Defence. (May 30, 2013). Cost Model Changes.

Department of National Defence. (n.d.). PMO NGFC Weapons Cost - An Approach for Consideration WME RDIMS #361427-v1A.

Department of National Defence. (Spring 2012). NGFC Infrastructure Program Costing Template V.6 Ground Rules and Assumptions.

Department of National Defence. (n.d.). Work Description for a Strategic DMS Financial Analyst. HR Internal Document.

Department of National Defence. (n.d.). Work Description for a Strategic Financial Analyst. HR Internal Document.

Directorate of Strategic Finance and Costing. (April 2006). Costing Handbook, Second Edition.

Government of Canada. (December 2012). Seven-Point Plan: Status Report National Fighter Procurement Secretariat.

Joint Program Office (JPO). (April 9, 2013). Production Cost Update.

Joint Program Office. (February 15, 2013). JSF Sustainment Ground Rules and Assumptions (GR&A) - 2012 Sustainment Cost Estimate v1.1.

KPMG. (November 27, 2012). Next Generation Fighter Capability: Independent Reivew of Life Cycle Cost.

KPMG. (November 27, 2012). Next Generation Fighter Capability: Life Cycle Cost Framework.

Mather, D. (April 30, 2013). Review of the NGFC Cost Model.

Secretariat, N. F. (June 10, 2013). NGFC SAR Report.

Spreadsheet Standards Review Board. (August 11, 2010). Best Practice Spreadsheet Modelling Standards, Version 6.1.

The Australian National Audit Office. (2012). Management of Australia's Air Combat Capability - F-35A Joist Strike Fighter Acquisition. Commonwealth of Australia.

US Department of Defense. (n.d.). F-35 SAR 2012 Communication Points.

7. Appendix C | Summary of Suggested SSRB Spreadsheet Standards to Implement

• 1. KPMG - NGFC Life Cycle Cost Framework, November 27, 2012
• 2. DND - Next Generation Fighter Capability Annual Update, December 2012
• 3. Government of Canada - Seven-Point Plan: Status Report National Fighter Procurement Secretariat, December 2012
• 4. KPMG - NGFC Independent Review of Life Cycle Cost, November 27, 2012
• 5. KPMG - NGFC Life Cycle Cost Framework, November 27, 2012.
• 6. DND – Next Generation Fighter Capability Annual Update, December 2012
• 7. KPMG - NGFC Independent Review of Life Cycle Cost, November 27, 2012
• 8. Spreadsheet Standards Review Board, Best Practice Spreadsheet Modelling Standards, Version 6.1, August 11, 2010
• 9. DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013
• 10. DND - Draft Work Plan v3. Apr 18 2013
• 11. NFPS -NGFC SAR Report, June 10, 2013
• 12. KPMG – NGFC Life Cycle Cost Framework, November 27, 2012, page 9
• 13. NGFC aircrafts are operationally ready to conduct sustain air defence operations at one main operating base. (DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013, page 6)
• 14. NGFC aircraft capability can be effectively employed to the full extent of the applicable plans. (DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013, page 6)
• 15. DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013, Executive Summary
• 16. Email dated June 19, 2013 at 5:14PM – RE: Walkthrough of documents/model
• 17. Email dated July 25, 2013 at 2:20PM – Assumption confirmation
• 18. KPMG – NGFC Life Cycle Cost Framework, November 27, 2012, page 23
• 19. KPMG - NGFC Life Cycle Cost Framework, November 27, 2012, page 24.
• 20. Government of Canada - Seven-Point Plan: Status Report National Fighter Procurement Secretariat, December 2012
• 21. Email dated July 15, 2013 at 13:23PM – FW: Additional Framework-Model Questions
• 22. KPMG - NGFC Life Cycle Cost Framework, November 27, 2012, page 27
• 23. Email dated July 15, 2013 at 13:23PM – FW: Additional Framework-Model Questions
• 24. Management of Australia’s Air Combat Capability – F-35A Joint Strike Fighter Acquisition
• 25. KPMG - NGFC Life Cycle Cost Framework, November 27, 2012, page 28
• 26. KPMG - NGFC Life Cycle Cost Framework, November 27, 2012
• 27. JPO - Production Cost Update, April 9, 2013
• 28. DND – Economic Model 2012-2013.
• 29. JPO - Production Cost Update, April 9, 2013
• 30. Consensus Economics – Foreign Exchange Consensus Forecasts, May 2013.
• 31. DND - CF188 Fleet Long Term Disposal Cost Estimates, RDIMS #1049050.
• 32. JPO - Production Cost Update, April 9, 2013.
• 33. JPO - Production Cost Update, April 9, 2013.
• 34. PMO NGFC Weapons Cost – An Approach for Consideration
• 35. DND - Project 2527:  NGFC Infrastructure Program Costing Template v.7 Ground Rules and Assumptions As of June 2013
• 36. JPO - 2012 v1.1 Sustainment Estimate, February 2013
• 37. DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013.
• 38. KPMG - Independent Review of Life Cycle Cost, November 27, 2012.
• 39. Email dated July 15, 2013 at 13:23PM – FW: Additional Framework-Model Questions
• 40. KPMG - Independent Review of Life Cycle Cost, November 27, 2012.
• 41. JPO - 2012 v1.1 Sustainment Estimate, February 2013
• 42. DND - CF188 Fleet Long Term Disposal Cost Estimates, RDIMS #1049950.
• 43. DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013.
• 44. DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013.
• 45. DND - NGFC Summary Risk, July 22 2013.
• 46. DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013.
• 47. Next Generation Fighter Capability Annual Update 2013, Draft WME-410294 Ver 10B
• 48. DND - Risk Analysis and Contingency for the Next Generation Fighter Capability 2013
• 49. Dod - F-35 SAR 2012 Communication Points
• 50. DND - Review of the NGFC Cost Model by Drew Mather, April 30, 2013, DRAFT
• 51. Email dated July 24, 2013 at 10:01AM – FW: NGFC Cost Model Review
• 52. KPMG - NGFC Life Cycle Cost Framework, November 27, 2012, page 38
• 53. DND – Next Generation Fighter Capability Annual Update 2013, Draft WME-410294 Ver 10B
• 54. KPMG – NGFC LCC Framework, November 27, 2012
• 55. DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013
• 56. DND - Review of the NGFC Cost Model by Drew Mather, April 30, 2013
• 57. Email dated July 24, 2013 at 10:01AM – FW: NGFC Cost Model Review
• 58. Data obtained from:  DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013
• 59. Data obtained from:  DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013
• 60. Data obtained from:  DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013
• 61. DND - Cost Plan and Summary of Findings to Support the Annual Update to Parliament and the Next Generation Fighter Capability – F35 Cost Model v1.0, July 11, 2013
• 62. KPMG - NGFC Independent Review of Life Cycle Cost, November 27, 2012
• 63. Spreadsheet Standards Review Board, Best Practice Spreadsheet Modelling Standards, Version 6.1, August 11, 2010