Detail of outcome

This compliance and review methodology (the methodology) has been subject to public consultation and we would like to thank respondents for sharing their views. All feedback received has been carefully considered. The consultation response document summarises the key issues raised and provides the SSRO’s response to them. We have published the individual feedback forms from stakeholders, in cases where we have been permitted to do so.

The methodology is being introduced to stakeholders in January 2020 and will apply from the start of the 2020/21 financial year. This is intended to allow stakeholders to become familiar with the changes before they come into effect and to avoid different versions of the methodology applying in a single year.

---

Original consultation

Consultation description

The SSRO has been operating its current compliance and review methodology since January 2017. In that time, it has gained a greater understanding of how the regulatory framework is being applied and has continued to develop its approach to implementing the methodology.

The SSRO is publishing a consultation paper covering the planned update to its compliance and review methodology. The methodology has been updated to acknowledge that compliance monitoring may inform the SSRO’s understanding of how the regulatory framework is being applied, as well as setting out the potential for compliance monitoring to impact more widely on the operation of the regime, specifically by:

• aiming to contribute to achieving good quality data from contractors;
• working to develop a shared understanding with stakeholders about identified issues and what appropriate action could be taken to address these issues; and
• informing changes to guidance issued by the SSRO, changes to the Defence Contracts Analysis and Reporting System or recommendations for legislative change.

We believe our proposed new compliance methodology makes best use of current resources and ensures there is no duplication with the work of the MOD or unnecessary burden on industry.

This is a public consultation, which is open to anyone with an interest in the SSRO’s two statutory aims of obtaining good value for taxpayers’ money and a fair and reasonable return for industry. We also welcome comments from people or organisations with a particular interest in defence (non-competitive) procurement.

Completed response forms should be sent:

• Via email to consultations@ssro.gov.uk (preferred); or
• by post to SSRO, Finlaison House, 15-17 Furnival Street, London, EC4A 1AB.

Responses to the consultation should be received by 6 December 2019. Responses received after this date will not be taken into account in finalising the methodology but may inform subsequent consideration of any future methodology updates.

The SSRO also welcomes the opportunity to meet with stakeholders to discuss the proposals during the consultation period. If you wish to arrange such a meeting, please contact us at the earliest opportunity via consultations@ssro.gov.uk.

1. Background

1.1 The SSRO launched a public consultation on a revised compliance and review methodology (‘the methodology’) in October 2019. This followed consideration of responses to a working paper that was issued to key stakeholders in relation to this topic in July 2019.

1.2 The consultation document was circulated widely and published on the SSRO’s website. This document summarises the key comments raised by respondents and the SSRO’s response to them.

1.3 We would like to thank respondents for sharing their views on the consultation. We have considered all submissions in drafting this consultation response document. This document summarises the main points raised by respondents and, where appropriate, the SSRO’s response.

1.4 Alongside this consultation response document, we have published the updated methodology. The methodology will take effect from 1 April 2020. This allows reasonable time for stakeholders to become familiar with the methodology before it takes effect.

2. The consultation process

Process

2.1 The SSRO sought feedback from stakeholders on its updated methodology through a public consultation. The consultation commenced on 14 October 2019 and closed on 6 December 2019, a period of eight weeks. Prior to consultation, the SSRO had issued a working paper on the topic during July 2019 and held a workshop with key stakeholders where we obtained feedback on the changes that were being considered.

2.2 The consultation set out the proposed updated methodology, alongside the SSRO’s reasons for making the update. It focused on the SSRO’s review obligations under sections 36(2) and 39(1) of the Defence Reform Act 2014 (the Act) and also on how, having gathered intelligence on the operation of the regime, we have continued to develop our approach to implementing the methodology published in 2017.

2.3 Consultees were asked to respond to a total of nine consultation questions, which are considered in detail in the sections below. Any changes made following the responses are also detailed. We have also taken the opportunity to make minor amendments to clarify some aspects of the methodology.

2.4 The responses gave the SSRO an understanding of views on the proposed methodology and suggested alternative wording to some paragraphs, as well as alternative approaches for the SSRO to consider. We have grouped the comments received into key issues by each question and responded to them in this document.

Breakdown of responses

2.5 We are grateful to all those who responded to the consultation. We received a total of seven responses to the consultation as set out in Table 1:

Table 1: Number of consultation responses

	Government	Industry	Trade Association
Number of responses	1	5	1

2.6 Four of the seven responses are published on our website alongside this response document. Three others were either marked confidential or did not provide affirmation that the response could be published. Respondents, other than the MOD and the ADS Group (‘ADS’), are not identified in the document but are referred to as ‘respondents’.

3. Key issues raised by respondents

3.1 The key comments raised in the responses are detailed below by question, alongside the SSRO’s response.

.2 The demonstration of how the SSRO will exercise its s36(2) function and how its s39(1) function may be informed through its compliance work was broadly considered to be appropriate and was understood by most respondents. Three comments were raised in relation to this:

• a respondent considered that by undertaking compliance work in this way, the SSRO may threaten its independence in the case of opinions or determinations, particularly if it was the SSRO who raised an issue that was later subject to referral, in the first place;
• ADS considered that while the methodology demonstrated how the SSRO fulfilled its obligations under s36(2) of the Act, it was more difficult to see how this work may link to its s39(1) obligations; and
• the MOD considered that the proposed performance indicators 2a) and 2b) should not refer to whether reports were in accordance with statutory guidance. This was on the basis that statutory guidance is not included in either of section 36(2) or 39(1).

The SSRO’s response

3.3 The legislation contains four clear roles in relation to reporting requirements, which combine to support a database of information that can be used for the purposes of achieving value for money and fair and reasonable prices:

Table 2: Roles in reporting compliance

Role	Responsibility
Submission	Contractors are required to submit the reports.
Review	The SSRO keeps under review whether reporting requirements are being met and keeps under review the provision of the regulatory framework.
Enforcement	The MOD is responsible for enforcing enforcement obligations.
Referral	The SSRO gives opinions and makes determinations if requested, including on penalty notices issued by the MOD.

3.4 The SSRO accepts that it should avoid exercising its review functions in a way that could prejudice its referral functions. It seeks to avoid any suggestion of predetermination and, with this in mind, has clarified in paragraph 2.3 of the methodology the basis on which it will measure its performance indicators 2a) and 2b), in essence relying on information provided by contractors and not arriving at an independent conclusion.

3.5 The SSRO’s view is that its approach to informing its s39(1) function through the methodology follows a similar process to that outlined above. An issue may be raised on a submission to allow the SSRO to better understand how the provisions of the legislative framework have been applied in practice, as set out in paragraph 3.8 of the methodology. Our view is that it would be inefficient and ineffective to apply the methodology in a silo fashion to exclude any intelligence gleaned from reviewing reports which may inform our s39(1) work.

3.6 The SSRO considers that our statutory guidance forms part of the reporting obligations as contractors must have regard to the guidance issued by the SSRO when preparing reports. As such, it is our view that it is appropriate for the two indicators to refer to the statutory guidance.

.7 All respondents noted that the approach was generally clear. There were two comments raised in relation to this:

• ADS stated that the effectiveness of the approach would depend on the ability of the SSRO and the contractor to build the trust to allow for collaborative working arrangement. ADS’s view is that to do this, the SSRO needs to be able to be flexible and potentially tailor its approach for different contractors; and
• the MOD stated that the SSRO’s work should not be an audit of contractor submissions as it is not the responsibility of the SSRO.

The SSRO’s response

3.8 The SSRO welcomes the input from ADS and accepts that trust is essential. One of our corporate objectives is to maintain effective and comprehensive engagement with stakeholders. Our aim is to ensure that the engagement we have with contractors and the support that we provide aligned to the methodology is both positive and constructive. There is a balance to be struck between automation to achieve an appropriately risk-based approach that can be applied across a variety of contractors, and the flexibility to deal with each contractor in a tailored way. We state in the methodology that our support can be targeted towards identified difficulties, for example through on-boarding sessions, helpdesk assistance and training on reporting and the use of DefCARS and in this way we may be able to tailor our approach, and our support, for different contractors.

3.9 We agree that care needs to be taken to ensure that the work under our methodology is not viewed as an audit of the submission, particularly as this is not one of our functions and as noted in the methodology our work is not an assurance exercise. We have updated paragraph 1.6 of the methodology to emphasize these limitations and note that our compliance review function does not involve providing assurance that individual contracts have been priced in accordance with statutory requirements, nor is it an audit of individual submissions.

3.10 All respondents noted that the review process was generally clear. There were three comments raised in relation to the review process:

• a respondent considered that the SSRO needed to ensure that it adhered to these timescales when reviewing reports;
• ADS considered that significant input from suppliers (referring to paragraph 5.2 of the methodology consultation document) would only be required if reports were substantially non-compliant when submitted; and
• the MOD’s view was that there needed to be a clear differentiation between reporting and pricing matters, and that the review process should not become an exercise in querying contract prices or become an audit of submissions. The MOD also considered that a diagram to outline the process would be helpful.

The SSRO’s response

3.11 The SSRO has implemented a process which actively monitors when submissions were made and when the initial 15 working days have passed. This should avoid issues being raised in advance of expected timescales. It should be noted that these timescales only apply to initial submissions and not to any correction reports that have been submitted which may give rise to additional queries. While the SSRO endeavours to review supplier reports on a timely basis, these are not currently subject to the 15 working day timescale, although we may endeavour to do so in the future. We have updated paragraph 3.5 of the methodology to make this clearer.

3.12 The SSRO accepts the aspiration from ADS that significant input from contractors should only be required if a report is substantially non-compliant. We have sought to promote a high degree of compliance with reporting requirements which, if achieved, will reduce the need for the SSRO to raise issues with the MOD or contractors. Issues are also now raised and responded to in DefCARS, which should help to streamline the level of input required from industry.

3.13 As stated in response to the comments raised for question 2, we have updated paragraph 1.6 of the methodology with respect to the review process not being an audit. We agree with the MOD that the line between reporting and pricing issues may be ambiguous, but our view is that it is important that issues are raised with contractors in the first instance to allow the SSRO to understand how the provisions of the legislative framework have been applied in practice.

3.14 We have clarified an aspect of our manual review process, making clear that it includes a manual ‘sense-check’ of the submission. We have updated bullet two of paragraph 3.3 of the methodology to state that our process considers issues that may have passed the validation check and have stated in paragraph 3.4 that our manual review will involve consideration of whether changes need to be made to its set of automatic validation checks.

3.15 We agree that a diagram may help to understand the review process and have added this as Figure 1, below paragraph 5.3 of the methodology.

3.16 Six respondents noted that the methodology was clear in how findings from compliance reviews would inform the SSRO’s other work. The following additional comments were raised:

• one respondent’s view was that a pragmatic approach should be adopted in cases where there may be some inflexibility in the reporting required in line with the Regulations; and
• ADS noted that there was complexity involved in meeting reporting requirements and that the SSRO should recognise this when considering findings from its compliance work. ADS suggested that simplifying and streamlining reporting should be considered by the SSRO as part of its work to review the legislation.

3.17 The MOD, however, considered that the process for how findings from compliance reviews may inform the SSRO’s other work was not clear with the approach appearing to be more of ‘tick-box’ exercise.

The SSRO’s response

3.18 The SSRO seeks to understand, as a matter of course, why reporting or pricing issues arise. In cases where the regulations do not allow flexibility in reporting or where the complexity of reporting becomes evident, these are the types of issues that need to be considered when making recommendations for legislative change or be considered as part of our work on reviewing reporting requirements.

3.19 Our approach is not intended to be a ‘tick box’ exercise, and findings from our work to date have informed development of our statutory reporting and profit rate guidance documents, as well as our recommendations for legislative change. To make the linkage clearer, we have updated paragraph 3.7 to state that where there is an issue that the SSRO needs to address, it will consider and log the issue and prioritise action in accordance with its prioritisation principles, as set out in section 7 of the methodology.

3.20 There were a wide range of responses to this question. A respondent and the MOD agreed that additional compliance activities could be undertaken without setting out the detailed process in the methodology.

3.21 Three respondents, however, stated that the section on additional compliance activities could be deleted. ADS, supported by two other respondents, considered that this was a natural progression in the review process and that it was expected that some aspects of the information provided would require further investigation or clarification to understand what was being reported. As such, ADS believe Section 4 of the methodology could be deleted without impeding the transparency of the review process.

3.22 Another respondent, however, considered that if the SSRO was to carry out such activities that it would be useful for it to inform the relevant stakeholders that a report had been selected for review, prior to the detailed review taking place.

3.23 Finally, two other respondents stated that setting out some of the detail behind the additional activities would be beneficial in order to ensure consistency, as at the moment the proposal appeared to be open ended for inclusion in the methodology and should be evidence based.

The SSRO’s response

3.24 The SSRO accepts that these activities represent normal incremental improvements and quality assurance activities that would be expected with this type of process, but we are committed to transparency and believe the methodology should reflect the various ways in which the SSRO discharges its functions. This will allow for incremental developments to take place in future, without having to amend the methodology.

3.25 The SSRO’s targeted reviews have, to date, focused on the work of the MOD delivery team and their review of a contractor’s submission, rather than seeking to involve the contractor any more than would be required under the standard review process. However, in any instance where additional compliance activities impact the contractor directly, we would seek to involve them at the earliest opportunity. We have updated paragraph 4.3 of the methodology to state that the SSRO will make contact with relevant stakeholders at the earliest opportunity to enable reasonable timescales for review and feedback.

3.26 We agree that additional reviews should be evidence based and have updated paragraph 4.1 of the methodology to state that before undertaking any such work, the SSRO would consider the evidence available from the reviews of individual reports.

3.27 All respondents agreed that the MOD should undertake a review in the first instance. There were three comments raised in relation to this:

• a respondent considered that the 15-day window for the MOD to conduct a review may not be an achievable time frame and if it was not achieved, the SSRO should not step in to perform what it believed to be the MOD’s job;
• ADS considered that for the process to be successful, an undertaking was required from the MOD to complete its review in a timely manner. ADS also noted that its view was that it was the MOD’s role to review the content and accuracy of the reports and the SSRO’s role to manage the reporting regime; and
• the MOD stated that it should be made clearer that the MOD should also be picking up on reporting issues as part of its process of reviewing reports.

The SSRO’s response

3.28 The SSRO accepts the point made by a respondent that it should not seek to perform the MOD’s job. The role of the MOD in reviewing reports has been emphasised in the methodology and the SSRO cannot substitute. The SSRO nevertheless has its review functions to discharge and any review carried out by the SSRO will be in discharge of those functions, as outlined in this methodology.

3.29 As reported in its commercial toolkit, the MOD seeks to adopt a consistent policy with respect to the collection and review of data on qualifying contracts. Chapter 5 of the toolkit makes clear that the MOD must review QDC reports submitted on DefCARS within 15 working days of their submission.

3.30 The SSRO agrees that the MOD should pick reporting issues as part of its use of the reports to manage the contracts and we have updated paragraph 5.2 of the methodology to state the MOD may identify reporting issues when reviewing reports for contract management purposes.

3.31 There was a range of responses to this question. The MOD and two other respondents agreed that the general approach of the SSRO notifying the MOD of issues was reasonable. There were two comments raised in relation to this:

• one respondent considered that only notifying the MOD of a non-submission of the ‘strategic reports’, may not be the most effective course of action as it would be the contractor who would be better informed as to the status of the report; and
• another respondent considered that the SSRO needed to remain independent in this respect, particularly considering its role in issuing opinions and determinations.

3.32 One respondent did not agree that the SSRO’s approach as drafted in the methodology was appropriate. It considered that the SSRO appeared to be undertaking a detailed operational compliance role, on behalf of the MOD. This respondent’s view was that the SSRO’s role should be more independent, identifying broader compliance themes, rather than individual issues for review.

3.33 ADS, and two other respondents supporting the ADS submission, considered that the tone of this section suggested that the SSRO would be extending its role into enforcement, something which should be left to the MOD. These respondents considered that the SSRO should not recommend to MOD the action it should take. There were two additional comments raised in relation to this:

• a respondent queried what the definition of a compliance issue was; and
• linked to this, in response to a previous question, a respondent noted that it would welcome a review of the measurement of non-compliance to confirm that it was not solely based on the number of questions raised on submissions.

The SSRO’s response

3.34 The responsibility to submit the strategic reports (the Strategic Industry Capacity Report ‘SICR’ and the Small or Medium Enterprise Report ‘SMER’) falls to the ‘designated person’, who may be the contractor or the Ultimate Parent Undertaking if the contractor is part of a group of companies. The SSRO agrees there may be instances where the contractor should be contacted if a strategic report has not been received as it may be able to provide a response on behalf of the designated person. We have updated paragraph 6.3 to state that where the delay relates to the submission of the strategic reports^{[footnote 1]}, the SSRO will seek to contact the designated person due to make the submission if known and will then inform the MOD.

3.35 As noted in our response in relation to comments made against question 1 of the consultation, the SSRO’s role is not to become an active participant in enforcement. Paragraph 6.1 of the methodology notes that the SSRO’s focus is on encouraging timely and good quality submission. The SSRO may bring matters to the attention of the MOD to gain a better understanding of the issues arising, but it is for the MOD to decide if any enforcement action is appropriate. We accept that care must be taken when raising issues to avoid expressing concluded views. As such, paragraphs 6.4 and 6.5 have been updated to make clear that it is the MOD who will consider whether any follow up action is required for any issues that may be raised by the SSRO.

3.36 We have reviewed section 6 of the methodology for tone and have updated the following paragraphs to recast them as follows:

• paragraph 6.4 “…The SSRO will refer unresolved reporting and pricing issues to the MOD, to allow it to consider whether any follow up action is required”; and
• paragraph 6.5 “…The SSRO will further note whether it considers that the issue forwarded to the MOD is of a high, medium or low priority to allow the MOD to consider the matter further. The priority assigned to an issue will be based on its potential impact on operation of the regulatory framework. This involves an assessment of relative priority. The SSRO takes seriously any possible non-compliance with the requirements of the regulatory framework and would likely mark such issues as ‘high priority’.

3.37 The SSRO’s review of submissions considers the timeliness of each submission, whether it contains the information prescribed in the Regulations, and whether it is consistent with statutory guidance issued by the SSRO. An issue may be raised where a submission appears to be incomplete, inconsistent, erroneous or lacking in detail, having regard to the reporting requirements. In raising issues, the SSRO also tries to maintain a standardised set of data and seeks to apply categories to issues as appropriate.

3.38 We agree there is merit in providing greater clarity regarding when the SSRO will consider a reporting requirement has not been met for the purposes of monitoring its performance indicators 2a) and 2b). As noted above, responses may inform either of the SSRO’s 36(2) or 39(1) responsibilities, however only an issue in relation to our 36(2) responsibilities will impact the quality rating of a submission. We have updated paragraph 2.3 of the methodology to state “…A reporting obligation will be identified as not being met for the purposes of indicators 2a) and 2b) if the SSRO or the MOD has raised an issue on a submission which has resulted in the contractor:

• submitting a correction report to rectify an error;
• providing additional information required by the legislation; or
• failing to respond to the issue raised.

3.39 Paragraph 3.11 has also been updated to state that for the purposes of applying the indicators, the SSRO will rely on the contractor’s response rather than forming its own conclusion about compliance. Any issues categorised as pricing issues will not impact these indicators nor will the SSRO form any concluded views about those matters. We will count the submission as a ‘pass or fail’ of an entire submission once made, regardless of the number of errors that may be apparent in the initial submission. We will report these indicators as a twelve-month rolling average proportion of report submissions.

3.40 All respondents noted that the elements of support or engagement reflecting in the methodology were appropriate. There was one additional comment in relation to this:

• a respondent considered that the work undertaken with the help of the SSRO’s Reporting and IT Sub-Group should be reflected within the methodology as feedback was often provided at these meetings from which improvements were made.

The SSRO’s response

3.41 The SSRO agrees that the SSRO’s Reporting and IT Sub-Group of its Operational Working Group provides a forum for stakeholder feedback and is an arena to provide support to stakeholders. We have updated paragraph 7.3 to state that we will engage with stakeholders, for example through the Reporting and IT Sub-Group, to help us identify, prioritise and take forward any required action.

3.42 Following publication of the 2019 Annual Compliance Report, the SSRO wrote to 14 companies who were included anonymously in the report under their Global Ultimate Owner to show their relative position on submission timeliness and quality. Respondents found these individual compliance letters useful and noted also that the Annual Compliance Report itself was useful.

3.43 One respondent also suggested that individual feedback on supplier report issues would be useful. One other respondent stated that it would welcome regular reports for those companies who have multiple DefCARS accounts.

The SSRO’s response

3.44 The SSRO states in the methodology that it will provide regular feedback to the MOD, identifying where matters have had an impact either on data quality or the operation of the regulatory framework as well as to contractors with multiple contracts on compliance themes identified from their report submissions.

3.45 We are grateful for the suggestions put forward by respondents on what the key areas of feedback should be. We have reordered the paragraphs in this section and have updated paragraph 8.1 to state that the SSRO will provide information about reporting issues at a contractor or designated person level, in addition to the information provided to the contractor’s administrator or the person submitting the report, if it is practicable to provide the information and it would assist with compliance.

3.46 The analysis of compliance issues is an area that the SSRO will keep under review as it considers future developments to DefCARS.

Other comments made by respondents

3.47 Respondents made two other specific comments linked to the SSRO’s compliance work:

• a respondent stated that issues are often reported against the contractor, but that the reporting would benefit from ownership attribution as not all actions were for the contractors to resolve; and
• the MOD stated that the information gathered through the compliance process should be used to identify and correct issues with DefCARS and the Statutory Guidance in a more timely fashion.

3.48 The SSRO thanks both respondents for this feedback. We will consider what action we can take to make it clearer within DefCARS to whom issues are attributed and whether the processes in place to update DefCARS and the Statutory Guidance can accommodate changes within a shorter timescale.

3.49 There were other comments made by respondents which do not directly relate to the methodology. These comments related to reporting functionality within DefCARS and a specific reporting issue in relation to provisionally priced contracts. The comments have been logged separately for consideration.

4. Next steps

4.1 The SSRO has commenced implementation of the updated compliance and review methodology. The published 2020 methodology is available on the SSRO website. The SSRO has produced internal guidance to ensure that the methodology application is objective and consistent.

4.2 Data produced from the application of the methodology will be collated ahead of the publication of the next applicable Compliance Report and will be used to analyse compliance trends.

4.3 The SSRO will therefore publish its next Annual Compliance Report in the summer of 2020 after completion of the 2019/20 financial year, using the 2017 methodology. This methodology will be implemented from 1 April 2020.

1. SICR and SME submissions ↩

Compliance and review methodology

1. Background

1.1 The SSRO is proposing to update its compliance and review methodology (‘the methodology’). We are consulting publicly on a revised version of the methodology and we welcome responses to the specific questions set out below, as well as any general comments on the methodology.

1.2 We issued a working paper to key stakeholders in relation to this topic on 1 July 2019. We received written responses to the working paper from five organisations, representing the views of three contractors, the MOD and a trade organisation. The working paper was discussed at a workshop held at the SSRO’s offices on 10 July 2019, attended by representatives from the MOD, trade and contractor organisations.

1.3 We would like to thank respondents for sharing their views on the working paper. We have considered all submissions in drafting this consultation document and we have provided a summary of the main points raised in response to the working paper, and how we have sought to address these.

1.4 This consultation document was issued on 14 October 2019. Any representations should be sent to consultations@ssro.gov.uk by Friday 6 December 2019, using the consultation response form as set out below. Members of the SSRO’s Reporting and IT Sub-Group of the Operational Working Group will have the opportunity to discuss this consultation at the SSRO’s offices on 5 November 2019.

1.5 Any comments will inform our consideration of the need for changes to the updated methodology as drafted. The SSRO aims to allow reasonable time between publishing any changes and when they come into effect. The current timetable for publication of the revised methodology is 31 January 2020, with implementation from 1 April 2020. However, as part of this consultation, we invite representations on the timing of publication and implementation.

2. Key changes to the methodology

2.1 The SSRO has been operating its current compliance and review methodology since January 2017. In that time, it has gained a greater understanding of how the regulatory framework is being applied and has continued to develop its approach to implementing the methodology.

2.2 The methodology has been updated to acknowledge that compliance monitoring may inform the SSRO’s understanding of how the regulatory framework is being applied, as well as setting out the potential for compliance monitoring to impact more widely on the operation of the regime, specifically by:

• aiming to contribute to achieving good quality data from contractors;
• working to develop a shared understanding with stakeholders about identified issues and what appropriate action could be taken to address these issues; and
• informing changes to guidance issued by the SSRO, changes to the Defence Contracts Analysis and Reporting System or recommendations for legislative change.

2.3 Other updates include the reference to, and use of, the key developments that have taken place since January 2017. In particular, these relate to:

• a more risk-based approach with the consideration, primarily, of automatic validation warnings from the system that remain unresolved;
• the ability to raise issues within the system, and importantly to seek direct feedback from contractors or the MOD on issues arising;
• a continuing programme of engagement with the MOD on compliance, which has led to a greater reliance being placed on the verification of report submissions by the MOD against contract information, and which will inform our understanding of the application of the regime and of the quality of data within the system; and
• supplementing our routine work, which is based on the validation warnings in the system, by undertaking a deeper examination of how reporting requirements are being met in selected reports, either through ‘targeted’ or ‘thematic’ reviews, which will also provide feedback on how well the MOD reviews are working in practice.

2.4 Through these changes and the proposed updates to the methodology we plan to increasingly target important compliance issues whilst maintaining a clear focus on improvements that are required to data quality.

2.5 There remains a continued focus within the updated methodology on notifying the MOD of unresolved issues identified through the compliance monitoring process, where issues are not resolved or responded to by the contractor, or where the response appears inconsistent with the statutory guidance or legislation. It will only be through clear feedback loops that the SSRO will be able to understand whether reporting requirements have been met, or whether other aspects of the regulatory framework are being met or not.

2.6 The current methodology sets two key performance indicators, which inform the overall approach to compliance and the contents of the annual compliance report. The indicators measure the timeliness and quality of report submission and their use has provided an important basis for the development of the Annual Compliance Report. These indicators have provided an important basis for the development of the Annual Compliance Report, but it has proved difficult for contractors to meet reporting requirements first time with the result that the second indicator provides limited information about the extent of compliance. We have retained these, with minor amendments and added an additional indicator to provide insight into how contractors perform after the first report submission:

• 1a) “All required reports have been submitted within the relevant deadlines”;
• 2a) “Reporting obligations have been met first time for all reports submitted, in accordance with the Regulations and relevant statutory guidance”; and
• 2b) “Reporting obligations have been met either first time or in subsequent submissions for all reports submitted, in accordance with the Regulations and relevant statutory guidance”.

2.7 The methodology also notes that the SSRO’s approach to stakeholder engagement is important to support timely, good quality submissions, particularly as early engagement will assist the SSRO to identify the difficulties that contractors may have understanding, or complying with, reporting requirements.

3. Introduction

3.1 The Defence Reform Act 2014 (the Act) places two review obligations on the SSRO. Firstly, the SSRO is required, in accordance with section 36(2) of the Act, to keep under review the extent to which persons subject to reporting requirements are complying with them. Secondly, under section 39(1) of the Act, the SSRO is required to keep under review the provision of the regulatory framework established by Part 2 of the Act and the Single Source Contract Regulations 2014 (the Regulations). Reviewing and seeking to understand reports that are submitted in compliance with the legislation and the SSRO’s statutory guidance provides the SSRO with relevant information to discharge these functions.

3.2 We seek to use the methodology to demonstrate how the SSRO exercises its function under section 36(2), but also how our function to keep under review the provision of the regulatory framework established by the Act and the Regulations may be informed by information obtained from compliance monitoring.

3.3 The methodology emphasises that the SSRO does not have a general role to provide assurance that individual contracts have been priced in accordance with statutory requirements, nor does it seek to use its compliance monitoring function for this purpose.

3.4 Our working paper did not specifically ask for stakeholder views on the linkage between the SSRO’s legislative obligations and the objectives that we are seeking to achieve through this methodology. Although no general comments were raised in relation to this by respondents, we are interested in understanding stakeholder views on the exercise of these functions.

4. The SSRO’s general approach

4.1 The SSRO proposes that the methodology should be updated to reflect actual practice and provide greater transparency regarding its approach. Some respondents to the working paper considered that the SSRO should detail the review process, while others felt that it should first be demonstrated that greater transparency as to how the SSRO logs issues and prioritises action will deliver benefits.

4.2 The SSRO is committed to transparency as one of its core values. We consider that working in partnership is more likely to lead to effective understanding of the extent to which reporting requirements are being met and result in data being reported that is used in support of the regulatory framework. Transparency will help all stakeholders to understand how the review process works and facilitate its delivery. We have drafted the methodology with distinct sections, detailing ‘the SSRO’s general approach’, ‘the review of report submissions’ and later ‘raising matters with the MOD’.

4.3 The first of these sets out the general approach that the SSRO will follow in reviewing submissions made by contractors, along with a commitment to rely on system automation as much as possible. It links the methodology to the vision set out in our data strategy, specifically that methodology aims to contribute to achieving good quality data from contractors that can be utilised in procurement decisions, contract management and in the development of the regulatory framework. It also sets out the three key performance indicators that the SSRO will apply when measuring compliance against statutory reporting requirements.

5. Review of report submissions

5.1 In response to the working paper two respondents stated that the SSRO should detail the review process, this section provides detail on the process that the SSRO will undertake when considering each submission. It notes that the SSRO will review all reports submitted and raise issues with contractors where appropriate. It also highlights the reliance we will place on the validation warnings within the system as well as the importance of the MOD’s assessment of a submission.

5.2 The exercise of our functions as detailed in the ‘Introduction’ section will only be properly achieved with significant input from stakeholders, particularly in response to the issues we raised on the system. These inputs are set out in the review process. It is our view that there should be a large degree of consistency in the reviews and in the categorisation of issues arising, irrespective of the submission being reviewed, as this will allow for a more complete record of issues to be analysed and considered. It is also our view that undertaking this work promptly will allow the opportunity for explanations to be provided early, informing potential actions to address issues.

5.3 The SSRO proposes that the methodology sets out the linkage between the compliance review process and the SSRO’s other functions. In particular it is suggested that the methodology makes clear that intelligence gathered from the compliance process will be used to inform future priorities for the SSRO, which should in turn improve the quality of the data held in the system.

5.4 Respondents to the working paper stated that it should first be demonstrated that greater transparency as to how the SSRO logs issues and prioritises action will deliver benefits. It is important to make clear that there may be ongoing work that arises from the findings of compliance reviews and that this will be appropriately prioritised.

5.5 Three respondents commented that the compliance process needs to be sufficiently flexible to deal with simple issues such as arithmetical or typographical errors, which may be resolved directly by a contractor, as well as to deal with more complex issues needing to be resolved in a different manner. The SSRO agrees with these representations and believes that the methodology appropriately acknowledges there may be a range of work needed by the SSRO to address complex issues.

6. Additional compliance monitoring activities

6.1 The methodology sets out how the SSRO may seek to supplement its standard compliance reviews with additional targeted or thematic reviews.

6.2 One respondent to the working paper agreed that if an automated process was to be relied upon as part of the compliance approach then a risk-based approach to undertaking targeted reviews would be a means of validating whether the automated process was correct and provided the right emphasis. Another respondent noted that both targeted and thematic reviews could be useful.

6.3 Three respondents felt that these activities represented normal incremental improvements, evolution and quality assurance activities that would be expected with this type of process. They considered that the SSRO could proceed with the activities without the need to formalise them in the methodology. The SSRO accepts this is the case, but is committed to transparency and believes the methodology should reflect the various ways in which the SSRO discharges its functions. These activities are referred to in the methodology, without excessively detailing how they may be undertaken. This will allow for incremental developments to take place in future, without having to amend the methodology.

7. MOD reviews of report submissions

7.1 One respondent to the working paper stated that the SSRO should detail the links between the MOD’s review process and the compliance work undertaken by the SSRO, ensuring that they are clear and transparent. Three respondents stated that it would be inappropriate to link the SSRO’s work to the MOD’s reviews and thought that careful consideration would be required to ensure that the circumstances and the reasons for the MOD’s issues were understood fully before any further action could be taken.

7.2 The SSRO agrees that care needs to be taken to differentiate between its responsibilities and those of the MOD. There are, however, clear roles for both the SSRO and the MOD in relation to understanding the extent to which reporting requirements are being met and how the provision of the regulatory framework is being applied. For example, only the MOD is able to highlight any inconsistencies between what has been reported by the contractor and what has been contractually agreed to, particularly in relation to allowable costs and the contract profit rate.

7.3 The identification of any inaccuracies or inconsistencies with contractual information by the MOD, followed by subsequent corrections by contractors, will lead to improved data quality within the system. By undertaking analysis of the queries being raised by the MOD, and any subsequent amendments that are made by contractors, the SSRO will have a greater understanding of whether the information being provided is sufficient for the MOD’s contract management purposes. Where it is not, it will be able to consider whether it needs to undertake further work, such as an update to its guidance, to address such issues to prevent reoccurrence.

7.4 The methodology sets out an appropriate interface between the SSRO’s compliance reviews and the work of the MOD. It identifies that the SSRO will only undertake its reviews after giving the MOD opportunity to do so first. The SSRO will analyse issues raised by the MOD following its review of submissions, specifically to understand data quality issues within reports and also whether the information being provided is sufficient for the MOD’s contract management purposes, adding to the evidence base available to support the regulatory framework.

8. Raising matters with the MOD

8.1 The methodology also sets out the circumstances in which the SSRO will notify the MOD of delayed submissions and of any specific issue that it has raised on a submission. This enables the MOD to provide feedback to inform the SSRO’s compliance reviews and to take any appropriate action.

8.2 Respondents to the working paper were generally of the view that the SSRO should not recommend compliance action to the MOD, as the SSRO would need to maintain an independent position should a matter be subsequently referred to it. The draft methodology does not provide for the SSRO to recommend that the MOD takes enforcement action, but ensures that the MOD has the relevant information to make an informed decision.

8.3 The methodology recognises that there may be a range of responses from the MOD to issues raised by the SSRO. It emphasises the importance of feedback from the MOD to ensure the SSRO understands the extent to which reporting requirements are being met and how the regulatory framework is being applied.

9. Supporting compliance and improving the regime

9.1 With a view to understanding the reporting issues that a contractor may face and in turn improving the timeliness and data quality of submissions, the SSRO seeks to engage early in the reporting process with those subject to reporting obligations and the methodology sets out how we seek to undertake this.

9.2 Respondents to the working paper, and at the workshop, agreed that early engagement to support and enable compliance was a constructive development to the approach. They acknowledged positive previous experiences with the process for initial set up of a qualifying contract onto the system and with our engagement in the reporting of these contracts. Respondents also noted that early engagement by the SSRO and the capacity to enable contractors to gain early access to DefCARS has improved the quality of submissions. Three respondents to the working paper did, however, question whether it was necessary to formally reflect this focus in the methodology as this was good practice and would be carried out as a matter of course.

9.3 The SSRO does not have an express support function. We are, however, focused on exercising the statutory functions linked to our compliance methodology to encourage the use and quality of reported data as this lends itself to promoting our statutory aims. To this end, we may only provide support to the extent that it facilitates, or is conducive or ancillary to, one of our functions. It is the SSRO’s view that our:

• help desk assists people to understand our reporting guidance and also helps us understand how the regime is operating through the queries we receive;
• Reporting and DefCARS training and the use of automated validation warnings supports how we facilitate electronic submission of reports, compliance monitoring and analysis and that the training and analysis of non-resolved validation warnings provides opportunities for us to understand how the regime is operating; and
• on-boarding is in a similar category to the Reporting and DefCARS training.

9.4 Our view is that we should refer to this early engagement and support in the compliance methodology as it is helpful to link monitoring of reporting compliance with how we help people to make their submissions in the first instance.

10. Reporting findings and providing feedback

10.1 The methodology sets out how the SSRO will provide feedback to the MOD and to contractors on issues identified.

10.2 The SSRO proposed some additional compliance metrics in the working paper and asked for stakeholder feedback on those and any other metrics that stakeholders would find useful. Respondents to the working paper, and at the workshop, questioned the value that additional metrics would add to the process and one respondent queried where ‘more strategic level’ reports would go. Respondents did, however, state that it would be useful to have a regular report to the contractor detailing themes arising in their report reviews and a summary of outstanding issues in order to allow them to gauge their own performance at an overall level. Respondents also highlighted that the ability to comment on outstanding issues and the ability to explain why they might still be outstanding would enrich the data available.

10.3 The methodology states that, in addition to the Annual Compliance Report, the SSRO will provide regular reports to the MOD, identifying where matters have had an impact either on data quality or the operation of the regulatory framework, and also to individual contractors with multiple contracts. One respondent noted that the provision of such reports will improve visibility of issues; understanding of error causes; the co-ordination of corrections and should ultimately lead to good quality data.

11. Stakeholder input and next steps

11.1 A summary table of the questions for stakeholders is provided at Appendix 1 for ease of reference.

11.2 The SSRO welcomes views from interested parties on these and any other aspects of this consultation in writing by 6 December 2019 to consultations@ssro.gov.uk. Members of the Reporting and IT Sub-group will have an opportunity to discuss this consultation paper at our offices on 5 November 2019.

11.3 The SSRO also invites stakeholders to discuss any of the issues raised in this working paper with it on an individual basis. If you wish to do so, please contact us via consultations@ssro.gov.uk.

Appendix 1: summary of questions.

Section heading	Question
Introduction	Question 1: Does the methodology clearly demonstrate how the SSRO will exercise its s36(2) function and how its s39(1) function may be informed though our work in this area?
The SSRO’s general approach	Question 2: Is the SSRO’s approach sufficiently clear from the methodology?
Review of report submissions	Question 3: Is the SSRO’s review process sufficiently clear from the methodology?
	Question 4: Do you agree that the methodology appropriately identifies how the findings from compliance reviews will inform the SSRO’s other work?
Additional compliance monitoring activities.	Question 5: Do you agree that these additional activities can be reflected in the methodology without setting out the detail of how each may be undertaken?
MOD reviews of report submissions	Question 6: Do you agree that the SSRO should only review a submission having given the MOD sufficient time to undertake its own review in the first instance?
Raising matters with the MOD	Question 7: What is your view on the general approach to the notification of issues to the MOD as reflected in the methodology?
Supporting compliance and improving the regime	Question 8: Are there any other elements of support or engagement that should be reflected in the methodology?
Reporting findings and providing feedback	Question 9: What are the key areas of feedback for the SSRO to provide to industry?