Challenges in using data across government

27 June 2019

A report from the National Audit Office (NAO) has found that data is not seen as a priority by government; the importance of investing in quality data is not well understood, and there is a culture of tolerating and working around poor data.


The report examines how government uses data to implement its policies and reform public services and draws together the NAO’s experience of auditing government, and recent work that has identified where the quality and availability of data has been an issue.


Data is crucial to how successfully government can: deliver services that work for the people who use them; improve its systems and processes; and support better decisions. The NAO has repeatedly highlighted the importance of evidence-based decision making in government, and the problems that arise when data is missing or low-quality.


The effectiveness of government programmes is often compromised because data is inadequate. The consequence of this is demonstrated by the Windrush situation, where the Home Office took action without fully assessing the impact of the quality of its underlying data. Unless the quality and availability of data is high before new policies are implemented, it can be difficult for departments to assess their effectiveness or secure hoped for benefits.


The report includes information about data projects that have sometimes been set aside when funding is under pressure such as the government funding projects to automate data feeds like the Department for Work & Pensions using HM Revenue & Customs real-time information to support benefit payments. This has the potential to provide benefits, but this has often been driven by new policies rather than fixing ongoing problems.


The NAO report states that the Government recognises the value of using data more effectively and that it plans to produce a new national data strategy in 2020, which will be led by the Department for Digital, Culture, Media and Sport (DCMS), working with Cabinet Office. To achieve its ambitions, government will need to resolve challenges around how to use and share data safely and appropriately, and how to balance competing demands for public money in a way that allows it to invest in data. The NAO has found government has lacked leadership on developing its use of data, and DCMS has acknowledged that it needs to do more to embed its leadership role.


The report finds that there are no cross-government standards on how to record data, which has led to inconsistent ways of recording information. The NAO found more than 20 different ways of identifying individuals and businesses across 10 departments and agencies, with no standard format for recording data such as name, address and date of birth. The problem is replicated for businesses and in local areas where information is recorded differently across boundaries. This makes it difficult for government to maximise its data, for example by allowing analysis across different sectors to help understand economic challenges or systematic problems.


Silo working can inhibit progress. While the Digital Economy Act 2017 has made it easier for departments to share data where it is appropriate to do so, many still lack confidence in how to share it legally. The NAO has found that civil servants are used to working within departmental boundaries, which is a hurdle to setting up and maintaining cross-departmental initiatives.


The NAO recommends in its report that the Cabinet Office and DCMS use the 2020 data strategy to identify and address the barriers to better use of data. They should set up clearer cross-government accountability, governance and funding for data to help deliver the strategy. The datasets that are critical to government should also be identified, and it should be established how they can be improved and shared more easily.


Gareth Davies, the head of the NAO said:


“Government has lacked clear and sustained strategic leadership on data, and individual departments have not made enough effort to manage and improve the data they hold. This can reduce public confidence in government’s ability to collect and use people’s data effectively.


“The right processes, systems and conditions must now be put in place, otherwise the new data strategy will become yet another missed opportunity.”



Read the full report from the NAO - Challenges in using data across government.