No Cover Image

Journal article 552 views 61 downloads

Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric

Alex Bradley Orcid Logo, Martyn Quigley

Studies in Higher Education, Pages: 1 - 14

Swansea University Author: Martyn Quigley

  • 63079.pdf

    PDF | Version of Record

    This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/),which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent.

    Download (2.17MB)

Check full text

DOI (Published version): 10.1080/03075079.2023.2196292

Abstract

The mass participation in higher education has led to greater spending by governments and students which has increased the focus on graduate outcomes. In England, the Office for Students (OfS) is planning to take regulatory action, using the Proceed metric, against universities and their courses whi...

Full description

Published in: Studies in Higher Education
ISSN: 0307-5079 1470-174X
Published: Informa UK Limited 2023
Online Access: Check full text

URI: https://cronfa.swan.ac.uk/Record/cronfa63079
Tags: Add Tag
No Tags, Be the first to tag this record!
first_indexed 2023-04-04T14:35:12Z
last_indexed 2023-04-06T03:23:55Z
id cronfa63079
recordtype SURis
fullrecord <?xml version="1.0" encoding="utf-8"?><rfc1807 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><bib-version>v2</bib-version><id>63079</id><entry>2023-04-04</entry><title>Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric</title><swanseaauthors><author><sid>45ba0b00b12b2a4cd533dcd42f0121d9</sid><firstname>Martyn</firstname><surname>Quigley</surname><name>Martyn Quigley</name><active>true</active><ethesisStudent>false</ethesisStudent></author></swanseaauthors><date>2023-04-04</date><deptcode>HPS</deptcode><abstract>The mass participation in higher education has led to greater spending by governments and students which has increased the focus on graduate outcomes. In England, the Office for Students (OfS) is planning to take regulatory action, using the Proceed metric, against universities and their courses which do not have 60% of students with positive outcomes within 15 months of graduation. This study uses simulations to explore how effectively the Proceed metric can (a) identify the true population level of positive outcomes, (b) explore the precision of those estimates, and (c) accurately classify courses below the 60% threshold. The simulation varied: level of positive outcomes within the population (20–90%), sample size (40–1000), and the percentage of the population sampled (30–90%). The bias (difference between sample and population estimate), coverage probability (proportion of true population estimates within the intervals), and precision of confidence intervals (average range of confidence intervals) were calculated. Three main findings were (a) the Proceed metric is accurate in terms of bias and coverage probability, (b) the estimates lack precision, especially with small sample sizes, and (c) the imprecision will impact the Proceed metric ability to correctly classify courses below the 60% threshold. Governments seeking to collect graduate survey data and use it to regulate universities should make every effort to maximise sample size and should resist the temptation to regulate at a micro-level (i.e. courses) since classification will likely be inaccurate, especially for those just below the threshold.</abstract><type>Journal Article</type><journal>Studies in Higher Education</journal><volume/><journalNumber/><paginationStart>1</paginationStart><paginationEnd>14</paginationEnd><publisher>Informa UK Limited</publisher><placeOfPublication/><isbnPrint/><isbnElectronic/><issnPrint>0307-5079</issnPrint><issnElectronic>1470-174X</issnElectronic><keywords>Employability, OfS, B3 regulations, proceed metric, graduate outcomes</keywords><publishedDay>4</publishedDay><publishedMonth>4</publishedMonth><publishedYear>2023</publishedYear><publishedDate>2023-04-04</publishedDate><doi>10.1080/03075079.2023.2196292</doi><url>http://dx.doi.org/10.1080/03075079.2023.2196292</url><notes/><college>COLLEGE NANME</college><department>Psychology</department><CollegeCode>COLLEGE CODE</CollegeCode><DepartmentCode>HPS</DepartmentCode><institution>Swansea University</institution><apcterm>SU Library paid the OA fee (TA Institutional Deal)</apcterm><funders/><projectreference/><lastEdited>2023-05-18T15:23:19.4804857</lastEdited><Created>2023-04-04T15:17:04.6970085</Created><path><level id="1">Faculty of Medicine, Health and Life Sciences</level><level id="2">School of Psychology</level></path><authors><author><firstname>Alex</firstname><surname>Bradley</surname><orcid>0000-0003-4304-7653</orcid><order>1</order></author><author><firstname>Martyn</firstname><surname>Quigley</surname><order>2</order></author></authors><documents><document><filename>63079__27243__954f2e1e7588454cbab8ffb3ada59ac8.pdf</filename><originalFilename>63079.pdf</originalFilename><uploaded>2023-04-27T14:58:03.3579322</uploaded><type>Output</type><contentLength>2270378</contentLength><contentType>application/pdf</contentType><version>Version of Record</version><cronfaStatus>true</cronfaStatus><documentNotes>This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/),which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent.</documentNotes><copyrightCorrect>true</copyrightCorrect><language>eng</language><licence>http://creativecommons.org/licenses/by/4.0/</licence></document></documents><OutputDurs/></rfc1807>
spelling v2 63079 2023-04-04 Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric 45ba0b00b12b2a4cd533dcd42f0121d9 Martyn Quigley Martyn Quigley true false 2023-04-04 HPS The mass participation in higher education has led to greater spending by governments and students which has increased the focus on graduate outcomes. In England, the Office for Students (OfS) is planning to take regulatory action, using the Proceed metric, against universities and their courses which do not have 60% of students with positive outcomes within 15 months of graduation. This study uses simulations to explore how effectively the Proceed metric can (a) identify the true population level of positive outcomes, (b) explore the precision of those estimates, and (c) accurately classify courses below the 60% threshold. The simulation varied: level of positive outcomes within the population (20–90%), sample size (40–1000), and the percentage of the population sampled (30–90%). The bias (difference between sample and population estimate), coverage probability (proportion of true population estimates within the intervals), and precision of confidence intervals (average range of confidence intervals) were calculated. Three main findings were (a) the Proceed metric is accurate in terms of bias and coverage probability, (b) the estimates lack precision, especially with small sample sizes, and (c) the imprecision will impact the Proceed metric ability to correctly classify courses below the 60% threshold. Governments seeking to collect graduate survey data and use it to regulate universities should make every effort to maximise sample size and should resist the temptation to regulate at a micro-level (i.e. courses) since classification will likely be inaccurate, especially for those just below the threshold. Journal Article Studies in Higher Education 1 14 Informa UK Limited 0307-5079 1470-174X Employability, OfS, B3 regulations, proceed metric, graduate outcomes 4 4 2023 2023-04-04 10.1080/03075079.2023.2196292 http://dx.doi.org/10.1080/03075079.2023.2196292 COLLEGE NANME Psychology COLLEGE CODE HPS Swansea University SU Library paid the OA fee (TA Institutional Deal) 2023-05-18T15:23:19.4804857 2023-04-04T15:17:04.6970085 Faculty of Medicine, Health and Life Sciences School of Psychology Alex Bradley 0000-0003-4304-7653 1 Martyn Quigley 2 63079__27243__954f2e1e7588454cbab8ffb3ada59ac8.pdf 63079.pdf 2023-04-27T14:58:03.3579322 Output 2270378 application/pdf Version of Record true This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/),which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent. true eng http://creativecommons.org/licenses/by/4.0/
title Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric
spellingShingle Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric
Martyn Quigley
title_short Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric
title_full Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric
title_fullStr Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric
title_full_unstemmed Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric
title_sort Governments harnessing the power of data to get ‘value for money’: a simulation study of England’s Office for Students B3 Proceed Metric
author_id_str_mv 45ba0b00b12b2a4cd533dcd42f0121d9
author_id_fullname_str_mv 45ba0b00b12b2a4cd533dcd42f0121d9_***_Martyn Quigley
author Martyn Quigley
author2 Alex Bradley
Martyn Quigley
format Journal article
container_title Studies in Higher Education
container_start_page 1
publishDate 2023
institution Swansea University
issn 0307-5079
1470-174X
doi_str_mv 10.1080/03075079.2023.2196292
publisher Informa UK Limited
college_str Faculty of Medicine, Health and Life Sciences
hierarchytype
hierarchy_top_id facultyofmedicinehealthandlifesciences
hierarchy_top_title Faculty of Medicine, Health and Life Sciences
hierarchy_parent_id facultyofmedicinehealthandlifesciences
hierarchy_parent_title Faculty of Medicine, Health and Life Sciences
department_str School of Psychology{{{_:::_}}}Faculty of Medicine, Health and Life Sciences{{{_:::_}}}School of Psychology
url http://dx.doi.org/10.1080/03075079.2023.2196292
document_store_str 1
active_str 0
description The mass participation in higher education has led to greater spending by governments and students which has increased the focus on graduate outcomes. In England, the Office for Students (OfS) is planning to take regulatory action, using the Proceed metric, against universities and their courses which do not have 60% of students with positive outcomes within 15 months of graduation. This study uses simulations to explore how effectively the Proceed metric can (a) identify the true population level of positive outcomes, (b) explore the precision of those estimates, and (c) accurately classify courses below the 60% threshold. The simulation varied: level of positive outcomes within the population (20–90%), sample size (40–1000), and the percentage of the population sampled (30–90%). The bias (difference between sample and population estimate), coverage probability (proportion of true population estimates within the intervals), and precision of confidence intervals (average range of confidence intervals) were calculated. Three main findings were (a) the Proceed metric is accurate in terms of bias and coverage probability, (b) the estimates lack precision, especially with small sample sizes, and (c) the imprecision will impact the Proceed metric ability to correctly classify courses below the 60% threshold. Governments seeking to collect graduate survey data and use it to regulate universities should make every effort to maximise sample size and should resist the temptation to regulate at a micro-level (i.e. courses) since classification will likely be inaccurate, especially for those just below the threshold.
published_date 2023-04-04T15:23:18Z
_version_ 1766242174173708288
score 11.037581

Similar Items