Department of Education

INTRODUCTION: We use a utility model to illustrate that, firstly, selecting an assessment method involves context-dependent compromises, and secondly, that assessment is not a measurement problem but an instructional design problem, comprising educational, implementation and resource aspects. In the model, assessment characteristics are differently weighted depending on the purpose and context of the assessment. EMPIRICAL AND THEORETICAL DEVELOPMENTS: Of the characteristics in the model, we focus on reliability, validity and educational impact and argue that they are not inherent qualities of any instrument. Reliability depends not on structuring or standardisation but on sampling. Key issues concerning validity are authenticity and integration of competencies. Assessment in medical education addresses complex competencies and thus requires quantitative and qualitative information from different sources as well as professional judgement. Adequate sampling across judges, instruments and contexts can ensure both validity and reliability. Despite recognition that assessment drives learning, this relationship has been little researched, possibly because of its strong context dependence. ASSESSMENT AS INSTRUCTIONAL DESIGN: When assessment should stimulate learning and requires adequate sampling, in authentic contexts, of the performance of complex competencies that cannot be broken down into simple parts, we need to make a shift from individual methods to an integral programme, intertwined with the education programme. Therefore, we need an instructional design perspective. IMPLICATIONS FOR DEVELOPMENT AND RESEARCH: Programmatic instructional design hinges on a careful description and motivation of choices, whose effectiveness should be measured against the intended outcomes. We should not evaluate individual methods, but provide evidence of the utility of the assessment programme as a whole.

A new theory of the development of expertise in medicine is outlined. Contrary to existing views, this theory assumes that expertise is not so much a matter of superior reasoning skills or in-depth knowledge of pathophysiological states as it is based on cognitive structures that describe the features of prototypical or even actual patients. These cognitive structures, referred to as "illness scripts," contain relatively little knowledge about pathophysiological causes of symptoms and complaints but a wealth of clinically relevant information about disease, its consequences, and the context under which illness develops. By contrast, intermediate-level students without clinical experience typically use pathophysiological, causal models of disease when solving problems. The authors review evidence supporting the theory and discuss its implications for the understanding of five phenomena extensively documented in the clinical-reasoning literature: (1) content specificity in diagnostic performance; (2) typical differences in data-gathering techniques between medical students and physicians; (3) difficulties involved in setting standards; (4) a decline in performance on certain measures of clinical reasoning with increasing expertise; and (5) a paradoxical association between errors and longer response times in visual diagnosis.

CONTEXT: Problem-based learning (PBL) is widely used in higher education. There is evidence available that students and faculty are highly satisfied with PBL. Nevertheless, in educational practice problems are often encountered, such as tutors who are too directive, problems that are too well-structured, and dysfunctional tutorial groups. PURPOSE: The aim of this paper is to demonstrate that PBL has the potential to prepare students more effectively for future learning because it is based on four modern insights into learning: constructive, self-directed, collaborative and contextual. These four learning principles are described and it is explained how they apply to PBL. In addition, available research is reviewed and the current debate in research on PBL is described. DISCUSSION: It is argued that problems encountered in educational practice usually stem from poor implementation of PBL. In many cases the way in which PBL is implemented is not consistent with the current insights on learning. Furthermore, it is argued that research on PBL should contribute towards a better understanding of why and how the concepts of constructive, self-directed, collaborative and contextual learning work or do not work and under what circumstances. Examples of studies are given to illustrate this issue.

In assessment a considerable shift in thinking has occurred from assessment of learning to assessment for learning. This has important implications for the conceptual framework from which to approach the issue of assessment, but also with respect to the research agenda. The main conceptual changes pertain to programmes of assessment. This has led to a broadened perspective on the types of construct assessment tries to capture, the way information from various sources is collected and collated, the role of human judgement and the variety of psychometric methods to determine the quality of the assessment. Research into the quality of assessment programmes, how assessment influences learning and teaching, new psychometric models and the role of human judgement is much needed.

We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory.

SUMMARY The question of whether problems of motor co‐ordination in early childhood recede with age has rarely been addressed. This paper reports the findings from a follow‐up study of 17 children, identified by their teachers as having poor motor co‐ordination at age six. Now age 16, these children and their matched controls completed a battery of assessments. The results suggest that the majority of children still have difficulties with motor co‐ordination, have poor self‐concept and are experiencing problems of various kinds in school. However, there are individual differences in the extent to which the children have learned to cope with their continuing difficulties over the years. RÉSUMÉ Maladresse chez l'enfant: disparait‐elle? Etude de suivi durant dix ans La question de savoir si les problèmes de coordination de l'enfance régressent avec l'ǎge a été rarement posée. L'article rapporte les données d'une étude longitudinale de 17 enfants, chez qui les enseignants avaient détecté une coordination motrice médiocre à l'ǎge de six ans. Ces sujets ont étéévalués par une batterie, avec des contrǒles appariés, a l'ǎge de 16 ans. Les résultats suggèrent que la majorité des sujets présentaient encore des difficultés en rapport avec une mauvaise coordination motrice, avaient une mauvaise image de leurs capacités et avaient eu des problèmes variés à l'école. Il y avait cependant des différences individuelles dans I'aptitude à intégrer les difficultés au cours des annees. ZUSAMMENFASSUNG Ungeschicklichkeit bei Kindern: wie stehen die Chancen für eine Normalisierung? Eine Verlaufstudie über 10 Jahre Es ist selten die Frage gestellt worden, ob Probleme der motorischen Koordination im frühen Kindesalter im Laufe der Jahre verschwinden. In dieser Arbeit werden die Ergebnisse einer Verlaufsstudie bei 17 Kindern dargestellt, die im Alter von sechs Jahren von ihren Lehrern eine schlechte Beurteilung ihrer motorischen Koordination bekommen hatten. Jetzt, im Alter von 16 Jahren, wurden diese Kinder, sowie ihre Kontrollen, einer Reihe von Untersuchungen unterzogen. Die Ergebnisse zeigen, da8 die Mehrzahl der Kinder noch immer Schwierigkeiten bei der motorischen Koordination, sowie ein mangelhaftes Selbstwertgefühl und verschiedene Probleme in der Schule hat. Es gibt jeoch individuelle Unterschiede, wie die Kinder gelernt haben, ihre fortbestehenden Schwierigkeiten im Verlaufe der Jahre zu meistern. RESUMEN Torpeza en niños:? Ilegan con la edad a libererse de ella? Estudio a lo largo de diez años Sólo muy raramente se ha planteado la pregunta de si los problemas de coordinación Ilegan a desaparecer con la edad. Este trabajo expone los hallazgos obtenidos en un estudio continuado de 17 niños, identificados por sus maestros como teniendo una coordinacion pobre a la edad de seis años. Ahora, con 16 años estos niños y un grupo control sa ha completado una bateria de exámenes. Los resultados sugieren que la mayoria de niños tienen todavia dificultades en la coordinacion motora. Tienen un autoconcepto pobre y sufren problemas de distinto tipo en la escuela. Sin embargo hay diferencias individuales concerniendo a lo aue el niño ha aprendido para adaptarse a sus continuas dificultades a lo largo de los años.

This article provides an introductory overview of findings from the past 40 years of research on bullying among school-aged children and youth. Research on definitional and assessment issues in studying bullying and victimization is reviewed, and data on prevalence rates, stability, and forms of bullying behavior are summarized, setting the stage for the 5 articles that comprise this American Psychologist special issue on bullying and victimization. These articles address bullying, victimization, psychological sequela and consequences, ethical, legal, and theoretical issues facing educators, researchers, and practitioners, and effective prevention and intervention efforts. The goal of this special issue is to provide psychologists with a comprehensive review that documents our current understanding of the complexity of bullying among school-aged youth and directions for future research and intervention efforts.

Abstract A little more than 10 years ago, the objective structured clinical examination (OSCE) was introduced. It includes several "stations,"; at which examinees perform a variety of clinical tasks. Although an OSCE may involve a range of testing methods, standardized patients (SPs), who are nonphysicians trained to play the role of a patient, are commonly used to assess clinical skills. This article provides a comprehensive review of large‐scale studies of the psychometric characteristics of SP‐based tests. Across studies, reliability analyses consistently indicate that the major source of measurement error is variation in examinee performance from station to station (termed content specificity in the medical‐problem‐solving literature). As a consequence, tests must include large numbers of stations to obtain a stable, reproducible assessment of examinee skills. Disagreements among raters observing examinee performance and differences between SPs playing the same patient role appear to have less effect on the precision of scores, as long as examinees are randomly assigned to raters and SPs. Results of validation studies (e.g., differences in group performance, correlations with other measures) are generally favorable, though not particularly informative. Future validation research should investigate the impact of station format, timing, and instructions on examinee performance; study the procedures used to translate examinee behavior into station and test scores; and work on rater and SP bias. Several recommendations are offered for improving SP‐based tests. These include (a) focusing on assessment of history taking, physical examination, and communication skills, with separately administered written tests used to measure diagnostic and management skills, (b) adoption of a mastery‐testing framework for score interpretation, and (c) development of improved procedures for setting pass‐fail standards. Use of generalizability theory in analyzing and reporting results of psychometric studies is also suggested.

From Pong to virtual reality, Understanding Video Games, 4th Edition, takes video game studies into the next decade of the twenty-first century, highlighting changes in the area, including mobile, social, and casual gaming.<br/><br/>In this new edition of the pioneering text, students learn to assess the major theories used to analyze games, such as ludology and narratology, and gain familiarity with the commercial and organizational aspects of the game industry. Drawing from historical and contemporary examples, the student-friendly text also explores the aesthetics of games, evaluates the cultural position of video games, and considers the potential effects of both violent and "serious" games.<br/><br/>Extensively illustrated and featuring discussion questions, a glossary of key terms, and a detailed video game history timeline, this new edition is an indispensable resource for students, scholars, and teachers interested in examining the ways video games continue to reshape entertainment and society.

As more Americans enter college than ever before, their pathways through the broadly differentiated higher education system are changing. Movement in, out, and among institutions now characterizes students' attendance patterns—half of all undergraduates who begin at a four-year institution go on to attend at least one other college, and over one-third take some time off from college after their initial enrollment. This study investigated whether there is social-class variation in these patterns, with advantaged and disadvantaged students responding to new postsecondary choices by engaging in different pathways. National longitudinal data from postsecondary transcripts were used to follow students across schools and to examine the importance of family background and high school preparation in predicting forms of college attendance. The results demonstrate that students from lower socioeconomic backgrounds are more likely than are economically advantaged students (net of prior academic preparation) to follow pathways that are characterized by interrupted movement. Such pathways appear to be less effective routes to the timely completion of degrees. Thus, difference in how students attend college represent an additional layer of stratification in higher education.

More than two hundred papers have reported genome-wide data from ancient humans. While the raw data for the vast majority are fully publicly available testifying to the commitment of the paleogenomics community to open data, formats for both raw data and meta-data differ. There is thus a need for uniform curation and a centralized, version-controlled compendium that researchers can download, analyze, and reference. Since 2019, we have been maintaining the Allen Ancient DNA Resource (AADR), which aims to provide an up-to-date, curated version of the world's published ancient human DNA data, represented at more than a million single nucleotide polymorphisms (SNPs) at which almost all ancient individuals have been assayed. The AADR has gone through six public releases at the time of writing and review of this manuscript, and crossed the threshold of >10,000 individuals with published genome-wide ancient DNA data at the end of 2022. This note is intended as a citable descriptor of the AADR.

CONTEXT: The move towards competence-based medical education has created a need for instruments that support and assess competence development. Portfolios seem suitable but mixed reports of their success are emerging. METHODS: To examine the effectiveness of portfolios, we searched PubMed and EMBASE using the keyword 'portfol*', PsychInfo and ERIC using the keywords 'portfol*' and 'medical education' and references of retrieved papers for empirical studies on portfolios in all phases of medical education. Thirty of 1939 retrieved papers met the inclusion criteria and were analysed. Data were collated against the research question, number of subjects, design, setting, findings and limitations, purpose and content, mentoring and assessment. We analysed impact using a modified version of Kirkpatrick's hierarchy. RESULTS: Because differences across studies precluded statistical meta-analysis, the data were analysed by context, goals and procedure. Positive effects were strongest in undergraduate education. Important factors for success were: clearly communicated goals and procedures; integration with curriculum and assessment; flexible structure; support through mentoring, and measures to heighten feasibility and reduce required time. Moderately good inter-rater reliability was reported and global criteria and discussions among raters were beneficial. Formative and summative assessment could be combined. Without assessment, portfolios were vulnerable to competition from other summative assessment instruments. CONCLUSIONS: For portfolios to be effective in supporting and assessing competence development, robust integration into the curriculum and tutor support are essential. Further studies should focus on the effectiveness and user-friendliness of portfolios, the merits of holistic assessment procedures, and the competences of an effective portfolio mentor.

Abstract. Socially shared regulation of learning (SSRL) has been recognized as a new and growing field in the framework of self-regulated learning theory in the past decade. In the present review, we examine the empirical evidence to support such a phenomenon. A total of 17 articles addressing SSRL were identified, 13 of which presented empirical evidence. Through a narrative review it could be concluded that there is enough data to maintain the existence of SSRL in comparison to other social regulation (e.g., co-regulation). It was found that most of the SSRL research has focused on characterizing phenomena through the use of mixed methods through qualitative data, mostly video-recorded observation data. Also, SSRL seems to contribute to students’ performance. Finally, the article discusses the need for the field to move forward, exploring the best conditions to promote SSRL, clarifying whether SSRL is always the optimal form of collaboration, and identifying more aspects of groups’ characteristics.

INTRODUCTION: Written assessment techniques can be subdivided according to their stimulus format--what the question asks--and their response format--how the answer is recorded. The former is more important in determining the type of competence being asked for than the latter. It is nevertheless important to consider both when selecting the most appropriate types. Some major elements to consider when making such a selection are cueing effect, reliability, validity, educational impact and resource-intensiveness. RESPONSE FORMATS: Open-ended questions should be used solely to test aspects that cannot be tested with multiple-choice questions. In all other cases the loss of reliability and the higher resource-intensiveness represent a significant downside. In such cases, multiple-choice questions are not less valid than open-ended questions. STIMULUS FORMAT: When making this distinction, it is important to consider whether the question is embedded within a relevant case or context and cannot be answered without the case, or not. This appears to be more or less essential according to what is being tested by the question. Context-rich questions test other cognitive skills than do context-free questions. If knowledge alone is the purpose of the test, context-free questions may be useful, but if it is the application of knowledge or knowledge as a part of problem solving that is being tested, then context is indispensable. CONCLUSION: Every format has its (dis)advantages and a combination of formats based on rational selection is more useful than trying to find or develop a panacea. The response format is less important in this respect than the stimulus.

Cognitive Load Theory (CLT) has started to find more applications in medical education research. Unfortunately, misconceptions such as lower cognitive load always being beneficial to learning and the continued use of dated concepts and methods can result in improper applications of CLT principles in medical education design and research. This review outlines how CLT has evolved and presents a synthesis of current-day CLT principles in a holistic model for medical education design. This model distinguishes three dimensions: task fidelity: from literature (lowest) through simulated patients to real patients (highest); task complexity: the number of information elements; and instructional support: from worked examples (highest) through completion tasks to autonomous task performance (lowest). These three dimensions together constitute three steps to proficient learning: (I) start with high support on low-fidelity low-complexity tasks and gradually fade that support as learners become more proficient; (II) repeat I for low-fidelity but higher-complexity tasks; and (III) repeat I and II in that order at subsequent levels of fidelity. The numbers of fidelity levels and complexity levels within fidelity levels needed depend on the aims of the course, curriculum or individual learning trajectory. This paper concludes with suggestions for future research based on this model.

AIM: Portfolios are often used as an instrument with which to stimulate students to reflect on their experiences. Research has shown that working with portfolios does not automatically stimulate reflection. In this study we addressed the question: What are the conditions for successful reflective use of portfolios in undergraduate medical education? METHODOLOGY/RESEARCH DESIGN: We designed a portfolio that was aimed at stimulating reflection in early undergraduate medical education, using experiences described in the medical education literature and elsewhere. Conditions for reflective portfolio use were identified through interviews with 13 teachers (mentors), who were experienced in mentoring students in the process of developing their portfolios. The interviews were analysed according to the principles of grounded theory. RESULTS: The conditions for successful reflective use of portfolios that emerged from the interviews fell into 4 categories: coaching; portfolio structure and guidelines; relevant experiences and materials, and summative assessment. According to the mentors, working with a portfolio designed to meet these conditions will stimulate students' reflective abilities. CONCLUSION: This study shows that portfolios are a potentially valuable method of assessing and developing students' reflective skills in undergraduate medical training, provided certain conditions for effective portfolios are recognised and met. Portfolios have a strong potential for enhancing learning and assessment but they are very vulnerable and may easily lead to disappointment. Before implementing portfolios in education, one should first consider whether the necessary conditions can be fulfilled, including an appropriate portfolio structure, an appropriate assessment procedure, the provision of enough new experiences and materials, and sufficient teacher capacity for adequate coaching and assessment.

CONTEXT: Although work-based assessments (WBA) may come closest to assessing habitual performance, their use for summative purposes is not undisputed. Most criticism of WBA stems from approaches to validity consistent with the quantitative psychometric framework. However, there is increasing research evidence that indicates that the assumptions underlying the predictive, deterministic framework of psychometrics may no longer hold. In this discussion paper we argue that meaningfulness and appropriateness of current validity evidence can be called into question and that we need alternative strategies to assessment and validity inquiry that build on current theories of learning and performance in complex and dynamic workplace settings. METHODS: Drawing from research in various professional fields we outline key issues within the mechanisms of learning, competence and performance in the context of complex social environments and illustrate their relevance to WBA. In reviewing recent socio-cultural learning theory and research on performance and performance interpretations in work settings, we demonstrate that learning, competence (as inferred from performance) as well as performance interpretations are to be seen as inherently contextualised, and can only be under-stood 'in situ'. Assessment in the context of work settings may, therefore, be more usefully viewed as a socially situated interpretive act. DISCUSSION: We propose constructivist-interpretivist approaches towards WBA in order to capture and understand contextualised learning and performance in work settings. Theoretical assumptions underlying interpretivist assessment approaches call for a validity theory that provides the theoretical framework and conceptual tools to guide the validation process in the qualitative assessment inquiry. Basic principles of rigour specific to qualitative research have been established, and they can and should be used to determine validity in interpretivist assessment approaches. If used properly, these strategies generate trustworthy evidence that is needed to develop the validity argument in WBA, allowing for in-depth and meaningful information about professional competence.

CONTEXT: Performance assessments, such as workplace-based assessments (WBAs), represent a crucial component of assessment strategy in medical education. Persistent concerns about rater variability in performance assessments have resulted in a new field of study focusing on the cognitive processes used by raters, or more inclusively, by assessors. METHODS: An international group of researchers met regularly to share and critique key findings in assessor cognition research. Through iterative discussions, they identified the prevailing approaches to assessor cognition research and noted that each of them were based on nearly disparate theoretical frameworks and literatures. This paper aims to provide a conceptual review of the different perspectives used by researchers in this field using the specific example of WBA. RESULTS: Three distinct, but not mutually exclusive, perspectives on the origins and possible solutions to variability in assessment judgements emerged from the discussions within the group of researchers: (i) the assessor as trainable: assessors vary because they do not apply assessment criteria correctly, use varied frames of reference and make unjustified inferences; (ii) the assessor as fallible: variations arise as a result of fundamental limitations in human cognition that mean assessors are readily and haphazardly influenced by their immediate context, and (iii) the assessor as meaningfully idiosyncratic: experts are capable of making sense of highly complex and nuanced scenarios through inference and contextual sensitivity, which suggests assessor differences may represent legitimate experience-based interpretations. CONCLUSIONS: Although each of the perspectives discussed in this paper advances our understanding of assessor cognition and its impact on WBA, every perspective has its limitations. Following a discussion of areas of concordance and discordance across the perspectives, we propose a coexistent view in which researchers and practitioners utilise aspects of all three perspectives with the goal of advancing assessment quality and ultimately improving patient care.

As a result of the COVID-19 pandemic, many school districts have closed for the remainder of the academic year. These closures are unfortunate because, for many students, schools are their only source of trauma-informed care and supports. When schools reopen, they must develop a comprehensive plan to address the potential mental health needs of their students. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

AIM: The aim of this paper was to describe the anthropometric and physiological characteristics of young soccer players (14-17 years old) which were associated with their being successful or not as soccer players. METHODS: Somatotype and body composition was calculated by measuring skinfolds, limb circumferences and joint diameters. VO(2max) was estimated by the Astrand's Test. Sprint, jump and endurance tests were also performed. RESULTS: The most relevant differences were obtained between selected and non-selected players belonging to the 14-year-old team. Selected players were taller, heavier, leaner and faster and they had higher absolute or relative VO(2max). In addition, a higher % of selected players was found among those born during the first 6 months of the year. In the rest of the teams, the agility was better in selected than in non-selected players. At later ages, there was also a predominance of players born during the first 6 months of the year. CONCLUSIONS: These results indicate that around the time of puberty, parameters associated with physical maturity such as height, size, speed, VO(2max), or chronological age are important to determine the success of a soccer player. At older ages, other factors such as agility seem to be more important. Nevertheless, players born in the 1st semester of the year are also more frequent in the older teams. These findings should be taken into account by trainers and coaches, in order to avoid biasing their selection choices.