Home   MosaicPresentation format
<http://cnfolio.com/LearningScorecard>
This report is supported by a grant from the Faculty Learning and Teaching Committee, Faculty of Technology, University of Portsmouth
Updated November 1, 2009

The Learning Scorecard

by Chi Nguyen



Objective

An important tenet of the Chinese philosophy tradition, Confucianism, is that everything needs to be given correct names. In the context of education, knowledge of methods and techniques for teaching has implicit impact on government policy, course development, compensation negotiations, student learning experience, continuing professional development and the public view of the teaching profession. This research investigated the perceptions of teaching methods within government, university and professional documents. The objective was an initial set of criteria to analyze and describe teaching methods, tentatively called a Learning Scorecard. The continued development of such a Learning Scorecard could assist the accurate identification ("naming") of teaching methods and critical analysis of the act of teaching.

Keywords: learning scorecard, teaching methods, impact of teaching, dimensions of teaching, order of precedence, units of measurement, pedagogy.


Elements of a Learning Scorecard

In both science and management, there is a common belief that measurement of an activity is necessary for analysis and improvement. The Learning Scorecard emphasizes 2 interdependent relationships. The first relationship reflects the necessity of identifying dimensions and units suitable for measuring teaching methods before attempting to assess their impact. The second relationship suggests that assessment of the impact of teaching methods is constrained or influenced by choices made about measurement dimensions and units. More importantly, accepted valuation of the impact for specific teaching methods directly affect their development and adoption.
Elements of a Learning Scorecard
Figure 1.   Elements of a Learning Scorecard

As an example, let's consider the teaching method of a teacher standing to lecture a class of 100 students who are expected to listen, ask questions and take notes. Furthermore, let's prescribe that the content of the lecture session is conducted on a whiteboard and is not distributed to students afterwards. But, this is not sufficient to specify a teaching method! Research published by the Mid-continent Research for Education and Learning (2000) described specific categories of instructional strategies which could be incorporated into a lecture session.
Categories of instructional strategies that strongly affect student achievement:
  1. Identifying similarities and differences
  2. Summarizing and note taking
  3. Reinforcing effort and providing recognition
  4. Homework and practice
  5. Nonlinguistic representations
  6. Cooperative learning
  7. Setting goals and providing feedback
  8. Generating and testing hypotheses
  9. Activating prior knowledge
(p. 4)
The dimensions and units for measuring the impact of a teaching method improve as it is specified in more details. But, the university and government documents reviewed for this report has a total absence of such details when making reference to teaching methods. Pedagogical strategy and method of delivery must be considered to determine suitable dimensions and units for measuring the impact of teaching methods.


Teaching Methods as Professional Skills

The Higher Education Academy Subject Centre for Education (2005) published a report by Bart McGettrick, Professor of Education at the University of Glasgow, which considered standards for the teaching profession. Item 2 of the list below made an important distinction between professional knowledge as compared to professional skills.
Within the past seven years the Quality Assurance Agency has spearheaded the publication of benchmarks for teaching. In the statements of benchmarks for the profession of teaching itself, there has been consideration given to three main areas. These are:
  • Professional knowledge and understanding
  • Professional skills and abilities
  • Professional values and personal commitment
(p. 8)

This distinction is clarified on page 10, which specified conceptual frameworks as a form of professional knowledge.
It is the expectation of professionals that they will undertake professional action which is rooted in one or more conceptual frameworks which are recognised and understood by fellow professionals. Their actions are therefore based on principles and knowledge that is authenticated by professional peers.

Later on page 17, the report suggested that there was a difference between teaching standards, academic awards and professional awards. Although the report indicated that a framework was required to reconcile the differences, it did not attempt to suggest such a framework!
The Higher Education Academy will have to align academic awards and professional awards. This also requires a framework to be established. The recognition of teaching standards at:
  • "Threshold Level",
  • "Lecturer Level", and
  • "Chartered Lecturer Level",
may be a suitably simple basis on which to begin. In time a greater degree of sophistication may be helpful, but a graded system of a simple kind would be useful at the beginning. These are indicated in Appendix 1 (p27), showing how they may relate to increased competence in teaching. This increase in competence is not only a matter of experience, but of effectiveness across different kinds of teaching in higher education.

The most direct reflection of teaching methods as professional skills is perhaps the official university guidelines for academic staff grading and pay. As of October 31, 2009, the Freedom of Information page (n.d.) on staff pay at the University of Portsmouth indicated that the guidelines are under review and not available!
Staff pay and grading structures
Information on the HE Role Analysis (HERA) scheme
  • An introduction to HERA - document currently under review but will appear here soon. Further information available from the Human Resources department.
  • Single pay spine for academic and HE support staff 2006-09 - document currently under review but will appear here soon. Further information available from the Human Resources department.

The only available alternative is the University of Portsmouth Academic Role Profiles document (2005). Below is an extract of selected requirements for professional levels 1 to 5, where 3 is a senior lecturer and 5 is a professor. A similar distinction has been made between professional knowledge ("Expertise") and professional skills ("Teaching and learning support" and "Research and scholarship"). Yet, this attempt at a comprehensive specification provides no help or guidance on how to apply the criteria in a consistent manner across all academic staff and all subject areas. This ambiguity is a source of disparity within the university and places university management in a disproportional position of advantage as "interpreters" of the specifications. Furthermore, the current document has no indication of relative priority or weighting amongst the criteria. If there is no priority, then all criteria are equally important, or worse, none are important. Yet, this is the main document for managing the professional career progression of academic staff.
ROLE PROFILES: TEACHING AND SCHOLARSHIP
LEVEL 1
  • Teaching and learning support
    • Develop own teaching materials, with assistance and support.
    • Set and mark assignments.
    • Contribute to the development of examination questions.
  • Expertise
    • Possess sufficient breadth or depth of specialist knowledge in the discipline and be developing further skills in and knowledge of teaching methods and techniques.
LEVEL 2
  • Teaching and learning support
    • Teach in a developing capacity in a variety of settings from small group tutorials to large lectures.
    • Transfer knowledge in the form of practical skills, methods and techniques.
    • Identify learning needs of students and define appropriate learning objectives.
    • Ensure that content, methods of delivery and learning materials will meet the defined learning objectives.
    • Develop own teaching materials, methods and approaches with guidance.
    • Develop the skills of applying appropriate approaches to teaching.
    • Challenge thinking, foster debate and develop the ability of students to engage in critical discourse and rational thinking.
    • Seek ways of improving performance by reflecting on teaching design and delivery and obtaining and analysing feedback.
  • Research and scholarship
    • Reflect on practice and the development of own teaching and learning skills.
  • Expertise
    • Engage in continuous professional development.
    • Able to engage the interest and enthusiasm of students and inspire them to learn.
    • Develop familiarity with a variety of strategies to promote and assess learning.
LEVEL 3
  • Teaching and learning support
    • Design teaching material and deliver either across a range of modules or within a subject area.
    • Use appropriate teaching, learning support and assessment methods.
  • Research and scholarship
    • Engage in pedagogic, practitioner and other scholarly activities as required to support learning.
    • Develop and produce learning materials and disseminate the results of scholarly activity.
  • Expertise
    • Use a range of delivery techniques to enthuse and engage students.
LEVEL 4
  • Teaching and learning support
    • Develop and apply innovative and appropriate teaching techniques and material which create interest, understanding and enthusiasm amongst students.
    • Ensure that course design and delivery comply with the quality standards and regulations of the university and department.
  • Research and scholarship
    • Engage in pedagogic, practitioner and other scholarly activities.
    • Significantly contribute to the strategies for learning and teaching.
  • Expertise
    • Required to be externally recognised scholar or teacher.
LEVEL 5
  • Teaching and learning support
    • Contribute to teaching.
    • Encourage the development of innovative approaches to course delivery and ensure that teaching delivery achieves the educational standards of the department.
  • Research and scholarship
    • Lead the development and implementation of teaching and learning strategy.
    • Conduct research into learning and teaching methodologies and disseminate best practice within and outwith the Institution.

In contrast to the general descriptions of the Subject Centre for Education report and the Academic Role Profiles document, the Higher Education Academy published a report in 2007 about research and teaching, which advocated a more specific framework for evaluating teaching. This report attempted to blur the glass wall between research and teaching groups. However, the framework has not been widely adopted. For example, there is no evidence to suggest that such a framework was being considered for the HE Role Analysis (HERA) scheme.
Boyer (1990) and Glassick et al. (1996) offer a now widely respected typology of 'scholarship', for them a preferred term to the binary classification of 'teaching' and 'research'. They identify the scholarships of:
  • Discovery (advancing knowledge)
  • Integration (synthesising knowledge)
  • Service or engagement (advancing and applying knowledge)
  • Teaching (advancing and applying knowledge about how to teach and promote learning).
We have found the framework developed by Griffiths (2004) effective in supporting staff to examine both their current courses and institutional policies and practices and in adapting innovations from elsewhere. According to Griffiths teaching can be:
  • Research-led: where students learn about research findings, the curriculum content is dominated by staff or current disciplinary research interests, and some or much of the teaching may emphasise information transmission
  • Research-oriented: where students learn about research processes, the curriculum emphasises as much the processes by which knowledge is produced as knowledge that has been achieved, and staff try to engender a research ethos through their teaching; or
  • Research-based: where students learn as researchers, the curriculum is largely designed around inquiry-based activities, and the division of roles between teacher and student is minimised.
(p. 28)



Institutional Strategy and Teaching Methods

Teaching methods are a common feature of institutional strategy documents. The University of Portsmouth Learning, Teaching and Assessment Strategy 2008–2012 document (2008) has 2 specific references.
1.1.2   Further developing approaches that promote engagement and interaction, and develop independent and reflective learning, critical thinking and problem solving skills, specifically:
  • investigating and developing strategies to promote and monitor student engagement; and
  • continuing to develop and disseminate approaches to develop independent and reflective learning, critical thinking, information literacy and problem solving skills.
1.1.3   Providing increased opportunities for students to experience pedagogically informed approaches to blended learning, including elearning, through the use of a variety of modes, media and formats, specifically:
  • ensuring that curricula are designed with explicit opportunities for elearning and ideally assessment appropriate to a range of different learning styles and affording maximum learning opportunities;
  • ensuring that all students engage in elearning at each level/stage of study and that progression through the course is characterised by increasingly autonomous and self-directed learning;
  • defining student entitlement to access core, specialist/experimental elearning software and how this software will be supported in terms of physical and human resources; and
  • ensuring that all units have a virtual learning environment (VLE) presence and classifying all units according to their current and potential student elearning engagement, thus facilitating and directing staff training and development.

The Higher Education Funding Council for England published a report about learning and teaching strategies (2001) with examples of using teaching methods to support institutional objectives.
Case studies 4-7 illustrate the ways in which strategies are being used to implement one of the most common agendas: the development of students’ employability, skills and 'graduate attributes'. Mechanisms include:
  • an institution-wide definition of what these skills consist of and what graduate attributes should be expected
  • requirements to re-draft course documentation to specify which key skills, transferable skills or graduate attributes are being addressed, in every module
  • requirements to specify where and how these skills are being assessed. This may include rules about the minimum proportion of marks allocated explicitly to skills, and the minimum number or size of courses involving specific skills
  • requirements to provide a record of achievement, portfolio or profile which logs the acquisition of skills for each student
  • requirements to build work experience of a specified minimum duration into all degree programmes
  • mechanisms to recognise and give academic credit to work-based learning and off-campus learning of many kinds
  • timescales for meeting these requirements and implementing these mechanisms, with methods to check their achievement, for example through modified course review procedures
  • staff development, funding, provision of expert consultancy and other support to enable teachers to make these extensive and rapid changes.
(p. 13)

These excerpts provide a reminder that the institution is itself a participant and stakeholder in the development of teaching methods. Yet, there is a gap between the top down approach in institution strategies and the earlier role specifications for individual lecturers. Specifically, one problem is the absence of guidelines and negotiations about necessary resources to apply and develop particular teaching methods.


Innovation and Teaching Methods

There are a number of documents that analyzed the issue of teaching methods and innovation. In 2006, the Teaching Commission published a report with surprising recommendations. On page 3, the report cited data from 2 states in the US which indicated that the pay difference between teachers of the highest and lowest aptitude as measured by standardized test scores was 37% in 1963, decreasing to 19% in 1979 and to only 4% in 2000.
Policies based on a simple linear growth over time in teacher effectiveness should be reexamined. If student achievement gains are a school district’s primary focus, little evidence supports compensation packages that raise salaries equally for each year of service without regard to other considerations. A few years of experience makes a teacher more effective; after that it’s unclear.

This point was further emphasized on page 28.
Research suggests that because teachers of high aptitude generally earn no more than teachers of low aptitude, it may be more difficult for teaching to recruit and retain the talent it needs.

Page 4 of the report advocated a non-traditional view of teacher training.
Education courses taken before teaching have little impact on teacher effectiveness ... Pre-service education courses may help some aspiring teachers to be more effective than they would have been otherwise, but there is no evidence to support policies that bar individuals from the profession because they lack such coursework. Other credentials or experience may add just as much or more value.

The most important recommendation was made on page 8 about the importance of literacy for effective teaching.
More effective teachers will score relatively higher on tests of literacy ... Clearly a prospective teacher’s level of literacy, however measured, should be a primary consideration in the hiring process.

The difficulty of measuring the effectiveness of teaching methods and the resources required to foster innovation was emphasized in a report published by the University Planning Committee for Science and Engineering at Harvard University (2006).
While the University does have some science education pioneers in its ranks (E. Mazur and others), it has no clear process for implementing new educational ideas, and for assessing their effectiveness. Harvard has no clearly defined resources for technical curriculum development, and its framework for allocating teaching fellows and class support does not recognize the demands of curricular innovation (p. 21).

Two consultants from the management consultancy, McKinsey & Co., wrote a report based on their analysis of data from the OECD Programme for International Student Assessment. The importance of teaching methods played a central role in their thesis, as indicated by items 1 and 2 in the following excerpt from page 2.
There are many different ways to improve a school system, and the complexity of this task and the uncertainty about outcomes is rightly reflected in the international debate about how this should best be done. To find out why some schools succeed where others do not, we studied twenty-five of the world's school systems, including ten of the top performers. We examined what these high-performing school systems have in common and what tools they use to improve student outcomes.
The experiences of these top school systems suggests that three things matter most: 1) getting the right people to become teachers, 2) developing them into effective instructors and, 3) ensuring that the system is able to deliver the best possible instruction for every child.
These systems demonstrate that the best practices for achieving these three things work irrespective of the culture in which they are applied. They demonstrate that substantial improvement in outcomes is possible in a short period of time and that applying these best practices universally could have enormous impact in improving failing school systems, wherever they might be located.

Teaching methods as an indication of teacher quality had more impact than reductions in class size.
More importantly, every single one of the studies showed that within the range of class sizes typical in OECD countries, "variations in teacher quality completely dominate any effect of reduced class size." (p. 11)

The report identified a close relationship between the status of the teaching profession, the quality of teachers and the quality of the educational institution.
The quality of an education system cannot exceed the quality of its teachers (p. 16).
In all of the systems we studied, the ability of a school system to attract the right people into teaching is closely linked to the status of the profession (p. 23).

An important conclusion of the report is found on page 26, which made a direct link between teaching methods and the future of education.
The only way to improve outcomes is to improve instruction ... You could define the entire task of (a school) system in this way: its role is to ensure that when a teacher enters the classroom he or she has the materials available, along with the knowledge, the capability and the ambition to take one more child up to the standard today than she did yesterday. And again tomorrow.

These sources support the important role of teaching methods in developing innovative and high quality educational institutions. Furthermore, they emphasize the importance of the teacher as a determinant of success in educational systems.


Initial Proposal for a Learning Scorecard

A thorough analysis of the impact of teaching methods must consider 4 key participants, as indicated by the axes on the radar chart:
  1. Student
  2. Teacher
  3. Institution
  4. Community

The measurement of values along each axis is influenced by Maslow's hierarchy of needs theory:
  1. Knowledge
  2. Resources
  3. Satisfaction
Initial Proposal for a Learning Scorecard
Figure 2.   Initial Proposal for a Learning Scorecard

The community variable depends on the specific context of the teaching method. Often, in the context of public higher education, the community is represented by the government sponsor or funding agency. In other contexts, the community could refer to a subject specific community, e.g. electronic engineers.

The measurement of values reflects the evaluation of a teaching method against 3 sets of criteria. Knowledge criteria set includes the familiar measures of assessment results, learning outcomes, average unit marks and pass rates. It also includes the less common measures of teacher knowledge required to use a teaching method and the amount of knowledge that is shared within an institution or the wider community as a result of the application of a teaching method. For example, there is an absence of criteria to measure various forms of literacy for both teachers and students, e.g. language, computer and Internet literacy. Resources criteria set includes the familiar tools required for delivery, such as notepads for students, whiteboards for institutions, technology literacy for teachers and Internet access for the community. Additionally, resources must also consider less tangible requirements such as suitability for student learning styles, time for teachers to prepare learning resources and the institutions capacity to support students who need help or motivation. Satisfaction criteria set includes familiar measures such as annual student feedback surveys and less common factors such as the teacher's satisfaction when using a teaching method. The criteria sets will determine the units of measurement for each axis.

Similar to Maslow's hierarchy of needs theory, the measurement of values along each axis has an order of precedence. A teaching method must meet the knowledge criteria before it is evaluated against the resources criteria. Likewise, a teaching method must meet both the knowledge and resources criteria before it is evaluated against the satisfaction criteria. The novelty of using such an order of precedence is that it is an explicit attempt to prioritize knowledge objectives first, resources objectives second and satisfaction objectives last.

The next steps for developing the Learning Scorecard require:
  1. Selection of specific criteria set for each axis.
  2. Units of measurement for each axis that reflects the criteria set.
  3. Sample scorecards of specific teaching methods in order to analyze the suitability and effectiveness of the proposed criteria sets and units of measurement.


Conclusions

The brief survey of 11 reference documents re-affirmed the importance of teaching methods to the education profession. At the same time, the cited excerpts also illustrated the ambiguity about how to measure the impact of teaching methods. The variety of the documents reflected sources in the UK, USA and European (OECD) regions. In all sources, there was a consistent urgency for innovation to the development and application of teaching methods.

The initial proposal for a Learning Scorecard has immediate practical applications. For example, the prioritization of knowledge above satisfaction would suggest that results of student feedback surveys should only be evaluated in conjunction with the relevant assessment data. As another example, the separation of resources requirements for the student, teacher and institution would suggest that universities should conduct a more holistic analysis of Internet teaching tools such as virtual learning environments, blogs and wikis. Often, universities spend money and time to develop the necessary technology resources, but, fail to consider whether the teachers and students have the necessary hard (e.g. computers, Internet access, mobile phones) and soft (e.g. computer literacy, Internet literacy, analytical maturity, software literacy) resources necessary for teaching methods using the Internet to be successful. A third example involves the separation of satisfaction requirements for the student, teacher and institution. The National Student Survey in the UK and the end of semester student feedback surveys at the University of Portsmouth clearly show awareness for student satisfaction, but, there is a lack of consideration about satisfaction for the teacher or the institution.

In a knowledge economy where the rate of education is declining relative to previous generations (Vaitilingam, 2009), teaching methods must be more rigorously developed. Teaching takes place at home, schools, universities and workplaces. The Learning Scorecard attempts to combine values long associated with the act of teaching with explicit variables of measurements and order of precedence which have previously been absent. The framework is necessary to identify teaching methods in a manner which enables critical analysis and innovation.



References


 VisitorsPage views
Today11
Recent week1023
Recent month1979