Home     Mosaic     CLIP     Octave     R Project     Processing     Contact  
Search
<http://cnfolio.com/LearningScorecard>
This report is supported by a grant from the Faculty Learning and Teaching Committee, Faculty of Technology, University of Portsmouth
Updated November 1, 2009

The Learning Scorecard

by Chi Nguyen



Objective

An important tenet of the Chinese philosophy tradition, Confucianism, is that everything needs to be given correct names. In the context of education, knowledge of methods and techniques for teaching has implicit impact on government policy, course development, compensation negotiations, student learning experience, continuing professional development and the public view of the teaching profession. This research investigated the perceptions of teaching methods within government, university and professional documents. The objective was an initial set of criteria to analyze and describe teaching methods, tentatively called a Learning Scorecard. The continued development of such a Learning Scorecard could assist the accurate identification ("naming") of teaching methods and critical analysis of the act of teaching.

Keywords: learning scorecard, teaching methods, impact of teaching, dimensions of teaching, order of precedence, units of measurement, pedagogy.


Elements of a Learning Scorecard

In both science and management, there is a common belief that measurement of an activity is necessary for analysis and improvement. The Learning Scorecard emphasizes 2 interdependent relationships. The first relationship reflects the necessity of identifying dimensions and units suitable for measuring teaching methods before attempting to assess their impact. The second relationship suggests that assessment of the impact of teaching methods is constrained or influenced by choices made about measurement dimensions and units. More importantly, accepted valuation of the impact for specific teaching methods directly affect their development and adoption.
Elements of a Learning Scorecard
Figure 1.   Elements of a Learning Scorecard

As an example, let's consider the teaching method of a teacher standing to lecture a class of 100 students who are expected to listen, ask questions and take notes. Furthermore, let's prescribe that the content of the lecture session is conducted on a whiteboard and is not distributed to students afterwards. But, this is not sufficient to specify a teaching method! Research published by the Mid-continent Research for Education and Learning (2000) described specific categories of instructional strategies which could be incorporated into a lecture session.
Categories of instructional strategies that strongly affect student achievement:
  1. Identifying similarities and differences
  2. Summarizing and note taking
  3. Reinforcing effort and providing recognition
  4. Homework and practice
  5. Nonlinguistic representations
  6. Cooperative learning
  7. Setting goals and providing feedback
  8. Generating and testing hypotheses
  9. Activating prior knowledge
(p. 4)
The dimensions and units for measuring the impact of a teaching method improve as it is specified in more details. But, the university and government documents reviewed for this report has a total absence of such details when making reference to teaching methods. Pedagogical strategy and method of delivery must be considered to determine suitable dimensions and units for measuring the impact of teaching methods.


Teaching Methods as Professional Skills

The Higher Education Academy Subject Centre for Education (2005) published a report by Bart McGettrick, Professor of Education at the University of Glasgow, which considered standards for the teaching profession. Item 2 of the list below made an important distinction between professional knowledge as compared to professional skills.
Within the past seven years the Quality Assurance Agency has spearheaded the publication of benchmarks for teaching. In the statements of benchmarks for the profession of teaching itself, there has been consideration given to three main areas. These are:
(p. 8)

This distinction is clarified on page 10, which specified conceptual frameworks as a form of professional knowledge.
It is the expectation of professionals that they will undertake professional action which is rooted in one or more conceptual frameworks which are recognised and understood by fellow professionals. Their actions are therefore based on principles and knowledge that is authenticated by professional peers.

Later on page 17, the report suggested that there was a difference between teaching standards, academic awards and professional awards. Although the report indicated that a framework was required to reconcile the differences, it did not attempt to suggest such a framework!
The Higher Education Academy will have to align academic awards and professional awards. This also requires a framework to be established. The recognition of teaching standards at:
may be a suitably simple basis on which to begin. In time a greater degree of sophistication may be helpful, but a graded system of a simple kind would be useful at the beginning. These are indicated in Appendix 1 (p27), showing how they may relate to increased competence in teaching. This increase in competence is not only a matter of experience, but of effectiveness across different kinds of teaching in higher education.

The most direct reflection of teaching methods as professional skills is perhaps the official university guidelines for academic staff grading and pay. As of October 31, 2009, the Freedom of Information page (n.d.) on staff pay at the University of Portsmouth indicated that the guidelines are under review and not available!
Staff pay and grading structures
Information on the HE Role Analysis (HERA) scheme

The only available alternative is the University of Portsmouth Academic Role Profiles document (2005). Below is an extract of selected requirements for professional levels 1 to 5, where 3 is a senior lecturer and 5 is a professor. A similar distinction has been made between professional knowledge ("Expertise") and professional skills ("Teaching and learning support" and "Research and scholarship"). Yet, this attempt at a comprehensive specification provides no help or guidance on how to apply the criteria in a consistent manner across all academic staff and all subject areas. This ambiguity is a source of disparity within the university and places university management in a disproportional position of advantage as "interpreters" of the specifications. Furthermore, the current document has no indication of relative priority or weighting amongst the criteria. If there is no priority, then all criteria are equally important, or worse, none are important. Yet, this is the main document for managing the professional career progression of academic staff.
ROLE PROFILES: TEACHING AND SCHOLARSHIP
LEVEL 1
LEVEL 2
LEVEL 3
LEVEL 4
LEVEL 5

In contrast to the general descriptions of the Subject Centre for Education report and the Academic Role Profiles document, the Higher Education Academy published a report in 2007 about research and teaching, which advocated a more specific framework for evaluating teaching. This report attempted to blur the glass wall between research and teaching groups. However, the framework has not been widely adopted. For example, there is no evidence to suggest that such a framework was being considered for the HE Role Analysis (HERA) scheme.
Boyer (1990) and Glassick et al. (1996) offer a now widely respected typology of 'scholarship', for them a preferred term to the binary classification of 'teaching' and 'research'. They identify the scholarships of:
We have found the framework developed by Griffiths (2004) effective in supporting staff to examine both their current courses and institutional policies and practices and in adapting innovations from elsewhere. According to Griffiths teaching can be:
(p. 28)



Institutional Strategy and Teaching Methods

Teaching methods are a common feature of institutional strategy documents. The University of Portsmouth Learning, Teaching and Assessment Strategy 2008–2012 document (2008) has 2 specific references.
1.1.2   Further developing approaches that promote engagement and interaction, and develop independent and reflective learning, critical thinking and problem solving skills, specifically:
1.1.3   Providing increased opportunities for students to experience pedagogically informed approaches to blended learning, including elearning, through the use of a variety of modes, media and formats, specifically:

The Higher Education Funding Council for England published a report about learning and teaching strategies (2001) with examples of using teaching methods to support institutional objectives.
Case studies 4-7 illustrate the ways in which strategies are being used to implement one of the most common agendas: the development of students’ employability, skills and 'graduate attributes'. Mechanisms include:
(p. 13)

These excerpts provide a reminder that the institution is itself a participant and stakeholder in the development of teaching methods. Yet, there is a gap between the top down approach in institution strategies and the earlier role specifications for individual lecturers. Specifically, one problem is the absence of guidelines and negotiations about necessary resources to apply and develop particular teaching methods.


Innovation and Teaching Methods

There are a number of documents that analyzed the issue of teaching methods and innovation. In 2006, the Teaching Commission published a report with surprising recommendations. On page 3, the report cited data from 2 states in the US which indicated that the pay difference between teachers of the highest and lowest aptitude as measured by standardized test scores was 37% in 1963, decreasing to 19% in 1979 and to only 4% in 2000.
Policies based on a simple linear growth over time in teacher effectiveness should be reexamined. If student achievement gains are a school district’s primary focus, little evidence supports compensation packages that raise salaries equally for each year of service without regard to other considerations. A few years of experience makes a teacher more effective; after that it’s unclear.

This point was further emphasized on page 28.
Research suggests that because teachers of high aptitude generally earn no more than teachers of low aptitude, it may be more difficult for teaching to recruit and retain the talent it needs.

Page 4 of the report advocated a non-traditional view of teacher training.
Education courses taken before teaching have little impact on teacher effectiveness ... Pre-service education courses may help some aspiring teachers to be more effective than they would have been otherwise, but there is no evidence to support policies that bar individuals from the profession because they lack such coursework. Other credentials or experience may add just as much or more value.

The most important recommendation was made on page 8 about the importance of literacy for effective teaching.
More effective teachers will score relatively higher on tests of literacy ... Clearly a prospective teacher’s level of literacy, however measured, should be a primary consideration in the hiring process.

The difficulty of measuring the effectiveness of teaching methods and the resources required to foster innovation was emphasized in a report published by the University Planning Committee for Science and Engineering at Harvard University (2006).
While the University does have some science education pioneers in its ranks (E. Mazur and others), it has no clear process for implementing new educational ideas, and for assessing their effectiveness. Harvard has no clearly defined resources for technical curriculum development, and its framework for allocating teaching fellows and class support does not recognize the demands of curricular innovation (p. 21).

Two consultants from the management consultancy, McKinsey & Co., wrote a report based on their analysis of data from the OECD Programme for International Student Assessment. The importance of teaching methods played a central role in their thesis, as indicated by items 1 and 2 in the following excerpt from page 2.
There are many different ways to improve a school system, and the complexity of this task and the uncertainty about outcomes is rightly reflected in the international debate about how this should best be done. To find out why some schools succeed where others do not, we studied twenty-five of the world's school systems, including ten of the top performers. We examined what these high-performing school systems have in common and what tools they use to improve student outcomes.
The experiences of these top school systems suggests that three things matter most: 1) getting the right people to become teachers, 2) developing them into effective instructors and, 3) ensuring that the system is able to deliver the best possible instruction for every child.
These systems demonstrate that the best practices for achieving these three things work irrespective of the culture in which they are applied. They demonstrate that substantial improvement in outcomes is possible in a short period of time and that applying these best practices universally could have enormous impact in improving failing school systems, wherever they might be located.

Teaching methods as an indication of teacher quality had more impact than reductions in class size.
More importantly, every single one of the studies showed that within the range of class sizes typical in OECD countries, "variations in teacher quality completely dominate any effect of reduced class size." (p. 11)

The report identified a close relationship between the status of the teaching profession, the quality of teachers and the quality of the educational institution.
The quality of an education system cannot exceed the quality of its teachers (p. 16).
In all of the systems we studied, the ability of a school system to attract the right people into teaching is closely linked to the status of the profession (p. 23).

An important conclusion of the report is found on page 26, which made a direct link between teaching methods and the future of education.
The only way to improve outcomes is to improve instruction ... You could define the entire task of (a school) system in this way: its role is to ensure that when a teacher enters the classroom he or she has the materials available, along with the knowledge, the capability and the ambition to take one more child up to the standard today than she did yesterday. And again tomorrow.

These sources support the important role of teaching methods in developing innovative and high quality educational institutions. Furthermore, they emphasize the importance of the teacher as a determinant of success in educational systems.


Initial Proposal for a Learning Scorecard

A thorough analysis of the impact of teaching methods must consider 4 key participants, as indicated by the axes on the radar chart:
  1. Student
  2. Teacher
  3. Institution
  4. Community

The measurement of values along each axis is influenced by Maslow's hierarchy of needs theory:
  1. Knowledge
  2. Resources
  3. Satisfaction
Initial Proposal for a Learning Scorecard
Figure 2.   Initial Proposal for a Learning Scorecard

The community variable depends on the specific context of the teaching method. Often, in the context of public higher education, the community is represented by the government sponsor or funding agency. In other contexts, the community could refer to a subject specific community, e.g. electronic engineers.

The measurement of values reflects the evaluation of a teaching method against 3 sets of criteria. Knowledge criteria set includes the familiar measures of assessment results, learning outcomes, average unit marks and pass rates. It also includes the less common measures of teacher knowledge required to use a teaching method and the amount of knowledge that is shared within an institution or the wider community as a result of the application of a teaching method. For example, there is an absence of criteria to measure various forms of literacy for both teachers and students, e.g. language, computer and Internet literacy. Resources criteria set includes the familiar tools required for delivery, such as notepads for students, whiteboards for institutions, technology literacy for teachers and Internet access for the community. Additionally, resources must also consider less tangible requirements such as suitability for student learning styles, time for teachers to prepare learning resources and the institutions capacity to support students who need help or motivation. Satisfaction criteria set includes familiar measures such as annual student feedback surveys and less common factors such as the teacher's satisfaction when using a teaching method. The criteria sets will determine the units of measurement for each axis.

Similar to Maslow's hierarchy of needs theory, the measurement of values along each axis has an order of precedence. A teaching method must meet the knowledge criteria before it is evaluated against the resources criteria. Likewise, a teaching method must meet both the knowledge and resources criteria before it is evaluated against the satisfaction criteria. The novelty of using such an order of precedence is that it is an explicit attempt to prioritize knowledge objectives first, resources objectives second and satisfaction objectives last.

The next steps for developing the Learning Scorecard require:
  1. Selection of specific criteria set for each axis.
  2. Units of measurement for each axis that reflects the criteria set.
  3. Sample scorecards of specific teaching methods in order to analyze the suitability and effectiveness of the proposed criteria sets and units of measurement.


Conclusions

The brief survey of 11 reference documents re-affirmed the importance of teaching methods to the education profession. At the same time, the cited excerpts also illustrated the ambiguity about how to measure the impact of teaching methods. The variety of the documents reflected sources in the UK, USA and European (OECD) regions. In all sources, there was a consistent urgency for innovation to the development and application of teaching methods.

The initial proposal for a Learning Scorecard has immediate practical applications. For example, the prioritization of knowledge above satisfaction would suggest that results of student feedback surveys should only be evaluated in conjunction with the relevant assessment data. As another example, the separation of resources requirements for the student, teacher and institution would suggest that universities should conduct a more holistic analysis of Internet teaching tools such as virtual learning environments, blogs and wikis. Often, universities spend money and time to develop the necessary technology resources, but, fail to consider whether the teachers and students have the necessary hard (e.g. computers, Internet access, mobile phones) and soft (e.g. computer literacy, Internet literacy, analytical maturity, software literacy) resources necessary for teaching methods using the Internet to be successful. A third example involves the separation of satisfaction requirements for the student, teacher and institution. The National Student Survey in the UK and the end of semester student feedback surveys at the University of Portsmouth clearly show awareness for student satisfaction, but, there is a lack of consideration about satisfaction for the teacher or the institution.

In a knowledge economy where the rate of education is declining relative to previous generations (Vaitilingam, 2009), teaching methods must be more rigorously developed. Teaching takes place at home, schools, universities and workplaces. The Learning Scorecard attempts to combine values long associated with the act of teaching with explicit variables of measurements and order of precedence which have previously been absent. The framework is necessary to identify teaching methods in a manner which enables critical analysis and innovation.



References

This website uses wikkawiki.org software.