Thanks to everyone who came to our latest Faculty Resource Workshop. Lindsay Schwarz (Pharmacy), Tony Frankino (Biology), Donna Pattison (Biology), and James Garson (Philosophy) spoke on “The Critical Mulitple Choice: Using Multiple Choice to Foster Critical Thinking,” and the audience included faculty members from every discipline. Some were interested in how to improve their multiple choice (MC) questions, but others were looking for new ways to use MC.
Lindsay Schwarz started things off with the question, where do we learn to write multiple choice questions? By taking MC tests, of course. Because so many of us think of MC as information recall only, we must have taken some bad tests. Lindsay, therefore, led us through best practices that link MC questions to course objectives, explaining how to move up Bloom’s taxonomy of learning domains. Lindsay recommends creating a test blueprint that maps out how much lecture time is devoted to each topic and then creating MC questions that mirror those time ratios. Even the types of questions can be based on the test blueprint: if she asks students to do critical thinking on three of the test topics, then those topics should use critical thinking questions. The point is that if instructors use good questions that connect to course lectures and objectives, then students will think that the course met its objectives.
Lindsay then went through methods of analyzing test data received from University Testing Services. The extended item analysis shows how the top, middle, and bottom performing students scored on each question. The analysis can help explain whether the question is valid and which distracters were easily omitted.
Lindsay ended by mentioning the advantages and disadvantages of MC questions. One disadvantage, she said, was that MC cannot be used to assess writing. The rest of the room began to discuss this idea with one participant saying that it is not appropriate to assess something that was not specifically covered in class lectures. Therefore, it is not okay to assess writing unless the class teaches writing. Therefore, it isn’t a disadvantage of MC that is doesn’t assess writing. Another participant claimed that assessment of writing is different from testing content, and that engineering assesses writing in many different classes that don’t actually include lectures on writing.
Tony Frankino then spoke about how he uses MC questions to foster critical thinking through his use of graphics to teach evolution. Instead of asking students to merely interpret or recall a formula, he asks questions that force students to recall, identify, interpret, and apply their knowledge. He uses CASA, as well, to switch response orders, but he retains distractor groupings. He reuses questions on the final exam, but they won’t see the same question twice because he has different versions of the same questions, each with different answers. He may have four different versions of each question. One participant asked about using different level questions from Bloom’s taxonomy for each topic, but Tony claimed that such a tactic penalizes the student twice if they don’t know a single answer or topic. Tony also uses “None of the above” for all of his MC questions, and versions of each question include “None of the above” as the correct answer. So it is always a choice, and it can’t be eliminated quickly.
Donna Pattison then led the group through a series of poorly written MC questions. The questions didn’t link to goals or objectives of the course, included opinions, used different structures or length for answers and distracters, or included clues such as article usage in the questions themselves. She then went through some best practices and mentioned why she doesn’t like the questions included in textbook question banks. Those questions don’t sound like the professor, so the students are at a disadvantage when they have been taught by someone other than the one that wrote the question.
Discussion after Donna’s workshop moved into ESL and how much professors should tailor their questions for these students. Donna mentioned that she encourages asking questions about words and language during the tests themselves, but another participant claimed that some students have been “beaten up” for asking questions, so they don’t. Tony Frankino then mentioned that one of the disadvantages of using CASA for his tests is that he can’t be there to answer questions. Then the discussion branched into whether professors give exams back to students, and there is no consensus there. Some do, and some don’t.
To wrap things up, James Garson presented briefly about his use of MC questions in a class on critical thinking. He asks students to make a diagram and then asks multiple choice questions about their diagram. In essence, he has made the rubric for the diagram into a question. Students are, in fact, self-reporting their diagram. He uses MC questions in way that suggests that the answer isn’t the point; instead, the critical thinking skill to arrive at the answer is what is important.
Lindsay then brought up MC questions with clickers where students are polled with correct answers, and another participant mentioned www.polleverywhere.com and Site44.com, which allow real-time polling, as well. Another option to allow for discussion of MC questions is http://peerwise.cs.auckland.ac.nz/, where students discuss questions asynchronously.
Overall, it was a great workshop, and everyone seemed interested in more workshops on writing multiple choice questions and innovative methods for their use. A future Faculty Forum will take up the topic again, we’re sure.
Chad A. Wilson
Please attend our Faculty Resource Workshop, April 18th, 1-2:30pm, on Developing Critical Thinking in the Multiple Choice FormatPosted: March 30, 2013
Center for Teaching Excellence Faculty Workshop
Thursday, April 18, from 1:00-2:30pm
M.D. Anderson Library, Room 306 (Faculty Senate Offices)
The Critical Multiple Choice: Developing Critical Thinking with Multiple Choice Format
This workshop provides a hands-on forum to help you design and evaluate multiple choice questions so that they engage your students’ critical thinking abilities.
We will cover basic principles of effective question design, as well as novel approaches that focus on fundamental pedagogical goals.
Please bring some of your own multiple choice questions to use in the workshop, and we will provide feedback and discussion on the best strategies to develop effective questions in your field.
For RSVPs or questions regarding this workshop, please contact Prof. Jim Garson at garson@Central.UH.EDU.
Our speakers today were Drs. Casey Due-Hackney (MCL), Donna Pattison (BIO), and Andrew Hamilton, our new Executive Director for Academic Innovation. The panel was moderated by Prof. Jim Garson (PHIL), who led off discussion with his own comments.
Garson noted that when he arrived here some time ago, he was surprised by the atmosphere surrounding instruction, which was dominated by cheap credits ($30/course) and “shopping for courses.” And he suggested that the student culture has not moved sufficiently beyond that attitude. He also noted that faculty attitudes then and now focused on aspects like students’ lack of preparation, which were essentially out of faculty control, rather than aspects of instruction that were in our power. And he concluded with a call for faculty and students to change the prevailing culture of low expectations, and build together a culture of high expectations and success. This cultural shift, however, would demand that both faculty and students embrace the risks and uncertainties of genuine learning.
Hamilton’s presentation, “Rethinking the Large Lecture” was organized into two parts: Problem and Solution. The large lecture has been in use ever since medieval lecturers found it the cheapest way to disseminate information from big, expensive, hand-produced books, as we can see below with the variously distracted and sleeping medieval students might suggest.
According to Hamilton, the large lecture represents a problem because it demands a level of engagement and private study time that contemporary students are simply unable or unwilling to provide. This, along with grade inflation and state and federal disinvestment in education, means that students will continue to fail in these important introductory courses in ever-growing numbers.
Hamilton did not propose any single “solution” to this mismatch of teaching approach to students, but instead offered a few design principles that would make success likelier than in teaching with conventional lecture models.
- restructuring time on task, so that students must do practice work and receive feedback while still in class, instead of passively listening to content-materials
- teaching intellectual skills, so that students can learn not just a particular content, but intellectual and critical skills (active reading, better writing, critical thinking, etc.) that could be developed further and used in other classes and contexts
- teach students how to collaborate and work together better, so they can accelerate the pace of their own learning
- support innovation structurally, so that systemic issues can be addressed in a similarly comprehensive way (this would include enhanced TA training, multi-use classroom space, new faculty orientation, and targeted faculty development)
Due-Hackney spoke from her 11 years’ experience in Classical Studies, and noted that her students had a very different schedule than she had when she was an undergraduate: they often work full-time, have family commitments, face significant commutes, and sometimes have responsibility for aging parents. They are also much less prepared than she was in school, particularly in their writing.
Due-Hackney made her own recommendations about “what works for her”:
- her enthusiasm for the material: this might seem obvious, but her evaluations have consistently shown that students enjoyed her classes because they could recognize her love of the material, and learned more because of her engagement with poems like the Iliad
- keep your class on students’ radar, by regularly assigning weekly, engagement-oriented assignments like brief, informal writing responses to their assigned readings, and providing equally regular, timely feedback to their writing, which could be done online or in-class
- student confidence grows with consistent encouragement and feedback, and with confidence comes better, more accomplished writing
- stand up and be willing to be seen as a human being who makes mistakes, sometimes needs to look up answers, or can learn things from your students
- acknowledge that the material is challenging, and that it should be hard for them to learn, and let them know that some of this material was difficult for you to learn, too
- Finally, even in the largest classes, try to model the kind of approachability, compassion, and flexibility you have periodically needed in your own life, career, and education. This can make an enormous difference for someone struggling with an issue in- or outside your course.
The third and final speaker, Dr. Pattison of Biology, outlined the comprehensive approach to student success that Biology Chair Dan Wells (a CTE board member) has directed in conjunction with a THECB grant, with the assistance of Pattison, Dr. Larry Williams, and Dr. Medrano, among others. The Biology student success program, which Pattison described as “tackling every problem all at once,” included the following features:
- tracking attendance through clicker questions
- increasing engagement through strategic use of clicker questions followed by think/pair/share activities for questions that caused confusion
- live demonstrations, with physical props, if necessary, to help students remember key concepts like meiosis/mitosis
- having undergrad TAs patrol the back of the lecture hall, and make sure students are not texting, on facebook, etc.
- arranging a dual syllabus, so that students scoring below 70% on a diagnostic must take a recitation-section version of the course with required attendance for recitations
- field trips and other activities, including a Biology talk/dinner that allows undergrads to hear current research topics discussed
- faculty workshops on pedagogy
Afterwards, the group discussed the problems of “disappearing students,” and the sometimes puzzling fact that students in difficulty will be able to finish classes they are engaged in, but fail the large lecture courses that are not motivating them. We also discussed the possibility of mandatory prerequisite checks, which are being instituted in a number of courses and departments, but which are complicated by our very large transfer population, whose courses sometimes require significant time to process. Finally, the group agreed that all of these changes would require structural changes that would encourage faculty development and continual reflection upon, and improvement of, one’s teaching and courses.
For further information, please contact Jim Garson at email@example.com.
See you there!
Thanks for Attending our 3/30/12 Workshop on the Future of Teaching in Higher Education (Summary of Discussions)Posted: March 31, 2012
Workshop March 30: What is the Future of Teaching in Higher Education?
Dave Mazella welcomed the participants to the final workshop of the Spring Semester, and urged everyone to share ideas they have about future workshops to be held in the 2012-2013 academic year.
Frank Holt opened the proceedings with a disclaimer. As an ancient historian, he might not be the best-equipped person to discuss the future. Still he offered a “water-cooler” viewpoint as a starting place for discussion. The vision he shared was somewhat bleak.
Pressures on higher education are growing to increase the ratio of college graduates in our population. There will be demands for more degrees, more access, more accountability, more relevance to employment, and less cost. (Pressures are already on us to create a B.A. that costs less than $10,000.) It is likely this will redefine the experience in institutions of higher learning.
There will be more part-time students, who will demand a self-service education with the speed and efficiencies of Ikea over Ethan Allen. There will be an increasing divide between research and teaching. The teaching function will be outsourced to for-profit retailers such as textbook and software vendors and online universities such as Phoenix University. Teaching will develop into a “Hollywood” model, where instructors become “stars” unaffiliated with any particular institution, and who offer their wares directly to the marketplace. Instructors will be expected to seek corporate sponsorship. The thousands of introductory courses now taught will eventually be serviced by a handful of star quality packages leased back to universities for use by their students. New entities will create institutions of higher learning with special slants on education, such as religious and government groups, corporations, and even political organizations. The ability to offer credentials such as the B.A, will end up in new hands. Brick and mortar universities will take on the role of the Grand Tour in Victorian England – a form of conspicuous consumption indulged in only by the very rich.
Holt ended his talk by saying that he hopes this vision is wrong, but processes that tend in these directions are already underway.
There was a lively discussion, which included the following points (not necessarily in chronological order).
President Khator’s call for UH to do a better job of servicing the needs of the City of Houston opens the question: what exactly does our city need?
The ability to offer credentials is beside the point. The real question is what exactly do our students need to do well in their lives? For-profit educators do a better job at focusing on education with demonstrable immediate payoffs. Maybe, but occupational training is not sufficient in an economy where needed job skills are rapidly evolving. What students need is training in general skills that can be applied to a fluid workplace.
At the moment, universities have a monopoly on credentials that blocks the development of independent global online educational sites. When this monopoly is broken, there will be a thunderbolt, and universities will be in deep trouble.
We need to learn how to partner with publishers and software vendors and ask for grants from them to create course materials. Yes but we may end up becoming a service organization for our sponsors. Like food contractors, we will become contractors for educational services. Well, we are already dependent on courseware vendors to some degree. Adoption of grading systems is being heavily marketed by vendors offering perks like conference travel.
What really matters for an institution is not its competitive pricing but its ability to compete in the race for institutional reputation. This is how we attract good faculty and good students. But limiting admission to only the best students restricts access. The new emphasis on accountability emphasizes fast and cheap education, but the students we have are not ready to handle college level work. Of 2,000 students entering our introductory math courses, half of them know next to nothing. This is a problem all the way up to the graduate level, where we are forced to dumb down what we teach. So called “efficiency” and lower DWIF rates are near impossible to achieve when students come poorly prepared. Is it my fault if I must award an F to a student who never showed up to class in the first place? Students display a YouTube philosophy where education means taking the path of least resistance to a degree. American culture does not value knowledge. On-demand education means to students that they do not need to learn it now. What they do not realize is their short-cut strategy does not pay off in learning the skills they badly need.
UH probably has as many students now as it can handle. In the future, the state is more likely to pay us for degrees granted than for class enrollment. As the total state contribution declines, it is in our interest to abandon the old model where we let in everyone to see who washes out. Instead, we might improve our faculty to student ratio, and concentrate on quality in admissions. On the other hand, it is not likely that we will be seeing fewer students, nor will they be significantly better prepared. We have to face that reality.
The challenge is to do more with fewer resources. This raises the problem of quality management, where we are way behind. The reason is that academic freedom limits he degree to which demands for improvement get converted into reform. All that is left is continuous quality improvement, which some of us engage in, but which is not widely enough practiced.
Outsourcing has already imposed a massive cost to our economy. Outsourcing in education will be a disaster. Yes, but we already engage in outsourcing to community colleges, since their credits automatically transfer to UH.
We need to make clear what it is that we do that is different from community colleges. One answer is that we generate knowledge, another is that we offer higher quality teaching where students are engaged in research and come to understand the importance of research. Excellence in undergraduate teaching can be part of what makes us unique. It is easy to sell the idea that quality of teaching in for-profit institutions is comparable to ours, but this is false. We should emphasize the skills (such as leaderships, and writing) that are hard to develop online. Some students are happy to remain anonymous in large classes. But students trained in small classes in basic skills are snapped up in the job market. There is no substitute for face-to-face teaching. The very fact that class attendance is so strongly correlated with good grades shows the value of face-to-face learning.
It is crucial to our mission that we do not decouple teaching from research, and that we make clear to the public the importance of research. But the reality is that the main value at UH is research and relatively meager resources are devoted to integrating research with teaching. Furthermore, the ratio of tenure track jobs is decreasing, so that more and more teaching is done by instructors without a research mission. Workload policies may take us in the same direction, by shifting the teaching load to faculty who are not productive enough researchers.
We could partner with high schools and community colleges to better prepare the students who come to us. The best students are ones who do not come directly from community colleges, but who have had a year or to get ready.
Most of the forces that will change higher education are out of our control, so let’s think about what we can control. We need communicate with people who will be making the crucial decisions, the people who are in control. The question was asked: how many people here belong to the AAUP? Almost no one did.
Another suggestion for what we can control is to refuse to cooperate with publishers who convert education into a commodity. Yes, but we need to learn how to use online instruction and other techniques in the best ways to teach large numbers of students well. We are not going to have the small classes that would be found in an ideal university.
The best argument to those in control is to cite events like this workshop. This shows our commitment to teaching, improves our reputation, and is something we can control.
Dave Mazella closed the session by saying that he hoped that the Center for Teaching Excellence (CTE) could take a more active role in communicating with administrators. As faculty, we tend to operate in splendid isolation. It is better to band together than suffer in silence. We are not alone. There are many universities like ours who are facing the same issues, so research on what they are doing can be helpful to us. You are invited to participate in the CTE blog, not to just to gripe about the issue, but to develop solutions to the problems that face us. We need to organize at the department and faculty levels to improve teaching at UH. Revising a course here or there is not enough. We need to work together in a committed and systematic way. If you are interested in helping by joining CTE, please get in touch, as we are opening up the membership of CTE.
DM postscript: On the subject of Distance Education and Hybrid Education models in the high-quality education of the future, I am going to quote myself from earlier blog posts, and ask everyone to consider Ed Glaeser‘s highly useful dictum about online interactions:
My rule of thumb, which I have taken from the economist and NYT columnist Ed Glaeser, is that online interactions do not substitute for interactions, but supplement them, in more or less-powerful ways. This is a view that seems to be getting corroborated in at least some of the research that I have seen. (See, for example, this 2003 study by some sociologists at Berkeley [Brint et al.]).
The study by Brint et al. linked above contains a description of online education that constitutes a warning, I think, for any institution that embraces online education as a means for cost-control and swelling enrollments:
Eventually a new “digital divide” will likely separate institutions that can afford to maintain control over the means of their online production and those that cannot. This divide will correspond to two educational cultures: one oriented primarily to training (“learning just the skills and content you need”) and the other to higher education in the traditional sense – with its stress on creative activity, capacity development, theory and methods, contexts for understanding, and critical approaches to existing knowledge.
It is imperative, then, to maintain quality in online programs as in all other forms of content delivery, if we wish to remain competitive with other institutions in reputational terms.
UPDATED: This Moody’s Higher Education Sector Outlook report for 2012 provides plenty of fodder for both optimists and pessimists. (Found in our Division of Administration and Finance’s remarkably useful Budget page)
Look particularly at frames 12-13, which talk about the “broken” Higher Ed Model, and the declines of governmental support.
Having highlighted all the challenges we face, I still think that questions of quality and reputation, especially in regard to instruction, are paramount for institutions like ours. See, for example, this quote: “Student demand and net tuition growth remain strongest for those that are most affordable, reputable, and programmatically diversified” (frame 7). The question becomes how, and for whom, we define “quality.”