Thank you for attending the 4/18/13 Faculty Resource Workshop on “The Critical Multiple Choice”!

Fac Res Mult Choice 4-18-13 c

Thanks to everyone who came to our latest Faculty Resource Workshop. Lindsay Schwarz (Pharmacy), Tony Frankino (Biology), Donna Pattison (Biology), and James Garson (Philosophy) spoke on “The Critical Mulitple Choice: Using Multiple Choice to Foster Critical Thinking,” and the audience included faculty members from every discipline. Some were interested in how to improve their multiple choice (MC) questions, but others were looking for new ways to use MC.

Fac Res Multiple Choice 4-18-13 a

Lindsay Schwarz started things off with the question, where do we learn to write multiple choice questions? By taking MC tests, of course. Because so many of us think of MC as information recall only, we must have taken some bad tests. Lindsay, therefore, led us through best practices that link MC questions to course objectives, explaining how to move up Bloom’s taxonomy of learning domains. Lindsay recommends creating a test blueprint that maps out how much lecture time is devoted to each topic and then creating MC questions that mirror those time ratios. Even the types of questions can be based on the test blueprint: if she asks students to do critical thinking on three of the test topics, then those topics should use critical thinking questions. The point is that if instructors use good questions that connect to course lectures and objectives, then students will think that the course met its objectives.

Lindsay then went through methods of analyzing test data received from University Testing Services. The extended item analysis shows how the top, middle, and bottom performing students scored on each question. The analysis can help explain whether the question is valid and which distracters were easily omitted.

Lindsay ended by mentioning the advantages and disadvantages of MC questions. One disadvantage, she said, was that MC cannot be used to assess writing. The rest of the room began to discuss this idea with one participant saying that it is not appropriate to assess something that was not specifically covered in class lectures. Therefore, it is not okay to assess writing unless the class teaches writing. Therefore, it isn’t a disadvantage of MC that is doesn’t assess writing. Another participant claimed that assessment of writing is different from testing content, and that engineering assesses writing in many different classes that don’t actually include lectures on writing.

Fac Res Mult Choice 4-18-13 e

Tony Frankino then spoke about how he uses MC questions to foster critical thinking through his use of graphics to teach evolution. Instead of asking students to merely interpret or recall a formula, he asks questions that force students to recall, identify, interpret, and apply their knowledge. He uses CASA, as well, to switch response orders, but he retains distractor groupings. He reuses questions on the final exam, but they won’t see the same question twice because he has different versions of the same questions, each with different answers. He may have four different versions of each question. One participant asked about using different level questions from Bloom’s taxonomy for each topic, but Tony claimed that such a tactic penalizes the student twice if they don’t know a single answer or topic. Tony also uses “None of the above” for all of his MC questions, and versions of each question include “None of the above” as the correct answer. So it is always a choice, and it can’t be eliminated quickly.

Donna Pattison then led the group through a series of poorly written MC questions. The questions didn’t link to goals or objectives of the course, included opinions, used different structures or length for answers and distracters, or included clues such as article usage in the questions themselves. She then went through some best practices and mentioned why she doesn’t like the questions included in textbook question banks. Those questions don’t sound like the professor, so the students are at a disadvantage when they have been taught by someone other than the one that wrote the question.

Discussion after Donna’s workshop moved into ESL and how much professors should tailor their questions for these students. Donna mentioned that she encourages asking questions about words and language during the tests themselves, but another participant claimed that some students have been “beaten up” for asking questions, so they don’t. Tony Frankino then mentioned that one of the disadvantages of using CASA for his tests is that he can’t be there to answer questions. Then the discussion branched into whether professors give exams back to students, and there is no consensus there. Some do, and some don’t.

Fac Res Mult Choice 4-18-13 d

To wrap things up, James Garson presented briefly about his use of MC questions in a class on critical thinking. He asks students to make a diagram and then asks multiple choice questions about their diagram. In essence, he has made the rubric for the diagram into a question. Students are, in fact, self-reporting their diagram. He uses MC questions in way that suggests that the answer isn’t the point; instead, the critical thinking skill to arrive at the answer is what is important.

Lindsay then brought up MC questions with clickers where students are polled with correct answers, and another participant mentioned and, which allow real-time polling, as well. Another option to allow for discussion of MC questions is, where students discuss questions asynchronously.

Overall, it was a great workshop, and everyone seemed interested in more workshops on writing multiple choice questions and innovative methods for their use. A future Faculty Forum will take up the topic again, we’re sure.

Chad A. Wilson

Please attend our Faculty Resource Workshop, April 18th, 1-2:30pm, on Developing Critical Thinking in the Multiple Choice Format

Center for Teaching Excellence Faculty Workshop

Event Date:
Thursday, April 18, from 1:00-2:30pm

Event Location:
M.D. Anderson Library, Room 306 (Faculty Senate Offices)


The Critical Multiple Choice:  Developing Critical Thinking with Multiple Choice Format
This workshop provides a hands-on forum to help you design and evaluate multiple choice questions so that they engage your students’ critical thinking abilities.
We will cover basic principles of effective question design, as well as novel approaches that focus on fundamental pedagogical goals.

Please bring some of your own multiple choice questions to use in the workshop, and we will provide feedback and discussion on the best strategies to develop effective questions in your field.

For RSVPs or questions regarding this workshop, please contact Prof. Jim Garson at garson@Central.UH.EDU.


Thinking Critically About Critical Thinking: Breakout Session Notes, Ppt, and Summary, Oct. 14, 2011 (Jim Garson)

[image from,+learning+styles+and+thinking+organisers%5D

Break Out Session:  Thinking Critically About Critical Thinking


Richard Armstrong, Associate Professor, Modern and Classical Languages, and Fellow in the Honors Program

Rebecca Forrest, Instructional Assistant Professor, Physics

Lonny Hoffman, George Butler Research Professor of Law, UH Law Center

Maria Solino, Associate Professor of Spanish Literature and Film

Department of Hispanic Studies

Moderator: James Garson, Full Professor, Department of Philosophy


The panel began with an exercise asking all the participants to write out answers  two questions (exercise and other ppt visual materials can be found here):

1. Give an example of an intellectual skill that qualifies as an example of critical thinking.

2. How do you help students master that intellectual skill?

Three or four answers to the first question were briefly discussed, and as expected, they illustrated very different intellectual abilities, among them, artistic creativity, seeing the value in another’s point of view, problem solving, and the ability to distinguish cause from mere correlation.

Then the panelists introduced themselves and explained how their own interests are connected to critical thinking.

At that point a brief, and then a very wordy, definition of critical thinking was offered as a way to try to focus on exactly what we are seeking when we hope to promote critical thinking.  Almost every cognitive skill other than rote memorization has been classified as critical thinking by someone, and even memorization drills are classified as critical thinking in some websites. So the puzzle over what critical thinking is remains.

However, a discussion of specific critical thinking skills and how they can be promoted brought out some interesting themes related to aims in the classroom:

*  to welcome asking and answering questions of the form: “What if?”, “How do we know?” and “Who cares?”

*  to have the flexibility to suspend ones own beliefs, and to see the world from another’s point of view

*  to be creative and voracious readers, with nuanced appreciation for ambiguity and novel context

*  to be endlessly curious

*  to welcome and to profit from being wrong

*  to be able to evaluate the accuracy of sources of information, for example information found online

*  to slow down, to see the wider view, to be circumspect and thoughtful

A common thread in many of these ideas is metacognition, or executive functioning, the ability to consciously evaluate one’s own learning process.  Organizing classroom goals around opportunities to practice such talents is a great way to improve teaching effectiveness, and to promote valuable higher level thinking skills.

Jim Garson