Measuring and improving Pedagogical Content Knowledge of student assistants in introductory physics classes

Developing PCK-Q questions in other STEM contexts

We applided the procedure of PCK-Q instrument development with elementary science Pre-service Teachers (PSTs), which justifies the feasiblity of applying the following six steps in other STEM contexts for novice teacher preparation.

Step 1: Collect classroom videos and analyze PSTs' questioning performance using the coding scheme.
Step 2: Identify representative interactive vignettes from videos and adjust them into free-response questions.
Step 3: Evaluate the correlation between PSTs' questioning practices measured by videos and their PCK-Q measured by free-response questions .
Step 4: Apply free-response questions with a relatively large sample of PSTs to summarize the patterns of common responses from the PSTs.
Step 5: Use the common responses as options for Likert-scale questions that can be automatically graded. Use PSTs' PCK-Q measured by free-response questions as the baseline to derive empirical equations for quantifying PCK-Q from ratings.
Step 6: Examine and revise the empirical equations in further empirical studies.

Elementary science: video coding

Vignettes of different levels in a given situation

Lesson objective: Students will understand the difference between reflection and refraction.
Episode: Students are conducting two labs about reflection and refraction. In the first lab, they shine torch light on a mirror. In the second lab, they place a pencil in a half-full glass of water.
Vignette The teacher approaches a group of three students who have finished the first lab and checks their findings.

Qa vignette: pq-pq-pq-di-gq-pq-tr-gq-tr-pq-tr-di
Teacher: What happened when you shined the torch light on the mirror? (pq)
Students: It bounced off.
Teacher: How did you know that? (pq)
Students: Because it was reflection.
Teacher: What is reflection? (pq)
Students: Bounce-off is reflection. (sign of conceptual difficulty)
Teacher: Let’s break apart this process a little. (di) Was there anything on the ceiling when you shined the torch light on the mirror? (gq)
Students: Yes, there was a light spot on the ceiling.
Teacher: How did the light get there? (pq)
Students: The light hit the mirror, got bounced.
Teacher: Exactly. (tr) Without the mirror, would the torch light reach the ceiling it was shined horizontally? (gq)
Students: No, because light moves straight.
Teacher: That’s right. (tr) Now do you know what reflection means? (pq)
Students: Yes, it means light changes direction, because it hits the mirror, gets bounced.
Teacher: Good thinking (tr). Now you can go on with the next lab. (di)

Qb vignette: pq-pq-pq-di-pq-le-cq-le-cq-gq-gq-tr-di
Teacher: What happened when you shined the torch light on the mirror? (pq)
Students: It bounced off.
Teacher: How did you know that? (pq)
Students: Because it was reflection.
Teacher: What is reflection? (pq)
Students: Because bounce-off is reflection. (sign of conceptual difficulty)
Teacher: Let’s break apart this process a little. (di) How does light travel? (pq)
Student: Like spread out.
Teacher: Light travels in a straight line, it cannot change direction on its own. (le) Does that make sense? (cq)
Student: Yes
Teacher: So you saw a light spot on the ceiling (le), right? (cq) Without the mirror, would the torchlight reach the ceiling when it was shined horizontally? (gq)
Students: No.
Teacher: Then what is the function of the mirror in this lab? (gq)
Students: Change the direction of light.
Teacher: Yes, that’s what bounce-off is. (tr) Now you can go on with the next lab. (di)

Qe vignette: pq-pq-pq-di-pq-le-cq-le-cq-gq-gq-tr-er-di
Teacher: What happened when you shined the torch light on the mirror? (pq)
Students: It bounced off.
Teacher: How did you know that? (pq)
Students: Because it was reflection.
Teacher: What is reflection? (pq)
Students: Because bounce-off is reflection. (sign of conceptual difficulty)
Teacher: Let’s break apart this process a little. (di) How does light travel? (pq)
Student: Like spread out.
Teacher: Light travels in a straight line, it cannot change direction on its own. (le) Does that make sense? (cq)
Student: Yes
Teacher: So you saw a light spot on the ceiling (le), right? (cq) Without the mirror, would the torchlight reach the ceiling when it was shined horizontally? (gq)
Students: No.
Teacher: Then what is the function of the mirror in this lab? (gq)
Students: Change the direction of light.
Teacher: Yes, that’s what bounce-off is. (tr) Only smooth surfaces like a mirror can bounce off light. (er) Now you can go on with the next lab. (di)

Da vignette: pq-pq-pq-le*-cq-le-gq-le*-pq-tr-di
Teacher: What happened when you shined the torch light on the mirror? (pq)
Students: It bounced off.
Teacher: How did you know that? (pq)
Students: Because it was reflection.
Teacher: What is reflection? (pq)
Students: Because bounce-off is reflection. (sign of conceptual difficulty)
Teacher: Reflection is the change of the direction of light when it hits a surface, like the mirror (le*). Did you see a light spot on the ceiling? (cq)
Student: Yes.
Teacher: Light travels in a straight line, it cannot change the direction on its own. (le) Without the mirror, would you be able to see that? (gq)
Student: No.
Teacher: So the mirror reflects the light and changes its direction. That’s what reflection is. (le*) What else could reflect light? (pq)
Students: The screen of an iPad.
Teacher: Exactly (tr). Now you can go on with the next lab. (di)

Db vignette: pq-pq-pq-le*-cq-di
Teacher: What happened when you shined the torch light on the mirror? (pq)
Students: It bounced off.
Teacher: How did you know that? (pq)
Students: Because it was reflection.
Teacher: What is reflection? (pq)
Students: Because bounce-off is reflection. (sign of conceptual difficulty)
Teacher: Reflection is the change of the direction of light when it hits a surface, like the mirror. Light travels in a straight line. It cannot change direction on its own. When light hits the mirror, the mirror reflects the light and changes its direction. Another example of reflection is when you see your image in a lake. (le*) Does it make sense? (cq)
Students: Yes.
Teacher: Now you can go on with the next lab. (di)

De vignette: pq-pq-pq-le*-er-cq-di
Teacher: What happened when you shined the torch light on the mirror? (pq)
Students: It bounced off.
Teacher: How did you know that? (pq)
Students: Because it was reflection.
Teacher: What is reflection? (pq)
Students: Because bounce-off is reflection. (sign of conceptual difficulty)
Teacher: Reflection is the change of the direction of light when it hits a surface, like the mirror. Light travels in a straight line. It cannot change direction on its own. When light hits the mirror, the mirror reflects the light and changes its direction. Another example of reflection is when you see your image in a lake. (le*) Reflection only happens on smooth surfaces (er). Does it make sense? (cq)
Students: Yes.
Teacher: Now you can go on with the next lab. (di)

N vignette: pq-pq-tr-pq-pq-pq-tr-di
Teacher: What happened when you shined the torch light on the mirror? (pq)
Student: It bounced off.
Teacher: How did you know that? (pq)
Student: There was light on the ceiling.
Teacher: Good observation. (tr) What do you think bounced off the light? (pq)
Student: The mirror.
Teacher: Is that reflection or refraction? (pq)
Student: Reflection.
Teacher: Why? (pq)
Student: Because light bounced off, that’s reflection. (sign of conceptual difficulty)
Teacher: Good thinking. (tr) Now you can go on with the next lab. (di)

Elementary science: free-response questions

Sample questions

Context: You are teaching a new unit about magnets in the first grade. You have prepared several items for exploration, including paper clips, nails, aluminum cans, pencils, crayons, and books. Students work in groups to examine which items can be attracted by a magnet. You approach a group of students and have a conversation with them as shown below:

You: Which items do you think can be attracted by the magnet?
Students: Paper clips, nails, and cans.
You: Why?
Students: Because they are metal, magnets attract metal.
You: Do you think magnets attract all metals?
Students: Yes.

a. What can you conclude from the information provided about the students’ scientific content knowledge, both the strengths (i.e., what they know) and difficulties (i.e., what they do not know)?
b. How would you respond to the students? Please use direct quote of what you would say. What is(are) your purpose(s) behind that response?

Elementary science: likert-scale questions

Sample questions

Context: You are teaching a new unit about magnets in the first grade. You have prepared several items for exploration, including paper clips, nails, aluminum cans, pencils, crayons, and books. Students work in groups to examine which items can be attracted by a magnet. You approach a group of students and have a conversation with them as shown below:

You: Which items do you think can be attracted by the magnet?
Students: Paper clips, nails, and cans.
You: Why?
Students: Because they are metal, magnets attract metal.
You: Do you think magnets attract all metals?
Students: Yes.

How would you respond to the students at this point? On a scale from 1(very unlikely), 2(unlikely), 3(neutral), 4(likely) to 5(very likely), please rate the likelihood of you giving each of the following responses.

A. Is a penny metal? Could a magnet attract a penny?
B. How does a magnet work?
C. You are correct. What makes you think so?
D. Magnets cannot attract all metals.
E. All metals are magnetic.

	CATEGORY	DEFINATION
A	Effective guiding question	An effective guiding question directly addressing students' difficulty.
B	General definition question	An overarching question regarding the definition of the key concept involved in a scenario without specific directions.
C	Circumventing difficulty question	A question prompting students to extend their thinking without resolving their difficulty yet.
D	Correction through direct instruction	Addressing students' difficulty by directly conveying the information that students need.
E	Erroneous statements	Direct instruction that conveys erroneous information to students.

Elmentary science: Emperical equations for Likert QUESTIONS

The following empirical equations were derived from one case study since they generated O-C-S-I that highly correlated with the same components measured by free-response questions. These equations need to be examined in further empirical studies.

Orientation(O) = (RateA-3) – (RateD-3)¹
Knowledge of curriculum (C) = (RateA-3) – (RateC-3) – (RateE-3)
Knowledge of students (S) = – (RateE-3) ²
Knowledge of Instructional strategies (I) = (RateA-3) + (RateD-3) – (RateB-3) – (RateC-3) – (RateE-3) ²
PCK-Q = O + C + S + I = 3×(RateA-3) – (RateB-3) – 2×(RateC-3) – 3× (RateE-3)

NOTE
1: This equation is used to measure Orientation (O) for Likert-scale questions used in both introductory college physics and elementary science.
2: These two equations appear in both contexts but are used to measure different components.