When research questions don’t make sense: use claims!

I need to write up this Very Small Thought in order to get it off my to-do list. The basic thesis is: when we ask students to do rich, open-ended projects, we often insist that they write “research questions.” Sometimes this is a terrible idea.

Don’t get me wrong: asking students to come up with research questions can be important. Many frameworks for how science works have “formulate a research question” as an early step. Furthermore, when you grow up, some grant proposal RFPs insist that you specify your research questions.

I attended a recent webinar where Kim Kastens described strategies for helping students develop research questions. We were presented with some intriguing graphics about world temperature changes and asked to come up with questions inspired by the graphics. This was a perfect use for research questions: it helped us brainstorm what we were wondering about. You can imagine students following up on the questions, collecting data, noticing patterns, developing hypotheses, and so forth.

Temperatures in 2011–2015 compared to 1950–1980 baseline (NASA). I wonder: Would we see reduced snow in Moscow? And what about the blue parts in the Antarctic… What’s going on there? Why?

But it’s not always so clear that research questions like these are what you want. Sometimes you know what you want to study, and developing the question seems stilted, like you’re forcing some idea into a box. Making it a question drains the life out of it.

Suppose instead of looking at temperature maps, you’re looking at Census microdata. And you notice that men earn more than women. If you’re a student, do you really have to make a question like, “Do men earn more than women?” or “Are there gender differences in pay?” These are weak questions; we know the answer: it’s yes. To be sure, we could make the lesson about asking better questions, but there are pitfalls there too, one of which is that it might take you away from the data analysis.

Making claims instead

As an alternative, I have sometimes asked students to write claims instead of questions. You can make a claim when you’ve been rummaging around in data and you notice something that interests you. It’s like skipping directly to the hypothesis: Men earn more than women.

A claim is a statement that is either true or false. When you make a claim, the next task is to create a visualization from the data that supports or refutes the claim.

For example, given Census microdata with sex and income, it’s easy to make a graph that shows that men, in general, earn more than women. (The fact that it’s easy doesn’t mean it’s unimportant—students need to be able to create visualizations for obvious things.)

Income by sex, 400 random Californians, 2013. Data from the American Community Survey, accessed using CODAP.

(Here is a link to a CODAP document with a plugin with which you can get your own data set.)

Then even more fun ensues. Having done that, you (or another student) can play “yes but” or “devil’s advocate,” and create reasoning that contradicts your conclusion; then you need to dig deeper and find data that supports or refutes that new argument.

The devil’s advocate might say, “sure, men in general earn more than women—but that’s because more men are actually employed than women.” The implication is that if you looked only at employed people, the gender gap would go away. We can check that! We have an employment variable for this data set, so we can filter (data move!) the data set to include only those people listed as employed.

Only “Civilian Employed.” The mean and median incomes are higher, but the gender gap is still there. And we no longer have those big stacks of points at income = 0.

As you can see, even though the devil’s advocate made sense, when you look at the data, you see that employment does not account for the gender gap.

You can go on, of course (does education make the difference?) and in doing so, learn more about the data and about how to use the data and the tools to address new considerations.

Which is not the topic of this post. The point here is that claims may give students an engaging entry point for doing tasks with data.

Some quick notes:

• I like having students just mess around in the data first. Look at the table. Make graphs. Make more graphs. Select things. With rich data, students inevitably see interesting patterns that can become claims.
• Just making a claim and developing a visualization to address it is a good first assignment.
• You can then develop the culture of the follow-up, the counterclaim, or devil’s advocate. That is, later assignments can insist on “dig deeper” components.
• If it turns out that research questions work for a student, that’s fine of course. Often, something beginning “I wonder” results in a question.
• Notice that our last “dig deeper” morphed into a question rather than a counter-claim. “What is the relationship between educational attainment and gender difference in income?” is a cool question.
• The “research question” tradition comes from science. Maybe we can think of “claims” as coming from law. You’re prosecuting gender inequality. What’s your evidence? You’re also the defense attorney: what evidence can you show in response?

A final point of reflection: claims are often powerful for students when they work with data about society or other topics they care about. That is, they have an agenda, and this is a chance to support their ideas with data. Of course we have to be critical of the data and the analysis techniques; but it’s not essential, in this context, that students be disinterested, ivory-tower, scientists-above-the-fray.

And are any of us? So often, in creating a research question, we know what result we want to find in the end. We have an idea of how to do something, and we want it to work. Paradigm-shifting results are rare and precious. When we get an unexpected result, our first thought is to explain it using the constraints of our world-view, our prejudices. The key to being fair in our use of data, it seems to me, is to develop those devil’s advocate muscles, to learn to poke holes in our own reasoning—and to follow up those pokes with additional data and analysis.