Education informatics and challenges of applied programs
EDIT: This is a re-posted entry from my previous blog.
The program from which I received my PhD had a lot of quantitative methods, as was (is?) common among many of the more research-heavy education policy programs at the time. My required courses included statistics, multiple classes on regression analysis, causal inference, and a year-long practicum course on quantitative analysis. The focus on quantitative methods was so much so that I don’t think it’s particularly a secret that some people were concerned that we, the students, were not getting quite enough exposure in our coursework to content issues or, alternatively, other ways of creating knowledge.
At the same time, given my interests, I also self-selected into a course plan that was heavily dominated by quantitative methods, which included a smattering of courses across various departments. In addition, I try to do my best to stay up on current trends in quantitative methods, both in terms of econometric ideas as well as the emerging fields of “data science” (ambiguously named) and machine learning. And while I as a faculty member do not have any particular desire to have students replicate my experiences (unless they want to), I do find it difficult to figure out what a quantitative methods program would look like that effectively covers the various kinds of questions and issues that students might encounter, giving them a deep understanding of what the methods are doing and why while avoiding making a quantitative-only focused program.
In general, my concern is that I do think a researcher who wants to do quantitative work should be adept and complex in their thinking about the three major branches of quantitative questions: descriptive, explanatory, and predictive (these categories are simplified, I know, but that’s an issue for another time). The tide I am seeing that concerns me is that while there may have been a point where fairly simple methods of analysis and presentation were good enough to accomplish these goals, there are now fairly complex, advanced ways of dealing with these issues that are hard to figure out how to fit into one or two applied statistics courses.
Descriptive: One of the big places I see advances in descriptive approaches, for example, is data visualization. The world of data visualization has evolved over time, and there are entire academic programs of study that focus on visualization and how to communicate data and gather insights from said data about what is happening in our world. Not only that, but the tools continue to evolve, whether in point-and-click interfaces like Tableau or coding interfaces like the ggplot2 suite in R. However, I think academic programs often cover through basic visualizations and descriptive methods (e.g., so many scatter plots), but don’t get very deep into having students understand how to be complex and flexible in their thinking and presentation.
Explanatory: At the end of the day, when it comes to explanatory questions in quantitative methods, I think we need to admit to ourselves that most of the time, we are talking about causal inference. The design of randomized experiments is certainly important, but there are also complex ways to infer causality from non-experimental data that are now part of the bread and butter of the policy analyst (e.g., regression discontinuity). Of course, I’m biased by virtue of my training from (mostly) economists, but I think advances in econometrics and other social sciences have raised the bar for what should be expected from an emerging education policy researcher.
Predictive: On the predictive questions note, I’m a little less sure about the future, as we exist in an era (it seems) where the focus on things like model fit and cross-validation are less emphasized than concerns about endogeneity. Not only that, but given the incentives and outcomes expected in academic life (e.g., academic journal articles), it’s not entirely clear to me what the outlets are for people who do work that is meant to produce, as an end product, a model. However, these methods are indeed a part of education policy life. Take, for example, machine learning studies and products focused on creating models that can predict whether or not a student drops out of e-learning. While I don’t think we’re at the point where an emerging education policy researcher needs to have an in-depth understanding of machine learning models (not to mention that I don’t get the impression that many machine learning users understand the models either), but shouldn’t students at least get a good introduction to those methods (e.g., decision trees, neural nets) so that they understand the way they can be used to answer important questions?
In addition to the issues above, the techniques for collecting data, organizing it, and analyzing it are getting more complex. For example, the world of measurement has evolved, yet we often don’t do much more than show our students a Cronbach’s alpha. (See this post from William Buchanan (Fayette County) regarding his frustrations with working with researchers that don’t have adequate training on these issues.) In addition, I think we have gotten to a point where there is broad consensus that the ideals of reproducible research necessitate a shift of quantitative tools training away from reliance on point-and-click interfaces to actual programming, whether that is by using SPSS syntax or R code. However, the learning curve for programming is more steep than we might have time to cover. I even found myself de-emphasizing programming in my own courses that I teach in an attempt to fit all of the material.
Of course, there is value in specializing, but if someone wants to do quantitative research, would covering a broad range of issues better allow them to flexible in their thinking and approach a variety of research questions, rather than trying to solve every problem with a multiple regression (with fixed effects)?
My biggest problem then becomes this: How do you design a program that can accomplish this goal? Certainly, there are some pretty severe downsides to having a program so heavily weighted towards quantitative methods. Upon exiting my program, I felt somewhat unprepared to engage with the education research community on strict content grounds. However, I collaborated, and continue to collaborate, with some very wonderful colleagues that indeed have the strong content knowledge, and I do believe that this open, compassionate collaboration allows me to make sure the content gaps are not lost in my work, while also giving me the bandwidth to focus my mental energies on being flexible with the quantitative approach. (My collaborators are welcome to disagree with me on this point.)
In other words, I wonder, sometimes, if we need to think about reorienting the field such that there is space for more specialized, quantitative methods people who can come from programs where they are indeed focusing very much on quantitative analysis, but also on collaborating with the people from more content-heavy educational backgrounds. Given, I’m biased because my undergraduate degree is in bioinformatics, but I do think we are in a moment where we should seriously consider the creation of education informatics as a specific field (I’m flexible on the name). The history of how bioinformatics was created, and the challenges that were facing biology at the time, feel similar to the issues we are facing now in education. Perhaps we are at a point where the depth of knowledge that exists in quantitative methods warrants some separation. We bemoan the fact that education, it seems, is often behind other fields in terms of the advancement of its methods. Perhaps creating a space where people could specialize and spend their energies focusing on those issues, while also working side-by-side with content experts (who should still have some baseline training) would allow for more robust, insightful, and important research. Just a thought.