Alan's writing -- even in comment form -- is so perfectly crafted that I thought i'd move it into its own post, and turn some of the words into hyperlinks, and added two [bracketed] notes.
I saw this a few years ago. They could be right, but there is nothing in the paper that substantiates it.
(How to do a short reply here?)
Notion 1: Good science can rarely be pulled off in an environment with lots of degrees of freedom unless the cause and effect relationships are really simple. Trying to assess curricula, pedagogy, teaching, and the learners all at once has lots of degrees of freedom and is *not* simple.
So for example we've found it necessary to test any curriculum idea over three years of trials to try to normalize as much as possible to get a good (usually negative) result.
Notion 2: Most assessments of students wind up assessing almost everything but. This is the confusions of "normal" with "reality".
For example, in our excursions into how to help children learn powerful ideas, we observed many classrooms and got some idea of "what children could do". Then I accidentally visited a first grade classroom (we were concerned with grades 3-6) in a busing school whose demographic by law was representative of the city as a whole. However, every 6 year old in this classroom could really do math, and not just arithmetic but real mathematical thinking quite beyond what one generally sees anywhere in K-8 [kindergarten and grades 1 through 8].
This was a huge shock, and it turned out that an unusual teacher was the culprit. She was a natural kindergarten and first grade teacher who was also a natural mathematician. She figured out just what to do with 6 year olds and was able to adapt other material as well for them. The results were amazing, and defied all the other generalizations we and others had made about this age group.
This got me to realize that it would be much better to find unusual situations with "normal" populations of learners but with the 1 in a million teacher or curriculum.
I found Tim Gallwey, who could teach anyone (literally) how to play a workable game of tennis in 20 minutes, and observed him do this with many dozens of learners over several years.
I found Betty Edwards who could teach (again literally) anyone to draw like a 2nd year art student in one intense week.
And so forth, because what the exceptional teaching is doing is actually allowing assessment of what general human beings from a typical bell curve can learn from crafted instruction.
And, I think some of the keys here are in the metaphor of bell curve. Students will exhibit distributions of talent, motivation, learning skills, style, etc., and one will see these show up right away in any simple-minded form of instruction and curriculum.
But if the battle cry is "Learner's First", then what we really want to know is what can be done to help the different types of learners. Some don't need any help. Some need to learn some things before they tackle the main subject. Some need to be shown different POVs so they can see a route for them to learn.
Really good teachers want to get all the students to be fluent, and they often find ways to do this. "Regular" teachers often just want to get through the material. Some school systems want to use education to sort the population rather than to educate the whole population. Etc.
I don't know the general answers here, but our research groups in the mid-70s [presumably the Learning Research Group at Xerox Parc] set a goal of 90% fluency for (say) 10-12 year olds, and then we proceeded to fail to achieve this until about 1998, when enough things had been done in the computer environment to provide hooks to many different kinds of children without losing the essential high quality of powerful ideas that was our goal.
I think as a teacher, one has to embrace the bell curve idea and be prepared to deal with at least three tiers of preparedness in the students. One could hope that a lot more general prep about thinking and symbolizing would have happened in K-12, but it doesn't in the US for sure.
There has been some very interesting work with respect to science teaching that seems parallel here (for example, by Tinker and others at Tufts). They not only found a pretest (could they interpret various kinds of graphs?) that would predict the grades of the 1st year physic students, but found that teaching the kids skills in doing well on the pretest (using some very creative ideas that Jerome Bruner would find familiar) would also vastly improve their performance in the physics class itself.
So the pretest was not just testing, but also finding some forms of relational and figurative thinking that some of the students needed skills in, before tackling physics.
I think every musician who is reading this will know what I'm driving at here. Music is a lot of skills and types of thinking and few musicians are naturally good at all of them. The desire to be a musician plus decent music instructors will find the things each learner will need to work on to get fluent. The result is that most skilled musicians can play advanced stuff, but they are all rather different on their outlook, how they practice, what they practice, etc.
(Sports and art also ... and almost certainly the more holy subjects sanctified by society, and those pretenders to the throne such as computing ....)
Cheers,
Alan