We Tell Ourselves Stories

Earlier this week, an essay written by Adam Kotsko was published in InsideHigherEd called “Making the Best of Assessment.” Two key sections of his piece stood out to me:

Would I really object if someone suggested that my institution might want to clarify its goals, gather information about how it’s doing in meeting those goals, and change its practices if they are not working? I doubt that I would: in a certain sense it’s what every institution should be doing. Doing so systematically does bear significant costs in terms of time and energy — but then so does plugging away at something that’s not working. Paying a reasonable number of hours up front in the form of data collection seems like a reasonable hedge against wasting time on efforts or approaches that don’t contribute to our mission. By the same token, getting into the habit of explaining why we’re doing what we’re doing can help us to avoid making decisions based on institutional inertia.

And a bit later on in the piece, referring to his own institution, department, and colleagues:
Despite that overall optimism, however, I’m also sure that there are some things that we’re doing that aren’t working as well as they could, but we have no way of really knowing that currently. We all have limited energy and time, and so anything that can help us make sure we’re devoting our energy to things that are actually beneficial seems all to the good.

Recently, I experienced an assessment phenomenon that I’ve started to affectionately call “when data challenges our belief system.” (My colleague has also written about this here; when I was telling another colleague about this, she referred to it as “when reality interferes with our denial.”) I’ve been helping my colleagues work with assessment methods and their findings for many years now, but it only became apparent to me recently that sometimes we don’t want to believe what we see in our assessment findings or in other sources of data. Sometimes we’d prefer to just erase the evidence or the findings of a inquiry project rather than face the reality that we might be able to do something better, something different.

Thanks to F. Jourde on Flickr for making this image available for use through a Creative Commons license.

Thanks to F. Jourde on Flickr for making this image available for use through a Creative Commons license.

Joan Didion wrote a book with the title We tell ourselves stories in order to live.

In trying to build an evidence-informed culture for improvement in higher education, I have come to believe that our willingness to interrogate the stories we tell ourselves might be one of the biggest challenges we face. Resources? Yep: we certainly need those! Buy-in that the process is worthwhile? Totally important! Support and learning to enact effective assessment practices? Absolutely necessary. But …

We tell ourselves stories.

And guess what! I am totally guilty of this! Here’s a recent example: Using findings from a database-informed report I received almost 7 years ago, I believed that 95% of students who took an introductory course offered in my department went on to take many more courses at the university. This became my department’s story (because it was true — in 2006). It was a great story, until it wasn’t. When I requested and received an updated report, this is what we found out, and what we did:

Of 388 students from Summer 2009 – Fall 2013, 90 took only this course.  Thus, 23% of students who took the course didn’t take anything else. And although this means 77% of students did take other courses (we can celebrate that – it could have been worse, after all), we needed to think about whether or not our original story (remember it? 95%?) still held water. My departmental colleagues and I discussed this report, and once we came to grips with the new story the data was telling us, we realized that we wanted to make some advising process improvements with the goal of increasing the number of students who go on to take more courses at the university.

What’s challenging from an assessment perspective is that this practice of storytelling can totally limit our ability to use information to make improvements; more significantly, it can limit our learning.  Of course we should be analytical about our findings; we should understand their limitations, reliability and validity, the circumstances, the context. But if we believe everything to be perfectly fine — or even quite good — reliable findings that tell a different story can be hard to stomach. Sometimes what happens is folks blame the data; worse is that they blame the messenger; far, far worse is that they blame students.

We tell ourselves stories.

One of the greatest powers of assessment and of an evidence-informed, improvement-oriented culture is that it can foster critical reflection on practice, but only if we can be — if we’re willing to be — critically reflective. I also think assessment can foster really important conversations about students’ experiences and learning among colleagues (such as the great conversation we had in our department when the new data no longer supported our outdated story).  In other words: assessment itself can foster our learning.  But when we get stuck with our stories and we can’t see that there might be a different reality that’s out there, we shut learning out, and we shut out the opportunities that can result from learning.

No doubt that stories are important — to culture, to humanity, to life. As Brene Brown said in this TED talk, “stories are data with soul.”  I do think our stories can inform us and teach us (hey – I was an English major; of course I think this!) But I also think that if we’re not open to alternative views, to other voices (namely those of students), and to new information, then we’re being irresponsible educators because we’re not walking our talk. And so I’ve come to realize that a big part of my work is to support people in being willing to engage in critical reflection on practice, and to consider changing their stories if needed using the information they learn from assessment (it may come as no surprise to some of you that my work with Prior Learning Assessment and interest in heutagogy greatly inform my perspective and my approach). After all, education is supposed to be about learning; let’s use information we have to learn, and to get better.

So Many Posts; So Little … Posting

I have about 50 posts started, but none finished — or at least none worthy to post. As usual, I’ve been prattling (though only in my head, so far) about change in higher education; about the importance of leaders-as-learners and teachers; about technology, learning, assessment, teaching; about heutagogy; about access; etc. etc. etc.

So many posts; so little posting.

While not posting, I have been contemplating the lessons in two poems by Marge Piercy: The Seven of Pentacles and To Be of Use.

And I did take this picture of a lovely spring flower while out on a bike ride with my kid:

Meditation on a spring flower in the forest.

Meditation on a spring flower in the forest.

So there’s that.

Instead of posting my own ramblings right now, I’ll share a few links to others’ pieces that have really inspired my nog as of late.  I’ll also share a few quotes from each that I mentally highlighted – may they inspire you to inquire and reflect, too.

How Disruptive Is Information Technology Really? by Judith Ramaley – EDUCAUSE Review

The act of teaching is becoming more about designing the educational context and engaging students as they learn to approach material in more insightful and demanding ways. We are not transmitters of knowledge very often today, although an occasional superb lecture by a remarkably perceptive and even prescient speaker or a carefully crafted blog contribution can open up new ways of thinking about things.

We all know these simple things about how the educational experience is changing, but how recently have we paused to think about how truly wonderful it is to be able to use our smartphones to answer a question right immediately? My real concern is that not all questions have a quick, well-researched, and easy-to-find answer. Many, perhaps most, questions in today’s world are hard to formulate, are seen in very different ways by different people, or simply do not have good answers at all. That is why we still need real people who interact with each other in real time in order to frame questions that matter, to explore the ideas that come from those questions, and to work together to find solutions. No longer, however, are those people confined to the knowledge and experience that they carry in their own minds or that they can bring along with them on paper. They can tap into a true universe of material whenever they wish.

Good Teachers Become Less Important by Bernard Bull – Etale – Life in the Digital World.

When I think about the role of what we traditionally think of as teacher, one of the most important roles of teachers is to work hard at making themselves as unimportant as possible, not unimportant in the sense of lacking value, but unimportant in the sense that they are eventually no longer needed.  In other words, the goal of the teacher is to aid the learners in becoming self-directed learners.

Notes on social learning in business by Harold Jarche

Training and instruction are all about control, with curricula, sanctioned learning objectives, and performance criteria. This works when the field of study is knowable. But fewer fields remain completely knowable, if they ever were. Many institutions and professions have been built on the premise that knowledge can be transferred in some kind of controlled process. If you question that premise, you threaten people’s jobs, status, and sense of worth. This is why you see some violent reactions to the notion of informal and social learning having validity within organizations.

A major difference between communities of practice and work teams is that the former are voluntary. People want to join communities of practice. People feel affinity for their communities of practice. You know you are in a community of practice when it changes your practice.

And finally, a post from the Heutagogy Community of Practice: The PAH Continuum: Pedagogy, Andragogy, & Heutagogy, by Fred Garnett

When we collaboratively developed the ideas of the open context model of learning, Wilma Clark had pointed out that in Russia the word ‘obuchenie’ means both teaching and learning, and the PAH Continuum might be seen as a way of scaffolding ‘obuchenie’ as a move from teacher’s control to learner’s control. I would see it as axiomatic, as I did when I was ‘brokering’ learning, that teachers, whilst delivering their subject expertise, should be enabling learners to better understand the process of learning for themselves.

So many posts; so much learning. With great appreciation for all my teachers out there. As Bernard Bull wrote:

In one sense, a teacher is anyone or anything that contributes to our learning.

Moving Past Wishes

Many many many moons ago, when I was teaching English in my first “real” job in a community college in Washington state, I decided to try a technique called “negotiated assessment” that I’d heard about at a conference. The conference was about learning outcomes assessment — a fairly new paradigm at the time — and when I heard this idea I immediately wanted to try it. Why should I be the only one who assessed my students’ learning; why couldn’t they do it? Wouldn’t that help them pay more attention to their learning process and outcomes; wouldn’t it also, perhaps, potentially, lighten the power I wielded with my pen and my gradebook?

Grade A

Many thanks to Steve Snodgrass on Flickr for making this image available to use.

At the time it was a risky experiment for me: I didn’t yet have tenure, and here I was, giving the power of grading over to my students (except that I wasn’t — more about that below). My department chair (who was also the chair of my tenure committee) was supportive of the experiment, and so I tried it out. This was long before I knew much about learning or assessment; long before I knew how to support students in taking more ownership over their learning; long before I formally knew much about reflection, about deep learning, about metacognition, about self-directed and self-determined learning; long before I had devoured Freire and hooks and Dewey and Kegan and Daloz and Palmer and Doll and so many others; and it was long before I had enough experience  to really design the approach so it worked well.

Nonetheless, here’s how it went in my English 101 – College Composition course that spring semester:

  • I created the course description, the learning outcomes (some of which were departmentally prescribed), and the learning activities and assignments for the course.
  • I led and facilitated the course activities and provided direct instruction when needed.
  • Students engaged (or not, in some cases) in the writing processes I set up (and required), including peer review, responding to feedback on multiple drafts from me and other students, learning about revision as “re-seeing” and editing as cleaning up fly specks on the window so readers could see the message clearly, etc.
  • I never graded a single paper; I provided LOTS of feedback on LOTS of drafts, but I never assigned an actual letter grade.
  • At the end of the term, students wrote a final essay self-assessing their learning through the lens of the learning outcomes of the course and feedback they received from me and others along the way. They put the paper in the front of a binder that contained all of their other drafts and final papers (a very fundamental form of portfolio at the time). Based on their “analysis” and the “evidence of learning” in the binder, they recommended a final grade for themselves.
  • The last week of the term, I met with each student individually to review the paper, the portfolio, and to discuss the grade they thought they should get, and to talk about how they should transfer their learning from this class to other classes and contexts (because surely, if I told them to, they would, right?)

It was a fascinating experiment in all sorts of things — students whom I considered to be “strong” graded themselves much harder than I would have; students I considered to be “weak” (or “flakey”) recommended A grades for themselves. (I am now ashamed that I had those thoughts about my students at that time, but I did.) During the meeting, we “negotiated” the student’s grade and came to agreement, using the portfolio as an anchor, on what that grade should be. (In actual fact I still actually did hold most of the power. For one thing, I totally over-ruled 2 of my 20 students. They most certainly did not earn an A let alone demonstrate much learning. What kind of pushover did they think I was!?!)

I was reminded of this experiment from many moons ago this past week when I read a new perspectives post about heutagogy contributed by Fred Garnett in the Heutagogy Community of Practice.  Read it here:

The PAH Continuum: Pedagogy, Andragogy & Heutagogy

In this post, Fred describes his practice of  “brokering:”

Writing the syllabus and developing the schedule of delivery along with the work to be completed meant that I was, in effect, building the framework of what I was teaching. Consequently I really understood what the boundaries were and so could better broker between the formal requirements of the education system and the personal desires of my learners; I had found that all these ‘failing’ students wanted to learn.  On the social impact course each student picked any technology that interested them to research and write about. I showed them how to “play” with the learning requirements, which can be used as creative constraints, and how best to meet them in their completed work. I also encouraged them to present that work in original ways rather than as just a written report. Although most presented reports a precious few tried original approaches, such as wall charts, cartoons, a class presentation with Q&A, and so on. Most importantly simply having the opportunity to present finished written work in ways that they determined meant that they thought about various ways in which to explain their ideas.

Here’s what I like about this concept and what I wish I had done in that class: I wish I had actually tapped into and understood the personal desires of my learners and showed them how to “play” with the learning requirements and how to best meet them in their work. I wish that I had encouraged them to produce artifacts that were original — that were personalized to their desires — that would meet the learning outcomes. And I wish that I had understood that my “failing” students — those that I unfairly deemed flakey, or lazy, or poor writers — probably did want to learn, but maybe not the way I thought they should. I wish … I wish … I wish…

This is why I am excited about heutagogy and wanted to start a Community of Practice focusing on it — it so nicely grounds and frames the approaches I try to use when I teach in a way that does not leave me wishing. It provides a way for me to organize and re-vision and re-enact my knowledge about self-directed learning; about authentic assessment; about learning outcomes, course design, and curricular structure; and about my preferred “teaching” approach.

I am so grateful to continue to learn about my practice — and to self-determine my path. Thanks to all who’ve joined the Heutagogy Community of Practice for already furthering my learning in ways I hadn’t ever anticipated.

To Africa

Lake Nakuru National Park, Kenya

Lake Nakuru National Park, Kenya

In October, I had the amazing opportunity to travel to Kenya to facilitate a week-long faculty and staff workshop about Prior Learning Assessment at Tangaza College in Nairobi.  I am grateful to my colleagues at the School for New Learning of DePaul University for inviting me to participate!

In a bookstore in Nairobi, I purchased Beryl Markham’s autobiographical account of her time in Africa titled West with the Night.  Here’s a particularly relevant and poignant paragraph that captured my attention:

“There are as many Africas as there are books about Africa — and as many books about it as you could read in a leisurely lifetime. Whoever writes a new one can afford a certain complacency in the knowledge that his is a new picture agreeing with no one else’s, but likely to be haughtily disagreed with by all those who believed in some other Africa. … Being thus all things to all authors, it follows, I suppose, that Africa must be all things to all readers. . . . Africa is mystic; it is wild; it is a sweltering inferno; it is a photographer’s paradise, a hunter’s Valhalla, an escapist’s Utopia. It is what you will, and it withstands all interpretations. It is the last vestige of a dead world or the cradle of a shiny new one. ”
― Beryl Markham, West with the Night

Read more about my trip here: Out to Africa

Fearing Assessment; Fearing Learning (And Fearing David Brooks)

O this learning, what a thing it is! ~Grumio in Taming of the Shrew, William Shakespeare

Twice this past month I’ve heard the word “fear” used by faculty when referring to their experience of assessing student learning in their courses. One person described it as fear of students disagreeing with their grade or feedback, or generally unhappy with the judgment the instructor made about their work and requesting explanation and justification (much of which could be alleviated, I thought, if the instructor made the criteria transparent to students, or even better, if the criteria were collectively developed with students, but I digress…).

Thanks to Jimee, Jackie, Tom & Asha on Flickr for making this photo available for use.

The other person described her fear that she lacks the ability to discern quality and to really be able to tell what a student has learned. She described her lack of confidence in using a writing rubric to “judge” what about a student’s writing, as exemplified in a single assignment, is exceptional and what is developing (and every shade of grey in between). I appreciated her honesty with this challenge; I’ve certainly faced it as well (though in my case, “fear” was not a word I used to describe what I experienced as a “bleepin’ assessment conundrum!”). Nonetheless, her description reminded me of something from Parker Palmer’s The Courage to Teach: “Teaching is a daily exercise in vulnerability.”

So, too, is learning.

I am pretty certain that learners fear assessment as well, which is truly unfortunate and totally not necessary, and in the end, adversely affects our ability to learn. When faculty work from a “gotcha” perspective, of course assessment is something to fear! I remember a Shakespeare course I took in college that made me have night terrors; I couldn’t sleep that term because that class and that prof were seriously scary. Our final grade consisted of our scores on 5 tests: a test after each unit (comedy, drama, history, and what was the other??? – poetry, I guess), and the big ugly final exam, 3 hours of closed book / closed notes mental torture. These tests were tricky because they were designed as “gotcha” tests (including the essay part of the tests, for which we could use only one side of a single piece of unlined white 8.5 x 11″ paper to address the topic, for no clear reason other than the prof didn’t want to read more than what could fit in this designated space). It was always obvious from the smirk on his face and comments under his moldy breath that the curmudgeonly old prof enjoyed this process. These tests didn’t in any way advance or enhance my learning (I memorized a lot of Shakespeare that term but I didn’t learn any of it except for a few random quotes I can pull out of my head for cocktail parties or blog posts); they didn’t help me appreciate Shakespeare in new ways, or connect important themes or ideas to topics I was interested in. They freaked me out! Why was that necessary?

And now there is something else to fear: David Brooks, writing this op-ed piece in The New York Times, has called for value-added assessments.

Colleges have to test more to find out how they’re doing . . . There has to be some way to reward schools that actually do provide learning and punish schools that don’t. . . This is the beginning of college reform.

To which I reply: ARE YOU SERIOUS? How will THIS advance learning on the part of students, and on the part of faculty and institutions? Punish schools that don’t. Really? Punishment creates fear; punishment creates distrust. And fear and distrust do not promote learning — for students or any of us! I don’t disagree that we need to know how we’re doing … we do! We really, really do! But I absolutely believe that this approach is completely antithetical to actually promoting learning (note that I didn’t say “producing” learning, the term Brooks used, as if learning were something that gets assembled on a conveyor belt). This approach will foster fear; fear inhibits learning. Period.

Colleges (and faculty) have to remove fear first — this should be the beginning of college reform. I think it was Shakespeare who once wrote:

Things done well and with a care, exempt themselves from fear.

This Makes Us Official

Marylhurst University

My colleagues in the Assessment Program and I finally got our program description and content on the Marylhurst website. We’ve been wanting to do so for a very long time, but we needed the time and space to get it all organized. Thanks to the great work of our Educational Assessment Specialist Sione, we’re now present, virtually.

I think that makes us official now, right?

Check it out:  Assessment at Marylhurst University

Liberating For Learning

As I continue to work through and ponder the readings for the Assessment Leadership Academy I am participating in, and as I talk with my colleagues about assessment, the conundrum of grading keeps surfacing. The key question seems to be:

What is the relationship between grading and assessment?

I recently responded to an AALHE blog post about grading in which the author wrote this:

If we reverse our assumptions, we can think critically about how grades can be used to assess student learning.  It seems to me that a few things need to be in place first, though — with a performance-based approach, that would include shared rubrics, mapped curricula, and faculty who regularly “norm” to the rubrics they use through some sort of appropriate validity process.  Given those conditions, grades could be used with sufficient validity to produce useful aggregate data.  (Why Not Use Grades?)

My response was this:

I think the relationship between grading and assessment is often one of the biggest conundrums for faculty — it certainly has been for me! And I think you’re right that we need to think differently about the role of grades in assessing (and dare I say promoting?) learning.

I like the explanation offered by Mary J. Allen in her book Assessing General Education Programs (2006): grades can be a tool to promote student attainment of outcomes IF grades reflect the extent to which students meet / master course outcomes. As she writes, “Grading procedures should align with course outcomes, and course grades should indicate the extent to which students have mastered them” (p. 100).

The concept of grades as an assessment tool — and thus a learning tool — helps change this direction. But the “A” word always comes back into play: Alignment. When I have this conversation with my colleagues, this is where I focus. To what extent are the assignments and other graded activities (projects, exams, etc.) aligned with the outcomes? If we can do good alignment work (and Allen shows an example of a Grading Alignment Matrix that I’d like to use), then we can feel more comfortable that grading can be good assessment.

But here’s the conundrum for me: Students often focus their efforts on the grade, not the learning. We have probably all heard our students say “What will it take to get an A?”

Thanks to frankchowrules.blogspot.com for making this image available.

Of course, for people who care about Learning (capital L intentional), this is the wrong question! An alphabetical letter usually cannot tell me what a student knows or can do. Nonetheless, the pressure our students experience for a good GPA, financial incentives (in the case of some of my students, their employers will only reimburse tuition if they earn a B or higher, for example), and the desire to be the “A” student get in the way of the bigger picture: Learning. We can work with our students to refocus on the intrinsic rewards of learning and how learning supports their own personal, professional, and educational goals, but the extrinsic rewards for good grades often overshadow these.

In the PLA program, I have the great and wonderful gift of teaching courses that are Pass/No Pass, so that we can focus on the learning experiences and outcomes and not “what will it take to get an A.” It’s freeing for me and I believe it’s freeing for my students (my students: please feel free to chime in here). The Pass/No Pass system does not affect a student’s GPA in any way – it is GPA neutral. The key to learning and integrity, though, is still in alignment and in assuring appropriate quality and rigor: all defined and supported by the learning outcomes and their corresponding activities and assignments.

An additional benefit of the Pass/No Pass system is that I get to focus on where I think the biggest-bang-for-my-teaching-buck can reside, in my formative assessment work — in the feedback I provide (and my students provide to each other) to improve or deepen their learning. I worry a lot less about making evaluative judgments on their performance in my class. (Frankly, I also have a lot fewer um…shall we say, “attempts to negotiate” in a Pass/No Pass system than when I teach a class that has A-F grades. I have a 3-year old; I don’t need any more “attempts to negotiate” in my life!)

There are several downsides to Pass/No Pass as well, of course (a very real one I’ve encountered is that some employers will not recognize a Pass as a legitimate grade for which to allow tuition reimbursement). I am in no way advocating for this kind of structure over an A-F system (nor am I about to launch into a critical comparison and contrast of systems — that would be a dissertation or at the very least a white paper, not a blog post). But I am saying that in many ways, the Pass/No Pass approach is liberating for me. I believe it is liberating for my students. And most of all, I am absolutely certain that it’s liberating for learning.