“A” is for actionable!

Assessment geeks like me refer to this as “closing the loop” — using our assessment findings (whether direct or indirect) to make improvements in our assignments, course design, instructional methodologies, programs, or our student services. Or, when appropriate, using our assessment findings to showcase what we are doing really well!

In the Assessment Program at my university, we constantly try to ask this question: What can we do with this data? We often all have the What? and we might even have the So What? But then …

NOW WHAT?

Simply put: How will what we learn here help us improve? What actions should we take accordingly?

So now we are writing about some of these assessment projects we’re working on making more actionable in my colleague’s Actionable Data blog. (By “we,” I mean “she.”)

(Oh – who is she? Her name is Kim Firth, and she’s an awesome data geek and a very cool colleague.)

Here’s part 1 of several posts to come about a project we are working on (again, by “we” I mean “she”): the revision of our Alumni Survey. Read it and weep. Ok, don’t weep – it’s not sad. If anything it should make you happy that we’re being so intentional about making this survey actionable.

Lessons from an attempt at action-focused survey revision

This is a more recent picture of him with his new bike at the age of 4 and 30 seconds:

Some pretty major things needed to happen to get from picture #1 to picture #2. For example, in our family:

• Four year-olds need to be able to put on their own socks.
• Four year-olds need to make their beds when they get up in the morning.
• Four year-olds need to take their plates to the kitchen after they are done eating.
• Four year-olds, for goodness sake, need to feed themselves!!!

That’s right. We instituted milestones.

Milestones are constructed to provide reference points along the road. This can be used to reassure travelers that the proper path is being followed, and to indicate either distance traveled or the remaining distance to a destination. ~Wikipedia

Milestones, in this way, promote learning and help us assess how well we are doing toward a learning outcome. In the world of educational assessment, we call milestones a kind of “formative assessment.”

In my family, it’s what got us sanely from three to four.

Ok, ok, in all fairness, we really created milestones so to preserve what’s left of our parental sanity, but we also wanted the kid to have a goal (being four), to work toward something significant and important (and I quote, “Mama, I fed myself my dinner all by myself; now I can be four!”), and to have a sense of accomplishment when he got there.

The payoff for reaching these milestones? Well, not only did the kid actually turn four (which frankly seemed significant enough to all of us), but he also got that new bike in picture #2.

And guess what! To prove to us that he had in fact turned four, he ate 2 pieces of this birthday cake!

All by himself!

First Official School Photo

The report card is not a card that says “A” or “C” or any other grade, but is, in fact, a rubric.  There is a description of where his learning levels are in several categories, accordingly:

(E) = Exceeding – consistently exceeding grade-level expectations; a strength

(M) = Meeting – developmentally appropriate or meeting grade-level expectations

(D) = Developing – working towards grade-level expectations

(X) – Not assessed at this time; not applicable

For each category, there are specific learning items that are assessed using this framework (some are skills; some are knowledge areas; and a lot is behavior, as you might expect for students who are 3 and 11/12ths years old). Here is an example from the listening category:

LISTENING

• Effort – M
• Demonstrates comprehension in the daily routine – M
• Listens attentively to spoken language – M

(No comment here about how I would assess his listening skills. Let me just say we might have a case of grade inflation happening here. Or an inability to transfer skills from one context to another. Either way … )

Thus, across several categories and skills, we now know where his teacher sees his strengths and where we can help support his improvement. For example, he can count from 1-6 (E) and sort objects by color, shape, and size (E), but he needs more work in demonstrating self-control (D) and accepting responsibility for his own actions (D). He is doing as expected in recognizing his own name in print (M) and cutting across paper with scissors (M).

This takes us to the parent-teacher conference, which was also about assessing his learning and was evidence-based. His teacher had an iPad with about 30 pictures of Mac taken from the beginning of the year. Together, we looked at evidence of how he held a marker in September, October, and November; we were able to see differences in technique by looking at actual letters, shapes, and pictures he had drawn in an accompanying portfolio of work. She also had samples of his writing in which we saw evidence of how he wrote M – A – C in September compared to how he writes M – A – C now (not much improvement there, frankly. The M is still upside down thus spelling WAC instead of MAC. No comment…)

My point?

THIS IS ASSESSMENT, AND IT’S AWESOME!

We pose questions: What and how well is he learning? What evidence do we have? And what do we need to keep working on? And in answering these questions, we learn and his teacher learns and his school learns! Assessment = Learning = Assessment = Learning and around and around we go through the learning cycle. And we love it because we care.

Overall, Mac seems to be learning and doing pretty well in school (which is awesome considering that he is 3 and 11/12ths years old and has trouble listening … but apparently only with us). To provide further evidence that he is learning, here is a conversation between Mac and his dad this morning:

Mac: I want to wear these pants today.

Dado: Cool! These are cords!

Mac: Oh – I can’t touch those. I am not supposed to play with cords. They’re dangerous!

This after yesterday’s moment of inquiry, accordingly:

Mac: After the champion wins, is that when they get the chips?

Dado: The chips? What chips?

Mac: Yeah, the champion chips, Dado.

(Yep – most certainly grade inflation.)

FIRST: WHAT DO YOU WANT TO LEARN?

The first step, of course, is to identify what you want to learn. What would the learning outcomes of a visit be? Surely you will learn stuff you didn’t know you wanted to learn too, but if you can identify a focus for yourself and your institution, proposing a visit and the visit itself will be a lot more focused.

For example, for my recent visit to a university in SoCal, I identified these focus areas and shared them with my colleague there:

• What you are doing with Liberal Arts assessment, specifically using the VALUE rubrics and your institution’s core revision process?
• What the librarians are doing with their assessment work?  (You’d mentioned that they were using SAILS – I’m wondering if it would be possible to meet with them to learn more about how they are doing assessment?)
• A sense of how the Assessment Committee(s) work — their structure, charge, participants, etc.
• More about your Program Review processes and outcomes
• Your role in building the culture of evidence / assessment

This list helped her know who to set up meetings with and what materials I might be interested in seeing. (Indeed, it was a rather large list, but in all fairness, I was coming from out of town and only had one day to meet with them. I was trying to be comprehensive.)

TYPES OF VISITS

In my experience, there are two main types of learning visits:

1) Problem-based: This kind of visit is intended to help you work on a problem you or your institution has. For example, I arranged a phone visit for myself and 3 colleagues from my university to talk with a person at another institution in Chicago about how they manage the assessment of student learning in an outcomes-based liberal arts curriculum; this was a very real problem for us at the time. (Turns out it was for them too!)

2) Topic-based:  This kind of visit is more focused on a specific shared topic, such as “general education learning outcomes” or “facilitating internships.”  A while back, I set up a session with a person using the Mahara ePortfolio system at a different institution because we wanted to explore this system and see it in action. We used a web-based desktop sharing system and the phone, and she kindly took us on a 45-minute tour of how they are using Mahara ePortfolios. Nice!

IDENTIFYING WHERE & WITH WHOM TO VISIT

Figuring out where and with whom to visit per your desired learning outcomes is likely the next step. Here are some ideas for how to do so:

1) Look to your local network. Are there colleagues with similar job responsibilities at institutions near you? Do you know anyone at an institution near you that can connect you? Do any of your current colleagues have connections at these institutions that they could leverage for you?

2) Look to your distant network. Have you connected with folks at conferences that you can reconnect with? Even if you can’t visit physically, with the phone, Skype, or with other technologies you can visit virtually. I have a “coffee date” about once a term with a colleague from another institution across the country who I met at a conference; we both get a cup of coffee and talk on the phone for about an hour, and just learn from each other.

3) Look to your virtual network. If you use LinkedIn or Twitter, search for and follow people who are in similar roles or who have identified projects they are working on that are similar to yours. This is how I connected with the Mahara ePortfolio person; I had learned a lot from her by just following her on Twitter, and then when my colleagues and I were ready, I sent her a message with a few questions. From this initial conversation came the idea to have a short virtual meeting in which she took us on a tour.

PROPOSING THE VISIT

First, make contact and make a simple initial proposal. It might look something like this:

Hi there – My colleague XX shared with me your contact information because I am interested in learning about what you are doing with blah blah blah at your university; we are trying to implement this at my university as well, and I wonder if we might be able to set up a time to chat briefly about what is working and what’s not. Maybe we can learn some strategies from each other. Etc etc etc …

As demonstrated here, a proposal for a visit might be more compelling if you identify what you can bring to the table. What will be in the visit for them? What might you be able to contribute to the conversation?

Also, start small. You don’t need to visit for a whole day. Maybe you just begin with a short phone conversation, or maybe (if you’re close) you meet for lunch.

Finally, I think it’s useful to learn with others, as a team. If you can, take a colleague or two from your institution with you, or invite them to the phone / Skype conversation. Propose the person you’re meeting with do the same. The more the merrier!

DID YOU LEARN?

After the visit, assess your learning. Did you achieve your learning outcomes? If not, what might be next steps? What additional questions were raised for you, or what other resources should you explore? Likewise, was the person / institution you met with interested in learning more and continuing the conversation? If so, then maybe you all set up another learning visit with each other.

ONE LAST TIP

In the spirit of academic integrity,1) don’t take and just start using what is not yours and 2) give credit where credit is due. We all adopt and adapt ideas and materials all the time in higher education; if you want to borrow something from someone you’ve visited, ask permission and then attribute it. There may be nothing more irksome than being visited and sharing a rubric, for example, and not knowing the visitor adopted it or re-purposed it.

Happy visiting – learn lots!

• Go interdisciplinary
• Go international
• Go outside your institution
• Go to a neighboring institution
• Go to your local community

What a great way to broaden my perspectives and learn new things, and bring back good ideas to my own institution; what a neat form of professional development; and except for “go international,” many of these things I would be able do with low impact on my to-do list and relatively low-cost to me or my institution.

But then I realized that I already do this kind of stuff all the time. These kinds of activities have been integral to my own scholarship for a long time now (if you subscribe to Boyer’s definition of scholarship, which I do). Perhaps these kinds of activities might be defined as networking activities (as University of Venus does) or could even be considered some funny form of academic tourism, but I prefer to think of them as I have experienced them: learning visits.

Learning Visits, Not Academic Tourism

Let me share some recent examples. I will begin, first, with an experience of being visited:

Several months ago, colleagues from an institution similar to mine (but way across the country) contacted me about coming to my campus for a learning visit. I had met a few of these good folks at a conference a year before; we had a healthy exchange of ideas then and had remained loosely in touch. Their institution was planning to take a team of folks to three universities in the Pacific Northwest, just to learn. They came and spent a day with us, learning from and sharing with several folks on my campus.  It turned into a learning exchange within a learning community. Indeed, we asked as many questions of them as they did us, and we learned as much from them as I hope they learned from us.

Since then, one of the members of that visiting team and I have had virtual coffee dates to continue to discuss shared challenges and opportunities (mostly about assessment, but also about implementing liberal arts programs as well as working with adult learners in higher education). To continue our shared learning, next week several of us are visiting again (though this time virtually) to talk about Prior Learning Assessment. And we are visiting with each other just to learn: What’s working and why? What’s not? What ideas might we come up with to improve our programs and our students’ experiences?

This past Friday I completed a learning visit of my own to a university in Southern California (and my university will host them in a learning visit this coming week). This was actually one of the assignments for the WASC Assessment Leadership Academy in which I am participating, but it was a great excuse to spend the good part of a day at another institution learning about what they are doing with assessment and how they are building their teaching/learning/assessment culture. I met with folks from a few academic areas, student services, and institutional research, and also learned how their cross-college assessment committee supports this important work at their institution. I learned about their progress, and their challenges. The visit gave me several ideas for strategies and tools I might bring back to my institution (with adaptations of course), and it also confirmed some of the work we are already doing. I learned.

I am learning so much from these learning visits that I am planning more. With colleagues from our Assessment Program, I am planning a learning visit to a local college to explore their experiential learning simulations lab and think about how such teaching/assessment systems might apply to other disciplines. With a colleague from our Service Program, I am planning to visit another local university to learn about their service-learning program.

I have taken the challenge to heart, and I intend to keep doing so.  I think learning visits might be unique opportunities to higher education (perhaps I am incorrect, but I can hardly imagine car or technology companies doing “learning visits” with other car or technology companies). I also think they just might help all of us get better. If higher education is about learning — our students’ learning and our own — then learning visits are one relatively simple way we can achieve great learning outcomes.

It is true that I flunked out of Girl Scouts long before I could get a vest like this.

I am trying to understand badges. Given my background with Prior Learning Assessment programs and other forms of experiential learning, I totally get the idea conceptually. (Hey – I know a few boy and girl scouts too! They’ve earned badges for things they learned and could do too, right?)

Apparently, with a badge:

You can get credit for learning outside of school, on the web, or from work and life experience.

So it sounds like some sort of technologically-mediated PLA, except the credit you earn is is not college credit. There is some other currency here, though what value it will hold remains to be seen.  It seems to me that there’s great potential here, as this article explains. What with the rising costs of higher education and access issues, and the learning opportunities presented with new technologies, combined with learning one can obtain through experience (hopefully with some expert guidance and reflection), this might be something quite worthwhile.  I can see it now: Post a few badges onto your robust ePortfolio and your avatar may actually scream: Hire Me!

But here’s what I don’t get — where is the “quality control?” (Ack – I hate that term applied to learning, but I think it’s a legitimate question).  My burning question with all of this is:

So, how, exactly, will learning be assessed?

Of course I ask this question of my faculty colleagues in higher education too — it’s certainly not a question specific to badges. I am honestly trying to figure it out!

Here is some information about assessment and badges, according to the Badge FAQs (a wiki, so what I’ve copied in blue will likely change soon):

Who can issue badges?
Badges can be created, defined and issued by a number of sources, including:

• Traditional educational institutions (e.g., x, y or z)
• Professional bodies (e.g. doctors, engineers, accountants)
• International credential assessment agencies
• Non formal, community learning organizations (e.g. Adult Basic Education, Literacy, Employability)
• Communities of practice (e.g., open education projects, peer learners, or the individual learners themselves)
• After-school programs and learning networks.
• Online courses and open courseware initiatives.
• Companies/organizations that employ people

How will the value of the badges be authenticated? In this system, a digital badge is more than just an image – it is essentially a collection of metadata that fully explains the badge and includes information such as the issuer, issue date, criteria for earning the badge, expiration if needed, the learner work or evidence behind the badge, etc. So the badge acts as a gateway or conversation starter, but the bulk of the information is in that metadata and it can act as an informal validation system itself.

And this is what the Mozilla and P2PU badge pilot project will address regarding assessment specifically (also from the FAQs):

Assessment

The pilot will explore a range of assessment types, including:

• peer assessment
• self-assessment
• portfolio assessment
• stealth assessment
• The Javascript badge assessment, for example, will require learners to submit work that demonstrates competency. Peers will then rate the work against a predefined rubric and set of criteria. Once the rating reaches a particular threshold, the badge will be issued.
• The Accessibility badge will require experience designing or developing for challenged users or accessibility technologies, plus a blog post with reflection and analysis of the experience. A group of accessibility gurus within the community will then assess the work and issue badges accordingly.
• Other badges may be aligned directly with courses, with course organizers able to assess work and issue badges.

And more:

How does assessment work?

• For badges to hold real value and carry the weight of more traditional grades or degrees, assessment and quality is critical.
• Badges can contain multiple levels of assessment, depending on the use case, community or intended audience
  • some will require distinct pre-defined assessment exercises and success criteria
  • others may be loosely defined and require learner reflection or peer recommendations.
• Hard skills may require standard or more rigid rubrics to compare learner work against.
• Softer skills can be more fluid and require more open and social assessments like peer reviews or endorsements.
• For certification badges, intended for audiences like hiring managers, admission boards, more rigorous assessments can be required
• For badges intended to simply build community or reward behaviors, simple assessments may be enough

How can badges provide greater flexibility and innovation in assessment?
Badges can help:

• drive innovation around new types of assessments (e.g., x or y)
• provide more personalized assessments for learners (e.g., x or y)
• move beyond out of date or irrelevant testing practices (e.g., x or y)

For example:

• Asynchronous assessment. Instead of being required to take an exam at a pre-determined time, for example, learners can seek out the assessment on their own time.
• “Stealth assessment.” Assessment and awarding badges can happen automatically and provide immediate feedback. [Need a half sentence summary of what "stealth assessment is.]
• Portfolio assessment. Work samples, projects and other artifacts the learner has produced or been involved in can demonstrate skills and competencies.
• Multiple assessors or group assessment. In traditional classrooms, an individual instructor generally does most of the assessing. An open badge system can support assessment from multiple contexts, including course organizers, peers, or learners themselves. This flexible and networked nature could mean that there are multiple paths or assessment options for earning a badge, making the system more flexible, ensuring that the needs of each learner are met and limiting the learning path constraints.

I  think this is all very exciting. But I’ll need to keep learning about this badge idea and seeing where and how it goes because my jury is still out. But hey – maybe I can earn a badge for learning about badges. Then I can get an outfit for my avatar that looks something like this:

Just like I say about earning a college degree, maybe it’s never to late to earn a badge!

Stephanie is an Interdisciplinary Studies major with concentrations in Human Sciences and English, Literature, and Writing. She wrote PLA essays on the following topics:

• ED 109: Library Procedures
• ED 114: Reference Materials
• CCM 325: Communication of Self-Esteem
• ILS 300: Literature for Children
• SP 227: Nonverbal Communication
• SP 105: Listening
• SP 111: Public Speaking
• PSY 222: Family and Intimate Relationships

Congrats Stephanie on learning a new way to learn, and on 30 credits toward your undergraduate degree!!!

1. You made your bed and now you have to sleep in it. Meaning: we all need to take responsibility for our actions, even when they come back to bite us. Own it – and learn from it. (Notice how when I say this proverb, I avoid the lay/lie confusion? Even for this English major, that one always stumps me!)
2. Let’s cross that bridge when we get to it. I’ve often said this to colleagues who start worrying about potential institutional changes well in advance of them needing too. A few weeks ago I said this to my 3-year old who was worrying up a storm 3 days before a swimming lesson he was going to. When I said this, suggesting that we postpone his worrying until a more relevant time, he said, “Which bridge Mama? Ross Island Bridge?” Funny, that boy. Good reminder that proverbs don’t always translate well.
3. Hello new day. Ok, so this isn’t really a proverb, it’s a lyric from one of my favorite Roger Clyne & The Peacemakers songs. It’s a nice and happy rock & roll song. I say it every morning (sometimes out loud and sometimes in song) because I need the reminder that I get to have a fresh start every day; that yesterday’s “stuff” doesn’t necessarily have to follow me. Check it out – you’ll want to sing along:

See? Sing along:

Well I feel lucky, I feel cool

What can I say?

Every time I give away a dollar or two

I find three more on the way

Now for better or worse, the whole Universe

Is singing along with every song I play

Hello new day!

Of course when I am not feeling so happy and optimistic, I might be heard grumbling this one, which actually may not be a proverb.

Yet. . .

Mr. Messy is a friend of my kid Mac, who for some reason is intrigued by Mr. Messy. We haven’t even read the book (so in fact I don’t know what it’s about), but we have seen his picture and sometimes it looks to Mac like Mr. Messy is:

a) wrapped up in pink silly putty, or

b) covered in strawberry jam, or

c) “like a flower with arms and feet!”

All of these scenarios might sound pretty neat if you are 3 — eh, hem, I mean 3 and 1/2 — years old.

For us grown-ups, though, messy may not be as fun. Messy means things are not simple, clean, or clear. Messy means being able to live in ambiguity (per my mother’s motto: “We’ll see…”). Messy means that emotions can be involved, that learning is tough, that life is sometimes challenging, that the world can be a difficult place. And as such, messy also provides opportunities. Learning to deal with messy means we get to problem-solve, think differently, find a way, and figure out how to navigate the rough and tumble seas, and then celebrate arrival to dry land and reflect on what worked and what didn’t.

Without messy, we’d likely be bored. And without messy we probably wouldn’t learn much.

To celebrate messy, let me share another goodie from The Heart of Higher Education: A Call to Renewal,by Parker Palmer and Arthur Zajonc:

If higher education cannot deal with the messiness of real life, educated people will not be prepared to use their knowledge amid the complexities and the cruelties that constantly threaten to undo civilization. And they clearly will not know how to use their knowledge with wisdom, compassion, and love. . . If higher education does not help people learn how and why to take the risks of love, its moral contributions to the world will fall short of its potential. (pgs. 38-39)

I like the idea of addressing messy with love because love represents embracing something fully and being devoted and caring. Dean Dad recently suggested that we should “tolerate” ambiguity by reframing it, and he has a great point in asking us to consider this:

Which sounds better: uncertainty or possibility? Failure or learning experience? Internal politics or growing pains?

Um, that’s easy: Possibility, learning experience, and growing pains, please!

Ultimately, for us folks who work and learn in higher education (hopefully there’s overlap there, right?), I think that love — in the way my kid seems to love Mr. Messy — is likely more sustainable and impactful than tolerance. To me, tolerance means I put up with something that might bother me (e.g., I tolerate my husband’s inherent need to reload the dishwasher after I’ve loaded it); love, though, means we can embrace the messy and, as Dean Dad proposes, work to make a difference.

My kid loves Mr. Messy.

And I am trying to, too. Even when he’s covered in strawberry jam (or in this case, nutella).

Meet Mr. Messy: Loving ambiguity one waffle at a time.

]]> http://prattlenog.com/2011/10/06/mr-messy-meets-higher-education/feed/ 0 melaniebooth messy nutella http://prattlenog.com/2011/10/04/this-makes-us-official/ http://prattlenog.com/2011/10/04/this-makes-us-official/#comments Tue, 04 Oct 2011 15:53:02 +0000 melaniebooth http://prattlenog.com/?p=3147 ]]>

Marylhurst University

My colleagues in the Assessment Program and I finally got our program description and content on the Marylhurst website. We’ve been wanting to do so for a very long time, but we needed the time and space to get it all organized. Thanks to the great work of our Educational Assessment Specialist Sione, we’re now present, virtually.

I think that makes us official now, right?

Check it out:  Assessment at Marylhurst University