Still Learning ... through #lrnchat

Where do I spend my Thursday nights?  Well, between 8:30 and 10:00 PM eastern time, I'm usually online participating in a twitter-based #lrnchat discussion.  For those of you who are not familiar with #lrnchat, it is a weekly gathering of workplace learning professionals, hobbyists, academic enthusiasts, and education junkies who get together to discuss learning topics.  A very geeky thing to do I'll admit, but it is a lot of fun.  It has also been an important part of my professional growth and development since I began participating in - and learning from - these chats in October 2009.  I wrote this blog post: I Have Learned ... Socially!!! at the time to share the excitement I had in that experience. I have been returning ever since.

I have to admit, not every topic, every week is a home run, but the batting average is pretty impressive.  And even in those weeks when to topic isn't the greatest, it is still fun to talk and share laughs with the other community members.  But every once in a while, a thought provoking question, a comment, or a discussion thread whacks me on the side of the head and a good idea penetrates my thick skull.  That happened again this week.   The topic was "Using Social Media in Projects."   The first question of the night was, "What projects are you working on that use social media?"   Happily, I was able to contribute a few responses here.  I and my team have been incorporating social media tools into our learning design for quite some time now.  But it was the second question that hit me upside the head:  "How are you using social media to gather formative data about your projects?"   This was a simple, straightforward question.  It absolutely made sense.  But the truth of the matter is I haven't been using social media to gather formative data.  I don't really know why.  I'm usually very diligent about analysis and data gathering in the early stages of a learning project.  In fact, I'm sometimes criticized for spending too much time doing it.  It never occurred to me that I could be using social media as part of my data collecting.  But as soon as I thought about it, it made absolute sense to do so.  I can cast a wider net and likely get a faster response than I can with traditional data collection methods.

The good news is that I can learn.  My company has recently re-outfitted our sales and service employees with an upgraded mobile device.  The time is right for us to begin deploying mobile learning.  We are in the early stages of developing our strategy.  So on Friday, I logged onto my company's yammer network and posted a few questions about mobile learning.  I asked people to share experiences they may have already had with mobile learning, and to share expectations about topics, tools and support they would like to see through mobile learning.   Responses were starting to trickle in by the end of the day.   I will continue the dialog next week.

Thanks again to my #lrnchat buddies for the whack in the head.  See you next Thursday.

Guilty of Learning Malpractice

Last week's #lrnchat discussion was on learning malpractice.  I love to join these Thursday night chats as often as I can. Unfortunately, I was unable to make this one.  What a pity.  From the looks of the transcript it was a lively discussion.  I definitely would have been able to contribute to this one.   After 22 years in workplace learning, I have seen just about every form of learning malpractice imaginable.  And I have to admit, that I am guilty of a few transgressions myself over the years.   Let's face it, at times Corporate America can be a tough place for a learning professional.   Business leaders often expect you to take their underperforming employees and run them through a training session as if it was a car wash. They want them to come out on the other side all squeaky clean and ready to go. In this type of environment, it is hard not to commit a few sins.

I wish I could say that after 22 years, I no longer commit learning malpractice. But the best I can do is tell you that I no longer commit felonies.  I'm still guilty of a few misdemeanors now and again.  Sometimes I do things that I know I shouldn't, just to please my internal clients.   My rationale is that if I commit a minor violation to appease a few key stakeholders, it creates some goodwill that helps us build our relationship.  This way I can get closer to them, their problems, their business issues, and their employee development needs.   I'm willing to make small trade-offs to be able to consult with them on the bigger issues where we can really help them with their business.   Some of you might be cringing as you read this.  You might be thinking, "As learning professionals we should never compromise our standards."  But others of you will relate to what I'm saying.   With the competitive pressures, pace of change, and depletion of resources through downsizings we all face, we do have to pick and choose our battles.

So here I write my latest confession:

Our flagship program is a three-month long onboarding for new sales executives that we first implemented in 2009.  I'm very proud of the work my team has done on this program. It is well designed and well-facilitated.  The metrics for program graduates have consistently been in the desired performance range.  When we first introduced this program, we did all the right things. We thoroughly analyzed the audience, work tasks, and expected outcomes.   We were thoughtful in how we set up our virtual learning environment.  We were diligent in our approach to developing and selecting learning activities.  We had lively discussion and healthy conflict among our stakeholders that helped to strengthen our decision-making.   All of this paid off as the program proved to be both effective, and cost-effective.   That is until this summer when we began rapidly transforming our organization and our workforce.   Suddenly, the number of participants enrolled in each new class began to jump through the roof.  And now we are seeing quite a variety of individuals many of them with very different skills and experiences than that of our original profile.  We had to make quite a lot of adjustments on the fly.   We've made changes to content and activities.  We've increased the maximum class size from 15 to 22 and added 25% more classes to the schedule.  Even with the additional classes we are still hitting our newly raised ceiling of 22 every time.  We've pressed new people into service to facilitate the additional classes with less preparation than we would like them to have had.

On all these counts, I'm guilty of learning malpractice.  But I throw myself on the mercy of the court.   I stayed as closely aligned to my business leaders as I could.  I wanted to stay alert and prepared to make changes as needed, but they just came too fast and too furiously this year.  Our onboarding program now resembles one of my local roads after a Connecticut winter: broken up with cracks and pot holes.  We've been filling them in as best we can, but it is still a bumpy ride for our learners right now.   In November, we are reconvening the team to discuss repaving the road for 2011.  I hope I'll be done serving my time by then.

Obsessing over Assessment

The law of attraction is at it again. Testing and assessment is running through my mind and it seems to be running through my life as well. There is a certification project currently being implemented in my company that involves training and testing. I am taking a course on Inquiry and Measurement that also involves testing. And, during this week’s #lrnchat, I was drawn in by a discussion thread in which the pros and cons (well, mostly cons) of assessment were being discussed. Why am I so preoccupied with Level 2 of Kirkpatrick’s Four Levels of Evaluation? It is mostly because of the certification project at work.

Certification is a term that gets thrown around a little too loosely in training departments these days. Put someone through a course and a post-test and BANG, you are certified – or worse, you are not. It is a tricky thing to put together a certification process that is valid and reliable. Very few companies can stand to wait out validation process, so they jump right in and begin training and “certifying” people.

Certifications do not guarantee that the person being certified has learned more than he or she would in a regular training program, but business leaders often feel it is operationally necessary to validate a level of knowledge or skill required to meet goals and targets. If certification is needed or required, here are some things to consider for the assessment process:

Test items should be directly related to learning objectives, which should be directly derived from performance requirements. This may seem obvious but I have seen many tests that have included filler material alongside valid questions.

Test only on important items, not obscure ones. It is not necessary to test someone on small details unless they are critical. Very few corporate employees are doing life-saving work that needs to be tested at a granular level.

Test items should be straight-forward. Don’t try to be tricky. What is the point? It only serves to confuse the learner and adds no value to the assessment.

Choose question formats that make sense for the level of learning you need to assess.  Multiple choice questions are commonly used on knowledge tests because they are easy to score and easy to tie to outcomes – but they are not always easy to write. Good multiple choice questions will have a clear premise in the stem of the question, a correct answer, and reasonable alternative choices. There should not be any throw away responses or convoluted choices such as “a and b, but not c” or “a and c” only. If your test item has more than one correct answer, then rethink the question format. Consider short answer questions or a matching column.

If possible, use randomized test questions. Most learning management systems have this capability. They allow you to create a bank of questions that can be drawn upon at random so that test-takers will be deterred from sharing answers. But don’t make the bank of test questions so large that everyone feels like they are taking a completely different test.

Pilot your test. This is the hard part, because it takes time and patience. You need to let a few people complete the learning experience and take the test to give you the opportunity to analyze the questions. You will want to take a second look at questions that everyone got right, or everyone got wrong, or questions for which many people chose the same incorrect answer.

Create rubrics for skills assessments.  Skills assessment usually requires direct observation.  It is important that all of your assessors are using the same criteria and weights when judging performance.  Validate the process by having multiple assessors review the same performance.  If they are more than a few points off from each other, either redesign the rubric or re-train your assessors.

Tweeting at Conferences: Etiquette vs. Impact

I had an epiphany during this week’s #lrnchat. Participating in these discussions has been tremendously helpful in my quest to understand the value social media tools provide for learning.

This week’s chat started off with questions about how media sharing tools (such as Youtube) and collaborative content creation tools (such as wikis) are impacting corporate learning. As this discussion ran its course, the next question that appeared in the chat scroll was about the role of social media at in-person conferences. In response to this question, one of the chat participants made the suggestion that we should “bring the backchannel forward” in conferences. He stated that, "social media does not have to be a post-event activity; instead it should be an attendee."

After reading his post, I pictured a speaker in front of a room full of people with Blackberries in hand, eyes down and thumbs flying as they twittered messages back to the members of their respective online communities. The next image that came to mind was of my teenage children pulling out their cell phones to respond to text messages from their friends at inappropriate times. Being a relatively new social media user (I started blogging and tweeting in August) I viewed these two situations as much the same, so I naively typed in this question in response to his post: “Isn’t it rude to tweet during conference events?”

My question garnered some interesting responses, such as: “Not at a good one,” “It’s engagement,” “These days it seems like an insult not to,” “It extends the conversation,” and “Is it rude to take notes?”

Reading these responses made me realize that I need to change my frame of reference. My “pre-social media” point-of-view was that it would be disrespectful and disruptive to do anything other than give the speaker my full attention. But as I thought about it some more, I thought, “Of course the speaker would want people to share his or her message beyond the conference-goers.” Then, I began to calculate the impact:

Imagine a conference presenter speaking to an audience of 100 people. Let’s say half of them are social media users. And out of that half, half of them were interested enough to tweet the speaker’s message back to their online communities. That would be 25 people extending the message beyond the in-person audience. Now let’s say that each of those 25 people had 100 followers on twitter. The speaker’s message has now potentially reached 2500 people. Even if only a small percentage of those 2500 people retweet the message it still extends the impact even further. And these are very conservative estimates. I currently have 88 followers and I have only been a twitter user since August. Factor in long time users who have huge followings and the numbers grow exponentially.

As a result of this, I have revised my mental model of social media activity at conferences.  Far from being rude, it is actually complimentary to tweet during presentations.  And never mind about about it being disruptive, it actually helps the speaker be more productive. So, next time you are at a conference, don’t be surprised to see me in the audience at with my mobile device in hand. I promise it won’t be because I am texting home to find out if we are having meatloaf or chicken for dinner.

Five Examples of Learning Metrics that Matter

This week I participated in a discussion on learning metrics with my new friends at #lrnchat. What a pleasure it is to be part of these weekly events. Where else can you say things like, “if we start analyzing the problem using the Six Boxes® model based on Thomas Gilbert’s theory, we are sure to make some progress” without people running away to see if someone refilled the punch bowl?

The questions that were tossed around this week were:

• Are learning metrics different from business metrics?
• What are you measuring in your organization?
• How do you tie organizational learning to business performance outcomes?
• What else besides metrics do you use to show impact?

I was excited and a little bit surprised as I watched the responses roll by. I get so caught up in what I’m doing at work that I lose sight of what is going on in the rest of the world. Some people seemed to be struggling with measurement basics, while others seemed to have a strong grasp on the topic. This got me thinking about what we are doing at my organization, which probably falls somewhere in the middle.

I thought a lot about this discussion after it was over. Really, the heart of the matter is: What value do we as learning professionals bring to the organization? How do we show it? The short answer is that we do this by helping business leaders meet their goals.

I have been fortunate throughout my career to have been involved in a number of projects that made a huge impact on the business. But I can also recall times when it was difficult to see connections between the work that I was doing and the impact it was having on the organization. These are the times when we all fall back on measures such as, “butts in seats,” level one evaluations, and elearning completion rates. These are important feedback indicators for the training department, but they usually don’t mean very much to business leaders. That being said, I can think of examples in which these measures alone have been important to the business. What it comes down to is that sometimes the measures associated with learning success will be obvious and glamorous, and other times they will not. Here are a few examples of both from my experience:

1. Sales Training is one of those cases where the metrics are obvious and easy to align to the business. We run a class on how to sell against our competitors effectively. The natural metric associated with this is the number of competitive takeaways. We have clear evidence that sales reps who take this class have a higher rate of competitive takeaways than the general rep population.
   
   
   
   
   
2. Equipment Service Training is another example with obvious metrics. Well-trained service reps fix problems more quickly and are able to tackle more service calls in less time.
   
   
   
   
   
3. Operations Training can have a variety of metrics. We ran a class on how to write work instructions for a part of the business that was very procedurally oriented. Every time they had people leave the department due to promotions or turnover, processes broke down. Our metric was the existence of well-written work instructions in the departments that participated in the training. This helped the business leader keep continuity when there was turnover.
   
   
   
   
   
4. Compliance Training metrics are not very glamorous. This is a case where counting up completions works to serve the measurement need. Our legal department wants to reduce or eliminate ethics violations, or failing that, be able to prove the company did its part in making employees aware of their responsibilities and the consequences. We make sure everyone goes through our business ethics course and provide a report shows that this has been done.
   
   
   
   
   
5. Soft Skills Training metrics are of course the fuzziest area. Yes we conduct level three evaluations to show behavioral change, but most people don’t get too excited over these. However, our Chief Human Resources Officer is very interested in employee engagement. She knows that offering soft skills training helps people with personal development which is important to engagement. She also knows that managers and supervisors who use what they have learned in soft skills courses are going to receive higher engagement scores. So yes, we count “butts in seats” for these courses.

The point I’m trying to make with these examples is that the metrics have to match the need.  When we help our organization’s leaders meet their goals by identifying and addressing the learning component of their business problem, we are truly adding value.

I Have Learned...Socially!!!

After two months of blogging and tweeting, I have actually begun to learn something through this experience.  Here is my story:

On Thursday night I participated in a #lrnchat session on Twitter with other people interested in learning.  The theme of the chat was, "learning from each other socially in different ways."  I logged on a few minutes early and was politely greeted by a few others who were already there.  At the appointed start time, the chat leader began tweeting the rules.  They appeared on screen one by one, interspersed with greetings from other chatters who were just arriving.  There were seven rules in all.  The gist of which I have repeated here:

1. Introduce yourself and your location
2. Share your favorite topics
3. Write complete thoughts so those seeing your tweets outside of the chat will understand them
4. Play nice (but sarcasm was welcomed!)
5. Periodically retweet so others will understand the context of your responses  (I didn't get this at first, but I got it now)
6. Remember to include #lrnchat in all posts (Putting this at the end of your tweets includes your post in the chat stream)
7. Five minutes before the end of the session, let the group know if you need anything from them and then close by reintroducing yourself

After all the rules were read, several threads of discussion took place.   I paid attention and participated in a few, but there were several others that went right by me.

The first thread I followed had responses to a prompt asking, "What have you learned this week?"   Some learned about instructional design, some about twitter; I typed that I learned there is a community of learners that meets through Tweetchat on Thursday nights.

Then we began to discuss social learning.  One person suggested social learning is the convergence of ideas. Someone else suggested that all learning is social.  Several people refuted this with examples and testimony.  I asked, "How does social learning fit into the corporate learning world with its security restrictions and data privacy concerns?"  One person sent me a direct response saying, "It doesn't...it's the very antithesis...uncontrolled by "the man" and self/group constructed."   I thought that was interesting, since one of my goals is to learn how to bring these tools into the corporate learning environment.  I'll be thinking about this one for a while.

Someone asked, "If Socrates coulda' tweeted...would that be social learning?"  This garnered the response: Socratweets!

After all of this, a good discussion thread got started on learning measurement.  Certainly it is difficult to get a handle on the impact that learning through social media could have on a business.  It was clear that this issue creates a barrier for acceptance in the corporate learning world.

In all, the chat lasted ninety minutes.  I had fun participating in it and afterwards - as a good learner should do - I took time to reflect.  So here is a brief summary of my thoughts about my experiences with social learning this week:

1. I have been blogging for about two months now.  During that time I have been reading and connecting with other bloggers.  It is through one of these connections that I discovered the existence of the #lrnchat group
2. By linking back to another blogger who read and commented on one of my blogs, I learned some of the vocabulary and syntax used to make tweeting more effective
3. Through #lrnchat, I learned that there is more to twitter than just isolated tweets
4. I learned that twitter can be effective for learning if you have people with common interests all tweeting together at the same time
5. I learned you can make your tweeting more meaningful by using techniques like retweeting, direct replies, and writing complete thoughts

So I think I'm beginning to get the idea of social learning.  I'm not ready to include it in my instructional strategies at work yet, but I'm hopeful that I will be able to at some point.

For you curious social anthropologist types, here is a link to Thursday's #lrnchat transcript.  Be warned, you have to navigate through the zigzag nature of the discussion threads.  As a matter of fact, someone in the group suggested coining a new term for this: Zigzag Learning.  You can also see my contributions to the discussion in isolation by going to http://twitter.com/mpetersell  and viewing my posts dated Thursday, October 1.