Product Hero: Jason Fournier, VP of Product Management at Pearson MyLabs
Preparando audio para descarga.
Escucha patrocinada. El audio empezará en pocos segundos...
Escucha sin anuncios y sin esperas con iVoox Premium
Pruébalo GratisiVoox Podcast & Radio
Comparte éste audio
Enlace directo
A continuación: Product Hero: Barbara Dumery, SVP of Product Management at Imprivata Cancelar 10
Preparando audio para descarga.
Escucha patrocinada. El audio empezará en pocos segundos...
Escucha sin anuncios y sin esperas con iVoox Premium
Pruébalo Gratis
Product Hero is our bi-weekly series to highlight outstanding members of the product management community. These industry leaders share tips on processes, team building, how to be a better product manager, and who they are outside of their careers. This week our product hero is Jason Fournier, Vice President of Product Management at Pearson MyLabs.
Listen to the Show: http://traffic.libsyn.com/thedirt/Product_Hero_Jason_Fournier.mp3 Download the MP3
Subscribe through RSS
Subscribe on iTunes
Subscribe on SoundCloud
Subscribe by E-mail
Show notes: Connect with Jason on LinkedIn.
Pearson’s Learning Design Principles
Ralph Koster’s Gritty Systems Design for Retention
Storytime with Jonathan Blow at PAX East 2016
Podcast Transcript: Jill: Welcome to The Dirt. My name is Jill. I am here with Jason. He is here from Pearson. He’s VP of Product Management. Welcome, Jason.
Jason: Thanks. Thanks for having me.
Jill: Thanks for coming out here today. Tell us a little bit about yourself.
Jason: I’m VP of product at Pearson. I’ve been there for 12 years. I’ve been all over the company. I’ve worked in the K-12 division. I’ve worked in higher education. I’ve been a product manager for more than half of that time. I’ve worked on a lot of big platforms, small products, new mobile efforts, big enterprise things. Really just spend lots of time trying to become a better product person.
Jill: We always like to get a little bit of personal context on someone instead of just jumping straight into business. One of my favorite questions is what’s one product that you really admire that’s not in the education space?
Jason: Right now I’m a big fan of Nintendo. They just launched the Nintendo Switch and I think what they’ve done with the format, the different style with the joysticks, the portability of it, the way they’ve tried to incorporate shared play together but not necessarily making everybody look at the screen or everybody be passing the joystick around I think is really cool.
As a company that’s been around for a long time they have a pedigree in the space but they’ve also struggled. I think it’s really interesting to see them try to reinvent themselves with the Switch. The detail that they put into building that device and the unique way that it forces people to engage with each other is really fascinating and interesting. If you’ve had a chance to play with them it’s a lot of fun too.
Jill: I haven’t but now I totally want to go play with one of those things.
Jason: You should look for a demo station. Check it out. Target, Best Buy, somewhere. They’re a lot of fun to play with.
Jill: Yeah, I’ll have to polish my Mario Kart skills. It’s been a while.
Jason: There you go.
Jill: You’ve been at Pearson for 12 years now. It sounds like you’ve worked in a lot of different areas there and you’ve kind of worked your way up in the company. Tell me a little bit about what led you to Pearson and how your progress through Pearson has been so far.
Jason: Sure. In college, we had a teaching learning technology center. At the time it was in Pennsylvania. I went to Juniata College. Everybody got paid minimum wage, which was $5.15. No matter what you did at the college, if you worked as a janitor, if you worked in a computer lab, you got the same money.
They were trying to find a way to incentivize students to get involved in some higher order types of work. They created this position to run the teaching learning technology center. I got involved in that. It started out it was just like manage the lab, make sure the keyboards are clean, and over time we got involved in bringing in new technology and helping instructors learn how to use it in the classroom.
By the time I graduated I was getting called out of class to help fix things or help instructors figure out how to use their PowerPoints more effectively. I got a real taste for what it was like to try to apply technology to education.
I graduated right around the time that the dot com boom burst. I was competing with a lot of other people for work who had been in the market at startups. I got into publishing. There was work there. It was not highly competitive. I was able to apply my IT skills to that. I’ve been in education ever since and been in the publishing space ever since.
At Pearson, I started out doing just customizations of content and really learning how content production happened. Then I moved into product and I started to think about how we were building products for customers, how we were delivering educational technology into the classroom, and all of the different ways that it could be used.
You have some instructors who really are afraid of technology and they’ll take the technology and turn it around and point it at the wall so that it’s not looking at them the whole time. You have other instructors that are constantly trying new things and looking for ways to improve the experience and everything in between.
For me, it’s a real passion. Education has helped me move forward in my life. It’s something that I have felt the impact of and my family the same experience. It’s an area I really believe in. It’s an area where, at Pearson in particular, we have a lot of opportunity to use our size and scale to change education in a worldwide way. I think that’s really cool.
I’ve been all over. I’ve worked on products for K-12 math where you can’t have a password because kids are just not old enough to be able to type one in to college products for the nursing space. We’re talking about simulations and trying to give students the ability to experience in a low-risk way the types of things that they’ll experience when they’re out in the clinical space. It’s great.
There’s so many ways that you can be making an impact in education, where you can really make a difference, and so many different disciplines to think about from hospitality and tourism to mathematics all the way up the curriculum. There’s always something new to work on.
Jill: Yeah, it definitely sounds like you don’t get bored. There’s enough versatility within the education space that you could do whatever you wanted and be really engaged and making some really cool progress.
When people hear about Pearson they think, Oh, the textbook company. You referenced content earlier being the product. I think there’s been a shift over time. You’re specifically working in a digital products division where content isn’t necessarily as much of the product and maybe it’s more of a SAS-based product. Can you tell me about what that transition has been like and the things that you’re doing to help that transition happen?
Jason: The transition is complicated. I think if you look at the music industry or the movie industry or now even the television industry as it’s disrupted by companies like Netflix and Amazon, there’s been a lot of transition in content and media-based industry.
For us in particular, we’ve been a little insulated from that because institutions are slow to change. As they think about what they’re going to do that influences the products that we bring to market. There’s always this tension between trying to innovate but not get out ahead of your customer base.
The content piece for us we focus on best of breed content. We want authors who have experience in the classroom, who have unique and novel ways of teaching, and who can help us turn that into a product that uses digital to make it even more effective and to scale it. Our product strategy is really content plus assessment, powered by technology and services, to deliver education at a scale.
For an industry like ours where you think about content it’s easy to think about Wikipedia or open educational resources and the impact that’s having. We really believe that there is value in curated content and in that space of making sure that the content aligns to the curriculum, that it has common voice throughout.
We also believe in using technology to deliver that in new and unique ways that helps students make progress against their outcomes. That’s really for us we’ve put a lot of investment into efficacy. Being able to measure in third-party, validatable ways that the products that we’re putting into market are really helping students make progress.
The transition has certainly been hard and getting the balance of what’s the value of the content versus the value of the technology is challenging. We think that there is still value in the content and we think that value increases when you leverage it in unique software experiences that help learners.
Jill: Yeah, yeah. I like that idea of content plus. You’re building on top of that. It definitely makes sense in the information age that we’re in that having the right content in front of you instead of having to go look for it that just saves a huge step for the professors and the instructional staff and then as well as the students in their learning process.
Jason: The other thing that I would add is institutions all have different amounts of resourcing. An institution that’s really large, that has 150 person instructional design staff, when they work with us that’s totally different than a small community college where there’s a lot of adjuncts and they’re trying to just figure out how to get their students to be successful.
One piece of it is just how do you provide the resources needed for institutions all across the continuum. The other thing is that students come to school for a whole bunch of reasons. I’m sure if you think back to your time in college there were classes you loved and you were super engaged and you wanted to do really great work and it felt natural because they were just topics that were really interesting to you.
Then there were the classes you just needed to get through. Your definition of success in both of those is really different. Like, I just want to pass this class so I don’t have to pay for it again versus the class where you love and you get an A just as a side effect of just loving the material and loving the domain. Those are different things. For one student you can have any mix of those.
I think it’s really challenging to figure out how to bring products to market that meet the needs of a class of students where every student in the class might be there for a slightly different reason. They might have a slightly different goal in mind or motivation that’s leading them to engage the way they are.
Jill: That’s one of the questions that I had. When I think about education, and education is getting more individualized, and that goes in line with design. When you think about great product design it’s very customer or user-centric. When you have a group of users that all have to use the same product but then you have all these individual needs, whether it’s a learning-based need, how they learn, or their level of interest, how do you balance that out and design for the user in that circumstance?
Jason: It’s a good question. I think there’s a question in education that parallels that that has to do with what’s the role of the educator if the technology can do everything?
I think those go hand in hand. The first thing I would say is there is always going to be a role for excellent teachers. That’s just the way it is. For the people who can help you make the connections and can help you find a way through the material that makes sense to you so that you can connect the dots and connect it to the real world and make it actionable for yourself there’s always going to be a role.
We see our job as helping, in some ways, instructors scale themselves. Instructors are strapped for time just like everybody else. They want tools that help them teach more effectively. They want tools that might auto-grade the basic content so that they don’t have to spend their weekends assessing quizzes and instead they can be thinking about the next debates they’re going to drive in the class to get students to think about the material differently.
Part of our goal is to personalize for the student in ways that help them understand the material at a level that a good instructor can then help them build on that. If you think about the social sciences, flipping the classroom, giving the student the tools they need to be able to master the material before they come to class so teachers can use the class time differently.
Use that time to have a great lecture that brings in current events and helps students connect the in-class material to what’s happening in the world. Or helping students have a great debate with each other about different perspectives on a topic. That’s where a lot of education becomes really meaningful. Personalization is about trying to help students through the material so those other great experiences can happen.
At the same time I think, again, if you reflect on your own education there was always that skill that you were a little weak on that was a prerequisite for the next skill that you needed to learn. Personalization can help every student no matter where they are stay caught up so that they can keep learning and keep being effective.
I think it’s actually really easy to balance those things in the sense that there’s a mix in education. There’s always a mix of what you’re trying to learn and what you already know in combination with the connections that you wouldn’t make if it weren’t for someone else helping you through that or bringing you around to a perspective you hadn’t imagined. That’s part of it.
The other thing is that if you look at technology, you look at machine learning, you look at big data, you look at all of the new technologies that are out there, there’s an opportunity for us to leverage those in education to help with some of the problems that we’ve had, whether it’s how do you help a student study more effectively or how do you assess a large group of students and understand what’s working and what’s not working. There’s benefit there.
It becomes this real virtuous circle where you put the product into market, you collect data, through that data you can make the product more effective, and thereby help students learn more effectively. An example that I really love we’ve been doing a lot of item response theory analysis of our test items. You can find the items where a great student is more likely to answer incorrectly than an average student, which indicates that the items are broken.
You can do an analysis of the data where you can see across all of your student responses is this item really indicative of student mastery of a concept or is it not working in some way? By using big data we can improve the product but to improve the product you need to have lots of students go through it so there’s scale in catering your items to a broad student base. Then you want to use those items in a targeted way to help any individual student given where they’re at to help them move forward. This is a topic I can spend a lot of time on. I think it’s really an interesting part of educational product design.
Jill: That’s awesome. When you were telling that story my mind instantly went back to when I was in college and I would have a multiple choice question in front of me and I’d be like, Oh, it could be either one of those two. Then the professor was hard, No, it’s C. It’s not D. Just wavering that out so that the critical thinking is still taking place.
Jason: That’s right. That’s right. Critical thinking has to take place but if you just have answers that are confusing or they become more confusing, to use your example, where you’re really pondering those and that’s what makes them confusing when you get into them, deep into them. You can have items that just don’t work where a great student who is trying to make those connections is actually less likely to get the answer correct.
The goal of items like that is to really understand something about what the student knows after giving them that item. If they’re not working then you need to know that. Historically, it’s been hard to figure out. In today’s day and age of being able to collect all of those things … Even down to in cases like did you click one answer and then change it? How long did it take you? Was there a pacing pattern there?
We’ve been doing some research recently where we have found patterns of students skipping through questions to get to the spot where they get the feedback and then they’ll go back and try again. There’s patterns of behavior there that you can identify with the data that, again, if you think about an ethnographic study of watching students take tests you would recognize that behavior because you might have done it. When you see it in the data and you start to analyze that and think about how you can use that to influence future design in subsequent versions of the product it becomes really interesting and there’s a lot of cool insights there.
Jill: Yeah. I’m curious. Going down this story path you find an item that can be improved. Is that something that you then have to go back and convince the professor that they need to change their process too? How do you balance that?
Jason: Yeah, that’s not really one of those spots. We do have those spots. In that case it’s about content quality. We’ll look at the item, we’ll recognize that it’s not a well-authored item. A lot of cases those are items that we’ve written. We don’t really do a lot of analysis of instructor-written items right now. We’ll just update the content.
In other cases, we know that one of the biggest impacters of performance in the classroom is how the product gets implemented. If you have a homework system think about all the ways you might use that. You could assign one homework before every class. You could make all of that homework a big part of the student’s grade, a small part of the student’s grade. That’s all going to influence whether the student spends a lot of time on it, a little bit of time on it. Implementation has a really big impact on whether the product is effective.
In that case, you often do run into the situation where you want to help an instructor think differently about how to use the product to be more effective. Instructors really value academic freedom. That’s one of the great things about higher education is that different instructors teach topics in different ways. You can learn different things by going through different instructor’s programs.
For us, there’s a balance there between saying, Hey, if you do it this way you might get better results which we obviously all want and trying to enforce our perspective on the process of teaching and learning. We want surface insights that instructors can use if they so choose to change the way that they do things in the class. We also want to recognize that each class is different and that each instructor has a different approach and we want to leave the door open for them to apply that.
Jill: Yeah. That makes a ton of sense. You’ve talked a bit about the education space. I’m curious. As a product leader, what are some of the challenges that are keeping you up at night. You have a huge team that you manage from what it sounds like. What are the things that motivate you?
Jason: Yeah. A couple of things wrapped up in that question. I think the things that keep me up at night are around how we stay relevant as all this change happens. Change is a constant now. It’s always going to be there. Our products have to work harder and harder to keep up and to make sure that we’re embracing new technology, that we’re trying to leverage new features, and try them out, try to leverage new processes.
In education, we have a very crazy cycle in that there’s really two main points where you can release. Everybody else in software is trying to release as fast and as often as possible so that you have smaller, less risky releases and you’re constantly delivering value to your customers. We had a really hard time with that because customers adopt the product ahead of fall semester and then they teach first semester. If you change too much along the way it can influence the student’s ability to learn.
I think what keeps me up is trying to understand how we can apply all of these best of breed approaches to product development and product design and also serve our customers well in the market that we’re in. It’s really hard. There’s a new flavor of product management process all the time. The one that we’re really focused on now is Jobs to Be Done theory and really trying to understand what job our product is being hired to do by the customer.
Really digging in on that is hard to do. It’s easy to stop at the surface level and not get the insights that you need to understand how one customer is different from another and how to tailor your product. We spend a lot of time trying to figure out if we’re doing that well. There’s not a lot of good benchmarks out there.
Really just trying to be a good team that’s using the best processes and the best approaches to be effective. That’s something we’re always working on. I want my team to be as effective as they can be. That’s always something that I’m thinking about. It’s also one of the things that makes me passionate but I think the real thing that I am passionate about is making a difference in people’s lives through learning.
If you can bring those things together the knowledge we have about education and a passion for product development and understanding your customer and responding well you can have a lot of fun doing what we do. That’s true in any market. Mine just happens to be education. Anywhere where you’re really trying to understand your customer and solve their problems and bring to bear great product development and expertise and approach you’re going to have fun. That’s something that I’m passionate about.
Jill: I’m curious. We also use Jobs to Be Done here quite regularly. I’m curious if you have any examples of a time where that tool was specifically useful to you?
Jason: Yeah. That’s a hard one to answer because we haven’t had a real win where we’ve used the process and we came to a conclusion we wouldn’t have come to otherwise. That sounds a little harsh to say. I think we’re just early in using it. Unfortunately when you’re early in using it you tend to use it to validate the things that you know.
On the other hand it’s really forced us to think differently about how we think about our customers. If I think about the biggest way it’s changed our thought process it’s in adding the emotional aspect. Understanding that an adjunct who wants great reviews so that they can stay on their career track is going to do something different in the classroom than a tenured professor who is really interested in their research and is really excited about the research project they want to get back to.
If you can think about what emotional context is motivating your customer it leads you to different insights about how to guide your product as you develop it. I don’t know that we’ve had a real win where Jobs to Be Done theory has changed the product yet. It has caused us to just think differently about the customers that we’re trying to serve.
Jill: That’s awesome. That’s huge how you think about the customers. Especially I’ve worked with enough different organizations where you kind of get some entrenched opinions or stereotypes of your customers. That can be really dangerous. Having something that busts that open a little bit and reminds you, Oh, yeah. We’ve got more than just that one stereotype is a really good thing.
Jason: Yeah, it’s true. I’ll connect it back to your question earlier about balancing the needs of the one against the many. It’s a slippery slope. You have to be careful that you don’t do the Jobs to Be Done analysis and then find this one unique special snowflake customer. That can be a challenge.
I think the underlying piece is how do you identify the themes then? How do you do enough research in the right ways to find those themes and then create fit for purpose solutions that solve those jobs? It’s really exciting. I think we’re still pretty early days in that process. It’s really influenced the way we think about customer engagement.
Jill: Very cool. One of the other things that you talked about you’d said earlier that in the education space it’s a little bit slow to change. Also, as you look at how the next generation of students how they are learning, the proliferation of mobile devices, and just access to information online, that’s moving really quickly but you have institutions, which are a little bit slower to change. You’re sitting in the middle of those two things, right?
Jason: Yeah.
Jill: How do you manage those two variable timetables?
Jason: That’s a really hard question. If you have the answer to it let us know. When you ask a question like that often folks are thinking about Con Academy or I’ve referenced Wikipedia earlier. The common refrain is, Well, calculus hasn’t changed in 50 plus years so why do we need a new calculus book? Why do we need a new edition of the book?
That can be true in some ways. In other ways, it’s like you need the test banks to be updated so that students are not cheating. You need other resources and assets to be updated. There are also approaches to learning that have changed. Pearson has a set of learning designed principles that we released under Creative Commons license a few months back. I’d be happy to share the link with those.
As we learn more about the brain and we learn more about how people learn and we learn more about how they make memories and form connections we want to be updating our educational products to take advantage of those things so that students can be more effective. We think we have the resources and the scale and the expertise and learning and efficacy and design to be able to capitalize on that. You don’t always get that in free resources or in other areas. You’re often looking for somebody to help apply that on top of the resources.
In a different way, my lawn mower stopped working last summer. I did some research online and I found a website where I could buy a carburetor kit and it came with a link to the YouTube video that told you how to rebuild the carburetor yourself. I sat there with my iPad next to my lawnmower and I tore the carburetor apart and I put it back together and dragged the lawnmower out and it started right up.
Jill: Nice.
Jason: Education is definitely changing. The ease of which we can access material like that and have it make a difference in our lives is also changing. Cooking is another great example. There’s tons of cooking shows, there’s all kinds of recipes, there’s videos on Facebook. You can’t open Facebook without a little video popping up of somebody making a crazy dessert.
You’re right to point out that educational resources are everywhere now, whether it’s math at Con Academy or it’s cooking, there’s tons of stuff out there. There’s always going to be a need for someone to help you with it. There’s gym equipment everywhere too but a lot of people still go get a trainer or sign up for a gym membership to create that little built of guilt that keeps them going back and working hard. I think instruction can be the same way. A good teacher can help you put that material together in an important way and can keep you accountable for your learning in ways that are important.
Jill: Even structuring learning experience so that it is tailored to the latest learning theories and learning approaches. I have a friend whose son is in elementary school and he’s like, I was helping my kid with his math homework and the way that they’re teaching mental math now is completely different from the way I learned. He’s teaching me how to do it in a different way and it works. He can do it in a way that I never could when I was that age. Just seeing those new learning things applied to the course, to the system, that already exists and making sure that it’s as awesome as it can be makes a ton of sense.
Jason: That’s right. Yeah. That’s it, right? There’s educational strategies. Like how you learn a given topic. I think that’s becoming more available too. There’s been a big increase in ways to learn playing guitar. There’s all kinds of tools. You can get a fake guitar you can plug into your Xbox. You can go on YouTube and buy videos. There’s tons of different approaches. You’re able to get access to teachers that you might not otherwise get access to.
It is changing in some fundamental ways and applying different strategies is a key part of it. I think there’s probably more disruption to come in terms of how we actually string the material together online to teach. That’s a great story. It’s one like some people will say that they hate the new ways of teaching math, others will say it’s more intuitive. I think in and of itself those examples are proof that there’s different ways to learn and we’re part of an ecosystem.
Jill: Yeah. Very cool. Nice. What are the things that you’re working on solving right now at My Labs?
Jason: Personalized and adaptive learning is really a key focus for us at the moment. Understanding what adaptive means because everybody has a different idea of what adaptive looks like.
Jill: Let’s assume that not everyone who is listening to the podcast, including myself, has a clear understanding of adaptive learning. How would you describe that?
Jason: Yeah, so I think about adaptive as any tool or technique that you’re applying to customize the experience that the student is getting. Now some people will be much more specific and they’ll say, It’s algorithms and it’s application of those algorithms to the learning graph so that you can better place the student in the domain and understand where they’re at and then give them a pathway to achieve on the outcomes that you’re trying to drive against.
Some will put it within a certain context. It’s a study plan or it’s some other tool. It’s a little bit all over the place. I think there’s a scale to it. It’s easy to do it in a very choose your own adventure kind of way. It’s much harder to build a very sophisticated learning profile that takes into account everything that you know about the learner and then tries to help them forward. Even harder still to do that cross-course, which is kind of the Holy Grail.
How do we take into account everything that you know in all of the domains and use that to guide you? The reality is that with any given specific course it’s often times not the material in that course that you’re struggling with. It’s the prereqs that you were supposed to have when you got into that course and so the ability to make cross-course connections I think is one of the places where there’s a lot of opportunity with adaptive still out there.
That takes a paradigm shift. College is very course-oriented right now. Trying to break that down and think more holistically about all the topics that need to be well-understood for a student to master the outcomes they’re working on is really important.
I think adaptive that’s the one I would say we’re really focused on. The other thing that we’re really trying to do a better job of is deliver consumer grade experiences to customers, whether you look at the finance space or the health space or education those industries have been a little slower to keep up with the types of experiences that people have when they use their Comcast X1 remote to say what show they want to hear or they use their iPhone or their brand new Android phone.
Consumer experiences have just come so far, so fast and we haven’t really been able to keep up. The other big thing is just how do you create smoother, lower friction experiences for customers so that it’s more about the learning and less about learning the tool to try to learn?
Jill: That makes so much sense. I was just talking with another client about this. You’re not competing with some of these other products but you have people who are using really, really easy to use tools like Slack. We were just talking about that before the podcast kicked off. Once you start using something regularly your expectations are then raised. It doesn’t matter if that’s not a direct competitor. There’s always going to be, Well, why isn’t this as easy for me to use as this other thing?
Jason: That’s right. Yup. That’s right. It’s not just ease of use. It’s up time and it’s bandwidth. There’s so many things that we now take for granted that if you have a legacy product that you’re trying to move forward you have to take into account and figure out how you’re going to respond to those things. It’s very easy to look dated these days if you can’t keep up with how quickly interfaces are changing.
Think about how quickly the hamburger icon became a common icon and interface staple. That happened so fast now that it’s really important for us to try to keep up because our applications stand out when they don’t. That’s a challenge for us. Adaptive in the learning space I would say that’s the big one that we’re focused on. Consumer grade experiences in terms of an overall product focus would be another one.
Jill: Yeah. When you were talking about the adaptive learning it kind of occurred to me that that must take an incredible amount of testing to get that right. How do you implement testing? Especially when you’re trying to serve so many different user groups across the world and student groups across the world at different learning levels?
Then at the same time this is one of those situations where I do think it’s a higher risk test if you’re putting it out there. It’s not like, Oh, you’re not going to be able to watch this Netflix movie right now. This is someone’s education. That’s something that’s pretty serious. How do you implement testing when you’re exploring something new like that?
Jason: Yeah. It’s a good question. I think that we try to think about testing in a variety of ways. There’s product testing, which we probably do the same way everybody else does. You build your product, you have your user acceptance testing, you have your QA process and your test plans, and you’re going through all of that and you have automated tests, which you’re running. That’s your basic functionality level.
We try to do a level of class tests and pilots that we can use to try to understand if it’s working the way that it should work. Sometimes that’s a small scale test so you do a chapter or two. You’re lower on the impact of the student’s overall course experience but you’re still getting some important feedback. We try to do much more rigorous testing so you have a control and you have a class where you’ve introduced the product, you try to understand the difference in performance.
It is challenging. I’ll be the first to admit that I have a really difficult time trying to figure out whether education is more like medicine or more like a consumer grade product. Do students want to have A/B testing done to them when the kid next to them might get a better grade because they got a tool that they didn’t get?
I wouldn’t have wanted that. I would have been really upset if there was any chance that another student in my class got something that made them more effective than me. On the whole, we want education to improve. We have to work through the challenges around these topics. For us, it’s trying to manage the scale early on. As you have more confidence that it’s working, you can ramp the scale up.
I’ll bring it back to the efficacy work I mentioned earlier. We have an efficacy framework that we’ve introduced that talks about the evidence you’re collecting, the outcomes you’re trying to achieve, the plan that you have to get there, and the capacity you have to deliver, and the capacity your customers have to implement.
We’re trying to use that framework to really better test our products and to prove the outcomes are being achieved the way that we say they’re being achieved. We have a goal to be able to report on that with the same level of accuracy that we report on our corporate finance.
We’re bringing in third-party auditors to look at the data and we’re doing real scientific testing to get our product into the classroom and to collect the data and work with partner institutions to measure the performance and then to make sure the products do what they’re supposed to do. Sometimes the data shows that they work and sometimes it doesn’t and then we have to go back and try to figure that out.
It’s a balance. I think the way we’ve really tried to address it is by breaking testing down, understanding why we’re doing different types of tests, and what the output and outcomes of those tests should be. Being very rigorous about the tests that have to do with academic achievement and trying to make sure that we’re doing that in appropriate ways.
Jill: Yeah. That’s awesome. A lot of times when we’re working with clients sometimes we’re doing some coaching around testing and how you build that up. It sounds like you guys are a little bit of a well-oiled machine when it comes to testing.
Jason: It’s always easy to make it sound like you’re doing awesome. I would just say that as a product professional who always wants to get better there’s always something that you can do to improve, whether it’s better application of automated testing or it’s better communication with customers as you’re working through that process. There’s always a way to get better. I think that’s one of the things we’re constantly looking for is how do we improve our overall process?
Jill: Yeah. Cool. All right. As you think about how you approach product leadership what are some things that people aren’t talking enough about?
Jason: One thing I think we don’t often talk a lot about is how hard it is to try to change a legacy process and historical process to adopt some of these things. I think really more case studies and more examples, more people talking about success of applying these things in big organizations would help.
I think we all feel the pace of change and how it’s accelerating and the pressure it’s putting on us. I don’t think we always know who to go to to say, What did you do when this happened to you? Or how are you responding and what’s working and what’s not working?
It’s one of the reasons I was interested in participating in this conversation because I’m hopeful that it will lead to additional discussion on some of these topics. When you have these challenges about being high stakes and high risk but also wanting to improve your process how do you balance that? Or when you have challenges around having to change faster than you’re comfortable with or to change the skillset of your product organization quickly to respond to a new initiative? How do you do those types of things?
For me, that’s the answer that I would give. I don’t think we talk enough about how to apply these things in places other than small organizations. We tend to look at the startups who are doing it and we say, Look how cool it is that they’re using guilds or whatever the hot topic is. Then we don’t think about the 35,000 employee company and how you would restructure them if they can’t wheel their desk next to the person that they’re going to be working with.
That’s the type of thing that I think we could do more about is bridging the differences between the small organizations and the big organizations and trying to find common ground and shared techniques for how we’re making these types of things be effective.
Jill: Yeah, and especially when I think about change and how I’ve seen it implemented at other organizations. People always want it to be a perfect process. It never is. There’s always some pains somewhere. I understand your point of let’s find out what’s worked at other organizations, what hasn’t worked, why was that? Would it actually work the same for my organization or not? Are we culturally a little bit different?
Jason: Yeah, I was out last night at a place where on the wall they had a saying that said, Your comfort zone will kill you. The trick is being far enough out of your comfort zone that you’re trying new things and you’re growing as an organization but not so far out that you can’t execute and you can’t deliver. Change should hurt a little bit but it shouldn’t hurt so much that it kills you.
Jill: Back to your fitness analogy. You should be a little bit sore after that workout or it probably wasn’t pushing you enough.
Jason: That’s right. That’s right. That’s the trick is finding the sweet spot. I think we could all work together more effectively if we were sharing some of those stories.
Jill: One more question for you. This is one thing we always like to close out the podcast with, which is if you had a new product manager in here right now what would be the one piece of advice that you would want to hand down to that person as they grow in their career?
Jason: Yeah. I spent some time thinking about this because you had given me this question in advance. The thing I settled on is work on your powers of observation. I think there’s inspiration everywhere you look if you just are looking for it.
Jill: Like at restaurants?
Jason: Yeah, restaurants. Every experience. The check-in system when I came in today. For me, I really like gaming as inspiration. I was thinking about some folks that I often recommend for people to check out. Jonathan Blow has a talk that he gave at the Penny Arcade Expo here in Boston about level design. He talks about all the ways that the game subtly is teaching the player how to play it.
I think if you really think about your own product experiences and that level of detail, like what are all the cues that I give to somebody as they’re using my product for the first time about how to use it? Where am I just leaving them to their own devices? It’s really eye opening.
Another thing that I really like to think about is gaming systems. How do you design a system that’s fun to play? Raph Koster did a great analysis of Pokemon Go when it first launched and how the leveling structure caused players to drop out at certain points in the game. If you actually did the math of how much time it would take you to get to the next level it was just so astronomical that people would stop playing.
You can actually model all of that out on paper and it looks a lot like the types of things we think about. What’s your funnel? How many people do you retain through the funnel? What are your drop-off points? What are your percentages as you navigate that?
His point was if you actually do the math and think it through you can improve it. You can design a system that will improve those metrics in the same way that we think about funnels and improving our pass through rates.
For me, I think gaming is an interesting spot right now because there’s so many new games all over the place. Mobile games and console games. There’s a lot of really interesting system design happening there that’s really inspirational. I think you have to be observant. You have to be looking for it.
If you are, whether it’s the weather app or the Twitter app that you use or the YouTube app, there’s tons of great examples of design. Everywhere you look from the console of your car to the cable system you use at home. You can be inspired by all of them. Be really observant and really study the world around you and look for inspiration in all the places you go and all the things you interact with.
Jill: That’s great advice. A lot of times when I’m starting a design sprint when I ask people to introduce themselves I want them to talk about a great product experience they had. That depends on that power of observation.
Jason: Yup.
Jill: Nice. Thank you so much, Jason, for coming in today. It’s been a blast talking with you and getting deeper into the education space. Any last words that you want to say?
Jason: No. You’re welcome for coming in and thank you for having me. I really appreciate the opportunity and the chance to catch up.
Jill: All right. Jason, folks. You can check him out. He’s at Pearson and he’s working on the My Labs product. He’s our latest product hero. Thank you very much.
Jason: Thank you.
Comentarios