ChatGPT Just Passed This Wharton Professor’s Final Exam. He’s Says He Won’t Ban The AI Tool

In your white paper, you said that you would change your personal syllabus and wouldn’t let students use it for tests. Have there been more formal conversations amongst Wharton faculty about changing the honor code or formerly banning it for exams?

Wharton is a place – and I give it great credit for this – where we have long discussions about these things, and then it’s left at the discretion of the instructor. I think that makes perfect sense because whether this thing is going to be an enabler or a barrier to learning will really depend on the specific class and how it is taught.

We have to ask ourselves why do we test in the first place? I think there are three reasons: First one is skill certification: You pass the test to become a certified public accountant. You pass a test to drive a vehicle at the DMV. You pass a board certified radiology exam. For that, we have to absolutely make sure that it is you passing the test and not the bot.

The second reason we test is to customize the learning to where the students are in their learning journey. Just before introducing new material, I have to make sure the students are comfortable with the old material. For that I need some form of a test — it could be a cold call in the classroom, or a little homework assignment. Again, I think, everybody loses if we have the bot take that test for the students because then I’m teaching week-eight material for students that are still struggling with week-three materials.

The third reason we test is to have students engage with the material, and as part of that engagement, to learn. When we have students write up a case analysis, or when high school students write a history essay comparing political leaders in the 18th century, nobody really cares about the outcome of that paper. That five-page essay is not going to go into the notes of history. What we care about is the process, engaging with the material and becoming a more knowledgeable person. That is at risk with this bot. Somebody can write a five page essay, and all they did was spend one minute with the bot. They haven’t engaged with the material at all, and we’ve just wasted a learning opportunity.

So I think for these types of tests and assignments, we have to find new ways to engage the student. I think the technology, if we’re creative about it, is actually our friend. I could have an eighth grader interview George Washington, for example. A student could have a French pen pal. I could have a student even crawl in the cell of the human body and wander around.

If what we’re solving for is engagement, we shouldn’t be scared about the test question. When testing for certification and customizing learning, we may want to ban ChatGPT. But for solving for engagement, we should find other ways of engaging the students. And there I think the technology does miracles.

So what will your Chat GPT policy be for your MBA classes?

For now, I just made it clear that on the exam and graded homeworks the bot is forbidden, because operations management is a little more bread and butter skill certification. It’s only a policy in the sense that students could, in theory, still use it on homework in the same way they could send an email to get answers from someone who has gotten an A+ in the course. The honor policy, in my view, always works.

Ethan Mollick (Wharton associate professor of management teaching innovation and entrepreneurship) has given this more thought and just published his AI policy, requiring students to use it in his classes. So, at Wharton we have agreed we would pass it down to the professors, and I think there’s a very healthy debate.

At the University of Pennsylvania, the School of Education has chimed in. In the medical school, we are aware of the fact that ChatGPT has passed all the three major medical licensing exams, so they are giving this some thought. This is clearly something that is intensely debated. But I think we’re not doing the problem justice if you’re looking for a single dimensional decision around how much should we regulate it or should we ban it. I think there’s just so much upside opportunity.

I think for very privileged business schools with resources, self motivated students, honor codes, all of these things we can now build on, is a good foundation to get to something better than a ban.

The Stanford Daily surveyed nearly 4,500 Stanford University students this month and found that 17% of students had used ChatGPT on their finals. Most used it for idea generation and brainstorming. But, nearly 13% of them either edited ChatBot’s answers or copy and pasted them directly. What is your reaction to that?

So, I’ve spent many, many years of my life researching ideation and brainstorming, and I hope that’s where the discussion is going. You ask the bot for, say, five first lines for an essay or five ideas for new business, and then the human being makes the selection decisions.

Any form of creativity always involves these two steps: Creation and selection. Selection is really hard. Even when you look at the best venture capital firms in the country, they all struggle with selection. Everybody has passed over amazing investment opportunities, and everybody has invested in crap.

What you have now is a device that can add additional options for you to consider. I always say in innovation that variance is your friend; You want to really push the ball towards making totally wacky suggestions. Even if those suggestions themselves are total nonsense, there was a spark in your brain.

So, what I saw in my experiment, the bot got three out of five questions on my final right. If I’m a clever student who wants an A or even a B+ grade, I can’t just copy and paste the whole homework in. If you are a doctor, three out of five is a really bad ratio. You have five patients, and two of them die? But in venture capital, three out of five is better than anybody.

We have to ask ourselves, what type of decisions am I making? Is there a validation after me? If I’m the last person in the decision line, I cannot rely on ChatGPT. There’s just too many unpredictable errors. But, early on in the decision line, and if I’m there to create crazy ideas that will get validated, improved and potentially recombined by human beings, I mean, I’ll take three out of five.

What kind of response did you get when you first released your experiment? I imagine it started a lot of conversations with colleagues, even outside of Wharton.

I sent out the paper to the operations management community first. I think, without any exceptions, it was positive feedback. I heard from three other colleagues who did similar things after they saw my white paper, and they all ended up in this B to B- range.

Some folks are pushing hard on making sure we have a cheating detection device, and I think that’s the wrong way for business schools to work. My colleague Karl Ulrich and I did a story with Poets&Quants many years ago about whether MOOCs would put business schools out of business. I think that was one of the few cases where our prediction actually was reasonably good. Initially it sounded provocative, but for the top business schools, I think MOOCs have been a blessing. And I think the same is going to happen with this technology here. If you are in the top business schools, it is an additional tool to create learning experiences. At the end of the day, that’s our job.

Have you thought yet about how you will use it in your classroom?

I’m totally open to having students use it in case discussions. McKinsey in the box, if you will. I teach an innovation class, and we’re going to run a brainstorming innovation tournament around generating ideas on taking advantage of the technology. In this tournament and others, I invite students to collaborate with ChatGPT to see if their ideation, concept generation, and brainstorming get better if supported by AI.

There is a really cool research study I hope to do this spring comparing multiple brainstorming groups. One of them will brainstorm as usual, the other gets a chatbot as a partner. We can just see which is better.

You also note in the paper that you’ve written more than 1,000 exam questions in your career, and this will be a tool to help you do that more efficiently. And, time faculty saves in such endeavors can be used to better serve students. Can you explain what you mean?

I for sure will use ChatGPT for question generation. As I mentioned in the paper, when it comes out of the press, you have to clean up the edges. It’s not good to go. I don’t think anybody pities me for that, but coming up with these exam questions is just hard. There are only so many DMVs and barber shops that you can write about. So I look forward to that part.

In the long run, as we all adapt and find the best usage, I think our productivity will go up.

For the sake of argument, let’s say our productivity will double. Whenever you have productivity go up in a process, you have two options: You could either cut the amount of teaching staff in half, or we could double student learning. I just very much hope, even the poorer parts of our educational system, can go down the path where we double the learning. That means additional courses, maybe more students, but I think an important part in a business school is just out-of-the classroom experiences.

All the top business schools have done a lot with global immersive courses, programs with faculty and students to interact outside the classroom, independent study projects, and that kind of thing. There are a lot of cool things you can do to turn faculty time into additional student learning. We should just really make sure that we continue to do that. And again, I’m totally aware of the fact that this is easier for the rich business schools than it is for community colleges.

Your paper listed several implications business schools should consider based on the results of the experiment. What do you think is the most important?

I think all of us need to find a way of rethinking work in a world where computers are so smart. There’s a story that we’re running out of work, and I think it’s nonsense. I mean, look at our schools, at our health care workers, at our environment; Do you really think we’re gonna run out of work? There’s just so much that needs to be done, and we have to find a way of solving these problems. Ask an eighth grade teacher whether they feel like they need more work.

We have an abundance of work. We have to find ways to improve productivity, and then lift everybody up to be healthier, better educated, and a happier person. I mean, if we have a machine that makes us 50% more productive, as in our previous example, let’s use it.

Read Terwiesch’s full white paper here.

DON’T MISS: THE P&Q INTERVIEW: NYU STERN’S JOHANNES STROEBEL, WINNER OF THE FISCHER BLACK PRIZE FOR FINANCIAL RESEARCH WITH IMPACT and SLOAN FELLOWS: AN ELITE MID-CAREER DEGREE AT 3 WORLD CLASS B-SCHOOLS