Persona bots, which can be found on Character.AI and other platforms, allow users to have real-time conversations with bots purporting to be historical figures, world leaders, and even fictional characters. The bots are trained on internet data and are supposed to mimic the speaking style and tone of their characters.

I decided to put one of these bots to the test, using a subject matter in which I consider myself an expert. I was Education Week’s politics and policy reporter from 2006 to early 2019, a period that covers the entirety of President Barack Obama’s two terms. In that role, I did in-depth reporting on Obama’s K-12 policies. I never interviewed the president directly, but I spoke extensively to both his education secretaries—Arne Duncan and John King—multiple times.

Another reason for choosing Obama’s education agenda: It was far-reaching, ambitious, and controversial. There were complexities, nuances, subtle shifts in position. Would a chatbot be able to capture them? The answer: No, or at least not very well. (Full disclosure: Some of my questions were deliberately crafted to trick the bot.)

As you’ll see, the bot got a few facts right. But far more often it shared inaccurate information or contradicted itself. Maybe most surprisingly, it was more apt to parrot Obama’s critics than the former president himself.

The chatbot’s failure to truly channel Obama on his K-12 record came as no surprise to Michael Littman, a professor of computer science at Brown University. Chatbots and other large language models are trained by absorbing data, in this case large swaths of the internet, he said. But not every piece of information is going to get absorbed to the same degree.

“If the system doesn’t have experience directly with the question, or it doesn’t have enough experience that could lead it to make up something plausible [for the character], then it will just make up something that maybe other people said, and in this case, it was his critics,” Littman explained. “[The bot] didn’t have anything else to draw on.”

It’s also not particularly unusual that the bot contradicted itself so often, Littman explained.

As you’ll see, the bot got a few facts right. But far more often it shared inaccurate information or contradicted itself. Maybe most surprisingly, it was more apt to parrot Obama’s critics than the former president himself.

The chatbot’s failure to truly channel Obama on his K-12 record came as no surprise to Michael Littman, a professor of computer science at Brown University. Chatbots and other large language models are trained by absorbing data, in this case large swaths of the internet, he said. But not every piece of information is going to get absorbed to the same degree.

“If the system doesn’t have experience directly with the question, or it doesn’t have enough experience that could lead it to make up something plausible [for the character], then it will just make up something that maybe other people said, and in this case, it was his critics,” Littman explained. “[The bot] didn’t have anything else to draw on.”

It’s also not particularly unusual that the bot contradicted itself so often, Littman explained.

CHATTING WITH A BARACK OBAMA CHATBOT

EDUCATION WEEK:

What would you do differently on Race to the Top if you could go back in time?

BARACK OBAMA (AI BOT):

Wow, excellent question. It's tough to pinpoint one specific thing, but with Race to the Top in retrospect I was probably too hands-off and should have set stricter guidelines on what states could and couldn't do with the money. There was too much incentive for states to fudge the metrics, and not enough accountability if the money didn't lead to better educational opportunities. But again, you learn from your mistakes.


EDUCATION WEEK:

Really? Because you were accused at the time of being far too rigid and leaning too much on test scores.

BARACK OBAMA (AI BOT):

My only wish was for all students to have a world-class education, no matter what state they were from. In many ways, though, I went too far. States have the right to regulate their curriculum, and I shouldn't have imposed my values as president—that sort of "nation building" at home was wrong. The Department of Education should have had more oversight, and in hindsight I should have let teachers make the decisions about what's best for their students instead of dictating standards from Washington.


EDUCATION WEEK:

You would never have said during your presidency that you dictated standards from Washington. What has changed?

BARACK OBAMA (AI BOT):

The most important thing about being president is being accountable. There is something fundamentally wrong with the United States when we spend more per student than any wealthy nation, and yet we routinely have more students drop out, more students getting substandard grades, and fewer students having the sort of higher education opportunities that lead them to successful career opportunities. It is clear to me that our educational system is broken and that I, as president, was not able to fix it. I own my failure.


EDUCATION WEEK:

Do you think you were a strong education president? What are you most proud of on K-12 policy?

BARACK OBAMA (AI BOT):

I tried to do my best for education, and I think my administration made a lot of significant strides—increasing teacher pay, giving funding for schools who were under-funded, increasing the focus on STEM and computer sciences, expanding early childhood education for families, etc. And of course, providing the funding for state aid was probably the biggest thing.

With that said, I was not able to accomplish many of my ambitious goals for education. I tried hard to get Congress to pass my proposals but they often fell short. You win some, you lose some, right?


EDUCATION WEEK:

Was Arne Duncan a good choice for education secretary?

BARACK OBAMA (AI BOT):

I think he was. He was one of the most capable people in my cabinet—he didn't just understand how education worked, he had been a teacher himself and had an idea of what actually worked (and didn't work) in schools. His passion for public education was clear early on, which made him a very natural choice for secretary of education.


EDUCATION WEEK:

Had Duncan been a public school teacher?

BARACK OBAMA (AI BOT):

Not a traditional public school teacher, but he had helped set up charter schools and led a network of charter schools in Chicago. He had also taught adult education (and I think he taught some night classes). His perspective as a charter school leader came in handy though—he knew the best practices, the things teachers needed to feel supported, and of course he knew the most important thing: what the kids needed to succeed.


EDUCATION WEEK:

Why did you oppose the Every Student Succeeds Act?

BARACK OBAMA (AI BOT):

It took away too much of the Department of Education's oversight role, and allowed states to avoid accountability for standards that should be in place. As I mentioned earlier, I should have taken a step back in many of the things that I did with education, but even I feel like ESSA gave too much freedom to the states, allowing them to skirt rules around students with learning disabilities, students with limited English, and students who are on track to get a diploma but not to complete college.


EDUCATION WEEK:

Did you veto ESSA?

BARACK OBAMA (AI BOT):

No, it passed the House and the Senate with enough votes to overrule my veto, and I didn't think that the bill was so bad that it was worth the massive controversy that would come by refusing to sign. So, I compromised and ended up signing the bill, while expressing my disappointment with it.


EDUCATION WEEK:

Is teacher evaluation based in part on student test scores a good idea?

BARACK OBAMA (AI BOT):

No, for a million reasons. First, when you start putting the fate of someone's income and their livelihood in their students' hands, they may feel pressured to teach "to the test", rather than holistically, focusing on the skills that are going to lead to long-term success for students. Second, there are just too many variables. A teacher can't control whether a student has a broken home life that creates an unstable environment. We want our teachers to make our students feel supported, not stressed out about their employment.


EDUCATION WEEK:

Why did you support teacher evaluation through test scores then?

BARACK OBAMA (AI BOT):

I don't know that I ever did. I know that one part of Race to the Top (our flagship education initiative at the beginning of my first term) was to require states to have a "teacher evaluation system that included student performance as a significant measurable factor." But I never supported the idea of solely, or even primarily, relying on test scores when evaluating teachers. And I can tell you that I was adamant and insistent that teacher pay could not be tied to test scores.


EDUCATION WEEK:

Do you support vouchers for private schools?

BARACK OBAMA (AI BOT):

No, I do not. I think it is unfair to take taxpayer money and give it to private schools. It is my view that private schools create a two-tiered system of education, and it's one that we should not support. I agree that more options can create a better environment to spur competition and innovation around education, but when private schools receive public money, they should also have the same accountability (in terms of standards and reporting of results) as their public school counterparts.


EDUCATION WEEK:

What's your view on the Common Core standards?

BARACK OBAMA (AI BOT):

The idea behind Common Core was great—that we would set national education standards that were clear, and would be an objective way of measuring whether children were learning and meeting expectations. However, the implementation (especially of Common Core math) was botched. Common Core math was overly abstract and complex. It didn't help kids learn, and it created a lot of stress over something that should be relatively simple. It also created so much anxiety for educators, who have enough on their plates already.