Guide to Integrating Generative AI for Deeper Math Learning

Sycophancy & OpenAI

SAP’s e² Instructional Practice Suite™

Amanda Bickerstaff

Amanda is the Founder and CEO of AI for Education. A former high school science teacher and EdTech executive with over 20 years of experience in the education sector, she has a deep understanding of the challenges and opportunities that AI can offer. She is a frequent consultant, speaker, and writer on the topic of AI in education, leading workshops and professional learning across both K12 and Higher Ed. Amanda is committed to helping schools and teachers maximize their potential through the ethical and equitable adoption of AI.

Jun Li

Jun is a Director on the math team at Student Achievement Partners. Prior to this role, she has served as a middle school math teacher, curriculum and assessment writer, and in various charter network leadership roles designing math programs and coaching teachers and leaders in math curriculum implementation. She holds a MSEd from the University of Pennsylvania. Jun is inspired by work that sits at the intersection of math, professional learning, and instructional practices that place student thinking at the center of instruction. Throughout her work, she has been driven by learning about and creating learning experiences that make doing equitable teaching practices more prevalent and accessible.

Jasmine Costello

Jasmine is a Product Manager at Student Achievement Partners. Prior to this role, Jasmine was a project manager, education researcher, and classroom teacher in Philadelphia. She has an MS in Effective Teaching and Learning from the University College of London where she attended as a Fulbright scholar. Jasmine is committed to finding creative solutions that serve students who have been historically and systemically underserved by our country’s education system. She is passionate about working towards greater educational equity and creating tools that support all students to thrive as their authentic selves.

Mandy DePriest

Mandy is a Curriculum & Content Developer at AI for Education. She has over 15 years of experience in public education, having served as a classroom teacher, library media specialist, and instructional coach. She has also taught education technology courses in higher education settings as well as professional development workshops for teachers on the transformative power of technology. She is committed to ensuring that students are prepared for the dynamic demands of the future by leveraging the power of technology-driven instruction.

00:01
Amanda Bickerstaff
Really excited to have you all here today. I'm Amanda Bickerstaff, I'm the CEO and co founder of AI for Education. And this is really cool because this is actually the second in a series of our partnership with Student Achievement Partners. We if you were able to join us, gosh, I think it was only a couple. Like late fall, like our winter, we released our AI Literacy, essentially a deeper learning or in. In literacy with SAP. And we absolutely love the experience so much that we're excited to have our second guide which is focused on deeper mathematics learning. And I will tell you, it has been a journey. In fact, we had planned on having this guide out about two months ago, but really hit some really interesting roadblocks and we learned so much the process.

00:46
Amanda Bickerstaff
So I'm really excited to have you all here today. I'm gonna, I am joined with Mandy from our team as well as June and Jasmine from SAP. But we're gonna go to the next slide. And as always, everyone, this is a. We want you to get involved, say hello in the chat, where you're from, what you do. If you have a great resource, please share that. And then finally, if you want, because we have almost 400 people already, if you want a question specifically to the team, just make sure to put that in the Q and A because that's where we're going to be reviewing those questions. I'll of course always review the chat, but just want you to be able to do that. And thank you. So many people from already.

01:21
Amanda Bickerstaff
We already see you're joining from all over the world, including Nambia, India, got some New Yorkers in the room as well where I am. But just really excited to have you all here today. So let's talk about the partnership. And so this is our first really big partnership around content. And I say content specific meaning like we've already done our guide on literacy and now on math. And so the reason why this happened is that the very first conversation I think that Jasmine and Carrie, who's not here today, had with Corey and our team was that there's just this opportunity to learn together and to take what we learn and actually present it to you all because there is a real lack of content specific guide strategies out there and we are not content.

02:03
Amanda Bickerstaff
You know, Mandy and I, we have our different expertise, but we are more your education and AI kind of folks in this world. And so to have the amazing opportunity to work with SAP has been wonderful. But we wanted to really to start thinking through the way we structured this really around productive struggle. And so June and Jasmine are going to talk a lot about this, but we wanted to really understand what are students going to be learning if they're going to be supported by generative AI, where is that productive struggle going to be important? And then where are the potential benefits and where are the potential pitfalls? And we're going to talk a lot about those kind of tensions during the session. But I really hope we're going to drop, I think we'll drop the guide into the chat.

02:43
Amanda Bickerstaff
We really hope that you'd find this as a really good starting place. I'm going to say none of this is fully like, we want you to experiment, we want you to try it. And so this is really meant to be about action research as much as anything else. So I'm going to be real, I'm really excited. We're going to move the next slide and I'm excited to introduce June and Jasmine who are going to take us through how we actually combined our forces here around the high quality mathematics instruction and then how we built the guide. So I'm going to hand it over to you all and thank you both so much for being here today.

03:13
Jun Li
Thank you, Amanda. And as we talked about content, we thought it was important to first start with what do we mean when we say high quality mathematics instruction? So what you're looking at here are the four key areas of high quality mathematics instruction. When we say essential mathematics, we're talking about students engaging in cognitively challenging mathematics that focuses on grade level or course level content standards. When we say connected mathematics, we're talking about students being able to relate new content to math content within or across grades that builds towards future mathematical concepts. And we're talking about students building procedural skill and fluency from conceptual understanding.

03:50
Jun Li
When we say meaningful mathematics, we're talking about students being able to use math flexibly for applications in real world context and involving this engagement with the demands of the challenging world instruction should affirm students identities, cultures and communities as they do mathematics. And when we say student centered mathematics, we're talking about positioning students as the mathematical experts, thinkers and doers in the classroom. Students are engaging in mathematical reasoning, discourse, communication, problem solving and modeling while engaging in the math context.

04:23
Jasmine Costello
And if you're interested in learning more about our expanded definition of high quality math instruction, we encourage you to check out our E squared, which stands for essential and equitable instruction landing page, which June's going to pop in the chat now. And there are some accompanying tools on there that can Help you bring the research into your practice as well. So now we're going to dig into this concept of productive struggle and counterproductive struggle that Amanda was referring to earlier. So we know that a big concern about using generative AI in the classroom is this risk offloading critical thinking or the cognitive load from the students to the tool.

05:07
Jasmine Costello
So when we conducted our exploration, which we're going to dig into in a bit, but we anchored that exploration in this framing as a way to identify use cases that support or encourage productive struggle and, or help students that might be in a place of counterproductive struggle move out of that and into a place of productive struggle. So we can start by defining and creating shared understanding of these terms so that we can know how to identify AI tools and uses that work in service of supporting students critical thinking and cognitive lift. So starting with productive struggle, what we really mean is students engaging in meaningful interaction with challenging and grade levels level content that supports their learning. Whereas counterproductive struggle is a challenge that creates a barrier to learning.

05:59
Jasmine Costello
And we know that this is a big concern in math, especially as when students get stuck in a counterproductive space, they can really develop increased math anxiety. And we see harmful effects on their development of math identities as well, which can have lasting and detrimental impacts. So during our gen AI tool and use exploration, we use this framing, as I said, to identify opportunities for use that would support students in engaging in productive struggle or help mitigate the concerns of counterproductive struggle and moving them out of that space and into a space of productive struggle.

06:39
Jun Li
Yeah, absolutely. And I want to add on to that. We talked about counterproductive struggle as the math may feel out of reach for students. There's also the case of counterproductive struggle when the math is too scaffolded. So we know that there are tools that exist out there that scaffold the learning for students. But sometimes that creates learning experiences that folks primarily on procedural skills absent of conceptual understanding. And it begins to reinforce this mindset around mathematics of there's one right approach or that math is really disconnected and it's a bunch of procedures that I need to learn. And so that counterproductive struggle, that struggle is counterproductive. And it also leads to the increased math anxiety and the hindrance of positive mathematical identity development that Jasmine was talking about.

07:19
Jun Li
And so instead we want to focus our attention on how can we use AI tools to really support student learning while creating an environment for productive struggle. And we're really excited that the resource we're about to shares examples of both.

07:32
Jasmine Costello
Yes, appreciate those specific math examples. And last thing I'll note is that in our exploration we also discovered the difference between specialized gen AI tools and foundational large language models when used specifically in math. So I'll pass to Mandy now to dig into that and walk through some examples that we discover.

07:57
Mandy DePriest
So we had a little fun with AI image generation to kind of summarize our learnings from our field testing of using AI to support math. And I think the temptation when we see a transformative technology like generative AI and ChatGPT can do so many things is to think that it can do all of the things and try to apply it into areas where it may not actually be suitable from a design standpoint for certain tasks. And we discovered that math would be one of them. So we put in our picture specifically LLMs or large language models can't do math. And that is because they are language tools. And Amanda might want to break down the technical aspects of this a little bit more, but essentially they are not computing numbers the way a calculator would or even the way a human brain would.

08:54
Mandy DePriest
They are interacting with the words that you input into it. So Amanda, do to speak a little bit more to the technicalities of this.

09:02
Amanda Bickerstaff
Yeah, I think it's in the name too everyone. Large language model.

09:05
Mandy DePriest
Right.

09:05
Amanda Bickerstaff
So these tools are actually really interesting because they are built on really complex statistical probabilities and so it actually is math. But when you ask this is why we think this is funny, by the way. This is for me, everyone, this image is for me particularly. But one of the things that happens is that when you ask ChatGPT 2/2, it is not going to a calculator and then going to be answering the question. What it is doing is it's going to its training data set and the probability of answer is going to be five. The most probable answer will be five. But we have been in a training where a college professor sat there for 45 minutes convincing that ChatGPT, the ChatGPT that 19 +4 was 28, which is not the right answer.

09:51
Amanda Bickerstaff
And in fact we talked to the people that have built Cognigo when they created Cogmigo, ChatGPT had very difficulties with just basic computation, but was very confident in its wrong answer. What's interesting here is that these tools, these large language models, if you're using an input where they have Wolfram Alpha or others that are going to be additive to the large language model, then it can do Math better, but just the large language model, the kind of vanilla large language model will make mistakes all the time.

10:26
Amanda Bickerstaff
And one of the things I think, Mandy, we've talked about in the book, the paper as well, is that while ChatGPT and other tools can do really well on like the Math Olympiad in terms of final answers, when you ask them to break down all the problem solving, it actually goes down from like 98% to like 10% in terms of how good it does because it has so many computational errors, so many reasoning areas and like it just cannot do math. And so while specialized tools we found are going to better for this, it is something that we're a little bit, we want you to be concerned about because there are a lot of teachers out there or even students that potentially can be getting inaccurate responses about answers and explanations.

11:09
Jasmine Costello
Yeah.

11:09
Mandy DePriest
And I think it can be a little bit misleading too, Amanda, just because the generative nature of the technology, it might get it right several times in a row and that can build this false sense of security that. Oh yeah, it's fine. And especially like with like lower level math, like with early math, single digit numbers, it's probably seen and it's seen in its training data, two plus two is four enough times to like confidently regenerate that. But when you get into more complex math things it might not have seen in its training data as much, that's when it's like going to hallucinate and totally recompute and regenerate a response each time that you ask. So you can, I just want to highlight that it's easy to like anecdotally say, well it's been right every time before this when I've used it.

11:54
Mandy DePriest
So it's going to be right every time. And that is not an assumption that we can have with this technology, which is a shift from like, you know, our calculator is going to add two plus two correctly every time. You know, our GPS is going to show us the route, assuming it has up to date data every time. Like, but this technology has problems with consistency and that was one of the things we surfaced. I'll go into the next slide and share an example. So this is a screenshot from a chat which we've linked there at the bottom. We've also in our, in the document we have all these worked examples with links to chats and we put in one as a negative example to demonstrate what we're talking about here.

12:35
Mandy DePriest
So in this case I was interacting with the chatbot with chatgpt as a tutor, and I was taking on the role of, like, a middle school math student. My son is in middle school, so I was working through some of the content that he was doing. They were calculating the area of triangles at that point last March. And so it had worked through several problems with me just fine. I was feeling good about the situation. And then we got and it asked me to calculate the area of a triangle that had, like, a base of 18.4 and a height of 9.2 inches. And I answered incorrectly on purpose to see what it would do, and it did catch that, and it broke me down. The the problem here, what do you get when you multiply eight? You know, multiply by half.

13:22
Mandy DePriest
And I doubled down on my wrong answer. I told it 18.4 times. 9.2 is 158.28. And that is not the correct response. It is actually 169.28. I wrote it down so that I wouldn't forget. So I tell it this wrong answer, and then I divide by half or multiply by half, and it gets 79.14, and it tells me I'm correct. Now, that is the correct response of half of 158.28 is 79.14. And I think that's where it got thrown off in the layers of mistakes that I was making, because I was making multiple mistakes here, and it just kind of lost the thread of what were originally talking about, and it told me that I was correct. But if I'm a student who's developing my understanding of calculating the areas of triangles, I'm not necessarily going to catch that.

14:10
Mandy DePriest
I'm going to be confused and think that I got it right, when in fact, I did not. And it's these kinds of misleading interactions that I think make the technology too unreliable to recommend that you use with students without specific things in place. Now, I do think maybe there's some value in an AI literacy context talking about the risk of incorrect answers and using that as a teaching moment and talking about why is this wrong, doing some error analysis and breaking it down. But I would not trust that to students to be using independently to develop their math understanding. June, do you have anything?

14:48
Amanda Bickerstaff
It's okay. June, this really quickly. I think one thing that most likely happened here is when a man, when Mandy said, you're wrong because of something called sycophancy. So sycophancy is that these models are designed to say yes and to agree with you in a lot of ways. And I think that this is where it gets very interesting. If anybody saw the sycophantic apocalypse, that was very funny to call it, but GPT4O came out a new version two weeks ago that had to be rolled back because it was telling you right every time. But the one thing is, if you say this is the wrong answer or this is a right answer, the bot will say you're right nine out of ten times. Right. Because it's designed to do that even if you are wrong. And so this is where it gets very tricky.

15:33
Amanda Bickerstaff
So I'll hand it over to June. But this sycophantacy component is another reason why this is not going to be significantly valuable. As Crystal said, like, they actually rolled back the entire update because of how sycophantic it was.

15:48
Jun Li
Yeah. Nothing new to add here. I see a lot of resonance in the chat about people having similar experiences when you did AI. I think as long as we know kind of what the limitations are or what the learning opportunities are. If you're reading for this purpose, I'm not kind of. It gives us food for thought about why we use AI, which type of AI tool, use for what kind of purpose. But we can talk about the next example just to show another way in which things might look right and then they look a little bit deeper. You're like, oh, wait, I should be more careful about how I use the tool. So here Mandy had created a prompt that was using Quad to create interactive visualization to see how changing the parameters of a function would also affect the graph.

16:31
Jun Li
And so she's going to click into the interactive so we can show you a little bit more.

16:45
Mandy DePriest
And this interactive looks really cool. Right. This is one of the features of Claude doing this kind of native interact. Oh, Amanda, you're on mute.

16:53
Amanda Bickerstaff
Yeah, Mandy, we can only see the slide deck, so you're on the wrong tab.

16:57
Mandy DePriest
Okay, hang on. Apologies. How's that?

17:04
Jun Li
Yep. Yeah, great. Thank you, Mandy.

17:07
Mandy DePriest
I'm normally pretty fluid with zoom. This is an aberration to use maybe a math word.

17:14
Amanda Bickerstaff
Thank you.

17:14
Mandy DePriest
Okay, so like I said, this is one of the features of Claude that people get really excited about when it came out. I personally am very excited about it. And it does have value. Certainly it can do some really cool things. And so we can put in. I asked it to create something that would show transformation of functions as we're going. So theoretically the student would be able to manipulate these variables and see how it changes the graph associated with the function, and we can change the nature of what's going on. Here it looks like a really amazing tool.

17:48
Amanda Bickerstaff
Right.

17:48
Mandy DePriest
And I'm especially thinking of, like, maybe first year teachers or novice teachers here who may be not quite as fluent with the subject area. I know math teachers are hard to come by in my area. Sometimes people end up teaching math when they didn't expect to. So if I'm building my confidence in my subject knowledge, to me, this looks great. But June surfaced some issues when she looked at it, and so I'd love to hear your take on it, June.

18:17
Jun Li
Yeah. So as Mandy was showing, when we play around with the sliders, I'm like, oh, it does stretch vertically or horizontally or shift vertically or horizontally. And you can see, like, the equation is also changing as you would expect it to change. And also as Mandy was dropping down in the base function, you can see like, this, it says it's a quadratic. It looks like a quadratic. If we try a linear function, we're going to see it looks like a linear function. If we try all the other ones, you'll see, like, the way the function looks like as it's described. But then when you look more closely. So we go back to the quadratic example, and you reset all the sliders to zero just because it's easiest to see it that way. So zero would be, like, right in the middle.

18:54
Jun Li
We can actually work here. No, let's reset it to zero because it'll be easiest to show. Sorry. To one. Maybe if you just. Let's reset it to one. That's what it'll.

19:19
Mandy DePriest
Reset it to one.

19:20
Jun Li
Okay. For the vertical stretch, I mean, I can even explain the example here. We'll see, like, there's that X squared for the quadratic. But then you're like, what's that extra X hanging out here? Doesn't that make it now an X cubed? And so an X cubed and an X squared graph would look very different. And so if we go through the other base functions, we're also going to see there seems to be this, like, extra power that's just hanging out every time with the equation. And so that goes to show, like, on first glance, it looks right. And when you play around with the sliders, it's expecting it to change the parameters. It follows the prompt and it looks right.

19:58
Jun Li
But then when we dig a little bit deeper, we're like, actually, that would be conceptually teaching students a very incorrect recognition between what the equation is and what the graph would look like. That would compound later into. Into the learning something that took a little bit deeper of a Look to see the error.

20:15
Mandy DePriest
Even you, June. I want like, not to call you out or anything, but like when, at first glance I remember when were talking about it, you were like, yeah, it's the graph is behaving the way I would expect it to. It wasn't until like a few days later when you'd really dived into and played around with it, that you caught it. So even like experience math teachers might sometimes not catch things, especially in like, the more complex disciplines. So to me, if it's not 100% reliable, then I'm not comfortable using it with students without, like, significant instructional framing around being on guard for errors, checking solution, viability, recognizing if something looks off and like kind of interrogating the results as a basic, like, habit, as in part of the calculation. See, and I can jump back into our presentation now.

21:05
Mandy DePriest
So there is a solution to this and I'll jump back into. And I know I won't.

21:11
Amanda Bickerstaff
Yeah, I think a lot of people are kind of identifying this already. Is that like they're, you know, we're talking, we're starting from a place where most teachers and students are using this, which is just the vanilla foundation model. So we want to be clear that like, this is the reason why we're spending so much time on this though, is that were so surprised by just how unreliable these tools were. But we're going to. There's questions about cogmigo, there's questions about some specialized tools that have tool calling and calculators. So they're absolutely going to have more.

21:44
Amanda Bickerstaff
But I think that this is where we just want to caution against this broad brush that like ChatGPT is going to be able to help my young person with their math homework, or I'm going to use this to lesson plan and build out deep explanations of math proofs, for example. And so that. So we're going to move now into the more practical approach of what we do think works, right, Mandy?

22:07
Mandy DePriest
Yeah. And you know, I'll mention there are prompting strategies and setting up your prompt in the right way that honestly I think take us into like a more nuanced level of use. And I think a lot of people are readily able to access. So if you're a power user and you know how to like set up your prompt in such a way that it's going to call certain tools or do things in a certain way, then you know, you're the best judge of how you want to deploy that in your circumstance. But we do find that there are a lot of specialized Tools already out there. Several of you have surfaced already, like Khanmigo and WolframAlpha and Snorkel. We have on there on the slide things like that have additional fine tuning and parameters that enable them to calculate more reliably.

22:50
Mandy DePriest
And one of the things we have, like in our. In our math guide we talk about, Khanmigo incorporates a calculator you can link to their site where they explain how they've got it all set up. Things like that make it like, more easy to. More easy. I was a language arts teacher. Make it easier to interact with math tools. They'll also have, like, specialized interfaces. Like, it's hard to enter, say the quadratic equation into ChatGPT's context window and just type it like it looks strange. And if you try to copy and paste and like share the chat, it'll reformat in a really wonky way. That doesn't look like a standard math problem.

23:24
Mandy DePriest
But specialized tools have like, keyboards and things that make it easier to interact with like decimals and fractions and symbols and things like that in a way that makes more seamless usage as well. So that removes an unproductive struggle from student. We might consider it an unproductive struggle if you're having to enter, you know, in a certain way. And we cite all of these tools not as endorsements necessarily, but just as examples of what we're talking about when we say specialized tools. There are a bunch of them out there. You may have your preferred ones if we don't mention it on the slides. Now, it does not mean that we're not saying that it's not good, just that these are the ones that kind of are commonly discussed and that came to mind immediately for us as were preparing our materials.

24:07
Mandy DePriest
So, Amanda, anything to add to the development of these tools?

24:10
Amanda Bickerstaff
I mean, I think that what I just also call out though, that like Snorkel and others, like they're using not just generative AI, so they're using vertical AI, meaning they're using like machine learning models. They're using just prompt. Like, they're using like user experience. They have prompt injections that, let's say, don't just give me give the student the answer. One of the big funny things about Kamiga when it first started is that if you said, I don't know, a couple times it kind of felt bad for you and gave you the answer, which is really not what we want to have happen. So I do think that one of the things that we really like about the specialized tools as they do also, someone did ask. They do make mistakes.

24:48
Amanda Bickerstaff
So cogmigo will make mistakes and will hallucinate anything that is generative AI will hallucinate that has not been solved. And so. But this is where it gets really interesting though is that there's some really good use cases here, especially for more like foundational math. And so I think that's what we're encouraged by.

25:08
Mandy DePriest
And can we. Let's see. Now, that's not to say that there aren't very valuable uses for language models like ChatGPT or Claude or what have you in the math classroom, but they're. They're going to be more around things like planning, you know, brainstorming, developing out complex projects and ideas, things like that, developing alternative explanations, coming up with ideas for interventions, things that are more on like the linguistic side of math planning, because there's plenty of that in there. We do have the link to the resource there. I know we've linked it in the chat as well. Thank you, Dan. So if you don't get it here on the screen before I click out, then no worries, it's in the chat or you can also find it on our website.

25:53
Mandy DePriest
We will also send things out and the way that is structured is with like an explanation of our thinking and then with the worked examples. We have these in our literacy guide as well. So what we found is that sometimes it's helpful for educators to see, well, what would this look like in practice if I wanted to use ChatGPT to enhance my instructional planning, like, what would the actual conversation look like? I spoke to a teacher yesterday who didn't realize ChatGPT was a website. They thought it was some kind of specialized software that they had to. So I think sometimes it's helpful to just see, this is how you might interact with these bots. And so these are the ones we have in the document.

26:33
Mandy DePriest
If you get in there, you will see that we have it divided up such that tasks that we feel like specialized tools would better for, we have signaled that in the document. So we only have these worked examples for tasks that we think large language module, large language models could handle, with the exception of we did put in a negative example of the AI tutoring with computational errors because we think that's helpful to see in action. But we also have use cases around creating lesson materials and interventions. And I do want to signal here, the example we have is fairly basic. It's generating practice problems. I used fourth grade as an example because it's easy to quickly check the math and make sure that they're computationally viable. But you could use this in a really transformative way.

27:18
Mandy DePriest
We talk about often using AI to not just automate outdated practices, but kind of go further and farther and think of new and innovative ways that we might teach. And so we've been on our content team kind of playing around with some use cases of using ChatGPT to help us plan out, like project based learning that might be really complex and highly differentiated and found it to be a really helpful thought partner there in terms of differentiation and individualized instruction. There's an example there that we used it to create a choice board around division tasks, which would normally be a very time consuming thing. I was able to complete it in under 45 minutes. Like from Idea conception to having formatted choice board that I could push out to students, which felt like a pretty efficient use of time there.

28:02
Mandy DePriest
And then finally we have a use case for breaking down tasks and concepts. So the example we put in is suggesting alternative teaching strategies, but it could also be things like unpacking prerequisite skills, looking at sample problems from like previous state or national exams that students had to take. It could be things like generating ideas for intervention for students who don't catch the instruction the first time around, and any number of other things. And like I said, we also have additional use cases in there for which we recommend specialized tools and they're all broken down in the document. Amanda.

28:39
Amanda Bickerstaff
So why don't we. Yeah, let's actually pull up the document so we can walk you through everything, because if you didn't see our first guide, it's very similarly designed. And I want to say thank you to Dan because last night were finalizing this and it literally didn't save. So everyone give Dan super hearts for finishing this up. And Mandy, just stepping in this morning. But the way that this has been designed, if you want to scroll down, Mandy, is that it really is a way. Even if you are using this in your context, we want it to always feel. I know. Thanks, Dan. We just want it to feel like it's a standalone document. So we've got the goals for the resource. And also there's some key assumptions here.

29:16
Amanda Bickerstaff
And the first one I think is so important is that if you're using AI at all, whether you're a teacher, a leader or student, AI, literacy has got to be the foundations here. Even the conversations we've had today around sycophancy and hallucinations. And this is something that I think is really important. And we also really believe very strongly, which is why we are working with SAP, is that we need to have it be everything that we do around this. Math instruction needs to be based in high quality instructional practices. And that's very important to us. We have a nice glossary here that goes beyond just generative AI, but also into some of the other things, like productive and counterproductive struggle, if you want to keep rolling, Mandy. And then we have a note on why we've developed it the way we are.

30:07
Amanda Bickerstaff
Because remember, this is something that we're giving to you all as a way to start to experiment and do action research. But we really wanted to focus on productive versus counterproductive struggle because of how many people think about generative AI as a cheating tool, as a cognitive offload tool, as a tool that just replaces thinking, whether that is on the teacher side of just developing something and not looking at it closely, to the student side of using this to do their math homework. And so we wanted to really couch this in a way that really focused on the quality interaction and quality thinking that can happen. And I think that this is really important. And so we keep rolling down. Also, we have these kind of.

30:46
Amanda Bickerstaff
This is where you're going to start to see those high quality instructional actions and then where it could be productive or counterproductive to use generative AI. And so, like. Or what we want to avoid students doing. And I think that. I don't know. June, do you have a favorite one of these that you want to talk about? Like, any of these that you really was like, your fate. I know, I. It's like hard to ask your favorite, but I know you probably have one. I think you're still on mute. There you go.

31:20
Jun Li
I was like, oh, can you guys hear?

31:23
Amanda Bickerstaff
Yeah, we can hear you. Yep.

31:25
Jun Li
I was like, I just need a minute. I was like, there. It is hard to choose.

31:35
Amanda Bickerstaff
I think I. I mean, if I might call it out, then I think it's pretty interesting is a student centered one, because I know that's something that we really struggle with. And so the idea here of, like, moving from, you know, just giving, you know, the kids, like, scaffolding is great, but sometimes we scaffold too much. Like, we are not representing what students experience in or that teachers are the ones that are actually leading all the explanation. So, June, you want to talk a little bit about how that shifts into a productive struggle? Yeah.

32:03
Jun Li
I also wanted to mention meaningful mathematics because in particular we're thinking about the use of AI tools. I have seen some really powerful uses of where by entering a prompt, you can kind of pull from local data or create contexts that feel more relevant for students or that engage students in data that they can analyze and then make sense of the world and make sense of the mathematics. And so kind of like that also makes a student centered because it's a part of affirming students identity and involving their thoughts on how they make sense of things while making the math meaningful.

32:37
Amanda Bickerstaff
Oh yeah. I think one of my favorites is that, you know, we often don't have enough like sample data or graphs or like for data literacy and so like to do one for like the Kendrick Drake, like, you know, conflict. That's a year everyone. Kendrick and Drake's conflict is a year old for those that know. But you know, to actually use something that was really part of like the zeitgeist to build like, you know, how many, you know, how many LinkedIn or how many YouTube video reviews and how quickly things happen to actually create data sets for kids to engage with. It's such. Was such a cool way of thinking about like an application that is high quality but also really student centered. So I love that. June.

33:18
Amanda Bickerstaff
So if we want to keep rolling through the documents we keep rolling through is that this is where went into the enhancing the math instruction and this is where we do talk and call out specialized tools. And as Mandy said, these are not like, we're not espousing any of these, but we do actually think that you should, if you're using it, please use a tool that you feel more confident can do the math. Going back to our silly photo and we have some examples here, so there's quite a few. Samantha, you want to talk through a couple that we learned through this process. I know that snorkel has gotten very popular, so. But Mandy, do you want to talk a little bit about any of these tools?

33:52
Mandy DePriest
Yeah. And I'll say Snorkel spoke to my heart as a former elementary teacher, just because students have the option to use voice to explain their thinking. And of course it has a feature that a lot of these have where you can upload like a picture, things like that. And it'll kind of extract the data for you as the teacher, so and summarize and help you make conclusions, which I think makes it a little more comprehensive than some of the other tools on this list. And definitely like a teacher facing, like helpful tool for the teacher to use with students. Khanmigo is the one. Everyone knows I speak to it a lot just because I use it with My son, like I said, he is in middle school.

34:32
Mandy DePriest
The math got a bit beyond me this year and so it's been helpful for him to have access to Khanmigo. He doesn't rely on it most of the time, but when he gets stuck it's been a great tool to get him over the hump. And I have never seen it give him the answer. In fact, sometimes frustratingly so like it'll force him to work through the problem Anecdotally I have heard other people say that it has returned inaccurate calculations, but that has not been my anecdotal experience. So I'll just add that. And then we have several like Math Way and PhotoMath and Goat Math Month will all support uploading a photo so that a lot of students really like that because they can work it out, you know, by hand and take a picture and get some error analysis.

35:18
Mandy DePriest
And in return I see these showcased a lot on social media specifically for so I lurk a little bit on Tick Tock for the teens and not in a creepy way but just to stay informed about like how they're talking about these AI tools. And the way I'm seeing it framed is like these are sacrosanct. They're always right. This will get your and it may be it will a lot of the time, but there doesn't seem to be any awareness that like we need to check these results. And so I think that's something math teachers really need to emphasize. If you're incorporating these tools and say, hey, just by the way, you can't assume that it's always going to get it right every time. I think that's a good opportunity to build some AI literacy. WolframAlpha and Desmos are much more sophisticated.

36:02
Mandy DePriest
I would use them for higher levels math. Wolframa in particular will integrate with like science and engineering and other things. They have a ton of resources on their site. And then Julius AI I like to highlight it's kind of a sleeper. I don't hear about it as much but I think it's been pretty good in the use cases I've tried it out on. It specializes specifically in data and Amanda mentioned generating data sets for things. And like we've used Julius before to generate synthetic data sets to demonstrate like prompt for trainings and things. You can just ask it, you know, hey, you know, generate a.

36:36
Mandy DePriest
I've used it to generate a false set of data or some synthetic data for a typical College class of 150 Intro to Lit students like simulate their grades or their GPA or whatever, their performance on the midterm versus their performance of the final. And then you can ask it to kind of connect, look for trends between the midterm and the final and like participation in discussion boards or something like that. So I would play around with Julius for sure and other things, but like I said, there are a million of these tools out there. I haven't been able to keep up with the chat and talk. It's like, you know, walking and chewing gum. I think at the same time it's hard for me to do.

37:12
Mandy DePriest
But I hope you guys have been dropping in the chat resources that you like that are these kind of application layers that are specialized tools that help you use more reliable masks.

37:24
Amanda Bickerstaff
Absolutely. And so I think there's a question and a couple of questions in the chat, but one that I'll call out is that remember what I said, they're vertical AI and that they're heavily coded. Large language model is going to be probabilistic and random every time you use it. The applications on top of this or the specific specialized tools have a lot more structure and are going to have in some cases always the same right answer or always going to a calculator or going to Python or R to code and do the analytical pieces. And so it is something that is.

37:58
Amanda Bickerstaff
That's why we are suggesting so strongly using the applications themselves versus just large language models because they have significantly more and they're still artificial intelligence driven in a lot of cases, a lot of different types of artificial intelligence models or algorithms, but they are going to be more reliable because they're built to be more reliable. Generative AI like ChatGPT, Gemini and Claude are built to be creative and to respond and say yes and to try and to answer everything. And so that's going to be quite different than even like Khanmigo. If a kid just now asks for the right answer, it will say no, why don't you try it this way? Or if someone gets discouraged, it has the, it has a prompt injection that says respond this way to encourage students to keep going.

38:42
Amanda Bickerstaff
And so it's going to be very different than just using ChatGPT as a tutor that can say anything at any time. So we want to keep rolling like Janie, do you have something there?

38:51
Mandy DePriest
Actually, Amanda, I wanted to, if we could go back a little bit and highlight some of the use cases that language models are because we list those first, we maybe went out of order. But so like I said earlier, it's not that you can never use these Ever in a helpful context. We have listed here tasks for brainstorming, like generating alternative teaching strategies, creative ways to address standards. If you're tired of teaching something the same old way every year. Clarifying abstract mathematical concepts. I've explained this every way I can think of to explain it. Give me 10 more alternatives. Designing innovative projects.

39:25
Mandy DePriest
I signaled earlier that we want to like start moving beyond like the way we've always taught math and maybe looking for ways, like June was saying, to make it more meaningful math, like connected to the students lived experience and things like that. Always with an eye to things like bias and developmental appropriateness and things like that. Designing student friendly communications. Like maybe the definition of the quadratic equation as presented in this textbook is not accessible for a variety of reasons to students in my class. And so maybe I can generate things they will understand. Certainly translation. Ideally you're having it checked with a native speaker, but you know, usually like you can get the job done with it. Identifying common misconceptions can be great for pre planning.

40:07
Mandy DePriest
So when you are working on materials and ways to present, especially if you are a newer teacher, it will help you kind of internalize where to anticipate those misconceptions and maybe how to proactively get out in front of them and address them. And finally, I mentioned translating before with the caveat that it's nice to have a native speaker review if you can. So I just wanted to signal those like we're not like totally down on ChatGPT or Claude or for any use case in the math classroom. It's just in kind of that those specialized circumstances.

40:36
Amanda Bickerstaff
Absolutely. So Jude or Jasmine, before we roll, we're going to go down to this last kind of, these last kind of pieces. But Jude or Jasmine, do you have anything to add around like the difference or like what we've just been speaking about? Are we good?

40:54
Jun Li
I would just emphasize what Mandy shared at the beginning or towards the beginning around learning opportunities. Like when we talk about AI literacy. I think that there still is a learning opportunity for 2% students as critical thinkers in the classroom and be like, well, here's what was produced. What do we think? You know, as opposed to always being an assistant in the sense of like supporting student thinking, but instead for the students we presented as a critical thinker to the AI output.

41:19
Amanda Bickerstaff
Absolutely. Totally agree. Jasmine, do you have anything?

41:22
Jasmine Costello
Yeah, I'm also just thinking about how when presenting being like kind of pulling back the curtain with students and being more transparent around like, oh, let's dig in this together, let's Analyze this together. Oh, this could be a way to use the tool that supports you in alignment with our learning goal and is not using the tool to take the cognitive lift away from you and engaging students in that process to really build their understanding of how they can, you know, use these tools in support of their learning goals and not as like a cheating tool as. As we're seeing concerns about.

41:56
Amanda Bickerstaff
Absolutely. And you can see that's a perfect segue, Jasmine, to these key principles of like, you know, support, not sick, prevent the. That thinking and that productive struggle and enhance, not replace. We talk about this all the time. Someone actually said it in the chat that they're finding that they're starting to, like, look for solutions instead of doing it on their own. Like, we would say that we'd love to see you enhance this. And sometimes maybe it is good to. Once you have everything in set, you can evaluate the outputs well to actually offload that, but not until you. You can really evaluate it. Understanding students developmental readiness to use these tools. Like, I've said it just today, like, I was on a futurist talk today in Canada. It's been a very funny day.

42:39
Amanda Bickerstaff
And it talks about don't give a kid a brainstorming tool. And so they can evaluate. They can actually brainstorm because they can't evaluate the output. The same thing I would say is that don't give a student a math tool until they can evaluate. Do the math that we're asking them to do so they can evaluate and double check and verify to June's point earlier. But also, we want this to not feel like it's replacing a teacher's pedagogical content knowledge and awareness, because that's going to be really important. That's how you're going to know. And like, one of my favorite things is like, when generative AI messes up, use that as a teachable moment. You know, like, this is like you, like what June was saying. Like, this is like, hey, does anybody notice anything interesting about this?

43:16
Amanda Bickerstaff
And then, like, what they do is you use this. Okay, so that's giving you the wrong answer. Let's actually use this and to get to the right answer. And then how will we do this next time? We would always have to double check. But losing those. Those opportunities actually to support deeper learning, like, because one of the things that we've seen, like, if you give kids an accurate ma. An accurate set of data and ask them to find where they have to, like, actually double check that it's accurate at all is a Great critical thinking skill. And also just having them do it all the time so they just don't blindly trust what you give them is going to be something that I think can be really powerful.

43:49
Amanda Bickerstaff
So the last part of the document, right, Mandy, or the biggest part is after we get through here. And of course. Oh, we have a note for elementary as well, but we'll keep rolling, is that this is the real key here. And so Mandy and Jean, do you want to talk a little bit about. These are where the real work of the document is going to be.

44:09
Mandy DePriest
Yeah. So this is. I mentioned the worked examples earlier and we have the slide where I was telling you the ones we have. So here's where you can see it in action, where we have the use case and we have them divided up by how students might use AI, assuming age appropriateness and parental consent and things like that. And then separate ones for where the teacher is using it, like maybe for planning or creating instructional materials and things like that. That. So the way it works is you can kind of click in and we have these worked examples, we call them, if you're not familiar from our. Our literacy document, where we kind of set up the scenario and then we put the conversation that we had with the chatbot side by side with our analysis.

44:52
Mandy DePriest
So you can kind of see why we framed it this way. This one is long. This is the tutoring one. So it has a really long initial prompt. They're not all that long, but if you want to see the actual chat, we've linked those there as well. You should be able to access just like that and kind of see for your. Yourself how the conversation went. But you can just kind of scroll down and see. Okay, this is a promising start. I like this. You know, here's what the chatbot did. That was good. Here's where there was a problem. I think this one went off the rails. The chatbot initially told me that I got it right, but then kind of walked it back and made me confused a little bit.

45:27
Mandy DePriest
So just kind of highlighting what we're thinking and how that influences our prompting. And we have that. Those for. Oops. Let's see. Like I say the. There we go all the way back down. Let's see. We have those for engaging with AI tutoring. Here we are recommending you use a specialized tool because it's error analysis. So something like snorkel or things like that might be more appropriate. Okay. Practice with procedural skills. Again, they're computing, so we suggest using a specialized tool. These are the considerations that you might want to take into account in these columns. So we have like the. The positives of using the tool like that. What is the benefit of using Gen AI in this way? This can be helpful for communicating with parents and other stakeholders. If anyone were to ask, why are you using AI in this way?

46:14
Mandy DePriest
You can say, well, these are the possible benefits, but then also some cautions and considerations to keep in mind. Things like, you know, the tool is going to be limited to its training data, so if things stray outside of that, it might get confused and return mislead responses. Kind of scaffolding things such that students are moving from highly scaffolded to more independent use and they're not dependent on the AI tool, things like that. Those considerations are all on the far right. And then for the teachers we have creating materials, lessons and interventions again, you'll see we click in, we get the conversation, you can see the chat here. And we use ChatGPT for all of these just because it's the most commonly used one.

46:54
Mandy DePriest
This would be a similar interaction with something like Claude or Google, Gemini or whatever else you might want to use it for. But that's essentially how we've set up the worked examples. Hopefully those would be helpful to you in your circumstance. Let me get back into the document. There we go. It always starts me back at page one. Amanda, anything that you would add?

47:19
Amanda Bickerstaff
No, I mean, I think that one thing to consider though is we did try on other tools, but we use ChatGPT just because it's the most common. And so like we showed you all Claude before Gemini, others and so just know. But this, I mean, I think that the most important thing. And if we want to come off share now to kind of. I'm going to. We'll end with just a couple of like final thoughts from the group. But I think the most important thing that we learned through this process is that it is so early, everybody, it is so early.

47:47
Amanda Bickerstaff
And we have this tendency to, like Amanda said at the beginning, like, if it can do this well and it is willing to do this, it doesn't necessarily mean that everything it's willing to do, like a generative AI model is going to be good or it's going to be capable. And so that requires that AI literacy component. But what we really hope and our goal for this document is that you as those on the ground are actually trying this stuff out and we want you to experiment. And there are some things that may have worked for us that don't work for you or may some things that work for you as the technology gets better.

48:25
Amanda Bickerstaff
So we do hope that this is a galvanizing action for you to feel like, more confident to try these things out, but maybe also talk to people around you that might be using these tools in ways that can create more misconceptions that could potentially harm student learning or to create a false sense of security. And so I think that's something that we really want to. Like I said everybody, we had the best. We were like, we're going to get this done for like March. It's going to be great. And then it got to this point where I think June and Mandy, you could talk a little bit about this. You had to be like, actually this isn't. We can't do it this way that we had intended to do.

48:59
Amanda Bickerstaff
So Mandy and June, you want to talk about that and maybe give a final thought?

49:02
Mandy DePriest
I would say it was more often than not that I would be running a conversation, having a conversation with chat GPT and just like face palm a little bit like you were doing so good. Chat GPT I can't help but anthropomorphize a little bit. But like, but now you went off the rails. And so like I said, I. I can't recommend with confidence using it without particular framing and guardrails in place. You know, as long as we. We have that inconsistency. I mean, June, you and I went back and forth quite a bit. What was your perspective?

49:37
Jun Li
Yeah, I would just emphasize like it is a thought partner and which means we as the human in the room of. And participate in my thinking similar to how we said it can't replace teachers. Like, I think it. We. I think the power is it create. It could share ideas. It could take us in directions that like on our own, we may not be due at that speed. But there's something. But we need to show up just as much to the other half of that partnership in analyzing the results and deciding what we want to do with the outputs.

50:07
Amanda Bickerstaff
Oh, Jude, I love that. I mean, I think that is absolutely the way to think about it. And you know, just because it's. It's something makes it easier, does it make it better? I think that's going to be a really big question that we have even when it does work. Like, and I think that June, you. You and Jasmine, everyone SAP recognizes how important like how more important the foundational math skills are going to be. Even though we have tools that are going to be able to do it, but the ability to do it and learn and Internalize this and evaluate and be able to be critical, you know, math, like people, you know, people. That's not the right way to say it, but those that have those financial skills can be really important.

50:45
Amanda Bickerstaff
So, Jasmine, you want to wrap us up and give a final word?

50:49
Mandy DePriest
Just. Yeah.

50:50
Jasmine Costello
Appreciation to you all, like engaging in this exploration. Like as Amanda said, you know, went in thinking this was going to go one way and it went a whole other way. And we got to learn so much more through that process and invitation to like everyone here to join us in that. As this is just the beginning, we want to learn from you all. So please try things out, fill out the survey, you know, connect with us as we're all kind of just learning together and. And need to apply our own critical lens as we do that.

51:22
Amanda Bickerstaff
Amazing. And I just want to say thank you. It goes both ways, but also thank you for everyone being a part of this with us. We are so lucky to have such an engaged group and sharing all the resources and thoughts and. And it really makes a big difference. We will be sharing the recording as well as the resources. But also please check out the guide. Give us feedback. Always reach out. Hopefully maybe there'll be a new one. Someone said in the chat, Jasmine and June that like a science one. So maybe there'll be more guides in the future. Although I don't know. I think after the craziness of this one we might take a little bit of time. But just appreciate everyone, have a beautiful morning, evening, afternoon, wherever you are, and we will see you next time. Thanks everyone.