Token maxing is one way of bragging, right? Right. It's like how many pounds do you lift and so on and so forth. But really, you know, are you getting
hernia or are you getting strong? We don't know. >> All that and more on this week's Mixer of Experts.
Hi, I'm McConnA and welcome to Mixer of Experts. Each week, brings together the sharpest minds working at the frontier of artificial intelligence
to take you through the week's news. And this week, it's New York Tech Week. So we are here at IDM's one Madison office and studio to bring you some
of the on the ground conversations and with me today I have Kowar El McGrali principal research
scientist in AI platforms Neil Sundares general manager automation and AI and Thuru Venatulum senior partner in enterprise AI transformation leader
after the panel we'll have a special section on AI and higher education with Justine Nixon Santel our VP of CSR and chief impact officer and today
there's three big stories we're going to cover. Um, token maxing is back in the news. We're also going to
talk about Nvidia has a new super chip. But first, let's get into the big conversation from New York Tech Week. And this is the future of software
engineering. And certainly um, on the various panels that I've sat in on this week around the city, there's been talk of course of AI replacing
entry-level coders. On the one hand, we have data on that front and then on the flip side, you know, people
are arguing that data engineers, you know, people who can supervise these systems, who can test them, you know, kind of deploy them and basically, you
know, have the human in the loop are, you know, even more important than ever. And so, this seems like a great topic to kick off with. And, um, Neil,
I know this is a topic that you're covering this week in tech weeks. I'd love to, uh, start with
you on the future of software engineering. >> I mean, we have seen this in software over years. Better tooling, better u uh infrastructure, better
ecosystem makes software developers productive always, right? So even before AI, we did that. We had metrics on how to measure developer productivity.
Years ago, it used to be lines of code, then number of PRs, um number of commits etc. And you know
when your metric becomes the output, then you know it be it gets tends to get gamed right. So we are seeing that with AI as well right AI is changing
the way we develop software. I mean increasingly software is AI agentic. So in some when I'm starting to design any software system I'm going to say
what are all the different components of it. Again this was not this is not new to software component
based software development or service based software development or architecture was always known but now we know that there's intelligence in each of
these components and intelligent in each of these agents and how do I orchestrate out how do I bring them together is at the design phase and then uh
below each of these systems is there AI which is translate to LLMs how do I bring these LLMs into
play now one big thing that has happened that has changed um over the last I would say seven or eight years in the way software is looked at. So for
example if you can I think 2016 was it when Andres Karpati said software 2.0 where data is code who needs code when you have gradient descent is kind
of the idea. So I could replace chunks of my software with data and then put a machine learning model
on top of it and change it. So if you take for example the Tesla self-driving car right um story goes that it was hundreds of thousands of rules which
said how to you know make the next decision when the car is a self-driving car if it's at a stop sign or there's a bicyclist passing etc. Now if I
have all the LAR data, all the camera data, all the video data, can I build a model on top of it and
make automatic decision? That was the big thing, right? But what changes with that changes that is we go from deterministic rule systems or
deterministic logic that built in the code to probabilistic system that intelligent agents tell you. So these agents are only as good or these models
are only as good as the data that's fed to them and what they learn from them. Right? So that was software
2.0. Software 3.0 We got into foundation models or frontier models where there is general intelligence. I mean quote unquote intelligence. I'm not
going to say human intelligence. It's like there's general knowledge and intelligence that's built into the system and I can extract from that those
frontier models to bring it to my ecosystem. So it's like when I have the GPT54s or the sets etc of the
world, how do I bring that? I mean this is the large language model. Small language models have a different play. Now when you go to software 4.0 go
which is kind of what I think of as the next generation of software. These software systems are going to be a bunch of agentic systems that are going
to orchestrate you know um either they're going to work with humans or they're going to work with
each other. So human language is a language of communication um and these agents will either communicate with humans or communicate with each other.
Today they communicate through human languages but tomorrow it could be something else. So that's where we are going into the future of software
development. That's fantastic and I'd love the others to jump in. You know, the even as we get more and
more complex with these agentic systems, you know, it seems like the role of the software engineer changes, you know, in terms of kind of
orchestrating all of these. Certainly I I have um a different angle to the same perspective that um Neil offered because I'm on the consulting side
where we do custom application development for our clients and in addition to the evolution of this uh capability
on commercial software development like Neil talked about the custom software development on the consulting side is dramatically changed and um what
we have as capabilities this quarter in Q2 of 2026 we did not even have it in 2024 last quarter 20 Q4 and uh it is a it is having a profound effect so
just to give you an example uh many of the coding engines that we talked about whether it is IBM's
amazing Bob and I've used it's fantastic cloud code which is very popular or codeex um and other coding engines we in IBM has created a harness on top
of it because if you think about software development uh and I'm a developer myself uh started my career in mainframe you do a number of things
outside of coding right you do requirements gathering at the front end uh you interact with other people
to collect that requirements you create design documents high level low-level design and then comes the coding and then post coding there is testing
deployment What we are able to do in commercial software development for our clients, we are able to automate this full end to end life cycle what
would take maybe a 6 to 8 months period now into a very cont uh constrained period right and um let's
say a month and it's getting better by the day to a level I'm one of the ones that are very skeptical about its possibility because being a developer
you're always questioning is it going to be as accurate as what I would code but it has um opened up a lot of possibilities. So what we have now that
is gamechanging is on top of these coding engines which can actually code we're able to automate the
full life cycle of the SDLC with context engineering and memory management at the front end. uh we provide as much input to the computer and AI so
that it is able to write the requirements document allowing us to iterate it's writing the high level low-level design allowing us to inspect and then
it writes the code like Neil said end to end and then it is doing the testing and even deployment I'm
able to and differentiate whether it should be deployed in Azure versus AWS or IBM cloud uh so it has come a long way is what I see >> that's
fascinating and yeah the strong consensus is software engineering is more important than ever as it evolves. And Katra, do you want to um add your >>
I think these are all great points and I totally agree uh with what was said. U but there is I think a
different angle here which is how is this the role of the software engineer or the developer changing over time you know right now it's not about you
know writing code or syntax and so on. And I think it's shifting to verification, a systemto-system architecture. It's it's shifting also to like
overall integration, the efficiency, the security, those are really important skills. But also this is
also bringing challenges to like the new uh you know like the students you know doing software engineering studying that what should we focus on you
know that on boarding right now is also kind of we're facing challenges like how do we onboard right now these uh new fresh graduates right and to get
them to that because right now those entry-level jobs that you mentioned they're disappearing
they're being replaced because it's all automated now with all these code generators and so on. So right now the skills we're really needing it's at a
higher level. It's the verification, the optimizations, the security, the hardware software could design. Those are things that these LLMs they still
struggle to do well, especially having that holistic view, integration, getting the efficiency,
performance, energy, all of these things are becoming super critical. And as you know now we're having more code generated at this rapid rate. We're
also hitting the point that okay we're generating more code more systems more you know uh holistic end toend pipelines and so on that are fully
automated that raises complexity more code more technical depth that we're getting. How do we manage all of
this? How do we make sure that you know we're are we generating all this code? Is it needed? Can we maybe shrink that you know code base? And that
requires somebody who's really understanding the system deeply. So that's on boarding. How do we you know I think get that hurdle off because
initially we had a sandbox where new software engineers they will build experiments and so on. That is sandbox
that we had initially for new beginners is disappearing and I think that also should revolutionize the way we educate and train software engineers. it
has to change so we can keep up with you know this new world of AI generated code. >> Yeah, just to add to that I think it's a excellent point about
the evolution of our roles. The developers are becoming agent builders that >> or orchestrators >>
orchestrators because it's not about coding it's about how do you define the AI agent to do the coding >> testers are becoming agent auditors. So when
you when AI develops the code, you can't go and edit the code that is gone, right? So what you could do is validate the output and the rules that it
has invoked to produce the code is what you could do. So it's definitely evolving and we have
evolved that we are creating that are auditing right so excellent >> even think that I mean we used to have these human computer interactions I think
now we're shifting toward agent computer interactions and the role of the human in the loop is also becoming kind of a verifier orchestrator and
deeper like system thinking skills are definitely going to become very important >> so if you look at IBM
right we have we doubling the number of entry- level engineers and you kind of started by saying hey software engineer jobs are disappearing actually
I have a very different view of that you can go back even two years teams that hired new software engineers what do they do in the beginning they're
given some document to read or they ask some testing to do most of them play ping pongs for the first
three months because the managers don't have time for them right and that is changing I mean we are running an experiment now we've onboarded
thousands of software developers into our division and we give them tools like Bob now Bob and automated augment in the entire SDLC life cycle. So
suddenly they feel more empowered right? So in some sense you know we have internal studies have shown right
IBM builds enterprise software right so if I if I were to say hey take the software and then get it fed ramp ready now entry- level engineers do not
know what that means or how to do it so they need a starting point and tools like Bob become their starting point so in some sense we often say I mean
this is again feedback coming from developers within the company which is you know Bob is the
distinguished engineer or the fellow or the senior engineer princip engineer for a beginner developer and for a principal developer or a fellow it's a
beginner engineer who would take those ideas and put them into code right so I think time will tell when we can talk in six months and we'll tell you
exactly what the journey has been but we see more and more people uh software developers when they
start fresh college right are able to do things that they were not ready to do or they were not empowered to do or they were not trusted to do right
so actually that's a positive side of the story, >> right? I like this idea of Bob having apprentices and you know we can unleash Bob on the all the
summer interns that have flooded into >> exactly what we do. So in their welcome bag they get a the
Bob swag and a Bob API key and then they go from there >> and that Bob Green can be involved in the upskilling as well >> and then you'll hear in the
afternoon in Justina we are working with university to create curriculum >> to get them ready for this AI style software development. I mean the idea
is not new right you can go back 40 years Donald N said well you know you know the idea for literate
programming was there before there was AI and it was like if I have an assistant and if I could speak to the assistant in English or human language
and say hey go code this up for me I just need to be a domain expert >> right I don't need to be that engineer so while people say it's taking away
software developer's job I think everybody's becoming a software developer and we see that >>
democratizing so internally yeah internally we are seeing Bob being you not just be used by software it's used by infrastructure consulting operations
finance I mean I was a CIO's office yesterday communications you know everybody's using Bob in their own way marketing everybody's using Bob's I mean
um so people are more empowered or and feel more capable right >> this is a rich topic and we'll
we'll move to the next we could talk and I'd love to follow up actually in six months and see how it's all evolved our next topic that we're tackling
is token maxing And this is back in the news. Um Uber's CTO made some comments that went viral about, you know, whether um it's going to be hard to
justify the AI spend, you know, when in the case of Uber, I think they blew through their token budget
for the year in 4 months. Um you know, and kind of leaders are rethinking, you know, how to incentivize employees to use AI. Um and I I know there's a
new term floating in the ether value maxing and uh in terms of uh how companies and need to be a bit smarter in terms of the AI spend and I feel like
this would be a rich topic and I I know uh Neil you were interviewed on it recently so why don't
you kick us off >> this is that is something called good arts law everybody talks about it these days I mean we are learning all kinds of new economic
terms devs paradox or good arts law but the point is if you create metrics and tell people to optimize on those metrics people are going to eventually
game it. I mean I got a great story from my economics friend who is to the he has three daughters
and uh he was treat um encouraging his older daughter to potty train his youngest daughter and he told her every time you take her to the bathroom
you'll get a jelly bean and suddenly he was seeing that the older daughter was eating a lot of jelly beans and the younger daughter was going to
bathroom all the time then he was seeing that she was feeding her water all day long. So you can create raw
incentives uh by you know creating a for reward system right so it's kind of like that we used to have lines of code as a metric so what would people
do they write a ton of lines of code in fact back then when you program in C++ people would argue about where to put the brace because we put a brace
below you get an extra line of code right uh counting PRs counting PRs we are withouting counting PR
reversals uh number of files that were changed I mean all these metrics have been used even with AI and every time this metric was used it was um it
was gamed or it was I mean it turns out to be incomplete metric we measure these things because we can measure them because there is no real measure
of productivity right um so token maxing is exactly the same things like hey if I use a lot of tokens
that means I'm very productive but you know it comes in various shapes number one I can write elaborate prompts for some simple questions I can make
it create elaborate outputs for some simple answers it should give the models can game too and we see that so for example the reason people talk about
tokens is because the model providers measure you know they charge you buy tokens so they say hey
mill per million tokens etc but they can be verbose as well so you can compare an older and newer model and you can just say for the same task it is
actually using a lot more tokens than before so it doesn't really matter how you price your million tokens right if you if you say oh my million
tokens are $10 and your million tokens are $20. I could all I need to do is generate twice as many tokens
and I'm good, right? I I can game the system. So model providers are getting system, the tools are gaining system. And when we talk about tools like
Bob, it is not just the models and not just the tokens. It's also the uh the context engineering that the developers have to do. So it is kind of the
harnesses the developer produce, the skills of the developers as well. So we with the same model we
can produce very different kind of results in terms of token consumption in terms of quality of the products in terms of the uh you know performance
of the product etc. So we always think about like you know in Bob we have we have multiple models that orchestrate and we measure in terms of cost in
terms of quality and in terms of performance and this parto frontier is what we try to optimize on so
it is not just the model just not the number of tokens um of course and token maxing is one way of bragging rights right it's like how many pounds do
you lift and so on and so forth but really you know are you getting hernia or are you getting strong we don't know right and a lot of people and I the
mistake was made where they created a leaderboard and moment there is a leaderboard people want to
be on top of the leaderboard right and we see that even with AI it's like oh we beat that model by 5%. because you're optimizing. I mean, we heard of
Plma 4, for example, we optimize to be on top of the leaderboard, not to solve the problem. So, the focus has shifted because of that. And I think
it's one of the many metrics. We should count tokens, but we have to count everything else as well. >>
I like the jelly bean maxing. I was one of five children and there's a lot of jelly bean maxing when uh when people were being potty trained but also
the you know the deeper themes of the kind of the challenge of measuring productivity but also you know kind of you know measuring many more outcomes
or you know outcome focus and um yeah I'd love to bring Karen Theoru uh you into the conversation on
this >> definitely token maxing by itself is not the right metric and I think Neil said it rightfully here uh because that also has implications on
the infrastructure token maxing means like huge GPU you know spends and massive you know infrastructure costs and energy and all of that you know had
environmental effects and so it's not just you know just looking at the model but looking at also the
entire environment and the hardware behind this and all of that is very important but again I think the productivity uh measures it takes multiple
facets here it's you know what's the ROI uh you know while all of these tokens what's the efficiency also that you're driving from this so I think it
is a complex metrics like you said it has to take multiple things into account and it's mostly are the
tokens I generate lead into useful solutions and you know things product that ship and bring you know money and solve real problems and so on those
are the real metrics that we should be focusing on >> yeah just to add to that I think the outcomes are probably the ones that are going to stick from
what I have seen um the cost of tokens is kind keep going down uh continuously over the last 18
months the cost of tokens have gone down but on the same uh period the consumption of tokens have gone up dramatically right so that's the two parts
of it and it I draw parallels to uh you know memory management and storage costs uh I come from the mainframe world right so we used to do programming
just to move chunks of code in and out of memory just to say memory and has become you know a non uh
yeah back then it was super important so I think that tokens are going to go through that evolution their costs of tokens will become very commodity
be very minuscule at the same time I also feel that the consumption of tokens would go dramatically up where I see outcomes play a role is uh one of
the clients uh conversation I had was very telling the client said to me you we've invested in these
tokens, invested in these LLMs and AI programs. Uh they are working on it but I am not really sure that leveraging AI to do a particular work is more
cheaper than the human. Right? More importantly, I want IBM to tell me how can I measure the productivity and efficiency of a AI enabled workflow is
better, faster, cheaper than my human-led work. Exactly. Let's say take a call center. Right. And
when we are in a position now to transform a human-led legacy workflow into an AI, we are still not in the place we're evolving there to measure the
ROI of a business workflow at the workflow level. Right? We are working on it. We are kind of developing solutions. But that's what the client was
asking. Tell me, you're asking me to modernize this workflow, remove humans, make it automated, but can
you tell me it was costing me X dollars before and Y dollar now, and it's cheaper, right? The conversation also went to, let's say I cut $2 billion in
labor on the old legacy workflow because I've eliminated these roles and automated them, but I spent 2 billion in tokens, right? Where does it take?
And we concluded that there is one advantage there. With the 2 billion of labor, you would take
about 8 months to develop that function. Whereas even if it cost 2 billion on AI, you could develop it in a month. Right? So there is a speed to
value. You could develop it faster but it cost you the same thing element is sticking. Right? But we're still not there at a point where we can
measure the outcomes. we can measure the the cost of a redefined workflow. So, it's a it's an area that is
evolving. But certainly tokens are a very narrow way of looking at productivity. >> And I think also it's very important to also have very smart and
intelligent orchestration of these models. I think we do this very well at IBM like we don't you don't want to always use the frontier models for all
the tasks. You have to choose carefully. There are certain tags that you just need a small LLM,
right? And then these intelligent like careful routing. I think it's going to be important to capitalize on that because that is a big costsaving.
Maybe I can just run things locally. I don't need to pay like the cost of a big frontier model API for all the tasks. So certain tasks I just need a
small model certain tasks. I can just maybe use some simple regax expression software, you know, that
could do these things for me ex in instead of calling a big AI model. I think that's going to be also very important because the key thing is getting
to the MVP with the tokens that gives me you know the right products at the right cost. >> Yeah, I think that's why with Bob we don't actually reveal
the model. We don't reveal because we don't let you choose the model but under the cover we
orchestrate between the frontier models even the multiple frontier models open source models and our own SLMs etc. I mean like that's why we
repeatedly say don't take your Ferrari to buy. Yes, I was going to say that fits perfectly with Kowar's prun. >> It's really important because it's
also you're not going to get the value even if you use like for example if I ask you what's your name? You
really don't want to be thinking for five minutes, right? You want an answer right away. Um I'm not asking for family history. So and a lot of the u
conversations about token maxing comes from I go to a tool and I pick the latest model that's available because you know I got this principal agent
problem. I'm not paying for it. Somebody else is paying for it. Why not why not choose the Ferrari,
right? And and then I just let it run rogue and you see the cost and that's why you see the Uber stories or Amazon stories or who is that latest story
about spending $500 million in a month. I don't know if that's true or not, but you know stories like that because it's it's I think it's
irresponsible. There is there is a thing to be said to ROI, right? Many times you have to invest in technology
because it makes sense and the ROI may not be there right away. So we should measure too soon then we'll have the wrong metrics right so we should
continue to invest while we create multiple measures I mean a statistician a while ago told me we measure this because we can measure this that
doesn't mean that it's useful that's the only thing we can measure but as we go further and we get mature we
can kind of develop a bunch of metrics when you can say now is it adding value or not if on day one if you tell me a or b show me the value you may
not be there because these technologies are developing I mean remember GPD3 we said AGI is here no Not really right and we are here today much more
capable models and still there is work to be done. So I think that's something that should remember in
the back of our minds. >> So for our last topic today we're talking about Nvidia's new super chip the Nvidia RTX Spark which I think is interesting
because of the fact that when you know combined with Microsoft Windows means that people can you know run agents uh accurately securely on their
personal PCs. Um, and so I'd love to start with you cowtower because I feel like we were having a
conversation or I was perhaps listening to a mixture of experts episode uh a little while ago, you know, where you were predicting, you know, this was
in the wake of the open claw, you know, kind of kind of I'd love to have you put this um development from Nvidia in a bit of context of what's
happened this year and whether it's, you know, kind of important going forward. Yeah, I think it's a big
development that actually these announcements are kind of shifting a lot of the focus to edge and AI, you know, in personal computers. I think also
the partnership between Microsoft and Nvidia, it's a big kind of revolutionizing the PC. So one thing you know if you look at what's happening in the
data centers right now uh and the cloud. So Nvidia is also kind of trying to find other markets
because uh many of its big clients like Apple and they're they're you know building their own silicon. They're building their own you know AI
accelerators and challenging also Nvidia's dominance with that. So now you know it's I think a strategic move for Nvidia to go into the personal
computing and kind of changing the way you know we interact with our computers like having agents running on your
personal computers with no you know cloud access and that's also like I think security here is a big aspect because what this implies is you know you
can have an agent go like watch every stroke that you have read you know have deeper access to all your files personal records etc. And then you know
the OS right now is moving from to traditional kind of just maybe like a police traffic officer like
managing resources and so on to also having intent understanding what you want to do because traditionally operating systems you know you just have
it's just an application launcher that manages all these resources and tries to you know make things run but it had no idea what you were doing. So
the end user has to orchestrate open the app, click here, move things from here to here. Now we're
changing that. Now we're asking, you know, with the RTX announcements having these powerful, you know, GPUs. So they even, you know, if you look at
the hardware store, it's pretty impressive to be able to run 120 billion parameter model locally, right? Having 128 GB of memory, right? having even
you know the NVL link between the uh the the CPU and the GPUs locally all of these things were not
possible before in a personal PC so I think this is going to bring a lot of the agentic workflow to the OS but also it's changing you know the nature
of operating systems and the role that the OS is playing so it's I think it's going to be interesting to watch how this you know folds or unfolds >>
but I I wonder if this is you know revolutionary technology or normal technology. I mean I was
talking to one of IBM's earliest forrron compiler developers and he said oh we had to fit a 4rron compiler in 4K of memory back in 1960 and that's how
we how we invented the halfbird we come a very long long way from there to 128 gigs of memory on your computer we still it will run slow it'll still
not be sufficient right so we come a long way and the IBM was there back then is there now you can
run a tin parameter model on your machine and that's great but you know it's part of the evolution. It'll >> be interesting to see where it goes and
theory. Do you want to close us out with some thoughts on the >> topic? I definitely feel there is um this is a big development in the world of AI and
to some extent rebirth of the PC and its relevance in the computing world because if you remember
you know back in the days the servers and the mainframe to distributed computing to mobile computing there was a forward momentum towards you know
continuously going towards that. But this actually sort of takes us back a little bit to making the PCs more relevant and I think that's going to then
evolve into again edge being more mobile right the same thing is going to happen in mobile chips. It
is super interesting for someone like me to look at it because in the age scenarios that I love cloud art we were all experimenting it's doing some
things that are extremely helpful. For example, for years I've struggled with finding that one email from my email inbox that I could not find. It's
able to find it, right? And it's able to uh, you know, I had like the storage maximum and it was able
to find the two videos that were hogging the maximum space and intelligently I actually copied that video one more time. for whatever reason he was
able to delete those two videos very quickly freeing up um space. So in terms of personal productivity this development is going to be a gamecher in
the role of PC u in what it used to be to where it's going. I'm very optimistic that there's going to
be a lot of uh future uh development in this space as people who are innovators come up with new use cases of how to use it cuz now with this chip uh
PC is now enable to and to do it people will come up with ideas that will be very gamechanging >> well theu karil thank you so much for joining the
panel I'm now going to move us on to our next segment which is looking at AI and higher education
I'm joined today by Justina Nixon Santiel, vice president of corporate social responsibility and our chief impact officer. Justina, thanks so much for
joining us on Mixture of Experts. And I'd love to start on the heels of New York Tech Week. We were just on a panel about AI and higher education.
What are some of the top challenges that you know administrators, students, professors are facing with
AI adoption at this kind of moment in time? Yeah, I mean this is so topical. Uh, you know, AI is a huge challenge for universities these days. Um,
number one, students are graduating into a workforce that's completely different from when they entered university four years ago. Professors are very
focused on how do I retain students critical thinking, right? and retain the information and knowledge
they've been gaining for years when they now have this AI tool to their disposal. Um, you know, from a governance perspective, right, administrators
and others are thinking, how do I protect student data? How do I make sure that students are being assessed the right way as well, which you know,
assessments have to change. Um and then how do I look at all the different tools that are being brought
into the college system and making some real decisions about safety and you know AI ethics and trust. So those are the things at the top of everyone's
mind. But I would say students are also thinking about this very carefully because again they entered university in a in a market that's very
different than what it is today and they're really thinking about how am I going to get a job? >> Yeah. you
know, so all of those things are coming together and you know, we're we're having a lot of discussions with higher ed institutions about some of the
best practices and some of the ways that we can help them and partner with them as well. >> Yeah. And I'd love to follow up on the partnership a
little bit later in the conversation, but I'm fascinated of those, you know, the various groups that are
wrestling with it, administrators, professors, teachers, students. Do you find that any of those groups are the most optimistic about the potential
for AI? you know it's it can be a tool to you know empower of course um as well as you know needing to be governed and regulated but yeah how do you
find that balance kind of >> I mean I think students are optimistic right this is technology that
they've grown up with overall right these are like digital natives and they've used technology ever since they were very young and now this is a new
tool at their disposal so on one hand I do think that they're very optimistic about AI do not know a university student that has not used AI and um
you know so they have access to the tools they're using them I think on the other side of that is a bit
of apprehension yeah >> right so although they're excited about having this tool they're using it in different ways they're also thinking about what
are their future work um prospects from a workforce perspective and what does this mean because the workplace is changing so quickly so I do see both
sides in many of the students that I meet with but just generally you know as you meet with
professors and higher ed um leaders. Those are the things that they're thinking about as well. >> And you mentioned uh earlier like assessment and you
know many facets of education having to change but I'd love to yeah dive in on you know kind of some of the ways that people or you know leaders in
the field are thinking about how assessment needs to evolve um you know given the prevalence of the
AI tools. Yeah, I think assessment is going to completely change. Um, you know, if you give an assess a test or a paper uh for students to complete,
they can use AI, right, to acquire the information they need to um, you know, create the answers or to write the paper. So, you really have to think
about assessment in a different way. A lot of schools are moving to oral assessments, right? making
sure the student can actually defend right the homework the assignment um without any technology or you know any tools um in front of them um and
others are actually accepting it and saying okay great use AI um and I think the best schools are moving in that direction it's okay you could use AI
but then you have to demonstrate to us right what have you learned from this when did you use human
judgment right what did you think about the output that you got from AI and does it make sense? So, I think the professors who are probing a bit more
on these assignments, um, it actually mirrors the workplace and I think that's where you're going to prepare more students to be successful when they
graduate and they get jobs. >> That's so interesting. And the sort of plays into the theme of the
sort of humans in the loop, you know, and kind of tying, you know, teaching students, you know, universities being places to hone critical thinking.
Of course, critical thinking applies to, you know, before it used to be the kind of how do we use the internet? How do you trust what's going on and
it sort of feels like the, you know, evolving the kind of pedagogy around, you know, critical thinking
to factor in AI. >> Yeah, absolutely. And I think a lot of schools are also bringing in more practical hands-on learning opportunities for students
because it goes beyond just understanding the information or um, you know, the content. It really is around how do you then demonstrate teamwork and
collaboration and the human judgment and you know your presentation skills that all needs to be built
into a university structure and you know many universities have been doing this you know for quite some time but I think it's even more critical now
that this is just integrated in every classroom in every uh situation and this hands-on learning is really how students can demonstrate competence
right ac across whether it's an AI tool, a topic, etc. Especially when they are moving into uh the
workplace. >> I like that in the sort of idea of extending evaluation to even like you were saying some of those skills like teamwork that you know
things that AI can't do where you know perhaps those had a role in the past in higher education but they weren't as you know as part of a formal a
part of the you know what students were being um evaluated on. I I'd heard one university as well or I
think there's maybe a movement of universities you know in the realm of assessment as well for you know actual exams having people kind of go back to
sort of paper and pen. So you used all the tools you know to prepare but now you know make sure that you can kind of also um demonstrate. So it's an
interesting kind of the combination of like um you know the new and old. >> Yeah and I agree. I think
all universities have to decide what makes sense for them and the governance structures right around AI use. What we're encouraging universities to
not do, yeah, >> is to ban AI. I mean, the ship has left, right? Like it has sailed and the workforce is using AI. Every major company's investing in
it. So, you want to make sure that you give access to those tools um in a responsible way to students
and make sure they are using it. And then you have to bring in the layer of the assessments and how do you uh make sure they're actually acquiring the
knowledge and they're they're demonstrating competence, but you don't want to ban those tools. Students are using it anyway. >> And it feels like you
are doing a disservice as you say for when they go to the workforce if they you know because I feel
like one of the amazing things about a you know center for higher education is that they can get access to these tools that you know if they're large
frontier models they might not otherwise you know be able to. Um and I feel like the you know many roles of course are evolving. Um but the I'm
fascinated to chat a little bit about how you know the role of a professor you know and the person who
imparts the knowledge um you know you see that evolving you know in these discussions about AI and kind of higher education. >> Yeah. I mean let's be
honest um you know higher education has historically been slow to move and you always have the early adopters. you have the professors and a lot of
them we work with because they're the first ones that are willing to say I want to bring in this new
tool or this new practice you know I want to try something different I want my students to learn in a different way um so you always have these early
adopters I think what's different now is the speed of change so although you're continuing to work with these university professors who are willing to
step in who are willing to try different things willing to bring these tools you have to actually
move very quickly to all professors ers, right? So, you usually use them as champions. You usually use them as part of your overall change management.
But I think what's going to be critical is this can't just be the computer science professor, right? Or the engineering professor. You have to look at
the humanities. You have to look across all of the different disciplines and schools um at a
university to really think about how AI could be implemented everywhere. It can't just stay in one area. So I think professors are looking at how do
we bring this in whether I'm in the school of business right whether I'm in the school of arts how do I bring in AI in a very um disciplined approach
with trust and governance around it and then how do I change my curriculum right so to incorporate
the use of AI and then how do I change assessments for a professor to think about all of this it's a lot it's a lot so I think this is where
partnerships with IBM M come in. I think a lot of schools are creating like rubrics and principles around governance and also showcasing the best
examples that professors especially the early adopters are using to quickly be able to bring this across an
entire school system. And then you have universities like Purdue University and um I we just met with them. They've actually incorporated AI as a
mandatory class across all of their freshmen. >> Right. So to me that's where you get real change. It's no longer an option. It is something everyone
needs to do. And we have a number of universities that we've worked with to incorporate um our skills
build program and our content um in a similar way. >> I love that in this sort of idea of like AI literacy like you know you have kind of writing
workshops AI literacy being you know kind of a mandatory for all freshmen. And I'm glad you mentioned you know IBM's work in this area because I was
wanted to chat about you know the role of the private sector in helping higher education sort of step up
because as you were describing I was imagining various you know poetry professors who you know trained you know they're not you know as used to
perhaps a computer science professor in you know thinking of techn technological tools to you know um convey their information but yeah what sort of
uh role do you think the private sector and particularly tech companies like IBM have as this space
evolves? I think they have a huge role. Um, number one, industry partnerships for higher ed, they are so critical. Right. >> Right. Higher ed
institutions really need to understand what's happening in the workplace and they need to understand the pace of change and how companies are
integrating AI, what their expectations are for an entry- level hire, and then what does that mean for higher ed
institutions and how they adjust their curriculum and prepare their students. So I think these partnerships with industry so critical for higher ed
high higher ed institutions. Um you know when I look at the work that we are doing right and the partnerships we have we have with universities we are
bringing our expertise and our tools and our skills build program to universities globally because
providing that access to content right is so critical for students but also making sure IBM are part of this right we have so many academic advisors
who are IBM who are actually mentoring students teaching classes is as guest lecturers sharing with them how work is changing within IBM so they could
be prepared. So I think it's not just the responsibility of companies to provide that access to the
training and content like we're doing but how do you make sure they also have access to your employee base >> to really demonstrate those types of
changes. Um and then also how do you partner that with talent acquisition? How do you make sure that you're also looking at the pool of students who
have acquired some level of AI competence right and fluency and how do you look at that as your pipeline
not just for your company but for your clients so those are the types of things that we are doing in our partnerships with higher education >> that's
wonderful and yeah and I feel like it for all those students we you know spoke earlier about the fact that they started university the skills that
they thought they needed the sort of contract of like you know if I work hard in this in this area I'll
have a job you has changed and you know it's combined with the you know pace at which it's changing it's good to hear that you know IBM you know I'm
sure some others as well are you know helping inform and educate students because it's like these roles are you know all of our roles are changing
kind of almost by the day so to >> exactly >> you know for students to know what the how they're kind of
redirecting um you know seems to be a critical >> I think one of the things that we are working with a lot of students around is how do you really
focus on those other skills that you've gained. So if you entered school and you were focused on data science right or computer science and some of
those uh you know disciplines have changed right over the last four years. How do you take what you've
learned especially the hands-on practical um experiential learning that you've done and parlay those skill sets into what the new job looks like? So
don't focus too much on what you know the actual degree that you've gotten. focus more on the skill sets that will enable you to be successful in any
of the new roles that are now available to you. >> Well, and especially as each of our roles will
kind of continue to change and evolve, you know, to translate. Justina, thank you so much for taking the time to chat with us. And thank you to all
our listeners. If you like what you heard, you can get us on Apple Podcast, Spotify, and podcast platforms everywhere. And we'll see you next week on
Mixture of Experts.