So we've spent the greater part of this afternoon discussing the business
opportunities of A I but obviously with great advancement comes disruption and
concerns for the future, as well as the economic impacts of this powerful
technology. In the long term. There are a whole range of questions to answer on
this subject. Some of which we just got into at the end there. What will jobs
look like in a decade? How do we upskill and reskill the workforce so
that it's still relevant and how do we measure productivity? So we're lucky to
be joined by a great panel now to discuss some of the possible answers to
these questions. So please welcome for our next session, Eric Bryn Nelson,
professor and senior fellow at the Stanford Institute for Human Centered A
I. And he'll be joined by Shane Luke, the vice president of product and
engineering and the head of A I and machine learning at work day. And Atit
Rafik, author of Decision Sprint, a book that implores its readers to
constantly be ready for the unknown. And Atit is also the founder and CEO of
Ritual an application that leverages A I to help teams, collaborate on
innovation and problem solving and they will be interviewed by fortunes crypto
editor Jeff John Roberts.
Hello everyone. Welcome to the workplace panel. Um I'm going to get
right into it um With something I heard be a cost. Uh one of our keynote
speakers tomorrow say recently and that was um 80% of 80% of all jobs will be
replaced or seriously disrupted by A I in the next 10 years. And that sort of
like, oh my goodness. So I just like to turn to our panel, you guys sort of
study this stuff closely. Is that an exaggeration eric, you know, I didn't
know you were going to ask that question. It was funny. I had Reno over
for dinner a couple of weeks ago and he made that statement and everyone at
dinner was sort of like, whoa, is that for real? But he made a really good
case that it is going to be something on that order. And I agree that this is
a tremendously disruptive technology people have compared on the scale of
the internet or even electricity or beyond. Um The technology is advancing
very fast. What we've seen already is impressive, but there are things in the
pipeline that are even going to blow that away. Um The bottleneck is in
transferring that into changes in business organization and capturing the
value and that's where we still need a lot of work. And uh what do you think
of that is, is that overstated or not? I mean, at some level, you could say
only 80% right? But, but all joking aside, I mean, I think the thing we're
not talking about is really true knowledge, work in organizations. So,
uh when you think about typical white collar work of strategic problem
solving, creativity and things about uh of, of that nature, I think we're
woefully unprepared organizations are not preparing their teams to be able to
tap into A I to do it better and smarter. But on top of that, you know,
it's a vulnerability for the workforce because A I I think is the ultimate bar
raiser when it comes to knowledge, work for sure. And Shane just go to you for
a sec, I think, you know, with work day, you have a window into all sorts of
companies and what do they want most or what are they coming to for? What sort
of A I guidance are you providing them? Yeah. So, you know, when you talk about
the disruption and whatever the numbers end up being, there's going to be
changes in the workforce. And one of the things we see with workday is that
a big desire for companies, what they want is to enable their workforce to be
able to upscale reskill. And I think one of the important and powerful
things we can do is while A I will be disrupting many jobs, it's also an
enabler to help you upskill and reskill to look at what the future is going to
look like. And we do a lot of that and we hear a lot of that from companies
who are thinking about, hey, in 10 years, I don't even know what my
workforce is going to need to be able to do. How do we get ahead of that
problem? And we can use A I to do it while A I is also changing the way
people work today. Yeah, and I want to go back to sort of what cos should do
and operationally, but I just do want to go to the future a little bit too in
terms of like, I think a lot of us in this room might be wondering how do we,
you know, future proof ourselves. And also, you know, I think some of us here
have Children, you know, in high school are going into college and what on
earth major should they take? You know, you don't want to send them in to get a
degree that's going to be obsolete in two years. So Eric, that's, you know,
your advice for sort of, you know, surviving the coming. Well, Shane is
exactly right that these technologies are incredibly powerful but they can be
used to augment, not necessarily replace people. I think not, not focus
is on this capability of allowing people to do jobs better and
differently. So I don't really see any mass unemployment or mass replacement
of whole occupations. What I see is specific tasks will be enabled by this
technology. And the people who can look at the large scale problem solving,
broad creativity, how to apply these technologies in new ways and have their
skills augmented are going to do really well. But this ultimately, I think for
most people is going to be something that allows them to do more and better
rather than replacing and downscaling.
So what, what should, what should my kids study in college? You guys, I
think critical thinking, problem solving and deductive reasoning because
A I is actually already pretty good at that, right? If you feed it a problem
statement, your Netflix, you, you're concerned about password sharing your
Uber. You want a loyalty program, you're mcdonald's, you, you think about
coffee subscriptions, right? Like today you gather a six person team and you
say come back to me in a month with our strategy and our and a set of
recommendations and be able to defend it. I mean, you can do a pretty good
job of breaking that down and distilling it into the key questions,
the unknowns, some key key answers and some conclusions you can draw from that
detective work. You can do that today probably in two days with today's
technology. So the question therefore is what are the people doing? And so we
need to bring the people together to actually maybe tap the A I as the first
draft and then make it better. That's not what we're doing in companies today.
So I think it could be a little bit bad in the short term. But I'm long on
human kind of contribution to kind of disambiguation things for companies so
that they can achieve their objectives. Yeah, I'm still a big believer in the
stem fields. I don't think that goes away but you hear a lot of people
talking about disruption where, hey, because you can generate code with
models, maybe software engineering goes away. I don't think so. At all,
software engineering creates the models. You have to data engineer them, there's
humans touching them at every step of the process. So it might change how
much code you're writing. That's, that's an assist. That's like what Eric
was talking about, it's going to enable you to do more, it doesn't go away,
you're going to be writing different kinds of code and less of it, but
you're still building systems. And so I think that I'm still a big believer in
those areas being really important, you know, just put our CEO hats for a
second. I mean, operationally, how do you sort of deploy resources? Because I
remember, you know, we used to have these conversations 10 years ago about
like starting a digital unit, you know, to handle the internet, you know, and
now it's just so pervasive, you know, you know, it's just part of everyday
operations and it sort of seems a, I, every company has kind of got like an A
I division and I mean, that's, you know, what you do, Shane, you know, is that,
is that going to stay that way for a while or a few of the CEO S should be
trying to sort of infuse it into all aspects. Um, anyone want to take that
one? Well, it is definitely affecting almost every aspect, especially the
knowledge work is the chief was saying, I mean, I get all these es coming to me
and they're overwhelmed by all these opportunities and I think you need to
do it in a very structured way. Um In my research, we've shown how you can
take any company, you break it down to its occupations that are being done in
the company. And then each of those occupations, you can break them down
into individual tasks. A typical occupation has about 20 to 30 distinct
tasks. And then at that level, you can start seeing whether or not the tool
can augment or help out with things. So when you do that, what you find is a
typical company, there are about 18,000 distinct tasks and you can just run
right through them and, and analyze which ones have tools that will help
out with that task with that. A CEO can have a game plan of where to prioritize
things, what parts are going to have the biggest effect on their overall
value add uh where to prioritize the first step and then the follow on step.
But it makes it a much more manageable and structured approach rather than
just, you know, being overwhelmed by all these different opportunities. What
I see over the next, you know, 3 to 5 years is companies will be taking these
technologies, applying this task based approach and getting some really
enormous productivity benefits we saw in one call center, there's about a 35%
productivity gain in some of the software engineering. They've gotten up
to 100% doubling of performance. There was just a paper about some management
consultants. They got about a 40% of productivity gain across the set of
different tasks and each of them um there was improvement quality. There
was also a effect that the less skilled workers often benefited even more than
the more skilled workers. Yeah, if I ask questions, I want to go to in a
second, but I just want to follow up what you're saying here, Eric, I mean,
I think that makes sense, systemically, approach it that way. But where do you
start? It's like, you know, what's the prompts? You know? OK. A I, I have
18,000 tasks. Please like tell me what to do next or like you guys want some
insight into, you know, what's step one? Well, I like what Eric's saying. I
think you can also pair that. I would co leaders to give A I or task teams to
work with A I to solve some of your most meaningful problems. So pick one
or two of the hard things, you know, new products, business model change
something really huge and ask them to uh write down the problem statement,
then ask A I to make it better or suggest three alternatives. And I think
you'll see that A I fine tune things makes things a lot more precise. We all
know that if a team spends weeks or months or even quarters on a different
understanding of the problem than the rest of the people in the company,
that's an issue because it's going to lead to things getting stuck. And it's
what holds companies like Google despite having great talent back
because it's big and it's become bureaucratic. So where I would start is
use A I all the way upstream define the problems come up with the right
questions that should be explored. It's very doable today. Yeah. And I think so
you're talking about sort of a top down approach about thinking about the
problems. I think that's, that's, that's something you need to be doing.
There's also the bottoms up approach, which models can help with, which is
understanding your data. So if you rewound, you know, five or 10 years ago,
executives of companies probably weren't, you know, understanding their
data in depth because you didn't need to. That was somebody else's department
these days, if your A I is really going to be powering your business, you're
going to want to get under the hood of that. And the nice thing is you don't
need to be super sophisticated technically anymore because you have
language models that can help you interpret and understand your data. So
really being able to kind of wrap your arms around that and understand. Well,
I have this information. What could I do with it and map that to problems
that you find at the top level is important. And often it's not just the
structured data that you've traditionally put in data sets, but now
we're getting to use language and other unstructured data. So it's unlocking a
lot of tacit knowledge that companies have in that call center analysis that
we did. There was just a ton of knowledge in the transcripts and all
the interactions that people were having that previously, there's no way
of studying that. But with the large language models, it was revealing the
best kinds of answers, the best way to solve problems and making that
accessible to the whole workforce. I want to go to just more granular for se
we're chatting backstage, Shane about work day. Apparently there's still a
lot of demands on the recruitment front, you know, parsing resumes. I mean, it's
just such, you know, so much out there both internal and external and you know,
to what degree is A I going to transform recruitment or maybe make
recruiters jobs obsolete. Yeah. So I don't think it makes their jobs
obsolete. I think it's, it's going to assist them a lot. You know, you, you
post jobs, if you're a big company and an attractive employer, you might have
hundreds of applicants for jobs, sometimes thousands, depending on, on
what the type of role is. And usually that's going to mean without A I and
without tools that's going to mean that a subset of the applicants get looked
at with A I, you can immediately look at all the applicants. You can, you can
look at everything that they have. You can focus on the skills, both skills
that are explicitly described in a resume or CV, as well as those that you
can infer using A I. And so you can take a skills first approach of mapping
between jobs and candidates using A I. And that to me is, is an enabler for
everyone. You're not looking at things like, hey, what school did you go to
first? What does your name sound like anything like that? What is any of your
demographics you focusing on skills and you're doing it at scale very quickly.
So I think A I is a big enabler there and the recruiter and the hiring
manager are still going to be making decisions about the people, but they're
going to be able to do that over the course of hundreds of candidates where
they might have only been able to look at a few dozen in the past and this
one's for anyone but like, is that possibly fraught too? It just seems
like it could be a legal minefield. You go ask the machine to go get you the
best candidate based on data and it pattern match in the last, you know,
three people at this job were white guys. So it, you know, you know, and it
goes and zeros in on that, like, how do you put guardrails to, you know,
promote like diversity and not strip over or break the law? You described a
classic challenge with a lot of machine learning is that it will often spit
back some of the same biases that you already have in your hiring decisions.
There's a paper by a couple of my former colleagues at Mit Daniel Lee and
Lindsay Raymond that looked at two different kind of A I powered systems.
One that used more traditional machine learning if you can call it that. And
it did tend to just some of the biases that the company already had and hire
more of the same kind of people that had been, been hired in the past. And
then they also looked at a separate, a different kind of system that was based
on reinforcement learning that did a lot of exploring of the different kinds
of people to hire and, and, and but, and that second system was able to not
only get applicants to perform better but also a more diverse set of people.
So it was a, it was a win by having it explore the space more thoroughly
rather than just kind of amplifying what you are already doing. Yeah. And
so a big factor here is how you design the system. So when we think about
doing candidate matching, we're not looking at profile data about people
and things about them because they are going to be highly correlated to
demographics. We're instead looking at skills from documents that they have.
And that's an important thing. It's a design decision not to use things like
the school you went to or your professional associations because they
have high correlation to demographics and they could learn those patterns in
a model. And so like with a brute force machine learning approach where you
just gave it everything it would over ft to demographic correlations. But if
you with a foil design approach, you can avoid doing that. And actually
again, take a skills first approach so that you're focusing more on what
skills people have not on who they are. I'm not in the hr space, but it strikes
me that a lot of what's going to happen with knowledge work follows the pattern
of software development. So when you hire for software developers, of course,
you give them exercises, right? Yes, you don't look at the resume, you don't
really give a damn about the resume you look at the code and the quality of the
work. So why not every job having exercises that you can interact with,
generated by these models to say what, what's your ability and your capability
around doing the work? That's probably the best signal of it, you know, for
the role and any o any other things. And I think we're on the cusp of things
like that. It's a good future idea. We might put that on the road. Take it.
Yeah, I, I know I have another question or two but we three me stuff. Does
anyone have a question out here?
And Mike is coming and please identify yourself?
Yeah. Hi, I'm Anita Vava. I'm head of Innovation, La Iona Insurance. Um My
question is, how do you, how are you incorporating synthetic data? Because
really like, you know, one way to bypass bias is like, don't touch
proprietary data with like all kinds of loopholes, particularly with the eu
regulation. Like that's kind of kicking in, but we can almost simulate
everything through synthetic data environment. So how are you taking that
into consideration already at the enterprise level right now? Synthetic
data is an important area. It's also a dicey area. You have to make sure that
when you generate the data, it's representative of the real data that
you're going to be, that you're actually going to be acting on. And so
it's kind of an open research area. It's true that you can generate large
data sets at scale. It is something that's really useful, especially if you
want to generate particular distributions or balance out
distributions. So we do it. It's a research area. I think it's going to
end up being one of the most prominent ones in the LM space. Simply because
the way that LMS are being trained today on these large corpuses that are
largely human generated on the internet has issues with it that you need to be
able to get around or balance out. A lot of that is done today with, with
RHF, which is having humans kind of supervise the models in the background,
but that doesn't scale very well over time. And so I think that synthetic
data is going to be a big part of like the LM world going forward and it's an
area that we're invested in. Yeah.
Um and just to wrap up, um you know, Fortune Col and Murray flagged an
interesting news item today, you know, one of our other partners Microsoft
announced they um they had signed an agreement with the A FL, you know,
America's biggest labor union, you know, to collaborate on these things going
forward. It is sort of, you know, encouraging, I wonder if it's a bit
Kumbaya because this thing just feels like a little disruptive. But, you know,
do you think that's a model worth pursuing, like leaning out and trying
to bring your, you know, kind of workforce in or are you just asking for
trouble and be honest here? Oh, I, I think 100% because in the end, uh the
good work in the organization happens at the team level, right? It's not the
C suite, it's not the CEO, right? You're very distant from the customer
and the specific issues. So, you know, what is sustainable work for teams? I
think we need to hear from a first person standpoint, kind of like in a
product mindset of like what actually makes work sustainable for all the
stakeholders, not only the organization, which is gonna get a boon here in
productivity but also for sustainable employment. It comes back to what you
said at the beginning and what we heard in the previous panel was Natalie about
using the technology to augment people or rather than replace them. And if you
bring labor in and say, OK, how can we help you do your job better? We're not
here necessarily to replace or maybe there's parts of your job that we can
make easier or less efficient, but keep the human in the loop ultimately as
impressive as these systems are and they are very impressive. They're still
not replacements for humans. In most cases, they Hallen, they make mistakes.
You want to have a human, not just in the loop but in charge and going into
that up front and saying, look, we want to work with professionals, with
laborers in all different categories and use the technology to allow us to
do new things better than we could before. That makes it a win, win. You
know, in the call center example that I mentioned earlier, the productivity
went way up, worker wages went up, customer satisfaction went up, customer
worker turn went down. So it was not one group benefiting at the expense of
the others. And it really doesn't have to be ok. Well, it seems a very
optimistic take and I hope you're right now. Just end with another thing, your
friend Viard Kla said, which was this is, you know, if all these jobs are
disrupted, it won't really matter because it's going to create so much
efficiency and productivity will only work if we want to. So that sounds nice
too. But, you know, so there's the optimistic look at the future. I'm sure
people have other takes. But thank you everyone, please. Thank our panelists.