I'm Emma McGrattan. I'm from Actian. I'm Chief Technology Officer at Actian. I've been in the data space for 30 years. You can probably tell from the
accent that I came from Ireland. And I actually had my very first job in America back in 1987 on a student visa working at the lab for artificial
intelligence at MIT. So that's how long AI has been around, right? But it got really, really good,
really, really fast and really accessible. So it's obviously the hot topic and role here to learn more about developing AI applications that we can
deploy across the enterprise. What I'm going to talk about is the context layer, right? Because as we deploy AI into the enterprise, by default, LLMs
know nothing about your business, right? So you need to give them the context of your business so they
can answer questions that are grounded in your business reality, right? Not some default from the internet. So what we're going to talk about is the
most important challenge that we have to address today, which is how do we engineer that data layer so that AI can deliver answers and business value
reliably and at scale. So since I guess the end of '22 when ChatGPT really burst onto our radars,
right? We've all been playing with it, right? We've been playing with AI applications. We've built some amazing pilots. We've rolled them out into
production. Many enterprises now are seeing success both with the genetic AI and with generative AI. But for some of us, there's a real problem in
building out an architecture to support AI within the enterprise. And there's three main pressures that
are being put on us that really spoil these beautiful architectural diagrams that we had in our heads as to what the AI and data architecture was
going to look like. The first is regulatory pressure, right? So, if we think about the current geopolitical climate, there's a lot of countries and
regions around the world, and even companies that are saying, "We need to unhook from our dependency in
American tech." Right? The US Patriot Act gives the US government the right to access any data that's on US soil or in a US based platform that they
think that might benefit um the security of our nation, right? And that is very scary to other nations, right? So, they're looking at, "How do we
unhook from American tech?" And sometimes that means moving to sovereign clouds. Amazon, for instance,
has a sovereign cloud in Europe, where European governments can set up their data and know that um it's never going to touch the confines of Europe.
So, not just the data residency, where does the data live, but also the people that touch that, right? Any of the operations that happen on that data
will never you leave the confines of that sovereign uh cloud space. The other team of people that
love to mess up clean data architectures are the regulators, right? So, if we look at industries like financial services and health care, they have
mandates that certain data can't leave the confines of those um data centers, right? So, in financial services, a lot of the customer data, they will
mandate through regulation that has massive fines attached to it, that data can't leave the data
center. So, it means that we have to look at on-premises deployment of some of our AI stack, right? Also, when it comes to autonomous vehicles, fraud
detection, the um latency requirements that we have there mean that we can't go all the way to the cloud and back again, right? That could take 50
milliseconds, that could take up to 200 milliseconds, and we need to be able to react in single-digit
milliseconds and in sometimes sub millisecond time frames, right? Cloud precludes that from happening. So, when you need to make truly instantaneous
decisions in real time, right? Cloud is not an option here. So, we've got to think about how do we deploy AI at the edge, right? How do we deploy AI
so that it's sitting in the device where the decision has been made. So, we're talking about
millisecond latency in making those decisions. And then the third is data gravity. So, every enterprise has a very messy data architecture, right? I
heard reference to mainframes earlier, right? And when you go into financial services, what you'll find is that they've got everything from the oldest
of mainframes to the most modern devices and they have data across all of them, right? According to
Gartner, the average enterprise has 400 different sources of data. 400. Now, imagine where that's all running. Imagine trying to wrap your arms around
it and bring the AI to that data. It's quite a challenge, right? So, data gravity also plays a role. Typically, you're going to have data that lives
on premises. You're going to have data that's in clouds. Maybe you're using, you know, Azure, maybe
using GWS or AWS, right? You're probably using some SaaS applications like Salesforce, right? You probably have some APIs that have data behind them
and you probably have data pipelines, right? The AI needs that data and the AI needs that data quickly. So, your AI all of a sudden has gravity,
right? It that data gravity is going to suck the AI along with it. So, you're going to have to have a very
messy AI architecture to support the data and support decision-making where the data is. Now, many of you, I'm sure, are very familiar with rag,
right? Rag stands for retrieval augmented generation, right? And basically, this comes back to the point I made earlier where we say that LLMs don't
have uh stateless, right? They also don't have any knowledge of your business. And what you need to do is
to provide them with that knowledge. To do that, you go to a vector database, right? The vector database is able to provide semantic meaning so that
you can deliver that to the LLM so that the answers that it provides are grounded in your customers or sorry, in your company's data. I'll give you a
quick example, right? So, let's say you've got an insurance policy. Let's say it's your car insurance
and it went up by 20% this year. And you haven't had an accident and you haven't had a ticket, right? And you want to ask your insurance, "Why did
this happen, right?" What the insurance company's going to do is take your question, right? It's then going to look up all of the data that influenced
the decision on your policy increase, right? So, it may be that you moved to a neighborhood where your
car is more likely to be stolen. Uh it may be that um your commute times have changed, right? So, you're going to be in the car more. It may be that
the roads in your neighborhood um have had a lot of potholes cuz you had a brutal winter, right? So, what that vector search is where we're going to
get all of that context, right? So, we're going to take your question, "Why did my car insurance go
up?" We're going to take all of that answers we have that influenced why your car insurance policy went up and we're going to provide you with an
answer um from the LLM interface that's going to make sense to you, right? So, we're going to cite the documents that caused your insurance policy to
go up and we're going to give you an answer that's grounded in the reality of why that decision was
made. So, that's rag, right? Retrieval augmented generation. And the challenge is that where all of that data that we look up to augment the uh the
LLM, right? lives, right? Where that resides and where we retrieve it from is going to impact the performance that we're getting when we ask these
questions um of our AI, right? So, where the search run and where the data lives is going to impact the
performance. And typically you want to get your answers back fairly quickly or you go off and do something else cuz we're easily distracted. So
getting this context layer right, deciding on what's an appropriate architecture to answer questions quickly is incredibly important. So we've got
three choices when it comes to the topology of our AI context layer, right? Cloud, my god, it's amazing,
right? It's has elastic scale, it's got global reach, it's it can flex as we need it to, right? If we're loading up a lot more information, if we're
doing a lot more work, the cloud can scale to meet those demands, right? It doesn't have any If we don't have any restraints around where the data
needs to live, cloud is a great answer, right? Um the challenges with cloud is first off it has latency,
right? So 20 to 200 milliseconds round trip time to get any answer. Egress co- egress costs can go up, right? So that is as we're taking data out of
the cloud, there's a charge associated with that. So depending upon how much data is being egressed from the cloud, those costs can be quite
significant. And then obviously if it's cloud, we've got a dependency upon connectivity, right? But cloud is a
really great default. Next up we've got on premises, right? So if you have data sovereignty requirements, if the data must live within a data center
either because, you know, of you're a sovereign nation and you required it or if because you're in financial services, health care, defense, and some
of these industries where they've got regulations that require data stays on premises, then you've
got to look at an on premises deployment of this semantic layer. Uh we here have um investments that need to be made in just the infrastructure to
support this. We've got delays in ordering all of that infrastructure. And then we've all of a sudden we've got to be able to manage, maintain, and
patch these environments. So that is um not nearly as attractive as cloud at where all of the
infrastructure and operations are handled for you. And then the third use case that we have for deploying our semantic layer is edge, right? So if you
need to make decisions in milliseconds, right? Edge is where you need to be thinking, right? Edge devices also can have no connectivity. So if you're
an environment, let's say you're down in Florida, our software is used for badge scannings on fast
pass, right? And every afternoon when you're in Orlando, there's a thunderstorm that comes through and every afternoon all of these devices lose
connectivity and people can cheat on the fast pass, right? But when the internet connectivity comes back up, it all gets reconciled. And but being
able to work from your disconnected is an important use case when it comes to edge, right? And also this is
a a great use case if the information cannot leave the device, right? So maybe it's personal health information and you don't want that, you know,
ever going to cloud. You want that to stay within the device, then edge gives you the ability to do that. The challenge with edge is that the compute
and the memory capacity that's available to you is quite limited. And also the freshness of the indexes
that you're working at can be quite spotty because you've got to refresh those. And there's also if you've got a large fleet of edge devices, managing
all of that together can be quite difficult. So getting back to what I said, you know, cloud is always a really good default choice. On prem, if you
have on prem, we think of it as legacy, right? But really we don't have an option when it comes to
environments where we've got to think about sovereignty, where we've got to think about security, and when we've got to think about stability at
scale, right? And so financial services when they're doing things like trade surveillance, like looking for the facts that um well, I won't even talk
about our current geopolitical situation, but you can imagine that there's some very dodgy trades that
are going on and that we need to be able to detect in real time, right? Also fraud, right? You want to detect that as soon as possible. And and as I
said earlier, in financial services, there are heavy penalties associated with breaching any of the regulations around data, so they've got to be able
to prove that the data stayed within the confines of the data center. Healthcare and life sciences,
similarly with uh regulations like HIPAA and PHI, right? That data needs to stay within a very hardened um stack within the um the confines of the
health network, right? So it might be within a a data center, it might be within a doctor's office, uh or hospital. And then in defense, these are
kind of very cool deployments where we need to have air-gapped systems, right? So that nobody can
infiltrate it. So you um we actually have software that's built into the warheads of nuclear submarines, right? Once that's deployed, no one can touch
it, right? So you can't have things like a license server that's checking to see if they're still licensed, right? So uh really important that we
understand as we deploy this, um you know, what makes the most sense. So, edge definitely when
milliseconds count, edge is the best option. What we're going to find is that there's no one great solution that's going to cover every use case that
you have, and hybrid is going to have to be what you design for. So even if today you're all in on cloud, as you design, design for a future where
you're going to have something on an edge and you're going to have something on prem. Also think about
intelligent query routing, right? So if you have classified data, right? If it's regulated data, then you're going to route that to your on prem tier.
If the latency needs to be less than 5 milliseconds, you're going to route that to your edge tier. If the freshness of the data is less than a minute,
definitely go into the cloud, right? And if you need to query all of it, right? If if your
question is going to span multiple domains, then you're going to have to fan it out, and then you merge the results in your application. Uh how do you
choose the right model? I would take a photograph of this slide cuz I have 1 minute left to speak and I'm not going to get through all of it. But but
uh refer to this. Look all the cameras going up. Uh yeah, refer to this, right? Make the decisions
for um each of the workloads that you're dealing with, right? Map all of them and then you'll very quickly come to a conclusion that you've got no
choice but to go hybrid. So plan for it. And where are they heading? Where are vector databases heading? At multimodal retrieval, right? So today it's
mostly text and text out. We're very going to very quickly going to get to a point and I think it's
within 12 to 18 months where when we're doing a retrieval, we will expect to retrieve uh audio and image data as well as text and time series data. So
multimodal retrieval is definitely in our future, so plan for it. Um managing indexes is difficult, right? Takes a little bit of brainpower. Uh I
think that's all going to be driven by AI again within the 12 to 18-month time horizon. At
governance-aware retrieval, right? So today, what happens is you retrieve the information and then you bolt govern on governance on after the fact,
right? So after the fact you decide who should have seen what and how you pass this on. That all needs to be built into the index. And then, you know,
having to understand SQL and other languages um like graph is uh it's going to be a thing of the
past. Uh what do you say back to your team? Take a photograph of this one as well, right? Topology, very important you get the architecture right.
Distributed AI requires distributed retrieval. Silent failures are incredibly dangerous, right? And the context layer is load-bearing, so treat it
like your business depends upon it. Uh I'm from Actian. We're launching a new vector AI database today.
It's uh designed for on-premises and edge deployment. We're just down here sandwiched between AMD and Oracle. I'm also very proud to say my second
book was published this morning and uh you can download that for free. It's on vector databases. Yeah, thank you. Uh it's available from O'Reilly. And
thank you for your time. Cheers.