00:01 Josh Bartels: We're going to talk about bridging clouds and building a multi-model LLM chatbot with Azure OpenAI and Amazon Bedrock. We're going to explore a couple different things.

We actually have a live demo as part of this and also the code that was created for that demo will be available and you'll see that at the end of the presentation today. Just as we get started, this is my least favorite slide to talk about. I don't like talking about myself, but I do think it's important for you to, like whenever I go to a conference or I'm listening to a talk, I like to understand the speaker personally.

So obviously at work I'm the chief technology officer at UDig, but at home I'm a girl dad. I am outnumbered three to one at home. I probably have more experience brushing hair or translating teenage slang than I need to and I know a lot of different princess movies that I've had to experience.

There have been after hours. I'm a hockey lover. In my late 30s, I decided to get off the couch from watching hockey and actually start to play ice hockey.

So playing in beer league here in the Richmond area, but still perfected my slap shot at 40 plus years old. Before we get into the meat of what we're going to be talking about today, I just wanted to spend some time talking about some things that I'm looking forward to in 2025. If 2024 was the year of kind of just general AI hype and what was happening with it, there's a lot more that's happening in 25.

01:53 Josh Bartels: Now that we're getting to real use cases and real implementations, and so I wanted to take a few minutes just to talk about some things I'm excited about, but also some things that I'm concerned or keeping an eye out for. So really looking at kind of AI's enablement of efficiency and code creation. Just being a CTO of a firm that helps organizations create custom software or data solutions, empowering our teams with AI, we've definitely seen a lot of efficiency in the capability to create code.

I think that's at varying levels depending upon the engineering skill set and years of experience, but senior developers or higher definite efficiency gains in code creation, lower level juniors and mids, some efficiency, but a lot more opportunity to potentially learn more about code using it as an explanation buddy rather than a helpful code assistant or writer. Democratization of data access by converting natural language to SQL, right?

So allowing people to just have a conversation but gain access to data I think is something I'm really excited about. We've had this previously, but I think it's just getting better and better by allowing models to access full schemas and understand the connections between tables, we can get more relevant answers to what we're looking for.

I really hope in 25 that we can continue to force innovation from large SaaS providers. I think that right now there's a large grab for continued licensing fees and increases, but they can't innovate as fast as some of the smaller firms. No dig against Salesforce, but I think agent force is okay, but there are more innovative things coming around the pike constantly.

03:58 Josh Bartels: So how do you continue to innovate as a large SaaS provider, which is going to give us value for our organizations or for in particular our clients' organizations. And then model performance gains are occurring every three to six months, right? So we go from 3.5 to 3.5 version two with Claude to 3.7 recently. Open AI just came out with GPT 4.5, right? But between 4 and 4.5 they had the O1 models.

So there are things that are just coming out quicker and faster and we're gaining more and more capabilities each time. Because we have the enablement of more efficient code creation and we have the large SaaS providers not innovating, we are seeing a lot more potential to in-source solutions internally, which I think is putting pressure on those large SaaS providers. On the flip side though, as we look at what are some of the potential negatives or potential things that I'm going to be keeping an eye out for 2025, as we write more and more code with AI, how do we maintain that code, right? Is it maintainable? How do we put some human in the loop on that to make sure that we don't just create sprawling large code bases that are not maintainable? As a CTO, part of my role is trying to keep pace with the rapid changes.

There's just so many changes that happen every day or every week that it is hard to keep pace with everything that's going on. Eventually I think we'll start to level out, but I think if you look between the providers of tools or the providers of models, every two and a half weeks somebody's coming out with something new and then two and a half weeks later somebody is trying to come up with a similar feature set. So we're getting parity almost in half a month.

06:04 Josh Bartels: It does increase integration complexity. The more and more things that we want to connect to LLMs or other models, we have more integration challenges. So how do we continue to tackle those? And then as we build things around these models, there's the dependency of evolving changes.

It's the wrapper problem. So creating really good wrappers is a good way to create unique experiences and it's a good way to add value to what some of the folks are providing. Perplexity is, you know, the CEO self-proclaims that they're a wrapper, but to be a wrapper that's able to keep up with those increasing changes, I'm sure the engineers there are working constantly to make sure that they can keep up with all the changes. And costs. A lot of organizations need to understand the costs. There seems to be increasing rapidly.

I think that's a pitfall of getting better and better performance, which means there's more and more model training. More and more model training isn't cheap and those costs have to be passed down to the consumer. So as we get more and more performance for it, is the value worth what the model providers are charging? So that was just some information I want to give an eye out for, looking at 2025 and moving forward.

07:39 Josh Bartels: Now we'll hop into what we're here to talk about today. So why would you want to bridge clouds? Why would you want to potentially build something with multiple models? So as an organization, I think there's a couple of different opportunities that are key to hit on, right? Different models perform differently and provide better results based upon what you're trying to achieve. So we know, or at least what we've seen, is models like the SONET-based models from Cloud seem to have higher proficiency in producing code, where other models like GPT-4 and 3.5 have a better potential to create generative content, along with, you know, they have the deep research that exists as part of those models that doesn't exist as part of Anthropic Cloud models.

So figuring out what you're trying to achieve might mean I need to use a different model for what I'm trying to accomplish. So by having access to multiple models in one interface, you can ask different questions and it can be fit for purpose. Reducing sprawl, right? So how do we simplify the use of these tools, right, where I don't have to context switch between Copilot over to OpenAI's website, over to Claude's website or Claude's desktop tool, right? Can I have one unified interface where I can get to that, right? I have engineers on my team who are coming to me, this is both cost and sprawl, right, coming to me and saying, hey, I have a subscription for Grok, I have a subscription for Claude, I have a subscription for OpenAI, right? Okay, well, we're up to $110 a month on just subscriptions in model providers, right? So by potentially hosting those yourselves, right, you get a better chance for control and you get a better chance for helping reduce sprawl, which leads into private deployments, right? So by putting them in AWS and Azure, you are deploying them on your technically, quote unquote, your infrastructure in the cloud, where the models are being brought along.

09:58 Josh Bartels: I caution everyone to read the end user license agreements just to make sure what it means by hosting those, but you do get some additional privacy and security by bringing those deployments private. So what are the differences between AWS and Azure? Just a quick overview here. Really, what it is, is there's a little bit of difference in what models you have access to and then the deployment time and how they calculate cost.

So from a model deployment time, Azure OpenAI can take anywhere from three to five business days, depending upon the model that you're selecting. AWS tries to give you, with Bedrock, almost immediate access to certain models. They both have a pay-as-you-go approach.

Azure looks at doing token-based pricing, while Bedrock employs more of an input-output token approach for different rates for each different model. But the reason that we used AWS and Azure for this is we wanted to give access to and show some OpenAI models, which GPT- 4.0 and 3.5 Turbo, and with AWS, we're showing some models with Titan, Claude, I believe Mistral, and Llama, but they also have the AL21 Labs, Jurassic, Cohere Command, plus a bunch of others. We'll see that here as I do a quick overview in the portals.

11:32 Josh Bartels: The biggest difference here is maybe just more naming conventions for the features. They both have foundational models that exist in the portals that you can deploy. They both have ways to put guardrails and do model evaluation.

They both have a chat playground for the models that you can exist with. I think if you want to do RAG in either one, in AWS, they call that knowledge-based support. In Azure AI, you're going to do AI search along with machine learning skills for retrieval, augmented generation.

So they both support RAG as well. They just have different naming conventions. So if you're diving deep on the AWS and Azure side of the house, just some naming conventions to look at.

The tool that we built is really just for chat. We did not implement RAG. If somebody wants to fork it, they want to do a pull request and implement it, feel free to do so, but we're just trying to give an example of how you can interact across the clouds.

12:41 Josh Bartels: So what did I kind of take time to build? What are the components of the solution? So we put together a chat interface, tried to make it user-friendly, that allows for interacting with the chatbot, real-time conversations. We have cloud-hosted models, right? That's the whole point of this example is basically leveraging both Azure and Amazon Bedrock to access different LLMs for different use cases. We do have an API backend that was written in Python, using FastAPI to manage those calls to both AWS and Azure, which handles the orchestration based upon which model you select and where it needs to go and how to retrieve it.

And then we also implemented conversational history. So if you want to go back and look at the different conversations that you've had, those are stored in a Redis store, and you can kind of go back and look at the conversations. And those are the four high-level blurbs.

If we go a little bit deeper, we have capability for streaming chat, prompt engineering. So we allow for changing system prompts for a couple of different scenarios that we'll go through. It is a monorepository, so I didn't want to have to give everybody two different repositories to go to.

We have one repository. We did use ShadCNUI components for trying to make it as pretty as we could without having to do a lot of styling and a lot of extra work. And then it is containerized, so there is Docker files in the code to make it really easy to stand up as long as you have your environmental variables set.

And you can kind of deploy that easily for testing locally. One thing you'll notice is there's no user management or security. So this is really just for internal testing and playing and learning.

14:38 Josh Bartels: I would not encourage anyone to deploy it into a production environment, but more just use it for learning. Or if you want to deploy it in production, make sure that you add security and authorization to it. There are a few things that I just want to point out that maybe completely didn't make it into the tool.

So there is an issue with Llama model streaming. When you enable streaming with the Llama model, the output comes out, but it's not always displayed in the correct way. That's just because of the streaming output return with Llama has some special characters or special character combinations that need to be handled better.

Markdown and code highlighting do work, but they can use some cleanup and there could be a cleaner approach. Also, there's some issues when you stream with it. If you run into an issue with a stream and it's trying to generate code or highlight it and you refresh the page, you probably get the actual return.

But in that middle of that interactive session, sometimes things can get hung up. And just more models, more complexity. So each family of models inside Claude, ChatGPT, Llama, Mistral, they all require different input output format handlings.

15:54 Josh Bartels: So this is more of just a known complexity. There's no issues with it. But if you wanted to integrate more models that are outside the families that are already integrated in the code base, just be aware that you're probably going to have to extend the code to be able to handle those different input and output models.

And then before I was giving this conversation, chat history was a late feature that maybe or maybe wasn't going to make it. It did make it in before the presentation. So it is in the code repository and we will be looking at it in the live demo.

16:28 Josh Bartels: So lastly, before we kind of get into the live demo, one thing I wanted to do was give more of a high level overview of the three, the two components that make up the code along with some environmental settings and maybe we'll do a quick jump over to the actual code just to give you guys a little bit of a deeper dive. But in the backend, we're doing model management. We're handling all the various AI models and supports for custom system prompts with them.

We have a services implementation, which allows us to reach out to each cloud provider and handle their models. And we have routing, which allows for routes for chat, streaming, model listing, and session management. In the front end, we have a component-based architecture, right, using the ShadCN components.

We wanted to do that because we wanted it to be modular and reusable. So if you wanted to extend on the functionality, some of those components already exist and are reusable. Some general styling with Tailwind CSS along with some theming that's built in.

When you go to create this yourself or when you go to stand it up yourself is a better way of saying it, you will need to manually add environmental variables. So when you deploy models on Azure OpenAI, you need to set an environmental variable for the models that you have. You don't need to do the same thing for AWS because the code cycles through the AWS models, test them quickly to understand which models you've actually stood up.

But for Azure OpenAI, we don't have that capability. There are a couple of secrets that you will need, which is for Azure, there's an endpoint and a key. And for AWS, you'll need a secret ID and a key that has access for Bedrock.

And then there is a Redis setup that's all containerized within Docker that establishes variables for host, port, and password. And that's really the chat history implementation.

18:37 Josh Bartels: And now we'll get into the demo. I'll kind of take you through some different aspects before we get into the demo, just so that you understand where we can get things in Azure, where we can get things in Amazon, do a quick look at the repo out on GitHub, and then we'll get into the demo. So I'm in Azure. I've logged into the instance that we're using to host the models for the demo.

When you are in Azure, you do have to go through a couple steps to get to what is called the foundry. You get there through your portal, establishing an Azure OpenAI endpoint, and then you can open up the foundry. When you get to the foundry, there's this deployments item over here on the left.

When you go to, you can deploy new base models. You can see access to all these different models that exist. One thing to point out, any of these models that have a lock require a special registration.

Usually you go and fill out a form, and then you wait a couple days, potentially longer, and you can then get access to that model. If you have a good Microsoft rep, you might be able to get access to those models quicker. One other thing to note is some of these models are now hosted not in East US, they're hosted in East US2, and if you need to be able to do that, they end up at a different endpoint.

20:09 Josh Bartels: We have an older instance that we set up for Azure OpenAI, which is in East US. Right now we're just going to be looking at 4.0 and 3.5, but if we wanted to use some of the 0.1s or the 0.3s or the 4.5, those might actually be hosted in East US2, which would require us to do some refactoring of the code. Just to show you where those things are for deployment, and then you would denote the model names from here, because those will be used in your environmental variable.

Going over to Amazon, once you get into your AWS account, you make your way over to Bedrock. When you're in the Bedrock service, there is a model catalog that shows you all of the different models that are available to you. They have this pretty card format, but there's also a list view.

With the list view, you can see what you've had access granted to, but you can also see there's seven pages of models. So tons of models to look at here from different providers, all on the AWS environment. A lot of them are serverless, so they'll spin up in the background and then spin back down, helping save costs.

21:22 Josh Bartels: But just wanted to give everyone an overview of where you would go and where you could request these things and get access to them. Last thing before the demo, this is the GitHub repository that exists, that's created, that's available for everyone. We have detailed readmes to understand how to get the project set up with a quick start, one with just using Docker, and then another quick start if you want to do manual setup, along with instructions on what do you have to do for Azure OpenAI, what do you have to do for Bedrock, and it walks through all of those things.

We also have separate readmes just for the front end and back end that you can access, but you can see Monorepo really with the back end organized here and the front end organized here, and everything works with everything that's in these folders. Moving to the demo, I already have this running on my machine. This is what the interface looks like.

Earlier today, I went out and asked just real quick, hey, this was for the RVA Tech Data and AI Summit participants, say hello. There is a default system prompt that tells the system to respond with markdown, and when it does, use Python or other fun greetings, along with being able to draw mermaid diagrams, because we wanted to show that capability. Response from the LLM was, hey, here's a friendly greeting that they've written in Python for us.

It shows the output of what that Python code would be, and then it visualized how that would work in a mermaid flowchart. But just to show how this works, we can start a new chat. Down here, we can say, hello, tell me about yourself.

And you can see we have a streaming capability where it's outputting the data to us. It's streamed. It's from GPT 4.0. We can switch models mid-flight.

We can go over to Claude 3.7 and say, all right, so switching models, going through different things, right, and answering questions. We can do new chats with different models. So we can do the haiku and say, tell me a poem.

24:15 Josh Bartels: And this is one of the issues. It's always fun to do a live demo. This is one of the issues that's related to the rendering of the stuff live stream, right? It wanted to put a mermaid in there, basically showing what the diagram was with the streaming, right? A couple other features that we built is you can search the conversations.

So you can say RVA data tech, and it'll filter the conversations. You can go back to the history of the conversations that you had. But two features that I think are relevant for people who are trying to learn about different LLM models are comparison and chaining, right? So in comparison, what I did here is I built some quick comparisons.

So you can select these to do a quick comparison between different things. So we have 4.0 and 3.7 with Sonnet. We have different kind of things that we can ask it, which are pre-built.

25:13 Josh Bartels: So we can say analyze the pros and cons of renewable energy. And we're going to send those responses off. And the reason this one is interesting to look at is the response on the left is actually going out to Azure OpenAI and retrieving a response.

The response on the right is actually going out to AWS and retrieving a response. This takes a little bit of a minute. So we're going to wait here for these things to respond.

But in the background, we kind of talked about what's happening, but really the GPT 4.0 response is going out. And so we see here that we have the activities returned. You can copy the entire response if you want.

You can actually also blow it up to read it if you want to. Or you can scroll side by side between the two responses to see how the answers came out. Remember, I did tell you that there is just a default system prompt built in for using a markdown and highlighting Mermaid.

26:19 Josh Bartels: So Claude 3.7 went ahead and said, hey, I'm going to draw a diagram as part of my output here, where 4.0 decided not to do that. I did talk about earlier that we do also have a system prompt that you can change. So over here under settings, you can use a custom system prompt.

And so for that, what we're going to do is we're going to make sure that we can adjust what the output looks like. So basically a helpful assistant, we're asking it to provide summary details, example key takeaways as part of its response. We'll save that and we'll just say, tell me about machine learning.

And you can see it's using that format of summary details, example and key takeaways. Last, we have the capability to do model chaining. So what we're going to do is we're going to put something into the first model with its own system prompt that has a bunch of information basically saying, you are the model that's responsible for generating the structure of the answer of what we're looking for.

And then the second model really takes the output of the first model and factors it into, hey, you got the output from the other model. You need to kind of beef this up and make sure it's a well-formatted response to the user's query. So what we'll do is maybe we'll start with Claude 3.5 over here.

28:04 Josh Bartels: And over here we might do GPT-4, right? For our prompt, what we'll do is we'll go ahead and ask it to design a sustainable urban garden for a small apartment. Once again, we're going to have to wait a little bit, but it's going out there using both of those system prompts to, one, come up with the intermediate result of the key takeaways. And then basically from these key points that it provided to us, now GPT-4 is taking that information provided from clod and generating an output.

And so you can kind of see how you can chain models together as part of this conversation, right? Last but not least, we'll go back to PowerPoint real quick. And that's the QR code to go ahead and grab from the GitHub repo. Obviously, no questions, because this is just a rerecording of the talk.

But appreciate everybody's time in taking a look at this. If you do have questions, feel free to reach out via LinkedIn. Appreciate your time, and hope everyone has a good day.