[0:01] Dustin Cleckler: Welcome to our LinkedIn live event. AI is everywhere and it's really getting a lot of attention, but it can be hard to see the real-world value beyond the hype sometimes. So today we're going to showcase a couple of use cases and demonstrate how AI enhances QA workflows. Um, so I'm going to cover a few slides, show you guys kind of the premise of what we want to present here, and then I'm actually turn it over to Josh Bartels for a demo of AI test automation.

Before we get started, I do want to note if you guys have any questions throughout the presentation, please add them to the chat. We would love to hear those and answer them. Add them a comment and we'll answer those when we can.

Ok, so my name is Dustin Cleckler. I'm a principal consultant here at UDig, and I primarily focus in the automation space. I've got around a decade of experience in the consulting world, and I'll be kind of doing the first half of the presentation today. Josh Bartels is joining me today. Josh Bartels, do you mind introducing yourself real quick?

[1:11] Josh Bartels: Yeah, so I look a lot different than the picture. I'm Josh Bartels. I'm the chief technology officer here at UDig. I've been in the Technology and consulting space for over 15 years, but what I really focus on here at UDig is trying to look at innovative ideas and solutions to partner with our clients and our practices to deliver things like what we're going to be showing today as a use case for AI as it relates to QA testing.

[1:40] Dustin Cleckler: Yeah, Josh Bartels and I both look really different. I made a joke last time that Josh Bartels looks better than his picture and I look worse, so it's like a net even, so we'll be sure to get those updated.

Ok, so let's jump in. One thing that we really focus on here at UDig is how AI can be used in practical ways to drive business value. There's a lot of hype around AI right now as I'm sure most of you guys are well aware, but we also recognize how powerful of a tool it is and we want to make sure we're leveraging this new tool in the most creative and useful ways to really drive impact for our clients. So one of the spaces we think AI can have a significant impact is the test automation and QA space.

On the screen, you're seeing kind of a snapshot of the history of test automation and quality assurance and software development. The discipline is rooted in the early days of computing, and it's really evolved significantly alongside advances in programming and computer science and really business needs. And as software has become increasingly integral to business operations and daily life, the need for reliable and efficient testing methods has grown. So, I think everyone in this space knows how important test automation and quality assurance is in the software implementation life cycle. And looking to the right of this timeline, you can see where we think test automation and QA is headed - and very quickly.

We have a vision of test automation and QA being accelerated or even completely automated by artificial intelligence. So at UDig, we've been helping to shape that. That's something that's exciting for us and we recognize that this is one of the many areas where AI can have a pretty significant impact.

Ok, so what you're going to see today is AI powered test script creation. We've built an AI powered test automation tool that converts natural language commands into executable test scripts. So imagine you have a new web application that you've been developing and it needs to be tested. What is typically done today is a team of developers all write various test cases and we either manually execute those test cases or code test scripts for automated testing.

So what we envision is, and what we're working towards with our tool, is a tool where anyone familiar with the website or the web application could enter test cases in English. So just like you would write a prompt for maybe like a ChatGPT or Claude, our system would take that prompt, and it would navigate to that specific web application and it would actually create a test script completely automated. So it would try to interpret your website or web application and then output that test script.

So the process now goes from someone writing code to create a test script to giving our tool instructions and the tool generates that test script for you. We're really excited about this and the possibilities that this tool has, and we think it can drive real business value in this case.

Ok, so anyone at all who is familiar with testing, I think has probably seen this test pyramid here. And I just put this up to really kind of help explain our tool and what we did a little better. So real quick for a little background on the test pyramid - the higher up you go, the more time consuming and maybe difficult to create these tests are, and then lower down on the pyramid represents more volume.

[5:30] Dustin Cleckler: And the amount of tests that are done. So at the bottom, developers as they're developing these applications are running tons of unit tests throughout development, and then at the very top, the UI test where you're actually interacting with the user interface, those get a little more difficult to create and time-consuming to create. So our tool is actually targeting kind of that top pyramid there. We do hope to expand this tool and have broader coverage of really the entire test pyramid, but right now we're focused on those UI tests.

Ok, so how it works, our tool, essentially what happens is you enter a prompt to a tool like I mentioned earlier, this could be just like a ChatGPT or Claude, and that prompt would contain information about the website and what you want to do. So you could say, for example, go to the UDig website and make sure the contact me form works correctly. So you would type that in and then the tool would actually navigate to the URL that UDig site that you give it, and it would display the HTML overlay on the page.

So a lot of these web elements that we typically don't see when we navigate to a web page would be shown and our tool would take a screenshot of that and it would then feed that via API to a large language model. Right now we are using the OpenAI LLM. And that would basically interpret like, hey, here's the task that I've been asked to do and here's the screenshot with the HTML overlay. I'm going to write a playwright script and try to do this next step in the task. So it would write one line of the playwright script, it would run that.

So now the website is at the next step in the process, and it would do that all again. So essentially what we have is a loop where we're sending information about your website to a large language model, and that large language model is interpreting and trying to understand your website and create a test script based off of that.

[7:34] Dustin Cleckler: The one caveat, one thing to note, the large language model, it does, the version must support those multimodal capabilities, allowing it to interpret and understand both text and images. But beyond that, you could plug and play different models as well. So we're looking to experimenting with Claude and a few other versions in the future.

All right, so what does this mean? Looking opportunity wise, like ideal opportunities, who would be a good fit for this and where would this drive value? We think really like maybe companies trying to save expenses by moving away from manual testing and adopting more automated testing solutions or large enterprises that have a wide ranging and thorough test coverage across multiple apps and environments. Agile teams who need to move really fast.

And then also, small development teams is another big one that we think would be really valuable. So if you have a team with limited resources or specialized QA staff who want to streamline and automate their testing process, we think there's a lot of good fits for this and we do think a lot of people will be using technology similar to this in the future.

Ok, so the value of this, one thing we want to do and we've kind of been talking about is how do we quantify the value of this right now and that's something we're working towards, but there's some kind of immediate obvious benefits that we're seeing, the first being the simplified automation process. So you can have less technical resources now do your testing, they just need to know the website and what needs to be tested. So if you have a list of test scripts and someone who doesn't necessarily have any playwright or test experience could simply type in, here's what I want you to test and the test script will be generated.

Faster test script creations, just going to park here for a minute. This one, when we run the automations, you guys will see here in a minute, it'll run about the same time as a human can create one test script. So that's probably a 1 to 1, but the benefit of this tool is you can run multiple test scripts in parallel. So you could feed a CSV in with 20 test scripts, hit run, and those would all be executing in parallel, so that's exponentially faster than a human could currently do.

Another one is that flexibility and adaptability. So maybe you have turnover or there's things you need to roll out fast test quickly. You go into your prompt and you get your test scripts generated. So it's really not only a cost savings, but it's really simplifying the test process and kind of skills needed. OK, so with that, I'm going to stop sharing and I will turn it over to Josh for a demo.

[10:41] Josh Bartels: All right. So before we kind of go and look into the tool in the demo, what I wanted to do is go over the use cases that we're going to test today, and kind of walk you through how you can provide a little bit of detail or you can provide a lot of detail depending upon the complexity of the use case.

This first use case got canceled. We pushed an update to our website yesterday about cookies, so we're not going to go through that use case, but we have about 5 use cases that we're going to go through. One is on our website we have a tool that's a business use case tool, and we basically provide some information and then fill out a form for a business use case generates use case, but here's our language prompt that we're providing it: write a test script that completes the form at this URL, complete each step of the form, use previous and next buttons to navigate, submit the form, letting it know what success is defined, and this expected result is important when it comes to playwright scripts and UI scripts.

So the expected result is really the assertion of hey you got to the point where you needed to get to and it's the validation or failure of the test. In earlier versions of this tool, we weren't doing expected results. We were just generating the playwright scripts that were executing the steps and then you would have to go in and manually put in your asserts, but we've started to improve and we continually improve and we now have assertions working for the most part.

So that'll be the first use case we go through. Then there's this demo like doctor's appointment website that's called CURA. We're going to go through that and you can see that in this one we gave it 5 steps cause it's a little bit more complex. We're trying to give it some more detail on what it's supposed to do.

In this one, we basically gave it two steps and we let the LLM and the screenshots kind of work through the process of what it believes the test script should look like, by just giving you a general make an appointment on this website and make sure that the visit date is set to this. And then this one we're going to show you how you can stack test cases in the CSV file. They won't run in parallel yet, but you'll see them run in serial, so we'll run the first test case for the appointment, and then it'll automatically, once the first test case completes, it'll run the second test case for the login.

As we get into a little bit more complex interactions, we have a couple of CarMax examples that we'll be running through. Really kind of the first one is a script to go to the CarMax website and basically search for a Jeep Cherokee and then select like one of the records to get detail on the record that came up. The second one is doing a search, coming up with, searching on a Toyota Camry, and then coming up and actually using the compare function and testing the compare function on the CarMax website.

And then the last use case that we have is, we made this one really simple. So we're asking the LLM and AI to do a lot here, but basically just asking it to write a test script that adds one item on Amazon.com to the shopping cart, and that item that we wanted to add should be from the bestsellers category.

[14:48] Josh Bartels: So those are the use cases that we'll go through in the demo. I'm going to give you a little bit of an overview of the project and the code and then we'll hop in to actually running some of the demos. Really just to kind of talk a little bit about the structure here, we're calling them demos, but in any case, these would be like your tests that you want to run and they're all organized into specific folders based upon the test that you're trying to execute.

So this business use case one is the first one that we're going to go over. There's a CSV file in there, that CSV file is written out as a CSV file. That's why I kind of showed you the pretty version of it in a text format cause it's hard to read in the CSV, but this lines up with what's in that use case document that I showed you, and as we run this.

We get different test runs and inside those test runs, we get something that matches the test. So test ID one test. We get a results format for that, and then we also get a set of screenshots that were generated throughout the process. We get the actual playwright script that was generated as part of that process. And then we have a log. And the result basically says, hey, this was successful, this is how long it ran, this is how many prompts.

It cost us 34 cents roughly to run that particular test. So we are doing some of that to try and figure out for complexity, we do have some kind of logging that goes along with what we're doing. That's what's kind of in this demo folder. All of our source code is written in TypeScript, so it's all a TypeScript project with a main script that runs everything, and then we have a prompt script where this is a lot of like our prompt that we're using that's going into AI. We have some playwright scripting, which is really helping us with that highlight function and highlighting things on the actual screenshots that get created, and then some utilities types.

This open AI stuff is really just to connect to the API for OpenAI and then some logging. So, as any live demo goes, we're going to hope that everything works appropriately in the live demo. We have turned on, for the demo purposes, what you'll see is when I go to run the demo, it'll open up a Chrome window and you'll actually see all the interactions. This could be run in a headless mode if we wanted to, so you don't have to have the browser window open and showing, but for demo purposes, this makes a lot more sense because you can kind of see what's happening in the interactions in the website.

[17:52] Josh Bartels: So we're going to start by running the UDig business case demo. So you can see in the first thing in the use case that we showed earlier was, hey, navigate to this website. And you can see the little black markers that are on the screen with this little key and index that's at the bottom. That is the highlighting function that we have put in place, and that highlighting function is basically saying, here are the different IDs or classes or other items that we found. And Awesome.

It seems to have gotten stuck. All right, I'm going to try again. That's a new one. OK, so now it's starting to go through it, it says, OK, I need to give it a user, so I'll give it a test user. I'll give it a test user email address and a test company, and I'll click on the next button, which then opens up the instructions page.

And it's going through, it's hit the next button. It sees that here's the objective page. It may or may not provide information for the objective. That's all based upon the LLM and what comes back from it, from the AI response based upon the screenshots.

So it is providing a test objective. So we’re going to see it kind of continue to go through this activity and exercise of highlighting the page, finding the IDs, filling in information, and moving onto the different steps.

Yeah. This, so we're going to be watching this stuff. I can continue talking about it, but if you do have questions, it could be a good time to ask questions and I can answer those while the demos are running. And I promise this demo is probably the longest demo that we have that we'll be going through.

And you can see down here on the screen all the different prompts that are running through in its cycles. We put in a lot of error handling into the process. So, if one of the playwright steps that it puts out doesn't necessarily execute, we give it a lot of information to continue to retry and move on to different things.

So we've really gotten through the entirety of the So that, that test case passed and ran through and generated everything, and we got to the summary page. So if I go into this business use case and I go into tests and I go to this last test run, and we go to the results, we can see all the information about the result there. We can look at the screenshots, with the screenshots kind of showing the steps that were taken.

So this - We haven't, we haven't really thought about this as a potential, additional effect, but because of all these screenshots that are taken, it could also be a very good tool in helping to potentially generate documentation because we are generating screenshots at each one of the steps that the use case provides. And so you can kind of see all the different things that happened as we were going through the different activities.

We do have the spec here, which is the actual playwright script with all the different fields that it went through along with our specs, which are our assertions of like, hey, these are the things that we expect to have.

And for those who are familiar with frontend tests, what I want to do is show you How this actually works in playwright. So, if we open up the playwright test UI in that folder that has the spec in it. It finds the spec and it finds the business use case, and we can actually run that test in playwright. And obviously playwright goes through and does it much faster, but it's going through and has the capability to execute all the different steps. So Just wanted to show that, that we do have that capability.

[23:08] Josh Bartels: So, moving on to the next demo, which was the CURA demo, which is the, hey, let's make an appointment demo. Which has, if we look at the test script inside of the CURA demo, we'll see that we have two different test cases in there and it's going to go through and execute those in iteration. For this one, because it's a demo website, it gives you the information to like kind of log in and create your account. We have this form that loads up. One of the interesting things that we may see as an issue with this is the use of a date picker form.

We've done a lot to kind of help it select the right field in the date picker, so it worked there, but that was always one of the things that, as we find things and we continue to improve this, we're always trying to improve the capability around certain technologies on certain pages.

[24:24] Josh Bartels: So we passed that first test case and we're going to the second test case and the second test case is really just testing the login. But we, once this completes, I can show you kind of the successful test scripts for that. So in the CURA tests, we have this, we have both test runs, we have the results, with both of them passing. Their associated costs, durations, number of tokens, completions, but same kind of layout that we had in the previous one where we have the screenshots of the process for making the appointment. And we have the outputs back, which is the playwright script. Same thing with test 2 where we have the playwright script.

I believe the next one that we're going to do is showing CarMax and showing the detail of CarMax. One of the fun things with the CarMax website, right now anyways, is if I run this, I believe there's going to be a modal on the page. And so we kind of have to use some language to say, hey, ignore that modal, like make sure you keep it out of the way. And so for the CarMax demo, we've basically CarMax runs pretty quickly in the demo side of the space.

We made it through all the steps for that particular thing, and we've completed the test of being able to say, OK, we searched for a car, and we were able to get to the detail. So pretty much so far, I've showed you, business case was probably medium complexity because of the number of steps that we had to go through. The CURA one was probably low to medium, just in the sense of we didn't give it a lot of information about the steps it should take to make an appointment. The CarMax detail one was probably pretty low, it's pretty easy in the sense of we just said, hey, search for this car and make sure you click on the first card that you find and give me the detail about that.

[26:39] Josh Bartels: This next one is a little bit higher complexity. And really what we're trying to do is once we search for cars on the CarMax website, we are actually trying to get to a comparison car feature and kind of test the capability to compare cars. So we have a, we're gonna get back a bunch of Camrys.

And click that little compare button on the screen over there, we add the first car to the compare, we add the second car to the compare. And we should be able to click the go button. And it looks like it's having an issue clicking the go button, so hopefully you can find the go button.

It'll run, I think what we have it doing right now is it retries up to 8 times before it fails, and so we've made it to the compare page and we've passed the test case on that. So, we have been improving this consistently. We're not a product company, right? We're a consulting firm, right? And so we've been using some more internal resources as they roll off projects or in between projects and it's giving us the opportunity to continuously improve this. And so that one I would say is like one of the higher complexity level ones. Because we are targeting a bunch of different things that we need to do to kind of look at that comparison.

The spec comes out to be very simple, but from an artificial intelligence or AI intelligence LLM model, there's a lot that kind of goes into it being able to figure that out and being able to use these right selectors that OK, I need to hit the zero article for add to compare and then the one article for add to compare. And that's all part of kind of the customized prompt that we've been able to build to enhance how it generates these one-line playwright outputs one after the other.

[29:16] Dustin Cleckler: Josh, you got a question in the chat real quick. Someone asked if you see this tool being impacted as new LLM models are released.

[29:25] Josh Bartels: I think it can only get better, right? One of the other things that we're potentially looking at doing is, with some of the new LLMs that are going to allow larger tokens, We might have the capability to move away from the screenshot process and actually feed in like scrape the HTML from the entirety of the page and give that to the LLM model, which might provide a better predictor for selectors. We run into token limits right now trying to do that, but as models allow for larger token sizes, That will allow us to potentially create even a better response because we won't be limited by the selectors that we're selecting and putting into that screenshot.

We'll have more discreet and finite detail to actually feed into the LLM to get a better answer. So I know O1 (?) has a larger token output and it's in preview right now, and I can see that being a very useful model to potentially look at using an HTML scraping method for this or scraping the HTML and creating those as files and then using those files potentially in a RAG model which is retrieval augmented generation.

If we know the website ahead of time and we potentially can crawl the website ahead of time, that potentially gives us an opportunity to stick all of those independent files into an index that the LLM could access, which allows us to potentially even do more complex things. I’m excited for model improvement because with each new model that comes out, or each new model version that comes out, we’re seeing kind of like doubling of potential that the models have to generate things. That might slow down a little bit here in the near future, but a lot of the newer models that have come out have been producing crazy results so I’m super excited for new models. Or even I'd be excited if like somebody came out with a fine tune model on playwright scripting or playwright testing, and we could use that model because it's highly trained or highly focused on playwright, that could also be another thing that we could look at as a benefit.

[31:57] Josh Bartels: I'm gonna run the last demo, which from a complexity standpoint, it was interesting in the sense of I was trying to be very non-specific on what it should do, and this is the Amazon demo.

[32:14] Dustin Cleckler: While this is running, we have one more question. Someone asked if my client isn't comfortable running their testing through a tool like this, could it be used to do the test case creation, so not the test script creation, but the test case creation.

[32:43] Josh Bartels: I think LLMs just in general, are a good potential model for writing test cases. I particularly I'll use all of them that exist. Claude Projects is a great way to potentially create a set of test cases because you can use the project function if you have a paid version of Claude, and that project function allows you to build content on top of content and so you're basically creating a mini repository of documents that are all relevant. So you could say, OK, help me generate a test case, an overarching test case Plan or document, right? And it would help you generate that in markdown. And then you could say, hey, here's, let's just say, here's a spreadsheet of test case names that I have, and here's some relevant information.

You don't potentially have to include client name information in that, but you could just provide enough information about what you want to test, and it could help write the test cases for those different things. So great way to kind of generate documentation for testing. So that was the last one, and other than that first run of the demo, we actually successfully completed everything, so for a live demo that actually went really well.

[34:23] Josh Bartels: I'm just going to show the output from the Amazon test run. So we have the spec of these are all the things that potentially we should be able to execute to get that spec. We have the results of it succeeding.

One other question that a lot of people generally ask with something like this is How expensive can it be or how expensive can it get? I think that all depends on the model that you're using. During development, sometimes we fall back to a lower model like a 35 model or something that's a little bit Cheaper but may not be as good or as accurate, but it can still help us with development of this.

When we're doing demos or when we're actually trying to apply this in real world scenarios, we're gonna go to the newer models, but it's still pennies, not, it's not, I mean, if you run enough tests, it can add up to dollars, but depending upon the set of test cases that you're trying or test scripts you're trying to execute and how complex they are. Like when we prepare for kind of going through this or doing other demos, I don't think we ever burn more than, you know, $15 or $20 but if you have a large set of tests that you want to do this on or a complex set, that's always something to look at and that's why we're also trying to get the number of tokens used and kind of calculate cost for that. That's all I had to share from a demo perspective.

[36:07] Dustin Cleckler: OK. OK. Well, if there's no more questions, uh, we do have one more. OK. Um, it looks like the tests generated are pretty vanilla playwrights, so they should be good to throw at a platform like artillery.

Yeah, so we do not give any guidance to the LLM on how, like, we're not shading how it should generate playwright, we're just asking for single playwright lines, so they should be pretty vanilla. And the screenshots are saved in the folders that of the test runs, so they can be shared with stakeholders as you're kind of like going through the process. Uh, so as you run it, you'll have all the screenshots that are generated.

Now, what you'll see is you probably want to go through and kind of analyze the screenshots that come out because in some of the failed steps, it'll be a blank page or you'll get duplicates, but for the most part, as you click through, it's almost like a step by step process of how you go through the page and how you execute the activity.

Cool. Any more questions before we wrap it up? Alright. Well I appreciate you guys joining. Have a good afternoon. Thanks.