Wayne Eckerson: Welcome, everyone, to another JAM session. We have an illustrious panel with us here today to talk about a very important topic, how to blend software and data engineers on a single team. If you've never had the luxury of doing that, you know it can be painful and challenging, so we're going to explore that.

So let me just give a little background here, re-read my abstract. So, as software applications become more data-driven, the worlds of software engineering and data engineering are colliding, and not always in a good way. These two sets of engineers often see and approach application development in different ways, and this can lead to friction and outright conflict that can torpedo a project if not managed properly.

Our panel of data architecture and engineering experts will examine why it's so hard to merge data and software engineers, describe best practices for doing so, explore the impact of AI on software and data development going forward, and provide some illustrative examples of how and when to create a blended team. So, I'm very proud to have three real experts with us, Carlos Bassi, CEO of Datalier, Elliot Cordo, founder of Data Futures, Josh Bartels, CTO of UDIG, all in the data and software consulting space. So, let's have each of them introduce themselves.

Carlos, do you want to start us off?

Carlos Bossy: Yeah, thanks a lot Wayne. I'm Carlos Bassi, and as Wayne said, I'm the CEO of Datalier, and I've been working in this field for all my career and really focused on the data side the last 20 years. I was just making a little bit of a joke earlier that I started consulting in 2004, and one of my goals was to sell machine learning and AI, and I'm finally doing it, but built a lot of data architectures and done a lot of the data engineering in the meantime, and focused on this area for quite a while at this point.

Wayne Eckerson: Okay, Elliott?

Elliott Cordo: Yeah, so, hi everyone. I am founder of Data Futures. I've been working in enterprise and data transformation for about 25 years or more, and during that time, and I think why this topic is really, really close to me is I've oscillated between software engineer, data engineer, data engineering, data transformation, more generalized enterprise architecture, right?

So, I've been kind of teasing this interplay of the two disciplines for quite some time. So, looking forward to the discussion.

Wayne Eckerson: Finally, Josh?

Josh Bartels: I'm Josh Bartels. I'm the Chief Technology Officer at UDig, and I started my career in data, and I transitioned from data to software engineering, right? So, interesting perspectives across both sides, and I've been on the teams and led the teams, and most currently in my role, right, but that's been a blend into AI and how do we use the different practices that we have, you know, inside our organization to blend teams to produce these solutions for our clients because they're often, they need aspects from both disciplines to ultimately create a solution of value.

Wayne Eckerson: Great, thank you, and I should introduce myself. I'm Wayne Eckerson. I was president of Eckerson Group, a small research and consulting firm, until March, and then I joined Data Layer, joined Carlos.

Happy to be there. I've been in the space, data analytics space, for over 30 years, hard to imagine. So, with that, I also want to encourage our attendees, our live attendees, that if you do have a question or a comment, please put it in the chat, and we'll introduce that into the conversation.

We'd love to have you part of this. All right, so Elliot, first question is for you, and then Carlos and Josh, if you want to add some additional comments, please do. So Elliott, are you seeing more instances of software and data engineers needing to work together on a single team, and what do you think is driving those trends?

Elliott Cordo: Yeah, so I think the forced collision of these two disciplines has been going on for some time, right? I think if you go back years ago, data was relatively siloed into business intelligence and decision support scenarios, but as we started to graduate into the sub discipline of data science and machine learning, supporting CRM initiatives and customer experience initiatives, personalization, we started needing to work more cross-functionally because you're starting to drive outcomes which are more aligned with product team objectives, right? And the way that we bring these things to light is actually working with software engineers. If we want to personalize the content in some website or trigger some next best action based on some LTV or defection calculation, we can't do that in the silo, right?

So I think when done best, teams have had to work together, right? Often in the same squad or the same scrum team, cross-functionally playing to the strengths of their discipline and their capability. So I think there's been a continuation and perhaps a slight acceleration of that in the age of AI, and the reason being is there's more of that, like AI is helping with those problems, but I think we're also entering a space where AI is being further embedded into applications and the business processes in new ways, right?

Like we're fundamentally affecting the way that companies are doing business, right? And this needs the skill set and the capability of data professionals, but it's much bigger and requires really close collaboration with application owners, business folks, software engineers, probably like never before, right? So it's been a slow slope and now it's definitely accelerating.

Wayne Eckerson: Josh, Carlos, any thoughts on why these two disciplines are converging?

Josh Bartels: I would say along the lines of what was just being discussed, we've moved from data being an important asset of any organization, right? And being part of business, right? And kind of moving up the line to data supports, decision support, right?

And so some of those decision support systems needed to be more interactive where that software team was blending with data, right? And now that we are moving to the age of AI, data is becoming this even larger asset because for AI to learn, it needs data, right? So, and for it to be able to respond or generate or model or decision anything, right?

That integration needs to come together, right? And I think what it's doing is it's breaking down this boundary of, okay, I developed the model and here's the inputs and outputs that I need for the model. Software engineer, just take the inputs and the outputs.

And they're coming together because the feature set that needs to be given to users, whether those are associates inside of your business or those are clients or customers, you have to, like the software engineer fundamentally has to understand what's happening with the engineering of the data. And the data engineer needs to know how the software engineer is exposing that to the user, right? So you're getting this blend of how do these things work together, right?

And so they're fundamentally being clashed together because of the experiences that can get created for the end user. And it relies on both sets of their skills and both sets of their outputs, right?

Wayne Eckerson: Well, I should clarify an assumption that I probably had in putting this together, that there's a movement or a trend to put these two folks, these two different types of engineers on the same team. But maybe we're also asking these different types of engineers to learn the other discipline. So I'm wondering which is more true than the other.

Are we creating a single team with different skill sets on them or are we expecting these engineers to increase their skill sets across these two disciplines?

Josh Bartels: So we like to think of folks as T-shaped individuals, right? Like you're deep in a certain area, but you have a broad understanding or spectrum across what you're doing. This is particularly beneficial in consulting, right?

Because the project that you're on today, two weeks from now, that project can be a completely different client, completely different outcome that you're trying to achieve, right? And so I think when we're blending these teams, what we're seeing is I might have deep expertise in the data side of the house, but I'm having to come up and learn a little bit more about software or the experience that software is trying to create. And on the flip side, if I'm a software engineer that's deep in, like we even see it just in software where there's a split between back-end and front-end, right?

Like, hey, I'm a guy who writes APIs versus I'm the guy who creates the web interface, right? They've had to learn how to work together as well. And now we're just blending this third T-shaped individual into the team to ultimately create these new, I mean, it's not like we haven't been doing it for years, but because of how fast we can move nowadays, these data enhanced experiences that we can create for associates and consultants infused with AI, these teams are having to work together even more, right?

So I think the thing, it's more about how can you see the perspective of the different engineering disciplines so that you work together in those different perspectives, because they do think about things differently and they solve problems differently, right? But the perspectives in kind of having those conversations across those perspectives are what creates tension, but also potentially create really good solutions.

Carlos Bossy: Yeah, I would agree. I think what Josh is saying, right? It's different skills, people with different skills coming together and working together.

And when I go back and answer like the original question, I think if I were to go back 15 years and look at, I don't know, maybe the start of the accessibility of being able to build a big data system for analytics in most enterprises. So going back to 2010, at the time there were maybe two reasons that I can think of that people, you might have people like this work together. And one was part of it was just timing.

I'm developing a new application and I also want my measures and metrics to be produced at the same, to be deployed at the same time. So as soon as the application's out, I can also measure what's going on and bring those measures together with other parts of my organization. The second reason is, should the application or did the application people need output from our big data system?

And in some cases they did, but that didn't happen very often. And now 15 years later, here we are where the output from the data system is output of machine learning models and AI models and things like that. And every application needs that today, or should be looking at that in the future.

So I think there's always a timing aspect of it. The data guys, the data people were always lagged behind an application and always maybe deployed afterwards. And it would be good to line those up better.

And I don't know that we've ever lined them up as well as we should in most organizations, but then the other side of it is the output. And now the output is getting very almost circular, right? That what goes into the big data system comes back out and it goes right back into the application.

So I think a lot of that is forcing us to do this.

Wayne Eckerson: Yeah. Yeah. Data was always the stepchild, but now it might be the tail wagging the dog.

So interesting. Yeah.

Elliott Cordo: So I liked where both you guys took the conversation. So I think, and I appreciate the T-shape. I think that makes sense.

This better be like a broad understanding and some baseline skillsets. There's some non-negotiables that I think we're developing. I think it's more so where software engineers are influencing data engineers.

So quietly over the past few decades, software engineering has become pretty close to a mature engineering discipline. And I'm a classical engineer, EE and ME by degree. So I know what that looks like.

But they've slowly gotten there and there's a lot of maturity to what they've done. And in many regards, that maturity for data has been decade or decades behind, right? Where software is.

So I as we're ending up in this integrated environment and we're thinking about transformation of roles for data professionals, data engineers, and the like, some things like you're going to use infrastructure as code. You're going to understand continuous integration and deployment and not just from an understanding, but you actually have to do it. It's part of your determination of done, right?

You're done when you have a repeatable way of creating the infrastructure, deploying your code, testing your code, right? And there's a lot of things there that we can unpack, but there is an influence for software engineering to help mature what data engineering is doing. In terms of the opposite, similar to what Carlos has said, Wayne, you just said the stepchild data being the stepchild or the afterthought.

I think there's a better understanding, right? Like the T-shape for the software engineer and software product professionals that data actually does matter. Then maybe we don't want to just treat data as an afterthought, something that comes out of the tailpipe of our application.

We want to be thoughtful about how we generate data. Heck, we're going to be accountable for a test and measurement plan on whatever product or feature we're creating. So how do we design for that during the process so it's not a month later when their boss comes up to them and says, hey, how's that product doing?

They say, geez, I don't know, right? And then there's convulsions of people trying to stitch together data that might not even exist, right? So I think a totally, totally great conversation, both kind of like lanes.

But yeah, those are some initial thoughts.

Wayne Eckerson: Yeah. So it is true that data has been behind the software space. They developed DevOps like 20 years ago, and we're just starting to figure out what that means on the data side.

But these two worlds, I want to talk about how they approach things differently. Josh, you kind of referred to this. We had a client a couple of years ago, and Carlos was actually working on this project.

It was a software application startup, and they had built a whole bunch of complex code. And they realized they were using more and more data. They were consuming data, and they were spitting out data, actually statistics for sports fanatics.

And so we went in there and helped them create an enterprise data model, which they were very thankful for. And then they said, well, I think we need to have a data engineer working with us. And we said, great, well, we can supply you with a data engineer.

Well, the poor data engineer didn't last very long because they just couldn't see eye to eye with the software engineers running the project. It was like oil and vinegar, right? So a poor data engineer got spit out pretty quickly.

It just was a clash of cultures that didn't work. So a long-winded lead-up to the question for Josh, which is, how are these two worlds different? How do they see and approach development projects differently?

Because if we can understand the differences in what they're thinking and when they approach a project, maybe we can better reconcile how they work together.

Josh Bartels: Yeah. I think being in both sides of this world, having actual experience in both, I've lived through the differences. I think the first thing to really look at is the differences in mindset.

So this is changing with AI, but I think if you look at the traditional practices as it relates to that, software engineering is all about, the core is about the feature. What's the functionality? What's the business logic?

That's what I need to drive towards. That core problem I'm trying to solve is all about the logic and the feature. Versus data engineering is always looking at data as the primary asset.

What do you want? What are we trying to capture? What are we trying to record?

Why is it important? How do we need to structure it? What are the relationships?

That's the two different mindsets. You take that contrasting mindset into how they look at development, their steps that they do from a development perspective. SDLC has been around for years.

There is a very structured way for SDLC, software development lifecycle, where you have requirements, you have design, implementation, testing, deployment, and maintenance. Data looks at it from a flow perspective. Where's the data being generated?

How are we going to ingest it? Where is it being stored? How are we transforming it?

How are we serving it? If you take both of those, here's my mindset, here's how I go through my process, you can see that there are some natural things that come together, but there's a lot of processes that are distinct in nature based upon how they think about different things. I did really like Elliot's comment of what we're starting to see is the spline of organizations of, hey, can you create a spline that is like a pipeline for deployment?

Both software and data are going to be using that spline for deployment. CI, CVE, and DevOps are now part of that core thing. That's fundamentally helping bring some things together.

I see those big differences in their background. I'm focused on features, I'm focused on developing core business logic versus I'm focused on capturing the information I need, transforming the information I need for consumption. They're both consumption-based models, but it's consumption of the feature and the logic versus consumption of the data for making a decision.

Those two things are different. Fundamentally different, but as AI drives these folks together, the consumption is overlapping because now it's data-driven decision-making and data-driven input and updating. Long story short, fundamentally, there are differences, but I think we are getting more to emerge.

I think we're probably going to talk about this later. I do think there are some things that you can do inside these teams to help blend those differences and start to create commonalities across the team from an understanding of if this was your approach and this is our approach, we say this, but what it really means is the same thing as what you're saying for this. If I'm doing data generation ingestion on the data side, that's really potentially some design aspect or some requirement aspect on the software side.

You just need to do a little bit of the translation between what they see as normal versus what software sees as normal, and you start to create some overlaps that will create natural conversations.

Carlos Bossy: Yeah, I think I would add, and again, probably saying the same thing a little bit differently, but in today's world, software engineers are trained and know how to manage code, and I think data engineers, that's not a focus often. Software engineers manage code so that they can produce those features in highly scalable ways, easy to deploy, make them a deliverable, testable. I think data engineers often come from a background of data analysis and how do you manage the data so it can be analyzed properly, and so in that sense, I think that creates a lot of the challenges because often, if you think about data engineering as a discipline, we really don't have a common or a standard training model for a data engineer, or if we do it, there's not a lot of history behind it. People get trained as software engineers and transition over to data engineers, or they get trained as data analysts, and they end up transitioning over to data engineers, so there isn't even a standard model for how data engineers should work. Often, they come more from the analysis side of things or tend to have that mindset at least, and so they're looking at it from that point of view, whereas the software engineer needs to know how to manage code and write it efficiently and effectively and make it, again, highly scalable and deployable in ways that often data analysts don't think about.

Elliott Cordo: I like what you're saying there, which is kind of like, and I think this is because this is evolving, this is a more recent acceleration to the synergy between the professions. I think that what good looks like for a data engineer, even in terms of some of the emerging tooling to support them better in this transition, is still in flux. What good looks like is in flux.

You get a software engineer and you're like, okay, I can give you a series of books which tells you what good looks like, and that those sort of things don't exist for the data profession. Just two other thoughts which kind of build upon Josh's comments, I think, and kind of where he was going. So in terms of impedance, I think two things come to mind.

One is kind of like the way we think about developing things, and often we do use code. We might be using Python, might be writing SQL pipeline as well. We're procedural programmers.

Data folks are procedural programmers, which is not the model for most software engineers, right? So they get all the goodness of dry, clean, reusable code, and we really struggle with that. And I think data engineers really struggle with that.

And some of it is the corner in which we've been painted because of the technologies that we've gravitated to to solve our problems, right? But not that I have necessarily a solution, right? There's a lot of folks in the DevOps community, and even in tooling DBT, which are trying to bring some level of maturity to our procedural development.

But that's one thing that I think of where we end up being apples and oranges with software engineers, where what we do is like the antithesis of what good looks like to them, right? And we could have a whole discussion on that. And I think the other thing is that we're generally not...

I think data professionals are generally not good at breaking down big problems into smaller ones, right? And as we start working together in agile execution, I often see a lot of data professionals just kind of falling behind. They have nothing to demo in the sprint end, right?

And I think some of that is the T-shape, right? The product managers, the folks leading the project don't know how to break down our work. We don't know how to reflect our work into more manageable chunks.

And I think there's also just the nature that some of the things that we do is just like innately, it's done when it's done, right? It comes in a big chunk, a big release. That's impedance in terms of us working together for sure.

We fall out of sync, even in a blended team, it seems a little alien sometimes, being the data person in the cross-functional team. So just some thoughts on that subject.

Wayne Eckerson: Well, I think you articulated exactly why this one client of ours, they rejected the data engineer, we said to them, this whole notion that our data engineer was working procedurally and building flows and their software engineers were really more focused on microservices and object-oriented stuff and it just did not gel.

Elliott Cordo: Yeah, they're like, your code stinks, what are you doing?

Josh Bartels: I do think you brought up a couple of good points though, right? So data engineering, if you go back, right? If you think about it, right?

Pipeline development was very graphical in nature. You're using tools like Informatica, IBM data stage, right? You're dragging things on and maybe scripting smaller things around it, right?

So you weren't thinking about this whole kind of, like if you're trying to create something in Airflow today, which is different, right? There's a whole bunch of code that goes around it, right? So because more data movement and data manipulation is now code, then kind of like these older application architectures, which were more of like GUI based, right?

I feel like we're seeing, that's kind of where data is starting to catch up from a software perspective, because we're writing more and more of what we're doing, right? It's kind of like where we're at from like an infrastructure perspective, like where infrastructure is starting to be released as code, right? Which is part of that DevOps pipeline and being able to release and stand up servers without having to actually physically click buttons, right?

And so I think that point that you made, like things like DBT, or things like some of these other frameworks that are coming out are making it way, are making it more, I guess, making it so you have to be more disciplined about what you're doing, right? And you're using Git more for your data, right? So you're having versioning, right?

You're following some of these processes, which also has allowed for larger data teams to work together, because now they're using that repository pattern, right? On the flip side of like, hey, it takes longer for data to kind of like do things and kind of, you know, I think there's, when I was data, data is all about perfectionism, right? Like there could be quality issues, but you don't want to show those quality issues to the user, right?

With software, there's this whole concept of demo, and if it completely work, it's totally fine, right? Like incremental build and incremental failure is part of software development, it hasn't really been part of data. I do think that is something that could be potentially added in that discipline as you're blending teams, especially when we're getting to AI, it's more about rapid prototyping, and showing the proof of value and proof of technology.

So that MVP concept that's come from software, and I think that can be applied more to data, right? Like, what's the MVP for something for data, and how do we incrementally get there? So you don't have the longer chunks in timelines, and you still have something to show to be part of the team when you're in a blended team.

Wayne Eckerson: And I just want to remind our audience that if you do have a question or comment, please put it in the chat and bring it into the conversation.

Carlos Bossy: And a quick comment to that is just that a lot of what we do in data is not very visual, right? And I think there's that hesitation, right? Okay, in this last two-week sprint, I ingested some data into a landing table, then I loaded into a staging table.

Well, who wants to see the staging table? You know, that would be like the developer pulling up the orders table and showing that the transaction is there, right? That's not really what's demoed.

So I think that's a little bit of a, you know, of something we have to overcome because- Yeah, that's what's new. Psychologically, right? Yeah.

I think folks are getting a little more used to that. So I think we got to, like, you know, if we've got a dev ops or people who have been charged with infrastructure, like, you know, they awkwardly muddle through showing something, which maybe is not a pretty UI. So I think we just get, we got to hit over.

Josh Bartels: Yeah. I think that also helps, like, who are, like, you might have stakeholders that care a lot about software. You might have stakeholders that care a lot about the data.

Both of those people should be at the demo, right? Like, that's also part of that process. Like, are you bringing the right people in?

Because there might be somebody on the team that's really excited that the data is there, right? Maybe the person who doesn't care as much, like there might be a person who's really, like, wants to see the thing spin on the screen and cool, that's a good demo. But there's going to be somebody in that business that really cares that the data is landing where it needs to land.

Wayne Eckerson: So I just wanted to check an assumption. We talk about how data is becoming more, adopting some more of the disciplined software development practices. And we call that data ops now, right?

You know, CICD testing as we go along. So you had mentioned that we're doing more development, more programming in the data side. Do you all agree that's a good thing, that we're getting away from GUI-based data development and going to more hand coding?

Carlos Bossy: I don't. I think GUI ETL development using GUIs was poorly implemented or poorly utilized often, right? Often the GUI was an orchestration engine and you still executed code under the covers, which I think wasn't good, right?

In the sense of you're not using the power of the ETL engine, but you're also kind of hiding the code a little bit, right? And so going in the direction of where it's a code first type environment, if for anything, it makes the code more visible, right? It's not, you know, often we would look at an Informatica job or, you know, you'd see, you know, this is where I merged these two tables together.

And you didn't, you didn't have the finest stored procedure that it was calling to see how it did that. Then the server user might've been thousands of lines long. So there was a lot that was kind of, you know, I think that wasn't a good environment, let me put it that way.

So making the code more visible is always good. And if you're going to use a GUI, use the GUI, right? Use the power of the built-in functions of the GUI, right?

So a lot of times Informatica and DataStage, SIS, all those tools, they were actually not truly being used as ETL engines, but, you know, they were really only being used to schedule and orchestrate a lot of times.

Josh Bartels: It's probably why we went from ETL back to ELT, right? So they're using the database engines underneath the hood for most of the work anyway. Right.

Wayne Eckerson: Okay. So let's take the next step here. We've talked about how these two worlds are coming together.

We've talked about how the different engineers approach things in the world differently. So how do we blend these two groups together? How do you build a team or teams that work side-by-side in an effective, functional way?

So let me throw that to Carlos here.

Carlos Bossy: Well, I think from a point of view of how they operate, right? There are strengths that I think data engineers can bring to software engineers.

I think one of the things that comes to mind, maybe two or three things come to mind. The first thing that comes to mind is software engineering, I think, could do better data testing. And they do very good feature testing or functional testing or application testing, but often the data testing side of it is it lacks.

And of course, the data engineer, that's often all they can do is data testing. So bringing that discipline a little bit more into software engineering, I think, is a plus. I think software engineers could also help the data engineer sometimes.

Sometimes there's decisions that are made on the engineering side to make the data engineer side much harder. And there could be ways to structure data and tables that are much more helpful. And that's often not in the software engineers mindset.

They're just trying to get their thing done and whatever works for them makes it as good. So those are two primary things, I think, on that side. I think software engineers could also understand that much of what happens on the data engineering side sometimes happens in the business.

There's maybe a heavy duty transformation process taking place business-wise that's been developed by business users and that needs to be integrated into the engineering practice. And that's a hard thing to do, right? That's not something a software engineer would typically tackle.

As far as what data engineers could do, we've talked about, obviously, I think we've had this conversation before about DevOps and other things. And the disciplines and the practices that could be better implemented that software engineers have used for years now. But if I would say one thing, and it's an easy thing, I wish data engineers would do code reviews, right?

That's like, let's implement a code review process. That'll improve things 100% right there. And in the software engineering world, often you can't do anything unless somebody else has reviewed your code.

Let's do that in the data engineering world.

Elliott Cordo: Yeah, so I agree with you. I think, you know, code review and kind of like development process, you know, I think can go a long way.

I think when you said data testing, that kind of made me think about a couple of things. One is in this blended team, if we were dropping just somebody there, but we fundamentally don't care about the outcomes related to data, we're not going to be successful. Like we can put the talent together, but if we don't have a team that's aligned on all the objectives, including the data ones, which also means creating good data, I think we're not set up for success.

That makes me also think about like some data responsibility too, right? Like, do we want to try to define a better contract with the software, right? We often like, you know, lift up the curtain to the relational database and go poking at its innards, right, underneath the applications being developed, even though that team would not allow anybody else to do that, you know, just a state of purpose, right?

So I think there's something about data contracts. I think there's about testing, fixture data, expectations, right, on the data. Like if, you know, there is a, once we define those contracts, right, we should make them testable.

So as the software goes and develops our code, and in the future, months from now, as they're maintaining their code, that if they do something that's fundamentally going to break data or cause its degrade, they should, you know, fail their pipeline, right? Just like they would any other bug or security risk or whatever other, you know, kind of programmatically enforced condition there was on their code as they were trying to release it. So, yeah, I think, you know, hopefully that makes sense.

But I think there's some, there's a little bit of a push and pull on kind of like process ownership, valuing data, right, as well as like maturation of the way that we interact with software systems.

Josh Bartels: I'll take a little bit slightly different tack about kind of team development. So norming, forming, storming, right? Creating a sense of a shared value and outcome of whatever the team's trying to achieve and making sure everybody understands what they're accountable for as part of that process is beneficial to the team, creating these shared experiences and shared collaboration on how they want to operate.

Like I did say, there's like a separation in disciplines and software between front-end engineers and back-end engineers. I feel like data engineers can partner really well with back-end engineers, right? Like they're the ones who are often creating the CRUD operations or, you know, create, read, update, delete on those tables, right?

So I could see those folks partnering together on things, right? So, hey, I'm responsible for writing the microservice API, but this is what my data structure needs to look like. How do we engineer that, right?

Should that be a Mongo, you know, JSON structure store, right? If I do that, what does that do to the data that we're trying to do from a consumption standpoint, right? Those conversations become more about, well, does that fit the vision of what our project is, right?

Like is that going to hit the value spot that the business is looking for or the consumer is for particular things? And so at the beginning of our projects, we like to kind of do an activity where the team gets together. They focus on the impact that they're trying to make, and then they poke holes in it, right?

They ask questions about is that truly valuable? Like how does that operate? And then the second thing that we do that's really valuable is setting team norms and then team communication.

Like how best do you like to receive feedback? How best do you like to kind of share ideas, right? And then setting those norms for like either having open hours or doing code reviews, right?

Like, hey, code reviews are now a shared responsibility. It's not just data and software. So it's taking it up a level of like some of the things that you can do from a team-building perspective, right?

And kind of organizing that team for that shared vision, which could help create a better, more consistent approach to how they tackle problems.

Wayne Eckerson: Yeah. Let me take this a little bit orthogonal direction, because I like what you both said. But in a lot of our clients, you know, the application developers are over a year, and they're building stuff, and they're changing stuff, and they're changing the schema of their application, and they don't tell anybody.

And then this stuff rolls downstream, and the poor data folks, you know, the pipelines break, and they get blamed, right? So we oftentimes start to recommend, you know, architectural review boards so that whatever is being developed on the software application side actually has a data person at the table. And I'm wondering if any of you could comment on that process.

Elliott Cordo: No, it never happens. Like, you know, upstream changes break data pipelines, right? We've never heard that.

You know, I think that the idea of like an architectural review board or some sort of like, you know, process that enforces, you know, a certain consideration, like across, you know, consumers, etc. is a really good idea. But I think I get back to like the naughtiness of what we do.

You know, reaching into, you know, so if you're an application developer, and you're developing some application, and you are, you know, trying to deliver some features, supporting specific outcomes to like customers or whatever, right? You are probably not worried about what's happening downstream, like your reward system is not set up for not breaking data, right? You're, you want to have an application that's stable, and you want to have an application that delivers features and development.

But, you know, getting back to like software engineering patterns, they've kind of figured this out, you know, mostly, if you think about microservices, the whole concept is that you communicate through well defined and tested contracts, right? So what that does is it gives these applications, the ability to evolve change, like I've been in microservice environments where I wouldn't say necessarily for good reasons, but I've seen teams rewrite the same service, like five times, doing a migrant, my language migration, at least twice, and their consumers, the services that they interacted with never knew the difference, right? So like, I think there's something to be learned there that we're maybe like relying on our old ways of like, okay, just give me like the JDBC credentials, and I'll go get my data, right?

You're going to be prone to failure. So I think like having like reviewed boards process, that kind of stuff is really great. But we should save that for very special things when we know we're going to have breaking changes, we need to talk about it, right, as a publisher and consumer group.

But I think in general, we've got to approach the problem differently. We've got to use different technologies, we've got to maybe think about, you know, how we enforce contract standards and in data consumption from these different applications. Because we've been fighting this fight for a long time, and it hasn't necessarily gotten any better.

So I think we have to approach it differently.

Josh Bartels: I like to think about it like, when we used to work with data, we were tight, tightly coupled to the object. So it's Eva, right? Yeah, super tight coupling, right?

So something changes, it breaks something because it's tightly coupled to whatever that changes, right. And so I feel like you start to see more consumption of like this loose coupling where, hey, my data engineering pipeline now uses the API, it doesn't use the actual access to the database, because the API contract can be versioned and changed, right? But if that API changes, it's not going to break my pipeline, right?

Because it just got versioned, I can use the new version, right? And I should get a deprecation notice. Like I do agree, there needs to be communication between the teams.

But I think, you know, that application of loosely coupled architecture, it's starting to apply it for how you move data or work with data from a pipeline perspective is kind of helping with some of that. But on the flip side, right, like what we're also potentially seeing to drive the right behaviors from a software engineering team perspective is, if you're a really high performing and high functioning organization, when you make software changes, and you migrate database changes, when you put that in the pipeline, that runs the DevOps CI CD pipeline, that should be doing the data pipelines as well.

And you should be failing tests from a data by my perspective, right? So the software team will know that they're breaking a data pipeline, if you have that, if you have a really good CI CD process that connects all of the strings for those changes.

Carlos Bossy: Yeah, I mean, I think you're you're talking about like a lineage, right, that, you know, let's make sure we understand the lineage. And I may be right at the start of the lineage as a software engineer. But you know, there's, there's a lot of people downstream, a lot of things downstream that that hold me up.

And I think, I think, you know, we often focus on schema, you have schema problems, let's just put it, you know, schema gets changed, schema gets updated, or it doesn't work for me anymore. Even a data type changing could cause a failure. But we also have the situation of, I kind of mentioned it earlier about data testing, you know, the software engineering team ran through their changes, and everything looks great.

But now they're not updating the modified timestamp anymore. And nothing fails. Except when we go pull data from that table, the modified timestamp doesn't tell us the data's changed.

And so those are the kind of things we also so there's the schema aspect and the structural aspect of it. And API's and contracts can certainly help us with that. But then there's that data testing aspect of it that I'm very reliant on sometimes.

And so those are the kind of things that today in today's world, the only way to solve it today, or the best way is we work together on the same team and, you know, talk to each other. But, but, you know, there'd be hopefully there are there's better ways to do that, so that we don't introduce those failures into the system, surprisingly.

Elliott Cordo: Yeah, I think I think we're, we're starting to get there, though.

So I think like, I think Josh's, you know, example of like, hey, we're going to use an API, because I know it's versioned, and I know it'll be, you know, well tested, hopefully. And, you know, if there's a breaking change, there'll be a new version that I can accept or not, you know, etc. You know, so there are patterns where like, kind of we've, we've, we've kind of like leaned into software engineering patterns, such as like, if you have an event based architecture shop, right, and they're publishing events of a strict and forced schema, and there's a schema registry, and, you know, you know, can data just be another consumer, right?

Can it consume events? Can it consume APIs, just like other microservices, you know? So there's that, right?

And then there's kind of like emerging, like we've ended up in, as I said, like in a certain corner, in the way that we approach problems and the tools that we have. So in that case, you know, can we start, you know, expecting the application owners to develop projections of their product that's in a space that we know how to deal with, right? Can they project a, you know, something in a relational database that is tested, like an API, right?

Can we get them to start building dbt expectations and tests into their pipeline, even though they might be like, why is this here, we've got to like, learn a little bit, expand the width of our T a little bit so that we know how to test data before we ship, right? So I think that, you know, the microservice patterns and, you know, the way that applications talk to each other, there's something to lean in there, and it works in the majority of cases. And then there's kind of the heritage that, you know, again, like there's tooling evolving, you know, and data ops movement, there's a good portion of it that's around like data testing, and seeing if we can move some of that upstream into software engineering pipelines, and not just lobbying it over, but having them assume ownership, because it's important, right?

So just a few ideas there.

Wayne Eckerson: Okay, gentlemen, we're going to go into our flash round, right, where you each get to put, look at your crystal ball and tell us how things will be different in the next 18 to 24 months. So specifically, how will application development change in the next 18 to 24 months, we've talked about AI, we've talked a little bit about tooling for doing development, but maybe this could be your summary statement of where do you think things will be going, or should be going?

Who wants to start?

Elliott Cordo: I'll go. So, you know, we went through this whole conversation without talking much about AI, right, other than like some of the AI-driven product outcomes.

So if I were to kind of like think about this from like a software and data engineer persona, right, and I'm also going to approach this from like the optimist perspective, because I know some that are less optimistic about how those, you know, roles evolve, right? So I think, you know, one thing we're going to have a lot more code being generated by models, it's already happening, than humans, right? But what I like is that there's also like a democratization around like some of these things that seem to be an impedance to developers, just like looking up like, hey, what's the documentation on this Python library, like AI is helping us do that, right?

Now, you know, as a data engineer needs to build the deployment pipeline or needs to develop IC to deploy their code on infrastructure, these are things that largely they have a toolkit to do. So, right, I think what we're going to see is folks that understand the objective and kind of understand what they need to build and have a broad T so they understand generally how these things work, we're going to see them, you know, with the ability to be much more capable in areas that felt uncomfortable, right? So everybody can be an infrastructure engineer, right, to some degree.

Everybody, you know, can build, you know, data tests, right, which we talked about, which we feel are really important, right? Generate, you know, data tests for my application so I don't break my downstream consumers, you know, these are all things that will come about. So I'll just kind of stay narrow to give everybody else a chance to talk.

But I think developer persona, I just think that it's going to normalize a little bit and people are going to, you know, I think this tension and this impedance in terms of skill set is going to, like, level out quite a bit.

Wayne Eckerson: So we're going to move from maybe a T-shaped person to more of a, like, you know, X that can do it all?

Carlos Bossy: You know, as I was, you know, as Elliot was talking, I was thinking, I'm not sure what my answer is here either, but I think the way I kind of was framing it in my own head is maybe we'll, I'm not sure it'll happen in two years, maybe it'll take longer, but you never know in this business how quickly things happen sometimes.

But maybe what we're really saying is we're going to blur the lines, right? Because, you know, there was kind of a pretty well-known division, I guess, or pretty well understood, maybe division between data engineering and what led to data analysis, business intelligence, and things like that, reporting versus what a software engineer was doing. But are we really blurring the lines now, right?

I mean, is there any AI or any software system without AI? And is AI a data system, right? And AI really is a data system.

And so, and I need to use the data, the output of that data system in my application, and I need to also send data to it. And so, maybe in the end, what we're really, what I'm really saying in that sense, and kind of similar to Elliot is we're blurring the lines. And two years from now, engineering is just engineering.

Yeah.

Wayne Eckerson: Okay, I like it.

Josh Bartels: And I kind of agree with that, right? Like, I think engineering is going to be the discipline.

I do think you could still have specialized knowledge inside of that, right? But I think 18, 24 months, what AI can do is also potentially help advance data up to the software space, right? Like, we're already getting AI code reviews that can be done on pull requests for software.

Why can't we do the same thing for data, right? Like, those platforms are going to help push and evolve folks in that way. So, we could see an acceleration, which is just getting more to this, like, hey, it's going to be blended, right?

Because we're going to be using the same tools. On the flip side, I think, on the flip side of that, what we're also potentially going to see is talking about the data generation, but data generation in context for my tests, right? Like, how can, right now, I might go into like a data generation tool and just say, hey, I need names and blah, blah, blah, but that doesn't give the real context for fixtures or add volume, right?

That's going to help, right? Digital twin is another thing that keeps coming up of, like, how do I make a digital twin of a physical thing, right? And then be able to test against that digital twin.

Big in, like, space planning and floor planning. So, I think we're just going to get more tools, but if I step back, right, in what we just talked about where AI is a data-based type of product, data has been blooming forever, right? Like, there's a reason that we had, like, vCAN fly file, et cetera, that bloomed into relational, that bloomed into big data, that's blooming into, you know, all this other thing.

With AI, it's just going to want more and more data, and it's going to create more and more data. So, as we, you know, this was the year of hype around agents, right? Well, maybe there's some promise there that hasn't really come out, but there's still promise there, but a lot of it's data-driven, right?

Like, and how do I align the data with the right model, with the right context, whether that's software perspective on agents, like, because now we're starting to get into, I have a managing agent, and then I have sub-agents for that managing agent, right? And so, there's some techniques in there about how do we manage flow, and how do we kind of, like, manage the messaging, like the loose couple between agents, right? So, I just see over 18 to 24 months, we're going to continue to get this confluence of these skills and knowledge that have to come together because the technology and the outcomes that business want are going to drive that, right?

Wayne Eckerson: Well, that seems like a great place to end. So, I want to thank all of our panelists for a great discussion, very insightful, a lot of great points raised. I'd like to thank our audience as well.

Even though you're very quiet, we certainly would love to hear from you after the fact. If you have a question for me or any of our panelists, please send it along. And just again, thank the panelists, thank the audience, and I hope everyone has a great day.