Heather “Lucky” Penney: [00:00:00] Welcome to the Aerospace Advantage Podcast, brought to you by PenFed. I’m your host, Heather “Lucky” Penney, here on the Aerospace Advantage. We speak with leaders in the DOD industry and other subject matter experts to explore the intersection of strategy, operational concepts, technology and policy when it comes to air and space power.

So AI is one of the hottest topics in defense. Just think about how AI can help process imagery for intelligence, support mission planning efforts for combatant commands, generate air tasking orders, and even support air battle managers.

We talk a lot about artificial intelligence when it comes to collaborative combat aircraft or CCA for short. But here’s the deal: while all of these efforts are impressive, we’re really just at the front end of the AI revolution. The kinds of AI that the Air Force and war fighters need is not Chat GPT or other large language models or even image generation models that most folks are familiar with, the kind of AI technology that war fighters need, although developing rapidly, is incredibly nuanced, complex. And while there are a [00:01:00] lot of potential use cases and even some that are ready for initial AI tools and functions now, there’s still a long way to go when it comes to subsequent development and broader applications. While much of the focus is on CCA, artificial intelligence autonomies and initial development.

We must also be thinking about and building in how war fighters will interact with the CCA agents when these aircraft are fielded. To be clear, this goes beyond the tactical user interface during mission execution. We need to be thinking about how we continue to train the autonomy after landing. This means debriefing.

How will we, and how should we debrief CCA after the mission? And how should we integrate debriefing CCA with the human formations? Because let’s face it, if we’re flying and fighting together, we can’t debrief separately. We have to understand how we operated as a team to learn the right lessons for both human pilots and the CCA agents.

At Mitchell, we’ve been following AI and its implications for several years, and it’s a fascinating topic. And the way we look at it is this, what are the strategic [00:02:00] objectives that we want to secure? Can AI better help secure those goals? And how might the use of AI fundamentally change the tasks and functions we execute in the achievement of those strategic and tactical objectives?

How do we manage risk through this transition? Plus, from a challenges and opportunities perspective, how do we weigh the relative benefit of implementing AI in these functions? Do we really understand the costs and limitations of AI and how the implementation of AI changes human responsibilities, taskings, and ultimately mission outcomes?

It’s important to recognize that no technology is a silver bullet. It’s all about ensuring net progress. How can we maximize the new positives and how do we mitigate any limiting factors? And if we use AI to debrief CCA and human CCA teams, then how else might we apply AI to improve mission performance for humans?

One area where we might get an outsize return is air crew readiness. For our regular listeners, you know that [00:03:00] readiness is a major concern of ours here at Mitchell. In the Cold War, we used to pride ourselves on how many hours we could fly well above what the Soviet crews got. We may have been a smaller air force numerically, but between the training and the tech, we were vastly superior.

But this position has eroded. Years of tight budgets have seen the Air Force grow smaller, older, and less ready. At the same time that the People’s Liberation Army Air Force in China fields an Air Force that is the largest youngest and most ready in its history. Clearly part of the solution for the Air Force lies in recapitalization with aircraft like F-35, F-15EX, F-47, B-21, and even CCA, but we also need more flying hours. However, even if we do all that, the question exists, can we make the flight training process more effective to maximize the benefits realized from each sortie?

Can AI be part of that process? Added onto that? What about sorties flown earlier in a pilot’s career during flight training, pilot training? [00:04:00] Does AI have a role there too? And could AI be used to accelerate the training of a pilot, increase their mission performance at each step, and ultimately make them a more lethal war fighter faster and with fewer sorties than the traditional pipeline?

So these are big questions. They’re super complex, but they’re crucial to explore because aircrew readiness is one of the most important attributes for the Air Force. And if there’s a way that we can help improve this part of the equation, we need to pursue it. We can’t keep letting standards slide. So joining us today to dig into this are two AI experts, Mike “Pako” Bonitas of Shield AI and Ray “Crypto” Amara, who recently joined Mitchell as a senior fellow.

Now, Pako is a longtime Strike Eagle WSO with a tremendous amount of experience, both in the operational and the test portions of the Air Force, and he’s now been working on these issues on the industry side of the equation. There is almost no one better than him who knows not only what the issues are when it comes to AI agents, as well as seeing it from the operational fighter pilot point of view.

Pako, it’s [00:05:00] always awesome to have you here.

Mike “Pako” Benitez: Thanks. I’m glad to be here. That was a really great intro. I hope I live up to it.

Heather “Lucky” Penney: Don’t disappoint me. Right. And Crypto is a C model Eagle driver. Alright, boys. No fights, right?

Ray “Crypto” O’Mara: Yep.

Heather “Lucky” Penney: So Crypto or his PhD focused on AI and autonomy at MIT. He was also involved in the Air Force Enterprise Capability Collaboration team that focused on air superiority.

And this effort led to the foundation of what would become CCA and the F-47. Crypto later served a DARPA and just wrapped up a tour in OSD. Sorry about that man. But we’re proud now to have a as Mitchell teammate. Crypto, welcome.

Ray “Crypto” O’Mara: Thanks very much.

Heather “Lucky” Penney: Okay, so Pako and Crypto, I tried to explain this in the opener, but you guys have so much experience and so I wanna hear your perspectives too, lay out the problem statement for us.

The Air Force is moving forward with CCA, but the focus is mostly on development. We need to look at operationalizing these aircraft, what that means, and what a day in the life looks like.

Mike “Pako” Benitez: First of all, I’d say [00:06:00] I love it the fact that the Air Force is moving out with incredible speed and vision on the CCA program. We’re seeing the prototypes first flight should happen any day now as this recording comes out. Knock on wood. But at the end of the day, the hardware is not new. We know how to build, we know how to test, we know how to integrate and operate and sustain aircraft in the Air Force. It’s what we do.

But software is what’s new. It’s the “C” and CCA, the collaborative combat aircraft. That’s the mission, autonomy, that’s what the difference is. So we’re not talking about building just a new type of aircraft. It’s a new type of team, and that’s a human machine team. And that’s really, I think what we really wanna talk to about today.

So the challenge, and really the opportunity that we’re gonna talk through is really the integration. So how do I integrate this new. Technology with the pilots. How do I integrate them with the maintainers? How do I integrate it with the larger force? And this is one of the few programs where I think it’s gonna hit every one of the letters in the “.mil PFP” acronym.

Ray “Crypto” O’Mara: Yeah.

Mike “Pako” Benitez: And so being thoughtful about all that, and I can’t remember what they all stand for. Doctrine maintainers organization. There’s a whole bunch of that [00:07:00] stuff. But it’s really gonna hit every single one of them. And what does that mean to address them is we have to create an environment to experiment and we’re gonna learn through the experimentation.

So I’m really looking forward. To discussion about the experimental ops unit, what they’re gonna be doing in the coming years. At the end of the day, mission autonomy has to earn the trust of the war fighters to earn a spot in the combat Air Force.

Heather “Lucky” Penney: I’m really glad you, mentioned that. Now for our listeners who are pretty savvy, we’ve been talking about CCA for a long time, when we’re talking about the aircraft, we also know that we need to be maintaining, and doing the logistics and using the aircraft in a different way.

We’re not gonna, if we maintain them and operate them and have the same certification standards as we do for manned aircraft, we’re gonna be on the losing side of that cost equation. But that’s not what we’re here focusing on today. So, Pako I’m really glad you focused on the integration of CCA as part of the human machine team.

Crypto, what are your thoughts?

Ray “Crypto” O’Mara: No, I think, understanding these things in the context of human machine team, like you laid out, is absolutely central. Getting that [00:08:00] right is central for the Air Force to be able to succeed at this because it comes down to, we’re actually not trying to replace humans. We’re trying to augment humans in a way. But we’re using machines to do that. So an important thing is to understand what we expect out of these machines. And I don’t think that’s as easy a question to answer as it seems on the surface, because humans have the ability to adapt, interpret things, and understand that, first, second, third order tasks based on what’s being asked. Like when we are all going to go up and rejoin at a certain point, implied within that is you’re actually gonna get to your airplane, you’re gonna start it, you’re gonna taxi out, you’re gonna take off and you’re gonna do all this without crashing. But, that’s not an integral part of, the quote, “mission autonomy.”

Until it is, because I spent a couple of years at, Tyndall, running the the Air Force, air-to-air Weapons System evaluation program. So I got a lot of experience running complex missions with both manned and, quote [00:09:00] drones. Don’t like the term, but with drones, unmanned things.

Now to be fair, these are not current generation types of things, and they’re operating in a different way, but it’s real, it’s another element of complexity. And it can be pretty hard when you’re dealing with people who are flying airplanes on the ground, with airplanes that are in the air, that, just the coordination and the ability to do things.

The adaptability is not baked in. It has to be built in.

Heather “Lucky” Penney: We’re talking about a day in the life, and so this goes beyond training the agent of the CCA in simulation and doing all of that and then, delivering it to the squadron, we’re talking about, okay, so you’re doing CT ops at the home base, we’re at Base X, we’re going out, we’re flying sorties in the first go, the second go, the third go.

Let’s talk about that kind of training because we’re still need to have the ability to refine and improve how we do that kind of teaming. So, Pako, you’ve been in this space for almost four years before CCAs were even a thing. How does that kind of training differ from the [00:10:00] notion of debriefing our CCA.

Mike “Pako” Benitez: Oh, that’s lot to unpack there.

Heather “Lucky” Penney: Yeah. Yeah.

Mike “Pako” Benitez: Well, I wanna back it up just a second. You bet. And talk about, training in the context of the engineering level, so before it ever hits the flight line, ’cause I think it’s really important to know, just like you wouldn’t go out and use a radar without having like, at least theory, radar theory.

Radar range equation and hey, here’s what the symbology is. And so you need them to have some context about the system that you’re going to integrate and operate with and again, that’s part of the justified confidence. So backing it up again, when you think about training autonomy, there’s.

You’re really talking about developing the algorithms and there’s reinforcement learning techniques that we won’t talk about today. And we use it where it makes sense. But really it’s the VV&A, the verification, validation, and accreditation process of building that mission autonomy before it even gets to the flight line.

And you need the tools to make that happen. and that gives you, that traceability again, for the justified confidence of the system that you’re going to have to use. Right.

Heather “Lucky” Penney: Unravel justified confidence, ’cause I think that’s a very important term for our [00:11:00] listeners.

Mike “Pako” Benitez: So, we’ve also used the term calibrated trust, so justified confidence.

The way that I think of justified confidence is the ability to match my level of confidence in a system that matches its ability to do what I expect of it. So it’s justified, and I have to justify that through, sets and reps, data edge cases and yeah, very important constraints.

And so there’s processes and methodologies to justify that confidence. And there’s entire PhDs that do this in a field called cognitive systems engineering, which I don’t have a PhD in. I don’t know if that’s what your PhD is in Crypto.

Ray “Crypto” O’Mara: It’s related to that.

Mike “Pako” Benitez: So Crypto is probably the guy to talk more about that but justify confidence is one of the enduring themes where the debrief helps that, and it really reinforces that and dials that in.

So, Crypto, what do you, any thoughts on that?

Ray “Crypto” O’Mara: No, I think that’s another key point in as we look at these, because this starts to get to what type of complexity can we expect the machine to be [00:12:00] able to handle? And that’s gonna be, based off of both technical development as we look at the advancement of quote “AI techniques” that are going to be used for these things, but also, the bounds within which we are going to allow the machine to make decisions. And, if we keep the bounds tight, it’s easier to build that level of trust that you’re talking about. Well, I should say it’s I’m not saying it’s easy to do it, it’s easier to do it,

Heather “Lucky” Penney: there you go.

Ray “Crypto” O’Mara: Than if you look at something that is more complex. So as we look at this, we’re gonna need to take a building block approach just like we do when we train humans to what we’re doing with the with the machines as far as when we’re training them.

We do have a lot of experience in training humans and there are a lot of analogs that we can leverage. There’re gonna be some things that’ll be a different but accurately or appropriately bounding what we’re asking the machine to do will help us be able to build a program where we can [00:13:00] develop the trust in what we expect it to do.

Mike “Pako” Benitez: Yeah, on that note, one of the things, we talked about VV&A, so verification is, did I build the thing, right? You have validation, did I build the right thing? And then the accreditation, is it acceptable for use by the end user? Right? Not, not the developer, not the tester, but the end user, right?

So the VV&A process is critically important. And when you look at the tools to be able to do that at Shield AI we have a product called Forge, that’s our autonomy forge if you will. That’s why it’s called Forge. But in that, we use a set of, the test methodologies and the tools we have within that to do that, generate those artifacts and the documentation you’re gonna need to have, traceability to everything from the domain randomization of how you actually do testing at scale. You can find those edge cases and you can tune for those edge cases in a way that really is not many other ways to do it right now. And so that’s one of the key things that we think we can bring, as we move the, mission autonomy forward into the combat Air Force is, that’s one of our unique value proposition as a [00:14:00] company, is not just the experience and the tech, but the way that we can actually test and validate the autonomy we are building.

So when we do get it to the flight line, we have essentially the bounding conditions and constraints. Just like if you have a radar, you know what the specs are, you go read the tech order. It says, Hey, your detection range that this type of RCS should be about this. Here’s the different modes.

And, and you can tell, you can kind of tell what it can do, what it can do. And then if it’s in a degraded mode, it’s gonna put different symbologies and things like that. So we have a way to, to do that from the software perspective and generate those artifacts for traceability and certifiability.

Heather “Lucky” Penney: So Crypto, what else should we be considering here when it comes to the human machine relationship?

Ray “Crypto” O’Mara: I think I would go back to the idea that we should not be thinking about these things in terms of replacing humans. That it is in fact a human machine team that we are in The Air Force aviation’s got a long history of integrating machines into tasks that humans are doing.

I mean, you [00:15:00] look at the first autopilot was put into a Ford C3, C5, Ford Tri Motor in 1927 and we allowed a machine to effectively fly an airplane. Very basic parts of it, but while the machine was doing that, the pilot could do something else. And that’s how we have increased the complexity and the capabilities that the Air Force and airpower brings to supporting, getting combatant commander objectives. In that the more, I will call them routine, they’re getting more and more complex and less, routine, but more routine things we can shift onto machines, the more humans can then shift their attention and the limited cognitive power that humans have, to be able to look at the larger picture and use the machine in a more effective way than you could before. William Lear had an, N+1 kind of thing and pilots will understand this in that there may be 15 things that you can do.

And, the idea of multitasking, which you know, is great to [00:16:00] millennials and that kind of stuff actually means that you do a bunch of things poorly, but when you get to N+1, so if I can do 15 things, when the 16th thing comes along, boom,

Heather “Lucky” Penney: Task saturation, right?

Ray “Crypto” O’Mara: Boom, I’m done. And I can’t do any of the first 15 well. But what if I could make, have a machine do that first five. Now I’m working on number six through 15, and oh, by the way, I can work my way up to 20.

Heather “Lucky” Penney: Well, and actually this is a lot of the value proposition of fifth generation and beyond type cockpits. So you’re no longer manually adjusting your L Strobe and and that’s just fourth gen, right? And you’re laughing Pako ’cause you have a tremendous amount of experience there. But the fifth gen cockpit is able to take all of those sensors and be able to fuse them together in a picture that is far more intuitive for human cognition to interpret than make decisions on. And so that’s something like that Crypto, when you’re talking about N + 1, that we very successfully used machines to create a human machine team in that fifth generation pilot to be far more effective. But let’s go back and let’s assume that some of that .mil [00:17:00] PFP has been done, right?

Because we’re still trying to figure out how are we gonna field CCA? Are they gonna be living with squadrons? Are they gonna be separate? Where are they gonna meet up? But at some point in time, you do have to create that human machine team that meets up in the airspace, goes out and does the JOB, and then comes back and lands and is gonna have to debrief.

So let’s kind of focus on that critical part of any sortie that aircrew experience, which is the debrief. Now, I don’t know about you guys, but I didn’t really love it at times. It’s crucial though, because it makes us better aviators, better pilots, better war fighters. So let’s help explain to our listeners what the debrief is, why it matters.

Crypto, let’s start with you.

Ray “Crypto” O’Mara: Okay. So the ultimate goal of the debrief is after the flight. You come back in, you accomplish the mission. The number one goal of the debrief is to improve future performance. That’s it. Plain and simple. Figure out number one, as from a process standpoint, understand what you did through all phases of the mission.

So have an accurate understanding of [00:18:00] what actually happened throughout the mission. Number two, compare that to what you wanted to do. So, in the brief, you lay out this is what we’re going out to do today. And it’s everything from we’re gonna start at this time. We’re gonna meet up at this point, we’re gonna execute this mission in order to go bomb these things.

Then we’re all gonna rejoin, we’re all gonna come back. So the whole thing from end to end is, comparing what we did to what we wanted to do. Then you look, you start to get into what, the important part is where did the two diverge? So you look at it, you go, if they diverged, if they didn’t, that’s great.

You know, you went out and you executed exactly what you talked about. However,

Mike “Pako” Benitez: which, which I’ll say in the history of the Air Force, there’s never been a perfect sortie.

Heather “Lucky” Penney: Absolutely. In the history of aviation, there has never been, things always happen.

Ray “Crypto” O’Mara: That’s right. But once you identify the places where things diverged, now you want to dig into why they diverged, because there’s a reason for that to have happened.

So, number one, you wanna look at it and determine, was that divergence [00:19:00] appropriate? ’cause sometimes it is. I was told to do this, but because of what was going on, I went and did that. And the learning point there is you, okay, was this covered under the contingencies?

And whoever’s leading a brief tries to lay in, if the normal isn’t happening, here are the things that could happen. They’re outta the ordinary and do this. So that that doesn’t fall into the divergence category. It would be the things that were not accounted for upfront. Now you start to peel back, okay, why did this happen that we didn’t discuss before?

Is it because of a lack of understanding a human make a mistake, lack of understanding. Why did they not not understand it, didn’t have the data necessary to do it?

Heather “Lucky” Penney: Is it perception?

Ray “Crypto” O’Mara: Exactly. Was it they didn’t have the experience to be able to handle things that were happening, knowledge they had never seen it before.

Or were they never trained to be able to do these things? So, and those are just kind of some of the, the blocks there. Know, I talked about that in terms of humans, but I think that we can look at that directly in, how a machine, those are the types of things I’m gonna wanna know from a machine if something diverges.

But, once you identify [00:20:00] those things, now you then peel back and go, okay, so the next time you see this, what is it? We can all learn from these things. Another thing from a divergence is that, what if your plan was bad? So that is, that goes to, yeah, we went out and we would’ve executed exactly what you said, but it was really dumb.

And I’ve been there, you know, been there before.

Mike “Pako” Benitez: Oh yeah, me too.

Ray “Crypto” O’Mara: And but, so now you get the opportunity where I think there is not and this kind of diverts a little bit from what we’re talking about here, but how do we use, “AI” to help us think about the way that we solve problems upfront. How does it shape the way that we think about doing this, which would then shape the way that we think about going out and executing?

Heather “Lucky” Penney: Well, and I’m gonna interrupt you, Crypto, because I think that’s a crucial point, right? I mean, and not just debriefing to what we said we were gonna do, but when we begin to team with CCA, team with machines, we know that they’re gonna behave and they’ve got different value propositions than the human.

We’re going to have to do business differently. So, just like you didn’t fly the Eagle the same way you would’ve flown an F-4 the same way you would’ve flown an F-86, [00:21:00] we are going to have to do things differently. That means tactically, that means our formations, that means roles and responsibilities, every part about that.

So we will need to have some debrief that comes back, and AI should be a component that helps us do that to figure out how do we grow in teams so we can maximize the value proposition of CCA.

Mike “Pako” Benitez: Yeah, absolutely. I think the backing up real quick, the brief is always interesting depending on scenario, you have, 60 to 65 minutes typically in a military aviation brief and that you have to cover a range of things that you don’t have time to cover. And so you kind of fall back to three things standards. So like, if I didn’t say it, it’s standard you all held to a standard, it’s written down somewhere and you’ve trained that there’s an expectation, there’s an expectation management of that standard, and then underlying all that is the trust in your team, right?

Ray “Crypto” O’Mara: Absolutely.

Mike “Pako” Benitez: So you’re able to spend an hour and brief the plan and then go fly the brief, assuming the plan was solid, like you said, which Yeah. It’s, I think all of us have interesting war stories for that, but [00:22:00] when you get back, you know, the debrief, there’s two types of fundamentally, there’s two types of debrief.

When I, and it took me a while to figure this out, I was not the sharpest knife in the drawer, growing up. But I was a, was an OPS instructor, was an FTU instructor, a weapons school grad, and then I was a weapons officer and ops squadron. It took me a while to learn this, but there’s two kinds of debriefs.

One of them is a more structured fundamental debrief where you’re focused on, and the really, the magic trick is perception, decision, and action. Like anything that went wrong is one of those three things. Did you perceive this, right? Yes. Okay. Did you comprehend what was going on to then?

Yes. Did you act accordingly? No. Okay. Well, here’s how you fix it. So it’s very, very structured, fundamental debrief. Interestingly enough, the way that we think about mission autonomy is perception, cognition, and action. It’s almost exactly the same. So it really starts to rhyme when you start looking at the basic fundamentals of debrief.

Ray “Crypto” O’Mara: [00:23:00] Absolutely.

Mike “Pako” Benitez: And we’re gonna learn a lot when we start flying with these things. And right now no one has done it. So we’re all, waiting and seeing we’re all gonna learn a lot. And I’m, like I said, I’m really excited to see in the next few months hold. The other type of debrief, which I think what you’re getting into Crypto is when you get into like your root cause analysis type debrief.

This is a missionized debrief of what actually happened. And that is a different level of complexity because the cognitive ability of a human, where when you go to causal factors and root cause analysis of a mission style debrief, and for those, aviators out there, you know what I’m talking about.

Someone died or someone ran outta the missiles or something, but the causal factor that, actually led to that happen, which made the mission go sideways, actually happened five and a half minutes earlier.

Ray “Crypto” O’Mara: Exactly.

Mike “Pako” Benitez: And there’s like four other conditions at that set in motion to make that happen.

So it takes a lot of experience and experience takes time for an instructor to be able to stand up on the fly in a group at a stadium seating sometimes

Heather “Lucky” Penney: Yeah.

Mike “Pako” Benitez: With a couple hundred people and walk [00:24:00] through that in a crowd of a few hundred people, with just in real time. That is an incredible skill. It’s magic when you see it happen. I’m always impressed when I see a human do it.

Ray “Crypto” O’Mara: No, and I think that there’s a critical piece in there that. So good instructors are the ones who are able to unpack those things. And where I think there’s real opportunity for AI to come in and help with the debrief, whether it’s just on the human side, whether it’s human machine teams that kinda stuff, is the reconstruction of what happened. Because starting with all of this is an accurate understanding of what actually transpired. And if you look at, the evolution of debriefs, and I’m old enough to have lived through flying with airplanes that just had film packs to putting three quarter inch tapes in the belly of the airplane, that could do one of two things. They could film your radar scope or they could film out the HUD. So you could only record certain things to be able to increasingly do it till we put pods on. We could, do more and more.

And all of this helped. The aviators at the [00:25:00] end reconstruct what happened. And part of the art was, alright, what actually did happen out there? And people drawing on boards with

Mike “Pako” Benitez: chalk, chalkboards,

Ray “Crypto” O’Mara: chalkboards

Mike “Pako” Benitez: back to begin with

Heather “Lucky” Penney: Yeah

Mike “Pako” Benitez: absolutely chalkboards. Yeah.

Ray “Crypto” O’Mara: Because whiteboards didn’t exist.

Yes. But there is a real, there’s a real skill in being able to, construct what happened accurately, but not for an end in itself. It’s so that we can get to the point we’re doing this. So if we look at that and that over time there was a lot of resistance in certain corners to incorporating things like pods that could lay things up because oh, if you can’t draw it, you can’t understand it.

And there’s an element I think of that that may have some truth. However, it takes a long time to do this stuff. And it kind of goes back to your point. I, you go out and do a four V four, four V eight or whatever, you’re looking at a 12 hour plus day. And by the time you get to the really important stuff, you’ve got up early,

Heather “Lucky” Penney: You’re at the very end of the day.

Ray “Crypto” O’Mara: You haven’t eaten well.

Mike “Pako” Benitez: Yeah.

Ray “Crypto” O’Mara: You’re tired because you flew. Now you’re looking at this stuff. Your ability to learn.

Mike “Pako” Benitez: Yeah, you’ve got crew rest, everyone’s [00:26:00] gotta go tomorrow.

Ray “Crypto” O’Mara: Exactly. So your ability to learn is going to be compressed.

Mike “Pako” Benitez: Yeah.

Ray “Crypto” O’Mara: If I could have something that can actually give me a picture quickly, what happened, who’s doing what to whatever degree of detail I want.

If I could have an idea of where somebody has their radar looking at any given time, or even better where their eyes are looking at any given time, I’ve got a lot more data now that I can get more quickly to that, through the root cross analysis process to be able to say, Hey, you were looking left, you should have been looking right and this is, these is the things that would’ve told you.

Got it?

Mike “Pako” Benitez: Yep.

Ray “Crypto” O’Mara: Got it. Good. Now let’s move to the next thing.

Mike “Pako” Benitez: Yeah.

Ray “Crypto” O’Mara: As opposed to, do you remember what you were doing?

Mike “Pako” Benitez: Yep.

Ray “Crypto” O’Mara: That’s, so I think there’s real opportunity there for us. And this is the same type of stuff if we look at it from that perspective. When we start looking at machines, CCAs, whatever, in this, we need to be able to.

Provide that input and get that understanding from what the machine is doing and what we would want it to do in the future.

Heather “Lucky” Penney: I’m gonna be a little bit of a troctolite here. ’cause I think the answer is a, is both end.

Mike “Pako” Benitez: What does that mean? That’s a $5 word.

Heather “Lucky” Penney: It’s like before the [00:27:00] dinosaurs. Right?

Mike “Pako” Benitez: Okay. Okay.

Heather “Lucky” Penney: Way before the dinosaurs. Because I do think there is still value in training the human cognition to be able to do that kind of peel back operation both real time when the flight’s happening. And I think you only get that through that process of reconstruction. So I wonder, what extent that when we just hand that reconstruction, ready made with all of the interest points and all the activities, are we diminishing our ability to train humans?

Because one thing that has really impressed me about the Air Force is the quality of our weapons officers, the quality of our instructor pilots. And their ability to get to that high level lessons. And we do this consistently. We’ve got a lot of them. It’s not like it’s just magic when one of those guys pops up, we train for that.

And so that’s just one of my concerns where I remain open-minded because I’ll tell you what I mean, I had the PFPS debriefing system. And boy did that accelerate and improve our lessons learned. But we still went through a process of teaching people how to draw circles because that was [00:28:00] deemed an important element of training the human cognition and their ability to peel that back.

And then we accelerated in. But Pako- I’m sorry, go ahead.

Mike “Pako” Benitez: No, I just go ahead. Crypto.

Ray “Crypto” O’Mara: I was say on that front, you just need to let evidence lead the way on it, because

Mike “Pako” Benitez: Yeah.

Heather “Lucky” Penney: Yeah.

Ray “Crypto” O’Mara: It is,

Heather “Lucky” Penney: I’m not saying it’s not possible. I just wanna make sure we don’t lose that element.

Ray “Crypto” O’Mara: Well, it’s an important thing.

It goes to I think when you get into the pilot training thing what are the important things just because we’re doing something. Does not necessarily mean that it’s causal in the past. Absolutely. But if we give people new tools to do things, it’s amazing what they can learn and what they can do with those tools that you didn’t necessarily expect and really get into it.

And I don’t expect machines to be identifying, root cause things right?

Heather “Lucky” Penney: Maybe they will at some point.

Mike “Pako” Benitez: We can talk about that in a little bit.

Ray “Crypto” O’Mara: Yeah, no, at some point potentially. But just this idea of where are the opportunities because this is a path that we are, have been on for a long time and we, I think are going to continue to [00:29:00] be on trying to jump straight to Act four when we’re in the middle of act two. So.

Heather “Lucky” Penney: Yeah, because I mean, the point of the debrief isn’t for the art of the debrief. The point of the debrief is to ensure that when you’re airborne, the next time you recognize those errors before they occur so you can execute appropriately.

Mike “Pako” Benitez: Like, I’ve seen this before. Don’t do that again.

Heather “Lucky” Penney: Yeah, you know, it’s the next time I will kind of thing. So, Pako, as a weapons officer with the debriefs and the Strike Eagle, it was a crew concept. so we’re talking about human machine teaming. You worked in a human- human teaming operation.

What about those dynamics shaped the process?

Mike “Pako” Benitez: Oh, boy, that’s a loaded question. No, that’s an interesting question. And, I really didn’t appreciate it at the time until I think you and I, we’ve been, you’ve been talk, we’ve been talking for years. Oh yeah. And you and I had a couple conversations and it really, you had asked me something like that, offline, a while back, and it really got me thinking.

I was like, oh. Never really thought about it that way. You’re right. So I I did put some thought into it now. Thanks to you. Thanks to you. a Strike eagle, it’s, it’s got two engines, two tails, got two [00:30:00] cockpits. And those, those two cockpits of two humans, those two humans have two sets of eyes, two sets of ears, two brains, and then two sets of, you know, hands and feet that are doing things.

And so the magic is we went do a mission. You go plan brief fly, and you come back and you debrief the mission. Now you have to fuse. Those two cognitive realities together or where they are fused and magic happens, like synergy one plus one equals three and it’s awesome when it happens or when, like they start to diverge a little bit and then not so many good things happen, right?

So it’s has a synergistic additive effect, but it’s also risks going the other way.

Heather “Lucky” Penney: Yeah.

Mike “Pako” Benitez: Where if like, man, if this was a single seat, this wouldn’t have happened. Like yeah, you’re right. But if you can get these things dialed in, like you can do a lot of really, really compelling things. And I remember, you know, back in the, in 2000, 2007, 2008 timeframe before, when Raptor was just coming through OT and, you go out to Nellis and Red Flag and some of those training events and the Raptor would show up [00:31:00] and he would have SA and Raptor has SA was kind of the big thing at the time and and as these scenarios got more and more just cognitively complex and the amount of just things going on that you have to task, manage and keep track of the cognitive burden of being able to offload that and task allocate in a two crew concept allowed the F-15E to do things in certain dynamic environments that a level of complexity that at fourth gen systems, it just couldn’t really do.

It was Human brains overcoming the fact that the way the system is set up is set up now, as you said before, the fifth gen moves a lot of that stuff from in front of the glass to behind the glass, which then frees up, you know, I’ve got 15 things to do that Crypto said, Hey, now I can focus on Yeah.

Heather “Lucky” Penney: The N+1, right.

Mike “Pako” Benitez: I can focus on those other things. And so the fifth gen integration is really where you start to see that shift, that value proposition of like, if software can actually like, reduce my cognitive burden to be able to do this, some really good, vignettes about a lieutenant flying an F-35 at red flag, being able to hold his own.

Heather “Lucky” Penney: Yeah.

Mike “Pako” Benitez: There, I mean there’s a lot of [00:32:00] truth to that. There really is like there is evidence if you go to the debrief, pulling a thread on the other part of it, I’m old enough to remember when we had tapes in the Strike Eagle and we had two screens, the whole thing. You had recorded two screens and it was a little eight mill tape.

When we moved to a digital debrief. You could see all the screens in the cockpit, both front and back all at the same time. So that’s four in the back and then three plus the head in the front, that’s eight screens. So when we’re debriefing a four vs. four DCA you just do the math. That’s 32 screens of data that’s going on and people are like, man, our debriefs are gonna take forever.

We have so much more data. Like that’s not actually true. Now you know where to look at what screen when and what happened, and so you can actually go through a debrief and so much more. Efficient manner when we move. And it took us a while to get there, but you know where to pull up the tapes, where to pull up the screen and go, okay, you’re seeing this at this time. And then did you see the symbology? Did you hear that radio call? So you can actually go back to the perception, decision, action. [00:33:00] So I was really, really happy to live through that. I was not happy to live through chalking up things. I was not a very good line draw. And I remember the last thing I have to draw for an upgrade ever was a TV six ACM fight.

Heather “Lucky” Penney: Oh man.

Mike “Pako” Benitez: At weapons school for a CM ride. It was painful. It was painful. But the ability to reconstruct that is a skill that is a learned skill. But it’s, the purpose of being able to do that is to validate your learning, not necessarily the debrief. Right. So,

Heather “Lucky” Penney: Yeah. Yeah.

Mike “Pako” Benitez: So the right tools for the right job.

Heather “Lucky” Penney: Yeah, I totally agree. Like I said, it’s not for the purpose of the art of the debrief.

Mike “Pako” Benitez: Yeah.

Heather “Lucky” Penney: It’s for training your mind for when you go fly the next time that you can anticipate because you’re like, I’ve seen this line before.

Ray “Crypto” O’Mara: Yep.

Heather “Lucky” Penney: And it did not end well. So speaking of that what about the debrief made you fundamentally better?

Because we all have memories and some of them are better than others. Generally we learned the most from the hard ones, the things that did not go well. When we look at the future of debrief, especially with these human machine teams, what should we seek to preserve? How should we begin to evolve this?

’cause, unlike the screens of [00:34:00] fifth gen when we’re operating with CCA, those are gonna be other entities out there in the airspace operating dynamically. What are the things that we need to think about preserving when we go forward?

Mike “Pako” Benitez: Well, I think we need to start small.

Ray “Crypto” O’Mara: Amen.

Mike “Pako” Benitez: Start small in a nice confined learning sandbox. And then we’re going to iterate. And fold that in. And then you increase the complexity and the variables that are in play. If we try to just, Hey, you know, CCA won first flights tomorrow, well then on Friday I wanna throw it in red flag.

That’s, that is not, that’s not what to do. So there, there is a process that I think the balance is going to be how do we speed to field these things, but also manage the time it’s going to take to do the learning that we’re gonna need through experimentation to then actually figure out the best way to do the integration like we talked about before.

But also, what are the enablers [00:35:00] to en to for that integration. So I’m gonna need some way to debrief to figure out what happened, so then I can go plan and fly these things and then expand the way that we integrate. So I think it’s gonna start at a, it sounds trivial, but the admin, tech admin basic stuff.

Heather “Lucky” Penney: Oh yeah.

Mike “Pako” Benitez: Let’s debrief that with the autonomy and then we can start adding and it know maybe one pilot, one CCA, and then, okay, let’s do the same thing with one pilot, two CCAs. And so your test and experimentation campaign plans. There’s people that are thinking about that right now.

It’s like, when these aircraft show up, what’s the best way that we’re gonna actually build a test plan and do that?

Ray “Crypto” O’Mara: No, I think that’s the basic blocks because all this is about learning, right? So I think the key question is who needs to learn from these debriefs. And this actually drives to, you know, we’ve been talking about human machine teams and you think about, okay, I’ve got an aircraft, I’ve got an air crew in it and I’ve got this CCA and there’s human machine team there.

The CCA also represents another human machine team. Because there is an operator of that CCA somewhere. At [00:36:00] some point they turn it over and it goes, but when we start to look at you talk about, debriefing the admin, everything from your ground ops all the way through landing and because when you land can have an impact on when the next authorities can happen.

And if you’re in some kind of combat rhythm, that’s important.

Heather “Lucky” Penney: And I think that the concept right, is that that the pilot or the flight lead is gonna be the quarterback of that CCA.

Ray “Crypto” O’Mara: Sure. But you’re also looking at this and I say sure somewhat tentatively ’cause. I, I’m not, I don’t, I think until we figure out the best way to operate these things that the answer to that remains open. But the bottom line is when you look at a sortie outside of just from push to knock it off, that everything, there are other people and entities involved in this now that are absolutely central to the successful execution of the mission. So you look at it and say, “okay, who needs to learn what?” When I look at the CCA, absent the mission execution piece and the admin stuff, it may be talking [00:37:00] about what somebody on the ground is doing and learning and how they need to interface with the rest of the human machine team. Or is it what is going on inside the brain of the CCA that I need to get to, which adds another level of complexity. How do you teach something so that on the next sort, if I teach you something tomorrow, I can feel pretty confident that. You’re gonna go out and, based on my experience with you, that you will, you’ll internalize that to a certain extent and you maybe won’t make that mistake again. Do we have that actual opportunity? ’cause it goes back to your VV&A discussion earlier. Can we do this in a quick way, in an accurate way as we do learn these things?

Mike “Pako” Benitez: And I think the test environment is I flew operational tests my last few years and, there’s a, you have these things flying in a, an environment where they have different types of software loads that allow you the freedom to do that VV&A iteration regression testing, bug fixing before you lock it down into a p-tape, a production tape [00:38:00] to actually send to the fleet. So I think there’s a lot of learning to be had outside of experimentation, but just a DT and OT process that, again, we’re all gonna learn a lot, I think in the next six months about what do we think? A lot of people have great ideas, a lot of great people with great ideas, but until we, we get hands on stuff, when we actually get data points, it’s mostly theory right now.

Ray “Crypto” O’Mara: Absolutely. No, and a decent amount of experience in the OT world and debriefs there are different than in an operational squadron.

Mike “Pako” Benitez: Yep.

Ray “Crypto” O’Mara: And, you know, that’s what we’re actually talking about. So the learning that happens in an operational squadron, is in the humans themselves.

And it, it does happen, very, you would hope more rapidly, but the range that you’re dealing with in divergence of experiences is less than in the test world where you’ve got a lot more, there’s a lot more that can be unknown as you go into it. So it’s going to be interesting.

The who needs to be in the debrief, I think is gonna change. And for old dinosaurs, like me [00:39:00] who, you know, I had the luxury of flying one airplane my entire career. There was nobody else in the airplane. We were, in fact, a team though, I mean as a flight we were 100% a team if we wanted to be successful.

But, there are a lot of myths and archetypes of, the single seat fighter pilot. And they are there because they’re largely true. there’s ego involved and the, so there’s gonna have to be a social evolvement of air crews and ground crews.

As we look at integrating, these things in to become overall more effective. And the key is understanding this is a human machine team, but understanding what that team actually is made up of.

Heather “Lucky” Penney: Yeah. So I think, you know that clearly the, the DOT&E is an important piece of maturing this capability, but if we work from the target backwards, let’s look at operationalizing this.

Let’s imagine that CCA are actually on, they’re at the squadron. They’re on the flight line, and it’s an average squadron where 50% of the kids are inexperienced. You know, you got your weapons officers, [00:40:00] your patches and all of that. Because there will still be value to debriefing the human as part of the human machine team.

Ray “Crypto” O’Mara: Absolutely.

Heather “Lucky” Penney: And debriefing the CCA as part of the human machine team, especially when we go into combat environments where we’re cut off from any data centers, anything like that, right? We’re operating in a distributed manner, forward edge as a team and experiencing the uncertainties, the unknowns, the surprises that occur, the war reserve modes that occur when we’re actually out in combat.

So that’s a far more stressing environment than you’re just your average. Oh, shoot, we didn’t expect that. That’s gonna be an audible, this is something we didn’t brief in the contingencies section. We still need to be able to debrief the CCA to continue to enhance, because I think that is gonna be a key component of providing us an edge in that human machine piece, especially when we go to combat, where we can then refine and share and promulgate those experiences across the entire fleet of [00:41:00] CCAs for that particular environment, just like we do with data libraries for EW and things like that.

We should be able to do that for CCAs as well. So what does that debrief look like?

Mike “Pako” Benitez: Ooh, that’s a lot to unpack there.

Heather “Lucky” Penney: Yeah,

Mike “Pako” Benitez: The. The first thing I’ll say is that we’ve been thinking about it, Shield AI for a while. And again, we’ll continue to learn and evolve as it goes. We’ve kind of indexed on three tiers of debrief, if you will.

Heather “Lucky” Penney: Okay.

Mike “Pako” Benitez: One of them, kind of talked about before with Hivemind Forge. That’s our engineering level debrief. You can get your VV&A and that’s gonna be important later. I’ll come back to it. Then you have, you know, all the way at the end, the operator debrief. There’s a product, and we could, we’ll talk about it a little bit later, but a product that we have to accelerate debrief.

Really we’re targeting pilot training the human, but it does have applicability for CCA as well. So there’s a lot of there’s a lot of goodness you had as cross pollination of that. That product.

Heather “Lucky” Penney: Very cool.

Mike “Pako” Benitez: We can talk about it later, but in the middle there’s this in between. Like, I need something more than a normal like ICADs type debrief.

But I don’t need like hardcore engineering logs to go [00:42:00] look at what does that middle look like, where I just need to get a little bit of inference of like, what happened, when did it see, when did it see it? So we have a product that we’ve made for that that’s, in experimentation right now.

And it allows you to have inference over things like, I’ve got a four ship of CCAs, for example. And, one does number three, get this piece of information from number one, and instead actually you can analyze data packets, message traffic, and you can actually start using that to frame out what did it know?

When did it know what did it do? Why did it do it? And you can actually bring up behavior trees and go, here’s where it was. And it’s in, in the cognition core. This is where it jumped to this behavior tree, which is what caused this line of thought, which is why it led to this action.

And so it’s a way that you can create, again, not necessarily. For testing, but for actually analyzing what happened. And probably in the near future, what you’re gonna see is something where you have some kind of field support rep in your debrief that’s like, Hey you know, maybe he’s the Spanish to English translator.

It’s like, Hey, [00:43:00] you’re gonna be CCA one in the corner here. When I ask, Hey, CC one, did you see this? And he is like, hold on a minute. Yes, I saw this, this is what I perceived at this time. And so I think there’s gonna be a little bit of that as we start to go down this path and develop the lexicon and the vernacular and really the skills to be able to articulate how we want to get the information out of the machine.

Heather “Lucky” Penney: That’s actually really interesting ’cause that opens up the potential for CCA behavior to be explainable and therefore teachable. Not only teaching humans, but also retraining or improving the performance of the CCA. So I think, like you said, having the contractor support there, or even if this becomes a new, specialty code within the Air Force is gonna be crucial.

Ray “Crypto” O’Mara: You see, and that’s, and I’m glad you lay that out that way because that’s really what I mean by who needs to learn what from this, earlier, I think you gave a little more credence to, some people who fly airplanes than they deserve.

When you talk about, you brought up the radar range equation, at one point.

Heather “Lucky” Penney: Because we’re gonna be doing that kind of public math in.

Ray “Crypto” O’Mara: Well, at one point I [00:44:00] was pretty pretty smart on the F 15 radar. And I flew with guys who weren’t, and to be honest, 99.98% of the time, it didn’t matter. I mean, the things that I really enjoyed about, excuse me understanding how the radar worked and on the impacts other people didn’t.

And I mean, and they. I wouldn’t say it out loud, but they might have been be better BFMRs than me. Nobody was a better BFMR than me. Of course. But this idea.

Heather “Lucky” Penney: Of I spoken like true fighter pilot.

Ray “Crypto” O’Mara: Exactly.

Mike “Pako” Benitez: Yeah.

Ray “Crypto” O’Mara: But this idea of how do we teach the CCAs to get better, you’re not gonna have Yeah.

I don’t want the air crew to be the guys in there dissecting and rebuilding the behavior trees. No, because I think that is a skill and there’s knowledge base that is required that would take away from being focus. Flying an airplane is hard. Yeah. Flying airplane, employing airplanes is hard.

We don’t need to make it hard. So I want the right people to focus on that. I want the right people to focus on this other thing, but they are an integral part of this team now. So the purchasing, I think the nearest analog that I came, I [00:45:00] could come up with on this is, you know, when we started flying the F-117 because of the stealth characteristics, the air aircraft, and the importance of presenting particular aspects to known threats

Heather “Lucky” Penney: To minimize the signature.

Ray “Crypto” O’Mara: We had, we actually brought EWOs into these are single seat aircraft in squadrons. Bringing EWOs into sit and do the mission planning.

Heather “Lucky” Penney: Mission planning,

Ray “Crypto” O’Mara: because it was so complex and important that, it was hard enough to fly that airplane that you’re not having the pilots necessarily do it.

You’re bringing experts in training to do it. And they were an integral part of this, what the success of the mission was. Very much contingent on not only pilot performance, but what the mission, what do we call them, but what the EWO did. When they put the thing together. So that’s the nearest analog that I can come with this, which I think is, so we’ve done a lot of things in the past that we can leverage as we look at doing this. It’s not, it doesn’t all have to be new.

Mike “Pako” Benitez: And I want to, I’m wanna pull a thread on [00:46:00] that. You know, not everything is, you know, just because it’s new doesn’t mean it’s new new, right? There’s a lot of,

Heather “Lucky” Penney: yeah,

Mike “Pako” Benitez: there’s a lot of the past that we can leverage into the future.

One of the closing the loop on the three tier debrief that we talked about. Yeah, so we finished this debrief and hey, feedback, you know, CCA one in the future makes you do this. Hey, here’s an edge case that we didn’t account for. Like, okay, great. Well now that field support rep, using, a kind of an analogy of the crowdsource flight data program.

Heather “Lucky” Penney: Yeah.

Mike “Pako” Benitez: At the Air Force started about eight or so years ago. You can feed that all of that fleet data back into a lab. You can ingest the data and then you can feed that back into your weights and your models back into the very, you know, back in the very beginning I talked about your engineering level back in your VVA loop. And if you, if you set a software release cadence using DevSecOps that has a CICD pipeline or something like that, where hey, every dude, every three weeks or every six weeks, you get a tape update.

Heather “Lucky” Penney: Yep.

Mike “Pako” Benitez: So if you debrief that, yesterday, dude, today, that’s still gonna happen.

’cause the tape hasn’t been updated yet. [00:47:00] Right? So there’s all these interesting nuances again about when we integrate with the force, there’s so much that has to happen to make this real. But it’s never been a more exciting time to, to see all this kind of come together.

Heather “Lucky” Penney: Well, you know, it’s really interesting.

I’m really glad you brought up like the DevSecOps and what it’s gonna need to do and going from that middle tier debrief back to the engineering level debrief, because I think there’s maybe a misperception that I’m gonna debrief it and then the next day the CCA is gonna be able to go out and learn from its mistake the day before.

And that’s just simply not the case. So whether or not we use today’s systems and processes to be able to release those tape updates, whether or not it’s standard, whether or not it’s an emergency release, whether or not it’s, you know, like a pacer wear back in the day.

Mike “Pako” Benitez: Oh yeah.

Heather “Lucky” Penney: We need to be able to handle that, to be able to address that.

But the other value of that is that then it proliferates across all of the CCA so you’re able to uplevel the standard of performance of all the CCA as different units are continuing to debrief, and then that all gets integrated. And so that’s something to me [00:48:00] that’s very exciting because you’re able to uplevel that performance and those mission outcomes.

Ray “Crypto” O’Mara: One of the things we’ll need to think about as we do that though, is when we would come out with a new OFP, radar tape, I was part of the team that would go around and do a roadshow.

Heather “Lucky” Penney: Yeah.

Ray “Crypto” O’Mara: And like, okay, this is what’s changed. This is what you got. And you know, these are things that happened in at best once a year because of the difficulty of doing this stuff.

However it kind of goes in the same thing. So when a new software load comes out of the the machine and goes into the CCAs. You now have an entire fleet of humans that have to be updated too, who need to be educated on this. And there’s gonna have to be a balance that in improving performance of the CCA and actually not decreasing the performance of the overarching system.

Heather “Lucky” Penney: Because every time we got a tape, we were having to learn the new technology.

Ray “Crypto” O’Mara: Exactly.

Heather “Lucky” Penney: And there was always a Yeah. Yeah. That’s a really good sort of social element that, that’s important to bring up. So, but I wanted Pako go back to the three [00:49:00] tiers of debrief and get to the operational piece. ’cause I also wanna talk about how AI can improve the human level of performance.

How might we apply AI to that process? How can we improve humans’ performance?

Mike “Pako” Benitez: I think it’s, stating the obvious. There’s a pilot crisis. It’s going on over a decade. The numbers haven’t fundamentally changed. Then the Air Force isn’t the only one with a crisis. The Navy has a pilot crisis and a crisis of production, absorption, experiencing, and then retention.

Heather “Lucky” Penney: Yeah.

Mike “Pako” Benitez: And if you were try to attack the whole thing is just, you know, it’s a wicked problem. But if you go all the way back to the beginning of like production, there’s the 1500 pilot production goal that the Air Force has had for a long time. It’s getting closer, but still not able to meet it. When you think about it, there’s, and here, and it’s a long wind up for the soft pitch.

There’s, call it 1300 pilots a year. And if you do the math, they get about 180 to 200 hours of training. That’s about 250 to 300,000 flight [00:50:00] hours a year. That are all just go thrown away. Like the data, there’s no data that gets captured and tracked and managed. And so what you end up with, if you go back to like UPT and pilot training next to where it started

Then it went to, UPT 2.5, and now we have 3.0. The fatal, I would, I would say the, the blind spot in that entire effort is that you basically ignored the principles of the scientific method. You started making changes to see, inputs versus outputs, but you didn’t have a control group.

And you didn’t have a control group because there’s no way to do objective based. Analytics of a human doing basic pilot training. It’s all subjective.

Heather “Lucky” Penney: Yeah. Because you’ve got an instructor pilot going out there and yeah, there are parameters like plus or minus a hundred feet plus or minus 10 knots plus or minus 10 degrees on heading and things like that.

So that’s somewhat objective, but in the end, the grade sheets that you get, whether or not that’s unsat good, excellent, outstanding, all that sort of stuff is still subjective.

Mike “Pako” Benitez: That’s right. And so what we have a product called Benchmark, [00:51:00] and it’s derived from some of the stuff we talked about earlier, but what it’ll, and we’re focusing it on undergraduate pilot training because it’s a very constrained, confined, syllabus oriented, event driven thing is like, what if you can move.

What is manual and subjective to something that is more dynamic and objective and it scales. And so think about a, our product. Think about a tool that you can go fly a basic syllabus sort at UPT and a T-6. You can come back, you can download your puck or your TIPC data, whatever you have.

And then you push one button and then 10 seconds it goes, here was your flight, here are all the different maneuvers that you did. And then here’s I am scoring how you. Did those maneuvers, nominal, off nominal. And that’s all based on the algorithms detecting and scoring, but also waiting by human ips going, this is what a good maneuver looks like.

Here’s what a bad one looks like. And you can actually accelerate the debrief to get to those highlighted, teachable moments where now the instructor, instead of hunting and pecking through a tape and go, Hey, here’s the three things, yada, yada, yada. And you kind of accelerate it [00:52:00] that way. And that lets you do a whole bunch of interesting things.

If you could move to that paradigm. It allows you to have control groups. I can track a pilot over time throughout his course, how he’s pre progressing so I can move to a competency based training curriculum, which we don’t have today. It’s event based. If I have a control group of a class, I can change and insert technology in another class and I can see are they actually, is it matter?

Can I get an ROI can, I actually justify that and then I can do I can do groups over time and eventually you could do it through, call it 1300 pilots a year from the day they show up to the day they get winged. And you can actually manage the entire ecosystem of pilot production with data on a dashboard.

You can pull it up at any time and you can compare who is actually better than who. And obviously there’s a human component that you subjectively weigh in, but from the objective data point. None of that. Everything I explained, none of it exists in that’s being used right now. And the unfortunate part is it could all, all of these products exist on the market.

And you, we were talking offline, you flew last week, you fly all the [00:53:00] time. Lucky. Like you use some of this kind of stuff.

Heather “Lucky” Penney: Yeah. I use this stuff, you know, the products, whether or not it’s ForeFlight and Cloud Ahoy and how they integrate, like I can actually score my performance. So if I haven’t flown for a while and wanna go do a proficiency flight and just beat up the pattern, I can turn those suckers on and actually see all of my data and all of my parameters and I will actually get scored as well.

Ray “Crypto” O’Mara: Yeah.

Heather “Lucky” Penney: You know, so this is something that, that the Air Force, I think definitely could leverage to improve the outcomes and create more standardized, behaviors and assessments for their pilots. Crypto from a socio technological standpoint. I mean, that’s part of what you’ve really focused on.

What are your thoughts here? What’s the potential benefits or LIMFACs?

Ray “Crypto” O’Mara: So, I think what you laid out is really, there can be tremendous benefit there. The one thing that I think is that we need to, so what is it that we should be having the AI evaluating? In other words, what tasks should the human be doing when they’re in the air? Because flying airplanes is expensive, right? So [00:54:00] what is it? And you know, there is anybody who’s flow an airplane knows that it is fundamentally different walking out, getting in, starting it on the flight line than it is in a simulator.

Heather “Lucky” Penney: Yeah.

Ray “Crypto” O’Mara: Okay. There are certain things, but what is it that’s different?

Heather “Lucky” Penney: My adrenaline level, right? Like stretch yourself into the jet.

Ray “Crypto” O’Mara: So human performance. Absolutely.

Heather “Lucky” Penney: Yeah.

Ray “Crypto” O’Mara: Why? Because there’s more threat. There’s a, once I take off I might crash and die. Okay? So don’t do that, number one, but the bottom line is, you know, what are the things that you can only experience in the air, the things that you experience that we need you to get better at so that you’re not wasting your time on. Things that you actually could do elsewhere. So you act, you maximize the amount of the amount of learning po potential in the things that cost the most, which is when you’re in the air.

And this, back in the 2008 ish timeframe when the predator fleet was second only to the F-16 fleet in [00:55:00] the Air Force there was this, tremendous thrash the Air Force went through about, you know, how do we train RPA pilots? And there were people like, oh, gotta go through pilot training.

Oh, well, do they like, what was true was that the things you needed to learn in order to employ an RPA, you learned by going through pilot training. However, was it only 30% of the things that you needed to learn in pilot training? We couldn’t go through the process of saying, alright, this is what an RPA pilot needs to do.

And there were tremendous, that was a social problem, that was plain and simple. That was nothing other than a, a social issue the Air Force was dealing with. So, I think that how we apply these things, and, you know, I was big on the whole, you know, measurement and capture thing.

Absolutely. And being able to get to, the best feedback possible to the human so that they can learn and improve the next time around. But also using this [00:56:00] larger, come from a larger perspective of making sure that we’re doing this at the right place. I live through. Again, being old F-15 simulators were actual cockpit mockups, but they were by themselves and the hydraulics for moving, it was all disabled because the stuff always broke.

And so you could do this single, you know, single seat, single airplane type of stuff to, you know, getting to WTT weapon tax trainers that were computer screens to then clamshell simulators to DMO kind of stuff. A lot of people probably have no idea what some of that stuff is, but but the bottom line was we were able to increase the learning opportunity by increasing the fidelity of what the pilots and the air crews were able to do in the simulators so that when you went out there and the weather was garbage or you didn’t feel well. You actually weren’t worried about the switch allergy stuff as much. You were worried about not throwing up.

Heather “Lucky” Penney: Yeah. And that’s what I mean. Simulators have progressed from just being tech, procedural trainers and now they’ve done tactical training and that, but there’s still that element of being airborne [00:57:00] that, you really just have to experience to be able to validate your capabilities when you get airborne.

Mike “Pako” Benitez: Yeah.

Heather “Lucky” Penney: And I think that’s one of the things that’s interesting about that, what the system that you’re talking about is measuring perhaps the difference between, ’cause you can probably have that both in a simulator.

Mike “Pako” Benitez: Yep.

Heather “Lucky” Penney: And airborne,

Mike “Pako” Benitez: So you can do sim and in the aircraft.

Heather “Lucky” Penney: Yeah.

Mike “Pako” Benitez: So you can actually look when you go from the sim to the flight line

You can actually see the difference in the delta, your drop off. Because again, you’re adding in this other dynamics of the human condition.

Ray “Crypto” O’Mara: Yep.

Heather “Lucky” Penney: Which could be interesting to then measure, okay, what’s the right ratio between simulation and flying that we need to have.

Mike “Pako” Benitez: That’s right. That’s right.

Heather “Lucky” Penney: So getting back to your scientific, evaluation of how we change things and why are we changing them and are they proving at our hypothesis for changing the way we do business? So what else could this system of evaluation and having quantitative metrics help us do in terms of generating pilots?

You mentioned a little bit about competency based training so you could profit advance, so proficiency, advance, [00:58:00] and move on to the next stage if you’ve demonstrated a certain level of consistency of scores and things like that. But what’s the ultimate benefit?

Ray “Crypto” O’Mara: Ooh. I’d say that there’s a few, depending on, on how you quantify it, you can accelerate the production of, of individual pilots, right?

For competency based training, ultimately you want to produce more pilots, you wanna do some more effectively and more efficiently. So I think we all acknowledge that, the biggest lever, cost, lever and speed is to just reduce the amount of flying, which I think all of us are like, well, that’s a slippery slope. It really depends so I wouldn’t go after that. But what I would say is optimizing the debrief and then can I? And this is where it gets a little second order effect. Can I optimize the instructor cadre that exists? And if they, if I can optimize the instructor cadre in the syllabus, then I can optimize the syllabus.

Does that mean that my demand signal of ops instructors going back to teach at the FTU can be decreased and I can leverage more FAPES and I can keep [00:59:00] those ops instructors in the ops squadrons and actually help with maybe the FTUs so now I have an absorption problem that I’m addressing.

So by by attacking the production problem this way, you are actually opening the door for solving. The next problem they’re gonna have is if I produce 1500 pilots, they have nowhere to go.

Heather “Lucky” Penney: Yeah. So we don’t have enough IP. I used to fly with them and, but there’s still like the social element where having an OPS instructor at the FTUs is crucially important.

Ray “Crypto” O’Mara: Oh, yeah.

Heather “Lucky” Penney: Because it shapes the. Shapes the culture, shapes the perspective. I mean, then having just a FAPE whose only experience has ever been pilot training. Now I will say some of the best fighter pilots I’ve known have come from the FAPEs. Yeah. Because of everything, you know, everything that they’ve been through, they were just stinking awesome.

But having that ops instructor go back to the FTUs could be really important.

Ray “Crypto” O’Mara: But even if you could free up, you know, and I don’t have the numbers in front of me right now, but if you invested in some software and you could quantify that, you could free up 20 instructor pilots in the Air Force to stay in the FTU, the op squadron and not have to go and teach because you’ve optimized [01:00:00] that.

That’s a pretty good ROI.

Heather “Lucky” Penney: Yeah, that’s,

Ray “Crypto” O’Mara: I mean, 20 pilots is a lot right now.

Heather “Lucky” Penney: Yeah. No, it is a lot. And then also gives you that quantitative basis for understanding exactly how much you’re gonna change that equation.

Mike “Pako” Benitez: Yeah.

Heather “Lucky” Penney: So gentlemen, where do you see things evolving in the future? And I’m curious whether or not we’re talking about pilot training and using, AI for debriefing syllabus, sorties, or whether or not we’re using AI to debrief pilots in the operational squadrons or debriefing the CCA or even debriefing the team together.

How do you see this happening in five and 10 years?

Ray “Crypto” O’Mara: So what I’ll say up front is there’s no such thing as technological determinism. There is no, there’s no path. We are not on a path that we cannot divert from. With that said the Air Force itself has a century of incorporating machines into the missions that it’s doing and shifting the burden of executing parts of those missions onto machines as they have improved. We’ve done it with [01:01:00] missiles, we’ve done it with autopilots, we’ve done all this kind of stuff. So we have a century’s worth of experience that we can leverage now as we look at the new opportunity. So the two things that I think are important, number one is recognizing that opportunity and recognizing what we’ve done.

’cause pivot off to something you said earlier just because we do something now doesn’t mean it’s wrong. You know, there’s a reason that we do the things that we do. Yeah. Now, just because we do something now doesn’t mean that it’s right either. You know, look, you go back at the root of reason for why we do something, and then you look at the tools that are now available and you say, you know what?

I can use this to help me do that. I don’t need to tear down the house and rebuild it. Maybe I’m building another wing on it. Maybe I’m building another floor. Maybe I’m just redecorating. But this is we are a service that has a rich field of experience in this area if we choose to take advantage of it.

And I think that’ll do two things. Number one it’ll enable us to get [01:02:00] better, but it’ll also help the people who are, entrenched in the way that we do things. Now, understand that, you know what, 10 years ago things were done differently. Understand that 30 years ago, to some abuse seems like a long time ago.

To me it doesn’t. And we did things differently and we grew through this stuff.

Heather “Lucky” Penney: Yeah. And ultimately it’s making our performance so much better. So if we care about mission outcomes, we need to, I think, take that approach Crypto that you just mentioned of evaluating. Why we’re doing something, what its benefit is, and what elements of that we need to preserve as we move to move forward, what elements of that we need to integrate and what elements of that we can simply let go.

Mike “Pako” Benitez: Yeah, absolutely. I think that I just end on air power as airmanship. The human condition and it sets in reps is how you evolve that. And so the more opportunities we have to debrief, we will reserve the right to learn along the way. And then what I think is, if you fast forward, if we do this right to, to your point, Crypto, we have we have this unique [01:03:00] opportunity, a generational opportunity. If we do it right, we’re gonna learn through the debrief process and the sets and reps, and we’re going to be able to proliferate the integration of AI and mission autonomy into other aspects, whether it’s contested or distributed logistics, whether it’s network collaborative autonomy for cruise missiles and things like that. And this is a really good forcing function to learn what we can out of this program, and what the Air Force is doing with the CCA program to learn those lessons, to then rapidly proliferate it to other mission areas, to really restore some of the capability capacity and readiness that the Air Force has been suffering through for probably a generation now that we’ve all been a part of.

So I just am with that. Yeah. Really looking forward to it.

Heather “Lucky” Penney: Gentlemen, thank you so much. Your insights and your expertise has been huge for this conversation and been a lot of fun looking forward to bringing y’all back again.

Mike “Pako” Benitez: Yeah. Thanks. Lucky. I appreciate the opportunity.

Ray “Crypto” O’Mara: Thank you.

Heather “Lucky” Penney: And with that, I’d like to extend a big thank you to our guests for joining in today’s conversation.

I’d also like to extend a big thank you to [01:04:00] you, our listeners, for your continued support and for tuning into today’s show. If you like what you heard today, don’t forget to hit that like button or follow or subscribe to the Aerospace Advantage. You can also leave a comment to let us know what you think about our show or areas that you would like us to explore further.

As always, you can join in on the conversation by following the Mitchell Institute on X, Instagram, Facebook, or LinkedIn, and you can always find us at mitchellaerospacepower.org. Thanks again for joining us and have a great aerospace power kind of day. See you next time.