Q&A: Major General Stephen T. Sargent
Implementing Test and Evaluation
Process That is Relevant and Timely
Major General Stephen T. Sargent
Commander
Air Force Operational
Test and Evaluation Center
Sargeant was born in Defiance, Ohio, and earned his commission from the U.S. Air Force Academy in 1978. He has served as the commandant of the USAF Weapons School at Nellis AFB, Nev., commanded the 8th Fighter Wing at Kunsan Air Base, South Korea, and the 56th Fighter Wing at Luke AFB, Ariz. He was also the Deputy Chief of Staff, Strategy, Plans and Assessment for Multi-National Force-Iraq. Sargeant has served at the Pentagon as military assistant to both the deputy secretary of defense and secretary of defense, and he was the deputy executive secretary to the National Security Council in the White House. Prior to his assignment to AFOTEC, the general served as the deputy chief of staff for the United Nations Command and U.S. Forces Korea at Yongsan Army Garrison in South Korea. He is a command pilot with more than 3,000 flying hours.
Q: Briefly describe AFOTEC’s mission.
A: AFOTEC tests and evaluates new weapon system capabilities in operationally realistic battlespace environments to provide factbased, decision-quality data to inform decision-makers on a range of accurate, balanced and timely assessments of effectiveness, suitability and mission capability.
Q: How do AFOTEC’s detachments support the command mission, in particular, for space and missile activities?
A: AFOTEC’s five detachments are the execution arms of the organization. Each detachment focuses on a different area of expertise. Detachment 2 at Eglin AFB, Fla. tests and evaluates weapon and electronic warfare systems. At Kirtland AFB, N.M., Detachment 3 focuses on command, control, communications, computers and intelligence [C4I] programs. Operational testing of space, missile and missile defense systems is the responsibility of Detachment 4 at Peterson AFB, Colo. Mobility, bomber, special operations and unmanned aircraft systems are tested by Detachment 5 at Edwards AFB, Calif. Finally, fighter programs are tested at Nellis AFB by our Detachment 6. For test programs in their focus area, each detachment is responsible for early influence activities, including requirements refinement, as well as planning, budgeting, test execution and report writing. The detachments are supported by headquarters personnel and draw operators and maintainers from operational units, as required, to conduct realistic testing.
Q: Your thoughts, please, on how to improve the flow and use of operational test and evaluation [OT&E] community data in DoD space acquisition programs?
A: Traditional acquisition and test models work well for systems with large-scale production decisions like aircraft. However, the lion’s share of investment for space systems occurs early in the program. Traditional models do not provide decision-quality data in time to support key space acquisition decisions.
AFOTEC’s space test initiative [STI] provides an operational test and evaluation model better suited to small-quantity, high-cost acquisitions typical for space systems. The space model provides early information to the acquisition and operational decisionmakers by moving OT&E activities earlier in the program, prior to launch. The space test initiative relies on three key tenets: early and continuous integrated testing involvement throughout the system’s life cycle, agile analysis and reporting, and focus on systemof- systems evaluations.
Q: How were you able to gain support for the space test initiative outside of AFOTEC?
A: In July 2008, we hosted the first-ever Space Summit at Kirtland AFB. The summit gathered senior leaders from the under secretary of the Air Force, Directorate of Space Acquisition, the National Geospatial-Intelligence Agency, Air Force Space Command, Space and Missile Systems Center and the Air Force Research Laboratory to discuss the space capabilities acquisition process, focusing on improving OT&E support to space acquisition. The summit helped to refine our approach and was well received by the DoD space community.
Afterward we also socialized the concept with industry leaders through the Industry Council for Operational Test & Evaluation to gain support and perspectives from the system developers. Then we moved out to codify these principles in an update to Air Force Instruction 99-103, “Capabilities Based Test and Evaluation,” that is currently in coordination.
Q: What are some of the OT&E lessons learned that led to the initial space test initiative?
A: About a year and a half ago, at my first commanders conference, we had our Detachment 4 commander—who deals primarily in the space programs—express a concern about the reports and how those reports were viewed of the OT&E work that we were doing. With some very healthy dialogue, we figured out that what we had done over the years was to take an OT&E model that had worked for major systems like aircraft where you buy a large number after you had built a few. In this model, most of the money flows at the back end of the program instead of right up front.
The space model is just 180 out from that. In the space business the key acquisition decisions and funding that come with those decisions are made very early in the program, and typically we are only funding a very few in terms of acquisition numbers. When we stepped back and looked at when those decisions were being made and when our OT&E reports were being made, we found that we were not really getting involved until after the space enabler—the satellite in most cases—was on orbit.
Once we get involved after all of the acquisitions have been made we have missed what we were really chartered to do. What this meant was that the work we were doing was having marginal impact on the warfighter and almost no impact on the acquisition decision-maker.
So we stepped back and did some very in-depth research across the space community, across the acquisition community and inside the T&E community itself. We discovered that what we were really doing was what I would call standardization and evaluation work. The work was beneficial, but marginally, and was really just informing the operator how to use the satellite better.
Notice I’ve not mentioned the ground system—the ground station that enables the warfighter to get the benefit of what the onorbit systems can provide. In many cases we weren’t really focused on that system. So we again backed up and asked ourselves when do we need to get involved—much earlier, how do we do that—by partnering very early on with the space community whether it’s the developer, the tester or the warfighter.
We began our search to pull the right people together, which resulted in the Space Summit that produced even better ideas. From that came our space test initiative.
From there we were really able to look at what OT&E really means to those systems and how to deliver a report—and maybe not even a traditional report—to the acquisition decision-makers at the right time to give them the data they need to be informed on not only the on-orbit system, but as important, those ground stations and their development. By doing this, we can avoid situations in the future where an on-orbit system may have used up 20 to 30 percent of its contracted life while the ground station is still being developed.
Q: When you went to the Space Command and some of the other organizations, what were their initial perceptions and how were the new ideas received?
A: I would have to say that, to our great surprise, when we approached every other command, we were met with 1) open arms and 2) with enthusiasm. What we were outlining and talking about at the time quickly made sense to the senior leadership of those organizations that this could actually help in their quest for acquisition excellence. Getting OT&E involved left of launch—with perhaps as much as 90 percent of our work—instead of doing most of our work right of launch would help inform them much better as to the effectiveness of the system, the suitability of the system, and, importantly, the mission capability of that system. This information would become available when decisions had to be made early on to make adjustments during the development phase, which is much more cost-effective in terms of time, schedule and cost.
I admit that there was some degree of skepticism that an OTA was coming forward and basically saying that we haven’t done, to the extent that we should, the work we were chartered to do.
It was self-examination that brought us to that conclusion. We have great people here at AFOTEC across all of the detachments, and the headquarters took this to heart and moved it forward at a very fast pace but with passion to ensure that the process is repeatable and done with as little extra cost as possible, in terms of time and dollars.
Q: What is your evaluation of the state of jointness in the space and missile programs?
A: As you know, most of the programs that we do are conducted under a multi-service construct. We are the lead for most of those programs.
We have a great relationship with the other operational test agencies [OTAs] across all of the services—perhaps especially with the Navy and the Army each having a lot of work in the space arena.
What we did right up front was to inform the other OTAs of what we were doing in our space test initiative and opened the doors immediately for them to come in and be partners. They have observed and been informed along the way, and now we are working with them to do what we have already done, which is to work the changes in the instructions [Air Force-level of instructions] such that the work done is codified for the future and standardized within our guidebooks. We are sharing those instructions with the Navy and the Army, which are taking them and individually tailoring to their own needs.
The other part is we are working on a more widely based joint and multi-agency environment working with the NSS [National Space Security Acquisition] 03-01. We know that there will be a review by the office of primary responsibility for NSS 03-01 later this spring. So what we did prior to the holidays was hold an action officers summit, with multi-agency participation, and take what we did with the Air Force instruction and tailor it to the broader national space instruction.
By doing so, we will be prepared to be the centerpiece for the work that is done later on when the entire instruction comes under review.
Q: Recently, AFOTEC was credited with completing the Space- Based Infrared System Highly Elliptical Orbit payload and operations center testing ahead of schedule. Please explain how you did this.
A: The Space-Based Infrared System [SBIRS] is our first program to use AFOTEC’s space test initiative [STI]. SBIRS supports an urgent Joint Space Operations Center operational need levied on Air Force Space Command. Using STI principles, we were able to increase the emphasis on the integrated testing already planned in the original test plan. While the original test plan made use of integrated testing for major end-to-end system tests and OT&E readiness, the STI changes allowed the test team to capitalize on earlier developmental tests to accelerate test procedure development and eventually allowed the team to collect over 40 percent of their effectiveness data prior to entering dedicated OT&E.
The integrated test approach accelerated warfighter “first use” and U.S. Strategic Command operational certification of the first satellite payload by six weeks. The test team also identified efficiencies during the execution of the operational utility evaluation [OUE] and reduced the length of the OUE from 75 to 60 days. Finally, our new agile reporting philosophy allowed AFOTEC to provide decision-quality data in an interim summary report to decision-makers 19 days from the end of the OUE instead of the 48 days typically needed to produce a full report. All of these changes together resulted in U.S. Strategic Command system certification eight weeks earlier than planned.
Q: Is it possible for you to quantify where you are in the space test initiative between the time you started to the point that you consider it to be full-up operational?
A: Yes, I can. We have two significant success data points already, both from programs that were already mature. By applying some of the principles and procedures from the space test initiative, we were able to reduce the amount of time that it was going to take for the warfighter to deliver information to the airmen, soldiers, sailors, Marines and civilians working in the field. I already mentioned the first, SBIRS, which was reduced by nearly two months and in the other was the Wideband Global SAT by 10 months. This is not insignificant in either case given the history of those systems.
Importantly, just before we sat down here I took a briefing from our Detachment 4 on a program that is on a very fast track, so fast that the acquisition decision-makers have elected to go right to KDP-B [key decision point] as opposed to KDP-A. Our space test initiative and its people are proving to be flexible enough to actually go in and do the requirements refinement work despite the normal amount of time available to get to a KDP-B decision. They have already worked those KDP-A requirements.
Now we are in this program that is very early on and marching through the anatomy we created of the space test initiative and demonstrating early in the program that the pieces are working and that we are continuing to refine them as we move forward with conducting hot washes after every stage of OT&E progress.
Q: What is the next step for the Space Test Initiative?
A: The next step is to match the National Security Space [NSS] 03-01 Acquisition Policy with the updates to USAF T&E policy described earlier. In December 2008, we held an action officer meeting to start integrating space test principles into the NSS 03-01. Representatives from the Service Operational Test Organizations, as well as deputy under secretary of the Air Force for Space Programs, the POC for national security space acquisition policy, spent four days drafting an annex to NSS 03-01. The annex to 03-01 will undergo formal coordination when NSS 03-01 begins its update cycle in the spring of 2009.
Finally, we’ll hold the 2nd annual Space Summit in March 2009 to re-affirm partnerships with national agencies like the National Geospatial-Intelligence Agency, National Reconnaissance Office, and National Aeronautics and Space Administration.
Q: What are the technology gaps and shortfalls that preclude higher fidelity and more accurate OT&E data from use in space and missile programs?
A: We face challenges in realistically testing space systems such as threat replication, space modeling and simulation maturity, and ground testing and stimulation of satellite sensors. However, we can overcome most of these shortfalls using laboratories on Earth before launch, through system-of-system testing as well as initial drafting of and modifications to development contracts.
Q: What space-enabled systems will AFOTEC evaluate to prepare the U.S. for future wars?
A: One of the SECAF and CSAF’s priorities is to modernize our air and space organizations, training and equipment. Two obvious examples of space-enabled systems are the F-22A Raptor and F-35 Lightning II Joint Strike Fighter [JSF].
The F-22A Raptor already provides many of the air dominance capabilities required for future conflicts. AFOTEC conducted IOT&E for the F-22A and follow-on OT&E for the increment 2 upgrade. We are currently in the test planning phase for increment 3.1, which will further enhance the F-22A’s capability against ground threats using precision weapons that rely on the Global Positioning System [GPS]. Increment 3.1 also includes synthetic aperture radar for ground target mapping and employment of the small diameter bomb [SDB]. One of the challenges we face in FOT&E of increment 3.1 is supersonic delivery of SDB.
The F-35 Joint Strike Fighter will deliver joint and coalition multi-role capabilities essential to many nations’ modernization plans and also needed to address 21st century threats. The F-35 will rely heavily upon space systems to provide intelligence information and long-range communications to support its many missions. AFOTEC will serve as the lead operational test agency for the F-35 multi-service and multi-national IOT&E, and we are already conducting operational assessments.
Q: What has AFOTEC done in OT&E to support the current war?
A: AFOTEC is completely committed to supporting today’s war by getting weapon system capabilities, whether space or traditional aircraft systems, to the field as soon as they are ready to deploy. Two recent examples where AFOTEC supported an Urgent Operational Need [UON] with rapid OT&E are the Laser Joint Direct Attack Munition [Laser JDAM] and the A-10C Thunderbolt II.
The Laser JDAM program supported an urgent operational need for moving target strike capability through an operational utility evaluation. In this case, AFOTEC applied an integrated developmental test [DT]/operational test [OT] philosophy to leverage 12 DT drops and reduce the number of OT drops from 10 to five. Integrated DT/ OT allowed us to reduce the test cost by $1.43 million and duration by 14 days. We produced a status report 21 days after the last drop to support a fielding decision six weeks ahead of schedule. The OUE from beginning to final report lasted less than three months and allowed delivery of the weapon to the warfighter within 17 months from concept to fielding. Three months after our test report, U.S forces destroyed their first target with the Laser JDAM.
The A-10C is the largest-ever upgrade to the A-10 and provides tremendous capabilities including delivery of precision weapons, data link, and video downlink to joint terminal attack controllers. The original operational utility evaluation was scheduled to support a fielding decision and a 2007 deployment to the area of operations. When several critical deficiency reports delayed the initial OUE, AFOTEC rapidly re-planned the OUE to use the Nellis Test and Training Range, jets and crews from Nellis AFB and the Maryland Air National Guard unit designated for deployment. We executed an aggressive “surge” to complete the original test events in half the time. While flying the test missions, operators and maintainers were able to train on the system and refine their tactics, techniques and procedures in preparation for their deployment. The OUE final report was published in time to support the fielding decision and an on-time deployment of the A-10C to the area of operations where it has proven to be invaluable to the global war on terrorism.
Q: Could you tell me a little more about the upcoming Air Force T&E Days conference and exhibition?
A: Last year was my first exposure to the Air Force Test & Evaluation Days when SMC [Space Missile Command] and the Air Force Flight Test Center held a fantastic event out in California. I was amazed to see how many testers, scientists, analysts, engineers and operators were all brought together under one roof. There were great discussions on relevant T&E issues.
The conferences have been held for about six years now. Typically, different product centers work with AIAA to host each event. What the conference really allows is for the Air Force to bring together all of those professionals and get involved for about three days and focus those elements of the T&E enterprise that can lead to better acquisition excellence. It is critical at this time in the Air Force’s history, especially noting our secretary’s and chief’s all-out focus and effort on improving our acquisition process at all levels.
The T&E community is such a vital part of that—from the development piece early in a program to the OT&E piece at the 12th hour. We are turning that around by the interaction of these types of conferences so that more and more people can say that at OT&E we are not only involved at the last hour. It starts back early on with the developer and the developmental testers and continues all the way through fielding.
At the conferences, the senior leaders come together, and this year I might add we have a great lineup of senior leaders that cut across all domains—air, space and cyberspace. The test community, product centers, Space Command, Air Force Materiel Command and DOT&E from OSD will all be there to lay out issues that are not only current today but will have an impact well into the future.
One thing we are doing differently for this conference is that we are going to establish work groups. We will have a keynote speaker followed by a panel. In the panels, once the issues are defined and laid out, we will have working groups that will go off for a day or two and examine and explore solutions to those problems and bring them back during the conference. The panel will update the attendees on their progress. If the solution cannot be found during the conference, then we will work on the Websites that are empowered by AIAA and the OTAs to continue to provide those participants and their colleagues an opportunity to continue to refine the solution. Some of this work may continue on well after the T&E Days have concluded. This will give OT&E a living and breathing start to 2009—solving T&E issues leading toward acquisition excellence.
Q: Any final thoughts?
A: The people within AFOTEC are top notch. We are blessed with a very savvy group of operators, a professional corps of analysts, engineers, scientists and support staff.
It’s a wonderful group of professionals, who quite frankly aren’t in the limelight very often—and that’s not all bad—but I would like to take the opportunity to pat them all on the back right now and put the spotlight on them.
Our people have done a lot over the past couple of years to really pull together the different domains of air and space. The common driver has always been acquisition excellence in both of those domains.
We are now launching more work, another initiative in the cyberspace arena to help empower those commands that are being stood up and those that have to meet the threat posed through cyberspace. This will enable those airmen, soldiers, sailors and Marines to defeat that threat.
We are looking for ways to amend the way AFOTEC does business, in concert with the other OTAs, to better serve that domain just like we have in the space domain.
We aren’t resting on our laurels in the air domain by any means. We have refined our processes there as well to ensure the warfighters get what they need, when they need it.
We will continue to do this. We are looking forward to the T&E Days to share some of those lessons learned and to learn from the lessons of others as they have worked diligently in their own areas. We will continue to keep that operational focus ensuring that our airmen have what they need in the field working side by side, in many cases, with our joint coalition partners. ♦