Charles River Analytics is advancing the reliability of autonomous systems with its Toolbox for Verification of Autonomous Systems with Neural Components (TOOL-VAN), designed to make deep learning verification more accessible and actionable.
Unlike everyday physical products such as refrigerators or chairs, which undergo extensive testing to ensure predictable performance, autonomous systems powered by Deep Neural Networks (DNNs) are difficult to evaluate comprehensively. Their inherent complexity reduces operational transparency, making it challenging to guarantee consistent behavior.
“Testing neural networks with every single permutation and combination of inputs is physically impossible,” says Dr. Jeff Druce, Senior Scientist at Charles River and Co-Principal Investigator on TOOL-VAN. “You just can’t manually perturb the system in every way and know what it’s going to do,” he added. TOOL-VAN solves this problem and enables verification and validation (V&V) of such autonomous systems at scale.
TOOL-VAN addresses this gap by enhancing existing verification algorithms while simplifying their use. Its intuitive drag-and-drop interface determines the appropriate Verification and Validation (V&V) methods and generates reports predicting system behavior across a range of scenarios. Developed with user experience in mind, the toolkit incorporates feedback from DoD analysts and other end users, lowering the barrier to adoption.
“The toolbox accounts for a wider class of inputs, which is a step up from existing DNN V&V approaches,” says Michael Harradon, Principal Scientist at Charles River and Co-Principal Investigator on TOOL-VAN. Most V&V solutions can only evaluate finite test instances, preventing analysis of environmental factors, Harradon explains. Harradon notes that TOOL‑VAN expands individual test inputs to evaluate performance over a broad range of transformations including factors like contrast, lighting conditions, and even weather. If an autonomous driving system has only trained on images in daylight, for example, it will be easier to find out how the system will behave at night and in the rain.
Funded with up to $2 million in Phase II SBIR support from the Office of Naval Research, TOOL-VAN enables the Navy to deploy autonomous systems with greater confidence in sensing, control, and decision-making. While initially focused on defense applications, the toolkit has broader potential across commercial autonomous vehicles and cyber-physical systems. Internally, Charles River plans to integrate TOOL-VAN into proprietary explainable AI software, further enhancing system transparency and reliability.
“TOOL-VAN is useful because it builds understanding in the situations under which you would want to use an autonomous system—and when you wouldn’t,” Harradon added.
