ApolloCore from Tocaro Blue is camera processing software for marine Electro Optical / Infrared (EO/IR) Pan/Tilt/Zoom (PTZ) systems, providing machine learning-based stabilization, detection, classification, ranging, tracking, and optional sensor fusion.
Designed for commercial, recreational, crewed, and autonomous marine platforms, ApolloCore combines PTZ imagery with vessel attitude data to maintain views of waterborne objects, classify objects, estimate range, and track contacts over time.
The software can operate with a single PTZ camera or integrate with Tocaro Blue’s ProteusCore Radar processing software, combining EO/IR classification and bearing data with Radar detection, ranging, tracking, and path information through a centralized fusion process.
Stabilization for Marine PTZ Cameras
Vessel motion presents a particular challenge for PTZ cameras operating at high zoom, where a narrow field of view can quickly move away from a contact.
ApolloCore on laptop.
ApolloCore uses vessel Attitude information from an NMEA 2000 Sat-compass or onboard IMU to control the PTZ platform. The software predicts motion affecting the camera and applies corrective control to maintain alignment with the horizon and keep relevant contacts in view.
The approach also addresses gyroscope drift in internally stabilized cameras, while native camera drivers are used to minimize control and data-stream latency. ApolloCore’s sea state Predictive Controller has been validated through hardware-in-the-loop simulations.
Machine Learning Detection and Classification
ApolloCore processes EO and IR imagery using dedicated Machine Learning (ML) models, detecting objects frame by frame before maintaining them as tracks over time.
The software distinguishes 15 object classes selected for navigation and ISR applications. These comprise eight COLREGS (Rule 18) vessel classes: Powerboat, Personal Watercraft, Sailboat, Ferry, Towboat, Ship, Barge, and Fishing Vessel.
Daymark and Buoy form two marker classes, with Bird, Human Face, Automobile, Debris, and Offshore Structure making up five additional specialized classes.
Independent models process EO and IR imagery. Where both channels are available, detections can be cross-validated between them to increase confidence when both sensors identify the same object.
The detection models support up to 50 objects within a frame, including contacts at different relative scales and objects that are partially obscured or passing close to one another.
ApolloCore can also identify shoreline or horizon features to help isolate waterborne targets.
Monocular Ranging from a Single Camera
ApolloCore uses a multi-stage ML process to estimate range without requiring a stereo camera arrangement.
The system supports cameras mounted as low as 3 meters, approximately 10 feet, above the waterline. Offshore, ranging can use the visible horizon, while operation in coastal areas can draw on Tocaro Blue worldwide shoreline maps embedded within ApolloCore. IMU-based ranging provides a fallback when visibility is degraded by weather.
This approach is intended to address the limitations of simple trigonometric ranging at the relatively low sensor mounting heights found on many USVs and workboats.
Search and Scan for Wide-Area Surveillance
The narrow field of view associated with PTZ cameras can provide long-range detail while limiting the surrounding area visible in each frame.
ApolloCore’s EO/IR Object Seeking Mode addresses this by searching for new objects while continuing to update existing tracks. Camera direction and zoom are adjusted automatically to increase area coverage.
The software also maintains a map of detection probabilities, recording areas that have recently been observed and identifying potential gaps in coverage. This allows the PTZ’s narrow field of view to be used for long-range detection while retaining information about where the camera has searched.
Fix and Follow for Object Tracking
For applications requiring sustained observation of a particular vessel, ApolloCore provides a Fix and Follow mode that maintains a lock on the selected object and can re-acquire it following a temporary loss.
SIFT descriptors retain feature-based information for detected objects, helping distinguish nearby objects, reduce track swaps, and support re-identification when the camera returns to a previously observed contact.
Broad-area trackers, dynamic ranging, and object searching can remain active while Fix and Follow is operating.
Combining Camera and Radar Data
ApolloCore can integrate with Tocaro Blue’s ProteusCore Radar tracking software to combine measurements from the two sensor types.
Radar provides 360-degree detection and ranging, as well as tracking and path prediction, while EO/IR cameras provide classification and precise bearing information. ApolloCore uses centralized fusion of these measurements rather than track-to-track correlation.
Radar object confirmation, also referred to as slew-to-cue, enables ProteusCore Radar tracks to direct the PTZ camera toward potential contacts. ApolloCore can then capture and catalog imagery associated with individual objects while incorporating Radar tracking and closest point of approach (CPA) information.
Multi-Object Fusion and Tracking
ApolloCore’s Fusion & Tracking module manages the lifecycle of tracked contacts and can handle hundreds of contacts simultaneously. As measurements accumulate, the software can refine classification confidence and improve estimates of future object movement.
Additional information can be incorporated from AIS over NMEA 2000, channel-marker light lists, Radar TTM data, LIDAR tracks, and customer-provided sources. Geographic polygons representing land and water can also be used to focus tracking on relevant waterborne contacts.
Outputs include estimated object position and uncertainty, SIFT identifiers, refined classifications, bounding-box and range overlays, and representations of track history and predicted movement.
Marine PTZ Camera Compatibility
ApolloCore is designed for integration with existing marine PTZ hardware. Compatibility is driven by customer requirements, with Tocaro Blue continuing to develop support for a range of Current Scientific camera models.
Performance is maximized with cameras incorporating a 640 x 480 Infrared IC or larger together with an EO daylight or low-light camera.
ApolloCore includes calibration maps for supported cameras to compensate for optical distortion. Where EO and IR lenses are offset, pixel-accurate mappings enable detections from the two image sources to be compared directly.
System Integration and Deployment
ApolloCore requires vessel pose and heading information to align EO/IR detections with an overhead view. These inputs can be received through NMEA 2000 from a satellite compass or from an onboard IMU, while USV and ASV applications can provide pose information directly through the WebSocket API.
HTTP and WebSocket interfaces are supported, with JSON messages providing access to camera configuration, sensor inputs, detections, tracker outputs, and ProteusCore linking functions.
For initial evaluation, the Tocaro Blue GPU Devkit uses Nvidia AGX Orin hardware to run ApolloCore and host the Tocaro Blue Configuration App aboard an existing vessel. For production deployment, ApolloCore can be compiled for the customer’s computing platform and can also operate within a Docker container.
These deployment options allow ApolloCore’s stabilization, machine learning classification, monocular ranging, active searching, tracking, and optional Camera+Radar Fusion functions to be incorporated into compatible marine PTZ systems.
