Sign up for our weekly emails to stay up to date on the latest news!
SoftBank Corp. has developed a multimodal AI system to enhance the safety and efficiency of autonomous vehicle (AV) operations, focusing on enabling effective remote monitoring by human operators. The system aims to tackle challenges posed by Japan’s traffic regulations, especially at unsignalized crosswalks, where a stopped vehicle requires the following vehicle to also come to a complete stop. By using multimodal AI, which processes diverse data inputs in real-time, SoftBank aims to reduce operational costs and enhance vehicle safety.
Key highlights include:
- Multimodal AI integrates text, audio, images, and sensor data for comprehensive processing.
- Real-time edge AI processing to support AV operations on low-latency servers.
- Field trials initiated in October 2024 at Keio University Shonan Fujisawa Campus.
- Enhanced safety and efficiency for AVs through a deeper understanding of traffic scenarios.
The multimodal AI collects and integrates different types of data, including text, audio, images, and sensor information, to provide a comprehensive analysis for remote operators. This enhanced understanding is crucial for human operators who monitor multiple AVs simultaneously, ensuring they can intervene effectively when necessary.
In October 2024, SoftBank launched a field trial at Keio University Shonan Fujisawa Campus in Fujisawa City, Kanagawa Prefecture. The trial aims to verify whether the multimodal AI can support smooth and safe AV operations, even when the vehicles face unexpected scenarios. This AI-driven solution is expected to provide more reliable support for remote monitoring.
The AI model was trained using a wide range of Japanese traffic information, including regulations and typical risk scenarios. By processing forward-facing footage from autonomous vehicles, such as dashcam videos, the system offers real-time insights into complex traffic situations and provides recommended actions to ensure safe driving practices.
Additionally, the remote support solution is being trialed by MONET Technologies, a joint venture involving SoftBank, Toyota Motor Corporation, and other Japanese automakers. This collaboration aims to further improve AV safety through continuous learning from real-world driving environments, allowing the AI to adapt to unpredictable risks and refine its recommendations over time.
By advancing the capabilities of multimodal AI, SoftBank is taking significant steps toward the goal of fully unmanned operations, ensuring AVs can navigate complex traffic environments safely and efficiently while minimizing human intervention.
Sign up for our weekly emails to stay up to date on the latest news!
