Tan Boon Leong

AVP / Hd Intelligence & Sensemaking, Mission Software & Services, ST Engineering

What if…we could save more lives by reducing the response time in handling emergency situations with application of Artificial Intelligence (AI)?

You will need to gather a multi-disciplinary team, comprising of an Ops-Tech lead, Data Scientists, Software Engineers and UI/UX Engineers.

Left to Right: Kenneth Won Jiunn Shyong, Zhuo Hui Yu, Tan Boon Leong, Chong Jin Kun, Bryan Low Zhen Xuan and Li Zhilong.
Absent from photo: Cherise Tan Pek Ying, Mark Gee Jun Wen and Joshua Ng Shen Geh

And you will need an AI Playbook, to come together and “Implement AI.” As the team sets out to develop a prototype, in this case study, an Artificial Intelligence (AI) enabled sensemaking system, they can leverage advanced technologies to exploit multimodal data to generate the situation picture for first responders in a timely manner.

Over the course of 2 years, our team did just that.

We worked closely with more than 60 Singapore Civil Defence Force (SCDF) officers to develop an AI system to sharpen SCDF’s operational edge in protecting and saving lives and property. The system comprises state-of-the-art AI modules developed in-house and by partners: Institute for Info-Comm Research, A*STAR; Nanyang Technological University and Singapore Management University.

Our AI Playbook

ST Engineering seeks to deliver maximum value to customers by applying and embedding analytics and Artificial Intelligence in its products and services.

Thus, the AI Playbook consists of three stages for customers with different needs along their digital transformation journey:

Figure 1. ST Engineering Artificial Intelligence Playbook for Customers
  • Stage 1 – Envision AI. Together, we envision how AI can be used to transform customer’s businesses for greater efficiency. Through Design Thinking workshops, we identify “as-is” pain-points in existing business processes, and also brain-storm for viable application of Data Analytics / Artificial Intelligence (DA/AI) solution to achieve “to-be” state of higher productivity. We also take the opportunity to review customer’s enterprise IT architecture to support DA/AI system implementation.
  • Stage 2 – Implement AI. We perform Proof-of-Concept projects on high impact use cases to develop working prototypes, through turnkey project or co-development with customer’s technical team.
  • Stage 3 – Deploy AI. We deploy DA/AI models on scalable platforms; and establish tools and Continuous Integration/Continuous Deployment (CI/CD) processes to monitor model performances and governance of data.

Key Challenges Encountered in Emergency Situations

In emergency situations, there is a need for decision to make accurate assessments and develop effective action plans based on available, information streaming in at different velocity and channels and they are often incomplete. In addition, these different types of information comes in different format, eg audio calls, video feeds, images, reports, social media feeds..etc.

Currently, the process is manual, man-in-the-loop and tedious to sieve through voluminous data. Without the AI enabled sensemaking engine, more trained operators are required to process and analyse the information in a timely manner to generate accurate actionable intelligence.

Our team identified two key bottlenecks:

  • SCDF Ambulance and Fire Service Emergencies services number 995 is the primary means from which an incident is first established. However, it is often challenging for the call taker to obtain essential information such as medical conditions or incident location especially from a panicky or distressed caller. It is critical for the call taker to engage the caller to avoid situations when caller is overwhelmed and become incapacitated to provide further information or unknowingly giving false information. At the same time, the call taker needs to ensure that the information is entered into the Emergency Response system accurately and as soon as possible.
  • To maintain situational awareness, the Ops Centre needs to resolve unique incidents from duplicate reports. On the other hand, hidden links between seemingly isolated incidents need to be uncovered and to alert the watch officer promptly. Consequently, the responders may review their course of action for better effectiveness or if it remains relevant.

The Central Idea

The team has developed a prototype AI rapid sensemaking engine that leverages on advanced technologies that can exploit multi-modal data to help generate insights for decision makers by helping to piece together a credible on-site real-time situation picture and amplify human capabilities through automated analysis.

With this, the Ops Centre is able to:

  • Reduce workload of the call taker, who can focus on addressing the needs of the caller
  • Eliminate need for dedicated manpower to monitor situation or interpret data
  • Allow the system the flexibility to be scalable and deployable – based on cloud native, microservice architecture using kubernetes.

The system is capable of the following:

  • Automatic Speech Recognition engine to transcribe telephony calls in Singlish
    • Supports 3 languages: English, Mandarin and Malay code switches under noisy conditions
    • ASR model trained for the emergency service response domain and local Singlish
    • Natural Language Processing (NLP) to automatically classify the incident category and extract key information
Figure 2: Call for help in medical emergencies (Enactment)

  • Video / Image Analytics to extract salient information for situational awareness and risk assessments.
    • Assess severity of fire through size estimate and fire characteristics
    • VA model to detect public disorder behaviour and classify objects of interest
    • Automatically retrieve relevant information from other database, eg Petroleum and Flammable Material Licences
    • Localise incident from an image taken at the site

Figure 3: Video analytics to classify objects and assess fire severity. (Source: SCDF)

  • Integrated Sensemaking Engine to fuse insights and information generated from video, image, speech and text.
    • Flag duplicated reports and uncover hidden links between incidents / events
    • Exploit social media for contextual information related to incidents
    • Multimodal data visualised on dashboard in intuitive manner
    • Accurate, Relevant and Timely situation picture for watch officer
Figure 4: Integrated sensemaking to accelerate decision making. (Enactment of a major Public Safety and Security incident)

Conclusion

ST Engineering’s expertise in system design, AI engineering and O&S has delivered a prototype system which demonstrated the implementation of AI in the ops center. At the conclusion of this project, the SCDF has a spring load solution for transition to operation and can better make informed decisions in operationalising the next generation Ops Centre, especially during the challenging period of the pandemic.

It’s a privilege for the team to be given the opportunity to work with SCDF frontliners. Together , we create new capabilities and innovations that contributes to SCDF’s mission to protect and save lives and property for a safe and secure Singapore.

Subscribe for notifications

Fill in the particular below to receive notifications of new insights and articles, delivered directly into your inbox.