The award-winning Xiaomi Mi Mix redefined smartphone design and proved that ultrasonic proximity sensing was viable in smartphones. It was a 0-to-1 product that sparked the full-screen trend, a design choice that remains the industry standard today.
However, it was not suited for mass-market adoption due to high costs, custom designs, and reliance on dedicated hardware components.
To create a viable business case in the highly competitive smartphone market, a robust, cost-efficient, and scalable software-based sensor solution was needed.
The product vision was clear: replace dedicated hardware with software by reusing the phone’s existing speakers and microphones, unlocking a true disruptive innovation that reduced costs and enabled greater industrial design flexibility.
The Problem & Design Constraints
- Business Model: Smartphone OEMs preferred a per-device licensing fee, meaning large-scale adoption was critical. Crossing the chasm from early adopters to mainstream customers was necessary to generate sufficient revenue.
- Hardware Limitations: The Xiaomi Mi Mix relied on a dedicated Murata ultrasonic transducer placed behind the screen, making it costly and difficult to scale. Re-using existing speakers and microphones limited the ultrasonic spectrum available.
- Performance Constraints: The solution had to be power-efficient with a minimal memory footprint to run alongside other processes on smartphone chipsets.
- Environmental noise: Since external sources of ultrasound can interfere with the detection, mitigation solutions needed to be developed
- Data Complexity: Unlike traditional IR sensors, the software-based ultrasonic solution required large amounts of structured data to function reliably in real-world conditions.
The Solution: Supervised Machine Learning at Scale
The transition from a hardware-dependent solution to a scalable software sensor using existing audio hardware was made possible through supervised machine learning.
Traditional signal processing techniques struggled with maintaining performance across multiple devices. When optimizing one area, unintended side effects were often introduced elsewhere. Machine learning allowed functionality to build incrementally without causing regressions.
Instead of defining explicit rules for signal processing, the models were trained using structured data and labeled examples. By continuously refining these models with real-world data, the software was able to adapt to new devices without requiring complete redesigns. The data pipeline, already in place from earlier development stages, provided a solid foundation for this transition. It enabled large-scale data collection, cloud-based processing, and iterative improvements based on structured product metrics.
My Role & Contributions
Unlocking Reliable Model Training Through Ground Truth Data
Based on the insight that the effectiveness of supervised machine learning models heavily depends on high-quality training data with labels, I focused on streamlining the data collection and building robust ground truth solutions:
- Envisioned and implemented an external ground truth solution that could be universally used across smartphones via USB connection.
- Addressed data collection challenges by reducing the need for highly repetitive human-controlled test processes, which had initially slowed progress.
- Enabled a unified benchmark that allowed consistent performance evaluation across different smartphone models.
Enhancing Product Metrics & UX Testing
- Established quantitative product and UX metrics to assess performance objectively.
- Implemented structured subjective scoring to define minimum acceptance criteria for release candidates.
- Built experience prototypes that showcased the final solution’s advantages to stakeholders.
- Incorporated gyroscope and accelerometer data into the data collection process, enhancing product performance through sensor fusion.
- Supported business development efforts in Asia, bridging technical and commercial discussions with OEMs.
Refining Data-Driven Development
- Designed scalable data collection apps that uploaded structured test data to a SaaS backend, improving model training efficiency.
- Enabled structured user testing that validated real-world performance and accelerated model refinement.
- Aided mitigation algorithms that allowed data collected from one phone model to be adapted to another, reducing redundant recordings and optimizing resources.
- Developed and refined data collection plans used to create a comprehensive data set
Trade-offs & Challenges
One of the biggest hurdles was the need for extensive structured user test data. Each new phone model introduced small variations in echo responses, requiring careful adaptation. Additionally, handling low-prevalence corner cases was a time-consuming process, demanding efficient methods to balance coverage and development speed. To mitigate these challenges, we developed algorithms that allowed training data from one phone model to be repurposed for another, significantly reducing the need for redundant recordings.
Since the solution did not rely on dedicated hardware, it had to coexist with existing smartphone audio processes. This required close collaboration with chipset vendors like Cirrus Logic, Qualcomm, and MediaTek to ensure smooth integration across multiple platforms. Another major challenge was quality assurance during the manufacturing process. Without a dedicated sensor to test, we had to invent new approaches to verify the performance of the ultrasonic software at the end of the assembly line.
Impact & Industry Adoption
By 2025, the ultrasonic proximity sensor had been implemented in over 500 million smartphones, with three of the top five smartphone manufacturers using Elliptic Labs’ software sensor. My contributions in establishing a high-quality ground truth framework and optimizing machine learning data pipelines were instrumental in making this transition possible.
The Power of Data-Driven Innovation
The success of Elliptic Labs’ mass-market ultrasonic sensing solution demonstrated the transformative power of supervised machine learning when paired with a well-structured data pipeline. By replacing dedicated hardware with software, we not only enabled full-screen smartphone designs but also proved that data-driven development could disrupt an entire industry.
