Robust Visual-Tactile State Estimation for Occlusion Handling
Develop a multi-modal state estimator that fuses visual input with proprioceptive and tactile history using a recurrent filter to handle severe occlusions.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The paper reports that 43.7% of real-world failures are caused by pose tracking loss using FoundationPose, specifically due to heavy occlusion by the robot hand or visual ambiguity of symmetric objects like markers.
“Conditioning on object pose goals alone is environment-blind, which can lead to collisions in cluttered scenes. We also currently assume tools are rigid. Pose alone can be insufficient to describe the state of non-rigid tools (e.g., scissors).”
Limitation
The system relies exclusively on third-person vision for state estimation. When the hand occludes the tool (common in dexterous manipulation), the visual tracker fails, causing the policy to lose track of the object state entirely.
Proposed Solution
Develop a multi-modal state estimator that fuses visual input with proprioceptive and tactile history. Implement a recurrent filter (e.g., Kalman Filter or LSTM) that predicts the object's pose update based on hand joint velocities and contact feedback when visual confidence scores from FoundationPose drop below a threshold.
Why This Matters
Solving occlusion is critical for deployment; a policy that goes blind during the most critical part of manipulation (grasping and using) is unreliable for production environments.
How to Get Started
How to test it: Hypothesis: Fusing visual input with tactile and proprioceptive history via an LSTM filter will prevent tracking loss during hand occlusions. Baseline: Vision-only FoundationPose tracking. Metric: Task progress percentage and object tracking loss frequency. Stopping criteria: Failure rate reduction of at least 30% under heavy occlusion.
- Collect demonstration data featuring heavy hand-object occlusions using a robotic arm equipped with a wrist camera and joint sensors.
- Implement an LSTM-based recurrent filter or Extended Kalman Filter that takes hand joint velocities and contact states as inputs.
- Integrate the filter with FoundationPose to predict object poses when visual confidence scores fall below a predefined threshold.
- Benchmark pose tracking accuracy and success rates against the vanilla visual-only pipeline on cluttered manipulation tasks.
Code linked to this paper on Hugging Face: github.com/tylerlum/simtoolreal
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed results discuss world models, surveys on robotic perception, deformable objects, and physical AI, but none of them cover the specific proposal of fusing visual input with tactile history using a recurrent filter when FoundationPose confidence drops. Therefore, this idea remains novel regarding these specific papers.
We searched OpenAlex for “Develop a multi-modal state estimator that fuses visual input with proprioceptive and tactile history using a recurrent filter to handle severe occlusions.” and “Robust Visual-Tactile State Estimation for Occlusion Handling” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool ManipulationKushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg +1 more · 2026
This paper presents SimToolReal, a framework for achieving zero-shot dexterous tool manipulation in the real world using a single Reinforcement Learning policy trained entirely in simulation. By training on procedurally generated "primitive" objects (simple combinations of handles and heads) to track random goal poses, the policy generalizes to diverse real-world tools and tasks (like hammering or sweeping) by conditioning on a coarse geometric abstraction rather than visual appearance.
Related Ideas
- Action-Conditioned Counterfactual Synthesis for Embodied Physical PlanningV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟠Medium
- High-Dimensional Articulated Control in Latent World ModelsWorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling🔴Hard
- Conformal Multimodal Outcome Verification to Mitigate Premature Completion BiasEnvisioning Agentic Robotics: A New Era of Robotic Systems🟡Fairly easy
- Trajectory-Guided Motion Encoders for Video InputOmniLottie: Generating Vector Animations via Parameterized Lottie Tokens🟠Medium
- Single-Pass Visual Dependence Estimation to Eliminate Dual-Pass Target ScoringMMFlash: Faster Speculative Decoding for Vision-Language Models🟡Fairly easy