Physics-based Scene-level Reasoning for Object Pose Estimation in Clutter

Chaitanya Mitash, Abdeslam Boularias and Kostas E. Bekris

International Journal of Robotics Research (IJRR) 2019
arXiV

This webpage provides a complete solution for object pose estimation in clutter. The following components are shared:

  • Autonomous data generation to train CNNs for object segmentation (synthetic data)
  • Rutgers Extended RGBD dataset (test dataset comprising real pose-labeled scenes)
  • Search-based object pose estimation process.

 

Autonomous data generation to train CNNs for object segmentation

The training dataset is generated by physical simulation of the setup in which the robot operates. The tool we developed for autonomous data generation, labeling, and training is shared below.

Dataset Generation toolbox: https://github.com/cmitash/physim-dataset-generator

Learned models for object segmentation:
Faster-RCNN (VGG16) Physics Simulation + Self Learning (Shelf): download
Faster-RCNN (VGG16) Physics Simulation + Self Learning (Table-top): download

Rutgers Extended RGBD dataset

Dataset download link: download
For each scene in the dataset, we share:

  • RGB Image
  • Depth Image
  • Segmentation mask
  • Parameters
    • camera_pose: pose of the camera in a global frame.
    • camera_intrinsics: intrinsic parameters of the camera.
    • rest_surface: pose of the resting surface such as a table or shelf bin.
    • dependency_order: physical and visual dependency of objects upon each other.
    • pose: ground-truth object pose in a global frame.

Examples of scenes in the dataset and results of pose estimation with physics-based reasoning.

Object Pose Estimation

Code: https://github.com/cmitash/PhysimGlobalPose

Related papers

Chaitanya Mitash, Kostas E. Bekris, and Abdeslam Boularias, A Self-supervised Learning System for Object Detection using Physics Simulation and Multi-view Pose Estimation
In Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), Vancouver, Canada, 2017
[arXiV] [video]

Chaitanya Mitash, Abdeslam Boularias and Kostas E. Bekris, Improving 6D Pose Estimation of Objects in Clutter via Physics-aware Monte Carlo Tree Search
In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 2018
[arXiV] [video]

Contact Information

Chaitanya Mitash, Kostas E. Bekris, and Abdeslam Boularias
Computer Science Department, Rutgers University, New Brunswick, NJ.
E-mail: {cm1074,kb572,ab1544}@rutgers.edu