Below you can find information regarding software packages that our research group has released or can provide upon request:


Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands

Many manipulation tasks, such as placement or within-hand manipulation, require the object’s pose relative to a robot hand. The task is difficult when the hand significantly occludes the object. It is especially hard for adaptive hands, for which it is not easy to detect the finger’s configuration. In addition, RGB-only approaches face issues with texture-less objects or when the hand and the object look similar. This paper presents a depth-based framework, which aims for robust pose estimation and short response times. It could be integrated with tracking-based methods to provide initialization or recovery from lost tracking. The approach detects the adaptive hand’s state via efficient parallel search given the highest overlap between the hand’s model and the point cloud. The hand’s point cloud is pruned and robust global registration is performed to generate object pose hypotheses, which are clustered. False hypotheses are pruned via physical reasoning. The remaining poses’ quality is evaluated given agreement with observed data.

Project Page


se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains

Tracking the 6D pose of objects in video sequences is important for robot manipulation. This task, however, introduces multiple challenges: (i) robot manipulation involves significant occlusions; (ii) data and annotations are troublesome and difficult to collect for 6D poses, which complicates machine learning solutions, and (iii) incremental error drift often accumulates in long term tracking to necessitate re-initialization of the object’s pose. This work proposes a data-driven optimization approach for long-term, 6D pose tracking. It aims to identify the optimal relative pose given the current RGB-D observation and a synthetic image conditioned on the previous best estimate and the object’s model. The key contribution in this context is a novel neural network architecture, which appropriately disentangles the feature encoding to help reduce domain shift, and an effective 3D orientation representation via Lie Algebra. Consequently, even when the network is trained only with synthetic data can work effectively over real images. Comprehensive experiments over benchmarks – existing ones as well as a new dataset with significant occlusions related to object manipulation – show that the proposed approach achieves consistently robust estimates and outperforms alternatives, even though they have been trained with real images. The approach is also the most computationally efficient among the alternatives and achieves a tracking frequency of 90.9Hz.

Project Page



Scene-level Pose Estimation for Multiple Instances of Densely Packed Objects

This paper introduces key machine learning operations that allow the realization of robust, joint 6D pose estimation of multiple instances of objects either densely packed or in unstructured piles from RGB-D data. The first objective is to learn semantic and instance-boundary detectors without manual labeling. An adversarial training framework in conjunction with physics-based simulation is used to achieve detectors that behave similarly in synthetic and real data. Given the stochastic output of such detectors, candidates for object poses are sampled.The second objective is to automatically learn a single score for each pose candidate that represents its quality in terms of explaining the entire scene via a gradient boosted tree. The proposed method uses features derived from surface and boundary alignment between the observed scene and the object model placed at hypothesized poses. Scene-level, multi-instance pose estimation is then achieved by an integer linear programming process that selects hypotheses that maximize the sum of the learned individual scores, while respecting constraints, such as avoiding collisions. To evaluate this method, a dataset of densely packed objects with challenging setups for state-of-the-art approaches is collected. Experiments on this dataset and a public one show that the method significantly outperforms alternatives in terms of 6D pose accuracy while trained only with synthetic datasets

Project Page


Robust 6D object pose estimation with stochastic congruent sets

Object pose estimation is frequently achieved by first segmenting an RGB image and then, given depth data, registering the corresponding point cloud segment against the object’s 3D model. Despite the progress due to CNNs, semantic segmentation output can be noisy, especially when the CNN is only trained on synthetic data. This causes registration methods to fail in estimating a good object pose. This work proposes a novel stochastic optimization process that treats the segmentation output ofCNNs as a confidence proba-bility. The algorithm, called Stochastic Congruent Sets (StoCS), samples pointsets on the point cloud according to the soft segmentation distribution and so as to agree with the object’s known geometry. The pointsets are then matched to congruent sets on the3D object model to generate pose estimates.StoCSis shown to be robust on an APC dataset, despite the fact the CNN is trained only on synthetic data. In the YCBdataset, StoCS outperforms a recent network for 6D pose estimation and alternative pointset matching techniques.



Stable Sparse RRT (SST): Efficient Asymptotically Near-Optimal Kinodynamic Motion Planning

We have developed the Stable Sparse-RRT (SST) algorithm, which provides favorable properties for systems where a steering function (BVP solver) may not be available, while being highly efficient. It can be shown that SST is asymptotically near-optimal and with a small modification (SST*) it can also provide asymptotic optimality for kinodynamic problems under some mild assumptions. In addition, SST maintains a sparse data structure, where the number of nodes stored in their trees is significantly lower than RRT providing computational benefits.

Multi-agent Path Finding: Push and Swap code

Our efforts in the area of Complete and Tractable Multi-Robot Path Planning started in 2011 with an algorithm called Push and Swap, which has polynomial running time and is complete but suboptimal for the multi-agent path finding problem. The algorithm has attracted attention and we received multiple requests for the source code. Since 2011 Ryan Luna has continued working on optimizing this implementation, in collaboration with Athanasios Krontiris and Kostas Bekris.

Download the latest version of the original Push and Swap algorithm.

The group has also worked on showing the relation between such efficient and complete but suboptimal planners and algorithmic contributions on the pebble motion problem. We will soon release the code that provides implementations for multiple variations of these methods.

The available code for Push and Swap above provides sequential solutions, i.e., where only one agent moves at a time. There is also work on a parallel version of Push and Swap. The implementation is not as optimized yet but we can provide it upon request.

Sparse Representations for Motion Planning: SPARS and SPARS 2 available on OMPL

Asymptotically Near-Optimal Roadmap-Based Planners SPARS and SPARS2 have been released as part of the Open Motion Planning Library maintained by the Physical and Biological Computing Group at Rice University. The benefit of these methods is that they correspond to sparse motion planning data structures that can be queried and communicated efficiently, while at the same time they are also able to provide guarantees regarding the quality of the resulting paths.

Download the latest version of OMPL.

The code is an implementation provided by Andrew Dobson. You can find SPARS and SPARS 2 under the list of supported planners by OMPL.

PRACSYS software: An Extensible Architecture for Composing Motion Controllers, Motion Planners and Task Planners

Our research group is working on a common software package that has been utilized for a variety of research projects over the last few years. Our objective is to provide an extensible software architecture so that the integration of low-level motion controllers with motion planners and higher-level task planers is streamlined.

An earlier version of the code is available through a Sourceforge repository, which corresponds to our paper at SIMPAR 2012.

We have since significantly revised and extended the functionality of our software package. Our current main focus is on the definition of proper interfaces for composing motion and task planners. Feel free to contact us if you are interested in the latest version and we will share a Mercurial repository.

Examples of publications that have utilized this software platform include the following: