Co-robots Human robot interaction Legible paths Manipulation Manipulator Pose Estimation Rearrangement Robot Perception Sociotechnological Systems tensegrity
2022 |
Morgan, A S; Hang, K; Wen, B; Bekris, K E; Dollar, A M Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planning Journal Article IEEE Robotics and Automation Letters, 2022. Abstract | Links | BibTeX | Tags: Manipulation @article{inhand_gaiting_multimodal, title = {Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planning}, author = {A S Morgan and K Hang and B. Wen and K E Bekris and A M Dollar}, url = {https://arxiv.org/abs/2201.07928}, year = {2022}, date = {2022-01-24}, journal = {IEEE Robotics and Automation Letters}, abstract = {Constraining contacts to remain fixed on an object during manipulation limits the potential workspace size, as motion is subject to the hand's kinematic topology. Finger gaiting is one way to alleviate such restraints. It allows contacts to be freely broken and remade so as to operate on different manipulation manifolds. This capability, however, has traditionally been difficult or impossible to practically realize. A finger gaiting system must simultaneously plan for and control forces on the object while maintaining stability during contact switching. This work alleviates the traditional requirement by taking advantage of system compliance, allowing the hand to more easily switch contacts while maintaining a stable grasp. Our method achieves complete SO(3) finger gaiting control of grasped objects against gravity by developing a manipulation planner that operates via orthogonal safe modes of a compliant, underactuated hand absent of tactile sensors or joint encoders. During manipulation, a low-latency 6D pose object tracker provides feedback via vision, allowing the planner to update its plan online so as to adaptively recover from trajectory deviations. The efficacy of this method is showcased by manipulating both convex and non-convex objects on a real robot. Its robustness is evaluated via perturbation rejection and long trajectory goals. To the best of the authors' knowledge, this is the first work that has autonomously achieved full SO(3) control of objects within-hand via finger gaiting and without a support surface, elucidating a valuable step towards realizing true robot in-hand manipulation capabilities. }, keywords = {Manipulation}, pubstate = {published}, tppubtype = {article} } Constraining contacts to remain fixed on an object during manipulation limits the potential workspace size, as motion is subject to the hand's kinematic topology. Finger gaiting is one way to alleviate such restraints. It allows contacts to be freely broken and remade so as to operate on different manipulation manifolds. This capability, however, has traditionally been difficult or impossible to practically realize. A finger gaiting system must simultaneously plan for and control forces on the object while maintaining stability during contact switching. This work alleviates the traditional requirement by taking advantage of system compliance, allowing the hand to more easily switch contacts while maintaining a stable grasp. Our method achieves complete SO(3) finger gaiting control of grasped objects against gravity by developing a manipulation planner that operates via orthogonal safe modes of a compliant, underactuated hand absent of tactile sensors or joint encoders. During manipulation, a low-latency 6D pose object tracker provides feedback via vision, allowing the planner to update its plan online so as to adaptively recover from trajectory deviations. The efficacy of this method is showcased by manipulating both convex and non-convex objects on a real robot. Its robustness is evaluated via perturbation rejection and long trajectory goals. To the best of the authors' knowledge, this is the first work that has autonomously achieved full SO(3) control of objects within-hand via finger gaiting and without a support surface, elucidating a valuable step towards realizing true robot in-hand manipulation capabilities. |
2021 |
Dylan S. Shah Joran W. Booth, Robert. Baines Kun Wang Massimo Vespignani Kostas Bekris Rebecca Kramer-Bottiglio L Tensegrity Robotics Journal Article Forthcoming Soft Robotics, Forthcoming. Abstract | BibTeX | Tags: tensegrity @article{tensegrity_survey, title = {Tensegrity Robotics}, author = {Dylan S. Shah, Joran W. Booth, Robert. L Baines, Kun Wang, Massimo Vespignani, Kostas Bekris, Rebecca Kramer-Bottiglio}, year = {2021}, date = {2021-12-01}, journal = {Soft Robotics}, abstract = {Numerous recent advances in robotics have been inspired by the biological principle of tensile integrity — or “tensegrity”— to achieve remarkable feats of dexterity and resilience. Tensegrity robots contain compliant networks of rigid struts and soft cables, allowing them to change their shape by adjusting their internal tension. Local rigidity along the struts provides support to carry electronics and scientific payloads, while global compliance enabled by the flexible interconnections of struts and ca- bles allows a tensegrity to distribute impacts and prevent damage. Numerous techniques have been proposed for designing and simulating tensegrity robots, giving rise to a wide range of locomotion modes including rolling, vibrating, hopping, and crawling. Here, we review progress in the burgeoning field of tensegrity robotics, highlighting several emerging challenges, including automated design, state sensing, and kinodynamic motion planning.}, keywords = {tensegrity}, pubstate = {forthcoming}, tppubtype = {article} } Numerous recent advances in robotics have been inspired by the biological principle of tensile integrity — or “tensegrity”— to achieve remarkable feats of dexterity and resilience. Tensegrity robots contain compliant networks of rigid struts and soft cables, allowing them to change their shape by adjusting their internal tension. Local rigidity along the struts provides support to carry electronics and scientific payloads, while global compliance enabled by the flexible interconnections of struts and ca- bles allows a tensegrity to distribute impacts and prevent damage. Numerous techniques have been proposed for designing and simulating tensegrity robots, giving rise to a wide range of locomotion modes including rolling, vibrating, hopping, and crawling. Here, we review progress in the burgeoning field of tensegrity robotics, highlighting several emerging challenges, including automated design, state sensing, and kinodynamic motion planning. |
Meng, P; Wang, W; Balkcom, D; Bekris, K E Proof-of-Concept Designs for the Assembly of Modular, Dynamic Tensegrities into Easily Deployable Structures Conference ASCE Earth and Space Conference 2021, Seattle, WA, 2021. @conference{228, title = {Proof-of-Concept Designs for the Assembly of Modular, Dynamic Tensegrities into Easily Deployable Structures}, author = {P Meng and W Wang and D Balkcom and K E Bekris}, year = {2021}, date = {2021-10-06}, booktitle = {ASCE Earth and Space Conference 2021}, address = {Seattle, WA}, abstract = {Dynamic tensegrity robots are inspired by tensegrity structures in architecture; arrangements of rigid rods and flexible elements allow the robots to deform. This work proposes the use of multiple, modular, tensegrity robots that can move and compliantly connect to assemble larger, compliant, lightweight, strong structures and scaffolding. The focus is on proof-of-concept designs for the modular robots themselves and their docking mechanisms, which can allow the easy deployment of structures in unstructured environments. These mechanisms include (electro)magnets to allow each individual robot to connect and disconnect on cue. An exciting direction is the design of specific module and structure designs to fit the mission at hand. For example, this work highlights how the considered three bar structures could stack to form a column or deform on one side to create an arch. A critical component of future work will involve the development of algorithms for automatic design and layout of modules in structures.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Dynamic tensegrity robots are inspired by tensegrity structures in architecture; arrangements of rigid rods and flexible elements allow the robots to deform. This work proposes the use of multiple, modular, tensegrity robots that can move and compliantly connect to assemble larger, compliant, lightweight, strong structures and scaffolding. The focus is on proof-of-concept designs for the modular robots themselves and their docking mechanisms, which can allow the easy deployment of structures in unstructured environments. These mechanisms include (electro)magnets to allow each individual robot to connect and disconnect on cue. An exciting direction is the design of specific module and structure designs to fit the mission at hand. For example, this work highlights how the considered three bar structures could stack to form a column or deform on one side to create an arch. A critical component of future work will involve the development of algorithms for automatic design and layout of modules in structures. |
Wang, Kun; Aanjaneya, Mridul; Bekris, Kostas E Sim2Sim Evaluation of a Novel Data-Efficient Differentiable Physics Engine for Tensegrity Robots Inproceedings IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021. @inproceedings{sim2sim_tensegrity, title = {Sim2Sim Evaluation of a Novel Data-Efficient Differentiable Physics Engine for Tensegrity Robots }, author = {Kun Wang and Mridul Aanjaneya and Kostas E. Bekris}, year = {2021}, date = {2021-09-27}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, abstract = {Learning policies in simulation is promising for reducing human effort when training robot controllers. This is especially true for soft robots that are more adaptive and safe but also more difficult to accurately model and control. The sim2real gap is the main barrier to successfully transfer policies from simulation to a real robot. System identification can be applied to reduce this gap but traditional identification methods require a lot of manual tuning. Data-driven alternatives can tune dynamical models directly from data but are often data hungry, which also incorporates human effort in collecting data. This work proposes a data-driven, end-to-end differentiable simulator focused on the exciting but challenging domain of tensegrity robots. To the best of the authors' knowledge, this is the first differentiable physics engine for tensegrity robots that supports cable, contact, and actuation modeling. The aim is to develop a reasonably simplified, data-driven simulation, which can learn approximate dynamics with limited ground truth data. The dynamics must be accurate enough to generate policies that can be transferred back to the ground-truth system. As a first step in this direction, the current work demonstrates sim2sim transfer, where the unknown physical model of MuJoCo acts as a ground truth system. Two different tensegrity robots are used for evaluation and learning of locomotion policies, a 6-bar and a 3-bar tensegrity. The results indicate that only 0.25% of ground truth data are needed to train a policy that works on the ground truth system when the differentiable engine is used for training against training the policy directly on the ground truth system. }, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Learning policies in simulation is promising for reducing human effort when training robot controllers. This is especially true for soft robots that are more adaptive and safe but also more difficult to accurately model and control. The sim2real gap is the main barrier to successfully transfer policies from simulation to a real robot. System identification can be applied to reduce this gap but traditional identification methods require a lot of manual tuning. Data-driven alternatives can tune dynamical models directly from data but are often data hungry, which also incorporates human effort in collecting data. This work proposes a data-driven, end-to-end differentiable simulator focused on the exciting but challenging domain of tensegrity robots. To the best of the authors' knowledge, this is the first differentiable physics engine for tensegrity robots that supports cable, contact, and actuation modeling. The aim is to develop a reasonably simplified, data-driven simulation, which can learn approximate dynamics with limited ground truth data. The dynamics must be accurate enough to generate policies that can be transferred back to the ground-truth system. As a first step in this direction, the current work demonstrates sim2sim transfer, where the unknown physical model of MuJoCo acts as a ground truth system. Two different tensegrity robots are used for evaluation and learning of locomotion policies, a 6-bar and a 3-bar tensegrity. The results indicate that only 0.25% of ground truth data are needed to train a policy that works on the ground truth system when the differentiable engine is used for training against training the policy directly on the ground truth system. |
Sivaramakrishnan, Aravind; Granados, Edgar; Karten, Seth; McMahon, Troy; Bekris, Kostas E Improving Kinodynamic Planners for Vehicular Navigation with Learned Goal-Reaching Controllers Inproceedings IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021. BibTeX | Tags: @inproceedings{learned_goal_reaching_controllers, title = {Improving Kinodynamic Planners for Vehicular Navigation with Learned Goal-Reaching Controllers }, author = {Aravind Sivaramakrishnan and Edgar Granados and Seth Karten and Troy McMahon and Kostas E. Bekris}, year = {2021}, date = {2021-09-27}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
Wen, Bowen; Bekris, Kostas E BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models Inproceedings IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021. Abstract | BibTeX | Tags: Pose Estimation @inproceedings{bundletrack, title = {BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models }, author = {Bowen Wen and Kostas E. Bekris}, year = {2021}, date = {2021-09-27}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, abstract = {Tracking the 6D pose of objects in video sequences is important for robot manipulation. Prior efforts, however, often assume that the target object's CAD model, at least at a category-level, is available for offline training or during online template matching. This work proposes BundleTrack, a general framework for 6D pose tracking of novel objects, which does not depend upon instance or category-level 3D models. It leverages the complementary attributes of recent advances in deep learning for segmentation and robust feature extraction, as well as memory augmented pose-graph optimization for achieving spatiotemporal consistency. This enables long-term, low-drift tracking under various challenging scenarios, including significant occlusions and object motions. Comprehensive experiments given two public benchmarks demonstrate that the proposed approach significantly outperforms state-of-art category-level 6D tracking or dynamic-SLAM methods. When compared against state-of-art methods that rely on an object instance CAD model, comparable performance is achieved, despite the proposed method's reduced information requirements. An efficient implementation in CUDA provides a real-time performance of 10Hz for the entire framework. }, keywords = {Pose Estimation}, pubstate = {published}, tppubtype = {inproceedings} } Tracking the 6D pose of objects in video sequences is important for robot manipulation. Prior efforts, however, often assume that the target object's CAD model, at least at a category-level, is available for offline training or during online template matching. This work proposes BundleTrack, a general framework for 6D pose tracking of novel objects, which does not depend upon instance or category-level 3D models. It leverages the complementary attributes of recent advances in deep learning for segmentation and robust feature extraction, as well as memory augmented pose-graph optimization for achieving spatiotemporal consistency. This enables long-term, low-drift tracking under various challenging scenarios, including significant occlusions and object motions. Comprehensive experiments given two public benchmarks demonstrate that the proposed approach significantly outperforms state-of-art category-level 6D tracking or dynamic-SLAM methods. When compared against state-of-art methods that rely on an object instance CAD model, comparable performance is achieved, despite the proposed method's reduced information requirements. An efficient implementation in CUDA provides a real-time performance of 10Hz for the entire framework. |
Si-Wei Feng Teng Guo, Kostas E.Bekris Jingjin Yu Team RuBot’s experiences and lessons from the ARIAC Journal Article Robotics and Computer-Integrated Manufacturing, 70 , 2021. Abstract | Links | BibTeX | Tags: Rearrangement @article{siwei_ariac_21, title = {Team RuBot’s experiences and lessons from the ARIAC}, author = {Si-Wei Feng, Teng Guo, Kostas E.Bekris, Jingjin Yu}, editor = {Craig Schlenoff, Zeid Kootbally, Erez Karpas (Special Issue: Agile Robotics for Industrial Applications)}, url = {https://www.sciencedirect.com/science/article/abs/pii/S0736584521000120}, year = {2021}, date = {2021-08-01}, journal = {Robotics and Computer-Integrated Manufacturing}, volume = {70}, abstract = {We share experiences and lessons learned in participating the annual Agile Robotics for Industrial Automation Competition (ARIAC). ARIAC is a simulation-based competition focusing on pushing the agility of robotic systems for handling industrial pick-and-place challenges. Team RuBot started competing from 2019, placing 2nd place in ARIAC 2019 and 3rd place in ARIAC 2020. The article also discusses the difficulties we faced during the contest and our strategies for tackling them.}, keywords = {Rearrangement}, pubstate = {published}, tppubtype = {article} } We share experiences and lessons learned in participating the annual Agile Robotics for Industrial Automation Competition (ARIAC). ARIAC is a simulation-based competition focusing on pushing the agility of robotic systems for handling industrial pick-and-place challenges. Team RuBot started competing from 2019, placing 2nd place in ARIAC 2019 and 3rd place in ARIAC 2020. The article also discusses the difficulties we faced during the contest and our strategies for tackling them. |
Morgan, A; Wen, B; Junchi, L; Boularias, A; Dollar, A; Bekris, K E Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks Conference Robotics: Science and Systems, 2021. Abstract | BibTeX | Tags: Manipulation, Robot Perception @conference{MorWen2021, title = {Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks}, author = {A Morgan and B Wen and L Junchi and A Boularias and A Dollar and K E Bekris}, year = {2021}, date = {2021-07-12}, booktitle = {Robotics: Science and Systems}, abstract = {Highly constrained manipulation tasks continue to be challenging for autonomous robots as they require high levels of precision, typically less than 1mm, which is often incompatible with what can be achieved by traditional perception systems. This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks with tight, industrially-relevant tolerances (0.25mm). The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant workspace. It adjusts the control reference of both the compliant manipulator and the hand to complete object insertion tasks via within-hand manipulation. Contrary to previous efforts for insertion, our method does not require expensive force sensors, precision manipulators, or time-consuming, online learning, which is data hungry. Instead, this effort leverages mechanical compliance and utilizes an object-agnostic manipulation model of the hand learned offline, off-the-shelf motion planning, and an RGBD-based object tracker trained solely with synthetic data. These features allow the proposed system to easily generalize and transfer to new tasks and environments. This paper describes in detail the system components and showcases its efficacy with extensive experiments involving tight tolerance peg-in-hole insertion tasks of various geometries as well as open-world constrained placement tasks.}, keywords = {Manipulation, Robot Perception}, pubstate = {published}, tppubtype = {conference} } Highly constrained manipulation tasks continue to be challenging for autonomous robots as they require high levels of precision, typically less than 1mm, which is often incompatible with what can be achieved by traditional perception systems. This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks with tight, industrially-relevant tolerances (0.25mm). The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant workspace. It adjusts the control reference of both the compliant manipulator and the hand to complete object insertion tasks via within-hand manipulation. Contrary to previous efforts for insertion, our method does not require expensive force sensors, precision manipulators, or time-consuming, online learning, which is data hungry. Instead, this effort leverages mechanical compliance and utilizes an object-agnostic manipulation model of the hand learned offline, off-the-shelf motion planning, and an RGBD-based object tracker trained solely with synthetic data. These features allow the proposed system to easily generalize and transfer to new tasks and environments. This paper describes in detail the system components and showcases its efficacy with extensive experiments involving tight tolerance peg-in-hole insertion tasks of various geometries as well as open-world constrained placement tasks. |
Wang, Rui; Gao, Kai; Nakhimovich, Daniel; Yu, Jingjin; Bekris, Kostas E Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search Inproceedings International Conference on Robotics and Automation (ICRA) 2021, 2021. BibTeX | Tags: Rearrangement @inproceedings{WangGNYB21_uniform_rearrangement, title = {Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search}, author = {Rui Wang and Kai Gao and Daniel Nakhimovich and Jingjin Yu and Kostas E. Bekris}, year = {2021}, date = {2021-05-30}, booktitle = {International Conference on Robotics and Automation (ICRA) 2021}, keywords = {Rearrangement}, pubstate = {published}, tppubtype = {inproceedings} } |
Wang, Rui; Nakhimovich, Daniel; Roberts, Fred S; Bekris, Kostas E Robotics as an Enabler of Resiliency to Disasters: Promises and Pitfalls Book Chapter Roberts, Fred S; Sheremet, Igor A (Ed.): 12660 , pp. 75-101, Springer, 2021. Abstract | Links | BibTeX | Tags: Sociotechnological Systems @inbook{WangNRB21_Resiliency, title = {Robotics as an Enabler of Resiliency to Disasters: Promises and Pitfalls}, author = {Rui Wang and Daniel Nakhimovich and Fred S. Roberts and Kostas E. Bekris}, editor = {Fred S. Roberts and Igor A. Sheremet}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Robotics_Enabler_Resiliency_Disasters.pdf}, year = {2021}, date = {2021-03-03}, volume = {12660}, pages = {75-101}, publisher = {Springer}, series = { Lecture Notes in Computer Science}, abstract = {The Covid-19 pandemic is a reminder that modern society is still susceptible to multiple types of natural or man-made disasters, which motivates the need to improve resiliency through technological advancement. This article focuses on robotics and the role it can play towards providing resiliency to disasters. The progress in this domain brings the promise of effectively deploying robots in response to life-threatening disasters, which includes highly unstructured setups and hazardous spaces inaccessible or harmful to humans. This article discusses the maturity of robotics technology and explores the needed advances that will allow robots to become more capable and robust in disaster response measures. It also explores how robots can help in making human and natural environments preemptively more resilient without compromising long-term prospects for economic development. Despite its promise, there are also concerns that arise from the deployment of robots. Those discussed relate to safety considerations, privacy infringement, cyber-security, and financial aspects, such as the cost of development and maintenance as well as impact on employment.}, keywords = {Sociotechnological Systems}, pubstate = {published}, tppubtype = {inbook} } The Covid-19 pandemic is a reminder that modern society is still susceptible to multiple types of natural or man-made disasters, which motivates the need to improve resiliency through technological advancement. This article focuses on robotics and the role it can play towards providing resiliency to disasters. The progress in this domain brings the promise of effectively deploying robots in response to life-threatening disasters, which includes highly unstructured setups and hazardous spaces inaccessible or harmful to humans. This article discusses the maturity of robotics technology and explores the needed advances that will allow robots to become more capable and robust in disaster response measures. It also explores how robots can help in making human and natural environments preemptively more resilient without compromising long-term prospects for economic development. Despite its promise, there are also concerns that arise from the deployment of robots. Those discussed relate to safety considerations, privacy infringement, cyber-security, and financial aspects, such as the cost of development and maintenance as well as impact on employment. |
Rahul Shome Kiril Solovey, Jingjin Yu Kostas Bekris Dan Halperin Fast, High-Quality Two-Arm Rearrangement in Synchronous, Monotone Tabletop Setups Journal Article IEEE Transactions on Automation Science and Engineering, 2021. Abstract | BibTeX | Tags: Rearrangement @article{shome_tase_two_arm_rearrange, title = {Fast, High-Quality Two-Arm Rearrangement in Synchronous, Monotone Tabletop Setups}, author = {Rahul Shome, Kiril Solovey, Jingjin Yu, Kostas Bekris, Dan Halperin}, year = {2021}, date = {2021-03-01}, journal = {IEEE Transactions on Automation Science and Engineering}, abstract = {Rearranging objects on a planar surface arises in a variety of robotic applications, such as product packaging. Using two arms can improve efficiency but introduces new computational challenges. This paper studies the problem structure of object rearrangement using two arms in synchronous, monotone tabletop setups and develops an optimal mixed integer model. It then describes an efficient and scalable algorithm, which first minimizes the cost of object transfers and then of moves between objects. This is motivated by the fact that, asymptotically, object transfers dominate the cost of solutions. Moreover, a lazy strategy minimizes the number of motion planning calls and results in significant speedups. Theoretical arguments support the benefits of using two arms and indicate that synchronous execution, in which the two arms perform together either transfers or moves, introduces only a small overhead. Experiments support these claims and show that the scalable method can quickly compute solutions close to the optimal for the considered setup.}, keywords = {Rearrangement}, pubstate = {published}, tppubtype = {article} } Rearranging objects on a planar surface arises in a variety of robotic applications, such as product packaging. Using two arms can improve efficiency but introduces new computational challenges. This paper studies the problem structure of object rearrangement using two arms in synchronous, monotone tabletop setups and develops an optimal mixed integer model. It then describes an efficient and scalable algorithm, which first minimizes the cost of object transfers and then of moves between objects. This is motivated by the fact that, asymptotically, object transfers dominate the cost of solutions. Moreover, a lazy strategy minimizes the number of motion planning calls and results in significant speedups. Theoretical arguments support the benefits of using two arms and indicate that synchronous execution, in which the two arms perform together either transfers or moves, introduces only a small overhead. Experiments support these claims and show that the scalable method can quickly compute solutions close to the optimal for the considered setup. |
Si-wei Feng Teng Guo, Kostas Bekris Jingjin Yu Team RuBot's Experiences and Lessons from the ARIAC Journal Article Robotics and Computer-Integrated Manufacturing, 2021. BibTeX | Tags: Manipulation @article{rubot_ariac, title = {Team RuBot's Experiences and Lessons from the ARIAC }, author = {Si-wei Feng, Teng Guo, Kostas Bekris, Jingjin Yu}, year = {2021}, date = {2021-03-01}, journal = {Robotics and Computer-Integrated Manufacturing}, keywords = {Manipulation}, pubstate = {published}, tppubtype = {article} } |
2020 |
Mitash, C; Shome, R; Wen, B; Boularias, A; Bekris, K E Task-driven Perception and Manipulation for Constrained Placement of Unknown Objects Journal Article IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE/RSJ IROS 2020), 2020. Abstract | Links | BibTeX | Tags: @article{231, title = {Task-driven Perception and Manipulation for Constrained Placement of Unknown Objects}, author = {C Mitash and R Shome and B Wen and A Boularias and K E Bekris}, url = {https://robotics.cs.rutgers.edu/task-driven-perception/}, year = {2020}, date = {2020-10-27}, journal = {IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE/RSJ IROS 2020)}, abstract = {Recent progress in robotic manipulation has dealt with the case of no prior object models in the context of relatively simple tasks, such as bin-picking. Existing methods for more constrained problems, however, such as deliberate placement in a tight region, depend more critically on shape information to achieve safe execution. This work introduces a possibilistic object representation for solving constrained placement tasks without shape priors. A perception method is proposed to track and update the object representation during motion execution, which respects physical and geometric constraints. The method operates directly over sensor data, modeling the seen and unseen parts of the object given observations. It results in a dynamically updated conservative representation, which can be used to plan safe manipulation actions. This task-driven perception process is integrated with manipulation task planning architecture for a dual-arm manipulator to discover efficient solutions for the constrained placement task with minimal sensing. The planning process can make use of handoff operations when necessary for safe placement given the conservative representation. The pipeline is evaluated with data from over 240 real-world experiments involving constrained placement of various unknown objects using a dual-arm manipulator. While straightforward pick-sense-and-place architectures frequently fail to solve these problems, the proposed integrated pipeline achieves more than 95% success and faster execution times.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Recent progress in robotic manipulation has dealt with the case of no prior object models in the context of relatively simple tasks, such as bin-picking. Existing methods for more constrained problems, however, such as deliberate placement in a tight region, depend more critically on shape information to achieve safe execution. This work introduces a possibilistic object representation for solving constrained placement tasks without shape priors. A perception method is proposed to track and update the object representation during motion execution, which respects physical and geometric constraints. The method operates directly over sensor data, modeling the seen and unseen parts of the object given observations. It results in a dynamically updated conservative representation, which can be used to plan safe manipulation actions. This task-driven perception process is integrated with manipulation task planning architecture for a dual-arm manipulator to discover efficient solutions for the constrained placement task with minimal sensing. The planning process can make use of handoff operations when necessary for safe placement given the conservative representation. The pipeline is evaluated with data from over 240 real-world experiments involving constrained placement of various unknown objects using a dual-arm manipulator. While straightforward pick-sense-and-place architectures frequently fail to solve these problems, the proposed integrated pipeline achieves more than 95% success and faster execution times. |
Wen, B; Mitash, C; Ren, B; Bekris, K E se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains Conference IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, 2020. Abstract | Links | BibTeX | Tags: @conference{232, title = {se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains}, author = {B Wen and C Mitash and B Ren and K E Bekris}, url = {http://arxiv.org/abs/2007.13866}, year = {2020}, date = {2020-10-26}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, address = {Las Vegas, NV}, abstract = {Tracking the 6D pose of objects in video sequences is important for robot manipulation. This task, however, in- troduces multiple challenges: (i) robot manipulation involves significant occlusions; (ii) data and annotations are troublesome and difficult to collect for 6D poses, which complicates machine learning solutions, and (iii) incremental error drift often accu- mulates in long term tracking to necessitate re-initialization of the object’s pose. This work proposes a data-driven opti- mization approach for long-term, 6D pose tracking. It aims to identify the optimal relative pose given the current RGB-D observation and a synthetic image conditioned on the previous best estimate and the object’s model. The key contribution in this context is a novel neural network architecture, which appropriately disentangles the feature encoding to help reduce domain shift, and an effective 3D orientation representation via Lie Algebra. Consequently, even when the network is trained only with synthetic data can work effectively over real images. Comprehensive experiments over benchmarks - existing ones as well as a new dataset with significant occlusions related to object manipulation - show that the proposed approach achieves consistently robust estimates and outperforms alternatives, even though they have been trained with real images. The approach is also the most computationally efficient among the alternatives and achieves a tracking frequency of 90.9Hz.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Tracking the 6D pose of objects in video sequences is important for robot manipulation. This task, however, in- troduces multiple challenges: (i) robot manipulation involves significant occlusions; (ii) data and annotations are troublesome and difficult to collect for 6D poses, which complicates machine learning solutions, and (iii) incremental error drift often accu- mulates in long term tracking to necessitate re-initialization of the object’s pose. This work proposes a data-driven opti- mization approach for long-term, 6D pose tracking. It aims to identify the optimal relative pose given the current RGB-D observation and a synthetic image conditioned on the previous best estimate and the object’s model. The key contribution in this context is a novel neural network architecture, which appropriately disentangles the feature encoding to help reduce domain shift, and an effective 3D orientation representation via Lie Algebra. Consequently, even when the network is trained only with synthetic data can work effectively over real images. Comprehensive experiments over benchmarks - existing ones as well as a new dataset with significant occlusions related to object manipulation - show that the proposed approach achieves consistently robust estimates and outperforms alternatives, even though they have been trained with real images. The approach is also the most computationally efficient among the alternatives and achieves a tracking frequency of 90.9Hz. |
Wang, R; Mitash, C; Lu, S; Boehm, D; Bekris, K E Safe and Effective Picking Paths in Clutter given Discrete Distributions of Object Poses Conference IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, 2020. Abstract | Links | BibTeX | Tags: @conference{233, title = {Safe and Effective Picking Paths in Clutter given Discrete Distributions of Object Poses}, author = {R Wang and C Mitash and S Lu and D Boehm and K E Bekris}, url = {https://arxiv.org/abs/2008.04465}, year = {2020}, date = {2020-10-25}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, address = {Las Vegas, NV}, abstract = {Picking an item in the presence of other objects can be challenging as it involves occlusions and partial views. Given object models, one approach is to perform object pose estimation and use the most likely candidate pose per object to pick the target without collisions. This approach, however, ignores the uncertainty of the perception process both regarding the target’s and the surrounding objects’ poses. This work proposes first a perception process for 6D pose estimation, which returns a discrete distribution of object poses in a scene. Then, an open-loop planning pipeline is proposed to return safe and effective solutions for moving a robotic arm to pick, which (a) minimizes the probability of collision with the obstructing objects; and (b) maximizes the probability of reaching the target item. The planning framework models the challenge as a stochastic variant of the Minimum Constraint Removal (MCR) problem. The effectiveness of the methodology is verified given both simulated and real data in different scenarios. The experiments demonstrate the importance of considering the uncertainty of the perception process in terms of safe execution. The results also show that the methodology is more effective than conservative MCR approaches, which avoid all possible object poses regardless of the reported uncertainty.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Picking an item in the presence of other objects can be challenging as it involves occlusions and partial views. Given object models, one approach is to perform object pose estimation and use the most likely candidate pose per object to pick the target without collisions. This approach, however, ignores the uncertainty of the perception process both regarding the target’s and the surrounding objects’ poses. This work proposes first a perception process for 6D pose estimation, which returns a discrete distribution of object poses in a scene. Then, an open-loop planning pipeline is proposed to return safe and effective solutions for moving a robotic arm to pick, which (a) minimizes the probability of collision with the obstructing objects; and (b) maximizes the probability of reaching the target item. The planning framework models the challenge as a stochastic variant of the Minimum Constraint Removal (MCR) problem. The effectiveness of the methodology is verified given both simulated and real data in different scenarios. The experiments demonstrate the importance of considering the uncertainty of the perception process in terms of safe execution. The results also show that the methodology is more effective than conservative MCR approaches, which avoid all possible object poses regardless of the reported uncertainty. |
Mitash, Chaitanya Scalable, Physics-aware 6D Pose Estimation for Robot Manipulation PhD Thesis Rutgers University, 2020. Abstract | Links | BibTeX | Tags: Pose Estimation @phdthesis{Mitash:2020aa, title = {Scalable, Physics-aware 6D Pose Estimation for Robot Manipulation}, author = {Chaitanya Mitash}, url = {http://www.cs.rutgers.edu/~kb572/pubs/thesis_chaitanya_mitash_final.pdf}, year = {2020}, date = {2020-09-30}, school = {Rutgers University}, abstract = {Robot manipulation often depend on some form of pose estimation to represent the state of the world and allow decision making both at the task-level and for motion or grasp planning. Recent progress in deep learning gives hope for a pose estimation solution that could generalize over textured and texture-less objects, objects with or without distinctive shape properties, and under different lighting conditions and clutter scenarios. Nevertheless, it gives rise to a new set of challenges such as the painful task of acquiring large-scale labeled training datasets and of dealing with their stochastic output over unforeseen scenarios that are not captured by the training. This restricts the scalability of such pose estimation solutions in robot manipulation tasks that often deal with a variety of objects and changing environments. The thesis first describes an automatic data generation and learning framework to address the scalability challenge. Learning is bootstrapped by generating labeled data via physics simulation and rendering. Then it self-improves over time by acquiring and labeling real-world images via a search-based pose estimation process. The thesis proposes algorithms to generate and validate object poses online based on the objects’ geometry and based on the physical consistency of their scene-level interactions. These algorithms provide robustness even when there exists a domain gap between the synthetic training and the real test scenarios. Finally, the thesis proposes a manipulation planning framework that goes beyond model-based pose estimation. By utilizing a dynamic object representation, this integrated perception and manipulation framework can efficiently solve the task of picking unknown objects and placing them in a constrained space. The algorithms are evaluated over real-world robot manipulation experiments and over large-scale public datasets. The results indicate the usefulness of physical constraints in both the training and the online estimation phase. Moreover, the proposed framework, while only utilizing simulated data can obtain robust estimation in challenging scenarios such as densely-packed bins and clutter where other approaches suffer as a result of large occlusion and ambiguities due to similar looking texture-less surfaces.}, keywords = {Pose Estimation}, pubstate = {published}, tppubtype = {phdthesis} } Robot manipulation often depend on some form of pose estimation to represent the state of the world and allow decision making both at the task-level and for motion or grasp planning. Recent progress in deep learning gives hope for a pose estimation solution that could generalize over textured and texture-less objects, objects with or without distinctive shape properties, and under different lighting conditions and clutter scenarios. Nevertheless, it gives rise to a new set of challenges such as the painful task of acquiring large-scale labeled training datasets and of dealing with their stochastic output over unforeseen scenarios that are not captured by the training. This restricts the scalability of such pose estimation solutions in robot manipulation tasks that often deal with a variety of objects and changing environments. The thesis first describes an automatic data generation and learning framework to address the scalability challenge. Learning is bootstrapped by generating labeled data via physics simulation and rendering. Then it self-improves over time by acquiring and labeling real-world images via a search-based pose estimation process. The thesis proposes algorithms to generate and validate object poses online based on the objects’ geometry and based on the physical consistency of their scene-level interactions. These algorithms provide robustness even when there exists a domain gap between the synthetic training and the real test scenarios. Finally, the thesis proposes a manipulation planning framework that goes beyond model-based pose estimation. By utilizing a dynamic object representation, this integrated perception and manipulation framework can efficiently solve the task of picking unknown objects and placing them in a constrained space. The algorithms are evaluated over real-world robot manipulation experiments and over large-scale public datasets. The results indicate the usefulness of physical constraints in both the training and the online estimation phase. Moreover, the proposed framework, while only utilizing simulated data can obtain robust estimation in challenging scenarios such as densely-packed bins and clutter where other approaches suffer as a result of large occlusion and ambiguities due to similar looking texture-less surfaces. |
Shome, R; Bekris, K E Synchronized Multi-Arm Rearrangement Guided by Mode Graphs with Capacity Constraints Conference Workshop on the Algorithmic Foundations of Robotics (WAFR), Oulu, Finland, 2020. Abstract | Links | BibTeX | Tags: @conference{229, title = {Synchronized Multi-Arm Rearrangement Guided by Mode Graphs with Capacity Constraints}, author = {R Shome and K E Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/multi_arm_rearrangement_capacity_constraints.pdf}, year = {2020}, date = {2020-06-16}, booktitle = {Workshop on the Algorithmic Foundations of Robotics (WAFR)}, address = {Oulu, Finland}, abstract = {Solving task planning problems involving multiple objects and mul- tiple robotic arms poses scalability challenges. Such problems involve not only coordinating multiple high-DoF arms, but also searching through possible se- quences of actions including object placements, and handoffs. The current work identifies a useful connection between multi-arm rearrangement and recent re- sults in multi-body path planning on graphs with vertex capacity constraints. Solving a synchronized multi-arm rearrangement at a high-level involves rea- soning over a modal graph, where nodes correspond to stable object placements and object transfer states by the arms. Edges of this graph correspond to pick, placement and handoff operations. The objects can be viewed as pebbles moving over this graph, which has capacity constraints. For instance, each arm can carry a single object but placement locations can accumulate many objects. Efficient inte- ger linear programming-based solvers have been proposed for the corresponding pebble problem. The current work proposes a heuristic to guide the task planning process for synchronized multi-arm rearrangement. Results indicate good scala- bility to multiple arms and objects, and an algorithm that can find high-quality solutions fast and exhibiting desirable anytime behavior.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Solving task planning problems involving multiple objects and mul- tiple robotic arms poses scalability challenges. Such problems involve not only coordinating multiple high-DoF arms, but also searching through possible se- quences of actions including object placements, and handoffs. The current work identifies a useful connection between multi-arm rearrangement and recent re- sults in multi-body path planning on graphs with vertex capacity constraints. Solving a synchronized multi-arm rearrangement at a high-level involves rea- soning over a modal graph, where nodes correspond to stable object placements and object transfer states by the arms. Edges of this graph correspond to pick, placement and handoff operations. The objects can be viewed as pebbles moving over this graph, which has capacity constraints. For instance, each arm can carry a single object but placement locations can accumulate many objects. Efficient inte- ger linear programming-based solvers have been proposed for the corresponding pebble problem. The current work proposes a heuristic to guide the task planning process for synchronized multi-arm rearrangement. Results indicate good scala- bility to multiple arms and objects, and an algorithm that can find high-quality solutions fast and exhibiting desirable anytime behavior. |
Shome, R; Nakhimovich, D; Bekris, K E Pushing the Boundaries of Asymptotic Optimality in Integrated Task and Motion Planning Conference Workshop on the Algorithmic Foundations of Robotics (WAFR), Oulu, Finland, 2020. Abstract | Links | BibTeX | Tags: @conference{225, title = {Pushing the Boundaries of Asymptotic Optimality in Integrated Task and Motion Planning}, author = {R Shome and D Nakhimovich and K E Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/asymptotic_optimality_task_motion_planning.pdf}, year = {2020}, date = {2020-06-15}, booktitle = {Workshop on the Algorithmic Foundations of Robotics (WAFR)}, address = {Oulu, Finland}, abstract = {Integrated task and motion planning problems describe a multi-modal state space, which is often abstracted as a set of smooth manifolds that are connected via sets of transitions states. One approach to solving such problems is to sample reachable states in each of the manifolds, while simultaneously sampling transition states. Prior work has shown that in order to achieve asymptotically optimal (AO) solutions for such piecewise-smooth task planning problems, it is sufficient to double the connection radius required for AO sampling-based motion planning. This was shown under the assumption that the transition sets themselves are smooth. The current work builds upon this result and demonstrates that it is sufficient to use the same connection radius as for standard AO motion planning. Furthermore, the current work studies the case that the transition sets are non-smooth boundary points of the valid state space, which is frequently the case in practice, such as when a gripper grasps an object. This paper generalizes the notion of clearance that is typically assumed in motion and task planning to include such individual, potentially non-smooth transition states. It is shown that asymptotic optimality is retained under this generalized regime.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Integrated task and motion planning problems describe a multi-modal state space, which is often abstracted as a set of smooth manifolds that are connected via sets of transitions states. One approach to solving such problems is to sample reachable states in each of the manifolds, while simultaneously sampling transition states. Prior work has shown that in order to achieve asymptotically optimal (AO) solutions for such piecewise-smooth task planning problems, it is sufficient to double the connection radius required for AO sampling-based motion planning. This was shown under the assumption that the transition sets themselves are smooth. The current work builds upon this result and demonstrates that it is sufficient to use the same connection radius as for standard AO motion planning. Furthermore, the current work studies the case that the transition sets are non-smooth boundary points of the valid state space, which is frequently the case in practice, such as when a gripper grasps an object. This paper generalizes the notion of clearance that is typically assumed in motion and task planning to include such individual, potentially non-smooth transition states. It is shown that asymptotic optimality is retained under this generalized regime. |
Sintov, A; Kimmel, A; Wen, B; Boularias, A; Bekris, K E Tools for Data-driven Modeling of Within-Hand Manipulation with Underactuated Adaptive Hands Conference Learning for Dynamics & Control (L4DC), Berkeley, CA, 2020. BibTeX | Tags: @conference{224, title = {Tools for Data-driven Modeling of Within-Hand Manipulation with Underactuated Adaptive Hands}, author = {A Sintov and A Kimmel and B Wen and A Boularias and K E Bekris}, year = {2020}, date = {2020-06-12}, booktitle = {Learning for Dynamics & Control (L4DC)}, address = {Berkeley, CA}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Wang, K; Aanjaneya, M; Bekris, K E Learning for Dynamics & Control (L4DC), Berkeley, CA, 2020. @conference{223, title = {A First Principles Approach for Data-Efficient System Identification of Spring-Rod Systems via Differentiable Physics Engines}, author = {K Wang and M Aanjaneya and K E Bekris}, url = {https://arxiv.org/abs/2004.13859}, year = {2020}, date = {2020-06-11}, booktitle = {Learning for Dynamics & Control (L4DC)}, address = {Berkeley, CA}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Sintov, A; Kimmel, A; Bekris, K E; Boularias, A Motion Planning with Competency-Aware Transition Models for Underactuated Adaptive Hands Conference IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020. BibTeX | Tags: @conference{221, title = {Motion Planning with Competency-Aware Transition Models for Underactuated Adaptive Hands}, author = {A Sintov and A Kimmel and K E Bekris and A Boularias}, year = {2020}, date = {2020-06-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, address = {Paris, France}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Kleinbort, M; Solovey, K; Bonalli, R; Granados, E; Bekris, K E; Halperin, D Refined Analysis of Asymptotically-Optimal Kinodynamic Planning in the State-Cost Space Conference IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020. Abstract | Links | BibTeX | Tags: @conference{222, title = {Refined Analysis of Asymptotically-Optimal Kinodynamic Planning in the State-Cost Space}, author = {M Kleinbort and K Solovey and R Bonalli and E Granados and K E Bekris and D Halperin}, url = {https://arxiv.org/abs/1909.05569}, year = {2020}, date = {2020-06-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, address = {Paris, France}, abstract = {We present a novel analysis of AO-RRT: a tree-based planner for motion planning with kinodynamic constraints, originally described by Hauser and Zhou (AO-X, 2016). AO-RRT explores the state-cost space and has been shown to efficiently obtain high-quality solutions in practice without relying on the availability of a computationally-intensive two-point boundary-value solver. Our main contribution is an optimality proof for the single-tree version of the algorithm---a variant that was not analyzed before. Our proof only requires a mild and easily-verifiable set of assumptions on the problem and system: Lipschitz-continuity of the cost function and the dynamics. In particular, we prove that for any system satisfying these assumptions, any trajectory having a piecewise-constant control function and positive clearance from the obstacles can be approximated arbitrarily well by a trajectory found by AO-RRT. We also discuss practical aspects of AO-RRT and present experimental comparisons of variants of the algorithm. }, keywords = {}, pubstate = {published}, tppubtype = {conference} } We present a novel analysis of AO-RRT: a tree-based planner for motion planning with kinodynamic constraints, originally described by Hauser and Zhou (AO-X, 2016). AO-RRT explores the state-cost space and has been shown to efficiently obtain high-quality solutions in practice without relying on the availability of a computationally-intensive two-point boundary-value solver. Our main contribution is an optimality proof for the single-tree version of the algorithm---a variant that was not analyzed before. Our proof only requires a mild and easily-verifiable set of assumptions on the problem and system: Lipschitz-continuity of the cost function and the dynamics. In particular, we prove that for any system satisfying these assumptions, any trajectory having a piecewise-constant control function and positive clearance from the obstacles can be approximated arbitrarily well by a trajectory found by AO-RRT. We also discuss practical aspects of AO-RRT and present experimental comparisons of variants of the algorithm. |
Wen, B; Mitash, C; Soorian, S; Kimmel, A; Sintov, A; Bekris, K E Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands Conference IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020. Abstract | Links | BibTeX | Tags: @conference{220, title = {Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands}, author = {B Wen and C Mitash and S Soorian and A Kimmel and A Sintov and K E Bekris}, url = {https://arxiv.org/abs/2003.03518}, year = {2020}, date = {2020-06-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, address = {Paris, France}, abstract = {Many manipulation tasks, such as placement or within-hand manipulation, require the object’s pose relative to a robot hand. The task is difficult when the hand significantly occludes the object. It is especially hard for adaptive hands, for which it is not easy to detect the finger’s configuration. In addition, RGB-only approaches face issues with texture-less objects or when the hand and the object look similar. This paper presents a depth-based framework, which aims for robust pose estimation and short response times. The approach detects the adaptive hand’s state via efficient parallel search given the highest overlap between the hand’s model and the point cloud. The hand’s point cloud is pruned and robust global registration is performed to generate object pose hypotheses, which are clustered. False hypotheses are pruned via physical reasoning. The remaining poses’ quality is evaluated given agreement with observed data. Extensive evaluation on synthetic and real data demonstrates the accuracy and computational efficiency of the framework when applied on challenging, highly-occluded scenarios for different object types. An ablation study identifies how the framework’s components help in performance. This work also provides a dataset for in-hand 6D object pose esti- mation. Code and dataset are available at: https://github. com/wenbowen123/icra20-hand-object-pose}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Many manipulation tasks, such as placement or within-hand manipulation, require the object’s pose relative to a robot hand. The task is difficult when the hand significantly occludes the object. It is especially hard for adaptive hands, for which it is not easy to detect the finger’s configuration. In addition, RGB-only approaches face issues with texture-less objects or when the hand and the object look similar. This paper presents a depth-based framework, which aims for robust pose estimation and short response times. The approach detects the adaptive hand’s state via efficient parallel search given the highest overlap between the hand’s model and the point cloud. The hand’s point cloud is pruned and robust global registration is performed to generate object pose hypotheses, which are clustered. False hypotheses are pruned via physical reasoning. The remaining poses’ quality is evaluated given agreement with observed data. Extensive evaluation on synthetic and real data demonstrates the accuracy and computational efficiency of the framework when applied on challenging, highly-occluded scenarios for different object types. An ablation study identifies how the framework’s components help in performance. This work also provides a dataset for in-hand 6D object pose esti- mation. Code and dataset are available at: https://github. com/wenbowen123/icra20-hand-object-pose |
Littlefield, Z Efficient and Asymptotically Optimal Kinodynamic Motion Planning PhD Thesis Rutgers, the State University of New Jersey, 2020. Abstract | Links | BibTeX | Tags: @phdthesis{230, title = {Efficient and Asymptotically Optimal Kinodynamic Motion Planning}, author = {Z Littlefield}, url = {http://www.cs.rutgers.edu/~kb572/pubs/LittlefieldThesisMay2020.pdf}, year = {2020}, date = {2020-05-01}, volume = {PhD}, school = {Rutgers, the State University of New Jersey}, abstract = {This dissertation explores properties of motion planners that build tree data structures in a robottextquoterights state space. Sampling-based tree planners are especially useful for planning for systems with significant dynamics, due to the inherent forward search that is per- formed. This is in contrast to roadmap planners that require a steering local planner in order to make a graph containing multiple possible paths. This dissertation explores a family of motion planners for systems with significant dynamics, where a steering local planner may be computationally expensive or may not exist. These planners fo- cus on providing practical path quality guarantees without prohibitive computational costs. These planners can be considered successors of each other, in that each sub- sequent algorithm addresses some drawback of its predecessor. The first algorithm, Sparse-RRT, addresses a drawback of the RRT method by considering path quality dur- ing the tree construction process. Sparse-RRT is proven to be probabilistically complete under mild conditions for the first time here, albeit with a poor convergence rate. The second algorithm presented, SST, provides probabilistic completeness and asymptotic near-optimality properties that are provable, but at the cost of additional algorithmic overhead. SST is shown to improve the convergence rate compared to Sparse-RRT. The third algorithm, DIRT, incorporates learned lessons from these two algorithms and their shortcomings, incorporates task space heuristics to further improve runtime per- formance, and simplifies the parameters to more user-friendly ones. DIRT is also shown to be probabilistically complete and asymptotically near-optimal. Application areas explored using this family of algorithms include evaluation of distance functions for planning in belief space, manipulation in cluttered environments, and locomotion plan- ning for an icosahedral tensegrity-based rover prototype that requires a physics engine to simulate its motions.}, keywords = {}, pubstate = {published}, tppubtype = {phdthesis} } This dissertation explores properties of motion planners that build tree data structures in a robottextquoterights state space. Sampling-based tree planners are especially useful for planning for systems with significant dynamics, due to the inherent forward search that is per- formed. This is in contrast to roadmap planners that require a steering local planner in order to make a graph containing multiple possible paths. This dissertation explores a family of motion planners for systems with significant dynamics, where a steering local planner may be computationally expensive or may not exist. These planners fo- cus on providing practical path quality guarantees without prohibitive computational costs. These planners can be considered successors of each other, in that each sub- sequent algorithm addresses some drawback of its predecessor. The first algorithm, Sparse-RRT, addresses a drawback of the RRT method by considering path quality dur- ing the tree construction process. Sparse-RRT is proven to be probabilistically complete under mild conditions for the first time here, albeit with a poor convergence rate. The second algorithm presented, SST, provides probabilistic completeness and asymptotic near-optimality properties that are provable, but at the cost of additional algorithmic overhead. SST is shown to improve the convergence rate compared to Sparse-RRT. The third algorithm, DIRT, incorporates learned lessons from these two algorithms and their shortcomings, incorporates task space heuristics to further improve runtime per- formance, and simplifies the parameters to more user-friendly ones. DIRT is also shown to be probabilistically complete and asymptotically near-optimal. Application areas explored using this family of algorithms include evaluation of distance functions for planning in belief space, manipulation in cluttered environments, and locomotion plan- ning for an icosahedral tensegrity-based rover prototype that requires a physics engine to simulate its motions. |
Shome, R Rutgers University, 2020. @phdthesis{226, title = {The Problem of Many: Efficient Multi-arm, Multi-Object Task and and Motion Planning with Optimality Guarantees}, author = {R Shome}, url = {http://www.cs.rutgers.edu/~kb572/pubs/rahul_shome_thesis.pdf}, year = {2020}, date = {2020-04-01}, volume = {PhD}, address = {New Brunswick, NJ}, school = {Rutgers University}, keywords = {}, pubstate = {published}, tppubtype = {phdthesis} } |
Alikhani, M; Khalid, B; Shome, R; Mitash, C; Bekris, K E; Stone, M That and There: Judging the Intent of Pointing Actions with Robotic Arms Conference Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), New York, NY, 2020. Abstract | Links | BibTeX | Tags: @conference{218, title = {That and There: Judging the Intent of Pointing Actions with Robotic Arms}, author = {M Alikhani and B Khalid and R Shome and C Mitash and K E Bekris and M Stone}, url = {https://arxiv.org/abs/1912.06602}, year = {2020}, date = {2020-02-01}, booktitle = {Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)}, journal = {Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)}, address = {New York, NY}, abstract = {Collaborative robotics requires effective communication between a robot and a human partner. This work proposes a set of interpretive principles for how a robotic arm can use pointing actions to communicate task information to people by extending existing models from the related literature. These principles are evaluated through studies where English-speaking human subjects view animations of simulated robots instructing pick-and-place tasks. The evaluation distinguishes two classes of pointing actions that arise in pick-and-place tasks: referential pointing (identifying objects) and spatial pointing (identifying locations). The study indicates that human subjects show greater flexibility in interpreting the intent of referential pointing compared to spatial pointing, which needs to be more deliberate. The results also demonstrate the effects of variation in the environment and task context on the interpretation of pointing. The corpus and the experiments described in this work can impact models of context and coordination as well as the effect of common sense reasoning in human-robot interactions.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Collaborative robotics requires effective communication between a robot and a human partner. This work proposes a set of interpretive principles for how a robotic arm can use pointing actions to communicate task information to people by extending existing models from the related literature. These principles are evaluated through studies where English-speaking human subjects view animations of simulated robots instructing pick-and-place tasks. The evaluation distinguishes two classes of pointing actions that arise in pick-and-place tasks: referential pointing (identifying objects) and spatial pointing (identifying locations). The study indicates that human subjects show greater flexibility in interpreting the intent of referential pointing compared to spatial pointing, which needs to be more deliberate. The results also demonstrate the effects of variation in the environment and task context on the interpretation of pointing. The corpus and the experiments described in this work can impact models of context and coordination as well as the effect of common sense reasoning in human-robot interactions. |
Goldberg, K; Abbeel, P; Bekris, K E; Miller, L Algorithmic Foundations of Robotics XII Book Springer, 2020, ISBN: 978-3-030-43089-4. Abstract | Links | BibTeX | Tags: @book{227, title = {Algorithmic Foundations of Robotics XII}, author = {K Goldberg and P Abbeel and K E Bekris and L Miller}, url = {https://link.springer.com/book/10.1007/978-3-030-43089-4}, isbn = {978-3-030-43089-4}, year = {2020}, date = {2020-01-01}, volume = {13}, publisher = {Springer}, organization = {Springer}, series = {Springer Proceedings in Advanced Robotics}, abstract = {Robotics is reaching an elevated level of maturity and continues to benefit from the advances and innovations in its enabling technologies. These all are contributing to an unprecedented effort to bringing robots to the human environment in hospitals and homes, factories, and schools, in the field for robots fighting fires, making goods and products, picking fruits and watering the farmland, saving time and lives. Robots today hold the promise for making a considerable impact in a wide range of real-world applications from industrial manufacturing to health care, transportation, and exploration of the deep space and sea. Tomorrow, robots will become pervasive and touch upon many aspects of modern life. The Springer Tracts in Advanced Robotics (STAR) was launched in 2002 with the goal of bringing to the research community the latest advances in the robotics field based on their significance and quality. During the latest fifteen years, the STAR series has featured publication of both monographs and edited collections. Among the latter, the proceedings of thematic symposia devoted to excellence in robotics research, such as ISRR, ISER, FSR, and WAFR, have been regularly included in STAR. The expansion of our field as well as the emergence of new research areas has motivated us to enlarge the pool of proceedings in the STAR series in the past few years. This has ultimately led to launching a sister series in parallel to STAR. The Springer Proceedings in Advanced Robotics (SPAR) is dedicated to the timely dissemination of the latest research results presented in selected symposia and workshops. This volume of the SPAR series brings the proceedings of the twelfth edition of the Workshop Algorithmic Foundations of Robotics (WAFR). WAFR went back to its roots and was held from December 18 to 20, 2016, in San Francisco, California, the same city in which the very first WAFR was held in 1994. The volume edited by Ken Goldberg, Pieter Abbeel, Kostas Bekris, and Lauren Miller is a collection of 58 contributions spanning a wide range of applications in manufacturing, medicine, distributed robotics, human–robot interaction, intelligent prosthetics, computer animation, computational biology, and many other areas. Validation of algorithms, design concepts, or techniques is the common thread running through this focused collection. Rich by topics and authoritative contributors, WAFR culminates with this unique reference on the current developments and new directions in the field of algorithmic foundations. A very fine addition to the series! More information about the conference, including videos of the presentations can be found under the conferencetextquoterights website: http://wafr2016.berkeley.edu}, keywords = {}, pubstate = {published}, tppubtype = {book} } Robotics is reaching an elevated level of maturity and continues to benefit from the advances and innovations in its enabling technologies. These all are contributing to an unprecedented effort to bringing robots to the human environment in hospitals and homes, factories, and schools, in the field for robots fighting fires, making goods and products, picking fruits and watering the farmland, saving time and lives. Robots today hold the promise for making a considerable impact in a wide range of real-world applications from industrial manufacturing to health care, transportation, and exploration of the deep space and sea. Tomorrow, robots will become pervasive and touch upon many aspects of modern life. The Springer Tracts in Advanced Robotics (STAR) was launched in 2002 with the goal of bringing to the research community the latest advances in the robotics field based on their significance and quality. During the latest fifteen years, the STAR series has featured publication of both monographs and edited collections. Among the latter, the proceedings of thematic symposia devoted to excellence in robotics research, such as ISRR, ISER, FSR, and WAFR, have been regularly included in STAR. The expansion of our field as well as the emergence of new research areas has motivated us to enlarge the pool of proceedings in the STAR series in the past few years. This has ultimately led to launching a sister series in parallel to STAR. The Springer Proceedings in Advanced Robotics (SPAR) is dedicated to the timely dissemination of the latest research results presented in selected symposia and workshops. This volume of the SPAR series brings the proceedings of the twelfth edition of the Workshop Algorithmic Foundations of Robotics (WAFR). WAFR went back to its roots and was held from December 18 to 20, 2016, in San Francisco, California, the same city in which the very first WAFR was held in 1994. The volume edited by Ken Goldberg, Pieter Abbeel, Kostas Bekris, and Lauren Miller is a collection of 58 contributions spanning a wide range of applications in manufacturing, medicine, distributed robotics, human–robot interaction, intelligent prosthetics, computer animation, computational biology, and many other areas. Validation of algorithms, design concepts, or techniques is the common thread running through this focused collection. Rich by topics and authoritative contributors, WAFR culminates with this unique reference on the current developments and new directions in the field of algorithmic foundations. A very fine addition to the series! More information about the conference, including videos of the presentations can be found under the conferencetextquoterights website: http://wafr2016.berkeley.edu |
Bekris, K E; Shome, R Asymptotically Optimal Sampling-based Planners Book Chapter Encyclopedia of Robotics, 2020. Abstract | Links | BibTeX | Tags: @inbook{219, title = {Asymptotically Optimal Sampling-based Planners}, author = {K E Bekris and R Shome}, url = {https://arxiv.org/abs/1911.04044}, year = {2020}, date = {2020-01-01}, booktitle = {Encyclopedia of Robotics}, abstract = {An asymptotically optimal sampling-based planner employs sampling to solve robot motion planning problems and returns paths with a cost that converges to the optimal solution cost, as the number of samples approaches infinity. This comprehensive article covers the theoretical characteristics of asymptotic optimality of motion planning algorithms, and traces its origins, analysis models, practical performance, extensions, and applications. }, keywords = {}, pubstate = {published}, tppubtype = {inbook} } An asymptotically optimal sampling-based planner employs sampling to solve robot motion planning problems and returns paths with a cost that converges to the optimal solution cost, as the number of samples approaches infinity. This comprehensive article covers the theoretical characteristics of asymptotic optimality of motion planning algorithms, and traces its origins, analysis models, practical performance, extensions, and applications. |
2019 |
Kimmel, A; Shome, R; Bekris, K E Anytime Motion Planning for Prehensile Manipulation in Dense Clutter Journal Article Advanced Robotics, 2019. Abstract | Links | BibTeX | Tags: @article{217, title = {Anytime Motion Planning for Prehensile Manipulation in Dense Clutter}, author = {A Kimmel and R Shome and K E Bekris}, url = {https://www.rahulsho.me/papers/ar_gmp.pdf}, year = {2019}, date = {2019-11-17}, journal = {Advanced Robotics}, abstract = {Many methods have been developed for planning the motion of robotic arms for picking and placing, ranging from local optimization to global search techniques, which are effective for sparsely placed objects. Dense clutter, however, still adversely affects the success rate, computation times, and quality of solutions in many real-world setups. The proposed method achieves high success ratio in clutter with anytime performance by returning solutions quickly and improving their quality over time. The method first explores the lower dimensional end effector’s task space efficiently by ignoring the arm, and build a discrete approximation of a navigation function. This is performed online, without prior knowledge of the scene. Then, an informed sampling-based planner for the entire arm uses Jacobian- based steering to reach promising end effector poses given the task space guidance. This process is also comprehensive and allows the exploration of alternative paths over time if the task space guidance is misleading. This paper evaluates the proposed method against alternatives in picking or placing tasks among varying amounts of clutter for a variety of robotic manipulators with different end-effectors. The results suggest that the method reliably provides higher quality solution paths quicker, with a higher success rate relative to alternatives. }, keywords = {}, pubstate = {published}, tppubtype = {article} } Many methods have been developed for planning the motion of robotic arms for picking and placing, ranging from local optimization to global search techniques, which are effective for sparsely placed objects. Dense clutter, however, still adversely affects the success rate, computation times, and quality of solutions in many real-world setups. The proposed method achieves high success ratio in clutter with anytime performance by returning solutions quickly and improving their quality over time. The method first explores the lower dimensional end effector’s task space efficiently by ignoring the arm, and build a discrete approximation of a navigation function. This is performed online, without prior knowledge of the scene. Then, an informed sampling-based planner for the entire arm uses Jacobian- based steering to reach promising end effector poses given the task space guidance. This process is also comprehensive and allows the exploration of alternative paths over time if the task space guidance is misleading. This paper evaluates the proposed method against alternatives in picking or placing tasks among varying amounts of clutter for a variety of robotic manipulators with different end-effectors. The results suggest that the method reliably provides higher quality solution paths quicker, with a higher success rate relative to alternatives. |
Kimmel, A; Sintov, A; Tan, J; Wen, B; Boularias, A; Bekris, K E Belief-Space Planning using Learned Models with Application to Underactuated Hands Conference International Symposium on Robotics Research (ISRR), Hanoi, Vietnam, 2019. Abstract | Links | BibTeX | Tags: @conference{213, title = {Belief-Space Planning using Learned Models with Application to Underactuated Hands}, author = {A Kimmel and A Sintov and J Tan and B Wen and A Boularias and K E Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/belief_space_learned_models_adaptive_hands.pdf}, year = {2019}, date = {2019-10-01}, booktitle = {International Symposium on Robotics Research (ISRR)}, address = {Hanoi, Vietnam}, abstract = {Acquiring a precise model is a challenging task for many important robotic tasks and systems - including in-hand manipulation using underactuated, adaptive hands. Learning stochastic, data-driven models is a promising alternative as they provide not only a way to propagate forward the system dynamics, but also express the uncertainty present in the collected data. Therefore, such models en- able planning in the space of state distributions, i.e., in the belief space. This paper proposes a planning framework that employs stochastic, learned models, which ex- press a distribution of states as a set of particles. The integration achieves anytime behavior in terms of returning paths of increasing quality under constraints for the probability of success to achieve a goal. The focus of this effort is on pushing the efficiency of the overall methodology despite the notorious computational hardness of belief-space planning. Experiments show that the proposed framework enables reaching a desired goal with higher success rate compared to alternatives in sim- ple benchmarks. This work also provides an application to the motivating domain of in-hand manipulation with underactuated, adaptive hands, both in the case of physically-simulated experiments as well as demonstrations with a real hand.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Acquiring a precise model is a challenging task for many important robotic tasks and systems - including in-hand manipulation using underactuated, adaptive hands. Learning stochastic, data-driven models is a promising alternative as they provide not only a way to propagate forward the system dynamics, but also express the uncertainty present in the collected data. Therefore, such models en- able planning in the space of state distributions, i.e., in the belief space. This paper proposes a planning framework that employs stochastic, learned models, which ex- press a distribution of states as a set of particles. The integration achieves anytime behavior in terms of returning paths of increasing quality under constraints for the probability of success to achieve a goal. The focus of this effort is on pushing the efficiency of the overall methodology despite the notorious computational hardness of belief-space planning. Experiments show that the proposed framework enables reaching a desired goal with higher success rate compared to alternatives in sim- ple benchmarks. This work also provides an application to the motivating domain of in-hand manipulation with underactuated, adaptive hands, both in the case of physically-simulated experiments as well as demonstrations with a real hand. |
Mitash, C; Wen, B; Bekris, K E; Boularias, A Scene-level Pose Estimation for Multiple Instances of Densely Packed Objects Conference Conference on Robot Learning (CoRL), Osaka, Japan, 2019. Abstract | Links | BibTeX | Tags: @conference{214, title = {Scene-level Pose Estimation for Multiple Instances of Densely Packed Objects}, author = {C Mitash and B Wen and K E Bekris and A Boularias}, url = {https://arxiv.org/pdf/1910.04953.pdf}, year = {2019}, date = {2019-10-01}, booktitle = {Conference on Robot Learning (CoRL)}, address = {Osaka, Japan}, abstract = {This paper introduces key machine learning operations that allow the realization of robust, joint 6D pose estimation of multiple instances of objects either densely packed or in unstructured piles from RGB-D data. The first objective is to learn semantic and instance-boundary detectors without manual labeling. An adversarial training framework in conjunction with physics-based simulation is used to achieve detectors that behave similarly in synthetic and real data. Given the stochastic output of such detectors, candidates for object poses are sampled. The second objective is to automatically learn a single score for each pose candidate that represents its quality in terms of explaining the entire scene via a gradient boosted tree. The proposed method uses features derived from surface and boundary alignment between the observed scene and the object model placed at hypothesized poses. Scene-level, multi-instance pose estimation is then achieved by an integer linear programming process that selects hypotheses that maximize the sum of the learned individual scores, while respecting constraints, such as avoiding collisions. To evaluate this method, a dataset of densely packed objects with challenging setups for state-of-the-art approaches is collected. Experiments on this dataset and a public one show that the method significantly outperforms alternatives in terms of 6D pose accuracy while trained only with synthetic datasets.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } This paper introduces key machine learning operations that allow the realization of robust, joint 6D pose estimation of multiple instances of objects either densely packed or in unstructured piles from RGB-D data. The first objective is to learn semantic and instance-boundary detectors without manual labeling. An adversarial training framework in conjunction with physics-based simulation is used to achieve detectors that behave similarly in synthetic and real data. Given the stochastic output of such detectors, candidates for object poses are sampled. The second objective is to automatically learn a single score for each pose candidate that represents its quality in terms of explaining the entire scene via a gradient boosted tree. The proposed method uses features derived from surface and boundary alignment between the observed scene and the object model placed at hypothesized poses. Scene-level, multi-instance pose estimation is then achieved by an integer linear programming process that selects hypotheses that maximize the sum of the learned individual scores, while respecting constraints, such as avoiding collisions. To evaluate this method, a dataset of densely packed objects with challenging setups for state-of-the-art approaches is collected. Experiments on this dataset and a public one show that the method significantly outperforms alternatives in terms of 6D pose accuracy while trained only with synthetic datasets. |
Sivaramakrishnan, A; Littlefield, Z; Bekris, K E Towards Learning Efficient Maneuver Sets for Kinodynamic Motion Planning Technical Report PlanRob 2019 Workshop of ICAPS 2019 2019. Abstract | Links | BibTeX | Tags: @techreport{211, title = {Towards Learning Efficient Maneuver Sets for Kinodynamic Motion Planning}, author = {A Sivaramakrishnan and Z Littlefield and K E Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/Learning_Maneuver_Sets_PlanRob2019.pdf}, year = {2019}, date = {2019-07-01}, institution = {PlanRob 2019 Workshop of ICAPS 2019}, abstract = {Planning for systems with dynamics is challenging as often there is no local planner available and the only primitive to explore the state space is forward propagation of controls. In this context, tree sampling-based planners have been developed, some of which achieve asymptotic optimality by propagating random controls during each iteration. While desirable for the analysis, random controls result in slow convergence to high quality trajectories in practice. This short position statement first argues that if a kinodynamic planner has access to local maneuvers that appropriately balance an exploitation-exploration trade-off, the plannertextquoterights per iteration performance is significantly improved. Furthermore, this work argues for the integration of modern machine learning frameworks with state-of-the-art, informed and asymptotically optimal kinodynamic planners. The proposed approach involves using using neural networks to infer local maneuvers for a robotic system with dynamics, which properly balance the above exploitation-exploration trade-off. Preliminary indications in simulated environments and systems are promising but also point to certain challenges that motivate further research in this direction.}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } Planning for systems with dynamics is challenging as often there is no local planner available and the only primitive to explore the state space is forward propagation of controls. In this context, tree sampling-based planners have been developed, some of which achieve asymptotic optimality by propagating random controls during each iteration. While desirable for the analysis, random controls result in slow convergence to high quality trajectories in practice. This short position statement first argues that if a kinodynamic planner has access to local maneuvers that appropriately balance an exploitation-exploration trade-off, the plannertextquoterights per iteration performance is significantly improved. Furthermore, this work argues for the integration of modern machine learning frameworks with state-of-the-art, informed and asymptotically optimal kinodynamic planners. The proposed approach involves using using neural networks to infer local maneuvers for a robotic system with dynamics, which properly balance the above exploitation-exploration trade-off. Preliminary indications in simulated environments and systems are promising but also point to certain challenges that motivate further research in this direction. |
Shome, R; Bekris, K E Anytime Multi-arm Task and Motion Planning for Pick-and-Place of Individual Objects via Handoffs Conference IEEE International Conference on Multi-Robot and Multi-Agent Systems (MRS), New Brunswick, NJ, 2019. Abstract | Links | BibTeX | Tags: @conference{212, title = {Anytime Multi-arm Task and Motion Planning for Pick-and-Place of Individual Objects via Handoffs}, author = {R Shome and K E Bekris}, url = {https://arxiv.org/abs/1905.03179}, year = {2019}, date = {2019-06-01}, booktitle = {IEEE International Conference on Multi-Robot and Multi-Agent Systems (MRS)}, address = {New Brunswick, NJ}, abstract = {Automation applications are pushing the deployment of many high DoF manipulators in warehouse and manufacturing environments. This has motivated many efforts on optimizing manipulation tasks involving a single arm. Coordinating multiple arms for manipulation, however, introduces additional computational challenges arising from the increased DoFs, as well as the combinatorial increase in the available operations that many manipulators can perform, including handoffs between arms. The focus here is on the case of pick-and-place tasks, which require a sequence of handoffs to be executed, so as to achieve computational efficiency, asymptotic optimality and practical anytime performance. The paper leverages recent advances in multi-robot motion planning for high DoF systems to propose a novel multi-modal extension of the dRRT* algorithm. The key insight is that, instead of naively solving a sequence of motion planning problems, it is computationally advantageous to directly explore the composite space of the integrated multi-arm task and motion planning problem, given input sets of possible pick and handoff configurations. Asymptotic optimality guarantees are possible by sampling additional picks and handoffs over time. The evaluation shows that the approach finds initial solutions fast and improves their quality over time. It also succeeds in finding solutions to harder problem instances relative to alternatives and can scale effectively as the number of robots increases.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Automation applications are pushing the deployment of many high DoF manipulators in warehouse and manufacturing environments. This has motivated many efforts on optimizing manipulation tasks involving a single arm. Coordinating multiple arms for manipulation, however, introduces additional computational challenges arising from the increased DoFs, as well as the combinatorial increase in the available operations that many manipulators can perform, including handoffs between arms. The focus here is on the case of pick-and-place tasks, which require a sequence of handoffs to be executed, so as to achieve computational efficiency, asymptotic optimality and practical anytime performance. The paper leverages recent advances in multi-robot motion planning for high DoF systems to propose a novel multi-modal extension of the dRRT* algorithm. The key insight is that, instead of naively solving a sequence of motion planning problems, it is computationally advantageous to directly explore the composite space of the integrated multi-arm task and motion planning problem, given input sets of possible pick and handoff configurations. Asymptotic optimality guarantees are possible by sampling additional picks and handoffs over time. The evaluation shows that the approach finds initial solutions fast and improves their quality over time. It also succeeds in finding solutions to harder problem instances relative to alternatives and can scale effectively as the number of robots increases. |
Sintov, A; Morgan, A; Kimmel, A; Dollar, A; Bekris, K E; Boularias, A Learning a State Transition Model of an Underactuated Adaptive Hand Journal Article IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE ICRA 2019), 2019. Abstract | Links | BibTeX | Tags: @article{205, title = {Learning a State Transition Model of an Underactuated Adaptive Hand}, author = {A Sintov and A Morgan and A Kimmel and A Dollar and K E Bekris and A Boularias}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Learning_a_State_Transition_Model.pdf}, year = {2019}, date = {2019-05-01}, journal = {IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE ICRA 2019)}, abstract = {Fully-actuated, multi-fingered robotic hands are often expensive and fragile. Low-cost, under-actuated hands are appealing but present challenges due to the lack of analytical models. This paper aims to learn a stochastic version of such models automatically from data with minimum user effort. The focus is on identifying the dominant, sensible features required to express hand state transitions given quasi-static motions, thereby enabling the learning of a probabilistic transition model from recorded trajectories. Experiments both with Gaussian Processes (GP) and Neural Network models are included for analysis and evaluation. The metric for local GP regression is obtained with a manifold learning approach, known as "Diffusion Maps", to uncover the lower-dimensional subspace in which the data lies and provide a geodesic metric. Results show that using Diffusion Maps with a feature space composed of the object position, actuator angles, and actuator loads, sufficiently expresses the hand-object system "configuration and can provide accurate enough predictions for a relatively long horizon. To the best of the authorstextquoteright knowledge, this is the first learned transition model for such underactuated hands that achieves this level of predictability. Notably, the same feature space implicitly embeds the size of the manipulated object and can generalize to new objects of varying sizes. Furthermore, the learned model can identify states that are on the verge of failure and which should be avoided during manipulation. The usefulness of the model is also demonstrated by integrating it with closed-loop control to successfully and safely complete manipulation tasks.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Fully-actuated, multi-fingered robotic hands are often expensive and fragile. Low-cost, under-actuated hands are appealing but present challenges due to the lack of analytical models. This paper aims to learn a stochastic version of such models automatically from data with minimum user effort. The focus is on identifying the dominant, sensible features required to express hand state transitions given quasi-static motions, thereby enabling the learning of a probabilistic transition model from recorded trajectories. Experiments both with Gaussian Processes (GP) and Neural Network models are included for analysis and evaluation. The metric for local GP regression is obtained with a manifold learning approach, known as "Diffusion Maps", to uncover the lower-dimensional subspace in which the data lies and provide a geodesic metric. Results show that using Diffusion Maps with a feature space composed of the object position, actuator angles, and actuator loads, sufficiently expresses the hand-object system "configuration and can provide accurate enough predictions for a relatively long horizon. To the best of the authorstextquoteright knowledge, this is the first learned transition model for such underactuated hands that achieves this level of predictability. Notably, the same feature space implicitly embeds the size of the manipulated object and can generalize to new objects of varying sizes. Furthermore, the learned model can identify states that are on the verge of failure and which should be avoided during manipulation. The usefulness of the model is also demonstrated by integrating it with closed-loop control to successfully and safely complete manipulation tasks. |
Shome, R; Tang, W N; Song, C; Mitash, C; Kourtev, C; Yu, J; Boularias, A; Bekris, K E Towards Robust Product Packing with a Minimalistic End-Effector Conference IEEE International Conference on Robotics and Automation (ICRA), Nomination for Best Paper Award in Automation Nomination for Best Paper Award in Automation, Montreal, Canada, 2019. Abstract | Links | BibTeX | Tags: @conference{207, title = {Towards Robust Product Packing with a Minimalistic End-Effector}, author = {R Shome and W N Tang and C Song and C Mitash and C Kourtev and J Yu and A Boularias and K E Bekris}, url = {http://robotpacking.org/}, year = {2019}, date = {2019-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, publisher = {Nomination for Best Paper Award in Automation}, address = {Montreal, Canada}, organization = {Nomination for Best Paper Award in Automation}, abstract = {Advances in sensor technologies, object detection algorithms, planning frameworks and hardware designs have motivated the deployment of robots in warehouse automation. A variety of such applications, like order fulfillment or packing tasks, require picking objects from unstructured piles and carefully arranging them in bins or containers. Desirable solutions need to be low-cost, easily deployable and controllable, making minimalistic hardware choices desirable. The challenge in designing an effective solution to this problem relates to appropriately integrating multiple components, so as to achieve a robust pipeline that minimizes failure conditions. The current work proposes a complete pipeline for solving such packing tasks, given access only to RGB-D data and a single robot arm with a minimalistic, vacuum-based end-effector. To achieve the desired level of robustness, three key manipulation primitives are identified, which take advantage of the environment and simple operations to successfully pack multiple cubic objects. The overall approach is demonstrated to be robust to execution and perception errors. The impact of each manipulation primitive is evaluated by considering different versions of the proposed pipeline that incrementally introduce reasoning about object poses and corrective manipulation actions.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Advances in sensor technologies, object detection algorithms, planning frameworks and hardware designs have motivated the deployment of robots in warehouse automation. A variety of such applications, like order fulfillment or packing tasks, require picking objects from unstructured piles and carefully arranging them in bins or containers. Desirable solutions need to be low-cost, easily deployable and controllable, making minimalistic hardware choices desirable. The challenge in designing an effective solution to this problem relates to appropriately integrating multiple components, so as to achieve a robust pipeline that minimizes failure conditions. The current work proposes a complete pipeline for solving such packing tasks, given access only to RGB-D data and a single robot arm with a minimalistic, vacuum-based end-effector. To achieve the desired level of robustness, three key manipulation primitives are identified, which take advantage of the environment and simple operations to successfully pack multiple cubic objects. The overall approach is demonstrated to be robust to execution and perception errors. The impact of each manipulation primitive is evaluated by considering different versions of the proposed pipeline that incrementally introduce reasoning about object poses and corrective manipulation actions. |
Feld-Cook, E; Shome, R; Zaleski, R; Mohan, K; Kourtev, C; Bekris, K E; Weiseil, C; Shin, J Exploring the Utility of Robots in Exposure Studies Journal Article Journal of Exposure Science and Environmental Epidemiology (JESEE), 2019. Abstract | Links | BibTeX | Tags: @article{215, title = {Exploring the Utility of Robots in Exposure Studies}, author = {E Feld-Cook and R Shome and R Zaleski and K Mohan and C Kourtev and K E Bekris and C Weiseil and J Shin}, url = {https://www.nature.com/articles/s41370-019-0190-x}, year = {2019}, date = {2019-01-01}, journal = {Journal of Exposure Science and Environmental Epidemiology (JESEE)}, abstract = {Advancements in robotic technology continue to help expand the use of robots in the workplace, research, and society. In this proof-of-concept study, a robotic platform was programmed to do a simple task, painting drywall, to help determine if robots are a plausible alternative to human subjects in exposure studies. For the exposure component, passive and active air samplers and direct-read monitors were placed by the robot to measure VOCs emitted from the paint and later compared to modeled estimates. A strong correlation of R2 = 0.85- 0.89 was found between increased paint used and increased total VOC air concentrations. Similar trends were observed for all painting trials for the direct read monitors with an overall low VOC air concentration (< 4 ppm), indicating a low exposure profile. Consistent results for the front (60.1 textpm 2.5 cm by 77.5 textpm 0.85 cm) and sides (60.1 textpm 2.5 cm by 60.1 textpm 2.9 cm) painted by the robot, the resulting exposure, and the amount of paint used per trial suggest that using a robot to perform an exposure study was successful. This study demonstrated how robots, compared to human subjects, are quicker and reliable way to perform exposure studies.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Advancements in robotic technology continue to help expand the use of robots in the workplace, research, and society. In this proof-of-concept study, a robotic platform was programmed to do a simple task, painting drywall, to help determine if robots are a plausible alternative to human subjects in exposure studies. For the exposure component, passive and active air samplers and direct-read monitors were placed by the robot to measure VOCs emitted from the paint and later compared to modeled estimates. A strong correlation of R2 = 0.85- 0.89 was found between increased paint used and increased total VOC air concentrations. Similar trends were observed for all painting trials for the direct read monitors with an overall low VOC air concentration (< 4 ppm), indicating a low exposure profile. Consistent results for the front (60.1 textpm 2.5 cm by 77.5 textpm 0.85 cm) and sides (60.1 textpm 2.5 cm by 60.1 textpm 2.9 cm) painted by the robot, the resulting exposure, and the amount of paint used per trial suggest that using a robot to perform an exposure study was successful. This study demonstrated how robots, compared to human subjects, are quicker and reliable way to perform exposure studies. |
Ricks, B; Dobson, A; Krontiris, A; Bekris, K E; Kapadia, M; Roberts, F Generation of Crowd Arrival and Destination Locations/Times in Complex Transit Facilities Journal Article The Visual Computer Journal, 2019. Abstract | Links | BibTeX | Tags: @article{216, title = {Generation of Crowd Arrival and Destination Locations/Times in Complex Transit Facilities}, author = {B Ricks and A Dobson and A Krontiris and K E Bekris and M Kapadia and F Roberts}, url = {https://rdcu.be/bUBid}, year = {2019}, date = {2019-01-01}, journal = {The Visual Computer Journal}, abstract = {In order to simulate virtual agents in the replica of a real facility across a long time span, a crowd simulation engine needs a list of agent arrival and destination locations and times that reflect those seen in the actual facility. Working together with a major metropolitan transportation authority, we pro- pose a specification that can be used to procedurally generate this information. This specification is both uniquely compact and expressive—compact enough to mirror the mental model of building managers and ex- pressive enough to handle the wide variety of crowds seen in real urban environments. We also propose a procedural algorithm for generating tens of thousands of high-level agent paths from this specification. This algorithm allows our specification to be used with tradi- tional crowd simulation obstacle avoidance algorithms while still maintaining the realism required for the com- plex, real-world simulations of a transit facility. Our evaluation with industry professionals shows that our approach is intuitive and provides controls at the right level of detail to be used in large facilities (200,000+ people/day).}, keywords = {}, pubstate = {published}, tppubtype = {article} } In order to simulate virtual agents in the replica of a real facility across a long time span, a crowd simulation engine needs a list of agent arrival and destination locations and times that reflect those seen in the actual facility. Working together with a major metropolitan transportation authority, we pro- pose a specification that can be used to procedurally generate this information. This specification is both uniquely compact and expressive—compact enough to mirror the mental model of building managers and ex- pressive enough to handle the wide variety of crowds seen in real urban environments. We also propose a procedural algorithm for generating tens of thousands of high-level agent paths from this specification. This algorithm allows our specification to be used with tradi- tional crowd simulation obstacle avoidance algorithms while still maintaining the realism required for the com- plex, real-world simulations of a transit facility. Our evaluation with industry professionals shows that our approach is intuitive and provides controls at the right level of detail to be used in large facilities (200,000+ people/day). |
Kleinbort, M; Solovey, K; Littlefield, Z; Bekris, K E; Halperin, D Probabilistic completeness of RRT for geometric and kinodynamic planning with forward propagation Journal Article IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE ICRA 2019), 2019. Abstract | Links | BibTeX | Tags: @article{202, title = {Probabilistic completeness of RRT for geometric and kinodynamic planning with forward propagation}, author = {M Kleinbort and K Solovey and Z Littlefield and K E Bekris and D Halperin}, url = {https://www.cs.rutgers.edu/~kb572/pubs/prob_completeness_rrt.pdf}, year = {2019}, date = {2019-01-01}, journal = {IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE ICRA 2019)}, abstract = {The Rapidly-exploring Random Tree (RRT) algorithm has been one of the most prevalent and popular motion-planning techniques for two decades now. Surprisingly, in spite of its centrality, there has been an active debate under which conditions RRT is probabilistically complete. We provide two new proofs of probabilistic completeness (PC) of RRT with a reduced set of assumptions. The first one for the purely geometric setting, where we only require that the solution path has a certain clearance from the obstacles. For the kinodynamic case with forward propagation of random controls and duration, we only consider in addition mild Lipschitz-continuity conditions. These proofs fill a gap in the study of RRT itself. They also lay sound foundations for a variety of more recent and alternative sampling-based methods, whose PC property relies on that of RRT.}, keywords = {}, pubstate = {published}, tppubtype = {article} } The Rapidly-exploring Random Tree (RRT) algorithm has been one of the most prevalent and popular motion-planning techniques for two decades now. Surprisingly, in spite of its centrality, there has been an active debate under which conditions RRT is probabilistically complete. We provide two new proofs of probabilistic completeness (PC) of RRT with a reduced set of assumptions. The first one for the purely geometric setting, where we only require that the solution path has a certain clearance from the obstacles. For the kinodynamic case with forward propagation of random controls and duration, we only consider in addition mild Lipschitz-continuity conditions. These proofs fill a gap in the study of RRT itself. They also lay sound foundations for a variety of more recent and alternative sampling-based methods, whose PC property relies on that of RRT. |
Surovik, D; Wang, K; Vespignani, M; Bruce, J; Bekris, K E Adaptive Tensegrity Locomotion: Controlling a Compliant Icosahedron with Symmetry-Reduced Reinforcement Learning Journal Article International Journal of Robotics Research (IJRR), 2019. Abstract | Links | BibTeX | Tags: @article{210, title = {Adaptive Tensegrity Locomotion: Controlling a Compliant Icosahedron with Symmetry-Reduced Reinforcement Learning}, author = {D Surovik and K Wang and M Vespignani and J Bruce and K E Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/reinf_learning_tensegrity_locomotion.pdf}, year = {2019}, date = {2019-00-01}, journal = {International Journal of Robotics Research (IJRR)}, abstract = {Tensegrity robots, which are prototypical examples of hybrid soft-rigid robots, exhibit dynamical properties that provide ruggedness and adaptability. They also bring about, however, major challenges for locomotion control. Due to high dimensionality and the complex evolution of contact states, data-driven approaches are appropriate for producing viable feedback policies for tensegrities. Guided Policy Search (GPS), a sample-efficient hybrid framework for optimization and reinforcement learning, has previously been applied to generate periodic, axis-constrained locomotion by an icosahedral tensegrity on flat ground. Varying environments and tasks, however, create a need for more adaptive and general locomotion control that actively utilizes an expanded space of robot states. This implies significantly higher needs in terms of sample data and setup effort. This work mitigates such requirements by proposing a new GPS- based reinforcement learning pipeline, which exploits the vehicletextquoterights high degree of symmetry and appropriately learns contextual behaviors that are sustainable without periodicity. Newly achieved capabilities include axially-unconstrained rolling, rough terrain traversal, and rough incline ascent. These tasks are evaluated for a small variety of key model parameters in simulation and tested on the NASA hardware prototype, SUPERball. Results confirm the utility of symmetry exploitation and the adaptability of the vehicle. They also shed light on numerous strengths and limitations of the GPS framework for policy design and transfer to real hybrid soft-rigid robots.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Tensegrity robots, which are prototypical examples of hybrid soft-rigid robots, exhibit dynamical properties that provide ruggedness and adaptability. They also bring about, however, major challenges for locomotion control. Due to high dimensionality and the complex evolution of contact states, data-driven approaches are appropriate for producing viable feedback policies for tensegrities. Guided Policy Search (GPS), a sample-efficient hybrid framework for optimization and reinforcement learning, has previously been applied to generate periodic, axis-constrained locomotion by an icosahedral tensegrity on flat ground. Varying environments and tasks, however, create a need for more adaptive and general locomotion control that actively utilizes an expanded space of robot states. This implies significantly higher needs in terms of sample data and setup effort. This work mitigates such requirements by proposing a new GPS- based reinforcement learning pipeline, which exploits the vehicletextquoterights high degree of symmetry and appropriately learns contextual behaviors that are sustainable without periodicity. Newly achieved capabilities include axially-unconstrained rolling, rough terrain traversal, and rough incline ascent. These tasks are evaluated for a small variety of key model parameters in simulation and tested on the NASA hardware prototype, SUPERball. Results confirm the utility of symmetry exploitation and the adaptability of the vehicle. They also shed light on numerous strengths and limitations of the GPS framework for policy design and transfer to real hybrid soft-rigid robots. |
Shome, R; Solovey, K; Dobson, A; Halperin, D; Bekris, K E dRRT*: Scalable and Informed Asymptotically-Optimal Multi-Robot Motion Planning Journal Article Autonomous Robots, 2019. Abstract | Links | BibTeX | Tags: @article{204, title = {dRRT*: Scalable and Informed Asymptotically-Optimal Multi-Robot Motion Planning}, author = {R Shome and K Solovey and A Dobson and D Halperin and K E Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/drrt_star_auro.pdf}, year = {2019}, date = {2019-00-01}, journal = {Autonomous Robots}, abstract = {Many exciting robotic applications require multiple robots with many degrees of freedom, such as manipulators, to coordinate their motion in a shared workspace. Discovering high-quality paths in such scenarios can be achieved, in principle, by exploring the composite space of all robots. Sampling-based planners do so by building a roadmap or a tree data structure in the corresponding configuration space and can achieve asymptotic optimality. The hardness of motion planning, however, renders the explicit construction of such structures in the composite space of multiple robots impractical. This work proposes a scalable solution for such coupled multi-robot problems, which provides desirable path-quality guarantees and is also computationally efficient. In particular, the proposed dRRT* is an informed, asymptotically-optimal extension of a prior sampling-based multi-robot motion planner, dRRT. The prior approach introduced the idea of building roadmaps for each robot and implicitly searching the tensor product of these structures in the composite space. This work identifies the conditions for convergence to optimal paths in multi-robot problems, which the prior method was not achieving.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Many exciting robotic applications require multiple robots with many degrees of freedom, such as manipulators, to coordinate their motion in a shared workspace. Discovering high-quality paths in such scenarios can be achieved, in principle, by exploring the composite space of all robots. Sampling-based planners do so by building a roadmap or a tree data structure in the corresponding configuration space and can achieve asymptotic optimality. The hardness of motion planning, however, renders the explicit construction of such structures in the composite space of multiple robots impractical. This work proposes a scalable solution for such coupled multi-robot problems, which provides desirable path-quality guarantees and is also computationally efficient. In particular, the proposed dRRT* is an informed, asymptotically-optimal extension of a prior sampling-based multi-robot motion planner, dRRT. The prior approach introduced the idea of building roadmaps for each robot and implicitly searching the tensor product of these structures in the composite space. This work identifies the conditions for convergence to optimal paths in multi-robot problems, which the prior method was not achieving. |
Littlefield, Z; Surovik, D; Vespignani, M; Bruce, J; Wang, W; Bekris, K E Kinodynamic Planning for Spherical Tensegrity Locomotion with Effective Gait Primitives Journal Article International Journal of Robotics Research (IJRR), 2019. Abstract | Links | BibTeX | Tags: @article{208, title = {Kinodynamic Planning for Spherical Tensegrity Locomotion with Effective Gait Primitives}, author = {Z Littlefield and D Surovik and M Vespignani and J Bruce and W Wang and K E Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/kinodynamic_tensegrity.pdf}, year = {2019}, date = {2019-00-01}, journal = {International Journal of Robotics Research (IJRR)}, abstract = {Tensegrity-based robots can achieve locomotion through shape deformation and compliance. They are highly adaptable to their surroundings, have light weight, low cost and are physically robust. Their high dimensionality and strongly dynamic nature, however, complicate motion planning. Efforts to-date have primarily considered quasi-static reconfiguration and short-term dynamic motion of tensegrity robots, which do not fully exploit the underlying system dynamics in the long term. Longer-horizon planning has previously required costly search over the full space of valid control inputs. This work synthesizes new and existing approaches to produce dynamic long-term motion while balancing the computational demand. A numerical process based upon quasi-static assumptions is first applied to deform the system into an unstable configuration, causing forward motion. The dynamical characteristics of the result are then altered via a few simple parameters to produce a small but diverse set of useful behaviors. The proposed approach takes advantage of identified symmetries on the prototypical spherical tensegrity robot, which reduce the number of needed gaits but allow motion along different directions. These gaits are first combined with a standard search method to achieve long term planning in environments where the developed gaits are effective. For more complex environments, the various motion primitives are paired with the fall-back option of random valid actions and are used by an informed sampling-based kinodynamic motion planner with anytime properties. Evaluations using a physics-based model for the prototypical robot demonstrate that modest but efficiently-applied search effort can unlock the utility of dynamic tensegrity motion to produce high-quality solutions.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Tensegrity-based robots can achieve locomotion through shape deformation and compliance. They are highly adaptable to their surroundings, have light weight, low cost and are physically robust. Their high dimensionality and strongly dynamic nature, however, complicate motion planning. Efforts to-date have primarily considered quasi-static reconfiguration and short-term dynamic motion of tensegrity robots, which do not fully exploit the underlying system dynamics in the long term. Longer-horizon planning has previously required costly search over the full space of valid control inputs. This work synthesizes new and existing approaches to produce dynamic long-term motion while balancing the computational demand. A numerical process based upon quasi-static assumptions is first applied to deform the system into an unstable configuration, causing forward motion. The dynamical characteristics of the result are then altered via a few simple parameters to produce a small but diverse set of useful behaviors. The proposed approach takes advantage of identified symmetries on the prototypical spherical tensegrity robot, which reduce the number of needed gaits but allow motion along different directions. These gaits are first combined with a standard search method to achieve long term planning in environments where the developed gaits are effective. For more complex environments, the various motion primitives are paired with the fall-back option of random valid actions and are used by an informed sampling-based kinodynamic motion planner with anytime properties. Evaluations using a physics-based model for the prototypical robot demonstrate that modest but efficiently-applied search effort can unlock the utility of dynamic tensegrity motion to produce high-quality solutions. |
Mitash, C; Boularias, A; Bekris, K E Physics-based Scene-level Reasoning for Object Pose Estimation in Clutter Journal Article International Journal of Robotics Research (IJRR), 2019. Abstract | Links | BibTeX | Tags: @article{209, title = {Physics-based Scene-level Reasoning for Object Pose Estimation in Clutter}, author = {C Mitash and A Boularias and K E Bekris}, url = {https://arxiv.org/pdf/1806.10457.pdf}, year = {2019}, date = {2019-00-01}, journal = {International Journal of Robotics Research (IJRR)}, abstract = {This paper focuses on vision-based pose estimation for multiple rigid objects placed in clutter, especially in cases involving occlusions and objects resting on each other. Progress has been achieved recently in object recognition given advancements in deep learning. Nevertheless, such tools typically require a large amount of training data and significant manual effort to label objects. This limits their applicability in robotics, where solutions must scale to a large number of objects and variety of conditions. Moreover, the combinatorial nature of the scenes that could arise from the placement of multiple objects is hard to capture in the training dataset. Thus, the learned models might not produce the desired level of precision required for tasks, such as robotic manipulation. This work proposes an autonomous process for pose estimation that spans from data generation, to scene-level reasoning and self-learning. In particular, the proposed framework first generates a labeled dataset for training a Convolutional Neural Network (CNN) for object detection in clutter. These detections are used to guide a scene-level optimization process, which considers the interactions between the different objects present in the clutter to output pose estimates of high precision. Furthermore, confident estimates are used to label online real images from multiple views and re-train the process in a self-learning pipeline. Experimental results indicate that this process is quickly able to identify in cluttered scenes physically-consistent object poses that are more precise than the ones found by reasoning over individual instances of objects. Furthermore, the quality of pose estimates increases over time given the self-learning process.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This paper focuses on vision-based pose estimation for multiple rigid objects placed in clutter, especially in cases involving occlusions and objects resting on each other. Progress has been achieved recently in object recognition given advancements in deep learning. Nevertheless, such tools typically require a large amount of training data and significant manual effort to label objects. This limits their applicability in robotics, where solutions must scale to a large number of objects and variety of conditions. Moreover, the combinatorial nature of the scenes that could arise from the placement of multiple objects is hard to capture in the training dataset. Thus, the learned models might not produce the desired level of precision required for tasks, such as robotic manipulation. This work proposes an autonomous process for pose estimation that spans from data generation, to scene-level reasoning and self-learning. In particular, the proposed framework first generates a labeled dataset for training a Convolutional Neural Network (CNN) for object detection in clutter. These detections are used to guide a scene-level optimization process, which considers the interactions between the different objects present in the clutter to output pose estimates of high precision. Furthermore, confident estimates are used to label online real images from multiple views and re-train the process in a self-learning pipeline. Experimental results indicate that this process is quickly able to identify in cluttered scenes physically-consistent object poses that are more precise than the ones found by reasoning over individual instances of objects. Furthermore, the quality of pose estimates increases over time given the self-learning process. |
2018 |
Shome, R; Solovey, K; Yu, J; Bekris, K E; Halperin, D Fast and High-Quality Dual-Arm Rearrangement in Synchronous, Monotone Tabletop Setups Conference Workshop on the Algorithmic Foundations of Robotics (WAFR), Mérida, México, 2018. Abstract | Links | BibTeX | Tags: @conference{199, title = {Fast and High-Quality Dual-Arm Rearrangement in Synchronous, Monotone Tabletop Setups}, author = {R Shome and K Solovey and J Yu and K E Bekris and D Halperin}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Fast_High_Quality_Dual_Arm_Rearrangement.pdf}, year = {2018}, date = {2018-12-01}, booktitle = {Workshop on the Algorithmic Foundations of Robotics (WAFR)}, address = {Mérida, México}, abstract = {Rearranging objects on a planar surface arises in a variety of applications, such as packaging. Using two arms can improve efficiency but introduces new combinatorial challenges. This paper studies the structure of dual-arm rearrangement for synchronous, monotone tabletop setups and develops an optimal MILP model. It then describes an efficient and scalable algorithm, which first minimizes the cost of object transfers and then of transitions between objects. This is motivated by the fact that asymptotically object transfers dominate the cost of solutions. Moreover, a lazy strategy minimizes the number of motion planning calls and results in significant speedups. Theoretical arguments support the benefits of using two arms and indicate that synchronous operation introduces only a small cost increase. Experiments support these points and show that the scalable method can quickly compute solutions close to optimal for the considered setup.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Rearranging objects on a planar surface arises in a variety of applications, such as packaging. Using two arms can improve efficiency but introduces new combinatorial challenges. This paper studies the structure of dual-arm rearrangement for synchronous, monotone tabletop setups and develops an optimal MILP model. It then describes an efficient and scalable algorithm, which first minimizes the cost of object transfers and then of transitions between objects. This is motivated by the fact that asymptotically object transfers dominate the cost of solutions. Moreover, a lazy strategy minimizes the number of motion planning calls and results in significant speedups. Theoretical arguments support the benefits of using two arms and indicate that synchronous operation introduces only a small cost increase. Experiments support these points and show that the scalable method can quickly compute solutions close to optimal for the considered setup. |
Surovik, D; Bruce, J; Wang, K; Vespignani, M; Bekris, K E Any-axis Tensegrity Rolling via Bootstrapped Learning and Symmetry Reduction Conference International Symposium on Experimental Robotics (ISER), Buenos Aires, Argentina, 2018. @conference{194, title = {Any-axis Tensegrity Rolling via Bootstrapped Learning and Symmetry Reduction}, author = {D Surovik and J Bruce and K Wang and M Vespignani and K E Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/any_axis_tensegrity_rolling.pdf}, year = {2018}, date = {2018-11-01}, booktitle = {International Symposium on Experimental Robotics (ISER)}, address = {Buenos Aires, Argentina}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Kimmel, A; Shome, R; Littlefield, Z; Bekris, K E Fast, Anytime Motion Planning for Prehensile Manipulation in Clutter Conference 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids 2018), Beijing, China, 2018. Abstract | Links | BibTeX | Tags: @conference{200, title = {Fast, Anytime Motion Planning for Prehensile Manipulation in Clutter}, author = {A Kimmel and R Shome and Z Littlefield and K E Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/gmp.pdf}, year = {2018}, date = {2018-11-01}, booktitle = {2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids 2018)}, address = {Beijing, China}, abstract = {Many methods have been developed for planning the motion of robotic arms for picking and placing, ranging from local optimization to global search techniques, which are effective for sparsely placed objects. Dense clutter, however, still adversely affects the success rate, computation times, and quality of solutions in many real-world setups. The current work integrates tools from existing methodologies and proposes a framework that achieves high success ratio in clutter with anytime performance by returning solutions quickly and improving their quality over time, measured in terms of end effectortextquoterights displacement. The idea is to first explore the lower dimensional end effectortextquoterights task space efficiently by ignoring the arm, and build a discrete approximation of a navigation function, which guides the end effector towards the set of available grasps or object placements. This is performed online, without prior knowledge of the scene. Then, an informed sampling-based planner for the entire arm uses Jacobian-based steering to reach promising end effector poses given the task space guidance. While informed, the method is also comprehensive and allows the exploration of alternative paths over time if the task space guidance does not lead to a solution. This paper evaluates the proposed method against alternatives in picking or placing tasks among varying amounts of clutter for a variety of robotic manipulators with different end-effectors. The results suggest that the method reliably provides higher quality solution paths quicker, with a higher success rate relative to alternatives.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Many methods have been developed for planning the motion of robotic arms for picking and placing, ranging from local optimization to global search techniques, which are effective for sparsely placed objects. Dense clutter, however, still adversely affects the success rate, computation times, and quality of solutions in many real-world setups. The current work integrates tools from existing methodologies and proposes a framework that achieves high success ratio in clutter with anytime performance by returning solutions quickly and improving their quality over time, measured in terms of end effectortextquoterights displacement. The idea is to first explore the lower dimensional end effectortextquoterights task space efficiently by ignoring the arm, and build a discrete approximation of a navigation function, which guides the end effector towards the set of available grasps or object placements. This is performed online, without prior knowledge of the scene. Then, an informed sampling-based planner for the entire arm uses Jacobian-based steering to reach promising end effector poses given the task space guidance. While informed, the method is also comprehensive and allows the exploration of alternative paths over time if the task space guidance does not lead to a solution. This paper evaluates the proposed method against alternatives in picking or placing tasks among varying amounts of clutter for a variety of robotic manipulators with different end-effectors. The results suggest that the method reliably provides higher quality solution paths quicker, with a higher success rate relative to alternatives. |
Calli, B; Kimmel, A; Hang, K; Bekris, K E; Dollar, A Path Planning for Within-Hand Manipulation Over Learned Representations of Safe States Conference International Symposium on Experimental Robotics (ISER), Buenos Aires, Argentina, 2018. Abstract | Links | BibTeX | Tags: @conference{195, title = {Path Planning for Within-Hand Manipulation Over Learned Representations of Safe States}, author = {B Calli and A Kimmel and K Hang and K E Bekris and A Dollar}, url = {http://www.cs.rutgers.edu/~kb572/pubs/within_hand_planning_over_learning.pdf}, year = {2018}, date = {2018-11-01}, booktitle = {International Symposium on Experimental Robotics (ISER)}, address = {Buenos Aires, Argentina}, abstract = {This work proposes a framework for tracking a desired path of an object held by an adaptive hand via within-hand manipulation. Such underactuated hands are able to passively achieve stable contacts with objects. Combined with vision-based control and data-driven state estimation process, they can solve tasks without accurate hand-object models or multi-modal sensory feedback. In particular, a data-driven regression process is used here to estimate the probability of dropping the object for given manipulation states. Then, an optimization-based planner aims to track the desired path while avoiding states that are above a threshold probability of dropping the object. The optimized cost function, based on the principle of Dynamic-Time Warping (DTW), seeks to minimize the area between the desired and the followed path. By adapting the threshold for the probability of dropping the object, the framework can handle objects of different weights without retraining. Experiments involving writing letters with a marker, as well as tracing randomized paths, were conducted on the Yale Model T-42 hand. Results indicate that the framework successfully avoids undesirable states, while minimizing the proposed cost function, thereby producing object paths for within-hand manipulation that closely match the target ones.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } This work proposes a framework for tracking a desired path of an object held by an adaptive hand via within-hand manipulation. Such underactuated hands are able to passively achieve stable contacts with objects. Combined with vision-based control and data-driven state estimation process, they can solve tasks without accurate hand-object models or multi-modal sensory feedback. In particular, a data-driven regression process is used here to estimate the probability of dropping the object for given manipulation states. Then, an optimization-based planner aims to track the desired path while avoiding states that are above a threshold probability of dropping the object. The optimized cost function, based on the principle of Dynamic-Time Warping (DTW), seeks to minimize the area between the desired and the followed path. By adapting the threshold for the probability of dropping the object, the framework can handle objects of different weights without retraining. Experiments involving writing letters with a marker, as well as tracing randomized paths, were conducted on the Yale Model T-42 hand. Results indicate that the framework successfully avoids undesirable states, while minimizing the proposed cost function, thereby producing object paths for within-hand manipulation that closely match the target ones. |
Littlefield, Z; Bekris, K E Efficient and Asymptotically Optimal Kinodynamic Motion Planning via Dominance-Informed Regions Conference IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 2018. @conference{197, title = {Efficient and Asymptotically Optimal Kinodynamic Motion Planning via Dominance-Informed Regions}, author = {Z Littlefield and K E Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/iros_dirt.pdf}, year = {2018}, date = {2018-10-01}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, address = {Madrid, Spain}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Zhu, S; Surovik, D; Bekris, K E; Boularias, A Efficient Model Identification for Tensegrity Locomotion Conference IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 2018. Abstract | Links | BibTeX | Tags: @conference{196, title = {Efficient Model Identification for Tensegrity Locomotion}, author = {S Zhu and D Surovik and K E Bekris and A Boularias}, url = {https://www.cs.rutgers.edu/~kb572/pubs/model_identification_tensegrity.pdf}, year = {2018}, date = {2018-10-01}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, address = {Madrid, Spain}, abstract = {This paper aims to identify in a practical manner unknown physical parameters, such as mechanical models of actuated robot links, which are critical in dynamical robotic tasks. Key features include the use of an off-the-shelf physics engine and the Bayesian optimization framework. The task being considered is locomotion with a high-dimensional, compliant Tensegrity robot. A key insight, in this case, is the need to project the space of models into an appropriate lower dimensional space for time efficiency. Comparisons with alternatives indicate that the proposed method can identify the parameters more accurately within the given time budget, which also results in more precise locomotion control.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } This paper aims to identify in a practical manner unknown physical parameters, such as mechanical models of actuated robot links, which are critical in dynamical robotic tasks. Key features include the use of an off-the-shelf physics engine and the Bayesian optimization framework. The task being considered is locomotion with a high-dimensional, compliant Tensegrity robot. A key insight, in this case, is the need to project the space of models into an appropriate lower dimensional space for time efficiency. Comparisons with alternatives indicate that the proposed method can identify the parameters more accurately within the given time budget, which also results in more precise locomotion control. |
Mitash, C; Boularias, A; Bekris, K E Robust 6D Pose Estimation with Stochastic Congruent Sets Conference British Machine Vision (BMVC) conference, Newcastle, UK, 2018. @conference{198, title = {Robust 6D Pose Estimation with Stochastic Congruent Sets}, author = {C Mitash and A Boularias and K E Bekris}, url = {https://arxiv.org/abs/1805.06324}, year = {2018}, date = {2018-09-01}, booktitle = {British Machine Vision (BMVC) conference}, address = {Newcastle, UK}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Shuai, H; Stiffler, N; Krontiris, A; Bekris, K E; Yu, J Complexity Results and Fast Methods for Optimal Tabletop Rearrangement with Overhand Grasps Journal Article International Journal of Robotics Research (IJRR), 2018. Abstract | Links | BibTeX | Tags: @article{193, title = {Complexity Results and Fast Methods for Optimal Tabletop Rearrangement with Overhand Grasps}, author = {H Shuai and N Stiffler and A Krontiris and K E Bekris and J Yu}, url = {https://www.cs.rutgers.edu/~kb572/pubs/optimal_tabletop_rearrangement.pdf}, year = {2018}, date = {2018-07-01}, journal = {International Journal of Robotics Research (IJRR)}, abstract = {This paper studies the underlying combinatorial structure of a class of object rearrangement problems, which appear frequently in applications. The problems involve multiple, similar-geometry objects placed on a flat, horizontal surface, where a robot can approach them from above and perform pick-and-place operations to rearrange them. The paper considers both the case where the start and goal object poses overlap, and where they do not. For overlapping poses, the primary objective is to minimize the number of pick-and-place actions and then to minimize the distance traveled by the end-effector. For the non-overlapping case, the objective is solely to minimize the travel distance of the end-effector. While such problems do not involve all the complexities of general rearrangement, they remain computationally hard in both cases. This is shown through reductions from well-understood, hard combinatorial challenges to these rearrangement problems. The reductions are also shown to hold in the reverse direction, which enables the convenient application on rearrangement of well studied algorithms. These algorithms can be very efficient in practice despite the hardness results. The paper builds on these reduction results to propose an algorithmic pipeline for dealing with the rearrangement problems. Experimental evaluation, including hardware-based trials, shows that the proposed pipeline computes high-quality paths with regards to the optimization objectives. Furthermore, it exhibits highly desirable scalability as the number of objects increases in both the overlapping and non-overlapping setup.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This paper studies the underlying combinatorial structure of a class of object rearrangement problems, which appear frequently in applications. The problems involve multiple, similar-geometry objects placed on a flat, horizontal surface, where a robot can approach them from above and perform pick-and-place operations to rearrange them. The paper considers both the case where the start and goal object poses overlap, and where they do not. For overlapping poses, the primary objective is to minimize the number of pick-and-place actions and then to minimize the distance traveled by the end-effector. For the non-overlapping case, the objective is solely to minimize the travel distance of the end-effector. While such problems do not involve all the complexities of general rearrangement, they remain computationally hard in both cases. This is shown through reductions from well-understood, hard combinatorial challenges to these rearrangement problems. The reductions are also shown to hold in the reverse direction, which enables the convenient application on rearrangement of well studied algorithms. These algorithms can be very efficient in practice despite the hardness results. The paper builds on these reduction results to propose an algorithmic pipeline for dealing with the rearrangement problems. Experimental evaluation, including hardware-based trials, shows that the proposed pipeline computes high-quality paths with regards to the optimization objectives. Furthermore, it exhibits highly desirable scalability as the number of objects increases in both the overlapping and non-overlapping setup. |