Kostas Bekris
Office Hours: Monday 11:30am – 12:30pm @ Hill 277
Email: kostas.bekris/AT/

Monday, 1:40 PM – 4:40 PM @ ARC-206 (Allison Road Classroom), Busch Campus

Focus on Robot Manipulation

In artificial intelligence, an intelligent agent is an autonomous entity, which observes its environment through sensors, acts upon the environment using actuators and directs its activity towards achieving goals, i.e. it is “rational”. A robot manipulator is the prototypical embodiment of an intelligent agent. Solving robot manipulation problems requires reasoning about the physical world and building upon foundational mathematical principles. At the same time, one can use as inspiration the impressive manipulation capabilities of biological organisms.

Description of manipulation from the book “Mechanics of Robotic Manipulation” by Matt Mason: Manipulation is the process of using one’s hands to rearrange one’s environment. There are many facets to manipulation. Manipulation is an art, since it is practiced by all of us, without any systematic or fundamental understanding of the process. Manipulation is also an engineering discipline, for there are some systematic tools for applying robotic manipulation to various problems. Finally, manipulation is also a science, since it is a process that engages our curiosity, which we can explore using scientific methods.

Motivation for Robot Manipulation

Commercially available robotic toys and vacuum cleaners inhabit our living spaces, and robotic vehicles have raced across the desert. These successes appear to foreshadow an explosion of robotic applications in our daily lives, but without advances in robot manipulation, many promising robotic applications will not be possible. Whether in a domestic setting or the workplace, we would like robots to physically alter the world through contact. Robots have long been imagined as mechanical workers, helping us in our daily life. Research on manipulation in human environments may someday lead to robots that work alongside us, extending the time an elderly person can live at home, providing physical assistance to a worker on an assembly line, or helping with household chores.

Within factories around the world, robots perform heroic feats of manipulation on a daily basis. They lift massive objects, move with blurring speed, and repeat complex performances with unerring precision. Yet outside of carefully controlled settings, even for the most sophisticated robot it would be difficult to get a glass of water. The everyday manipulation tasks we take for granted would stump the greatest robot bodies and brains in existence today. Why are robots so incompetent in unstructured, human environments?

The above motivating material for robot manipulation comes from the paper “Challenges for Robot Manipulation in Human Environments” by Kemp, Edsinger, Torres-Jara, IEEE Robotics and Automation Magazine, March 2007.

Course Description

This Computer Science seminar will cover algorithmic techniques on robot planning, perception and learning that are important for achieving autonomous manipulation with modern robots. The course will be split into the following components:

  • Mathematical foundations of robot manipulation: i.e., mathematics of rigid motion, forward and inverse kinematics solutions, the notions of a manipulator Jacobian, velocity kinematics and manipulability will also be discussed.
  • Motion and task planning for manipulation: i.e., computing paths for robotic arms, trajectory generation, pick-and-place manipulation, rearrangment and multi-arm manipulation, quasi-static and dynamic manipulation, considering more complex hands, etc.
  • Perception for Manipulation: i.e., object detection and pose estimation, manipulation under uncertainty, visual servoing foundations, etc.
  • Learning for Manipulation: i.e., modeling complex interactions, non-prehensile manipulation, end-to-end approaches, etc.

Beyond lectures, the course will utilize a semester long project and presentations of related papers from the literature.

NIST ARIAC Challenge

Monitor the following website for details:

Proposed Papers for Presentations

Check Rutgers’ Key Dates.

  • Motion Planning for Robotic Arms
    1. “CHOMP: Covariant Hamiltonian Optimization for Motion Planning” by Matthew Zucker, Nathan Ratliff, Anca Dragan, Mihail Pivtoraiko, Matthew Klingensmith, Christopher Dellin, J. Andrew (Drew) Bagnell and Siddhartha Srinivasa, International Journal of Robotics Research, May, 2013
    2. “Single- and dual-arm motion planning with heuristic search” by Benjamin Cohen, Sachin Chitta and Maxim Likhachev, International Journal of Robotics Research (IJRR), 2013. [Seth Karten – April 29]
    3. “Motion Planning with Sequential Convex Optimization and Convex Collision Checking”, by John Schulman, Yan Duan, Jonathan Ho, Alex Lee, Ibrahim Awwal, Henry Bradlow, Jia Pan, Sachin Patil, Ken Goldberg, Pieter Abbeel. In the International Journal of Robotics Research (IJRR), Vol. 33, No. 9, pp. 1251-1270, Aug. 2014.
  • Manipulation Planning
    1. “Manipulation planning with probabilistic roadmaps” by T Siméon, J Cortés, JP. Laumond, A Sahban,. The International Journal of Robotics Research, Vol.23, N°7-8 (2004). [Bhavya Tiwari – May 6]
    2. “Task Space Regions: A Framework for Pose-Constrained Manipulation Planning” by Dmitry Berenson, Siddhartha Srinivasa, and James Kuffner, International Journal of Robotics Research (IJRR), Volume 30, Number 12, pp. 1435 – 1460, October, 2011.
    3. “Global manipulation planning in robot joint space with task constraints” by M Stilman, IEEE Transactions on Robotics 26 (3), 576-584.
    4. “Elastic Roadmaps – Motion Generation for Autonomous Mobile Manipulation” by Yuandong Yang and Oliver Brock, Autonomous Robots 28(1):113-130, 2010.
    5. “A planning framework for non-prehensile manipulation under clutter and uncertainty”, by MR Dogar, SS Srinivasa, Autonomous Robots 33 (3), 217-236, 2012
    6. “Efficiently Solving General Rearrangement Tasks: A Fast Extension Primitive For An Incremental Sampling-Based Planner”, Krontiris, A, and KE Bekris, In International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 2016. [Siwei Feng – April 29]
    7. “A Framework for Optimal Grasp Contact Planning” by K Hang, JA Stork, NS Pollard, D Kragic, IEEE Robotics and Automation Letters 2 (2), 704-711, 2017.
  • Learning for Manipulation
    1. “A Data-Efficient Approach to Precise and Controlled Pushing”, M. Bauza*, F. Hogan*, and A. Rodriguez, CoRL, 2018 [Manvi Agarwal – May 6]
    2. “Differentiable Physics and Stable Modes for Tool-Use and Manipulation Planning”, Marc Toussaint, Kelsey Allen, Kevin Smith, Joshua Tenenbaum, RSS 2018. [Matthew Purri – April 29]
    3. “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations”,
      Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, Sergey Levine, RSS 2018. [Kunal Prakash Mokashi – May 6]
  • Perception
    1. “Super4PCS: Fast Global Pointcloud Registration via Smart Indexing”, by Nicolas Mellado,
      Dror Aiger, Niloy J. Mitra, Computer Graphics Forum, 2014 [Himanshu Jain – April 29]
    2. “PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
      Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, Dieter Fox, Robotics: Science and Systems (RSS), 2018 [Abhishek Bhrushundi – April 29]
    3. “Improving 6D Pose Estimation Of Objects In Clutter Via Physics-Aware Monte Carlo Tree Search”, Mitash, C., A. Boularias, and KE Bekris, In IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 2018. [Bowen Wen – April 29]
    4. “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation” by Julian Ibarz et al., CoRL, 2018. [Ridhima Sakhuja – May 6]
    5. “Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation” by Lucas Manuelli (Massachusetts Institute of Technology); Peter R Florence (MIT); Russ Tedrake (MIT), CoRL, 2018. [Chen Jia – May 6]

Text and Reading Material

No specific textbook is required. Students are expected to study the papers that are recommended for each lecture. Regular attendance and active participation during the meetings is expected. Specific material related to the corresponding topics will become available via Sakai.

Deliverable & Grades

The first half of the course will focus on book-based material. The second half of the course will focus on research papers and presentations. There is also a semester long project. You will be evaluated based on: a) in-class participation (20% of the course grade) b), the project’s quality, presentation and report (50% of the course grade) and c) your in-class presentation of a research paper (30% of the course grade). Extra credit can be awarded for impressive effort and performance in the project.

The students should submit typesetted reports using LaTeX in relation to their project. Resources on using LaTeX are available below. You are expected to use IEEE RAS’ double column format for conferences.

Submissions will be processed through Sakai. Late project reports will not be accepted. Final project presentations cannot be easily rescheduled. Unless there are unprecedented circumstances, no incomplete grade will be awarded and students will be graded based on the submitted reports and presentations up to the exam week.

As a rough guide, the following rule may be used for the final grade (this is tentative):

  • A: >= 90
  • B+: 80-89
  • B: 70-79
  • C+: 60-69
  • C: 50-59
  • D: 40-49
  • F: less than 40

Academic Standards

Discussions on the class material, on project challenges and programming practices are highly encouraged. Furthermore, students should try to exchange feedback on drafts of their project reports. Nevertheless, each student needs to eventually independently develop one’s proposed methodology, code the corresponding solution and execute the corresponding experiments.

You should carefully study the website of Rutgers University on Academic Integrity and the corresponding policy, as well as the corresponding policy from the department of Computer Science. Your continued enrollment in this course implies that you have read these policies, and that you subscribe to the principles stated therein.

Latex Resources

General info on what you can do with LaTeX:
Getting Started with Latex
The Not So Short Introduction to Latex
Comprehensive List of Latex Symbols
Latex for Logicians

Tex on Mac OS X

The first link describes many alternatives that are available for installing Tex on a Mac. The second link forwards to the MacTex package, one of the alternatives mentioned in the first website. MaxTex provides everythink that you need to use Latex on Mac except from a text editor. It is, however, compatible with a wide variety of popular editors (e.g., Alpha, BBEdit, Emacs, VIM, iTeXMac, TeXShop). Note that MaxTex is a large package.

Carbon Emacs has been succesfully tested with MacTex. After installing MacTex, it is possible to directly compile and view *.tex files from Carbon Emacs’s UI.

Note for Mac users: You will probably have problems previewing your PDF output when using the postscript images provided by the instructor for developing the notes. Nevertheless, the PDF file can be printed properly. Prepare your document without the images and then add them. You will probably still be able to preview the intermediate .dvi output file with the “xdvi” program.

Linux (Ubuntu)
Latex on Ubuntu
Tex Live

You just have to download and install the proper packages described above (e.g., through apt-get), use your favorite editor (e.g., emacs) to prepare a *.tex file and then you compile (run at least two times: “latex filename.tex”) to get the *.dvi output. You can go from dvi to postscript with the command “dvips” and you can convert postscript to pdf with the command “ps2pdf”.

Latex for Windows help
MikTex (Latex for Windows)

If you follow the instructions on the first link you should be able to get it working on a Windows system.

Below you can find Windows executables (32 bit) for the following programs (follow the order when installing):