Humanoid Robotics
Overview
For my robotics class, our team was tasked with using dual UR5 arms to perform a series of humanoid tasks, including manipulating objects such as plates, cups, and a microwave handle and performing pick-and-place operations with these items. As lead of the software team, I was responsible for the full software implementation — including the initial strategy, computer vision, pick-and-place execution, and coordination of all tasks across both arms. I received the Most Valuable Engineer award for my contributions to our team's success.
Strategy
I brainstormed a range of competition strategies, evaluating each on feasibility within the time limit, risk, and achievable points. We settled on picking and placing a plate and bowl into a microwave and onto a tray, and pouring a liquid into a cup and stirring with a stirrer. We also decided to perform all tasks purely using computer vision to maximize our point total.
Software Implementation
Computer Vision
Objects were isolated using HSV filtering, edge detection, and Hough transforms to detect colored shapes in the camera frame. I calibrated a transformation between the camera and robot frames by placing and detecting an AprilTag at multiple end effector locations. The detected object positions were then used to move the end effector to the exact object location, correcting for any object shifts before competition.
Manipulation
I used waypoint-based position control to move the arms into safe configurations before using computer vision to precisely locate and grip objects. This required extensive time spent exploring and recording robot arm configurations. Force control was also used to exert a desired contact force on the microwave handle and button. All programming was written in Python.
Orchestration
I built a web interface with a state machine to run tasks simultaneously across both arms and enable restart from any task in the event of an incomplete execution. This enabled clean coordination between the arms to minimize time lost between tasks and provided a level of safety in case of failure during competition.
