Last year we introduced the MenoRobot (Menorah Robot), which utilized an arm mechanism to light a Chanukiah* in celebration of Hanukkah. While it was quite impressive, it had its constraints, particularly because the arm's movements to access each of the eight principal candles and the shamash were manually pre-programmed, lacking any automated adaptability.
This year, however, my aim was to elevate MenoRobot to version 2.0, really pushing the envelope by incorporating a robust array of Viam's latest advancements. I was determined to demonstrate the potential of our cutting-edge computer vision and motion capabilities, alongside our data and machine learning (ML) services – not to mention the convenience of our mobile application.
With the holiday approaching fast, I was working against both ambition and a pressing deadline. Time to get building!
Putting together the hardware
I started by collaborating with a coworker to design a brand new 3D printed Viam Chanukiah.
This included simplifying some conditions, like purchasing oil filled glass candles so they wouldn’t melt and get shorter as I iterated. An added benefit: it looked much sleeker.
Configuring the robot
Once I had my hardware ready to go, I jumped into configuring all the bits and pieces using Viam's web app. It was unbelievably easy, all thanks to its super user-friendly interface!
My setup included a UR5 robot arm, an Intel RealSense camera, and a simulated component to mimic the candle attached to the arm.
The configuration of my hardware in Viam’s web app.
On the services side, I got a Data Management Service up and running for data capturing and syncing – thinking ahead for future training.
Setting up the Robot Frame Service
To make sure our Vision and Motion Services worked flawlessly, I set up a special frame for MenoRobot 2.0. It's like a roadmap, showing the robot where its arm is in relation to the camera.
I began by using the Config builder to integrate the key components of our system: the robotic arm, the candle attached to it, and the 3D camera.
Next, I defined the 'world state' - basically, mapping out the environment. Here, I pinpointed the obstacles, namely the table and the Chanukiah, almost like plotting a safe path in a maze. The goal? To ensure our Motion Service can skillfully guide the robotic arm, steering clear of these obstacles.
It's a crucial step because we definitely don't want any mishaps, like the robot accidentally knocking over the Chanukiah!
Training a machine learning model
I dove into training an ML model, aiming to detect candles in scenes.
Using Viam, I configured data capture and retrieved about 20 images, tagging each with bounding boxes around the candles. After adding these to the dataset for this project, I trained my object detection model.
The initial results were good, but I wanted perfection. So, I added more tagged images, refining the model further. The best part? Since I trained it as an updated version, it automatically synced with my robot, set to always grab the 'latest' version. In the end, I was really pleased with the spot-on results!
I then configured the ML Model Service to bring our trained machine learning model into play, and a Vision Service to handle all the image processing tasks.
Converting 2D images to the 3D world
Once my vision model spotted nine bounding boxes, it was time for some quick math magic. I calculated where the wick should be within each box – think top middle – and turned that into a 2D coordinate. This was like marking a spot on a flat map.
Next, I took this 2D coordinate and projected it onto my depth image. It's a bit like adding a third dimension to our flat map, which gave me the Z coordinate. Then I converted the 2D X and Y coordinates to coordinates in the 3D world and then voilà, I had the full X, Y, Z coordinates of each of the candle wicks in the robot's world state, like finding a precise location in a 3D world.
With these coordinates in hand, I was all set to start moving the arm – the real action begins!
Making the arm move with Python
I got to work with our Python SDK (which is my go-to toolkit) to maneuver the arm precisely to the X, Y, Z coordinates of each candle wick. I arranged the coordinates to ensure the Chanukiah candles were lit in the traditional order, from left to right, saving the central candle, or shamash, for last.
The program's pretty smart, too – it asks for the night of Hanukkah. This way, the robot knows exactly how many candles to light each night, reflecting the holiday's progression (one candle on the first night, two on the second, and so on).
I also programmed in a little 'rest' time between lighting each candle to ensure the match stays lit long enough to light a candle. And there's more: I factored in the melt rate of the candle doing the lighting, adjusting the coordinates accordingly as time ticks by.
Creating a last minute mobile app for the robot
Being shockingly ahead of schedule I coordinated with our Mobile Engineer, Clint, to help build a flutter app that streams from my robot my scene with candle bounding boxes and lets a user of the app select what night of the holiday it is and then run the robot!
The mobile app I created, in collaboration with Clint, using Viam’s Flutter SDK.
With a bit more time I would train another model that detects if a candle is lit and I would program the arm to retry until the candle is.
Within about a week I got everything working! A true miracle.
Reflecting on the build experience
Reflecting on this project, here's what stood out to me:
- Speed and Efficiency: Completing this project in under a week with less than 500 lines of code was a big deal, especially as an engineer without formal robotics training. It's a testament to what Viam can do.
- Exploring Capabilities: I've always been curious about integrating computer vision with motion planning to pinpoint and move to specific positions in the real world. This project proved that it's totally doable with Viam – a big win in my book!
- AI Chat Support: The AI chat feature in the Viam app was a game-changer. It was incredibly helpful and made programming and configuring the system much smoother than I expected.
- Future Possibilities: This experience has sparked a bunch of ideas. I'm particularly excited about the potential of creating a painting robot next using the arm and the Intel RealSense.
- Teamwork Makes the Dream Work: I can't stress enough how amazing my coworkers are. Their support, answers to my endless questions, and brainstorming help were invaluable. It's incredible working alongside such talented and brilliant people.
This project wasn't just about building something cool; it was a journey of learning, validation, and collaboration that has left me excited for the next challenge!
Eager to explore more in robotics? Dive into our Viam Documentation for in-depth tech insights, follow us on social media for the latest projects and updates, and join our Discord community to chat with our team and get inspired. In the meantime, watch the video below of MenoRobot 2.0 in action!
* Chanukiah is colloquially known as a Menorah, the candelabra for Hanukkah
