OpenAI server node for ROS2 Applications
If you don't have an image processing pipeline or a GPU on your robot, calls to OpenAI can replace much of that functionality. As of July 2024 we generally recommend yes/no questions, otherwise you will need to parse the response yourself. This package includes a simple filter which parses for yes/no. For example:
- Is it safe for the robot to drive forward? Please respond in one word, yes or no.
- Is the door open? Please respond in one word, yes or no.
- Is there a customer at that table? Please respond in one word, yes or no.
- Did the robot successfully grasp the cube? Please respond in one word, yes or no.
- Does the control panel look identical to the last time I checked? Please respond in one word, yes or no.
Some of these examples are useful for reinforcement learning, i.e. for determining if a reward was earned.
Requires an OpenAI key as described here: https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key
Set it as an environment variable: export OPENAI_API_KEY="..."
sudo apt install -y libcurl4-openssl-dev
sudo apt install -y nlohmann-json3-dev
sudo apt install -y libb64-dev
Here's a quick example:
ros2 run robosoft_openai openai_server
Send it a prompt. Here's an example that returns a bool from a yes/no question. Note that the image field of the service request may be left empty...
ros2 service call /openai_server ai_prompt_msgs/srv/StringImagePrompt prompt:\ "Are you a pirate? Please respond with a one-word answer, yes or no"
Here's an example that returns a full string...
ros2 service call /openai_server ai_prompt_msgs/srv/StringImagePrompt prompt:\ "Are you a pirate?"
Here's an example client which sends an image of a wooden table and prompts whether it is indeed a wooden table:
ros2 run robosoft_openai example_client
If you use this work, please cite it like so:
- Zelenak, A., Lock, J., & Aldrich, B. (2024) An OpenAI Server for ROS2. Github. https://github.com/robosoft-ai/robosoft_openai