Research Institute Creates AI Framework and Tools for Autonomous Industrial Robots

San Antonio, TX, USA – The Southwest Research Institute (SwRI), a non-profit R&D organization focused on basic and applied engineering and research, has introduced an image reconstruction framework and tools for AI-based classification systems that allow industrial robots to visually scan and reconstruct an object, then perform tasks on it.

The solution “intelligently classifies regions and surface textures of parts at different stages of the job,” says Matt Robinson, robotics R&D manager at SwRI (San Antonio, TX, USA; It can be applied to grinding, painting, polishing, cleaning, welding, sealing and other industrial processes.

In traditional robotic programming, a computer-aided design (CAD) expert designs the work, sometimes including robotic processing information, but this approach can be slow and inflexible. To overcome this hurdle, SwRI developed its approach using Scan-N-Plan, a set of open-source tools that use 3D scanning techniques “to generate part geometry and location in real time,” according to one. brochure of the ROS-Industrial consortium. -Americas (San Antonio, TX, USA,, which manages the project. The engineers also used ROS 2, a suite of open source middleware tools, which are maintained by the non-profit organization Open Robotics (Mountain View, CA, USA;

The SwRI approach involves an image reconstruction framework that creates high-fidelity mesh maps of objects using the Open3D 3D data processing library. This is an update to an earlier reconstruction framework, which proved difficult to set up and resulted in precision issues at the edges of the mesh, according to a blog post on the website of the ROS-Industrial consortium.

Using the framework, the first step is to scan a part with an RGB-D camera, mounted on a robot. “We calibrate the location of the camera, so we know precisely where it is in space at all points,” says Tyler Marr, research engineer at SwRI. The software creates the 3D mesh using color and depth images combined with position data.

Marr said the system is flexible and will work with any 3D camera. “The requirement is that we have in-depth images,” says Marr, adding that the process can be scaled to color or grayscale images.

As the camera scans a part, a live feed of the process is shown on a monitor, allowing human operators to go back and program the system to re-scan areas, if needed, before exporting the completed mesh .

Robinson says this approach would be especially useful in organizations with a wide range of activities to automate.

Once the mesh is created, simple menus help users generate toolpaths for the robot to do the work. These toolpaths are then converted into robot paths, which the robot then executes, Marr explains. The software also allows an operator to monitor the robot’s work while it is in progress.

To show industrial companies how to use the reconstruction framework and classification tools, SwRI has created a demo of an aircraft part sanding robot. In this application, SwRI engineers used the RealSense™ D455 depth camera from Intel (Santa Clara, CA; They created a video of the demonstration as a teaching tool, and it is available below.

The demo software is an open-source resource. Additionally, the ROS Industry Consortium plans to develop and publish on its website advanced workshops and training sessions on using ROS 2 to automate robotic processes involving sanding, painting, and other common industrial processes. . The goal of the project, says Robinson, was “to provide a general framework to help people develop these kinds of solutions.”

Mavis R. Bernier