Gonzalez BOHME, Luis Felipe

This study introduces a novel, hand-drawn language designed to foster human-robot collaboration in wood stereotomy, central to carpentry and joinery professions. Based on skilled carpenters’ line and symbol etchings on timber, this language signifies the location, geometry of woodworking joints, and timber placement within a framework. A proof-of-concept prototype has been developed, integrating object detectors, keypoint regression, and traditional computer vision techniques to interpret this language and enable an extensive repertoire of actions. Empirical data attests to the language’s efficacy, with the successful identification of a specific set of symbols on various wood species’ sawn surfaces, achieving a mean average precision (mAP) exceeding 90%. Concurrently, the system can accurately pinpoint critical positions that facilitate robotic comprehension of carpenter-indicated woodworking joint geometry. The positioning error, approximately 3 pixels, meets industry standards.

In the field of construction, human-robot collaboration and mixed reality (MR) open new possibilities. However, safety and reliability issues persist. The lack of flexibility and adaptability in current preprogrammed systems hampers real-time human-robot collaboration. A key gap in this area lies in the ability of the robot to interpret and accurately execute operations based on the real-time visual instructions and restrictions provided by the human collaborator and the working environment. This paper focuses on an MR-based human-robot collaboration method through visual feedback from a vision-based collaborative industrial robot system for use in wood stereotomy which we are developing. This method is applied to an alternating workflow in which a skilled carpenter lays out the joinery on the workpiece, and the robot cuts it. Cutting operations are instructed to the robot only through lines and conventional “carpenter’s marks”, which are drawn on the timbers by the carpenter. The robot system’s accuracy in locating and interpreting marks as cutting operations is evaluated by automatically constructing a 3D model of the cut shape from the vision system data. A digital twin of the robot allows the carpenter to previsualize all motions that are required by the robot for task validation and to know when to enter the collaborative workspace. Our experimental results offer some insights into human-robot communication requirements for collaborative robot system applications in timber frame construction.