This blog post is part of a series summarising my PhD project and the steps to complete my degree. The information in this blog post is an overview of the human-centric approach of Chapter 3 used to inspire a novel gripper design. Refer to thesis for full details.
Introduction
In the previous post of this series, it was noted that current fabric grippers have limitations in their applications and often involve complex components such as tendon-driven mechanisms [Koustoumpardis 14, Von Drigalski 17], compliant joints [Donaire 20, Le 13], or microneedles [Ku 20]. This complexity arises due to the challenge of designing end-effectors for fabric manipulation, which involves multifaceted considerations such as the desired application, intended capabilities, and technical feasibility. Researchers have shown interest in assessing the manipulation capabilities of fabric grippers, as demonstrated by Borras et al. [Borras 20]. Their work was motivated by the author’s recognition that existing grasp taxonomies must be revised to characterise textile manipulation. In addition, their developed schema led directly to the gripper presented by Donaire et al. [Donaire 20].
The framework of Borras et al. [Borras 20] defines fabric grasps from an object-centric perspective by studying previous devices. This framework allows researchers to outline desired grasp contacts and inform the necessary geometries of new grippers. Cutkosky [Cutkosky 89] defines object-centric characteristics as observing information from the manipulation target or task context. While the object-centric perspective is valuable, it does not directly correlate to a gripper’s behaviour and ability, as noted by Bullock et al. [Bullock 12]. They further note that hand-centric studies can directly compare the capabilities of different grippers and formalise manipulation primitives. Given these facets, the first research chapter of my thesis proposed a systematic approach to using taxonomies that define human grasp poses and behaviour. This approach established unique gaps that would inspire a unique gripper design.
I utilised a selection of grippers from previous research and analysed them from a hand-centric perspective. The hand-centric approach focuses on the manipulator’s pose and motion to assess the grasp or manipulation technique. I adapted two general anthropomorphic hand-centric manipulation taxonomies to concentrate on fabric handling [Bullock 12, Feix 15]. This method offers insight to guide gripper design through formalised manipulation definitions and human morphological features. In addition, the technical features of surveyed grippers were considered, including the grasping forces, control mechanisms, design features, and device applications. The goal was to create a gripper design that simplifies complex features and encompasses the full range of manipulation skills and features required for sorting textile waste in pick-and-place tasks, thereby having an immediate practical application in textile recycling facilities.
Studying Grippers with Taxonomies
Grippers Surveyed
The surveyed grippers are part of a subset of the literature that meets specific selection criteria. These grippers were from research projects and were explicitly designed for fabric manipulation. They have advanced capabilities beyond simple grasping, such as “rubbing” held materials to identify tactile features (haptic exploration). Additionally, some grippers can make collision-rich grasping motions that leverage the environment to robustly grasp flattened materials, known as environmentally constrained (EC) grasping. Another considered skill is the ability to adjust the grasp point on a held garment without letting go, also known as grasp gaiting.
Due to the selection process, several devices were omitted, including grippers with extra components such as electromagnets to attract metal components [Marullo 20, Dragusanu 22], microneedles with a suction mechanism inspired by the lamprey fish [Ku 20], or roller fingertips with a brush-like coating to assist with the grasping of flattened materials [Abe 20, Yamazaki 21]. This survey aims to qualify previous grippers designed for fabric manipulation under human-centric definitions. Including devices with these complex features would be problematic when describing the anthropomorphic taxonomies. This survey focuses on directly relevant grasping skills and human-comparable grippers to establish unique gaps and a proposed gripper design scope.
Eight research projects (10 grippers overall) were evaluated as shown in the list and image below:
- CLOTHILDE Project Gripper [Donaire 20]
- CloPeMa Gripper [Le 13, Le 15b]
- 3 Fingered Under-Actuated Gripper [Koustoumpardis 14]
- NAIST M2S Openhand Gripper [Von Drigalski 17a, Von Drigalski 17b]
- Cloth Gripper Based On CAM Mechanism [Koustoumpardis 17]
- 3 Fingered Anthropomorphic Gripper [Ono 01]
- Dual Pincers Prismatic Gripper [Shibata 12]
- Prismatic Edge Tracing-Based Grippers [Sahari 10]
- Basic Gripper
- Roller Gripper
- Inchworm Gripper
The surveyed grippers described above.
Customised Taxonomies
My research approach was to create fabric-specific manipulation taxonomies using hand-centric characteristics. First, the F-GRASP, a derivation of the GRASP taxonomy [Feix 15], describes grasp characteristics in a matrix structure, covering a comprehensive range of anthropomorphic hand configurations we use daily. The original GRASP taxonomy described 33 hand configurations and represented a significant step in further understanding anthropomorphic dexterous manipulation in a robotic context.
These grasping configurations are described by the neuroscientific characteristics of virtual fingers, opposition types, and grasp type [Iberall 97], alongside the thumb rotation, known as thumb abduction. My research used the GRASP taxonomy to create the F-GRASP taxonomy, which uses the same framework but targets the grasp configurations the surveyed grippers use to manipulate fabric. As the image below shows, only six grasps from the original 33 of the GRASP taxonomy are used while manipulating deformable materials.
A derived version of the GRASP taxonomy from Feix et al. [Feix 15], modified for fabric handling specifically (F-GRASP).
The F-Grasp taxonomy adequately describes grasp poses, but an additional classification system is needed to describe the motions of the gripper during dexterous acts such as grasping, gaiting, and haptic exploration. The In-Hand Dexterous Manipulation (IHDM) taxonomy by Bullock et al. [Bullock 12] presents a binary tree based on parameters defined in in-hand dexterous manipulation and contact descriptors surrounding a manipulation task. This taxonomy conveys the complexity of the manipulation while describing critical motion elements. Unlike other taxonomies that focus on specific manipulation primitives or grasps, the IHDM taxonomy provides an abstract description of anthropomorphic behaviour without relying on object properties.
Therefore, my first research contribution also presented a derived taxonomy specific to fabric manipulation for dexterous manipulation motions called the F-IHDM taxonomy. Actions are classified by the contact nature and motion parameters from a hand-centric viewpoint, enabling the definition of various robotic end-effector sequences of manipulation. Five parameters define a dexterous manipulation action, as listed below. The abbreviations indicate whether the parameter is active in defining manipulations, as shown below in the visualisation of the F-IHDM taxonomy.
- Contact/No Contact (C/NC) - Refers to whether an interaction makes contact with an item or the environment.
- Prehensile/Nonprehensile (P/NP) – An interaction with a single contact point represents nonprehensile interactions, while multiple contact points represent a prehensile action. Alternatively, if the grip can stabilise an object regardless of external forces, it is also considered prehensile.
- Motion/No Motion (M/NM) – Refers to an interaction where movement relative to a world coordinate frame occurs.
- Within-Hand/Not Within-Hand (W/NW) – If a motion is present, this parameter describes whether the movement comes from the wrist or occurs within the manipulator.
- Motion at Contact/No Motion at Contact (A/NA) – If contact and motion are present, this parameter details whether contact is significantly translating or rotating a held object concerning coordinate frames based on the contact points.
In addition, one can describe the direction of in-hand dexterous manipulation actions between the finger motion and hand coordinate frame. As shown in the image below in scenarios 7 and 8, the collision direction is also defined in the F-IHDM framework.
A derived version of the IHDM taxonomy from Bullock et al. [Bullock 12], modified for fabric handling specifically (F-IHDM).
The two taxonomies capture different aspects of human-like skilful manipulation. Bullock et al. [Bullock 12] propose that combining the IHDM taxonomy with a grasp classifier can effectively describe grasp pose and behaviour during manipulation sequences. This suggestion informs the basis of the taxonomy survey, which examines and compares how manipulators designed for fabric manipulation are applied in practice.
The F-GRASP taxonomy provides definitions that can be applied to a manipulator’s pose while holding a fabric item in different manipulation stages. The F-IHDM taxonomy then provides a series of classifications to describe manipulator behaviour throughout a manipulation sequence. As a manipulator moves from a stationary position to grasp the fabric, assigned tags can formalize the stages of manipulation. For example, moving toward fabric would be classified as (1) NC-M-NW before grasping the fabric with a two-fingered pinch grasp using the (8) C-P-M-W-A action.
My research aims to identify a unique gripper design. To achieve this, we study the grasp poses used by previous grippers and their dexterity. The F-GRASP and F-IHDM taxonomies serve as our guide, providing a framework for examining previous devices developed for deformable manipulation and identifying gaps in the current state-of-the-art. These gaps will then inform the requirements for building a new manipulator aimed at fabric recycling applications.
Evaluating Gripper Characterisitics
The first step of investigating the surveyed grippers was to discuss the grasp characteristics; the F-GRASP taxonomy is first used to describe the grasp poses the various manipulators are capable of. In addition to the grasp pose characteristics, I also looked at the grasp forces applied by the grippers to grasp clothing in their varied applications. The grasp characteristics of each of the surveyed manipulators are capable of is shown in the image below, highlighting the possible configurations and grasp force ranges. One can refer to the thesis document for the in-depth study of the grasp characteristics. However, after examining the grasping details, a few takeaways became apparent.
The anthropamorphic grasp poses the surveyed manipulators are capable of.
First, manipulators specifically targeting deformable manipulation did not require a wide range of grasp poses. Considering that the grippers with the more diverse applications only used precision grasps, this reinforces the perspective of point-to-point grasping being sufficient for a wide range of fabric manipulation tasks. Meanwhile, the manipulators with a broader range of configurations did not demonstrate wider applicability.
Next, from the examined manipulators, the gripper designed by Le et al. [Le 13] was found to have the widest range of grasping forces, with a maximum of 30 newtons. It also had the ability to adjust the strength of its grip. These technical features were incorporated to enable the gripper to extract clothing from a tangled pile of laundry while also allowing for haptic exploration. Many grippers typically use a force of up to 5 newtons to handle individual pieces of clothing. Overall, these details suggest that a manipulator designed for fabric handling should be able to temporarily withstand higher grasping forces for untangling scenarios or heavy garments. However, reduced grasping forces of up to 10N should suffice for most simple tasks.
Fabric can present itself in states that make the initial grasping action difficult. Grasps that leverage the environment allow humans to combat uncertainty and perform a robust grasping motion. While in a flattened state, fabric requires this collision-rich skill during manipulation. One can observe two broad techniques of environmentally constrained (EC) grasping applied throughout the surveyed manipulators. Biomimetic grasping is an action that involves dragging a finger across a flattened textile item, producing a protrusion in the fabric’s body for the effector to grasp. Alternatively, an insertion approach grasps by sliding a thin fingernail-like appendage between the garment and the surface it lies upon. Both approaches are shown in the image below.
An example of biomimetic grasping (left) and insertion grasping (right, ).
Using the F-IHDM taxonomy, both EC grasping motions performed by the surveyed manipulators are evaluated throughout their execution. The image below shows an example of two manipulators [Donaire 20, Koustoumpardis 14] performing these motions with classification tags from the F-GRASP and F-IHDM taxonomies. A few key details became apparent while evaluating both approaches to EC grasping.
Example grasping pipelines of biomimetic grasping (left, [Koustoumpardis 14]) and insertion grasping (right, [Donaire 20]).
First, both approaches have distinct benefits and limitations. Biomimetic grasping occurs when all fingers are in contact with a flattened garment or the surface environment, thereby deforming a garment into a grasp. Thus, contact points constantly change between the fabric and the environment surface. However, little to no motion occurs at the contact between the manipulator and the fabric as grasping occurs. Alternatively, insertion grasping preserves the initial state of the garment by performing a motion that constantly shifts the contact between the manipulator and the fabric. Usually, devices performing insertion grasping require joints with compliant joints to collide safely with the surface before sliding under a garment. In addition, grasping actions are limited to locations where the edge of a garment meets the table.
While biomimetic grasping also requires compliance in the joints, grasping at any point on a flattened body remains desirable for the pick-and-place sorting of textile waste. In addition, rather than integrating compliance into joints by using deformable silicon on springs onto the mechanism, one could use an actuator control mechanism that incorporates impedance elements to provide the same functionality while conforming to rigid body kinematics and dynamics modelling. Such a feature would also allow biomimetic grasping to control the force exerted on the environment surface while dragging inwards, a capability none of the surveyed grippers could achieve. Furthermore, the surveyed grippers executing biomimetic grasping could not perform the action from a diverse range of wrist orientations.
Many grippers also can perform grasp gaiting and haptic exploration. For both skills, grasp force modulation is an essential capability. While some of the surveyed grippers use unique approaches to explore grasp gaiting, the simplest method is to grasp force modulation, where the material is held at a force that allows it to slide between the fingertips. Haptic exploration was also seen in several devices to infer tactile features of held garments, in which they would rub two fingers together. Such capabilities were also associated with grasp force modulation. This discussion has further details to consider, such as the friction parameters. These are omitted from the discussion for brevity and can be found in the thesis.
Finding a Unique Solution
The lateral grasp is described as the thumb pressing into the side of the index finger, as shown below. Based on the literature survey and taxonomy analysis shown above, it remains a unique inspiration for a gripper design.
The lateral grasp hand configuration in two different poses, with the index finger's MCP joint both retracted and extended.
The design based on the lateral grasp offers several key advantages. First, a single serial-chain structure can represent actuation and must mainly replicate thumb motions. However, as the image above shows, the rotation of the index finger’s MCP joint can play a role in the lateral grasp’s configuration. Such motion can be accounted for in the serial-link design structure while only impacting the thumb-like appendage’s motion.
The primary motion for forming lateral grasps originates from the behaviour of the thumb. Additionally, actuation can occur at each joint without necessitating subordinate joints or complex actuation mechanisms. Grasps are formed by pressing the thumb into the side of the index finger in a point-to-point grasping configuration. As mentioned, this method should be adequate for various textile manipulation tasks. While this grip may not be suitable for more delicate deformable manipulation tasks such as folding or placing an item on a coat hanger, it remains an acceptable grasping configuration for pick-and-place sorting.
In addition to these characteristics, the lateral grasp was a hand configuration capable of performing biomimetic grasping, grasp gaiting and haptic exploration. Through the use of an actuator and control mechanism, the proposed gripper will achieve these capabilities through grasp force modulation. Previous devices from literature and the taxonomy survey could not perform the skills of biomimetic grasping, haptic exploration, and grasp gaiting skills simultaneously. While promising, the taxonomy evidence alone does not confirm that such a gripper based on the lateral grasp is an ideal solution. Therefore, the final component of this research set out to validate the proposed design via a human observation study.
Using Human Observation to Validate the Observations
The proposed observational study explores how peoples’ grasps affect their ability to manipulate clothing. The study involved 20 participants who were instructed to perform various tasks involving clothing handling. Firstly, they were asked to manipulate a single t-shirt on a table by grasping and folding it. Then, they were asked to manipulate clothing in a heap by extracting a garment from under the pile.
After that, they performed a sorting task and a combined folding and sorting task. These instructions were repeated three times, with participants freely manipulating the fabric, using only their index finger and thumb, and finally using only a lateral grasp. The participants were observed using a four-camera setup, and the duration of each task was recorded. A survey was also provided to the participants after completing the tasks.
An individual participating in the human observation study.
The study analyzed the task duration data and participant surveys. One key finding was that regardless of grasp constraints, all human participants could complete tasks involving folding, sorting, and manipulating clothing. However, participants did not react positively to the lateral constraint. Some mentioned that it felt unfamiliar or counter-intuitive or described needing additional steps to reach an ideal configuration under the lateral constraint. The box plot distributions below show that grasp constraints generally increased task duration when folding was involved. However, sorting tasks did not show this effect. In fact, the duration decreased as the tasks were repeated.
The boxplot distributions of durations for tasks under the three grasping conditions.
Folding a Shirt Sorting Clothing
Folding and Sorting Clothing
The data indicates that folding garments is more complex than two fingers can be involved. Approaches involving more than two fingers and a point-to-point grasp will be required. While participants found difficulties using the lateral grasp, it remains a simple mechanical configuration that could be useful for sorting applications. Grippers using the full capabilities of the human hand or the pincer grasp could remain complex if replicated in a gripper design. In contrast, the lateral grasp mainly uses the motions of the thumb to press into the side of the index finger.
Confirming a Design Inspiration
After analysing the taxonomy and conducting a user study, it became clear that the next stage of my thesis would involve designing a gripper based on the lateral grasp configuration. The manipulator should be capable of exerting a grasping force of 30 newtons while being able to adjust the applied force. These features allow the gripper to extract garments from tangled scenarios, explore tactile features, grasp gaiting and environmentally constrained grasping. In developing this manipulator, the goal is to maintain a simple mechanical system, such as a serial-chain manipulator, while retaining a wide range of dexterous skills. Additionally, the gripper only needs to perform point-to-point grasps due to its scope of application in sorting. The next step of the thesis involves turning this conceptual design into a hardware prototype.
Use this link to view the next post surrounding my PhD project. This next post outlines how I took the observations of this post and used them to model and fabricate a novel gripper.
References
[Borras 20] Julia Borras, Guillem Alenya, and Carme Torras. “A grasping-centered analysis for cloth manipulation”. IEEE Transactions on Robotics, 36(3):924–936, 2020.
[Bullock 12] Ian M Bullock, Raymond R Ma, and Aaron M Dollar. “A hand-centric classification of human and robot dexterous manipulation”. IEEE Transactions on Haptics, 6(2):129–144, 2012.
[Donaire 20] Sonia Donaire, Julia Borras, Guillem Alenya, and Carme Torras. “A versatile gripper for cloth manipulation”. IEEE Robotics and Automation Letters, 5(4):6520–6527, 2020.
[Feix 15] Thomas Feix, Javier Romero, Heinz-Bodo Schmiedmayer, Aaron M Dollar, and Danica Kragic. “The grasp taxonomy of human grasp types”. IEEE Transactions on Human-Machine Systems, 46(1):66–77, 2015.
[Koustoumpardis 14] Panagiotis N Koustoumpardis, Kostas X Nastos, and Nikos A Aspragathos. “Underactuated 3-finger robotic gripper for grasping fabrics”. In 2014 23rd International Conference on Robotics in Alpe-Adria-Danube Region (RAAD), pages 1–8. IEEE, 2014.
[Ku 20] Subyeong Ku, Jihye Myeong, Ho-Young Kim, and Yong-Lae Park. “Delicate fabric handling using a soft robotic gripper with embedded microneedles”. IEEE Robotics and Automation Letters, 5(3):4852–4858, 2020.
[Le 13] Loan Le, Matteo Zoppi, Michal Jilich, Raffaello Camoriano, Dimiter Zlatanov, and Rezia Molfino. “Development and analysis of a new specialized gripper mechanism for garment handling”. In ASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pages V06BT07A013–V06BT07A013. American Society of Mechanical Engineers, 2013.
[Von Drigalski 17] Felix Von Drigalski, Daiki Yoshioka, Wataru Yamazaki, Sung-Gwi Cho, Marcus Gall, Pedro Miguell Uriguen Eljuri, Viktor Hoerig, Ming Ding, Jun Takamatsu, Tsukasa Ogasawara, et al. “NAIST Openhand M2S: A Versatile Two-Finger Gripper Adapted for Pulling and Tucking Textile”. In 2017 First IEEE International Conference on Robotic Computing (IRC), pages 117–122. IEEE, 2017