Capturing forceful interaction with deformable objects using a deep learning- powered... 翻译

• 该文档由Doc2X翻译提供解析与翻译, 想看更多论文翻译欢迎来Doc2X
• This document is provided with parsing and translation by Doc2X. For more translated papers, feel free to visit Doc2X.
• 原文地址 https://www.nature.com/articles/s41467-024-53654-y
• 项目地址：https://github.com/jeffsonyu/ViTaM

Capturing forceful interaction with deformable objects using a deep learning- powered stretchable tactile array

利用深度学习驱动的可拉伸触觉阵列捕捉与可变形物体的强力交互

Chunpeng Jiang ${\text{@}}^{1,7}$ ,Wenqiang ${\mathrm{Xu}}^{2,7}$ ,Yutong ${\mathrm{Li}}^2$ ,Zhenjun ${\mathrm{Yu}}^2$ , Longchun Wang ${}^1$ ,Xiaotong Hu ${}^{1,3}$ ,Zhengyi Xie ${}^{1,3}$ ,Qingkun Liu O’,Bin Yang ${}^1$ , Xiaolin Wang1,Wenxin Du2,Tutian Tang ${\text{O}}^2$ ,Dongzhe Zheng ${}^2$ ,Siqiong Yao ${}^4$ , Cewu Lu ${}^{5,6}⊠$ & Jingquan Liu ${}^1⊠$

Capturing forceful interaction with deformable objects during manipulation benefits applications like virtual reality, telemedicine, and robotics. Replicating full hand-object states with complete geometry is challenging because of the occluded object deformations. Here, we report a visual-tactile recording and tracking system for manipulation featuring a stretchable tactile glove with 1152 force-sensing channels and a visual-tactile joint learning framework to estimate dynamic hand-object states during manipulation. To overcome the strain interference caused by contact with deformable objects, an active suppression method based on symmetric response detection and adaptive calibration is proposed and achieves $97.6\%$ accuracy in force measurement, contributing to an improvement of $45.3\%$ . The learning framework processes the visual-tactile sequence and reconstructs hand-object states. We experiment on 24 objects from 6 categories including both deformable and rigid ones with an average reconstruction error of $1.8\mathrm{cm}$ for all sequences, demonstrating a universal ability to replicate human knowledge in manipulating objects with varying degrees of deformability.

在操作过程中捕捉与可变形物体的强力交互，有利于虚拟现实、远程医疗和机器人等应用。由于物体变形的遮挡，复制完整的手-物体状态及其几何形状是具有挑战性的。在此，我们报告了一种视觉-触觉记录和跟踪系统，该系统配备有1152个力传感通道的可拉伸触觉手套，并采用视觉-触觉联合学习框架来估计操作过程中的动态手-物体状态。为了克服与可变形物体接触引起的应变干扰，提出了一种基于对称响应检测和自适应校准的主动抑制方法，并在力测量中实现了 $97.6\%$ 的精度，贡献了 $45.3\%$ 的改进。学习框架处理视觉-触觉序列并重建手-物体状态。我们在6个类别中的24个物体（包括可变形和刚性的物体）上进行了实验，所有序列的平均重建误差为 $1.8\mathrm{cm}$，展示了在操作不同程度可变形物体时复制人类知识的通用能力。

Human-machine interaction (HMI) systems serve as gateways to the metaverse, acting as bridges between the physical world and the digital realm. A natural user interface in HMI allows humans to perform natural and intuitive control ${}^1$ . Although the non-forceful interfaces such as hand gestures (Fig. 1A(i)) can be tracked using technologies like inertial measurement units ${\left(\mathrm{IMU}\right)}^2$ ,electromyography (EMG) ${\text{sensors}}^{3,4}$ ,strain sensors ${}^{5,6}$ ,video recording ${}^7$ and triboelectric sensors ${}^8$ , the forceful interfaces such as interaction with objects, i.e., the human manipulation,are less explored ${}^{9,10}$ . Capturing forceful human Nature Communications | (2024)15:9513 manipulation has extensive potential applications, such as virtual reality ${\left(\mathrm{VR}\right)}^{11,12}$ ,telemedicine ${}^{13}$ ,robotics ${}^{14,15}$ ,and contributes to real-world understanding for large artificial intelligence (AI) models ${}^{16}$ . Replicating the hand-object interplay is the first step to applying human manipulation knowledge in these applications. However, the hand-object states captured in previous research were far from complete. They mainly explore tasks like semantic recognition and spatial localization to predict object category and position (Fig. 1A(ii)) ${}^{17-20}$ . Imagine a general manipulation case: when a human rubs plasticine to

人机交互（HMI）系统充当通往元宇宙的门户，作为物理世界和数字领域之间的桥梁。HMI中的自然用户界面允许人类进行自然和直观的控制 ${}^1$。尽管非力反馈界面（如手势（图1A(i)））可以通过惯性测量单元 ${\left(\mathrm{IMU}\right)}^2$、肌电图（EMG）${\text{sensors}}^{3,4}$、应变传感器 ${}^{5,6}$、视频录制 ${}^7$ 和摩擦电传感器 ${}^8$ 等技术进行追踪，但力反馈界面（如与物体的交互，即人类操纵）却较少被探索 ${}^{9,10}$。捕捉力反馈的人类操纵具有广泛的应用潜力，例如虚拟现实 ${\left(\mathrm{VR}\right)}^{11,12}$、远程医疗 ${}^{13}$、机器人学 ${}^{14,15}$，并为大型人工智能（AI）模型提供现实世界的理解 ${}^{16}$。复制手-物互动是应用人类操纵知识于这些应用的第一步。然而，先前研究中捕捉到的手-物状态远不完整。它们主要探索语义识别和空间定位等任务，以预测物体类别和位置（图1A(ii)）${}^{17-20}$。想象一个一般的操纵案例：当一个人摩擦橡皮泥时

---

${}^1$ National Key Laboratory of Advanced Micro and Nano Manufacture Technology,Shanghai Jiao Tong University,Shanghai,China. ${}^2$ School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China. ${}^3$ IFSA-DCI Team, Department of Micro/Nano Electronics, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China. ${}^4$ SJTU-Yale Joint Center of Biostatistics and Data Science, National Center for Translational Medicine, MoE, Key Lab of Artificial Intelligence, AI Institute Shanghai Jiao Tong University, Shanghai, China. ${}^5$ School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China. ${}^6$ Present address: School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China. ${}^7$ These authors contributed equally: Chunpeng Jiang, Wenqiang Xu. $⊠$ e-mail: lucewu@sjtu.edu.cn;

${}^1$ 国家先进微纳制造技术重点实验室，上海交通大学，上海，中国。 ${}^2$ 电子信息与电气工程学院，上海交通大学，上海，中国。 ${}^3$ IFSA-DCI 团队，微纳电子学系，电子信息与电气工程学院，上海交通大学，上海，中国。 ${}^4$ 上海交通大学-耶鲁大学生物统计与数据科学联合中心，教育部转化医学国家中心，人工智能关键实验室，上海交通大学人工智能研究院，上海，中国。 ${}^5$ 人工智能学院，上海交通大学，上海，中国。 ${}^6$ 现地址：电子信息与电气工程学院，上海交通大学，上海，中国。 ${}^7$ 这些作者贡献相同：姜春鹏，徐文强。 $⊠$ 电子邮件：lucewu@sjtu.edu.cn；

jqliu@sjtu.edu.cn

---

Fig. 1 | The tactile array-based glove, enhanced by deep learning, with an advantage of capturing the forceful interaction with deformable objects. A The (i) non-forceful and (ii) forceful interactions that involve human manipulation in the field of HMI and their response results demonstrating the progress from low to high dimensions. B The overview of our proposed ViTaM system: (i) A human-inspired joint perception method of processing cross-modal visual and tactile signals simultaneously during manipulation to realize state tracking; (ii) The sensing error caused by the strain at stretchable interfaces, which deteriorates the accuracy of force measurement and the application effectiveness of tactile sensors; (iii) The force recording solution that includes a high-density, stretchable tactile glove with active strain interference suppression and a VR interface to show the results of distributed force detections; (iv) The applications of object state estimation powered by a deep learning architecture, enabling the reconstruction of overall object geometry and the fine-grained surface deformation in the contact area, particularly for deformable items. a desired shape, he needs to sense the surface deformation and track the object’s geometric states. Tactile perception takes precedence in analyzing deformations within the contact area ${}^{21}$ ,while visual perception is utilized to estimate the overall object states (Fig. 1B(i)) ${}^{22}$ . Thus, to capture the comprehensive information during manipulation in a human-like ${\mathrm{way}}^{23}$ ,a system that can record the visual-tactile sensory data and estimate the fine-grained hand-object states is desired.

图 1 | 基于触觉阵列的手套，通过深度学习增强，具有捕捉与可变形物体强力交互的优势。A (i) 非强力交互和 (ii) 强力交互，涉及人机界面 (HMI) 领域中的人类操作及其响应结果，展示了从低维到高维的进展。B 我们提出的 ViTaM 系统概览：(i) 一种受人类启发的联合感知方法，在操作过程中同时处理跨模态视觉和触觉信号，以实现状态跟踪；(ii) 可拉伸界面应变引起的传感误差，这会降低力测量的准确性和触觉传感器的应用效果；(iii) 力记录解决方案，包括一个具有主动应变干扰抑制的高密度、可拉伸触觉手套和一个 VR 界面，以展示分布式力检测的结果；(iv) 由深度学习架构支持的对象状态估计应用，能够重建整体对象几何形状和接触区域的细粒度表面变形，特别是对于可变形物品。为了达到所需的形状，他需要感知表面变形并跟踪对象的几何状态。触觉感知在分析接触区域内的变形时占主导地位 ${}^{21}$，而视觉感知则用于估计整体对象状态（图 1B(i)）${}^{22}$。因此，为了以类似人类的方式 ${\mathrm{way}}^{23}$ 捕捉操作过程中的全面信息，需要一个能够记录视觉-触觉感官数据并估计细粒度手-物状态的系统。

Recording tactile data in hand-object interplay is challenging, particularly when recording forces on stretchable interfaces during interactions with deformable objects. On the one hand, the tactile array should feature a high-density distribution of force-sensing units to cover multiple contact areas during object operation. A natural choice is to integrate a tactile array into a wearable glove using textile ${\text{techniques}}^{24-26}$ . On the other hand,the tactile array should be a stretchable interface, ensuring conformal contact with deformable ${\text{objects}}^{27}$ . However,when in contact with a deformable surface,the extension or bending of the stretchable tactile array could encounter undesired interference with output signals due to the increasing strain (Fig. 1B(ii)) ${}^{28-32}$ . Unlike traditional rigid-rigid or flexible-rigid interfaces without stretchability, it is suggested that the normal force cannot be independently measured at the stretchable surfaces, otherwise the strain will significantly impair precise force measurement ${}^{33}$ . To achieve strain insensitivity, previous studies have typically solved the problem from structural or material perspectives. Structurally, methods include stretchable geometric structures ${}^{34-37}$ ,stress isolation structures ${}^{38-41}$ , and Negative Poisson’s ratio structures ${}^{42-44}$ . Material strategies include strain redundancy techniques ${}^{45-48}$ ,localized microcracking ${\text{techniques}}^{49-51}$ ,and nanofiber network encapsulation technique ${}^{52-54}$ . These techniques belong to the source-protection approach, concentrating on the “input end” of the sensor and aiming to reduce or eliminate the influence of strain interference on tactile arrays in an open-loop non-adaptive manner ${}^{55-57}$ . However,these techniques have three limitations: no quantitative assessment of the strain received, no quantitative assessment of the strain suppressed and no measurement standard considering real-time testing conditions. Thus, a closed-loop adaptive tactile data recording approach is desired, where the closed-loop monitoring quantitatively detects and suppresses the strain interference and adaptive force estimation highly suits the deformable interface with unpredictable or high degrees of freedom.

记录手-物体交互中的触觉数据具有挑战性，特别是在记录与可变形物体交互时在可伸展界面上的力时。一方面，触觉阵列应具有高密度分布的力传感单元，以覆盖物体操作过程中的多个接触区域。一个自然的选择是将触觉阵列集成到使用纺织材料 ${\text{techniques}}^{24-26}$ 的可穿戴手套中。另一方面，触觉阵列应是一个可伸展的界面，确保与可变形物体 ${\text{objects}}^{27}$ 的共形接触。然而，当与可变形表面接触时，可伸展触觉阵列的延伸或弯曲可能会由于应变增加而遇到不希望的输出信号干扰（图1B(ii)）${}^{28-32}$。与传统的无伸展性的刚-刚或柔-刚界面不同，研究表明在可伸展表面上无法独立测量法向力，否则应变将显著损害精确的力测量 ${}^{33}$。为了实现应变不敏感性，以往研究通常从结构或材料角度解决问题。在结构上，方法包括可伸展几何结构 ${}^{34-37}$、应力隔离结构 ${}^{38-41}$ 和负泊松比结构 ${}^{42-44}$。材料策略包括应变冗余技术 ${}^{45-48}$、局部微裂纹 ${\text{techniques}}^{49-51}$ 和纳米纤维网络封装技术 ${}^{52-54}$。这些技术属于源保护方法，集中在传感器的“输入端”，旨在以开环非自适应方式减少或消除应变干扰对触觉阵列的影响 ${}^{55-57}$。然而，这些技术有三个局限性：没有对接收的应变进行定量评估，没有对抑制的应变进行定量评估，且没有考虑实时测试条件的测量标准。因此，需要一种闭环自适应触觉数据记录方法，其中闭环监测定量检测和抑制应变干扰，自适应力估计高度适用于具有不可预测或高自由度的可变形界面。

Although closed-loop adaptive perception of tactile data facilitates the estimation of object deformation at the contact area, it is insufficient for recovering the complete object state, as some parts of the object remain out of contact. Thus, it is naturally necessary to adopt the assistance of global signals, such as visual perception, to obtain the full object geometry ${}^{58}$ . Previous works on visual-tactile joint learning have primarily relied on camera-based tactile sensors ${}^{59-61}$ ,benefiting from their locally high-resolution and 2D grid-like data format. Along with visual images, they have been utilized for hand pose estimation, in-manipulation object pose estimation,and geometric reconstruction ${}^{62,63}$ . However, these visual-tactile models capture the hand-object interaction in static settings and do not consider the temporal consistency of hand movements and object deformation ${}^{64}$ . Supplementary Table 1 compares visual-only, tactile-only, and visual-tactile modalities for object understanding and manipulation.

尽管闭环自适应触觉数据感知有助于估计接触区域的物体变形，但仅凭此无法恢复完整的物体状态，因为物体的某些部分仍然处于非接触状态。因此，自然需要借助全局信号，例如视觉感知，以获得完整的物体几何形状 ${}^{58}$。先前关于视觉-触觉联合学习的工作主要依赖于基于相机的触觉传感器 ${}^{59-61}$，得益于其局部高分辨率和二维网格状数据格式。结合视觉图像，它们已被用于手部姿态估计、操作中的物体姿态估计以及几何重建 ${}^{62,63}$。然而，这些视觉-触觉模型在静态设置中捕捉手-物体交互，并未考虑手部运动和物体变形的时间一致性 ${}^{64}$。补充表1比较了仅视觉、仅触觉和视觉-触觉模态在物体理解和操作方面的差异。

To capture the forceful human manipulation of deformable objects, this paper proposed a visual-tactile recording and tracking system for manipulation named ViTaM, which employs a high-density, glove-shaped stretchable tactile array for force recording and a deep learning framework for visual-tactile data processing and hand-object state estimation. Especially, the stretchable tactile array works in an output-focus sensing paradigm by measuring forces under different strains in a closed-loop adaptive manner. Based on the proposed negative/positive stretching-resistive effect, quantitative symmetric response detection and suppression evaluation of strain interference are achieved, enabling accurate force measurement on the stretchable interface with an accuracy of $97.6\%$ ,which has an improvement of $45.3\%$ compared with the uncalibrated measurements. Meanwhile, a point cloud sequence as visual observations captures the entire interaction process. During data processing and hand-object state estimation, the learning framework adopts two distinct neural network branches to encode visual and tactile information respectively, and reconstructs the fine-grained surface deformation and the complete object geometry. To demonstrate the generalization ability of the learning framework, we select 24 objects from 6 categories, including both deformable and rigid ones, and we can achieve an average reconstruction error of $1.8\mathrm{cm}$ over all sequences. This work marks a revolutionary advancement of perception tools for human manipulation, takes a step towards a more generic recording approach for both rigid and deformable objects, completes the last mile of forceful interaction by machine intelligence, and improves the learning framework for linking the physical world and the digital realm.

为了捕捉人类对可变形物体的强力操纵，本文提出了一种名为ViTaM的视觉-触觉记录和跟踪系统，该系统采用高密度、手套形状的Stretchable触觉阵列进行力记录，并使用深度学习框架进行视觉-触觉数据处理和手-物状态估计。特别是，Stretchable触觉阵列通过在不同应变下以闭环自适应方式测量力，工作在输出聚焦传感范式下。基于提出的负/正拉伸-电阻效应，实现了定量对称响应检测和应变干扰抑制评估，使得在Stretchable界面上进行精确力测量，精度达到$97.6\%$，相较于未校准测量提高了$45.3\%$。同时，点云序列作为视觉观测捕捉整个交互过程。在数据处理和手-物状态估计过程中，学习框架采用两个不同的神经网络分支分别编码视觉和触觉信息，并重建精细表面变形和完整物体几何形状。为了演示学习框架的泛化能力，我们从6个类别中选择了24个物体，包括可变形和刚性的物体，我们可以在所有序列上实现平均重建误差为$1.8\mathrm{cm}$。这项工作标志着人类操纵感知工具的革命性进步，朝着更通用的刚性和可变形物体记录方法迈出了一步，通过机器智能完成了强力交互的最后一英里，并改进了连接物理世界和数字领域的学习框架。

Results

结果

Overview of the ViTaM system

ViTaM系统概述

The design of the ViTaM system is rooted in the idea of capturing fine-grained information during forceful interaction with deformable objects. It records the manipulation process with a proposed high-density, stretchable tactile glove, and a 3D camera, and estimates the hand-object state at the geometric level with a proposed visual-tactile joint learning framework. When the user interacts with objects, the hand-object state at the contact area is recorded by the tactile glove. It has high-density tactile sensing units with a maximum of 1152 channels distributed all over the palm and can accurately capture the force dynamics with a frame rate of $13\mathrm{Hz}$ on the stretchable interface between the hand and object during interaction (Fig. 1B(iii)). Meanwhile, the hand-object state at the non-contact area is recorded by a high-precision depth camera. The captured force measurement and point cloud sequence are processed by the visual-tactile learning model proposed in this article, facilitating cross-modal data feature fusion, ultimately enabling the tracking and geometric 3D reconstruction of manipulated objects with varying deformability, including both deformable (e.g., elastic and plastic) and rigid ones (Fig. 1B(iv)).

ViTaM系统的设计理念在于在强力交互过程中捕捉与可变形物体之间的细粒度信息。它通过提出的高密度、可拉伸触觉手套和3D摄像头记录操作过程，并利用提出的视觉-触觉联合学习框架在几何层面上估计手-物体状态。当用户与物体交互时，触觉手套记录接触区域的手-物体状态。该手套配备高密度触觉传感单元，最多1152个通道分布在整个手掌，能够以$13\mathrm{Hz}$的帧率在手掌与物体之间的可拉伸界面上精确捕捉力的动态变化（图1B(iii)）。同时，非接触区域的手-物体状态由高精度深度摄像头记录。捕获的力测量值和点云序列由本文提出的视觉-触觉学习模型处理，促进跨模态数据特征融合，最终实现对不同变形性操作物体的跟踪和几何3D重建，包括可变形物体（例如，弹性和塑性）和刚性物体（图1B(iv)）。

Design and fabrication of the tactile glove

触觉手套的设计与制造

The tactile glove contains the following modules (Fig. 2A): tactile sensing blocks, a fabric glove, flexible printed circuits (FPCs), a multichannel scanning circuit, a processing circuit, and a bracelet. The naming labels of tactile sensing blocks are given in Fig. S1. Three types of FPCs connect the finger and palm sensing areas with the multichannel scanning circuit and processing circuit (Fig. S2). Fixing and encapsulation methods are detailed in Fig. S3. The modular design allows for optimal performance, on-demand density expansion, and detachability. The multi-channel scanning circuit (Fig. S4), which contains a force sensing circuit (Fig. S5A) and strain interference detection circuits (Fig. S5B), supports up to 1152 sensing units per frame, with the prototype demonstrating 456 sensing units. Additionally, a custom data transmission protocol ensures efficient and adaptable data transfer (Fig. S6). In Fig. S7A, the tactile glove demonstrates excellent wearability and conformability through two distinct gestures. To validate the yield rate of the prototype, external forces were applied to each area of the five fingers (Fig. S7B) and palm (Fig. S7C) of the tactile glove. Upon calculation, the yield rate was determined to be $97.15\%$ ,which is sufficient to meet the requirements of most human-machine interaction applications. Besides, the estimated costs of the tactile glove and hardware are $3.38 (Supplementary Table 2) and $26.63 (Supplementary Table 3), respectively, fostering widespread public acceptance ${}^{16}$ . In the future,due to the simplicity of the processing procedure and automation advancements in processing equipment (such as sewing machines), there is significant potential for mass production of this tactile glove.

触觉手套包含以下模块（图2A）：触觉传感块、织物手套、柔性印刷电路（FPCs）、多通道扫描电路、处理电路和手环。触觉传感块的命名标签见图S1。三种类型的FPCs将手指和手掌传感区域与多通道扫描电路和处理电路连接（图S2）。固定和封装方法详见图S3。模块化设计实现了最佳性能、按需密度扩展和可拆卸性。多通道扫描电路（图S4），包含力传感电路（图S5A）和应变干扰检测电路（图S5B），支持每帧高达1152个传感单元，原型展示了456个传感单元。此外，自定义数据传输协议确保了高效和适应性的数据传输（图S6）。在图S7A中，触觉手套通过两种不同的手势展示了出色的可穿戴性和适应性。为验证原型的产出率，对外力施加于触觉手套的五指（图S7B）和手掌（图S7C）的各个区域。经计算，确定产出率为$97.15\%$，足以满足大多数人机交互应用的需求。此外，触觉手套和硬件的预估成本分别为3.38美元（补充表2）和26.63美元（补充表3），促进了广泛的公众接受${}^{16}$。未来，由于处理程序的简便性和处理设备（如缝纫机）的自动化进步，这种触觉手套的大规模生产具有巨大潜力。

The tactile array consists of multiple tactile sensing blocks, and each block includes a positive strain sensor, a negative strain sensor, and a force sensor array (Fig. 2B(i)). The negative and positive strain electrodes are connected with the assembled composite film, respectively (Fig. 2B(ii)). Besides, the conductive fabric wires are sewn onto the assembled composite film, forming the row electrode array and column electrode array of the tactile force sensor array, with the rows and columns positioned perpendicular to each other. The fully woven wiring method is given in Fig. S8A and the overlap of a row electrode, an assembled composite film, and a column electrode forms a tactile sensing unit (Fig. S8B). As shown in the cross-sectional view of Fig. 2B(iii), the electrodes are tightly shuttled between the films, avoiding the use of an adhesive layer and showing better assembling. Different from conventional techniques like photolithography or screen print, which assemble the top and bottom electrode layers on the two sides of the sensing film, this fully woven wiring method requires no adhesive for layer contact, so interlayer delamination will not occur, leading to better reliability, conformality, and wear resistance. Adjacent blocks share the row and column electrodes (Fig. S9).

触觉阵列由多个触觉传感块组成，每个块包括一个正应变传感器、一个负应变传感器和一个力传感器阵列（图2B(i)）。负应变电极和正应变电极分别与组装的复合薄膜连接（图2B(ii)）。此外，导电织物导线被缝制到组装的复合薄膜上，形成触觉力传感器阵列的行电极阵列和列电极阵列，行和列相互垂直排列。完全编织布线方法如图S8A所示，行电极、组装复合薄膜和列电极的重叠形成一个触觉传感单元（图S8B）。如图2B(iii)的截面图所示，电极紧密地在薄膜之间穿梭，避免了使用粘合层，显示出更好的组装效果。与传统的光刻或丝网印刷技术不同，后者将顶部和底部电极层组装在传感薄膜的两侧，这种完全编织布线方法无需层间接触的粘合剂，因此不会发生层间剥离，从而具有更好的可靠性、一致性和耐磨性。相邻的块共享行电极和列电极（图S9）。

The above assembled composite film is composed of stacked positive effect membrane (the orange layer) and negative effect membrane (the blue layer), each linked to a strain detection module via electrode pairs. Figure S10 illustrates the fabrication of negative and

上述组装的复合薄膜由堆叠的正效应膜（橙色层）和负效应膜（蓝色层）组成，每层通过电极对连接到一个应变检测模块。图S10展示了负效应膜和正效应膜的制造过程。

Fig. 2 | Design, fabrication, and testing of the tactile glove with the capability of strain interference suppression. A The blow-up schematic of the high-density and stretchable tactile glove with a maximum sensing channel of 1152; B (i) The structure of a tactile sensing block with two pairs of strain electrodes, row electrode array, and column electrode array; (ii) the enlarged view showing the positions of strain electrodes; (iii) the side view of the tactile sensing block showing tight assembling. C The relative resistance variation curves of the positive effect membranes and negative effect membranes when subjected to a strain increasing from 0 to $50\%$ ,which is named a symmetric response to the sensing error of strain. D The closed-loop and quantitatively adaptive system for the detection and suppression of strain interference on the stretchable interface. positive effect membranes. The negative stretching-resistive effect was first proposed in our previous work ${}^{65}$ . The positive or negative effect of a film is determined by the content of carbon nanotubes (CNTs). A higher weight ratio of CNTs than $3.3\mathrm{wt}\%$ in the natural latex substrate leads to a negative stretching-resistive effect, while a lower CNT weight ratio results in a positive effect. In this experiment,the CNT contents of $5\mathrm{wt}\%$ and $2.9\mathrm{wt}\%$ are chosen because they show comparable resistance changes with similar amplitudes but opposite changing trends when the strain increases gradually from 0 to $50\%$ (Fig. 2C). This phenomenon enables the measurement of the strain variation across a wide range, covering the maximum extension region of the fingertip of $40\%{}^{66}$ .

图2 | 具有应变干扰抑制能力的触觉手套的设计、制造和测试。A 高密度和可拉伸触觉手套的放大示意图，最大传感通道数为1152；B (i) 具有两对应变电极、行电极阵列和列电极阵列的触觉传感块结构；(ii) 放大视图显示应变电极的位置；(iii) 触觉传感块的侧视图，显示紧密组装。C 正效应膜和负效应膜在应变从0增加到$50\%$时的相对电阻变化曲线，这被称为对称响应于应变传感误差。D 用于检测和抑制可拉伸界面上的应变干扰的闭环和定量自适应系统。正效应膜。负拉伸-抗性效应在我们之前的工作中首次提出${}^{65}$。薄膜的正或负效应由碳纳米管（CNTs）的含量决定。在天然乳胶基材中，CNTs的重量比高于$3.3\mathrm{wt}\%$会导致负拉伸-抗性效应，而较低的CNT重量比则产生正效应。在本实验中，选择CNT含量为$5\mathrm{wt}\%$和$2.9\mathrm{wt}\%$，因为它们在应变从0逐渐增加到$50\%$时显示出可比较的电阻变化，振幅相似但变化趋势相反（图2C）。这种现象使得能够测量大范围内的应变变化，覆盖指尖的最大延伸区域$40\%{}^{66}$。

Adaptive strain interference suppression method

自适应应变干扰抑制方法

To improve the accuracy of force measurements when manipulating deformable objects, an adaptive strain interference suppression method is proposed for the detection and suppression of strain interference on the stretchable interface (Fig. 2D). Generally, the tactile array outputs a signal variation with the applied force changing, and a curve depicting the relationship of the forces and the outputs is obtained (called force estimation curve). Inferring a force based on the specified and pre-tested force estimation curve is the conventional open-loop measurement method for tactile sensors. However, in this work, in addition to the forces exerted on the composite film, the strain interference is also detected in a closed-loop manner. Prior to force calibration, several tests are performed to measure the force estimation curves under several different strains (shown in the next section). The following steps shown in Fig. S11A offer the detailed process: (i) First, how can the presence of strain interference be determined? The positive and negative effect membranes are connected with the strain detection module, respectively, constituting a novel dual input mode. As soon as an increased strain interference occurs, the output voltage of the strain detection circuit promptly shows a synchronized rise edge and further generates a warning flag to alert the existence of strain interference; (ii) Second, how can the magnitude of the strain interference be quantitatively determined? If strain interference exists,the strain ${\epsilon }_{\mathrm{x}}$ can be inferred from the relative resistive variation curve based on Fig. 2C. This stage determines the appropriate calibration coefficient in the subsequent calibration step; (iii) Third, how to obtain the force estimation curve under ${\epsilon }_{\mathrm{x}}$ ? Since it is impractical to enumerate force estimation curves for all possible strain interferences, this study proposes a method called “local domain curve interpolation” to update the force estimation curve corresponding to ${\epsilon }_{\mathrm{x}}$ . As shown in Fig. S11,identifying the two strain values closest to ${\epsilon }_{\mathrm{x}}$ among the known conditions (named ${\epsilon }_{\text{\_up }}$ and ${\epsilon }_{\text{\_bottom }}$ ) and their corresponding force estimation curves (named curve_up and curve_bottom). Then, a new curve is interpolated between curve_up and curve_bottom in proportion to the distance between ${\epsilon }_{\mathrm{x}}$ and ${\epsilon }_{\text{\_up }},{\epsilon }_{\text{lottom }}$ . This interpolated curve (named curve_x) represents the force estimation curve corresponding to the strain interference ${\epsilon }_{\mathrm{x}}$ ; (iv) Fourthly,using curve_x to calculate the force under the strain interference ${\epsilon }_{\mathrm{x}}$ . The examples in Fig. S11B

为了提高操作可变形物体时力测量的准确性，提出了一种自适应应变干扰抑制方法，用于检测和抑制可拉伸界面上的应变干扰（图2D）。通常，触觉阵列输出随施加力变化的信号变化，并获得一条描述力与输出关系的曲线（称为力估计曲线）。基于特定且预先测试的力估计曲线推断力，是触觉传感器的传统开环测量方法。然而，在本研究中，除了对复合膜施加的力外，还以闭环方式检测应变干扰。在力校准之前，进行多次测试以测量在不同应变下的力估计曲线（见下一节）。图S11A所示的以下步骤提供了详细过程：（一）首先，如何确定应变干扰的存在？正负效应膜分别与应变检测模块连接，构成一种新颖的双输入模式。一旦出现增加的应变干扰，应变检测电路的输出电压立即显示同步上升沿，并进一步生成警告标志以提示应变干扰的存在；（二）其次，如何定量确定应变干扰的大小？如果存在应变干扰，根据图2C的相对电阻变化曲线可以推断出应变 ${\epsilon }_{\mathrm{x}}$。这一阶段确定了后续校准步骤中的适当校准系数；（三）第三，如何获得 ${\epsilon }_{\mathrm{x}}$ 下的力估计曲线？由于不可能列举所有可能应变干扰的力估计曲线，本研究提出了一种称为“局部域曲线插值”的方法来更新对应于 ${\epsilon }_{\mathrm{x}}$ 的力估计曲线。如图S11所示，识别已知条件中最接近 ${\epsilon }_{\mathrm{x}}$ 的两个应变值（命名为 ${\epsilon }_{\text{\_up }}$ 和 ${\epsilon }_{\text{\_bottom }}$）及其对应的力估计曲线（命名为curve_up和curve_bottom）。然后，在curve_up和curve_bottom之间按 ${\epsilon }_{\mathrm{x}}$ 与 ${\epsilon }_{\text{\_up }},{\epsilon }_{\text{lottom }}$ 之间的距离比例插值一条新曲线。这条插值曲线（命名为curve_x）代表对应于应变干扰 ${\epsilon }_{\mathrm{x}}$ 的力估计曲线；（四）第四，使用curve_x计算应变干扰 ${\epsilon }_{\mathrm{x}}$ 下的力。图S11B中的示例

—— 更多内容请到Doc2X翻译查看——

—— For more content, please visit Doc2X for translations ——