Assembling Features Reading Features

为提升特征叠加工具如联合和交集等在处理大量数据时的性能和可扩展性,引入了自适应细分处理逻辑。当数据无法在物理内存中处理时,通过将原始范围细分为增量处理的子区域来保持在物理内存限制内,从而提高性能。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

In order to improve the performance and scalability of feature overlay tools such as Union and Intersect, operational logic called adaptive subdivision processing was added. The use of this logic is triggered when data cannot be processed within the available amount of physical memory. In order to stay in the bounds of physical memory, which greatly improves performance, processing is done incrementally on subdivisions of the original extent. Features which straddle the edges of these subdivisions (also called tiles) will be split at the edge of the tile and then reassembled into a single feature during in the last stage of processing. The vertices introduced at these tile edges will remain in the output features.

Why subdivide the data?

The overlay analysis tools perform best when processing can be done within your machine's physical memory (or RAM). This may not always be possible when working with datasets that contain either a large number of features or very complex features that contain hundreds of thousands or millions of vertices. Previously, when the physical memory was exhausted virtual memory was used, and when it was exhausted an internal paging system was used. Each successive mode of memory management is slower than the previous by an exponential factor.

How do I know when the process was tiled?

Review the messages returned by a tool during or after execution to determine if the input data was tiled. The third line will state "Processing Tiles..." if adaptive subdivision processing occurs, otherwise the input data was not subdivided and the fourth line will state "Cracking Features..."

Example of the messages from a process that was subdivided.

Executing (Identity_1): Identity c:\gp\fgdb.gdb\rivers c:\gp\fgdb.gdb\pf_watersheds c:\gp\fgdb.gdb\rivers_ws

Reading Features...

Cracking Features...

Assembling Features...

Executed (Identity_1) successfully.


Example of the messages from a process that was subdivided.

Executing (Identity_1): Identity c:\gp\fgdb.gdb\rivers c:\gp\fgdb.gdb\pf_watersheds c:\gp\fgdb.gdb\rivers_ws

Reading Features...

Processing Tiles...

Assembling Features...

Executed (Identity_1) successfully.


What do the tiles look like?

Every process starts with a single tile which spans the entire extent of the data. If the data in the single tile is too large to be processed in physical memory, it is subdivided into four equal tiles (using a quadtree approach). Processing then begins on a sub-tile, which is further sub-divided if the data in this second level of tiles is again too large. This continues until the data within each tile can be processed within physical memory. See the example below.

 

The footprint of all the input features.

 

The process begins with a tile that spans the entire extent of all datasets. For reference this is called tile level 1.

 

If the data is too large to process in memory, the level 1 tile is subdivided into four equal tiles. These 4 sub-tiles are called level 2 tiles.

 

Based on the size of data in each tile, some tiles are further subdivided, while others are not.

The tiles are output to the following shapefile c:\Documents and Settings\UserName\Local Settings\Temp\OverlayTile.shp upon completion of a process that required tiling.

Which tools use subdivisions

The following tools from the "Analysis Tools Toolbox" have subdivision logic when dealing with large data.

-Clip

-Erase

-Identity

-Intersect

-Union

-Split

-Symmetrical Difference

Process fails with an "Out of memory" error

The subdivision approach will not help process extremely large features. These are features with many millions of vertices. Splitting and reassembling extremely large features multiple times across tile boundaries is very costly in terms of memory, and may cause "Out of memory" errors if the feature is too large. It is recommended that these features be broken up into smaller features. Road casing for an entire city or a polygon representing a river estuary are examples of very large features with many vertices.

The "Out of memory" error could also happen if a second application is run while a tool is processing. This second application could adjust the available amount of physical memory and render the boundary calculation of the currently processing tile as incorrect, thereby causing the tool process to demand more physical memory than is possible. It is recommended that no other operations be performed on a machine while overlay processing large datasets.

What data format is recommended when working with large data?

The file size limit for the Windows operating system is 2 GB. If the result of a process is going to be a large feature class, writing that output to a shapefile (which is made up of several files) or personal geodatabase (consisting of a single .mdb file) could exceed this 2 GB file size limit. Enterprise and file geodatabases do not have this limitation so they are recommended as the output workspace when using very large datasets.

See Also

·  Temporary files created by Geoprocessing Tools


转载自:

http://resources.esri.com/help/9.3/ArcGISDesktop/com/Gp_ToolRef/analysis_toolbox/tiled_processing_of_large_datasets.htm

import logging import time from functools import cached_property from typing import Any from lerobot.cameras.utils import make_cameras_from_configs from lerobot.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError from lerobot.motors import Motor, MotorCalibration, MotorNormMode from lerobot.motors.dynamixel import ( DynamixelMotorsBus, OperatingMode, ) from ..robot import Robot from ..utils import ensure_safe_goal_position from .config_koch_follower import KochFollowerConfig logger = logging.getLogger(__name__) class KochFollower(Robot): """ - [Koch v1.0](https://github.com/AlexanderKoch-Koch/low_cost_robot), with and without the wrist-to-elbow expansion, developed by Alexander Koch from [Tau Robotics](https://tau-robotics.com) - [Koch v1.1](https://github.com/jess-moss/koch-v1-1) developed by Jess Moss """ config_class = KochFollowerConfig name = "koch_follower" def __init__(self, config: KochFollowerConfig): super().__init__(config) self.config = config norm_mode_body = MotorNormMode.DEGREES if config.use_degrees else MotorNormMode.RANGE_M100_100 self.bus = DynamixelMotorsBus( port=self.config.port, motors={ "shoulder_pan": Motor(1, "xl430-w250", norm_mode_body), "shoulder_lift": Motor(2, "xl430-w250", norm_mode_body), "elbow_flex": Motor(3, "xl330-m288", norm_mode_body), "wrist_flex": Motor(4, "xl330-m288", norm_mode_body), "wrist_roll": Motor(5, "xl330-m288", norm_mode_body), "gripper": Motor(6, "xl330-m288", MotorNormMode.RANGE_0_100), }, calibration=self.calibration, ) self.cameras = make_cameras_from_configs(config.cameras) @property def _motors_ft(self) -> dict[str, type]: return {f"{motor}.pos": float for motor in self.bus.motors} @property def _cameras_ft(self) -> dict[str, tuple]: return { cam: (self.config.cameras[cam].height, self.config.cameras[cam].width, 3) for cam in self.cameras } @cached_property def observation_features(self) -> dict[str, type | tuple]: return {**self._motors_ft, **self._cameras_ft} @cached_property def action_features(self) -> dict[str, type]: return self._motors_ft @property def is_connected(self) -> bool: return self.bus.is_connected and all(cam.is_connected for cam in self.cameras.values()) def connect(self, calibrate: bool = True) -> None: """ We assume that at connection time, arm is in a rest position, and torque can be safely disabled to run calibration. """ if self.is_connected: raise DeviceAlreadyConnectedError(f"{self} already connected") self.bus.connect() if not self.is_calibrated and calibrate: self.calibrate() for cam in self.cameras.values(): cam.connect() self.configure() logger.info(f"{self} connected.") @property def is_calibrated(self) -> bool: return self.bus.is_calibrated def calibrate(self) -> None: logger.info(f"\nRunning calibration of {self}") self.bus.disable_torque() for motor in self.bus.motors: self.bus.write("Operating_Mode", motor, OperatingMode.EXTENDED_POSITION.value) input(f"Move {self} to the middle of its range of motion and press ENTER....") homing_offsets = self.bus.set_half_turn_homings() full_turn_motors = ["shoulder_pan", "wrist_roll"] unknown_range_motors = [motor for motor in self.bus.motors if motor not in full_turn_motors] print( f"Move all joints except {full_turn_motors} sequentially through their entire " "ranges of motion.\nRecording positions. Press ENTER to stop..." ) range_mins, range_maxes = self.bus.record_ranges_of_motion(unknown_range_motors) for motor in full_turn_motors: range_mins[motor] = 0 range_maxes[motor] = 4095 self.calibration = {} for motor, m in self.bus.motors.items(): self.calibration[motor] = MotorCalibration( id=m.id, drive_mode=0, homing_offset=homing_offsets[motor], range_min=range_mins[motor], range_max=range_maxes[motor], ) self.bus.write_calibration(self.calibration) self._save_calibration() logger.info(f"Calibration saved to {self.calibration_fpath}") def configure(self) -> None: with self.bus.torque_disabled(): self.bus.configure_motors() # Use 'extended position mode' for all motors except gripper, because in joint mode the servos # can't rotate more than 360 degrees (from 0 to 4095) And some mistake can happen while assembling # the arm, you could end up with a servo with a position 0 or 4095 at a crucial point for motor in self.bus.motors: if motor != "gripper": self.bus.write("Operating_Mode", motor, OperatingMode.EXTENDED_POSITION.value) # Use 'position control current based' for gripper to be limited by the limit of the current. For # the follower gripper, it means it can grasp an object without forcing too much even tho, its # goal position is a complete grasp (both gripper fingers are ordered to join and reach a touch). # For the leader gripper, it means we can use it as a physical trigger, since we can force with # our finger to make it move, and it will move back to its original target position when we # release the force. self.bus.write("Operating_Mode", "gripper", OperatingMode.CURRENT_POSITION.value) # Set better PID values to close the gap between recorded states and actions # TODO(rcadene): Implement an automatic procedure to set optimal PID values for each motor self.bus.write("Position_P_Gain", "elbow_flex", 1500) self.bus.write("Position_I_Gain", "elbow_flex", 0) self.bus.write("Position_D_Gain", "elbow_flex", 600) def setup_motors(self) -> None: for motor in reversed(self.bus.motors): input(f"Connect the controller board to the '{motor}' motor only and press enter.") self.bus.setup_motor(motor) print(f"'{motor}' motor id set to {self.bus.motors[motor].id}") def get_observation(self) -> dict[str, Any]: if not self.is_connected: raise DeviceNotConnectedError(f"{self} is not connected.") # Read arm position start = time.perf_counter() obs_dict = self.bus.sync_read("Present_Position") obs_dict = {f"{motor}.pos": val for motor, val in obs_dict.items()} dt_ms = (time.perf_counter() - start) * 1e3 logger.debug(f"{self} read state: {dt_ms:.1f}ms") # Capture images from cameras for cam_key, cam in self.cameras.items(): start = time.perf_counter() obs_dict[cam_key] = cam.async_read() dt_ms = (time.perf_counter() - start) * 1e3 logger.debug(f"{self} read {cam_key}: {dt_ms:.1f}ms") return obs_dict def send_action(self, action: dict[str, float]) -> dict[str, float]: """Command arm to move to a target joint configuration. The relative action magnitude may be clipped depending on the configuration parameter `max_relative_target`. In this case, the action sent differs from original action. Thus, this function always returns the action actually sent. Args: action (dict[str, float]): The goal positions for the motors. Returns: dict[str, float]: The action sent to the motors, potentially clipped. """ if not self.is_connected: raise DeviceNotConnectedError(f"{self} is not connected.") goal_pos = {key.removesuffix(".pos"): val for key, val in action.items() if key.endswith(".pos")} # Cap goal position when too far away from present position. # /!\ Slower fps expected due to reading from the follower. if self.config.max_relative_target is not None: present_pos = self.bus.sync_read("Present_Position") goal_present_pos = {key: (g_pos, present_pos[key]) for key, g_pos in goal_pos.items()} goal_pos = ensure_safe_goal_position(goal_present_pos, self.config.max_relative_target) # Send goal position to the arm self.bus.sync_write("Goal_Position", goal_pos) return {f"{motor}.pos": val for motor, val in goal_pos.items()} def disconnect(self): if not self.is_connected: raise DeviceNotConnectedError(f"{self} is not connected.") self.bus.disconnect(self.config.disable_torque_on_disconnect) for cam in self.cameras.values(): cam.disconnect() logger.info(f"{self} disconnected.")
07-26
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值