Tolerances, Units, and Tessellation Settings: Control Geometric Precision at Export

Precision is not one number. A model’s kernel tolerance, a mesh’s surface deviation, a coordinate unit, a numeric representation, a point spacing, and a measurement uncertainty answer different questions. An exchange is controllable only when those questions are stated separately.

Define the requirement first

Start with four inputs:

• the smallest feature or distance that matters;
• the maximum deviation the receiving task can accept;
• the operation that will decide acceptance;
• the units and scale context understood by the receiver.

“Use a high-quality export” is not an acceptance requirement. A CAD boolean, a printed part, an inspection comparison, and a render may need different representations and checks. Manufacturing and measurement limits must come from the applicable process owner or specification; generic geometry advice cannot supply them.

Keep precision layers separate

Precision layer	What it describes	What to record
B-rep or kernel tolerance	How a geometry kernel treats coincidence and local gaps	Kernel/tool, units, local or global values, and the operation for which the value matters
Tessellation deviation	How far a faceted surface may depart from an exact surface	Chordal or other stated error measure, settings, and whether the exporter actually reports an bound
Mesh density	Sampling density or triangle size	Density rules, feature protection, simplification, and task-specific acceptance
Coordinate unit and scale	The numeric interpretation of positions	Source unit, file/profile unit, receiver setting, and a known-size check
Numeric precision	How coordinates and calculations are represented	Format and application limits where material; do not confuse digits with geometric accuracy
Scan point spacing and uncertainty	Acquisition sampling and measurement confidence	Scanner context, resolution/distance, registration, reported uncertainty, and reconstruction effects

Open CASCADE’s current ShapeAnalysis_ShapeTolerance reference demonstrates why a model tolerance should not be collapsed into one value. It can compute minimum, maximum, and average stored tolerances and find sub-shapes within a range or over a threshold, with scope for vertices, edges, faces, shells, or all shapes. That is an OCCT inspection capability, not a universal tolerance definition or a correct threshold for another kernel.

OCCT’s current BRepCheck_Analyzer can also report an invalid tolerance value as one of its shape-status conditions. That is evidence of a defect under OCCT’s implemented checks, not a selection of the tolerance that should replace it. The correct value still depends on the model, receiving operation, units, and acceptance purpose.

Units can change the shape you receive

Record units at both ends of the transfer. The 3MF Core 1.4.0 specification defines a model coordinate-space unit, defaults it to millimeter, and lists micron, millimeter, centimeter, inch, foot, and meter as valid values. That tells you how a 3MF model declares its coordinate space; it does not guarantee that an unrelated consumer interprets every package correctly.

glTF 2.0.1 defines POSITION as unitless XYZ vertex positions. A glTF handoff therefore needs an explicit scale convention in the exchange record or surrounding delivery context. A visually plausible asset can still be the wrong physical size.

SOLIDWORKS 2025 documents a receiver-side choice between using the file-specified unit and the document-template unit for neutral-file imports. The selected option must be part of the test record. A unit mismatch can change bounding-box dimensions by a factor while leaving the model recognizable.

Inventor 2026 illustrates how a repair tolerance should be treated as a controlled change boundary. Heal Errors uses an Allowable Change value and can preview the extent of repair. Its Refit Face workflow defines a Maximum Tolerance as the maximum deviation allowed when generating a replacement face, recommends starting small and increasing slowly, and allows a repair attempt to be undone. These controls are product-specific; none supplies a universal export or acceptance tolerance.

Geospatial point-cloud deliveries add another bounded example: the USGS 2025 Rev. A lidar specification prefers consistent horizontal and vertical units, says mixed units are not recommended and must be noted in spatial metadata, and expects source identity and reported accuracy controls. Those are USGS lidar-delivery requirements, not mechanical-CAD tolerances.

Do not rely on defaults

Defaults vary by producer, consumer, representation, version, and downstream purpose. Before export, capture:

1. source application and version;
2. model units and coordinate context;
3. exact, tessellated, mesh, or point representation;
4. format/profile/version and extension set;
5. tessellation, simplification, or tolerance settings;
6. receiver import settings and expected interpretation;
7. the acceptance measure and its allowed tolerance.

If the software does not expose an error bound, say so. Do not convert a quality preset or an “accurate” label into a numeric guarantee.

For STEP exchanges, NIST’s current STEP File Analyzer User Guide (Update 7) groups validation properties into geometric, PMI, assembly, annotation, attribute, and tessellated reports. It describes comparing properties written by the originating system and recomputed by the receiving system within an agreed tolerance. Area, volume, and centroid are examples, not a universal property set, and the result is bounded to the properties actually written, recomputed, and compared.

An export-to-check workflow

1. Name the purpose. Identify whether the result is for editing, display, additive work, fabrication, measurement, or another task.
2. Choose the smallest relevant feature. Use it to set the representation and required precision conversation.
3. Set and record units. Include a known dimension or reference object that can be checked after import.
4. Set tessellation or precision controls. Preserve the option names, values, and version; include a stated deviation if the tool provides one.
5. Import with recorded receiver settings. Do not silently accept a template unit or default profile.
6. Compare the result. Check scale, bounding dimensions, selected geometry properties, topology, and the downstream operation.
7. Record exclusions. A geometric match does not establish structure, feature history, PMI, materials, or other data layers.

For a mesh, include attribute and resolution checks from mesh fidelity. For source-versus-received geometric comparisons, see validate a geometry handoff.

What cannot be generalized

There is no universal tessellation tolerance, triangle density, scan resolution, or CAD-kernel tolerance that guarantees an acceptable result. A geospatial accuracy class does not become a mechanical-part tolerance, and a matching volume does not prove local surface or topology health. Use the requirement of the actual destination task and preserve any unresolved applicability as an explicit limitation.