Precision Measurement for Manufacturing

Any measurement is a claim that a part is a certain size. Before you stake a good part or a bad one on that claim, you need to know how much the number can be trusted. This is the thinking behind every measuring tool: what the print asks for, which tool can answer it, and how close the answer really is.

What You Will Be Able to Do

• Work out the limits of a dimension from its tolerance.
• Explain resolution, accuracy, precision, and uncertainty in plain words.
• Choose a tool that fits a tolerance using the 10-to-1 rule.
• Control the temperature and cleanliness problems that spoil a reading.
• Read a surface finish callout and pick a way to check it.

Units and Numbers

Most US shops work in decimal inches. Automotive, medical, and many global customers work in millimeters. Check the title block or notes before you measure, because a metric dimension read in inches means a scrapped part.

• 1 inch = 25.4 mm exactly.
• 0.001 inch is one "thou" or thousandth, or 0.0254 mm.
• 0.0001 inch is one "tenth," or 0.00254 mm.
• 0.001 mm is one micron, also written as 1 micrometer.

Say numbers the way the shop does. 0.250 is "two-fifty." 0.0005 is "five tenths" or "half a thou."

Reading a Tolerance

A tolerance is how far a size may vary and still be good. The limits are the largest and smallest allowed sizes. Always work out the limits first.

Worked example: a groove is 0.750 +/- 0.004 and you read 0.753. It is in tolerance, but it has used 0.003 of the 0.004 on the high side. It is good, and it is near the upper limit. Report that to whoever runs the job, because the next one may not be good.

Four Words People Mix Up

A tool can be precise and still not accurate. Picture ten readings that all land within 0.0002 of each other but all sit 0.002 above the true size. The tool repeats well but it is wrong, and calibration exists to catch this.

Uncertainty adds up everything that can move a result: the tool, the master it was set to, the temperature, and the person. It matters most near the limits. If a reading is 0.5040 on a part with an upper limit of 0.5040 and your uncertainty is 0.0005, you cannot honestly say the part passes or fails. You need a better measurement.

The 10-to-1 Rule

The common shop rule is that your measuring system should be about ten times finer than the tolerance you are checking. Some shops write this as the tool's resolution or uncertainty using no more than 10 percent of the tolerance band.

Worked example: a bore is 1.0000 +/- 0.0005, so the total band is 0.0010. Ten-to-one calls for a tool good to 0.0001. A caliper is out. A standard micrometer that reads only to 0.001 is out as well. You need a tool that reads and holds tenths, such as a bore gauge set to a master or a tenths micrometer for an outside feature.

Some tight tolerances make 10 to 1 impossible at a fair cost. When that happens, a quality engineer decides what ratio is acceptable, and may back that up with a gauge study.

Choosing a Tool

These are general starting points. Your control plan or inspection sheet comes first when it names a tool.

A variable tool gives you a number. An attribute tool gives you pass or fail. Use numbers when you need to see drift. Use attribute gauges when speed matters and the plan allows it.

Temperature

Dimensions on a print apply at the standard reference temperature of 68 F (20 C). A part warmer than that reads large. Steel grows about 0.000006 inch per inch per degree Fahrenheit. Aluminum grows about twice as much.

Worked example: an aluminum plate is 12 inches long and sits at 78 F, which is 10 degrees over standard. Using about 0.000013 inch per inch per degree, it is 12 x 10 x 0.000013 = 0.0016 inch longer than at 68 F. On a tolerance of +/- 0.002 inch, that is most of the band.

To control it:

1. Let the part reach room temperature before a close measurement. Warm parts read larger than their true size.
2. Keep the part and the tool together in the same room for a while, so both are at the same temperature. Steel parts measured with steel tools at the same temperature nearly cancel out.
3. Keep close measuring away from sun, heaters, and coolant.
4. Handle parts and tools as little as you can. Your hands warm them.

Cleanliness and Handling

A chip between a tool face and a part adds its full thickness to your reading. Coolant film and fine grit do the same. Before every close measurement:

1. Deburr the feature. A burr pushes the tool off the real surface.
2. Wipe the part and the measuring faces with a clean, lint-free cloth.
3. Use a light, steady force. Too much force bends the tool or the part. Thin parts can deflect and read small.
4. Measure where the part functions. Bores are rarely perfectly round. If a pin goes in one way, measure in that same orientation.
5. Take more than one reading on anything critical, in different spots.
6. Write down the actual value right away. Do not trust memory and do not round to make it fit.

Surface Finish

A surface finish symbol on a print looks like a check mark with a number. The number is usually Ra, the roughness average, which is the average height of the surface's peaks and valleys measured from its mean line. In US prints it is often in microinches. For example, 63 means no rougher than 63 microinches Ra. On metric prints it is in micrometers.

• Comparator: a set of sample surfaces you compare by sight and fingernail. It is fast and good enough for many loose callouts.
• Profilometer: a stylus drags across the surface and reports an Ra number. Use it for tight or disputed finishes, and measure across the tool marks, not along them.

Common Mistakes

Check Your Understanding

1. A shaft is 0.8750 +0.0000 / -0.0010. You read 0.8742. Is it good?

Yes. The limits are 0.8740 to 0.8750, so it is in tolerance but near the lower limit.

2. Your caliper reads to 0.0005. Can you use it on a +/- 0.001 dimension?

No. The display step is not its accuracy, and the caliper's accuracy is about +/- 0.001. The 10-to-1 rule calls for a tool good to about 0.0002 for this 0.002 band.

3. Ten readings agree within 0.0001, but a gauge block check shows the tool reads 0.0015 high. Is the tool precise, accurate, both, or neither?

Precise but not accurate. Take it out of service for calibration.

4. A steel part comes out of a warm process and measures 0.002 inch over its limit. Do you scrap it?

Not yet. Let it reach room temperature and measure again. Then decide.

Key Takeaways

• Work out the limits before you pick up a tool.
• Resolution, accuracy, precision, and uncertainty are four different things.
• Use the 10-to-1 rule to pick a tool, and ask a quality engineer when you cannot meet it.
• Standard temperature is 68 F (20 C), and warm parts read large.
• Clean and deburr first, measure where the part functions, and record the real number.