An understanding of what distortion is is elemental to audio. If you don't know this, it's like you are running a checking account, a credit card and a mortgage without knowing what cash is.
The Ferrograph RTS2 test set is an ancient but venerable piece of equipment that allows key measurements to be performed by traditional methods, quickly and easily. There are other similar units that will do the job just as well, perhaps in combination rather than all-in-one, and more modern equipment will take similar measurements automatically.
You don't need a Ferrograph RTS2 to understand the method, which is general to all test equipment.
Distortion is the bending of the input waveform, causing additional frequencies to be generated. A distortion meter measures the percentage of additionals compared to the overall level of the distorted signal. Plainly, to make a meaningful measurement, you have to apply a clean, undistorted sine wave (the simplest audio signal consisting of only one frequency) to the equipment under test.
So, use the signal generator to generate a sine wave of 1000 Hz. (Other distortion meters may be able to measure distortion over a range of frequencies but the RTS2 only works at 1 kHz). Apply it to the device to be tested at a known level. For digital equipment this could be 0 dBFS (full level), or -10 dBFS to represent a typical signal. For professional analog equipment the level could be 0 dBu (0.775 V); for semi-professional and domestic equipment -10 dBV (316 mV).
If you want to test a 'boutique' mic preamp to measure how much 'warmth' it produces, then switch off the phantom power and supply it with an input of 10 millivolts for starters.
If you want to see significant distortion, perform the test on an analog recorder or cassette deck. You have to record the test signal, then play it back.
Apply the output signal from the device under test to the input of the test set, switched to measure distortion.
At this point it is necessary to 'tell' the test set or distortion meter what the level represents 100%. This is the full level of the signal containing the distortion. On the Ferrograph RTS2 there is a control that is turned so that the meter reads full scale. Readings will then be with respect to that level being 100%.
Now comes the clever part. The signal, even with distortion, still consists mostly of 1 kHz. This must be canceled out. The RTS2, and any distortion meter working by the same method, will have two controls. One governs the frequency of the canceling signal to make it exactly the same as the signal under test. There will be slight inaccuracies in frequency, so this tuning stage is essential.
So carefully tune the canceling signal and watch the reading on the meter of the test set. The needle will go down and down, then start going up again. Center on the position where it is lowest. At this point, you will probably need to make the meter more sensitive so that you can still see the reading. Any meter will have a sensitivity switch to do this.
Although you have tuned the canceling frequency as accurately as you can, it also needs be 180 degrees out of phase so that it cancels with the signal under test. So adjust this control until you get the lowest reading you can. Make the meter more sensitive so that you can still see the reading.
Now you can return to the frequency control and further fine tune it, then to the phase again. Keep swapping back and forth between the two and refine the cancellation, making the meter more sensitive when necessary.
Eventually you will come to a point where you can't get the reading any lower. Congratulations! You have canceled out the 1 kHz component of the signal under test. All that remains is the distortion.
You can now read off the distortion directly as a percentage. From an analog recorder, reel-to-reel or cassette, you can expect something around one percent distortion. From a tube mic preamp, the percentage could be from 0.1% up to something over 1%, depending on the input level and the degree of 'drive' of the preamp.
By the way, if you apply for a job as a sound engineer and you say in your interview that you know how to measure distortion, you've got the job! They will be impressed.