Talkback is used in all sound engineering situations. Reverse talkback is common only in broadcasting, although it may have applications in theater and live sound.
The normal flow of signal in the recording studio is from the instruments to the microphones to the mixing console to the recording device and monitoring system. So the 'back' in talkback means a signal that is flowing in the opposite direction to that.
In recording this would be from the engineer or producer to the musicians or musical director in the studio. So if the producer thinks a take didn't go too well, he can press the talkback button and communicate directly with the musicians through their headphones or a studio loudspeaker. The musicians can often talk to the producer through the microphones set up to record their instruments – this type of communication is normally not given a special name, even if an additional mic is set up dedicated to this purpose.
In television however, there is much more of a requirement for communication, and in fact a sound engineer working in television can spend much of his or her time working on communications rather than the audio to be broadcast or recorded.
A particular instance is the camera operator. The camera operator needs to work to the director's commands. So there is a 'production talkback' system that handles comments of an artistic nature. It would be quite rare for the camera operator to need to communicate back to the director.
The vision mixer however also needs to talk to the camera operators. The vision mixer is the director's technical 'right hand man'. The director has the artistic insight, the vision mixer makes it happen. So the vision mixer has a button which they can press to add their voice to production talkback.
But then the camera operator might need to talk to the vision mixer. The director will not want to hear this technical communication, so it is channeled through a different route.
Where the camera operator speaks to the vision mixer, or anyone downstream of the main signal flow, this is known as 'reverse talkback'. So the phrase reverse talkback may sound almost like a double negative, however it is an important function that helps get the job done.