Coaxial=s/pdif
s/pdif=Sony/Phillips digital interface.
It's just a way to transfer audio digitally. If that's confusing, think of it as transferring data and not audio. You're not limited to audio signals but typically, that's what you use it for.
The advantage is that other than jitter, you transfer the audio exactly as it is without any quality loss. So if you have a workstation keyboard or anything else, an audio recorder, etc... instead of going from the audio 1/4" outputs into the inputs of the audio device, you go digital to digital and have no quality loss. Converters are not perfect and going from the keyboard analog audio outputs takes 1 digital to analog conversion and then to get it into the DAW it takes an analog to digital converstion. To avoid errors and noise from the analog audio transfer and conversions, you go digital to digital and avoid that altogether.
The reason why your digital to digital transfer is not working is probably because you're not using the right type of cables. RCA cables that are used for home stereo systems and other consumer-level devices are not the same as the cables needed for digital audio transfters. You need to get a cable for digital audio or a video RCA cable might work also. The impendance is different from what I can remember so that's why any RCA cable won't work.
You might also have to choose a sync source, whether internal or external, when it comes to d/pdif, adat, tdif, etc... This is why there's devices such as the apogee Big Ben, which is used to "clock" a series of digital devices.