A very helpful and widely recommended paper (and accompanying book) in this domain is:
– particularly the editions that include the MATLAB/Simulink companion or the dedicated "Digital Communications Using MATLAB and Simulink" by Sklar and Ray. Digital Communication Systems Using Matlab And Simulink
