Computer Vision with Deep Learning presents an intriguing combination in the field of Medical Image Processing and Analysis. Recent advancement in neural networks has made high-quality research plausible. Many models are developed, targeting specific application, within the Deep Learning community. However, very little has been explored for mapping the saliency information using the gaze information.We present a Convolutional Neural Network (CNN), targeting a specific application, which foretells the saliency information on the image.Reading and interpreting a chest x-ray is a challenging task. We record the gaze data of radiologist who is deciphering the chest x-ray, using an eye tracker device, that acts as the ground truth. We perform various operations in the convolution neural network on every chest x-ray image in the dataset. Plotting the saliency information on the chest x-ray for easier interpretation is the eventual goal of this thesis project.