Automatic and accurate segmentation of medical images is an important task due to the direct impact of this procedure on both disease diagnosis and treatment. Segmentation of ultrasound (US) imaging is particularly challenging due to the presence of speckle noise. Recent deep learning approaches have demonstrated remarkable findings in image segmentation tasks, including segmentation of US images. However, many of the newly proposed structures are either task specific and suffer from poor generalization, or are computationally expensive. In this paper, we show that the receptive field plays a more significant role in the network's performance compared to the network's depth or the number of parameters. We further show that by controlling the size of the receptive field, a deep network can instead be replaced by a shallow network.