Pedestrian intention recognition is crucial to the safe and efficient working of autonomous vehicles in busy, urban scenarios. However, it is a much more challenging problem, as opposed to intention recognition of cars or cyclists, because of the absence of pedestrian rules of the road like lane driving or staying within road boundaries etc. The problem complexity is increased further in urban intersections with traffic lights and tightly packed sidewalks. The objective of this project is to accurately predict pedestrian motion while embedding the environmental context into account. The embedding of semantics would further aid in applying learnt prediction models to prediction in new intersections as well, that the model was not trained on. We presented this work in the 2017 NIPS Machine Learning for Intelligent Transportation workshop.