The lack of automated cars above SAE level 3 raises challenges for conducting User Experience Design (UXD) and behaviour research for automated driving. User-centred methods are critical to ensuring a human-friendly progress of vehicle automation. This work introduces the Immersive Video-based Automated Driving (IVAD) Simulator. It uses carefully recorded 180/360° videos that are played back in a driving simulator. This provides immersive driving experiences in visually realistic and familiar environments. This paper reports learnings from an iterative development of IVAD, and findings of two user studies: One simulator study (N=15) focused on the immersive experience; and one VR study (N=16) focused on rapid prototyping and the evaluation of Augmented Reality (AR) concepts. Overall, we found the method to be a useful, versatile and low budget UXD tool with a high level of immersion that is uniquely aided by the familiarity of the environment. IVAD’s limitations and future improvements are discussed in relation to research applications within AutoUI.