Although Computer Science (CS) is gaining popularity in early education settings in the US and worldwide, there is a lack of agreement about how to assess learning in young children, particularly preschoolers. The current study presents the design and pilot of a developmentally appropriate assessment tool, the Coding Stages Assessment-KIBO (CSA-KIBO), to evaluate preschool children’s coding skills with a robot kit (KIBO) designed for young children. Using a design-based research framework we developed evidence-based design criteria to inform our iteratively-tested assessment tool. In this paper, we address the following research question: How does the mode of administration of the CSA-KIBO robotics assessment impact performance among preschool students? We administered CSA-KIBO to 151 coding naive preschool students ages 3-5 years, from ethnically, socioeconomically, and linguistically diverse backgrounds. (author abstract)