Break Out the Silverware: Semantic Understanding of Stored Household Items

“Bring me a plate.” For domestic service robots, such a simple command exposes a fundamental challenge: inferring where everyday objects are stored when they are not directly visible, such as inside drawers, cabinets, or closets. While recent advances in vision and manipulation have improved robotic perception, robots still lack the commonsense reasoning needed to predict plausible storage locations. We introduce the Stored Household Item Challenge, a benchmark task for evaluating a robot’s ability to infer likely storage locations given a household kitchen scene and a queried item. The benchmark comprises two datasets: (1) a real-world evaluation set of 100 item–image pairs with human-annotated ground truth from participants’ kitchens, and (2) a development set of 6,500 item–image pairs with polygon-level storage annotations on public kitchen images. To address this task, we propose NOAM (Non-visible Object Allocation Model), a hybrid vision–language pipeline that converts visual input into structured natural-language descriptions of spatial context and visible containers, and then prompts a large language model to infer the most likely hidden storage location. We evaluate NOAM against random baselines, vision–language pipelines, state-of-the-art multimodal models, and human performance. NOAM substantially improves prediction accuracy and approaches human-level results, demonstrating the value of structured vision–language reasoning for cognitively capable service robots in domestic environments.