Artificially Intelligent Food Assistant for the Visually Impaired


  • Sarthak Jain Los Gatos High School
  • Evan Brociner Mentor, Los Gatos High School



artificial intelligence, visually impaired, food assistant, computer vision, optical character recognition


Vision disability is a prevalent condition that affects the lives of many adults and children. Previous research has established that it is harder for the visually impaired to evaluate the nutritional value of food. Therefore, by applying concepts used in agriculture and optical character recognition, we engineered a system that can make these perceptual evaluations on a variety of fresh and packaged foods and linguistically relay that info to a visually impaired user. We utilized some of the most memory-cheap and accurate object detection models to evaluate and then detect lesions in peaches, apples, tomatoes, and strawberries. Table 3 shows that our models performed with high accuracy as EffecientDet d0, SSD Lite Mobilenet, and Faster RCNN MobileNet all had higher than 40% mAP and 50% mAR. Figure 2 portrays how our interface was able to detect, determine, and relay surface spoilage percentage(s) to a user. Figure 3 shows that our OCR integration successfully was able to gather nutritional data on packaged goods and relay nutritional information to a user. Our system provides the brain for future applications that plan to deploy our code to devices like smart glasses or other hardware. We have made our source code available on GitHub through this link: Our repository provides instructions about running our system through the command line and also a notebook demo that a less technical person can run to see how one of our models performs on a computer webcam with no video optimization.   


Download data is not yet available.

References or Bibliography

Blindness Statistics. The American Foundation for the Blind. Accessed October 14, 2021.

Fast Facts of Common Eye Disorders | CDC. Published June 9, 2020. Accessed October 14, 2021.

Bilyk MC, Sontrop JM, Chapman GE, Barr SI, Mamer L. Food experiences and eating patterns of visually impaired and blind people. Can J Diet Pract Res Publ Dietit Can Rev Can Prat Rech En Diet Une Publ Diet Can. 2009;70(1):13-18. doi:10.3148/70.1.2009.13

Jones N, Bartlett H. The impact of visual impairment on nutritional status: A systematic review. Br J Vis Impair. 2018;36(1):17-30. doi:10.1177/0264619617730860

Kostyra E, Żakowska-Biemans S, Śniegocka K, Piotrowska A. Food shopping, sensory determinants of food choice and meal preparation by visually impaired people. Obstacles and expectations in daily food experiences. Appetite. 2017;113:14-22. doi:10.1016/j.appet.2017.02.008

Zhu L, Spachos P, Pensini E, Plataniotis KN. Deep learning and machine vision for food processing: A survey. Curr Res Food Sci. 2021;4:233-249. doi:10.1016/j.crfs.2021.03.009

Koyama K, Tanaka M, Cho B-H, Yoshikawa Y, Koseki S. Predicting sensory evaluation of spinach freshness using machine learning model and digital images. PLOS ONE. 2021;16(3):e0248769. doi:10.1371/journal.pone.0248769

Du C-J, Sun D-W. Learning techniques used in computer vision for food quality evaluation: a review. J Food Eng. 2006;72(1):39-55. doi:10.1016/j.jfoodeng.2004.11.017

Karakaya D, Ulucan O, Turkan M. A Comparative Analysis on Fruit Freshness Classification. In: 2019 Innovations in Intelligent Systems and Applications Conference (ASYU). ; 2019:1-4. doi:10.1109/ASYU48272.2019.8946385

Horiguchi S, Amano S, Ogawa M, Aizawa K. Personalized Classifier for Food Image Recognition. ArXiv180404600 Cs. Published online April 8, 2018. Accessed October 14, 2021.

Memon J, Sami M, Khan RA, Uddin M. Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR). IEEE Access. 2020;8:142642-142668. doi:10.1109/ACCESS.2020.3012542


Zamir MF, Khan KB, Khan SA, Rehman E. Smart Reader for Visually Impaired People Based on Optical Character Recognition. In: Bajwa IS, Sibalija T, Jawawi DNA, eds. Intelligent Technologies and Applications. Communications in Computer and Information Science. Springer; 2020:79-89. doi:10.1007/978-981-15-5232-8_8

Ilya F. ImageNet 1000 (Mini).; 2020.

Wightman R. Efficientdet-Pytorch.; 2021.

Cloud Vision documentation | Cloud Vision API. Google Cloud. Accessed October 14, 2021.

CalorieNinjas - Easy, Free Nutrition Facts Search. Accessed October 14, 2021.

Shorten C, Khoshgoftaar TM. A survey on Image Data Augmentation for Deep Learning. J Big Data. 2019;6(1):60. doi:10.1186/s40537-019-0197-0

Buslaev A, Parinov A, Khvedchenya E, Iglovikov VI, Kalinin AA. Albumentations: fast and flexible image augmentations. Information. 2020;11(2):125. doi:10.3390/info11020125

Cocodataset/Cocoapi. cocodataset; 2021. Accessed October 15, 2021.

apple-5265125_1280.jpg (1280×853). Accessed October 15, 2021.

4857682389_7b13e44deb_b.jpg (1024×768). Accessed October 15, 2021.

front_en.10.full.jpg (1125×2000). Accessed October 15, 2021.



How to Cite

Jain, S., & Brociner, E. . (2021). Artificially Intelligent Food Assistant for the Visually Impaired. Journal of Student Research, 10(4).



HS Research Articles