Artificially Intelligent Food Assistant for the Visually Impaired
Keywords:artificial intelligence, visually impaired, food assistant, computer vision, optical character recognition
Vision disability is a prevalent condition that affects the lives of many adults and children. Previous research has established that it is harder for the visually impaired to evaluate the nutritional value of food. Therefore, by applying concepts used in agriculture and optical character recognition, we engineered a system that can make these perceptual evaluations on a variety of fresh and packaged foods and linguistically relay that info to a visually impaired user. We utilized some of the most memory-cheap and accurate object detection models to evaluate and then detect lesions in peaches, apples, tomatoes, and strawberries. Table 3 shows that our models performed with high accuracy as EffecientDet d0, SSD Lite Mobilenet, and Faster RCNN MobileNet all had higher than 40% mAP and 50% mAR. Figure 2 portrays how our interface was able to detect, determine, and relay surface spoilage percentage(s) to a user. Figure 3 shows that our OCR integration successfully was able to gather nutritional data on packaged goods and relay nutritional information to a user. Our system provides the brain for future applications that plan to deploy our code to devices like smart glasses or other hardware. We have made our source code available on GitHub through this link: https://github.com/SarthakJaingit/Artificially-Intelligent-Food-Assistant-for-the-Visually-Impaired. Our repository provides instructions about running our system through the command line and also a notebook demo that a less technical person can run to see how one of our models performs on a computer webcam with no video optimization.
References or Bibliography
Blindness Statistics. The American Foundation for the Blind. Accessed October 14, 2021. https://www.afb.org/research-and-initiatives/statistics
Fast Facts of Common Eye Disorders | CDC. Published June 9, 2020. Accessed October 14, 2021. https://www.cdc.gov/visionhealth/basics/ced/fastfacts.htm
Bilyk MC, Sontrop JM, Chapman GE, Barr SI, Mamer L. Food experiences and eating patterns of visually impaired and blind people. Can J Diet Pract Res Publ Dietit Can Rev Can Prat Rech En Diet Une Publ Diet Can. 2009;70(1):13-18. doi:10.3148/70.1.2009.13
Jones N, Bartlett H. The impact of visual impairment on nutritional status: A systematic review. Br J Vis Impair. 2018;36(1):17-30. doi:10.1177/0264619617730860
Kostyra E, Żakowska-Biemans S, Śniegocka K, Piotrowska A. Food shopping, sensory determinants of food choice and meal preparation by visually impaired people. Obstacles and expectations in daily food experiences. Appetite. 2017;113:14-22. doi:10.1016/j.appet.2017.02.008
Zhu L, Spachos P, Pensini E, Plataniotis KN. Deep learning and machine vision for food processing: A survey. Curr Res Food Sci. 2021;4:233-249. doi:10.1016/j.crfs.2021.03.009
Koyama K, Tanaka M, Cho B-H, Yoshikawa Y, Koseki S. Predicting sensory evaluation of spinach freshness using machine learning model and digital images. PLOS ONE. 2021;16(3):e0248769. doi:10.1371/journal.pone.0248769
Du C-J, Sun D-W. Learning techniques used in computer vision for food quality evaluation: a review. J Food Eng. 2006;72(1):39-55. doi:10.1016/j.jfoodeng.2004.11.017
Karakaya D, Ulucan O, Turkan M. A Comparative Analysis on Fruit Freshness Classification. In: 2019 Innovations in Intelligent Systems and Applications Conference (ASYU). ; 2019:1-4. doi:10.1109/ASYU48272.2019.8946385
Horiguchi S, Amano S, Ogawa M, Aizawa K. Personalized Classifier for Food Image Recognition. ArXiv180404600 Cs. Published online April 8, 2018. Accessed October 14, 2021. http://arxiv.org/abs/1804.04600
Memon J, Sami M, Khan RA, Uddin M. Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR). IEEE Access. 2020;8:142642-142668. doi:10.1109/ACCESS.2020.3012542
Ahmed M, Abidi A. REVIEW ON OPTICAL CHARACTER RECOGNITION. In: ; 2019.
Zamir MF, Khan KB, Khan SA, Rehman E. Smart Reader for Visually Impaired People Based on Optical Character Recognition. In: Bajwa IS, Sibalija T, Jawawi DNA, eds. Intelligent Technologies and Applications. Communications in Computer and Information Science. Springer; 2020:79-89. doi:10.1007/978-981-15-5232-8_8
Ilya F. ImageNet 1000 (Mini).; 2020. https://www.kaggle.com/ifigotin/imagenetmini-1000/metadata
Wightman R. Efficientdet-Pytorch.; 2021. https://github.com/rwightman/efficientdet-pytorch.git
Cloud Vision documentation | Cloud Vision API. Google Cloud. Accessed October 14, 2021. https://cloud.google.com/vision/docs
CalorieNinjas - Easy, Free Nutrition Facts Search. Accessed October 14, 2021. https://calorieninjas.com/
Shorten C, Khoshgoftaar TM. A survey on Image Data Augmentation for Deep Learning. J Big Data. 2019;6(1):60. doi:10.1186/s40537-019-0197-0
Buslaev A, Parinov A, Khvedchenya E, Iglovikov VI, Kalinin AA. Albumentations: fast and flexible image augmentations. Information. 2020;11(2):125. doi:10.3390/info11020125
Cocodataset/Cocoapi. cocodataset; 2021. Accessed October 15, 2021. https://github.com/cocodataset/cocoapi
apple-5265125_1280.jpg (1280×853). Accessed October 15, 2021. https://cdn.pixabay.com/photo/2020/06/06/03/52/apple-5265125_1280.jpg
4857682389_7b13e44deb_b.jpg (1024×768). Accessed October 15, 2021. https://live.staticflickr.com/4136/4857682389_7b13e44deb_b.jpg
front_en.10.full.jpg (1125×2000). Accessed October 15, 2021. https://world.openfoodfacts.org/images/products/930/060/145/1272/front_en.10.full.jpg
How to Cite
Copyright (c) 2021 Sarthak Jain; Evan Brociner
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright holder(s) granted JSR a perpetual, non-exclusive license to distriute & display this article.