Growth information in smart farms is essential to optimize the farm environment for crops and livestock. In this study, we verified the performance of a system for the collection of paprika growth information using semantic segmentation. Although most current deep learning methods recognize objects along the outline, this study investigated the recognition of objects hidden by obstacles during the growth investigation process. In addition, the ability of the artificial intelligence method to recognize only the stems necessary for growth investigation among several stems of paprika was investigated. Furthermore, the application of the deep learning method to areas where errors may occur depending on the observer was examined. The experimental results revealed that the deep learning models recognized and estimated objects covered by obstacles, and only objects necessary for growth investigation were recognized within a certain error. For practical use, an optimized model was proposed, and the model exhibited an mIoU of 0.7369.