Descaling is an important part of freshwater fish processing. The detection of descaling rate is of great significance in evaluating the descaling equipment and monitoring the equipment in real time during operation. In view of the situation that the calculation of descaling rate in China is mainly artificial at present, which is time-consuming and laborious. this paper proposes a descaling calculation method of silver carp based on image segmentation, which divides the descaled fish body into two regions: the region with scale and the region without scale. The regions are distinguished by the improved Mask R-CNN. The Deep learning methods have good anti-interference ability and multi-dimensional feature extraction ability, can be applied to complex environment. In this paper, the SENet module is added into the feature extraction network with ResNet50 as Backbone, and the operation time is reduced while the network pays more attention to the valuable features. By improving the Loss function, the network optimization direction is forced to favor mask segmentation. Conclusion：The AP and Recall of the modified Mask R-CNN model reached 90.6% and 91.9% respectively in the data set of 1950 silver carp images. Compared with the original network, AP increased by 19.7% and Recall increased by 16.8%. The experimental results show that the average error is 4.7%. The fish scales with small areas are still unclear, and the mask boundary segmentation is not smooth. However, compared with the manual detection, this method realizes the calculation of descaling rate with convenience, speed, non-destructive, low cost and high accuracy. The method has good ability in distinguishment of different regions of fish body and real-time computing ability, and can be used for evaluating descaling equipment and real-time monitoring.