During fish farming, the number of fish in the ponds needs to be monitored regularly. Aiming at the problem of missing detection in the existing methods, this paper proposed a local optimization method based on the YOLOv5s model and a fish quantity statistics method with an improved output scale. By adding local information such as the head and tail of the detected fish, the category with the largest number was selected from the three categories of the whole body, head, and tail of the fish as the result of quantitative statistics to solve the problem of missing detection. At the same time, for the case that the whole body, head, and tail of fish were displayed as large-scale or mesoscale targets in the image, the feature output of these two types of targets was increased to improve the target detection ability of the model, so that the model can be suitable for quantitative detection under the current conditions. The results showed that compared with manual counting, the error of the quantity counted by this method was small, the accuracy was 96.3%, and the detection frame rate was 111 FPS. Based on the YOLOv5 model, the application of local optimization strategy increased the number of statistics by 37.4%, and the improvement of output scale increases the number of statistics by 4.9%. The study can be applied to the statistics of fish stocks in fishery and fish detection.