The data obtained from the breeding environment has defects such as blurring and occlusion, rendering poor detection of the farmed fish. To solve the above problem, the farmed fish detection model ESB-YOLO (ECBAM-SPPF-BiFPN-YOLO), based on a non-channel-downscaling attention mechanism (ECBAM) and improved YOLOv5, is proposed. ECBAM is adopted to obtain more detailed features. To alleviate the slowdown of detection speed caused by adopting ECBAM, SPPF is used to replace SPP to reduce model computation, decrease model detection time, and improve detection speed. BiFPN is used to perform weighted feature fusion to enhance feature fusion by increasing the proportion of functional features and reducing feature loss. To verify the effectiveness of the proposed modified module for YOLOv5, ablation tests were designed. The test results show that the average precision of ESB-YOLO is improved by 2.40% compared with YOLOv5 while maintaining a similar detection speed. To demonstrate the competitiveness of ESB-YOLO, comparison tests were designed. Compared with advanced underwater object detection models such as FERNet, SWIPENet, and SK-YOLOv5, the average precision of ESB-YOLO is 3.10%, 3.90%, and 0.70% higher, respectively. The study shows that the proposed model is more effective and capable of farmed fish detection.