The application of target detection and recognition based on deep learning in aquaculture has problems such as low data set quality, high network computing complexity, and slow inference speed, which is not easy to meet high real-time application scenarios. This study collected and annotated 10,042 image datasets of 83 species of aquarium fish, and then explored network optimization methods on the basis of ensuring target detection and recognition capabilities, reducing network computational complexity, and improving inference speed. Redesign the backbone network of YoloV4 network using "depth separable convolution" and compare the optimization effects of different data enhancement methods such as Mixup, Cutmix, Mosaic, Mish, Swish, and ELU on the network. According to the comparison results, the combination of data enhancement method and activation function is selected to optimize the network. The results show that the prediction accuracy of the network optimized by this method on the test set reaches 94.37%, and the computational complexity (BFLOPS) is only 5.47, 93.99% lower than that of YoloV4. The result shows that the method of optimizing the network in this study can greatly reduce the computational complexity of the network and improve the inference speed on the premise of ensuring the accuracy of detection and recognition, which provides a reference for fish target detection and recognition in high real-time application scenarios.