In massive Multiple-Input-Multiple-Output（MIMO） system

traditional detection algorithms suffer from more performance loss and higher complexity with increase in the number of antennas. In order to solve the complexity problem

a likelihood criterion based on neighborhood search as a branch strategy is proposed. Firstly

the quadratic programing model is constructed

and the model is used to search tree as the root. Then

the node with the smallest value of the objective function is found as the branching node. Finally

the likelihood criterion is applied to the branching nodes for reducing the complexity of the branch. Specially

when the bit error rate achieves 10

-4

the performance gain of the proposed algorithm is increased by 1.5 dB and the complexity of the proposed algorithm is reduced by about 69.84 percentage points compared with the traditional search tree algorithm in 16 quadrature amplitude modulation. The simulation results show that the proposed algorithm has better error performance and lower complexity.