In this paper, the finite-time and fixed-time cluster synchronization problem for complex networks with or without pinning control are discussed. Finite-time (or fixed-time) synchronization has been a hot topic in recent years, which means that the network can achieve synchronization in finite-time, and the settling time depends on the initial values for finite-time synchronization (or the settling time is bounded by a constant for any initial values for fixed-time synchronization). To realize the finite-time and fixed-time cluster synchronization, some simple distributed protocols with or without pinning control are designed and the effectiveness is rigorously proved. Several sufficient criteria are also obtained to clarify the effects of coupling terms for finite-time and fixed-time cluster synchronization. Especially, when the cluster number is one, the cluster synchronization becomes the complete synchronization problem; when the network has only one node, the coupling term between nodes will disappear, and the synchronization problem becomes the simplest master-slave case, which also includes the stability problem for nonlinear systems like neural networks. All these cases are also discussed. Finally, numerical simulations are presented to demonstrate the correctness of obtained theoretical results.