With the dramatic expansion of large-scale videos, traditional centralized face recognition methods cannot meet the demands of time efficiency and expansibility, and thus distributed face recognition models were proposed. However, the number of tasks at the agent side is always dynamic, and unbalanced allocation will lead to time delay and a sharp increase of CPU utilization. To this end, a new distributed face recognition framework based on load balancing and dynamic prediction is proposed in this paper. The framework consists of a server and multiple agents. When performing face recognition, the server is used to recognize faces, and other operations are performed by the agents. Since the changes of the total number of videos and the number of pedestrians affect the task amount, we perform load balancing with an improved genetic algorithm. To ensure the accuracy of task allocation, we use extreme learning machine to predict the change of tasks. The server then performs task allocation based on the predicted results sent by the agents. The experimental results show that the proposed method can effectively solve the problem of unbalanced task allocation at the agent side, and meanwhile alleviate time delay and the sharp increase of CPU utilization.