Making accurate predictions of subway passenger flow is conducive to optimizing operation plans. This study aims to analyze the regularity of subway passenger flow and combine the modeling skills of deep learning with transportation knowledge to predict the short-term subway passenger flow in the scenarios of workdays and holidays. The processed data were collected from two months of Automated Fare Collection (AFC) data from Xizhimen station of Beijing metro. The data were first cleaned by the established cleansing rules to delete malformed and abnormal logic data. The cleaned data were used to analyze the spatial characteristics in passenger flow. Second, a short-term subway passenger flow prediction model was built on the basis of long short-term memory (LSTM). Determining that the error will be relatively high in peak hours, we proposed gradual optimizations from data input by dividing one whole day into different time periods, and then used particle swarm optimization (PSO) to search for the optimal hyperparameters setting. Finally, inbound passenger flow of Beijing Xizhimen subway station in 2018 was selected for numerical experiments. Predictions of the LSTM-based model had higher accuracy than the traditional machine learning support vector regression (SVR) model, with mean absolute percentage error (MAPE) of 21.97% and 4.80% in the scenarios of workdays and holidays, respectively, which are both lower than those of the SVR model. The optimized PSO-LSTM model has been verified for its effectiveness and accurateness by the AFC data.