In this paper, we develop a fast linearized virtual element method (VEM) for the approximation of the nonlinear time-fractional diffusion equations on polygonal meshes. The L1-scheme with graded meshes is used to deal with the non-smooth system, the Newton linearized method is adopted to handle the nonlinear term and VEM is employed to discrete the spatial variable. Then the error splitting approach is used to prove the unconditional optimal error estimate of the fully discrete linearized L1-VEM scheme. In order to reduce the storage and computational cost caused by the nonlocality of the Caputo fractional operator, a fast memory-saving L1-VEM is developed. It is proved that the difference between the solution of the L1-VEM and the fast L1-VEM can be made arbitrarily small and is independently of the sizes of the time and/or space grids. Finally, numerical results are implemented to verify the theoretical results.