Infrared (IR) imaging can highlight thermal radiation objects even under poor lighting or severe sheltering but suffers from low resolution, contrast, and signal-to-noise ratio. While visible (VIS) light imaging can guarantee abundant texture details of targets, it is invalid in low lighting or sheltering conditions. Therefore, IR and VIS image fusion has more extensive applications, but it is a still challenging work because conventional methods cannot balance dynamic range, edge enhancement, and lightness constancy during fusion. To overcome these drawbacks, we propose a self-supervised dataset-free method for adaptive IR and VIS image fusion named deep Retinex fusion (DRF). The key idea of DRF is first generating component priors that are disentangled from a physical model using generative networks; then combining these priors, which are captured by networks via adaptive fusion loss functions based on Retinex theory; and finally reconstructing the IR and VIS fusion results. Furthermore, to verify the effectiveness of our reported physics driven DRF, qualitative and quantitative experiments via comparing with other state-of-the-art methods are performed using public datasets and in practical applications. These results prove that DRF can provide distinctions between day and night scenes and preserve abundant texture details and high-contrast IR information. Additionally, DRF can adaptively balance IR and VIS information and has good noise immunity. Therefore, compared to large dataset trained methods, DRF, which works without any dataset, achieves the best fusion performance.