Recently after a large update, Onedrive apparently enabled a new feature "Files On-Demand". After I disabled this option, photo previews were not loading anymore in Windows 10 file explorer (either as thumbnails or in the preview pane). Clearing the thumbnail cache with disk cleanup did not solve the problem. Enabling the File On-Demand option again solved the preview problem.
However, I really want to have all my file stored locally, so this is not a solution for me.