Obviously not s-f as such, but - quite a few novels by Nabokov. "The Luzhin Defence" (aka "The Defense"), "The Eye", "Bend Sinister", "Transparent Things" - all very immersing, and posing the question of what's the true reality, its underpinnings and seams. Challenging reads for sure, every detail matters. Some might recommend Pynchon as well, personally I'm not convinced but to each his own.
Also recommended in this reply