Energy is not the only thing that is intrinsically coupled to matter. So is space-time. Without matter there can be no space, or time, or space-time. This was pointed out by Einstein himself.

Just ask yourself what it would mean to measure space or time in the complete absence of material objects. Space and time are relational, not a priori.

Fractal Cosmology