Photoshop: flip brightness while preserve the color

as the title is saying. Using Photoshop, I want to invert the luminosity of my image using adjustment-layers, while the colors stay the same.

So here is my example image:

• Dark Background (it's totally black in this example)
• Light Text (it's totally black in this example)
• two glowing edges

To explain it a little further. The expected result based on this image should look like the following:

• White Background
• Black Text
• Colors are the same

The Problems

I don't know how I can prevent the colors from being manipulated as well.
If I try to invert the image with an Invert Adjustment Layer with the blend mode of luminosity (which is the last in the list) it won't give me the desired result.

As you can see here, the Hue of the glow is still the same.
But the brightness has been flipped too.

I also tryed to calculate Chaels in LAB-Mode and could not come up with a decent looking result.

The Solution

I don't want to manually select the text & the shape and fill it with the opposing color. That destroys the edges and looks awful.

Quick thought at the end: I need an selection by saturation if that's possible ... ;-)