Apache2 serves all files in the wordpress directories tree. Did I misconfigure it?

First I noticed that if I go to http://example.org/wp-config.php, it gives me an empty page instead of 404 error. I tested it with another file - wp-config-example.php, and it gave me a page with the following text: "Error establishing a database connection". So it definitely just executed that php script. If I go to http://example.org/wp-content/themes/TheFox/screenshot.png, apache2 serves me that file.

This seems very dangerous to me. I didnt know about it for some time and kept a .sql backup of the whole wordpress database in the wordpress directory - anyone could've downloaded it.

I've just configured apache2 with <Directory /path/to/wordpress/dir/, and it worked, so I thought everything's done right, but now I am in doubt.

- - , .