How can use One media convert to give internet coection to 4 houses?

Can I use one media converter to give inteet access to 3 houses near my. My ISP saying he will be needing 2 media converts at each home with 8 port switch at each home. is it necessary? I mean can't we coect like

Fiber optic cable -> Media Converter -> 8 port switch -> (via LAN Cable) WiFi Routers ?