My wife did the same exact thing your are thinking of doing. that was back in 2010, when we moved here from Brooklyn. MLG – stands for Montgomery, Liberty, Grand, and they are three streets running parallel to the hudson river, also known as the Historic District. It is only one of many cool neighborhoods in the City of Newburgh. I think the previous person who replied to you suggested you move there while you “test the waters,” is because it is very commutable. My wife and I used to just walk to the ferry from our apartment on Grand Street, which used to take about 8 minutes, then take the ferry to
Beacon (that pass was $11/mo. back then), and finally a beautiful train ride into Grand central Station. All told it was 1hr and 40 minutes from my door to Grand Central Station.
A one bedroom is possible for about $700/month. Ours was $750 and had the most amaing views of the river.
Then the City of Newburgh is pretty walkable, although there aren’t any great supermarkets right now. There are however, farmer’s markets and CSAs that you can walk to easily.
We bought a house on Liberty Street and renovated it for $140,000. It is over 4000′ sq and it’s a two family. You can easily buy a smaller property for less (although most of the houses in the historic district are quite big).
A great realtor to talk to is Chris Hanson from Historic Newburgh Homes.