In winter months the river freezes and the ferry service stops and is replaced with a bus service. One of the buses that goes to NYC leaves from Broadway but that is out in the town, far from the city. I am not too sure if there are buses that leave from the city of Newburgh straight to NYC, but I doubt it. But here is a link to check out with bus service schedules.
A great weekend to head up will be this weekend. Columbia students will be out at 12 different sites for their urban design studio, and it is Last Saturdays in Newburgh. There will be extra exhibitions and longer business hours all around to help promote the area. Hope this helps!