Julian's House By Judith Hawkes

Sally and David Curtiss knew that the old New England house was supposed to be haunted. But they weren't afraid of ghosts. That was before they learned there was nowhere in the house they were safe. Not even in each other's arms. "Especially not there."
 
- Goodreads Summary