Why haven't you answered my phone? Can we use didn't instead of haven't

Yes, you can. But after that, the mean of sentence will be changed and you will also change the verb answered in the first form of verb in 'answer'.