Should I use poison to kill the squirrels?

Absolutely not. There are no registered or legal poisons for killing squirrels. Using rat poison is illegal and ineffective, and it poses a huge risk of secondary poisoning to pets, owls, hawks, and other wildlife that might eat the poisoned squirrel. Furthermore, a poisoned squirrel will likely crawl into a wall void to die, leaving you with an unbearable smell and a costly removal job.