Does the m of the bob on the pendulum affect the time it takes to move back and forth?

No. It is the same principle that two objects will fall at the same speed, whatever their m. Air-resistance neglected, of course,