No, "let's count to three" is correct. It is a common phrase used to suggest that a group of people count together to achieve a common goal or to synchronize an action.