I was removed from all groups before when I changed my number. Will the same happen if I change my phone? This time, I have the same number but a different phone.
You retain your groups if you have the same number.
Off-topic, you may also want to check the official documentation on how to back-up and restore your chat history to the new phone.
Just backup your chat from WhatsApp-> Settings-> Chat settings-> Backup conversations. This will back up your chats to a database on the WhatsApp folder of your SD card.
Copy the folder to your new phone and just install WhatsApp and activate your account from your number. Restore old chats when asked.
No, you won't be removed from the group unless you don't change to a new phone number. Your groups will remain the same with the same phone number as earlier. If the group admin deletes the group or removed you, it depends.
In case you want to restore the message from the old phone to the new phone, back up all data from the Whatsapp backups directory which contains all your message backups as a
.crypt file encrypted, and copy this file to your new phone and restore all your message.