Yes, assuming you had some "bad stuff" in the backup to begin with. For example, if you installed a malicious 3rd party app you found in some random site, then backed it up along with its settings to an SD card, then wiped/reset the phone to factory settings, then re-inserted the SD card with your backup and restored the malicious app from your backup, you're essentially back to where you started.
If you suspect that you have an infected/compromised system, and are factory resetting/wiping everything to start with a "clean" system, it may be prudent to re-download and then re-install all the apps from a known good source rather than restore from a backup.
If you suspect that the malware may have had root access, you may consider re-downloading and re-flashing the original stock ROM itself as an extra precaution using an update downloaded directly from the manufacturer. (This is because the malware may have installed itself in the /system directory-- that is, the firmware software itself.)
This is not a 100% failsafe method for being sure to have removed all traces of malware (once it has root, malware theoretically could install itself into other unusual partitions such as the radio/baseband, etc. but this would be difficult, device-specific, and I have not heard of this ever happening... yet), so I would suspect this should be effective.
But yeah, don't wipe everything only to restore from a backup of your system that has the original problem...