It does matter. Using 5V could blow up one of the two pins and then it would not work anymore. TTL voltages are actually 1.2, 1.5, 1.8. 2.5, 3.3, or 5V depending on the processor technology. But they are not interchangeable, meaning all of them will not work just because they are referred to as TTL.
You must use 3.3V just as the System Reference Manual states.
.
Gerald