That would be too easy to spot.
Now, replace the ball with one that is just slightly undersized...
When the victim picks-up the mouse and returns it to see what is happening, that should be enough for the ball to get in contact with the rollers and the cursor would move when the ball is rotated. Put the mouse back in the right orientation, the ball falls and loose contact and the cursor is immobile.