Have you tried going into your sound control application (not sure what desktop you're using) and selecting the other port? Video seems to go automatically to whichever port you plug a cable into, but I found that sound has to be selected manually (at least under Gnome 3). There are two choices for output device, both of which look exactly the same, but one corresponds to each HDMI port.
Thanks for the suggestion. I'm using the cinnamon desktop. There's a sound-control widget on the toolbar which has an option to set the output device, but there's only one device listed "Built in audio digital stereo (HDMI)", so that doesn't help much.