どうも!
Learning techniques/methods vary from person to person, and as you know, not every person picks up on things in the same way or time frame. So, unfortunately, answering the question would be a suggestion at best. With the obvious stated and out of the way, please allow me to give you my suggestion: Turn them off now! haha.
It seems we might be finding ourselves focusing on the displayed translations from time to time, despite our best efforts in wanting to focus on the auditory dialogue. This diversion happens in the conscious and subconscious realms, and it all has something to do with our comfort zones. (disclaimer: i'm no psychological expert or anything; this is just my amateurish speculation) We want to focus on hearing the Japanese, but because we fear not understanding what is being heard, we "glimpse" at the subtitles for understanding. Over and over and over. The bottom line is, something takes precedence over another. Almost always. In our situation, reading English trumps listening to Japanese (because of our familiarity with the former).
Instead of listening to understand, try listening to become comfortable. Comfortable with the sounds and inflections of the voice actors... of the language. As mmmansonさん mentioned, choose something you have already watched (though it doesn't have to be). That way, you reduce your distractions because you do not have to worry about plot, some other type of development, or reading any subtitles. Therefore, listening to Japanese takes priority over anything else. Like mrsspookyさん, I've seen Cowboy Bebop x-amount of times, but this time around, it is solely being watched in Japanese, though my knowledge is still at a beginner's level. But that is not an issue. What matters is that the language is becoming familiar, comfortable, accustomed, or even "domestic." Words that are known stick out and my ears are getting the hang of breaking the rest up in syllables. And because I know my ひらがな and かたかな fairly well, I sometimes grab my English-Japanese dictionary and search for the word(s) that was supposedly heard in that "decoding." Sometimes it is a success, other times it is not. (however, my dictionary is mostly romanized, so it too is another distraction in my eyes)
Force yourself out of your comfort zone and worry not about grasping comprehension from the beginning. Embrace 日本語 like a new born and let her melodious voice caress your ears as she tenderly holds you in her bosom of warmth and security. がんばって!