Things are getting tense between Hugh Jackman and Jake Gyllenhaal in Prisoners.

The thriller (out Sept. 20) features Jackman as a frantic father who is searching for his kidnapped daughter and is frustrated with the police detective (Gyllenhaal) on the case. For Gyllenhaal and French-Canadian director Denis Villeneuve, it’s their second collaboration in a row, with their thriller An Enemy due out later this year.

Gyllenhaal says his Detective Loki and Jackman’s Keller Dover are working on different levels in Prisoners. “This is instinct meeting institution, and neither of us looks very happy. But there are similarities between the two of them, which is why they butt heads.”

A scene in which Jackman confronts Gyllenhaal in his car, about keeping a suspect in custody, also marks a key point in the unfolding drama.

“They have a chance to work together, but at this moment, my character goes off on his own,” says Jackman. “This is where there is a parting of the ways. And it’s the beginning of some pretty hairy stuff.”

Torture to be exact. Jackman kidnaps the man he believes committed the crime (played by Paul Dano in full creepy mode). It’s a desperate, criminal effort to extract vital information about the disappearance of his daughter and another missing girl.

“The movie doesn’t condone violence,” says Gyllenhaal. “But at that same time it’s so relatable to need to solve the situation yourself as a parent, taking things into your own hands.”

It’s a feeling that Jackman, a parent to two children, can relate to. Even thinking about the concept of a child going missing causes a physically powerful reaction inside of him.

“As a parent, you immediately feel sick to your stomach at the thought of it. It’s so devastating when that happens. And so unnatural,” says Jackman. “That normal desire to protect is so strong. When a child goes missing like that, it’s almost too much to bear.”

“I am not someone who feels the end justifies the means,” he adds. “But then again, when it comes to my child, I might feel differently.”

The momentous act spins the volatile situation even more out control. Meanwhile, the question lingers about who actually committed the crime. Even Gyllenhaal’s brooding detective, who’s obsessed with the criminal mind, is not above suspicion.

“It is a full-on whodunit,” says Gyllenhaal. “By the end, you’re thinking anyone could have done it.”

Prisoners’ supporting cast includes Maria Bello (as Dover’s wife), along with Terrence Howard and Viola Davis (playing parents of the other missing girl) . Villeneuve says he needed a deep bench on the Atlanta set.

“This is about a parent’s journey, what they are willing to do to find their daughter,” says Villeneuve. “We had to dig into the human soul. And we needed the right actors. The movie is only as good as their performances.”

Jackman worked hard to make his sleep-deprived character look and act real. Not only was the subject harrowing, but filming took place over the winter while he was in the middle of the 2013 awards season for his Oscar-nominated role in Les Misérables.

“So I was flying to red carpets and dressing up and then going back to this movie. It was very schizophrenic,” says Jackman, “And I was catching a lot of red-eyes (flights) on my way back from Los Angeles or London. So there was not much acting required for that look.”