When director Denis Villeneuve first approached Jake Gyllenhaal to star as a detective in his new dramatic thriller, “Prisoners,” the actor was skeptical. The movie contained plenty of genre elements — heinous crime, creepy suspect — that could have turned it into pulp cliché. And besides, Gyllenhaal, fresh off the LAPD street story “End of Watch,” had just played a cop.

“The initial conscious response was, ‘Another drama where I played a member of a law enforcement? I don’t know,'” Gyllenhaal said. “But to me, this is a movie about the unconscious.”

Indeed, there is much going on beneath the surface of “Prisoners,” both with the detective and the man who plays him. Det. Loki is one of the more complicated police presences on the big screen recently. Moving beyond the conventions of the typical screen gumshoe, he’s a man brimming with confidence but boiling with rage, as firmly convinced of the need for justice as he is frustrated by his inability to bring it about.

The film, which opens Friday, is the result of another mental journey to a dark place for the actor, who has been submerging his consciousness in roles like this recently and in the process reenergizing his career. Eight years after he landed a supporting Oscar nomination for “Brokeback Mountain,” Gyllenhaal, at 32, looks to be making good on the promise he showed in that film and creating work to match his level of popularity.

Gyllenhaal has an an unusually high degree of association with cultural phenomena (“Brokeback,” “Donnie Darko”) as well as critical darlings (“Zodiac,” “The Good Girl”). But big studio movies haven’t always been kind to him. Two years after his turn as the naive love interest in 2002’s “The Good Girl,” there was the Roland Emmerich kitchen-sink effects tale “The Day After Tomorrow.” And his well-regarded role as the author Robert Graysmith in 2007’s “Zodiac” led to the maligned desert action-adventure “Prince of Persia: The Sands of Time” several years later. Some actors grow into themselves when they move to a bigger stage (think: Denzel Washington). But for Gyllenhaal, these movies didn’t play to his strengths, and he seemed dwarfed by his surroundings.

This has left him with a kind of restless questioning, a desire to do things grittier, more his own way, not simply to jump on the next sleek passing vessel.

He demonstrated the point recently when he opted out of a Disney adaptation of the big-budget fairy tale musical “Into the Woods” to sign on to something more moody — an indie thriller called “Nightcrawler” in which he’ll play a Los Angeles crime journalist.

It’s not an accident that he’s taken on the wounded masculinity of movies such as “End of Watch” and “Prisoners” just as he’s taken up boxing and mixed-martial arts, the ultimate sports for actors craving both swagger and pain. As with those endeavors, Gyllenhaal believes if he just picks his targets with enough foresight and engages in enough preparation, he can get to an optimal career place and land his punches.

“I can’t totally put it into words, but I feel like I’ve begun to prepare in strange ways in the past three or four movies, starting with ‘End of Watch,'” said the soft-spoken but intense actor. “I don’t think I’m sharp enough to not prepare and come on set and kill it. I’ve learned over the years that freedom is just the other side of discipline.”

To play an L.A. cop in David Ayer’s “End of Watch” last year, Gyllenhaal spent months on the beat with police observing “extraordinary and horrific things.” Soon after that, he began working with the French Canadian auteur Villeneuve, whose “Incendies” was nominated for 2011’s foreign-language Oscar and was notable for its brutal revelations.

Villeneuve first cast him in “Enemy,” an erotic thriller adapted from a José Saramago novel. Gyllenhaal plays both lead parts — a teacher and an unemployed actor who may be his doppelgänger. The movie, which does not yet have U.S. distribution, has a Jungian quality and evokes the brooding, paranoid spirit of other double-trouble movies such as “Dead Ringers” and “Black Swan.”

Shortly after they started working on “Enemy,” Villeneuve began pushing Gyllenhaal to take on “Prisoners” as well. Though it’s rare for a high-profile actor to make such a back-to-back commitment with a little-known director, Gyllenhaal believed it was a risk worth taking and signed on long before “Enemy” was completed.

“The most success I’ve had in my career comes from those shots, those gambles — ‘Brokeback,’ ‘Donnie Darko.’ You have to know what games you’re good at. You have to know what position you play,” Gyllenhaal said. “I love movies that are saying things that people might find odd at times. I don’t find them odd at all. They give me comfort.”

Both films played at the Toronto film festival that ends Sunday, and both received strong reviews. “Prisoners,” in particular, has been a favorite; Variety called the actor’s turn a “career best” and noted the “full breadth of his impressive range.”

Directed from a script by Aaron Guzikowski, the movie slyly explores questions of revenge, parental responsibility, individualism and the suspicions that rattle around the human brain. Gyllenhaal’s Loki swaggeringly tries to crack the difficult case of two missing girls on behalf of a contractor named Keller Dover (Hugh Jackman), whose daughter disappears along with a friend outside their small-town Pennsylvania home. Convinced Loki is not doing enough, Dover secretly kidnaps the man he believes abducted the girls (Paul Dano) and tortures him to find the truth. As Loki is thwarted by the case, he and Dover become mistrustful of each other.

Gyllenhaal plays the part with a mixture of sympathy and bravado; prominent tattoos and references to a tough childhood suggest a criminal past that further colors the character. The performance is rife with subtle contradictions. Loki is full of bravado about his ability to find the kidnapper but also is manifestly unsure of how to do it.