Jake Gyllenhaal made a deal with himself 10 years ago: For every three movies he made, he would perform in a play.

It was a deal he didn’t keep.

He was 21 back then, on a high from his London stage debut as a sensitive slacker in Kenneth Lonergan’s “This Is Our Youth,” the sort of hangdog character that had turned him into an indie darling in movies like “Donnie Darko” and “Lovely & Amazing.” But going off to do plays isn’t part of the Hollywood fast track for young actors still proving themselves at the box office. So Mr. Gyllenhaal tested for the Spider-Man and Batman franchises and other roles that might transform him into an action hero or leading man.

What happened? The critically derided disaster movie “The Day After Tomorrow” happened. The much-mocked “Prince of Persia: The Sands of Time” happened. Acclaimed films happened too, like “Brokeback Mountain” and “Zodiac.” But Mr. Gyllenhaal was uneasy.

“I wasn’t really listening to myself about the kinds of projects I wanted to do,” he said in a recent interview, reflecting on the past decade. “I had to figure out what kind of an actor I wanted to be and feel confident going for that.”

He has now come to a few conclusions, and they were evident last month at a table reading for his first outing in New York theater, “If There Is I Haven’t Found It Yet,” a dark comedy about an overweight British teenager and her troubled family. The project itself was telling: The play, which will begin performances on Friday from Roundabout Theater Company, is an Off Broadway ensemble work by a little-known writer rather than a famous Broadway drama by a prizewinner like Arthur Miller — the vehicles of choice for Hollywood stars these days.

Hunched over a script beside his cast mates and director, Mr. Gyllenhaal rolled through questions on his mind about a scene in which his character — Uncle Terry, brokenhearted and charmingly roguish — reveals a few of his many problems.

“When was the last time I talked to Rachel?” Mr. Gyllenhaal asked, referring to Terry’s ex-girlfriend. “Did I see Rachel at the funeral, or after?” And then: “I must’ve done something that made her say, ‘I’m tired of this guy.’ What was it?”

These questions, and the many that followed, were the sort that classically trained actors ask as they probe layers of their characters to puzzle out intentions, tones and emotional shades for imbuing a performance. Mr. Gyllenhaal studied at Columbia University for two years before dropping out to become a movie star, and some directors on earlier films, like Ang Lee of “Brokeback Mountain,” have described him as a freestyle actor more than a methodical one.

Mr. Gyllenhaal, who was nominated for a supporting actor Oscar for “Brokeback,” said he still revels in experimenting with his take on characters from scene to scene and performance to performance. But acting rigor is increasingly his goal, and perhaps the respect that comes with it.

“Early in your career it’s hard to know everything for yourself, and asking questions isn’t always a welcome thing in Hollywood, where everyone seems like they know what they’re doing,” he said during an interview over tofu salad and squid sashimi at a Japanese restaurant in the West Village.

“Around the time I hit 30, I asked myself if I was respecting acting as a craft,” he continued, frequently brushing his fingers through his thick hair, free of its black-on-black Yankees cap, “and if I was doing the right projects that deserved my attention and where I’m learning in a way that you might not feel at 15. So now it’s like I look at acting more as building little delicate cricket cages, with care and more thought.”

In the past 18 months Mr. Gyllenhaal parted ways with his longtime manager, signed with a new agency and began devoting more time to selecting and preparing for projects. He recently wrapped a role as a history teacher in another coming film, “An Enemy,” for which he e-mailed frequently with one of his old Columbia professors about the art of delivering classroom lectures. He spent five months observing and training with Los Angeles police officers for his new movie, the September release “End of Watch,” in which he plays a hotshot cop with street smarts in South Central Los Angeles.

Mr. Gyllenhaal went on repeated ride-alongs, witnessing gang shootouts, and immersed himself in weapons and martial arts training. He also gave feedback on casting and was on set constantly for scenes he wasn’t in and to watch dailies. His commitment to “End of Watch” was so complete that John Lesher, one of its producers, decided after filming to make Mr. Gyllenhaal an executive producer. (Mr. Gyllenhaal comes from a family of successful filmmakers: His father, Stephen, is a director; his mother, Naomi, is a screenwriter; and his sister, Maggie, is an Oscar-nominated actress.)

Mr. Lesher had long been an admirer of Mr. Gyllenhaal’s but only came around to casting him after “Jake mounted a campaign to meet with us and be in the movie.” Asked if Mr. Gyllenhaal was trying to prove something with “End of Watch,” Mr. Lesher continued:

“I just think it’s a very grown-up part, and something very different from Jake’s own personal life and upbringing, and we all wanted the movie to feel as real as possible. I don’t think we’ve seen all that Jake’s capable of, and this is a good example of how he’s trying to show all that he can do.” (The two men plan to work next on a movie about gambling addicts in the South.)

Mr. Gyllenhaal was careful to say that he didn’t think he needed a career reboot, but the fact remains that his continued celebrity is more about his raw talent and good looks than a proven record of hit films. Hollywood hasn’t seemed entirely sure what to do with him, and Mr. Gyllenhaal sounds at peace with that, saying he wouldn’t want to be pigeonholed as an actor at a time in his life when he is still finding his way.

“What I loved most about working in London, in the theater, there was a real appreciation of potential,” he said. “No one comes out of the gate 100 percent perfect. No one. I have a great sense of comfort onstage because I know taking risks is appreciated.”

Growing up in Los Angeles, with frequent trips to New York to see relatives, Mr. Gyllenhaal was a theatergoer long before he stood on a stage himself. Sitting in nosebleed seats as a boy, he was dazzled by Patti LuPone vamping through the opening number “I Get a Kick Out of You” in “Anything Goes” for Lincoln Center Theater; he then promptly fell asleep. From other musicals he graduated to serious drama like “Angels in America,” though mostly he went to school plays featuring Maggie, now 34.

Both Gyllenhaals live in New York now — Maggie in Brooklyn, Jake downtown in Manhattan — and Ms. Gyllenhaal has had high profile turns in Chekhov plays at Classic Stage Company in recent years. Mr. Gyllenhaal saw those and said he came away envious of his sister being exhausted and enthralled by theater work.

As with “End of Watch,” Mr. Gyllenhaal aggressively pursued a role in this new play — a bit strange for a celebrity, given that many theater producers are desperate to cast one, but perhaps not so unusual for an actor determined to reorient his career. Mr. Gyllenhaal came across “If There Is I Haven’t Found It Yet” a couple of years ago, after attending a play reading of a friend and asking around for new scripts to read. He flew to London to meet with the writer, Nick Payne, and continued talking about options for staging the work in the United States.

Then last winter Todd Haimes, the Roundabout artistic director, saw Mr. Payne’s critically acclaimed “Constellations” in London; he looked into mounting that work at Roundabout, but it wasn’t available, so he read “If There Is” and heard that Mr. Gyllenhaal was interested. They held a reading in March; Mr. Haimes said he quickly offered to stage the play, and Mr. Gyllenhaal enthusiastically embraced the idea of doing it Off Broadway, where celebrities seldom tread these days.

Broadway is more the province of movie stars, with its higher profile, bigger paychecks and eligibility for Tony Awards. (Off Broadway plays like Mr. Gyllenhaal’s are not eligible for Tonys.) Usually young movie stars come to Broadway and choose renowned works; Andrew Garfield did “Death of a Salesman” last season, while Scarlett Johansson won a Tony for “A View from the Bridge,” and Katie Holmes played a featured role in “All My Sons.”

Mr. Payne said that he and Mr. Gyllenhaal first bonded over the staccato structure of Terry’s dialogue. While the other three characters in the play are given to well-formed sentences and occasional speeches, Terry speaks mostly in fragments and leans frequently into pauses as he struggles to express deep-seated anger and self-loathing.

“Jake really got the rhythms and desperation in the dialogue, and its switches — the way a few words could be heartsick, and then there would be a period, and then the next few words could be furious,” he said.

Terry is far from a glamorous character, though he has a bad-boy appeal not unlike another beer-guzzling layabout named Uncle Terry: Mark Ruffalo’s character in Mr. Lonergan’s movie “You Can Count on Me.” But Mr. Gyllenhaal’s Terry is a rougher sort. He turns on a dime in several pivotal scenes with his niece, Anna, who is being bullied at school because of her weight.

The actress playing Anna, Annie Funke, said that the emotional intensity of rehearsals had been eased for her by Mr. Gyllenhaal’s kindness. Recently, she said, they were working on a scene in which Terry is giving Anna a hard time about her weight and sulkiness. In an acting exercise suggested by the director, Michael Longhurst, Mr. Gyllenhaal touched or poked Ms. Funke every time he said something to her, to physicalize the psychological effect of taunting. Ms. Funke eventually broke down in tears.

“I just reached a breaking point because I hadn’t quite realized before, until Jake was poking me, what it felt like to be picked on and bullied and how all of that must make Anna feel,” Ms. Funke said. “I was just completely overwhelmed. And Jake grabbed me and hugged me, and we finished the scene. He has looked out for me completely.”

It was Terry’s capacity for cruelty that appealed to Mr. Gyllenhaal most of all. “The intentions of Terry are very different from anything I’ve played before, especially his vicious side,” he said. “It intrigued me so much, and that was the sign. I want to come home at the end of the day and be wiped out and feel I’ve torn my heart out from acting and feel fulfilled. At this point I don’t have the desire to do anything other than projects that make me feel that way.”