In May 1935, he married singer Louise Tobin, and they had two children but were divorced in June 1943. Shortly after, joining the Pollack band he was approached by Benny Goodman, and began working for him by the end of 1936. By December 1937 he had begun to make recordings under his own name for Brunswick Records (later Columbia Records). In early 1939, he left Goodman and had his own orchestra. That spring, he heard the then unknown Frank Sinatra on a radio broadcast and hired him but when the more successful bandleader Tommy Dorsey made Sinatra an offer at the end of 1939, James did not stand in his way. In April 1941, he first reached the Top Ten with "Music Makers." (His band was sometimes billed as Harry James and His Music Makers.) A second Top Ten hit, "Lament to Love," featuring Dick Haymes on vocals, followed, and late in the year he reached the Top Five with an instrumental treatment of the song "You Made Me Love You." This was the record that established him as a star. James was second only to Glenn Miller as the most successful recording artist of 1942. During the year, seven of his recordings peaked in the Top Ten:
I Don't Want To Walk Without You, with vocals by Helen Forrest; "Sleepy Lagoon"; "One Dozen Roses," with vocals by Jimmy Saunders; "Strictly Instrumental"; ; He's My Guy","Mister Five by Five"; and "Manhattan Serenade," the last three with vocals by Helen Forrest.

When Miller went into the armed forces and gave up his radio show, Chesterfield Time, he handed it over to James. (James was ineligible for military service due to a back injury.) He appeared in films, Syncopation in 1942 and Private Buckaroo in June and Springtime in the Rockies in November.

His next hit, I Had The Craziest Dream, with vocals by Helen Forrest, was featured in the film Springtime in the Rockies; it hit number one in February 1943. The film also starred Betty Grable, whom James married in July 1943; they had two children but were divorced in October 1965. This tune was succeeded at number one in March 1943 by another James record with a Helen Forrest vocal, "I've Heard That Song Before." "Velvet Moon," , did almost as well. Frank Sinatra had recently emerged as a solo star, and Columbia reissued All Or Nothing At All, a song he had recorded as James' vocalist in 1939; the song reached the Top Five. Next, Columbia released "I Heard You Cried Last Night," a year-old recording with a Helen Forrest vocal; it too reached the Top Five. Again, James ranked as the second best recording artist of the year, just behind Bing Crosby. His instrumental "Cherry," , became a Top Five hit; "I'll Get By," recorded in 1941 with Dick Haymes on vocals, hit number one in June; and he had eight other chart records during the year. He also had two films, Two Girls and a Sailor and Bathing Beauty, on release in June.

He then had seven Top Ten hits in 1945: the number one "I'm Beginning to See the Light"; "I Don't Care Who Knows It"; "If I Loved You"; "11:60 P.M." ; The Top Five "I'll Buy That Dream"; "It's Been a Long, Long Time"; and "Waitin' for the Train to Come In." "If I Loved You" had vocals by Buddy DiVito; all the rest had vocals by Kitty Kallen. That was enough to make him the third most successful recording artist of 1945, behind only Bing Crosby and Sammy Kaye. James scored two Top Ten hits in early 1946 -- the Top Five "I Can't Begin to Tell You," featured a vocal by his wife Betty Grable, and "I'm Always Chasing Rainbows", with a vocal by Buddy DiVito -- but then his recording success began to decline, though he managed one more Top Ten hit, "This Is Always," with Buddy DiVito on vocals. He signed a contract with 20th Century Fox, resulting in bigger parts in Do You Love Me?, and If I'm Lucky, out in September. The declining popularity of the big bands led him to disband his band. But in January 1947, his All Time Favourites collection was at the top of the album charts, indicating he was still broadly popular, and he re-organized his band, taking a more jazz-oriented approach.

He scored only one Top Ten hit in 1947, "Heartaches",with vocals by Marion Morgan. And he appeared in the film Carnegie Hall . James appeared in the film A Miracle Can Happen (aka On Our Merry Way).In February 1950, his trumpet playing was heard in the film Young Man with a Horn, though the man fingering the trumpet onscreen was Kirk Douglas. Then Columbia teamed James with Doris Day for "Would I Love You," which hit the charts in March 1951 and reached the Top Ten. Similar success was achieved with Castle Rock, which had Frank Sinatra with Harry James and reached the charts in September.
James had his own TV series, The Harry James Show, which ran for the six months in 1951.
He played himself in the film biography The Benny Goodman Story in 1955. He was back onscreen in November 1956 in the film The Opposite Sex. He made his first major tour of Europe in 1957, and in following years he alternated national and international tours with lengthy engagements at Las Vegas. There were two more film appearances, The Big Beat (1958) and The Ladies Man (1961).
James performed regularly through the early '80s. He was diagnosed with cancer in 1983, but continued to play, making his last appearance only nine days before his death at 67.