Members of this band had already released an album in 1968 as The Rockets, and had appeared on record twice with Neil Young as Crazy Horse. The core trio from the Rockets, Danny Whitten, Billy Talbot, and Ralph Molina, provided instrumental backing for Young's 1969 album Everybody Knows This Is Nowhere, and performed on some songs from Young's 1970 album After the Gold Rush. During sessions for the latter, they met teenage guitar prodigy Nils Lofgren, and producer/keyboardist Jack Nitzsche, who had worked with Phil Spector's Wrecking Crew and The Rolling Stones. They meshed, and Crazy Horse expanded into a quartet to back up Young for a short tour in early 1970, and then into a quintet with Lofgren for this album, picking up a contract with Reprise Records after the exposure garnered from Gold Rush.
This was Whitten's last album release before his death from a drug overdose in 1972. For the recording of this album in the fall of 1970, they also recruited Ry Cooder, who had worked previously with Nitzsche on sessions for the Stones. Cooder plays on three tracks.
The album contains compositions from four principal writers. Whitten's ballad, "I Don't Want to Talk About It", would be covered by a variety of artists, including Rita Coolidge; Everything but the Girl on their 1988 album Idlewild; and Rod Stewart, who had a hit with the song in the U.K., taken from his 1975 album Atlantic Crossing. Neil Young's "Dance Dance Dance," was covered by The New Seekers in 1972 and has yet to be released by Young himself (see also "Love Is a Rose", a related song). Randy Newman had already performed Nitzsche's "Gone Dead Train" on the soundtrack for the 1970 film Performance by Donald Cammell and Nicolas Roeg. The song was also covered by the Scottish hard rock band Nazareth on their 1977 album Expect No Mercy. The Lofgren-penned "Beggar's Day" was previously covered by the same band (Nazareth) on their 1975 album Hair of the Dog.
A live version of "Downtown," by Whitten and Young, would appear on Young's 1975 album Tonight's the Night. On Neil Young's 2007 vault release Live at the Fillmore East 1970, the song is credited to Whitten alone.
Crazy Horse was released on compact disc on March 22, 1994, as part of the Warner Brothers archive series, produced for compact disc by Lee Herschberg. It appeared in its entirety as part of Rhino Handmade's Scratchy compilation from 2005, which also included outtakes from the sessions for this album. That compilation is no longer in print.