The often-repeated story goes that Brian Wilson was so enamored with the Beatles' Rubber Soul that he made the Beach Boys' Pet Sounds as a reaction to it. (And that album in turn inspired Sgt. Pepper's Lonely Hearts Club Band.) But there's so much more about Pet Sounds than just a little friendly competition between two of the era's most forward-thinking artists.

The story of Pet Sounds is one of heartbreak, joy, madness, rock 'n' roll, the past, the future, the beginning of the end of the Beach Boys and a whole lot more. It's the perfect mid-'60s album in ways that go deeper than the music – which is among the most gorgeous and influential ever made – itself. Rubber Soul opened the door to a new period; Pet Sounds was the album that made itself at home and never really left once it got inside.

When Wilson started work on the project in July 1965 (by recording a backing track to "Sloop John B"), he had already began branching out from the Beach Boys' early days as admirers of the barbershop harmonies favored by vocal groups like the Four Freshmen and the commercially successful surf music the Wilson brothers' dad and manager, Murray, encouraged them to record.

Brian's public shyness, fear of flying and panic attacks made the studio a natural environment for him to hide away with his passions. Obsessed with producer Phil Spector's intricately arranged pop songs, he started incorporating more ambition in the Beach Boys' music, like in Top 40 hits "Don't Worry Baby" and "California Girls." By the time The Beach Boys Today! was released in March 1965, Wilson was ready to take his next creative step.

In January 1966, he started dedicating every working minute to Pet Sounds, and for the next three months he labored over the record, laying down track after track, employing tons of studio and session pros (including string and brass sections) and fine-tuning the record that would become his masterpiece and one of the most heralded pop albums ever made. (It was also one of the most expensive at the time, tallying more than $70,000.)

Working with Tony Asher, who wrote many of the songs' lyrics, Wilson took over the construction of the album – producing, writing the music, arranging and orchestrating all of those musicians – while the rest of the Beach Boys were touring without him. (He abandoned the road after a particularly harrowing flight experience in late 1964.) Besides vocals, the group members performed on very few of the album's 13 tracks, one of which – the LP-closing "Caroline, No" – was originally released as Wilson's debut solo single.

As the excellent four-disc 1997 box set The Pet Sounds Sessions documents, Wilson's making of the album went beyond the time, passion and intensity his idol Spector put into his own Wall of Sound records. Arrangements were fleshed out and then scuttled; lead vocals by one Beach Boy were replaced by another member. Wilson worked his bandmates and particularly the session musicians who were bringing his visions to life. He dragged everything from accordion, harpsichord and mandolin to Theremin and ukulele (not to mention all of the strings and horns, as well as less-conventional instruments like bicycle bells and Coke cans) into the studio.

Listen to "God Only Knows" From 'The Pet Sounds Sessions'

But they all followed him. And it all paid off. Pet Sounds jump-started a number of genres, some that didn't even have names on May 16, 1966, when the album was released: art rock, baroque pop, chamber pop, psychedelia. It sounded little like the Beach Boys albums that preceded it, though all signs were pointing to its development and fruition. It's an amazing piece of work. Fifty years after its release, it continues to influence scores of artists and musicians.

Four of its songs reached the singles chart: "Caroline, No," (credited to Wilson, it came out in March and made it to No. 32); "Sloop John B," which climbed to No. 3; "Wouldn't It Be Nice," a No. 8 hit; and "God Only Knows," the flip side of "Wouldn't It Be Nice" that stalled at No. 39. The album itself made it to No. 10, the group's lowest-charting record since the filler-filled Shut Down Volume 2 two years earlier. (It was a bigger success in the U.K., where it peaked at No. 2.)

But its impact was almost immediate (Paul McCartney said there'd be no Sgt. Pepper without it) and its legacy has only grown since its release. Even the album's most fully formed outtake, "Hang On to Your Ego," an early take on the album track "I Know There's an Answer," is considered a classic these days. Pet Sounds unwinds unlike so many other records of its time. In a way, it's one of pop music's first concept albums, a song cycle that amounted to "a teenage symphony to God," to borrow the phrase Wilson later used to describe his next project, the troubled and ultimately shelved Smile.

The record eventually tore apart the Beach Boys. Wilson's increasingly erratic behavior during the album's formation (which only intensified afterward, leading to his well-documented breakdown and the Smile debacle) and its move away from more chart-friendly material pulled the group in several directions. It was a personal record about alienation, heartbreak, drugs and, most of all, the beauty in sounds. (The album's title is a pun; Wilson's "pet sounds" – as in his favorite noises – include actual animals, like the barking dogs that end the LP.) Even though the preferred mono mix, the way Wilson made it, didn't take advantage of the burgeoning stereo industry, the musical palette was the most expansive of its time ... until Sgt. Pepper came along a year later.

The music is beyond pop; there's jazz, folk, classical, experimental noise, exotica. The best songs – and really, the entire 36 minutes should be experienced in one sitting – point to Wilson's brave new future. The groundbreaking "Good Vibrations" was started during the sessions and completed during the making of Smile, the direct link between the two celebrated projects. From there, it was a wide open world that Wilson was just starting to explore when he mentally collapsed, stopping his progression for years, even decades. The breathtaking beauty of Pet Sounds stands as his hard-earned legacy. It was a lot to live up to. Then and now.

See the Top Rock Albums of 1966