Why is the Soft Machine missing in the Top 100?

Pedro! Don't you know? Though music was devised to express one's self in one's leisure time, as a trade, it eventually (or quickly?) evolved to garner attention, notoriety, and, eventually, money (or food and favor). In order to garner attention, notoriety, and favor/food/money, musically-expressive men found that if they created riffs, ditties, or "songs" that people would/could recognize that their esteem and, naturally, prosperity increased. Sure, the jazz free-form expressiveness was a part of jazz, but a very small part. As you yourself said, they always started "together" with a common thread or theme and then came back to that common thread at the end, otherwise they would loose their audience (and means to life-sustaining goods) (unless, of course, the audience was made up solely of either other musicians or aristocratic dilettantes with lots of leisure and disposable property [food, money, etc.]). Audiences move toward an artist because of the familiar emotion-provoking themes that these artists have created--be it one memorable "hit" or a particular sound (like the unique and refreshing sound of early Soft Machine). The improvisational bits that artists are able to squeeze into their live performances are only permitted/tolerated in proportion to the esteem that the audience member(s) hold for that artist. A person dragged along to a concert by a friend that really admires the band they're going to see might be very bored or wholly sucked into what they see and hear--there is no accounting for individual tastes. I see that more prog fans remain "loyal" to their old heroes and those old reproductions of music than to newer, lesser known bands. sure, we all want to see/hear the "hot" new sensation, but we are still being mostly informed by our cumulative "histories" with other bands, other musics, other stories, our own stories. The people who have heard The Soft Machine's albums are undoubtedly fewer in number than those who have heard Yes, Genesis, Steven Wilson, etc. I have a feeling that smaller percentages of people that have heard Soft Machine are won over and sucked into fanship than those who hear the more familiar sounds, subjects, and structures of Yes, Genesis, and Steven Wilson--thus, the lower ratings to support Soft Machine's albums into places among the Top 100. 

Don't get me wrong: I love to be blown away by the technical and lyrical wizardry of improvising musicians while they're performing live, but I like to have the familiar framework (e.g. a rendition from a studio recording) as it helps me to better appreciate the creativity of the instrumentalist, but 100% free form noodling? I have to be in the right mood--prepared and receptive.

     I once attended a Utopia concert in a hockey arena in which Starcastle opened (impressively) before Todd and his quartet took the small stage. They were dressed in white undershirts and black pants with no stage decor; everything was black and white and grey, the stage and show was absolutely void of color. They played only songs from their new release Oops! Wrong Planet which was so new that not many people had a chance to get to know the music. Then they left. The lights came up and easily 3800 of the approximately 4000 audience members in attendance left--some quietly, others booing. My roommate and I stayed, in disbelief that what we had just seen was it--was all that Todd had for us. After about 20 minutes, or faith and patience was rewarded as suddenly a curtain behind the simple little stage that we had been standing in front of was lifted to reveal a huge golden pyramid topped with a gold Tutankhamen mask framed by an even larger 40-foot pyramid frame in the back of a massive stage. The band came out dressed in wild psychedelic/Egyptian-themed regalia, proceeded to open with Todd saying, "It's nice to see a few Utopians remain in the fold" or something to that effect before launching into the guitar theme and vocal section the "Utopia Theme" and then playing virtually every song from "Ra" and part of "The Ikon." 

     My point is, the familiar stuff is what brought many people, not hearing it, they left. Those of us who were more deeply invested in Todd--"fans"--who had developed more trust and faith in his decision-making--were able to be more patient and trusting of the artist's process (for which we were amply rewarded). Sometimes it's just to be in a famed artist's presence, sometimes it's to really tune in and revel in the craftsmanship, sometimes it's just to hear and see the familiar (or to compare to that which you already know). Despite the Soft Machine's talent, I doubt that many people came to those shows on the 1968 U.S. tour with Hendrix that were there for the Brits (though, hopefully, some became converts and "fans" because of it). My dad used to go with his college buddies to a club in Georgetown that had become Louis Armstrong's "home" venue in the 1950s--despite the fact that Louis's lip cancer prevented him from playing much (or any) trumpet. (He sang and led the band.) My dad went cuz he loved jazz (esp. drumming), loved drinking with his buddies, and because it was "Louis Armstrong." Louis performed cuz that was his life, his passion, and that's how he earned the money to pay his bills. I have a feeling that his audience came to expect "songs" that the Maestro was known for as well as to see some of the band's creative artistry. "Songs" serve their purpose.

but your talking about Black Sabbath and Led Zep . They could get away with anything back in the day but that also flows through to the present.IMO