Luna was invented after the major Gryffindor characters were established in the 1st and 2nd books. Given how quirky she was it seems unreasonable to expect her to have been a Gryffindor without having been noticed much earlier than she was.
In the books and films, Ron and Ginny Weasley, Neville Longbottom were all assigned to the courageous House of Gryffindor, while Luna Lovegood was sorted to the cerebral Ravenclaw, but the online sorting hat on Pottermore had other ideas for the actors that played them.
She's seen as a bit absent-minded, is easily distracted, and is extremely eccentric in both her disposition and general outward appearance. But that's what makes her a perfect Ravenclaw. She loves learning, but unlike many of her housemates (or Hermione, for that matter), she can never know enough.
She is perceptive
Ravenclaws are perceptive, and Luna noticed all kinds of things others didn't during her time at Hogwarts. There was that time she explained Ron in one sentence: 'He says very funny things sometimes, doesn't he?
And this is why Hermione does not fit into Ravenclaw, since she lacks their creativity of thought. When you also add in her fearlessness and her strong convictions about right and wrong, which are inherently Gryffindor traits, then there was even less of a chance she'd end up wearing blue and bronze.
Harry Potter battled with the Sorting Hat over his desperation not to be placed in Slytherin. Years later, his son Albus arrived at Hogwarts with similar preconceptions. Meanwhile, Neville Longbottom was so intimidated by Gryffindor that he nearly became a Hufflepuff.
Moaning Myrtle
Born in the late 1920s to Muggle parents, Myrtle Warren started her Hogwarts life in the early 1940s, where she was sorted into Ravenclaw. She had no friends, and was bullied relentlessly because of her glasses and acne. But she would soon have all of eternity to get her own back.
And to answer the first question, there is no “best” house outside of personal preference and history within the universe. Gryffindor was where Harry, Ron and Hermione were placed, the brave heroes. Ravenclaws are smart. Slytherins are power-hungry, often evil and sometimes racist.
Luna is asked, "Which came first, the phoenix or the flame?" to which Luna says "I think the answer is that a circle has no beginning." When Amycus Carrow seeks entrance, he is asked, "Where do Vanished objects go?" Amycus is unable to answer, but eventually Professor McGonagall answers with "Into non-being, which is ...
It's not in-canon, but I'm speculating that they originally kept Luna in the basement for ready access to keep Xenophilius under control. After Xenophilius was sent to Azkaban I think they discovered that Luna was helpful in keeping Ollivander alive.
J.K. Rowling did not explicitly write the character of Luna Lovegood as a person with Asperger's. But because she is a good writer, she created a character with a distinct personality that is also familiar in many ways.
After being on the run from a bunch of Snatchers, the gang were taken to Malfoy Manor, where Bellatrix proceeded to torture Hermione using the Cruciatus Curse. It was a testament to Hermione's toughness that she was able to walk away without lasting damage.
Harry describes Umbridge to Sirius Black, saying that she's vile enough to be a Death Eater, despite not being one. Umbridge is also depicted to be a prejudiced person, despising half-breeds and Muggle-born witches and wizards to the point of firing Hagrid due to his half-giant birth status.
After a fellow Slytherin, the next best match for this shrewd house would probably be Ravenclaw. A Slytherin could use Ravenclaw's intellectual and inquisitive nature to their advantage, plan their world domination in minute detail.
Gryffindor's often thought of as the best house, as it's the focus of Harry's time at Hogwarts. Hufflepuff is stereotyped as a house for pushovers and Ravenclaw for bookworms. Finally, Slytherin is thought of as the 'evil' house.
You cannot “get into" a house. The sorting hat will decide it for you, after analysing your skills. If you are asking which house seeks the rarest qualities in students, then it must be Slytherin.
Hufflepuff house is haunted by the Fat Friar, who was executed because senior churchmen grew suspicious of his ability to cure the pox merely by poking peasants with a stick, and his ill-advised habit of pulling rabbits out of the communion cup.
Even J.K. Rowling says that Hufflepuff is her favorite House. It's produced the fewest Dark wizards of all four Houses. Hufflepuffs are down-to-earth. Literally, their House colors represent wheat and dirt, symbolizing their humility (rather than the precious metal secondary colors of the other Houses).
Hufflepuff is an open, egalitarian house, the most inviting of all the houses at Hogwarts. Where Gryffindor, Slytherin and even Ravenclaw strive to make their houses feel like an exclusive club, Hufflepuff has what is essentially a doors open policy.