We see in the Harry Potter universe, wizards or witches keep obliviating Muggles/No-Majs, because otherwise it may lead to chaos or persecution of witches/wizards. However, the Dursley family cannot be obliviated since they have to give Lily's protection to Harry.
They were escorted to an undisclosed location. J.K. Rowling has confirmed that: "Harry and Dudley would still see each other enough to be on Christmas-card terms, but they would visit more out of a sense of duty and sit in silence so that their children could see their cousins."
Dudley's last line is also found in a deleted scene.
His last words in the movie are, "I still don't understand why we have to leave." But he also has a deleted film scene where he has a final conversation with Harry, and his last line is, "I don't think you're a waste of space."
Simply put, Harry couldn't be harmed by the Dark Lord at the Dursley household because it was the place where his mother's blood dwelled. Chapter 37 of Harry Potter and the Order of the Phoenix reveals Dumbledore himself explained what he'd done in the letter that he left with Harry on Petunia's doorstep.
Vernon, Petunia, and Dudley Dursley were forced from their home to be placed under the protection of the Order of the Phoenix due to the threat of Lord Voldemort and the Death Eaters. Harry's cousin Dudley was the only member of the Dursley family to show any interest in reconciling with him.
The fan asked the author if Aunt Petunia was a Squib (a non-magical person who is born to at least one magical parent) and Rowling replied: "Good question. No, she is not, she is not a Squib. She is a Muggle.
Figg is a Squib and a member of the Order of the Phoenix. Apparently, she was stationed to live near Harry in order to keep an eye on him and the Dursleys.
'Remember my last'
Petunia's disdain for Harry was most likely born from a long-standing jealousy of her magical sister, Lily. A Howler from Dumbledore, addressed to Petunia, said 'Remember my last' – no doubt referring to the letter which he left with baby Harry, on the doorstep of Privet Drive.
Tragically, the last time Petunia ever heard from Lily was an announcement of Harry's birth (which Petunia threw away). Rowling also explained that Vernon hated Harry for the same reason Severus Snape pretended to: He looked too much like his dad.
She made a promise to Dumbledore
As Lily's last remaining blood relative, if Petunia were to take Harry in it would seal the enchantment and provide him with protection from Voldemort. This would last as long as he could call the place where his mother's blood existed his home.
While Vernon consistently praised his son as if he was extremely masculine and robust, Petunia infantilised Dudley, calling him "Diddykins" well in to his teen years, appearing heartbroken to have to put Dudley on a much-needed diet and even bursting into tears when Dudley showed marginal affection for Harry.
The aftermath of this saw Dudley change to an extent; he became less belligerent and even civil to Harry. The Dementor attack was the catalyst for a change in Dudley's personality that started in Order of the Phoenix, and we saw the end result in Harry Potter and the Deathly Hallows.
Petunia never showed any love and affection at all towards Harry because he reminded her of Lily and the wizarding world. She treated him as a waste of space, giving him the cupboard under the stairs at first and making him do all the chores. She, along with Vernon, ignored most of Harry's birthdays.
Dear Petunia, It is with great sadness that I must inform you of the death of your sister and her husband. They were murdered by a Dark wizard known as Lord Voldemort, whose cruelty and violence they had been fighting for years. I tried my best to protect them, but, alas, I regret to say that I failed.
One of the most beautiful moments in the entire Potter series came from when Dudley and Harry were able to make amends. This occurred in Harry Potter and the Deathly Hallows when Dudley told Harry that he didn't think he was a waste of space. They even shook hands!
Harry lost his temper and unknowingly used an Inflating Charm on her. Aunt Marge then blew up like a balloon, floated to the ceiling, then outside to the open sky. The Ministry of Magic eventually sent the Accidental Magic Reversal Squad to retrieve and puncture her before wiping her memory of the event.
As an unqualified wizard who didn't finish his education, Hagrid was not technically allowed to perform magic.
He defended Harry because he feared the emotional repercussions that he would impose upon himself if he didn't; the Dursleys defended him because they feared retribution.
IN: Waitress Flirting
Not so in the movie, for the Dursleys are cut from Half-Blood Prince entirely. According to Yates, his decision to drop them from this movie had everything to do with their comic relief stigma.
From sisters to strangers
No doubt Petunia would have been saddened by the news of Lily's death but those complicated emotions – grief, regret and unresolved jealousy – were probably buried in the distractions of housework, gossip and pandering to her son Dudley's every whim.
Petunia wanted to be a witch: Rowling reveals that, “Petunia had some latent feelings of guilt about the way she had cut Lily (whom she knew, in her secret heart, had always loved her) out of her life.” She also says that Petunia always held out hope that she would show signs of magic “and be spirited off to Hogwarts.”
Petunia learned that Lily died by a note and a baby on her doorstep. How incredibly cruel of Dumbledore. The sisters may have been estranged, but they were still sisters. Dumbledore also dumped a baby on her without asking if it was okay, and then seemed to blackmail her into keeping Harry.
Though nominally a Muggle-born witch, Hermione is descended from the Dagworth-Granger family through her father's squib heritage.
Albus Severus is the main character in "Harry Potter and the Cursed Child." He's the ugly duckling of the family, is sorted into Slytherin, befriends Scorpius Malfoy, and isn't good at magic. He's nicknamed "Albus Potter, the Slytherin Squib."
During the first books of the series, Neville is portrayed as an incompetent wizard and as poorly skilled at school (with the only exception of Herbology). The character himself reveals in the second book that it was even feared by his family, especially his grandmother, that he was a Squib during his childhood.