Can Kaio-Ken be used with Ultra Instinct? Yes it can be done if they're Mastered. No,UI Calmness and Kaio-Ken Rage can't mix.
The highest multiplication for the Kaio-ken ever seen was the Kaio-ken x100, which Goku used against Lord Slug.
It's vastly weaker than Ultra Instinct. Blue Kaio-Ken was weaker than Jiren by a fair margin, Jiren literally blinked SSJBKKX20 away: Jiren before breaking his limits was comparable to the GoD's of destruction, In fact he's stronger than his own GoD who's apparently superior to Beerus according to Whis.
In Dragon Ball Z, Goku has the Kaio-ken as one of his special moves, and he is able to use it even in his Super Saiyan form. In Super Dragon Ball Z, Goku is able to use Super Kaio-ken by transforming into a Super Saiyan and then using Kaio-ken.
We know Kaioken was invented by King Kai and goku is the only person to learn/use it. However Jiren is in an entirely different universe so it's possible he learned and honed his own kaioken. It would explain how he's able to attack and move so fast, plus it explains the red aura he displays.
Ultra Instinct like the one you're talking about will never use Kaioken, because Goku's body is already at it's max potential when using it. If he were to add Kaioken, he would likely die immediately.
Kaio-ken - is a fighting technique invented by King Kai. However, Goku and chi chi (not technically kaioken) are the only people ever able to successfully use it.
Super Saiyan Blue Kaio-ken, also known as Super Saiyan God Super Saiyan: Kaio-ken, is a technique first used by Goku during his fight against Hit only in the Dragon Ball Super anime. The technique consists of using the Kaio-ken while transformed as a Super Saiyan Blue.
Kaioken x100 multiplies your level of power by 100, hence the '100x' beside the 'Kaioken'. If a fighter is able to handle it, they will be able to match a Super Saiyan 2, if their base power levels are somewhat equal.
Anyone can learn how to use Kaioken in theory. However, it's a very difficult move to control let alone master. Despite training with King Kai for a long time, Yamcha, Piccolo, and Tien couldn't use Kaio Ken at all. Remember, these guys are some of the toughest fighters from Earth!
It's said by many that Super Saiyan Blue Evolution is not as strong as Ultra Instinct.
A final and most likely scenario is that there's simply nothing after Ultra Instinct. Goku will eventually learn Autonomous Ultra Instinct, and this will be his most powerful form through the rest of Dragon Ball.
The fact that Goku could not defeat Jiren with Super Saiyan Blue combined with the Kaioken technique, but was later able to defeat him using Ultra Instinct proves that Ultra Instinct is vastly more powerful. Don't get too attached, though - it's only a matter of time before Goku tops that as well.
Ultra Instinct is taught by Whis during his training with his martial arts students, Beerus, Son Goku, and Vegeta. However, the art is very difficult to master, taking Beerus millions of years to do so but still has not mastered it. The art allows the person to be able to act without thinking.
Xeno Goku can use up to Kaioken x100 which normally would put strains on his body. Range: Extended melee range. Multiversal+ with ki blasts and attacks.
One way to put it is that the False Super Saiyan transformation occurs when a Saiyan has the required physical power but lacks the required mental strife to transform into a Super Saiyan.
The highest multiplication used was X20. Both instances was against Freeza, which inevitably was the final time Kaio-ken was used in battle. Uniquely in Universe, Goku's maximum multiplication is just X10. At age 17, Uub was at the same level as Goku where he can use it without any visible changes and too use the X20.
"Kaioken and Kaioken x2 are the exact same, Toriayama just retroactive changed base Kaioken's name to "Kaioken x2" in that chapter, to avoid confusion, and emphasize that x3 and x4 were multipliers." "Kaioken x2 does not exist. It's used by Goku one time in one chapter, and then never seen or used again.
Despite the fact that it is also the most controversial Dragon Ball series and that SS4 is not even canon anymore, the power-up was so different and had such an amazing design that fans keep loving it. Super Saiyan 4 still shows up in the games, fan-fiction, and non-canon stories.
The power increase in Omen is far beyond that of Super Saiyan Blue: roughly equal to Kefla's Super Saiyan 2 state and allowing him to initially fight evenly with Jiren. Whis describes the power of this omen as the "limit-breaking power of the gods" (限界越えるネ申の力, genkai koeru kami no chikara).
[Meaning in Dragon Ball] The Kaio-ken technique was taught to Goku by Kaio North to defend Earth from the invading Vegeta and Nappa. By controlling all of the Ki in their body and temporarily amplifying it, the user can drastically increase their power, speed, destructive capabilities, defensive abilities, and more!
It's possible King Kai didn't think the Earthlings with their weaker physiology could handle such a demanding technique -- or more likely, King Kai saw the disastrous effects that the Kaio-ken had on Goku's body during his fight with Vegeta and decided to shelve the technique altogether for its potential for danger.
Reason being, Goku went Super Saiyan Kaioken in the Babidi saga. That was definitely anime only and was non-canon, and should be considered non-canon.