Green gem’s argument is obvious. Approaching a comfortable critical threshold is personal feel, and as you approach the highest levels, it becomes tougher to reach the theoretical crit cap for the highest level monsters’ critical resistances.
Red gem is for those who won’t crit as often. It’ll help push past physical defense (with ease, as monsters don’t have high physical resist to begin with). It’s also universal, applies to both physical and magical, applies to both primary and secondary damages, and is multiplied by critical hits. It’s almost always safe to choose red.
Yellow is for those who crit often. The nice thing about critical hits are the damage is added after defenses are factored, so you’ll always receive the full amount in damage formulas. The amounts of course are still augmented by your typical modifiers with the exception of T0 modifiers like Split Shot (as in, critical damage is nigh useless for builds centered around these skills).