Me… I don’t really understand what @troy_s is writing about. 
Anyway, I just made a Pull Request.
However I DISABLED default HCY blending for this. To test it out you need to (un)comment those lines starting at 733 (or reverse the latest commit). But I hope this will make it easier for other people to contribute. A GUI is needed primarily.
^ Basically that the standard 0…255 “sRGB” colorspace is not mathematically linear and ideally should not be treated as linear. For example, ‘mid-grey’ is not 128,128,128 in sRGB, but 186,186,186.Supposing you paint black at 50% opacity onto white, you should get the latter color, not the former. Or red onto cyan, etc.
The 2 images in the introduction section here:
Provide some good examples of the visual results of that.
This affects all dab-drawing code, because the colorspace that MyPaint uses is a scaled up version of sRGB (with values 0…32767 rather than 0…255). You are calculating the color changes via HCY, which IIRC is correctly linearized, but the actual dabs applied are sRGB values mixed via sRGB colorspace
Happily explain it, with imager friendly demos. As stated, try painting a purely saturated red fuzzy brush atop of cyan, all at full intensity.
See the nasty soot dark fringe? Welcome to nonlinear colour blending. Now extend that to every operation from blurs, to smudges, to overlays, to you name it.
When performing math on light values, nonlinear reference spaces are inherently broken.
Ah thank you, I see it! (and now I can’t unsee it 
). At least with the HCY blending it is less obvious at the beginning of strokes, since I go round the Hue circle first. But when red is really next to cyan it shows up…
Anyway, I don’t think I can do something against it. 
Yeah, it’s beyond the scope of your patch, don’t worry about it too much. Really a little off topic IMO, since your patches are entirely independent of the underlying image representation – it’s just if we end up changing that representation in the future, then the results should automatically become generally nicer.
Hi, I’m not an expert on this subject nor in programming, but perhaps you guys may want to check out a youtube video from “minutephysics”, “Computer Color is Broken”:
It focuses on the reasons and possible fix for this problem of greying/darking of color blendings.
Finally got around to trying this. The screenshots are hard to appreciate, but trying it out I have to say it is very fun and refreshing! I have a couple questions:
Won’t using any brush settings that affect transparency introduce the RGB blending effects that this is trying to avoid? So to get the best “real pigment” blending we should disable radius-by-random, pixel feather, hardness, eraser, opacity, and any other settings that might create transparent dabs?
My other question is probably (very likely) naive, but I was thinking about the saturation problem you mentioned. If you convert to RYB space couldn’t you do a straight-line to determine the proper chroma and hue of the dab? Ignoring the tinting strengths of real pigments, of course, unless we want to reference a table of every hue and its strength as part of the formula? Yellow is weak, Blue is strong, etc.
Handprint talks about this problem quite a bit.
There’s some info on converting to RYB and back to RGB here:
http://nishitalab.org/user/UEI/publication/Sugita_IWAIT2015.pdf
If using RYB wouldn’t that solve all the “wrong way” CW vs CCW issues? Red and Cyan and Blue/Yellow are not anywhere near 180 on the RYB wheel, more like 120. Likewise, if we use RYB the 180 degree complements should all mix toward a grey-- that’s what we want if we are talking about pigment/real mixing, right? Red+Green == Grey. Purple+Yellow == Grey, etc?. Of course if your two opposite colors are of different Chroma then the result should lean towards one or the other.
Then again, maybe we don’t like grey at all. I do find it fun that if painting an intense yellow over a dull blue I will get intense greens popping out here and there unexpectedly. It’s not necessarily “realistic” but maybe that’s not the goal after all. . .
Did I mention this was really fun? Here’s a video demoing the rake brush with smudge settings. Skip to here:
to see what seems to be a bug in the blending code. More fun:
This brush is basically a stair-stepping pattern on the Custom input/random. Then you can apply the custom input to both Offset Mirrored and Smudge. This way each “bristle” can have a different smudge factor so it seems like there is more paint on in the inner bristles than the outer bristles.
It really does look fun. I have to try this myself one of these days.
I’d also think that the mixed color should be on the straight line. I’ve been looking into Gurney’s gamut masking stuff recently, and everything seems to fit best with straight lines? I.e. how mixing a palette from any three colours as primaries results in a triangular gamut mask.
I think so, straight lines within the RYB color wheel though. MyPaint has an RYB wheel (edit-preferences-color-wheel-traditional). I switched to the RYB wheel and created HCY masks to demonstrate how RYB mixing might look. Yes it is muddy, but I started with fairly desaturated colors and I think this is really how it should look if they are pigments. I had to pluck the RYB colors from the mask manually so it is approximately evenly-spaced along the masks. The last HCY row I nudged the values of the yellow and blue a tiny bit so that the hue rotation went “the other way”. So, I don’t really understand anything BUT maybe @AnTi code can be tweaked to somehow just use the RYB wheel’s hue values? Does it make sense to say HCY space with RYB primaries?
I deleted the previous post; that research paper seemed to have a mistake: Cyan didn’t seem to be reversible from RYB back to RGB. Maybe I’m just terrible at math. Well, I made another branch using a different formula (http://www.deathbysoftware.com/colors/index.html) and it seems to work really well. I also added a step to desaturate colors more and more depending on how different they are on the color wheel. Otherwise, you could mix 100% Blue with 100% Yellow and get a 100% Green-- which isn’t how real pigments work (otherwise they’d only sell 3 pigments in the art stores).
Here’s a link to the branch
RGB:
RYB:
RGB:
RYB:
Here’s a really long youtube demo
@AnTi pointed out that there is too much desaturation with the yellow/magenta. I think I need to tone down the desaturation step, but I really need to also convert the color hue angles to RYB before the angle comparison.
So I think I will copy over the distort_hue stuff from MyPaint (gui) adjbases.py
_HUE_DISTORTION_TABLES = {
# {"PREFS_KEY_WHEEL_TYPE-name": table-of-ranges}
"rgb": None,
"ryb": [
((0.0, 1 / 6.0), (0.0, 1 / 3.0)), # red -> yellow
((1 / 6.0, 1 / 3.0), (1 / 3.0, 1 / 2.0)), # yellow -> green
((1 / 3.0, 2 / 3.0), (1 / 2.0, 2 / 3.0)), # green -> blue
],
"rygb": [
((0.0, 1 / 6.0), (0.0, 0.25)), # red -> yellow
((1 / 6.0, 1 / 3.0), (0.25, 0.5)), # yellow -> green
((1 / 3.0, 2 / 3.0), (0.5, 0.75)), # green -> blue
((2 / 3.0, 1.0), (0.75, 1.0)), # blue -> red
],
}
def distort_hue(self, h):
"""Distorts a hue from RGB-wheel angles to the current wheel type's.
"""
if self._hue_distorts is None:
return h
h %= 1.0
for rgb_wheel_range, distorted_wheel_range in self._hue_distorts:
in0, in1 = rgb_wheel_range
out0, out1 = distorted_wheel_range
if h > in0 and h <= in1:
h -= in0
h *= (out1 - out0) / (in1 - in0)
h += out0
break
return h
Well, this is a bit depressing. It’s not my extra desaturation step at all that’s causing the magenta/yellow issue. I think it’s actually “working” fine. Yellow+Magenta should make a muddy grey- in the RYB model. I think this is because (a fully saturated) Magenta doesn’t really exist in RYB and is just an artifact of mapping/cramming ALL of RGB into RYB values instead of starting off with a limited gamut of only “possible” RYB values.
See the MyPaint RYB wheel which has magenta and yellow opposite each other. From my understanding of color theory, any two colors opposite of each other on the color wheel should produce a grey when mixed.
Ultimately maybe the best solution is hinted at in this thread by Scott Burns:
There is not enough information provided by RGB values alone to perform a true Kubelka-Munk computation, as you need both absorbance and scattering curves across the visible spectrum. Instead, you could generate representative reflectance curves from RGB values, and then use the reflectance information to perform the subtractive mixture, for example, by computing the weighted geometric mean of the two reflectance curves.
Scott Burns goes on in quite detail on how to get from RGB to “real pigment”:
http://scottburns.us/subtractive-color-mixture/
and
http://scottburns.us/reflectance-curves-from-srgb/
It seems possible, but very complicated and possibly too computationally expensive to be useful. Although he does provide MatLab code. If we could pull it off I think we’d have something very special. Any takers?
CMYK is actually going to be just as weird. It’ll be hard to get B+Y=green (You’ll have to use C+Y) and it messes up red and cyan mix by making them complements (you’ll get grey). This is literally a game of color wack-a-mole! 
So, there might be a compromise, or maybe a solution depending on your perspective. The problem with RYB (or any other scheme besides RGB) is that you have the opportunity to pick colors that don’t make sense, that are outside the gamut of the primaries. So, if you created a gamut mask and apply that to the HCY wheel, you might have a pretty decent workflow for using RYB and natural pigment blending. Yes, you’ll be limited in colors but that’s not necessarily a terrible thing.
The steps to create the gamut mask are:
Hope that helps and sheds some light (light? no, not light!) on this color business. Still much to learn. . .
LAB has this property (but in relation to LCH hue term, not HCY hue term). LAB and LCH are more expensive than HCY (but less than Kubelka-Monk – IMO KM is just not a thing we can do here)
Kubelka-Monk ‘colorspace’ used to be supported by Krita, but bit-rotted and was removed. It was also the case that you couldn’t throw arbitrary colors down, they had to be strictly colors from a limited set (or mixes of them…).
More generally, what is going on with your RYB idea? Is it turning out worse or better overall than HCY? (I think we should prefer a visually consistent algorithm that doesn’t achieve B+Y=G to a less consistent algorithm that does)
Oh wow, I saw a video of Krita using that, but didn’t realize it was removed. I wasn’t very excited since it seemed to be in its own little window (like a palette), and in the video I saw blue+yellow made a muted yellow instead of green! Which, come to think of it. . .makes perfect sense for CMYK. Did you see my links to the work by Scott Burns? He has a method to map virtually every single RGB color to a spectral “pigment”. It sounds perfectly possible if we had enough CPU
The RYB patch is getting pretty interesting, I think. I’ve fixed a few more errors and the color blending seems to be very consistent.
@AnTi already mentioned a few issues with HCY that I’m not sure how to fix; there is very little desaturation of mixed pigments, and there are some odd blending intervals caused by the shifting of the hues- see image below.
That made me think that maybe the main issue was trying to contort the RGB color wheel into an RYB color wheel while still using RGB as the primary colors. So RYB solves that issue and it has desaturation, but it still retains all the other problems, which are pretty tremendous. Namely, everything else about MyPaint is RGB, particularly anything to do with opacity. So for the best RYB results you need to work on one layer, set brush opacity to 1, set hardness to 1, avoid use of Radius_by_random, and set pixel feathering to 0. All of those things introduce RGB blending and detracts from the RYB blending. It’s not terrible to leave them on but it can be noticeable.
I added a setting to control the way the desaturation works-- either accelerating additional desaturation or even reversing the process:
Anyway, I definitely feel RGB smudge is here to stay, but pigment blending might a place too for “painterly” effects. @achadwick had the good idea to make it a slider so you can switch from one way to the other so it’s not a major change to MyPaint.
@briend, @achadwick pushed in a change that address the saturation issues libmypaint had with grays. Probably want to rebase your PR and see how it affects your code.
Thanks @odysseywestra, it looks good and I rebased without any merge conflicts or anything.
I added yet another setting, mostly just to help test different modes of mixing the paints. I wanted to try HCY blending, too, so I copied the functions from @AnTi’s branch and made the mode selectable with the Smudge RYB Mode setting. Ranges 0- <1 use just RYB for the mix without messing with saturation or luma. 1-<2 uses HCY, 2-<3 uses HSL, and 3-4 uses HSV. These latter three modes use RYB for just the hue, and then mixes the saturation (unless sat=0) and luma with the selected mode. I rather like the plain RYB mix, but really there are things I like about all of them so it’s hard to pick a favorite.
This image shows some mixes with the 4 modes across 3 ryb smudge sat settings (0, .5, 2) and finally just rgb for comparison. The ryb columns in each row should be identical since that mode doesn’t use the sat setting. Its just pure RYB mixing.
There are interesting differences between each kind of mix. Notice how HSV goes towards white when mixing luma and HSL goes towards dark?
I’m changed the whole RYB Smudge setting to a generic Smudge Mix Mode setting and abandon the crossfade in the process (blending RYB / RGB is generally pretty ugly). So basically you’d have:
Smudge Mix Mode: Range 0-1: RGB, 1-2: RYB, 2-3: CMYK, 3-4: (new fancy Spectral mode by Scott Burns), etc
Smudge Adjustment Mode: 0-1: Native (no adjustment), 1-2:HCY, 2-3: HSL, 3-4: HSV, 4-5: LAB, etc
Smudge Desaturation: -10 to +10. Assuming you picked an adjustment mode, the saturation is decreased or increased based on hue angle difference
Only RGB and RYB are implemented at the moment, for mix modes (I want to add CMYK soon and Spectral… maybe never). For Adjustment modes HCY, HSL, and HSV are working.
Ok some pretty interesting updates. I learned a bit more about RGB and implemented that method of mixing that does the square root stuff so that red and green make orange. I have this implemented with the Smudge Linear mode selector setting. So you can choose a plethora of options now. 5 settings to switch from linear/non-linear, RGB and RYB models, adjustments with HCY/HSL/HSV, that Desaturation thing, and finally additive vs subtractive (not tested yet). In case you’re wondering where CMYK went, it dawned on me that CMY was just RGB flipped around (1-color). So, there’s not point to implement a CMY model; just make a selector for additive vs subtractive.
edit “SUB” below is really additive just the using the squared root blending (the 2nd additive model- Oh, it’s called linear mode?). I think the blending is much nicer with this mode for both RYB and RGB. You don’t get that weird darkening effect.
“Subtractive” blending is kinda impossible I think, without Scott Burns spectral method. Reason is, everything turns to black (cue Rolling Stones or Pearl Jam). When you multiply (which is how to do subtractive blending) Red * Green, or Red * Blue, etc etc, you get black. It’s really annoying. So, I think I’ll try the “merge grain” formula which is A+B - 0.5. Still additive, but subtracts 50% grey from the mix… Probably just have to figure out a way to “fake it”. edit no such luck. Subractive mode will have to be a slider between Additive and the Scott Burns subtractive method, I think.
Well for Subtractive mode, I wasn’t using the Weighted Geometric Mean that Scott Burns talks about. That seems to fix most of the issues it, and it can be pretty neat.
The image below shows the different blend modes. THere is a column for Additive and a column for Subtractive. Then there is a row for RGB primaries and a row for RYB primaries. For each cell there are four blends using RYB, RGB, OGM, and CMY colors.
Here’s a demo of the subtractive mode along with the desaturate and darken modes that operate on the hue angle difference:
