Color Management And Linear Light

Mypaint already has a wiki covering Linear Light and Color Management:

There is a great GIMP thread here that really shines a light on all of the issues with keeping backwards compatibility and appearances. MyPaint should listen carefully and consider the longer picture and go with a simpler approach:

1 Like

I think anyone interested should watch this presentation and try to grasp the concepts. hit present to have animations:



Scene Linear might be really awesome. You could work on a painting that could be displayed dynamically to allow the viewer to “explore” your painting. That is, normally information is lost in the shadows and highlights. Your workflow could be to do your painting in the “normal range”. Then, adjust the exposure either up or down and add more details to the shadows and highlight areas.

If we implemented OCIO and Linear reference we could have a very flexible and modern painting tool.
More thoughts

OpenColorIO doesn’t appear to handle ICC profiles at the moment, but it is on the roadmap for 2.0. However, ICC profiles aren’t anywhere near as good as 3D LUTs (I don’t quite understand why, but look it up)

So, if I understand (not that likely) one would need a 3D LUT of their monitor to get end-to-end color management.

More background:

So here’s a (naive) gameplan:

  1. standardize on sRGB Rec 709 for the internal reference, (leave it non-linear for now)
  2. implement OCIO to transform from this space to your display device using your 3DLUT for your monitor (if you don’t have a LUT just pass through and assume your monitor is Rec 709 sRGB)
  3. THEN go through and remove all the non-linear code or make non-linear an option with linear the default.
  4. Build GUI similar to Krita and Blender’s to adjust these settings such as display device and config, exposure, etc.
  5. Figure out what to do with the color picker GUIs which are built with Cairo (which doesn’t do CM). Maybe we can read the cairo widgets into a pixbuf and blast through OCIO? Gdk.pixbuf_get_from_surface?

Well here’s my attempt so far to modify the pixbuf of a cairo surface. Haven’t had a lot of success other than being able to fill the pixbuf with any color I want. Here’s a workflow by @troy_s (who’s an invaluable resource!) to help setup OCIO (3d lut, etc)

So here I’m affecting every GUI interface color picker thing, which is pretty nifty. For the actual drawing surface a different technique will be needed, I believe (it might actually be a lot easier). Also, I’m re-creating a new “processor” for every single call, I think this is totally wrong. I should be reusing the processor and keeping it around, so I’ll have to figure out how to do that.

Ok, this is getting much better. Almost all the gui picker things are now color managed by OCIO. In the below screenshot you can see all the color thingies on the sides have slightly “duller” colors than those on the canvas. Go ahead, use the color picker and check the saturation and hue angles. The ones “on canvas” are perfect sRGB colors and the colors on the side panels have been corrected for an “Adobe RGB” monitor.

When I use MyPaint on my Adobe RGB monitor, the colors on the side panels look like the colors you see on canvas here. Likewise, the colors on the canvas are ridiculously saturated and hue-shifted. Next step is to get the actual canvas color-corrected, oh, and the palette in the lower right there is not color corrected yet.

Ok I’m getting a bit farther, Palette is corrected now (still a few more guis to fix). I’ve defined 4 OCIO processors. These should be able to let us choose an arbitrary internal reference space such as linear or flmic_log, while still using the normal cairo widgets that are sRGB. Whenever a pixel is displayed on the screen it will take one of two paths. Either it will be a cairo/gui pixel and take the sRGB to Display path, or it will be a pixel from the drawing surface which is “in reference”, so that pixel will take the “reference to Display” path.

Before we’re able to use a different reference we’ll have to switch to floating point and remove a lot of stuff that might CLAMP the values, thankfully @maxy has pulled out of the attic some old branches that should really help with this.

But the next step is still to get the actual tiled surface to be processed through the ‘reference to display’ OCIO processor, and at least MyPaint will look good on any monitor (assuming you have a 3DLUT)

Funny, while dealing with a cairo pixelbuffer is a pain in the neck, getting the pallete to be corrected for display was just these two lines:

  •        hi_rgb = app.sRGB_to_Display.applyRGB(hi_col.get_rgb())
  •        fill_bg_rgb = app.sRGB_to_Display.applyRGB(col.get_rgb())

a note on performance: If you have just one color gui thingy I can’t tell if it is any slower, but if you add 3,4 or ALL, it definitely goes slower. However, the color picker doesn’t need to be super high performance, what I’m really worried about is the actual drawing surface… hopefully I’ll know soon how that goes!

Chatting a bit more with @troy_s, I learned that we could support wider gamut like Adobe RGB (or other colorants) pretty easily by switching to Cairo’s RGB30 mode that allows 10 bits per pixel (ints). Since we don’t need an alpha channel for UI elements like color pickers, this should be fine. TODO figure out if cairo is using a straight 2.2 gamma or the actual 2-part. If it is straight 2.2 it will be a bit easier.

Ok I have the drawing surface color-corrected. However at the moment the color picker is sampling the display-referred space instead of the internal reference, resulting in this madness:

This video is actually pretty interesting because you can see exactly how the color management is working (Besides the obvious malfunction). For red, each time I pick the color the new color is a different “red” It’s a bit darker but also the hue angle shifts just a bit. Same with Blue. This is because Adobe RGB is “supposed” to have the exact same colored LEDs for Red and Blue. Obviously my monitor is just a bit off. However, Adobe RGB has a very different green compared to sRGB (rec 709). So, when I color pick my green, each subsequent correct changes the hue angle a LOT. It’s almost turning red by the time I get to the middle. Here is the spread for each RGB:

Red: 0 degrees to 32 degrees
Green: 120 degree to 49 degrees (and I only did a few steps compared to R and B)
Blue: 240 degree to 247 degrees

Performance is really bad when panning, moving layers. The most important thing, drawing, is actually pretty fast and almost not an issue. Right now I’m using Python OCIO and they hint that you should use C++ for high performance applications. MyPaint already uses C++ here and there for pixel operations so I’m going to look at moving it all over to c++ for OCIO stuff.

In the video above, the colors look a dulled because it is meant for an Adobe RGB monitor (MY monitor, specifically). However, when I save the file to disk and open in GIMP, for instance, the color picker returns 255,0,0 for red, 0,255, 0 for green, etc. This is GOOD, it is preserving the internal reference without any display correction.

Here’s a few quick profiling snapshots:

moving layer:

color picking:

panning view:

Ok Panning/zoom/rotate/flip are now 100% speed. I was confusing rendering with caching and had the OCIO transform in the wrong spot. Thanks @achadwick!


@troy_s has been an enormous help sorting this all out. The end goal is to hopefully have the input colors come from 10 bit Cairo RGB non-linear perceptual space (hijacking the 2.2 gamma hopefully) and then transform to float32 numpy for everything in whatever space you want configuring via OCIO (linear or filmic-log probably). All operations internally will just work with these floats and not do any conversions at all. Finally, just before rendering to screen the numpy array will be transformed for the display device, converted to 8bit int and stuffed back in the mypaint surface/gdkpixbuf and then blitted

God I hope I’ve got that all right.

This might be interesting-- float32 is way faster than float16 numpy:

So for 2 days I didn’t even try flipping my setting in my OCIO config from Reference: SRGB OETF to Reference: Linear. Just flipping that switch is enough for colors to start blending like this:

Instead of this nastiness :slight_smile:

Oh, I didn’t even notice this… but look at the GUI elements. OCIO is now managing the top bar and the icons. With the sRGB reference it looks washed out, while the linear reference (top) it looks normal (actually maybe normal should be washed-out). I guess this is expected when you literally jam OCIO into every place you see pixel buffers :slight_smile:

I started replacing 8bit numpy arrays with 32bit numpy. I resurrected Jon Nordby’s floating point libmypaint branch, too.
Even with OCIO active, performance seems to be actually much better than normal mypaint, which I suppose is partly because numpy32 is faster, but also there is less conversions going on, like all the fix15… Clearly, color for the tiles is horribly broken but at least it is compiling and running!

Here is the required OCIO libmypaint branch (really nothing to do with OCIO just keeping the name consistent):
and the mypaint branch:

Memory usage for one layer is about 150MB for a 1080P screen. zoomed out to 50% (4X more area) and filling in the entire screen with a big air brush, the memory is 500MB. So, really not that bad at all IMO. I created 10 more layers and did some painting on each and my memory is at 1.5GB. I did basically the same thing in Krita (2000x4000 32bit) and created 10 more layers and it was about the same memory usage, so at least that much makes sense.

My assumption is that the color data is being skewed by some horrible incorrect data mismatch going from 8bit to 32bit, so hopefully the performance is not going to vanish as soon as I fix that.

So I was a little bit too eager to replace every uint8 with a float, when GDK buffer definitely needs to remain 8 bit. So I undid a lot of stuff and finally got something that actually doesn’t look broken until closer examination:

I’m not really doing anything to convert from 8bit int 0-255 and floats, which might explain this. It looks pretty neat but not useful yet. I’m hesitant to clamp the float into values 0-1, since I’m looking to implement scene-referred painting. If I can ever get my head wrapped around it. . . .

Update, I’ve got the background colors handled, I think, and alpha is working as expected. I’m constraining alpha to 0-1.0, which makes a lot of sense. There’s no reason to allow that to be bigger. However, I’m feeding bad values into libmypaint, I think, which expects RGB to be floats between 0-1. So, right now I have no color data but it’s looking nice:

More progress! Fixed the color issue, and got the 32bit float numpy tiles “managed” by OCIO, and flipped my OCIO config to Linear Reference to do a blend test. This is with the “noise” disabled for now, so there is some banding.

More progress. The internal model is almost completely transitioned to linear light and associated alpha (premultiplied color). Here in this video you can see how you can build up light intensity beyond what the display can show (without adjusting exposure, still need to add that OCIO controller). Then you can add opaque “darkness” back to bring back the original lines (since they were there all the time just brighter than the white).

Painting with a “zero opacity” or zero alpha brush is not intuitive at first glance, so I’m looking at either adding an addition control (alpha) or possibly changing the default blend mode to Multiply, and make this current “normal” mode into a new option like “additive mode”.

From the OpenEXR technical introduction:

Calling the color channels “premultiplied” does not mean that the color values in an image have actually been multiplied by alpha at some point during the creation of the image, or that pixels with zero alpha and non-zero color channels are illegal. Non-zero color with zero alpha is legal; such a pixel represents an object that emits light even though it is completely transparent, for example, a candle flame or a lens flare.

1 Like

Here’s a better demo, now I have an “Alpha” slider. This is just a boolean toggle to make the Opacity slider adjust ONLY the alpha channel (thanks @troy_s for that tip!). This lets you paint normally, or flip it to paint with control of the alpha (occlusion). So, if you want to paint a flame or something that isn’t blocking light, you flip that alpha switch and then you can reduce opacity to zero and paint with light.

Also, you can go back and “erase the occlusion” from things on canvas already.

and a continuation:

1 Like

I’ve changed the setting to “Occlusion” (for now, for lack of a better word) and made a brush that gradually becomes transparent (loses occlusion) towards the end of the stroke. Once it is backlit you can see the difference between Opacity and Occlusion:

How do you convert from 10 bit Cairo RGB to float32 numpy? Maybe you have some fast trick using numpy.right_shift() or something. Here’s my solution, possibly I confused the order of colors, maybe your solution is faster or cleaner:

surface = cairo.ImageSurface(cairo.FORMAT_RGB30, width, height)
buf = surface.get_data()
bgr = np.ndarray(shape=(height, width, 1), dtype=np.uint32, buffer=buf)
bgr_bg_b = np.right_shift(bgr, (((0,10,20),),))
bgr_bg0_b00 = np.left_shift(bgr_bg_b, (((0,10,20),),))
r_g_b = bgr_bg0_b00
r_g_b[:,:,0:2] -= bgr_bg0_b00[:,:,1:3]
r_g_b = r_g_b/2**30
r_g_b *=255
r_g_b = r_g_b.astype(np.uint8)
cv2.imwrite("r_g_b.png", r_g_b)

What do you mean by “Cairo RGB non-linear perceptual space”? I thought that Cairo can’t correctly handle non-linear/perceptual color spaces such as sRGB.

Thank you!

Probably better than subtracting shifted versions is to use bitwise_and.

The question about non-linear/perceptual color spaces remains.

I didn’t understand anything, but it’s very interesting