Advanced Layer Mask Creation in Photoshop

One of my readers, Lee, left a comment on the Dark Architecture post asking how I created the layer mask to separate the sky from the building. Good question Lee, and I didn’t get into it there because it’s a process in itself. I probably spent more time trying to get the mask right on that photo than anything else.

In my post a couple of weeks ago called Digital Photo Editing With Layer Masks in Photoshop, I covered the basics of masking and touched on the advanced stuff. If you haven’t read it, now’s a good time. Now, I’d like to get into the advanced techniques for creating masks right from the photo that’s being processed. I probably won’t cover every single method for doing this, but I’ll at least explain the one that I frequently use.

Why should you learn how to create advanced layer masks? I must say, I don’t use this on every single photo I process, but it’s awfully nice to have the ability when the need arises. I use this technique for photos that need different levels of editing on distinct parts of the image. For instance, you might be toying around with a curves adjustment and find a setting that makes the sky look outstanding. Unfortunately, it makes the other subjects in your photo too contrasty or dark. This is a case where separation of the sky from the rest of the photo would be beneficial.

Mask Process

So you’ve identified the areas you want to separate from each other (1). Before you do this, make sure you’ve done all your touch-ups so you don’t start working with the wrong image. Now convert your image to LAB color (Image >> Mode >> Lab Color). Go into the channels, select the Lightness Channel, and duplicate it to a new document (2) (Right Click >> Duplicate Channel…). The lightness channel is going to contain the best quality information, but you can use one of the RGB/CMYK channels just the same. Now delete the LAB conversion in your history palette of the original image to go back to RGB without doing another conversion.

In your new image, convert from Multichannel to Grayscale (Image >> Mode >> Grayscale). Now you should have one layer and one channel that look exactly the same. From this image, we want to darken the darks and lighten the lights to the point that we have just a black & white stencil of one region in the photo. Cool, now add a levels adjustment layer to the image. You’ll want to bring the black point up (3) and the white point down (4) until you have a mask. It’s okay to have some off-spots in either the white or black region, because you can go back over them with a pencil (5). This particular image required a black point around 147 and a white point around 229, plus a little penciling-in to darken some of the grays inside the building.

This levels adjustment is the most important step in the process. The goal is to bring the black point up just far enough to get a good solid outline around your subject. Same with the white, bring it down just far enough to whiten around the boundaries. If you take either adjustment too far, you’ll end up converting the remaining mid-tones into blacks or whites. This will produce a mask that’s too sharp and either too bulky or skimpy for the original image. This matters because the mask will cause your image adjustments to give off halos if it’s not created just right.

The mask is ready to go back home, but it’s a good idea to keep this document open in case you need to make some adjustments to the mask or the levels adjustment layer. Just duplicate the channel (not layer) back into the original document (Right Click >> Duplicate Channel…). This channel will be available in every layer as an “alpha” channel, or extra channel. You can use it to create layer masks for specific adjustment layers such as levels or curves.

To do this, pick your mask channel and load it as a selection (Select >> Load Selection…), or you can select it’s inverse if need be. In the Channels palette, click on the RGB channel to bring back the original image. Now, with your selection, when you apply a new adjustment layer it will automatically be masked off based on your selection. You can also apply the mask to existing adjustment layers by duplicating the channel into the mask channel.

That’s it. It’s pretty simple, but there are a lot of variations on this you can use to create masks for isolating certain tones or subjects within the image. You can also fine tune the mask by applying things like blur, sharpening, burning, dodging, etc. To see how I applied this mask to the photo, see my post titled Dark Architecture.

Also, a big thank you to Lee for asking the question. If any of you have other questions about how I do something, or just questions in general, don’t hesitate to ask. Chances are, the question will result in a whole post dedicated to the subject. And there’s also a good chance that you’re not the only one who has the question. Ask, ask, ask.

This entry was posted in Features, Photoshop Tips on by .

About Brian Auer

a photography enthusiast from North Idaho. He's also the guy behind the Epic Edits Weblog. As a hobbyist photographer since 2003, his passion has been to constantly improve his photography skill set, to share his own knowledge with others, and to become an integral part of the photographic community.

6 thoughts on “Advanced Layer Mask Creation in Photoshop

  1. Brian Auer Post author

    You’re certainly welcome for the tips! You’re right, the ol’ blown out sky look just won’t do, and this is one way to take care of it. rules! digg sucks!

  2. Charles Boehm

    “The lightness channel is going to contain the best quality information, but you can use one of the RGB/CMYK channels just the same. Now delete the LAB conversion in your history palette of the original image to go back to RGB without doing another conversion.”

    Deleting LAB conversion in the History palette does revert back to RGB, but also deletes the lightness channel at the same time!??? Do you mean dragging the lightness channel to a new document and dragging that onto document number 1?

    “In your new image, convert from Multichannel to Grayscale (Image >> Mode >> Grayscale). Now you should have one layer and one channel that look exactly the same.”

    Multichannel??? I’ve never had an image in multichannel. I converted from RGB/8bit to grayscale. The closest I can get to what you describe is a document with one layer and two channels, a gray channel and a lightness channel that was dragged, resized and dragged again.

    Wish you could have been more clear in your description because it seems like a useful method.


  3. Brian Auer Post author

    Hey Charles, thanks for the questions! It’s difficult to put Photoshop operations into words sometimes… this one could probably benefit from a video tutorial. As for your questions…

    When you duplicate that Lightness channel, you have to make sure you duplicate it to a NEW document rather than on the original. Now when you go back to the original document, the last thing in your history tree should be the LAB conversion.

    Also, when you duplicate that Lightness channel into a new document, it automatically pulls it in as Multichannel because you created that document with a single channel from the original photo.

    Like I said, I should probably do a video tutorial on this to make it more clear.

  4. Charles

    Ok. I’ve got it (maybe). So you’re creating the mask using the new (Lightness) document then moving (applying) it to the original. I forgot about Destination in the Duplicate dialog box.

    What I’m wanting to do, and don’t know how, is create multiple masks with different selection methods and add them together to form one major mask Yeah, my pics are that bad! ;-)


  5. Brian Auer Post author

    If you create multiple masks as separate channels, you load one mask as a selection and add it to another mask. If I do a video on this stuff, I’ll be sure to include that too.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>