Masking Hair from difficult background

Hi
Just had a quick go with your tiff download. It’s a bit fiddly, but quite easy to improve on what you have in the example. I duped the blue channel, actually there wasn’t too much to choose between the channels. I then ‘apply image’ using the blue channel on to itself in overlay mode. I used the ‘apply image’ 3 times. Did a rough fill of black into the bulk of the head area and a rough deletion of the background that isn’t too close to the hair. I then used dodge and burn to differentiate between hair and background a bit, then finally repeated the ‘apply image’ step again. that left me with a channel I could use as a layer mask.
It took about 5 minutes.
Hope it helps

< http://www.2shared.com/file/5191400/296eed/forum_pic_copy.ht ml> How about this? it still need a bit of work but i’d say it’s mostly finished. I’ve saved my progress as alpha channels so you can more easily follow if you like my solution:

(I worked off of the second image)Look at the channels and pick which of them has the most degree of difference in the background or in the subject(Red and Blue in this case). Go to Image->Calculations and set Source1 and Source2 to the current image, Layer should be set to Background(or Merged if you have more layers), Channel should be Red for one of them and Blue for the other and set the blend mode to Difference. Image->Adjustments->Levels and set the white point to 85 and then get the Burn tool with range set to Shadows and about 50% exposure and paint in the black areas to get rid of the white junk. Get the Lasso tool and broadly select the white part and fill with white. Get the Dodge tool set to Highlights with about 50% exposure and paint in the image to turn the rest of the guy to white. Press ~(the tilda key, its next to 1) to get a sense of where you are(red is the deselected areas).

Most of it is done, all we need is the bit of hair from the left side which got obliterated by the calculations command(too little difference between the channels in that area). Go again to the Calculations command and this time set the channels to Green and Blue as those contain the most detail in that area. Set the blend mode to Add and set Offset to -150 and Scale to -1.2. Make a rectangular marquee selection around the hair(or a normal selection but in that case make sure to turn of anti-alias) and go to Levels and set the Black point to 40 and the white point to 140 then Command+C to copy the selection, select the alpha channel that you worked on until now with the burn and dodge tools and hit Command+V to paste. With the Burn and Dodge tools refine the area like you did before.

Hope this method isn’t too slow for your needs but masking usually involves a lot of time. On the bright side, the more you do it, the faster you become at it.

Wow to both of you. I post the question up, and two hours later have two different geniuses (genii?) working on the problem. It’s amazing how often this happens on this forum. Either alot of people have alot of time on their hands, or there’s an awfully lot of very good people willing to drop whatever they’re doing at a moment’s notice to solve the problem of a complete stranger. For now I will presume it is the latter. 🙂

This is going to take me some time to make sense of. I always think I understand the in’s and outs of Photoshop…. until I meet up with people here. I’ll try following both these instructions and see how I come out on my own and get back to you both in a few days. What is interesting right off is that neither of you are using either a masking plug in or Photoshop’s own Extract. I’m guessing your steps are simply doing "manually" what a masking filter does.

Thanks so much.

Masking is fun and relaxing, for me anyway 😛
The extract filter could work but there’s a lot of guesswork involved and its very hard to reproduce the same effect. The most important thing to know when masking is to dodge highlights and burn shadows, the rest of it are just steps to make it happen faster/better. Its also useful to deselect Use Shift key for Tool Switch in the general preferences tab so that you can easily switch between the dodge/burn tools by just pressing O as you’ll do it a lot

What’s also interesting is that when you ask your question and I try to help, I also learn something new from Zeno. That’s what’s great about the forum. There are so many ways to do these things. I try to keep my thoughts very broad from the outset. I’m looking for qualities within parts of an image that have differences. Is it tone, colour, (which colours), saturation, anything I can exploit. There are always ways to pull apart these differences and that’s all that’s needed to make a mask/edge. It’s perhaps a bit old school with so many great plugins available now, but understanding the root of a problem is the key to the solution and that’s comforting.

OK, I’ve tried both of your suggestions. Bakerjohn, I could get your process to work fairly well, although not completely clear what I’m doing. But that’s nothing new. I do want to know if there are specific dodge and burn settings that might be best to use ( I just did 100% midtones for both and it worked fairly well). Also, a dumb question which should suggest why I’m having so much trouble with this. I end up with a black face image (with a white halo that was left of the background) on one layer and the original image on the other. How do I use a mask like this to cut out the background of the original image? I want to be left with a face/hair layer with a transparent background.

Zeno, you get the complexity award. I don’t know where to begin to ask you questions because you are having me use tools and settings that I’ve never used before. I don’t want to bother you to define all these things, but basically without an understanding of what I’m doing, I’m really not getting to the point of a real black mask. Just a rather gruesome high contrast image. The hair stands out but the background is also quite pronounced.

Basically what you are doing is making the thing you want selected white and what you want deselected black. Grays indicate partially transparent areas and you need them only on the edges to make sure that you have a smooth transition and in the hair to get a nice selection of it. Use Image->Adjustments->Invert if you get the background white and the subject black(this happens very often when masking).

The reason you use the burn and dodge tools is because when set right they will only affect the parts you need them to. Burning set to Shadows will make the dark portions of the image even darker while leaving the whiter portions relatively intact(and the other way around with the dodge tool set to Highlights). The Exposure setting is basically how strong the effect is. When working around the hair its usually a good idea to keep it a bit lower because you want to keep the grays in the hair, otherwise you’ll get jagged stumps instead of nice transparent hair

Like zeno says, initially it doesn’t matter whether you make the head black and the background white or viceversa. You can easily invert it once you have completed the mask. Sorry I missed that step out in my explanation (kinda crucial).
Another thing to bare in mind is that if you want to lay this cutout image onto another coloured backround if you have a pure black and white mask you are likely to get some white fringing on the cutout, because the mask isn’t allowing for reflected light from the background in the very edges of the mask. It will probably be best to leave the very edges of the hair as light grey not pure white. This will form a semi transparent edge to your mask. You can do this by blurring the mask by a pixel or by being extra careful when using the dodge and burn tools

OK Zeno… but I should be more clear as to what was unclear, or confusing, to me with your process. I didn’t understand the need to go into it twice with the calcuations. I didn’t understand how just going through once wouldn’t be enough. Or in other words, what the two different calcs are doing. I also don’t understand what the Offset and Scale factors are. Also it wouldn’t let me enter negative 1.2 for the scale. Only a positive value.

And, again for either one of your processes, I’m scared to say I don’t know what to do with whatever mask I come up with to create a final image. The OnOne filter did it all for me. I mean I understand how to mask an image, but that’s usually with a stark b&w mask. This one has fuzzy grey edges. How do I get from here to there….?

Sorry about the negative scale, that was a typo, it’s 1.2, not -1.2 The second calculations is "needed" because the hair on the right side didn’t get selected very well because there wasn’t a lot of difference between the channels in that part. If you feel that it’s good enough with just the initial step then by all means, stop there.

My reasoning when using the second calculations was that i needed a way to make contrast in the part of the hair that i didn’t like how it was selected. Usually its better to use two channels when trying to bring out details so i used the green and blue channels because there was the biggest contrast between the hair and the background in that particular area. For the blend mode, i just went through all of them to see what i was getting. I settled on the Add blend mode but there were others that you could also use (Linear Light was also a candidate), it’s just that you get more control when using Add because of the Offset and Scale settings. What Add basically does is add the luminance values of the two channels together, ads to this the offset value and then multiplies the result by the scale amount. Usually you first set the Offset and then darken the image with the Scale setting and then go back to the Offset to tweak a little bit more. I inverted both channels in the calculations command to get a black background and white subject.

Av
Once you have the channel made up. In the channels palette, command click the channel to select it. Go to your layers palette and click on the layer mask icon. to create a layer mask from the selection. Is that what you needed to know?
John

Oh I don’t know. I think maybe I messed up some where. When I did your steps John, I was left with a solid black face and hair with whispy grey ends and then a little edge of white which was what is left of the background.

So, it not being a solid black mask, I don’t understand how to select it out without ruining all the hair. We’re not talking feathered selection are we? That would seem counterproductive.

Loving this thread, some nice info. I have found that sometimes it’s necessary to give the person a "haircut" when cutting these masks. I have used everything from channel chops to vector tracing to Color Range (or some other color selection tool) to cut masks around hair. And sometimes I just have to play the part of a hairdresser.

Best of Luck!

pbc

Hi av
Every surface that isn’t face on and matt black picks up some degree of reflection from it’s background. Imagine a chrome sphere as an extreme example. If it was set against a blue background and you made a crisp selection around the edge of the sphere you would have a sphere with blue reflections, (not very useful if you want to lay it on a different coloured back drop).In order to pick up the correct colour reflection for any chosen background the selection of the sphere should include elements of transparency, i.e greys not blacks.
As your example involves masking grey hair from a greyish background it doesn’t matter in this instance.

John, I think you’re misunderstanding where I’m going amiss. Sorry that this is probably boiling down to Masking 101, but as I mentioned I’ve been relying on the OnOne masking program to do all this automatically and I just don’t have any other experience with masking. It’s something I’ve never had to do, believe it or not.

So I’ve posted a picture of what I have. I think what is obvious to you is mud to me. But I’m still at a loss of how to use the mask I end up with to cut out the actual background of the actual picture. The mask I end up with is a separate layer now on the PSD (or the middle picture of the three originally posted.)

<http://uploading.com/files/VUVCY4IK/mask> sample.tif.html

Sorry I’m so dense on this!!

I see you still have problems with the image, let me give you another alternative, maybe this one will be more beginner friendly 😛

As before, i’ve made a tiff with a bunch of alpha channels so you can compare your effect with what i’m getting: < http://www.2shared.com/file/5200151/3faa1f6f/forum_pic_new.h tml>

For this version you’ll only use the blue channel so make a copy of it by dragging it to the new channel icon and then make a marquee selection around the upper part of the image that includes everything above the subjects neck hair. I’ve made an alpha channel to show you my selection. To load an alpha channel as a selection you Command+Click on it in the channels panel. With the selection active and the copy of the blue channel selected go to Image->Adjustments->Curves and select the tool that looks like a pen. With it selected, click and drag at the top of the curves all the way to the middle(most of the face should have turned white). Then drag down the other half and the rest of the image turned completely black. So you have something like this -_ in the curves panel. Just to make sure, if you look at the Output and Input values for the top line you’ll have 255 output and the input goes from 0 to about 128 while the other part of the line has 0 for output and for input goes from where it left off(about 128) all the way to 255 and the image should look horrible. Click the Smooth button 3 times and you should get something similar to the first saved alpha channel.

With the selection still active, go to Select->Inverse to invert the selection and then go to Curves. Same thing as last time, only now you drag from input 0 to about 90 all the way to the top while the rest is down. Click on Smooth 4 times this time and check to verify that the channel you just made looks similarly to my second alpha channel.

Next up get the Lasso tool(or the Polygonal Lasso tool) and clean up the image till you get it to look something like my third alpha channel.

Now we come to the dodging(no burning this time around). Try to affect as little of the hair as possible with the Dodge tool. Invert the image by going to Image->Adjustments->Invert because the human eye sees very light gray on white much better than very dark gray on black. Select the dodge tool set to Highlights and 100% Exposure and a big brush with 0% hardness, i used a 600pixel brush but you’ll need a bigger one if you’re working on a bigger image then what i’m working with. Paint outside the gray areas so that only the drop off of the brush hits the gray background you want to remove. I was painting so that the edge of the brush was at about half of the brushes radius away from what i wanted to affect. You’ll have to use a smaller brush to completely get rid of the background patter that’s above the right shoulder. Check to see if you have something similar to my next saved alpha channel.

Next up, cleaning the subject. Go to Image->Adjustments->Invert to have the subject white again and clean up the rest with the Dodge tool. After you roughly cleaned up the image, get the brush tool set to about 35px size, 100% hardness. You’ll now paint away with the brush tool the parts that the dodge tool couldn’t handle like the small black dot in the subject’s ear and around the hair. Periodically check how the image looks with the RGB shown as a rubylith overlay by hitting the ~ key so that you can see what should be part of the image and what shouldn’t. You’ll notice for instance that there’s a small chip at the top of the subject’s head that you’ll have to fake out using the brush tool. Also at the right side, where the selection that you made ended you’ll also have a small area to fix up with the brush tool.

That’s about it, you should be at a point similar to my last saved channel. To actually use the alpha channel you have to Command+Click on the channel to load it as a selection, go to the Layers panel and with the layer selected press the Add Layer Mask icon at the bottom. To check your work make a layer below and fill it white or black to see how it looks. If you want to cut out the subject and not just have a mask that hides the background then Control+Click on the Layer Mask thumbnail and chose Apply Layer Mask.

Well, this was long. It took longer to write this than to actually make the mask :p

You’re not being dense. It’s just tricky for me to put into words that are clear. Looks like you’re getting quite close by the look of your link. Still need to dodge away some of the background using the highlights setting to avoid disturbing any of the hair. The file that you sent me the download link to should be a modification of a channel copy residing in the channels palette. That is to say, you chose a channel and duplicated it and then modified it to get to the stage I see in the download you supplied.
If you hold down the command key and click on the small square preview of the channel you have made it will create a selection of that channel. Then go to the layers palette with the selection still live. Highlight the layer you have of the man with background and click on the layer mask icon at the bottom of the layers palette second from the left. It looks like a grey square with a circle in it.
You must remember to not be working on a layer named background. It’s good practice to duplicate your original layer so that you are always working on a layer copy. I’m not sure that layer masks work on background layers.
Please don’t hesitate to ask if any or all of this is still unclear.

Forgot to mention. You will also need to invert the layer mask to delete the background and not the head. Select the layer mask and go to ‘Image’ ‘Adjustments’ ‘Invert’.

Ok… I just read the help manual and I’m getting closer to explaining to you what isn’t working! Because I did exactly what you said, and after clicking on the mask button a second square appears to the right of the image icon in the layers pallet. IT IS ALL WHITE.

So I followed instructions on making a plain old mask out of an image. I selected the layer that has the actual image (that I’m trying to cut the background out of). I made a simple rectangular selection on the guy’s nose. I hit the Mask button, and voila, his nose remained, and the unselected parts disappeared. And next to the icon image of the whole face in the layers pallet, I see a black rectangle with a little white square noting where the mask over the nose is. I understand all that.

But this thing I’m creating a layer mask of is not the image. It is already black and white. Again, if it had solid edges (which it doesn’t) I would just do something like Magic Wand select the white or the black and then go over to the layer of the guy’s faace and easily cut out what I don’t want. But I don’t think that Magic Wand would work too well in this case with whispy hair edges. And that’s where I’m stuck – how to use this layer to mask the layer underneath it. Because I don’t want to just mask itself, which seems to be all it does when I cmd-select the layer (which just selects EVERYTHING by the way).

Sigh.

I understand inverting. But not how to APPLY IT to the layer below!

Zeno’s Paradox states: If you want to cut out the subject and not just have a mask that hides the background then Control+Click on the Layer Mask thumbnail and chose Apply Layer Mask.

I’ve tried this. All it does is make the little white box next to the B&W image of the head go away. It doesn’t apply to the layer below where all the action (actual photographic image of the face) is.

That part is for after you made the alpha channel. Open the file that i linked in my big post, in the Channels Panel, Command+Click on the last channel(called "inverted back and cleaned subject"); you should have an active "marching ants" selection. Go to the Layers Panel, make sure that the layer is selected and the selection is still active and click on the Add Layer Mask icon(the dark gray square with a white circle in it from the bottom of the Layers Panel). The background should be gone, replaced with a gray/white checkerboard which indicates transparency. And only now if you want to permanently remove the background do you Control+Click on the Layer Mask thumbnail from the Layers Panel(the black and white one) and click Apply Layer Mask.

But like i said, you do this after you have made the final alpha channel.

to zeno: ahhhhhhhhhhhhhhhhhh………….. 🙂

Thanks for everyone’s help and patience on this. You’ve all been alot of help.