Fun Edge Effects Using Layer Masks and Filters Tutorial

in Adobe® Photoshop® and Photoshop® Elements

In an earlier training video, we looked at how to add a vignette effect to your photos.

In this tutorial, we’re going to take that idea a step further as we look at how you can use layer masks and filters to add other really cool edge effects to your photos.

First, let’s talk about layer masks. Layer masks are a very powerful feature that has been a part of the full blown Photoshop program for many years and is now a part of the latest version of Photoshop Elements, Photoshop Elements 9. But, there is a work around that you can use to create a fake layer mask in earlier version of Elements.

A layer mask is a mask that is linked to a layer and allows you to hide or reveal what’s on that linked layer based on the color of the mask. So, the parts of the linked layer that are behind portions of the mask that are white show through. The parts of the linked layer that are behind portions of the mask that are black don’t show through.

This concept is probably easier to understand visually.

Here is an image open in Photoshop Elements version 9. As we mentioned before, you can create a layer mask with the click of a button in Elements 9. And, relative to layer masks, pretty much everything we’ll be doing here in Elements 9 can also be done in the Full Photoshop program.

So, let’s add a layer mask to this photo. First, we have to convert this Background layer to a regular layer and we’ll do that by double clicking on the name in the Layer’s palette. Then, just click ‘OK’ in the New Layer dialog box that comes up.

Now, we’ll add the layer mask. We do this by clicking on the icon at the bottom of the Layers palette, the ‘Add layer mask’ button. This icon looks like a circle inside of a box.

When you click this button, you’ll see a layer mask has been added to the photo layer. You can see it in the Layers palette. You’ll also notice that the mask is all white and that the entire photo is still visible.

Let’s look at how a layer mask works.

To do this, select the brush tool from the toolbox on the left. Set the brush tool to a large, soft brush (I used a 200 px brush here).

Select black for the foreground paint chip:

Select the layer mask by clicking on it in the Layers palette:

Now, start painting on the layer mask with the black brush. As you do, you will see that the photo disappears from the places where the mask is painted black. Below, you can see what happens as we paint around the edges of the layer mask with the black brush:

But, if you disable the layer mask by right clicking on its thumbnail in the Layers palette and choosing ‘Disable Layer Mask’ in the pop-up menu:

You can see the entire image again. So, the image is all still there. But, the layer mask is hiding portions of it based on the color of the layer mask. That’s the idea behind layer masks.

Now, let’s move over to Elements 8.

As mentioned before, the versions of Photoshop Elements prior to version 9 did not include the layer mask feature. But here’s a work around that you can use to create a ‘fake’ layer mask in the versions of Elements before version 9.

Below, you can see that we have the same photo opened in the workspace of Elements 8:

As we did before, we need to convert the background layer to a regular layer by double clicking on the layer in the Layers palette and then clicking ‘OK’ in the New Layer dialog box that pops up:

Now, we’re going to add an adjustment layer to this photo. To do this, click on the ‘Create a new fill or adjustment layer’ icon at the bottom of the Layers palette. It looks like a black and white circle:

When you click on this button, you will see a pop-up menu showing the different kinds of adjustment layers that you can add. You can actually choose any of them for what we’re doing here, but I usually choose ‘Solid Color’:

This will bring up a color picker dialog box. Choose any color you wish, if doesn’t matter for what we’re doing here, and then click ‘OK’:

You can now see a solid color adjustment layer above the layer of the photo. It looks like a layer filled with a solid color linked to a layer mask:

Now, move the adjustment layer below the layer of the Photo by clicking and dragging it in the Layers palette:

To get:

And, now, group these two layers together by first selecting the top layer with the photo on it and then choosing Ctrl+G. The layers will now be grouped:

And, the layer mask attached to the solid color layer will act like a layer mask for the photo.

You can check that this works by using the paintbrush tool to paint on the layer mask as we did before. Select the brush tool, choose a large round brush and choose black as your foreground color. Before painting, click on the layer mask to select it. Then start painting with black on the layer mask. As you do, you will see the photo disappear in the areas where you paint:

Now that we’ve looked at how to create a layer mask in Photoshop and Photoshop Elements version 9 and a fake layer mask in the case of previous versions of Photoshop Elements, lets’ look at how we can use these layer masks to give photos some fun edge effects.

Now what we’ll be doing here is applying various filters to the layer mask. We’ll be working in Photoshop Elements 8, but this technique works whether you are using the fake layer mask like we created here or you’re using a regular layer mask in Elements 9 or regular Photoshop, as long as you’re applying the filter to just the layer mask.

Let’s undo the painting that we did on the layer mask here by using Ctrl+Z to undo the paint strokes.

Before we apply our filters, let’s add in a layer of color at the bottom of our layers stack so that it will be easier to see what we’re doing. The color doesn’t really matter, we just want to be able to see the effects that we’re adding to the photo and that’s hard to with a transparent background.

To do this, select the adjustment fill layer, the bottom layer and hold down the Ctrl button on the keyboard as you click the ‘New Layer’ icon at the bottom of the Layers palette:

This will add a new blank layer below the adjustment layer:

Fill this new layer with color by choosing Edit>Fill Layer, select a color in the color picker dialog box and click ‘OK’:

You won’t be able to see this color yet.

Now, we’re ready to start working on our edge effects.

Draw a selection around the area of your picture that you want to frame. Use any shape that you want to. You can use the elliptical or rectangular marquee tools for this or you can go freeform and use the polygonal lasso tool if you want. Whatever shape you like works here.

For our example, we’ll use the rectangular marquee tool to draw a selection outline on our photo:

Change to the inverse of this selection by choosing Select>Inverse from the top menu:

Now, select the layer mask by clicking on it:

And, fill the selected area with black by choosing Edit>Fill Selection and the choosing Black. Click ‘OK’:

The areas of your photo outside of your framing area will disappear and you will see the colored layer below it:

Ctrl+D to deselect.

Now, we’ll apply a filter to the layer mask. Here’s where you’re going to want to experiment with the different filters available to you in the program. And, don’t be afraid to use more than one filter to get some really cool effects.

Don’t forget to make sure that the layer mask is selected before applying the filter so that the filter acts on the layer mask.

Here are a few examples:

Spatter Filter:

Choose Filter>Brush Strokes>Spatter.

This example uses these settings:

And the result:

Glass Filter:

Choose Filter>Distort>Glass.

The Glass Filter has a texture drop down box where you can use different setting for the glass texture. Experiment with these different textures.

This example uses these settings:

And the result:

Crystallize Filter:

Choose Filter>Pixelate>Crystallize.

The Glass Filter has a texture drop down box where you can use different setting for the glass texture. Experiment with these different textures.

This example uses these settings:

And the result:

Displacement Filter:

Choose Filter>Distort>Displace.

In the dialog box that comes up, you’ll need to enter the amount of displacement you want the program to apply. In this example, we set both settings to 20 and click ‘OK’:

The next dialog box asks you to navigate to the displacement map that you want to use. Photoshop Elements has several displacement maps built into the program that you can try. They are located at:

Program Files>Adobe>Photoshop Elements 8.0>Presets>Textures

Here’s the result when we use feathers.psd for the displacement map:

So, that’s it. There are tons of framing effects that you can create using this method. So experiment and enjoy!