Displacement + Image map= Much longer render times (SOLVED)

[Triker]

If I use a texture map in the diffuse channel of Vray material and use displacement tag I get render times that are huge compared to just using color in the diffuse channel.

For example I'm creating a rug: Use displacement with noise to create the yarn, and RGb for color and it renders in 26 seconds, add a texture (image) in diffuse and it renders in 8 minutes.

Any advice here?

(the rug looks great by the way)

EDIT: My bad. Do Not put the image map in the diffuse TRANSPARENCY layer, put it in the COLOR layer.