Melt butter and peanut butter on medium high heat. Mix in powdered sugar and vanilla. Only add a small amount of the sugar at a time, while stirring. If it becomes too hard to mix, add a little milk. Press mixture into 8x12 inch greased pan; refrigerate for about 3 hours. Turn pan over onto wax paper, tap bottom of pan, cut fudge into 1 inch pieces.