Cream butter until it is soft. Add almond paste a little at a time, alternating with milk. Stir in sugar and ground nuts. Cook over medium heat, stirring constantly until cream thickens. Stir a few tablespoons of hot, thickened cream into egg yolks.
Then pour egg-yolk mixture into remaining cream, stirring briskly. Return to low heat. Continue to cook and stir until cream thickens a little more, being careful not to let it boil. When mixture is cool, add vanilla.