In a well-seasoned 10-inch cast-iron skillet over moderately low heat melt the butter. Sprinkle the brown sugar evenly over the bottom of the skillet and stir to combine. Remove skillet from heat and arrange peach halves, cut sides up, in a decorative pattern over the sugar and butter. Sprinkle whole pecans evenly around peaches.

Preheat oven to 375 degrees F.

In a large bowl whisk together flour, baking powder, baking soda, salt, cinnamon, and nutmeg. In another bowl using an electric mixer cream together butter and sugar until light and fluffy. Beat in vanilla. Add the eggs, one at a time, until well incorporated. With mixer on low beat in the buttermilk until just combined. Mix in the dry ingredients in two batches until just combined.

Spoon batter evenly over the peaches. Bake in preheated oven for 35 to 40 minutes, or until golden brown and a tester comes out with crumbs adhering. Cool cake in skillet on a rack for 10 minutes. Run a thin knife around the edge of skillet and carefully invert the cake onto a platter.