Preheat oven to 350F. Lightly grease and flour a 10-inch fluted tube pan. Set aside.
Place butter and both sugars in a large bowl or the bowl of a stand mixer attached with paddle attachment. Use an electric mixer or stand mixer to beat butter and sugar 1-2 minutes until completely creamy.
Beat in vanilla and eggs one at a time, beating well after each addition.
In a separate medium bowl, combine flour, baking powder, cinnamon, nutmeg, and salt. Add half of flour mixture to butter mixture with half of milk, folding gently until combined. Add remaining flour mixture and milk, folding until batter is smooth. Fold in mashed bananas.
Scrape batter into prepared pan and smooth top. Bake pound cake at 350F 55-60 minutes until a skewer inserted in center comes out clean.
Allow cake to cool in pan 10 minutes, then run a knife along the edges and carefully invert cake onto a wire cooling rack. Allow to cool completely.