Preheat the oven to 350°. Spray a sheet pan lightly with cooking spray. Cut the pound cake into 1” cubes and spread on the sheet pan. Bake for about 10 minutes until lightly toasted.
Beat the cream cheese with the brown sugar and vanilla until light and fluffy. Using a spatula or a wooden spoon, gently fold in the whipped topping until nicely combined.
Layer the trifle in a glass bowl or trifle dish like this: half the cake cubes, half the peaches, half the cream cheese mixture, repeat the layers and garnish with berries or mint if using. Refrigerate for a couple of hours before serving.