Roast whole yams in preheated oven until heated through but not quite tender, 35 to 40 minutes.
Melt butter in a large pot over medium heat. Stir brown sugar, orange juice, and nutmeg with the melted butter to dissolve the sugar completely; bring to a boil, reduce heat to low, and cook until the liquid thickens into a syrup, 10 to 15 minutes.
Peel cooled yams. Cut yams into bite-size chunks; arrange in a single layer in a 20x9-inch baking dish. Drizzle syrup over the yams.
Bake in the preheated oven, twice opening the oven to spoon syrup over the yams, until the chunks are fork-tender, about 30 minutes.
Arrange marshmallows over the yams and continue baking until the marshmallows are puffy and browned, about 10 minutes more.
2 1/2 pounds yams
2 tablespoons butter
2 cups brown sugar
1 cup orange juice
1 teaspoon ground nutmeg
3 (10.5 ounce) packages marshmallows