Zest all the limes into the bowl of a food processor. Then cut away the remaining peel and pith for each lime like this: cut the ends off of a lime, set the fruit on one cut end. With a sharp knife, and working from top to bottom, cut the pith away in strips all the way around, exposing the bare ends of the segments. Throw away the peel and pith.
Toss the now naked limes into the food processor. Cut the orange into eighths, and discard any seeds. Add the sugar and ginger and pulse until the fruit is finely chopped.
The lime even without the pith will make this a little bitter at first. Let the relish age for 12 to 24 hours in the refrigerator, and the bitterness will calm down significantly. And the ginger will be more pronounced.