They don't... They are spaced 12 and 14 studs apart respectively. So there are only two ways Jabba's palace will fit on the Rancor Pit. Even the hinges in Jabba's Palace (allowing the back wall to open up)
don't match up with the hinges on the Rancor Pit
Also, the fact that the Rancor Pit hinges only allow the gate to be opened at 90°, combined with the non-squared holes, means that the Rancor Pit's gate is not
able to support the tower in any way.
One small redeeming aspect: the space under the 8x8 grate is totally open when the gate part on the Rancor pit is opened: this also allows the portcullis to open and close freely. It's going to look a bit weird, though...
I also checked out my own theory about the tower being split up into two pieces, so as to put in a third set in between the two halves:(Pictures courtesy of Brickset.com)
This theory doesn't look as good as I hoped it would be. The bottom half of the tower is about 12 bricks high, while the Rancor pit is at least 15 bricks high. Secondly, I had forgotten that the locking mechanism for the Jabba's Palace entrance gate is actually on the top half of the tower, meaning the 'theoretical new set' would have to allow for a new mechanism for the entrance. It's possible, but unlikely, that Lego would choose this path...
I guess MOC'ing a decent transition will have to be the way to match these sets up.