Among the structures you can admire in the gardens, I suggest you take a look at the ruins of the Tenshukaku, the central tower of the Edo Castle and stretches of the castle walls.
The only three buildings dating to the Edo Period still standing are the lookout towers in the area along the moat. They have three floors, and you can spot them thanks to their characteristic pagoda roofs.
An interesting fact: the keep of the Edo Castle had a base of 41 by 45 meters and a height of 11 meters, which was the foundation of a 51-meter, five-story fort, making it the tallest tower in Japan. It was destroyed by fire in 1657, however, and was never rebuilt.