The tip of the rock in Fort Frederick, known as Swami Rock, is crowned by this Hindu temple dedicated to Shiva. On the same site, the Thirukonamalai Konesar temple, built in 200 BC and once attracting pilgrims from all over Asia, was destroyed by the Portuguese in 1624. This was done in order to reuse the material for the construction of the fort that would become Fort Frederick. One can only imagine the splendor of the monument if one considers that its destroyers themselves called it "the temple of a thousand pillars". It was rebuilt in 1963 under the name of Koneswaram Kovil. Although smaller in scale than its predecessor, it remains an important place of worship for Hindus and a popular place to visit for followers of other religions. A large statue of Shiva watches over the site. It is a busy place on poya days.

Swami Rock is the northern part of the rock on which Fort Frederick and Koneswaram Kovil are built. Its name means "Rock of the Lord", and Swami Rock rises 130 m above the water, offering a picturesque view of the two surrounding bays, Back Bay and Dutch Bay.

Lover's Leap, Swami Rock's highest point, hides a tragedy that actually happened here. It was here, on April 24 1687, that Francina Van Reed, the daughter of a Dutch officer, threw herself into the waves after watching her fiancé's ship disappear over the horizon. The date of this tragedy is inscribed on a pillar, almost obliterated.