St Mary's Episcopal Church, near the main entrance to Dalkeith Palace, was consecrated in 1845 and a memorial chapel to the 5th Duke was added by William, the 6th Duke (1831 - 1914) in 1890. The remains of the 5th Duke and several family members lie in a crypt below. Most of the Dukes of Buccleuch (the 3rd, 4th, 5th, 6th, 7th) are buried in the Buccleuch Memorial Chapel. The 2nd Duke (died 1751) is buried in Eton College Chapel. The most recent Dukes (the 8th and 9th) are buried among the ruins of Melrose Abbey in Melrose.

St Mary's remained a family chapel until 1915, when it became the responsibility of the Diocese of Edinburgh, within the Scottish Episcopal communion of churches. In 1958 the church building was given to the congregation and the Diocese by the 8th Duke.

The graphic on the right is an early photograph of the 5th Duke, Walter Francis Montagu Douglas Scott who was not only 5th Duke of Buccleuch but also the 7th Duke of Queensberry KG, PC. When his older brother, George Henry, died at the age of 10, Walter became heir apparent to the Dukedoms of Buccleuch and Queensberry. He was only thirteen when he succeeded his father to these titles in 1819.

In 1878 he became Chancellor of the University of Glasgow, a post he held until his death in 1884. He was also responsible for creating a harbour at Granton on the river Forth estuary near Edinburgh.

King George IV spent some days in 1822 as the Duke's guest at Dalkeith Palace, the first visit of a reigning Hanoverian monarch to Scotland. The Duke as only 16 years old at the time. The visit had been masterminded by the historian and novelist Sir Walter Scott who was a close friend of the 5th Duke.