Hermitage Castle is remote. Even travelling there today by motor car, it seems to take forever to reach it, along narrow, single-track roads. It must have been even more remote on horseback or on foot. But it is in a key location controlling Liddesdale - and is close to the English border.
The first fortification was probably a timber structure made around 1240 by Sir Nicholas de Soules who was the King's butler or chamberlain. He was a later a claimant to the throne of Scotland and his son was later accused of conspiring to murder King Robert the Bruce and seize the throne himself.
When the Wars of Independence broke out in 1296, Hermitage was on the front line and the castle changed hands several times. Sir William Douglas captured the castle in 1338 but King David II subsequently made Sir Alexander Ramsay sheriff of Teviotdale. That did not please Douglas, who promptly imprisoned Ramsay and starved him to death. Douglas was so powerful that King David then made him sheriff instead! However, when the English invaded in 1346, Douglas did a deal with them which involved him gaining control of Hermitage in return for not impeding the English armies. That was a step too far and King David granted the castle to Sir William's godson instead.
Douglases Take Over
Ownership of Hermitage later passed through marriage to Sir Hugh de Dacre, an English lord who began the first stone castle, which now forms the heart of the present castle. However, William, the first Earl of Douglas, eventually took over the fortification in 1371 and then in 1388 it was inherited by George Douglas, first Earl of Angus and founder of the "Red" Douglases. He extended the castle still further, adding corner towers to the already massive keep. The "Red" Douglases were often mistrusted by the monarch, even though they helped both King James II and IV - sometimes siding against the "Black" Douglases as a result. In 1492, however, the fifth Earl was involved in an intrigue with the English and when this came to light he was forced to exchange the strategic Hermitage for Bothwell Castle in Lanarkshire.
Earl of Bothwell Takes Over
Patrick Hepburn, the first Earl of Bothwell, now took over Hermitage Castle and also owned Crichton Castle in Midlothian, not far from Edinburgh. The fourth Earl of Bothwell, James Hepburn, one of the most colourful figures in Scottish history, became a favoured adviser of Mary Queen of Scots in the 1560s. In 1567 he was implicated in the plot to murder Mary's husband, Lord Darnley. The previous year, Mary had ridden 30 miles from Jedburgh across rough country to Hermitage to see Bothwell who had been wounded in a skirmish. She could not stay at the castle and on the journey back, her horse stumbled in a bog and Mary contracted a fever. She recovered after a week, but the house in Jedburgh where she recuperated is still known as "Mary Queen of Scots' House".
Bothwell's nephew, Francis Stewart, 1st Earl of Bothwell received a new creation as Earl of Bothwell, and Keeper of the castle. A grandson of James V, albeit through an illegitimate line, he was viewed by some as a potential replacement for James VI. In 1591, Bothwell was arrested, tried, gaoled and forfeited hid titles and lands for his supposed involvement with the infamous North Berwick Witches. He obtained a pardon in 1593 but again became involved in intrigue and he was again attainted, by Act of Parliament, on July 21, 1593. The Hermitage once again reverted to the Crown.
The Scotts Take Over
The following year, James granted the castle to Sir Walter Scott of Buccleuch,("the bold Buccleuch") a notorious Border reiver, Warden of the western marches, Keeper of Liddesdale, and leader of the daring and infamous attack on Carlisle Castle to rescue Willie Armstrong of Kinmont.
In the 19th century, the romance of Hermitage appealed to the imagination of Sir Walter Scott. When he had his portrait painted by Sir Henry Raeburn, Hermitage can be seen in the background. Scott's publisher Archibald Constable, commissioned the portrait from Sir Henry Raeburn. Conceived with reproduction in mind, in the next ten years it became the most frequently engraved and widely circulated image of Scott. For the first time Scott is explicitly personified as a poet in a setting full of allusions to his own work. In the background are the hills of Liddesdale and Hermitage Castle, which are featured both in Marmion and Minstrelsy of the Scottish Border.
When his publisher Archibald Constable was bankrupted in 1826 the painting was purchased by Scott's patron the 5th Duke of Buccleuch. In a letter to the Duke in 1826, Scott wrote: 'I must say I was extremely gratified by seeing Raeburn's portrait (which was like what the original was some two or three years before your Grace was born) hanging at Dalkeith and feel sincerely the kindness which placed it there. One does not like the idea of being knocked down even though it is only in effigy' The painting still hangs at Bowhill, home of the Dukes of Buccleuch.