The rack was a medieval torture device. The crude but, one presumes, effective racks often tore the victim’s limbs from their bodies. It isn’t surprising that ‘rack’ was adopted as a verb meaning to cause pain and anguish.
The first recorded use of this being specifically applied to brains is in William Beveridge’s Sermons, circa 1680:
“They rack their brains… they hazard their lives for it.”
Amazon.com – Read eBooks using the FREE Kindle Reading App on Most Devices