Penalty Fares rules specifically exclude the issue of a
PF▸ when travelling on a time restricted ticket at a time it is not valid.
From the
(Strategic Rail Authority - about)%20-%20Penalty%20Fare%20Rules%202002.pdf#page=12" target="_blank">Penalty Fares Rules:
7.6 An authorised collector must not charge a penalty fare to a person whose ticket is not valid only because of a published restriction, as described in condition 12 of the National Rail Conditions of Carriage.
Put simply, if you board a Peak train with an Off Peak ticket you should only be sold the excess up to the Anytime fare, or the Anytime Single for travel in one direction, if that is cheaper than excessing a Return ticket. A Penalty Fare should not be issued.
John R. If this was a day trip today, and is something you are likely to repeat in the future, PM me and I can give you some fares advice that could potentially save you money, versus the Anytime and/or Off Peak fares between Swindon<->Paddington, and allow travel from Paddington in the evening peak cheaper than the through Anytime fare.