Without knowing your location, I would suggest you contact the selling dealership.
Toyota didn't offer the fastest "TRD" leather from the factory

, it had to have been installed at port (if acquired in a distributorship area) or locally by your particular dealer's vendor.
There is a good chance that the leather installer may still have the piece of kit for re-fitting the optional (accessory) passenger armrest. If not, Classic Soft Trim and Katzkins do a lot of Toyota work and may be able to hook you up with a cover, which you can apply DIY.