He was not allowed to judge on process and guilt. If he had made his own determination of guilt his ruling would have been a slam dunk to be overturned.
The relevant federal law -- the Federal Arbitration Act -- that allow arbitrators' decisions to be vacated essentially does not allow judges to put themselves in the place of the arbitrator. The judge is basically forced by the law to defer to the abitrator's determination of (a) what the facts are, and (b) if those facts support the given arbitral award. All the judge can do is look to the process/procedure of the arbitration (that it was conducted fairly, that it didn't break any laws, and are the relevant contract provisions enforceable) and whether or not the arbitrator was "evidently partial".
I imagine but do not know that perhaps some factual determinations could be looked at as part of determining if an arbitrator is evidently partial. For example, if a person could be punished for holding yen and standing in direct sun, and the person in question was found standing against the west wall of a building at sunrise (thus impossible to be in direct sun) in East Bumblef**k, NE (thus very unlikely anyone would have yen), and the arbitrator said "since the sun rises in the west and the legal currency of the USA is the yen, then person X had to be in the sun and almost definitely had yen and therefore is punished", I would not be surprised if the judge could use those repeated clear misstatements of fact as evidence of bias.