"The only reason anyone would want to keep it secret is if it's doing something it shouldn't be such as profiling by race."
A few other possibilities. One is that it's such a pile of crap that they wouldn't be able to sell it or those that bought it would want their money back. Alternatively, it's such a pile of crap that the victims would take them for everything they've got and more in damages. More likely it's another of those AI things where nobody knows how its arriving at conclusions so it's not so much they don't want to disclose anything, more a case of they can't.
When NI had the judge-only courts the judge had to give a reasoned account of how he came to his decsion (which, of course, is more than a jury has to do). If S/W were to be a tribunal of fact I'd expect failure to give a reasoned decision to be basis for appeal against conviction. I'd also be interested in how S/W instructed itself in matters of law; with a jury trial this is always done in open court and can be a basis of appeal on the grounds that the judge made an error in law.
If the S/W is determining sentence then I'd expect lack of explanation there to be the basis of an appeal against sentence.