I arrived at the viewpoint I stated at least partly from the experience of having used an ancient compiler (Algol68R) that produced code that could do more or less exactly what I want any autonomous decison-making system to do. Forty years later I still haven't seen anything to match it: after a crash it produced a report that displayed the execution path from the start, not only dumping data but showing details such as which way execution had passed through conditional statements, how many times each loop had executed and why they'd exited - all cross referenced with source line numbers. In short, it took you by the hand and led you to the crash site, pointing out interesting events along the way.
A few years later, I designed a music planning system for BBC Radio 3 - a surprisingly complex job until you understand the complexity of the way orchestral musical works and the parts thereof are named and referenced. This was designed for easy use by both the dedicated music planners as well as others, e.g. producers and studio managers who might use it once a week at the most, and so it needed a decent help system that could show a user where they'd got to and what to do next. The system needed a menu structure that was 7-8 levels deep and some of the menus were tens of pages long, yet it still had to be fast and easy for the planners to use. It had just ten context-sensitive commands, designed so that they could be strung together along with user-chosen abbreviations of anything that could appear on any menu. The system had no fixed abbreviations, so a user could use any abbreviation they found easy to remember. So, we gave it a help system that could tell a user exactly where they were in the menu structure and how to get back on the track of what they were looking for. The database wasn't just a comprehensive catalogue of musical compositions, it also catalogued composers, performers and stored program details which tracked progress while programs were being made, where the recordings were stored and the broadcast history of each piece of music in each program.
So, if a compiler and its runtime environment can do what the Algol68R system could, and I was able to provide concise and readable help to the non-technical users of an application that let them search through and manipulate the content of a very complex and quite large database, I fail to see why the authors of an 'AI' system can't do the same, though of course this ability would need to be designed into it from the outset and not bolted on later as an afterthought.
Yes, I know this probably excludes most neural network systems because nobody, least of all their authors, understands how a trained network makes decisions, but think about it: would you really want such an untestable and unverifiable system making life or death decisions?