<ahref="https://aaai.org/Symposia/Spring/sss22symposia.php#ss06"target="AAAI2022">Ethical Computing: Metrics for Measuring AI's Proficiency and Competency for Ethical Reasoning</a>
<ahref="https://aaai.org/Symposia/Spring/sss22symposia.php#ss06"target="AAAI2022">Ethical Computing: Metrics for Measuring AI's Proficiency and Competency for Ethical Reasoning</a>
</p>
</p>
<p>
<p>
<i>Symposium workshop overview.</i>
The prolific deployment of Artificial Intelligence (AI) across different applications have introduced novel challenges
The prolific deployment of Artificial Intelligence (AI) across different applications have introduced novel challenges
for AI developers and researchers. AI is permeating decision making for the masses: from self-driving automobiles,
for AI developers and researchers. AI is permeating decision making for the masses: from self-driving automobiles,
to financial loan approval, to military applications. Ethical decisions have largely been made by humans
to financial loan approval, to military applications. Ethical decisions have largely been made by humans
...
@@ -188,34 +189,105 @@ more-effective supervision of operations involving lethal or life-saving force.
...
@@ -188,34 +189,105 @@ more-effective supervision of operations involving lethal or life-saving force.
With AI making decisions, those ethical responsibilities are now being pushed to AI designers
With AI making decisions, those ethical responsibilities are now being pushed to AI designers
who may be far-removed from how, where, and when the ethical dilemma occurs.
who may be far-removed from how, where, and when the ethical dilemma occurs.
Such systems may deploy global "ethical" rules with unanticipated or unintended local effects or vice versa.
Such systems may deploy global "ethical" rules with unanticipated or unintended local effects or vice versa.
</p>
<p>
While explainability is desirable, it is likely not sufficient for creating "ethical AI",
While explainability is desirable, it is likely not sufficient for creating "ethical AI",
i.e. machines that can make ethical decisions. These systems will require the
i.e. machines that can make ethical decisions. These systems will require the
invention of new evaluation techniques around the AI's proficiency and competency in its own ethical reasoning. Using traditional software and system testing methods on ethical AI algorithms may not be feasible because what is considered "ethical" often consists of judgements made within situational contexts. The question of what is ethical has been studied for centuries. This symposium invites interdisciplinary methods for characterizing and measuring ethical decisions as applied to ethical AI.
invention of new evaluation techniques around the AI's proficiency and competency in its own ethical reasoning. Using traditional software and system testing methods on ethical AI algorithms may not be feasible because what is considered "ethical" often consists of judgements made within situational contexts. The question of what is ethical has been studied for centuries. This symposium invites interdisciplinary methods for characterizing and measuring ethical decisions as applied to ethical AI.
</p>
</p>
<p>
<p>
One of nine workshops for
This event is one of nine workshops for
<ahref="https://aaai.org/Symposia/Spring/sss22symposia.php"target="AAAI2022">Association for Advancement of Artificial Intelligence (AAAI) Spring Symposium</a>,
<ahref="https://aaai.org/Symposia/Spring/sss22symposia.php"target="AAAI2022">Association for Advancement of Artificial Intelligence (AAAI) Spring Symposium</a>,
21-23 March 2022,
21-23 March 2022,
organized by our collaborating colleagues at Raytheon.
and is organized by our collaborating colleagues at Raytheon.
Work building on these capabilities is presented in three sessions.
<ahref="documentation/papers/SSS-22_paper_117_TieredApproachEthicalAIEvaluationMetrics.pdf"target="_blank">A Tiered Approach for Ethical AI Evaluation Metrics</a>,
<p>
<br/>
<ahref="documentation/papers/SSS-22_paper_117_TieredApproachEthicalAIEvaluationMetrics.pdf"target="_blank">A Tiered Approach for Ethical AI Evaluation Metrics</a>,
Peggy Wu, Brett Israelsen, Kunal Srivastava, Hsin-Fu "Sinker" Wu, and Robert Grabowski.
<br/>
Peggy Wu, Brett Israelsen, Kunal Srivastava, Hsin-Fu "Sinker" Wu, and Robert Grabowski.
</p>
<p>
<i>Abstract.</i>
Advances in machine learning are enabling autonomy to operate
in environments of increasing complexity, including
scenarios with ethical concerns. For many Artificial Intelligence (AI)
systems, decisions are driven by the goal to maximize reward.
Policies may contain unintended consequences
known as reward hacking. The AI is optimizing within the
constraints defined by the domain and goals and does not
have the capability to distinguish between benign and
negative consequences beyond specifications. This paper
describes an ongoing effort to develop an application-agnostic
framework for AI systems to simulate actions, characterize
potential outcomes, and perform introspection to articulate
the motivations for action. Such a framework provides the
foundational work for higher-level ethical reasoning using
consequential and deontological ethics than other approaches
in AI ethics. This enables metrics from consequential ethics
to be used to assign ethical value of actions based on outcomes.
Simultaneously, metrics from deontological ethics
can be applied to evaluate the universality of its motivations.
A Trolley Problem -inspired maritime search-and-rescue scenario
is used to operationalize and demonstrate this framework.
</p>
</li>
</li>
<li>
<li>
<ahref="documentation/papers/SSS-22_paper_118_DoctrineEthicsCompliantAutonomyOntologicalFramework.pdf"target="_blank">Doctrine and Ethics Compliant Autonomy Using An Ontological Framework</a>,
<p>
<br/>
<ahref="documentation/papers/SSS-22_paper_118_DoctrineEthicsCompliantAutonomyOntologicalFramework.pdf"target="_blank">Doctrine and Ethics Compliant Autonomy Using An Ontological Framework</a>,
Don Brutzman, Curt Blais, Hsin-Fu "Sinker" Wu, Richard Markeloff and Carl Andersen.
<br/>
Don Brutzman, Curt Blais, Hsin-Fu "Sinker" Wu, Richard Markeloff, and Carl Andersen.
and methodologies designed to guarantee it. Our approach extends
frameworks already used by the U.S. military
to ensure human ethical and doctrinal behavior by human beings.
These have built in advantages of being able to express
complex plans and constraints, yet remaining intelligible to
humans, a requirement for ethical responsibility and liability.
To extend the framework to machines, mission constructs
are expressed using an Autonomous Vehicle Command Language (AVCL)
expressing mission actions and outcomes that
can readily be translated to runnable source code in several
programming languages. Missions written in AVCL can be
validated via translation to an RDF/OWL Mission Execution
Ontology (MEO) supporting queried proofs of ethical correctness.
MEO ensures that missions are both semantically
valid and compliant with ethical constraints. These technologies
implement a simulation, testing, and certification regime
that can serve as a foundation for human authority over and
trust in robots capable of lethal force.
</p>
<p>
This paper is dedicated to the memory of Rich Markeloff who made substantial contributions
towards our understanding, adaptation and usage of advanced Semantic Web capabilities
supporting ethical control of unmanned systems.
</p>
</li>
</li>
<li>
<li>
<p>
<ahref="documentation/papers/SSS-22_paper_122_MeaningfulMetricsDemonstratingEthicalSupervision.pdf"target="_blank">Meaningful Metrics for Demonstrating Ethical Supervision of Unmanned Systems</a>,
<ahref="documentation/papers/SSS-22_paper_122_MeaningfulMetricsDemonstratingEthicalSupervision.pdf"target="_blank">Meaningful Metrics for Demonstrating Ethical Supervision of Unmanned Systems</a>,
<br/>
<br/>
Don Brutzman and Curt Blais.
Don Brutzman and Curt Blais.
</p>
<p>
<i>Abstract.</i>
Metrics for AI are important, as illustrated by the workshop topics of interest. We note that
commonplace gaps in applied AI derive from Here are the measurements we know how to take which are too
easily over-extrapolated into conclusions of interest. In other words, such precise metrics are necessary and
appealing but may not broadly apply to general situations. We assert that necessary subsequent questions are
How do we define meaningful objectives and outcomes for a current unmanned system, How do we measure
those characteristics that indicate expected success/failure, and Once we can measure meaningful results,
how do we assemble exemplars into test suites that confirm successful completion across ongoing system life
cycles?
</p>
<p>
This discussion session seeks to find common threads among all workshop contributions that may help advance
progress on these fundamental challenges.
</p>
</li>
</li>
</ul>
</ul>
</td>
</td>
...
@@ -1392,7 +1464,7 @@ more-effective supervision of operations involving lethal or life-saving force.
...
@@ -1392,7 +1464,7 @@ more-effective supervision of operations involving lethal or life-saving force.
