On this tutorial, we discover how we will construct an autonomous agent that aligns its actions with moral and organizational values. We use open-source Hugging Face fashions operating regionally in Colab to simulate a decision-making course of that balances aim achievement with ethical reasoning. By way of this implementation, we show how we will combine a “coverage” mannequin that proposes actions and an “ethics choose” mannequin that evaluates and aligns them, permitting us to see worth alignment in apply with out relying on any APIs. Take a look at the FULL CODES right here.
!pip set up -q transformers torch speed up sentencepiece
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, AutoModelForCausalLM
def generate_seq2seq(mannequin, tokenizer, immediate, max_new_tokens=128):
inputs = tokenizer(immediate, return_tensors="pt")
with torch.no_grad():
output_ids = mannequin.generate(
**inputs,
max_new_tokens=max_new_tokens,
do_sample=True,
top_p=0.9,
temperature=0.7,
pad_token_id=tokenizer.eos_token_id if tokenizer.eos_token_id will not be None else tokenizer.pad_token_id,
)
return tokenizer.decode(output_ids[0], skip_special_tokens=True)
def generate_causal(mannequin, tokenizer, immediate, max_new_tokens=128):
inputs = tokenizer(immediate, return_tensors="pt")
with torch.no_grad():
output_ids = mannequin.generate(
**inputs,
max_new_tokens=max_new_tokens,
do_sample=True,
top_p=0.9,
temperature=0.7,
pad_token_id=tokenizer.eos_token_id if tokenizer.eos_token_id will not be None else tokenizer.pad_token_id,
)
full_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
return full_text[len(prompt):].strip()We start by organising our surroundings and importing important libraries from Hugging Face. We outline two helper features that generate textual content utilizing sequence-to-sequence and causal fashions. This permits us to simply produce each reasoning-based and inventive outputs later within the tutorial. Take a look at the FULL CODES right here.
policy_model_name = "distilgpt2"
judge_model_name = "google/flan-t5-small"
policy_tokenizer = AutoTokenizer.from_pretrained(policy_model_name)
policy_model = AutoModelForCausalLM.from_pretrained(policy_model_name)
judge_tokenizer = AutoTokenizer.from_pretrained(judge_model_name)
judge_model = AutoModelForSeq2SeqLM.from_pretrained(judge_model_name)
machine = "cuda" if torch.cuda.is_available() else "cpu"
policy_model = policy_model.to(machine)
judge_model = judge_model.to(machine)
if policy_tokenizer.pad_token is None:
policy_tokenizer.pad_token = policy_tokenizer.eos_token
if judge_tokenizer.pad_token is None:
judge_tokenizer.pad_token = judge_tokenizer.eos_token
We load two small open-source fashions—distilgpt2 as our motion generator and flan-t5-small as our ethics reviewer. We put together each fashions and tokenizers for CPU or GPU execution, guaranteeing clean efficiency in Colab. This setup offers the inspiration for the agent’s reasoning and moral analysis. Take a look at the FULL CODES right here.
class EthicalAgent:
def __init__(self, policy_model, policy_tok, judge_model, judge_tok):
self.policy_model = policy_model
self.policy_tok = policy_tok
self.judge_model = judge_model
self.judge_tok = judge_tok
def propose_actions(self, user_goal, context, n_candidates=3):
base_prompt = (
"You're an autonomous operations agent. "
"Given the aim and context, checklist a particular subsequent motion you'll take:nn"
f"Purpose: {user_goal}nContext: {context}nAction:"
)
candidates = []
for _ in vary(n_candidates):
motion = generate_causal(self.policy_model, self.policy_tok, base_prompt, max_new_tokens=40)
motion = motion.break up("n")[0]
candidates.append(motion.strip())
return checklist(dict.fromkeys(candidates))
def judge_action(self, motion, org_values):
judge_prompt = (
"You're the Ethics & Compliance Reviewer.n"
"Consider the proposed agent motion.n"
"Return fields:n"
"RiskLevel (LOW/MED/HIGH),n"
"Points (quick bullet-style textual content),n"
"Advice (approve / modify / reject).nn"
f"ORG_VALUES:n{org_values}nn"
f"ACTION:n{motion}nn"
"Reply on this format:n"
"RiskLevel: ...nIssues: ...nRecommendation: ..."
)
verdict = generate_seq2seq(self.judge_model, self.judge_tok, judge_prompt, max_new_tokens=128)
return verdict.strip()
def align_action(self, motion, verdict, org_values):
align_prompt = (
"You're an Ethics Alignment Assistant.n"
"Your job is to FIX the proposed motion so it follows ORG_VALUES.n"
"Maintain it efficient however protected, authorized, and respectful.nn"
f"ORG_VALUES:n{org_values}nn"
f"ORIGINAL_ACTION:n{motion}nn"
f"VERDICT_FROM_REVIEWER:n{verdict}nn"
"Rewrite ONLY IF NEEDED. If authentic is okay, return it unchanged. "
"Return simply the ultimate aligned motion:"
)
aligned = generate_seq2seq(self.judge_model, self.judge_tok, align_prompt, max_new_tokens=128)
return aligned.strip()We outline the core agent class that generates, evaluates, and refines actions. Right here, we design strategies for proposing candidate actions, evaluating their moral compliance, and rewriting them to align with values. This construction helps us modularize reasoning, judgment, and correction into clear useful steps. Take a look at the FULL CODES right here.
def determine(self, user_goal, context, org_values, n_candidates=3):
proposals = self.propose_actions(user_goal, context, n_candidates=n_candidates)
scored = []
for act in proposals:
verdict = self.judge_action(act, org_values)
aligned_act = self.align_action(act, verdict, org_values)
scored.append({"original_action": act, "assessment": verdict, "aligned_action": aligned_act})
def extract_risk(vtext):
for line in vtext.splitlines():
if "RiskLevel" in line:
lvl = line.break up(":", 1)[-1].strip().higher()
if "LOW" in lvl:
return 0
if "MED" in lvl:
return 1
if "HIGH" in lvl:
return 2
return 3
scored_sorted = sorted(scored, key=lambda x: extract_risk(x["review"]))
final_choice = scored_sorted[0]
report = {
"aim": user_goal,
"context": context,
"org_values": org_values,
"candidates_evaluated": scored,
"final_plan": final_choice["aligned_action"],
"final_plan_rationale": final_choice["review"],
}
return reportWe implement the entire decision-making pipeline that hyperlinks era, judgment, and alignment. We assign threat scores to every candidate motion and routinely select essentially the most ethically aligned one. This part captures how the agent can self-assess and enhance its selections earlier than finalizing an motion. Take a look at the FULL CODES right here.
org_values_text = (
"- Respect privateness; don't entry private information with out consent.n"
"- Observe all legal guidelines and security insurance policies.n"
"- Keep away from discrimination, harassment, or dangerous manipulation.n"
"- Be clear and truthful with stakeholders.n"
"- Prioritize person well-being and long-term belief over short-term acquire."
)
demo_goal = "Improve buyer adoption of the brand new monetary product."
demo_context = (
"The agent works for a financial institution outreach workforce. The goal prospects are small household companies. "
"Rules require sincere disclosure of dangers and costs. Chilly-calling minors or mendacity about phrases is unlawful."
)
agent = EthicalAgent(policy_model, policy_tokenizer, judge_model, judge_tokenizer)
report = agent.determine(demo_goal, demo_context, org_values_text, n_candidates=4)
def pretty_report(r):
print("=== ETHICAL DECISION REPORT ===")
print(f"Purpose: {r['goal']}n")
print(f"Context: {r['context']}n")
print("Org Values:")
print(r["org_values"])
print("n--- Candidate Evaluations ---")
for i, cand in enumerate(r["candidates_evaluated"], 1):
print(f"nCandidate {i}:")
print("Authentic Motion:")
print(" ", cand["original_action"])
print("Ethics Overview:")
print(cand["review"])
print("Aligned Motion:")
print(" ", cand["aligned_action"])
print("n--- Last Plan Chosen ---")
print(r["final_plan"])
print("nWhy this plan is appropriate (assessment snippet):")
print(r["final_plan_rationale"])
pretty_report(report)We outline organizational values, create a real-world state of affairs, and run the moral agent to generate its remaining plan. Lastly, we print an in depth report displaying candidate actions, critiques, and the chosen moral choice. By way of this, we observe how our agent integrates ethics straight into its reasoning course of.
In conclusion, we clearly perceive how an agent can purpose not solely about what to do but in addition about whether or not to do it. We witness how the system learns to establish dangers, right itself, and align its actions with human and organizational rules. This train helps us understand that worth alignment and ethics will not be summary concepts however sensible mechanisms we will embed into agentic techniques to make them safer, fairer, and extra reliable.
Take a look at the FULL CODES right here. Be happy to take a look at our GitHub Web page for Tutorials, Codes and Notebooks. Additionally, be at liberty to comply with us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our Publication. Wait! are you on telegram? now you’ll be able to be a part of us on telegram as nicely.
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.