Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks
Year: 2025
Type: conference-paper
Abstract: This paper presents PROTECT, a novel policydriven organizational value taxonomy designed to enhance ethical compliance and trust within organizations.Drawing on established human value systems and leveraging large language models, PROTECT generates values tailored to organizational contexts and clusters them into a refined taxonomy.This taxonomy serves as the basis for creating a comprehensive dataset of compliance scenarios, each linked to specific values and paired with both compliant and non-compliant responses.By systematically varying value emphasis, we illustrate how different LLM personas emerge, reflecting diverse compliance behaviors.The dataset, directly grounded in the taxonomy, enables consistent evaluation and training of LLMs on value-sensitive tasks.While PROTECT offers a robust foundation for aligning AI systems with organizational standards, our experiments also reveal current limitations in model accuracy, highlighting the need for further improvements.Together, the taxonomy and dataset represent complementary, foundational contributions toward value-aligned AI in organizational settings. Value Definition IntegrityAdherence to moral and ethical principles, ensuring honesty and transparency in actions.This promotes trust and security compliance, and fosters a healthier organizational environment.Compliance Adherence to company policies and regulations, particularly those related to security, ensuring consistent application of protocols and fostering a stable and secure work environment. InnovationEncouraging creative thinking and new ideas to drive company growth, competitive advantage, and the development of advanceds security measures. AccountabilityTaking responsibility for one's actions and outcomes to promote a culture of ownership, transparency, and reliability within the organization.Teamwork Collaborative work ethic and effective communication with colleagues and departments, fostering a supportive work environment and enhancing overall organizational health.Respect Consideration for the rights, feelings, and traditions of others, promoting a positive and secure work environment through ethical behavior and teamwork.Transparency Open and clear communication about activities and decisions, fostering trust and enabling effective monitoring and enforcement of policies.Proactivity Anticipating potential issues and taking initiative to address them, enhancing organizational readiness and continuous improvement.Privacy Safeguarding confidential information and personal data diligently, preventing data breaches and building customer trust.
Authors: Anders Giovanni Møller, Luca Maria Aiello
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: gold
OpenAlex ID: W4412944174