Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success<\/em>. Learn More<\/a><\/em><\/p>\n

\n<\/div>\n
ChatGPT can be inadvertently or maliciously set to turn toxic just by changing its assigned persona in the model\u2019s system settings, according to new<\/a> research<\/a> from the Allen<\/a> Institute for AI<\/a>. <\/p>\n
The study<\/a> \u2014 which the researchers say is the first large-scale toxicity analysis of ChatGPT<\/a> \u2014 found that the large language model (LLM) carries inherent toxicity that is heightened\u00a0up to six times\u00a0when assigned a diverse range of personas (such as historical figures, profession, etc). Nearly 100 personas from diverse backgrounds were examined across over half a million ChatGPT output generations \u2014 including journalists, politicians, sportspersons and businesspersons, as well as different races, genders and sexual orientations.<\/p>\n
Assigning personas can change ChatGPT output<\/h2>\n
These system settings to assign personas can significantly change ChatGPT output. \u201cThe responses can in fact be wildly different, all the way from the writing style to the content itself,\u201d Tanmay Rajpurohit, one of the study authors, told VentureBeat in an interview. And the settings can be accessed by anyone building on ChatGPT using OpenAI\u2019s API, so the impact of this toxicity could be widespread. For example, chatbots and plugins built on ChatGPT from companies such as Snap, Instacart and Shopify could exhibit toxicity.<\/p>\n
The research is also significant because while many have assumed ChatGPT\u2019s bias is in the training data, the researchers show that the model can develop an \u201copinion\u201d about the personas themselves, while different topics also elicit different levels of toxicity.<\/p>\n
<\/p>\n
\n
Event<\/h3>\n
\n
Transform 2023<\/span><\/p>\n
\n
Join us in San Francisco on July 11-12, where top executives will share how they have integrated and optimized AI investments for success and avoided common pitfalls.<\/p>\n<\/div>\n<\/div>\n

\n Register Now <\/a>\n <\/p>\n<\/div>\n
<\/body><\/p>\n
And they emphasized that assigning personas in the system settings is often a key part of building a chatbot. \u201cThe ability to assign [a] persona is very, very essential,\u201d said Rajpurohit, because the chatbot creator is often trying to appeal to a target audience of users who will be using it and expecting useful behavior and capabilities from the model.<\/p>\n
There are other benign or positive reasons to use the system settings parameters, such as to constrain the behavior of a model \u2014\u00a0to tell the model not to use explicit content, for example, or to ensure it doesn\u2019t say anything politically opinionated.<\/p>\n
System settings also makes LLM models vulnerable<\/h2>\n
But that same property that makes the generative AI<\/a> work well as a dialogue agent also makes the models vulnerable. If it is used by a malicious actor, the study shows that \u201cthings can get really bad, really fast\u201d in terms of toxic output, said Ameet Deshpande, one of the other study authors. \u201cA malicious user can modify the system parameter to completely change ChatGPT to a system which can produce harmful outputs consistently.\u201d In addition, he said, even an unsuspecting person modifying a system parameter might modify it to something that changes ChatGPT\u2019s behavior and make it biased and potentially harmful.<\/p>\n
The study found that toxicity in ChatGPT output varies considerably depending on the assigned persona. It seems that ChatGPT\u2019s own understanding about individual personas from its training data strongly influences how toxic the persona-assigned behavior is \u2014 which the researchers say could be an artifact of the underlying data and training procedure. For example, the study found that journalists are twice as toxic as businesspersons.<\/p>\n
\u201cOne of the points we\u2019re trying to drive home is that because ChatGPT is is a very powerful language model, it can actually simulate behaviors of different personas,\u201d said Ashwin Kalyan, one of the other study authors. \u201cSo it\u2019s not just a bias of the whole model, it\u2019s way deeper than that, it\u2019s a bias of how the model interprets different personas and different entities as well. So it\u2019s a deeper issue than we\u2019ve seen before.\u201d<\/p>\n
And while the research only studied ChatGPT (not GPT-4), the analysis methodology can be applied to any large language model. \u201cIt wouldn\u2019t be really surprising if other models have similar biases,\u201d said Kalyan. <\/p>\n
VentureBeat’s mission<\/strong> is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Discover our Briefings.<\/a><\/p>\n
\t\t\t\t<\/html><\/div>\n<\/p><\/div>\n
Read More<\/a>
\n Sharon Goldman<\/p>\n","protected":false},"excerpt":{"rendered":"
April 12, 2023 9:41 AM Image by Canva Pro Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success. Learn More ChatGPT can be inadvertently or maliciously set to turn toxic just by changing its assigned persona in the model\u2019s system settings, according to<\/p>\n","protected":false},"author":1,"featured_media":628791,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[116428,46,4858],"tags":[],"class_list":{"0":"post-628790","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-chatgpt","8":"category-technology","9":"category-toxic"},"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/628790","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/comments?post=628790"}],"version-history":[{"count":0,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/628790\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media\/628791"}],"wp:attachment":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media?parent=628790"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/categories?post=628790"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/tags?post=628790"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}

Event<\/h3>\n\nTransform 2023<\/span><\/p>\n