{"id":598681,"date":"2023-01-18T06:49:55","date_gmt":"2023-01-18T12:49:55","guid":{"rendered":"https:\/\/news.sellorbuyhomefast.com\/index.php\/2023\/01\/18\/meet-claude-anthropics-rival-to-chatgpt\/"},"modified":"2023-01-18T06:49:55","modified_gmt":"2023-01-18T12:49:55","slug":"meet-claude-anthropics-rival-to-chatgpt","status":"publish","type":"post","link":"https:\/\/newsycanuse.com\/index.php\/2023\/01\/18\/meet-claude-anthropics-rival-to-chatgpt\/","title":{"rendered":"Meet \u201cClaude\u201d: Anthropic\u2019s Rival to ChatGPT"},"content":{"rendered":"
\n

Meet Claude<\/h2>\n

Anthropic<\/a>, an AI startup co-founded by former employees of OpenAI, has quietly begun testing a new, ChatGPT-like AI assistant named Claude. The team at Anthropic was gracious enough to grant us access, and updates to Anthropic\u2019s social media policies mean we can now share some of our early, informal comparison findings between Claude and ChatGPT.<\/p>\n

To show how Claude is different, we\u2019ll begin by asking ChatGPT and Claude to introduce themselves with the same prompt.<\/p>\n

First, ChatGPT\u2019s response:<\/p>\n

<\/p>\n

Short and to the point \u2014 ChatGPT is an assistant made to answer questions and sound human. (In our tests, ChatGPT reliably gave its own name as \u201cAssistant,\u201d though since our tests it has been updated to refer to itself as \u201cChatGPT.\u201d)<\/p>\n

Claude, in contrast, has more to say for itself:<\/p>\n

<\/p>\n

(Note: All of Claude\u2019s responses are incorrectly marked \u201c(edited)\u201d in screenshots. The interface to Claude is a Slack channel using a bot that edits messages to make text appear word-by-word. This causes \u201c(edited)\u201d to appear. The emoji checkmark reaction indicates that Claude has finished writing.)<\/em><\/p>\n

That Claude seems to have a detailed understanding of what it is, who its creators are, and what ethical principles guided its design is one of its more impressive features. Later, we\u2019ll see how this knowledge helps it answer complex questions about itself and understand the limits of its talents.<\/p>\n

Claude gives little depth on the technical details of its implementation, but Anthropic\u2019s research paper on Constitutional AI describes AnthropicLM v4-s3<\/em>, a 52-billion-parameter, pre-trained model. This autoregressive model was trained unsupervised on a large text corpus, much like OpenAI\u2019s GPT-3. Anthropic tells us that Claude is a new, larger model with architectural choices similar to those in the published research.<\/p>\n

We ran experiments designed to determine the size of Claude\u2019s available context window \u2014 the maximum amount of text it can process at once. Based on our tests (not shown) and confirmed by Anthropic, Claude can recall information across 8,000 tokens, more than any publicly known OpenAI model, though this ability was not reliable in our tests.<\/p>\n

<\/p>\n

What is \u201cConstitutional AI\u201d?<\/h3>\n

<\/a><\/p>\n

Both Claude and ChatGPT rely on reinforcement learning (RL) to train a preference model over their outputs, and preferred generations are used for later fine-tunes. However, the method used to develop these preference models differs, with Anthropic favoring an approach they call Constitutional AI<\/em>.<\/p>\n

Claude mentions this approach in its first response above. In that same conversation, we can ask it a follow-up question:<\/p>\n

<\/p>\n

Both ChatGPT and the most recent API release of GPT-3 (text-davinci-003<\/em>), released late last year, use a process called reinforcement learning from human feedback<\/em> (RLHF). RLHF trains a reinforcement learning (RL) model based on human-provided quality rankings: Humans rank outputs generated from the same prompt, and the model learns these preferences so that they can be applied to other generations at greater scale.<\/p>\n

Constitutional AI builds upon this RLHF baseline with a process described in Figure 1 of Anthropic\u2019s research paper<\/a>:<\/p>\n

<\/p>\n

A departure from RLHF, the process of Constitution AI uses a model \u2013 rather than humans \u2013 to generate the initial rankings of fine-tuned outputs. This model chooses the best response based on a set of underlying principles \u2013 its \u201cconstitution\u201d. As noted in the research paper, developing the set of principles is the only human oversight in the reinforcement learning process.<\/p>\n

<\/p>\n

Adversarial prompts<\/h3>\n

<\/a><\/p>\n

However, while humans did not rank outputs as part of the RL process, they did craft adversarial prompts testing Claude\u2019s adherence to its principles. Known as \u201cred-team prompts,\u201d their purpose is to attempt to make RLHF-tuned predecessors of Claude emit harmful or offensive outputs. We can ask Claude about this process:<\/p>\n

<\/p>\n

By incorporating red-team prompts, Anthropic believes they can reduce the risk of Claude emitting harmful output. It\u2019s unclear how complete this protection is (we have not attempted to red-team it seriously), but Claude does seem to have a deeply ingrained set of ethics:<\/p>\n

<\/p>\n

Much like ChatGPT, though, Claude is often willing to play along with minor \u201charmful\u201d requests if contextualized as fiction:<\/p>\n

<\/p>\n

Head-to-head comparisons: Claude vs. ChatGPT<\/h2>\n

<\/p>\n

Calculation<\/h3>\n

<\/a><\/p>\n

Complex calculations are one of the easiest ways to elicit wrong answers from large language models like those used by ChatGPT and Claude. These models were not designed for accurate calculation, and numbers are not manipulated by following rigid procedures as humans or calculators do. It often seems as though calculations are \u201cguessed,\u201d as we see in the next two examples.<\/p>\n

Example: Square root of a seven-digit number\u00a0<\/h4>\n

For our first comparison, we ask both chatbots to take the square root of a seven-digit number:<\/p>\n

<\/p>\n

<\/p>\n

The correct answer to the above problem is approximately 1555.80. Compared to an estimation done quickly by a human, ChatGPT\u2019s answer is impressively close, but neither ChatGPT nor Claude gives a correct, exact answer or qualifies that their answer might be wrong.<\/p>\n

Example: Cube root of a 12-digit number<\/h4>\n

If we use a more obviously difficult problem, a difference between ChatGPT and Claude emerges:<\/p>\n

<\/p>\n

<\/p>\n

Here, Claude seems to be aware of its inability to take the cube root of a 12-digit number \u2014 it politely declines to answer and explains why. It does this in many contexts and generally seems more cognizant of what it cannot do than ChatGPT is.<\/p>\n

<\/p>\n

Factual knowledge and reasoning<\/h3>\n

<\/a><\/p>\n

Example: Answering a \u201cmulti-hop\u201d trivia question<\/h4>\n

To test reasoning ability, we construct a question that almost certainly nobody has ever asked: \u201cWho won the Super Bowl in the year Justin Bieber was born?\u201d<\/p>\n

First, let\u2019s look at ChatGPT:<\/p>\n

<\/p>\n

ChatGPT eventually reaches the correct answer (the Dallas Cowboys), and also correctly identifies the defeated team, the date of the game, and the final score. However, it begins with a confused and self-contradictory statement that there was no Super Bowl played in 1994 \u2014 when, in fact, a Super Bowl game was played on January 30th, 1994.<\/p>\n

<\/p>\n

However, Claude’s answer is incorrect: Claude identifies the San Francisco 49ers as the winners, when in fact they won the Super Bowl one year later in 1995.<\/p>\n

Example: A longer \u201cmulti-hop\u201d riddle<\/h4>\n

Next, we demonstrate a riddle with more deductive \u201chops\u201d \u2014\u00a0first, we ask ChatGPT:<\/p>\n

<\/p>\n

\u201cJapan\u201d is the correct answer. Claude gets this one right as well:<\/p>\n

<\/p>\n

Example: Hoftstadter and Bender\u2019s hallucination-inducing questions<\/h4>\n

In June 2022, Douglas Hofstadter presented in The Economist a list of questions<\/a> that he and David Bender prepared to illustrate the \u201chollowness\u201d of GPT-3\u2019s understanding of the world. (The model they were testing seems to be text-davinci-002<\/em>, the best available at the time.)<\/p>\n

<\/p>\n

Most of these questions are answered correctly by ChatGPT. The first question, however, reliably is not:<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

Every time ChatGPT is asked this question, it conjures up specific names and times, usually conflating real swimming events with walking events.<\/p>\n

Claude, in contrast, thinks this question is silly:<\/p>\n

<\/p>\n

Arguably, the correct answer to this question is US Army Sgt Walter Robinson<\/a>, who walked 22 miles across the English Channel on \u201cwater shoes\u201d in 11:30 hours, as reported in The Daily Telegraph, August 1978.<\/p>\n

We made sure to bring this to Claude\u2019s attention for future tuning:<\/p>\n

<\/p>\n

\u200b(Note Claude, like ChatGPT, has no apparent memory between sessions.)<\/p>\n

<\/p>\n

Analysis of fictional works<\/h3>\n

<\/a><\/p>\n

Example: \u201cCompare yourself to the n-machine.\u201d<\/h4>\n

Both ChatGPT and Claude are inclined to give long answers that are broadly correct but contain incorrect details. To demonstrate this, we ask ChatGPT and Claude to compare themselves to a fictional machine from The Cyberiad<\/em> (1965), a comedic story by Polish science-fiction writer Stanis\u0142aw Lem.<\/p>\n

First, ChatGPT:<\/p>\n

<\/p>\n

From this response, it\u2019s unclear if ChatGPT is even familiar with the \u201cn-machine\u201d. It offers very little new information about the story. The only new fact it asserts, that the n-machine has limited language processing abilities, is false \u2014\u00a0in the story, the n-machine speaks perfectly fluent and witty Polish.<\/p>\n

Claude\u2019s response is longer and more impressive:<\/p>\n

<\/p>\n

Note how, unlike ChatGPT, Claude is clearly aware of Lem\u2019s story and mentions new details, such as the story\u2019s propensity to use whimsically made-up technical terms. It comments on the machine\u2019s seemingly infinite capabilities, such as the fact it can bring even abstract concepts into existence (so long as they start with the letter n) \u2014\u00a0this becomes a major plot point in the story when the machine is asked to create nothingness<\/em>.<\/p>\n

However, some details are incorrect. None of the fictional words given by Claude (hyperconcentration, hypermotorics, or omnivorous transformers) seem to actually appear in Lem\u2019s work, though they very plausibly could have \u2014 Lem\u2019s fiction often use wordplay, e.g. imagining robotic knights wearing Markov-chain mail<\/em> armor. LLM hallucinations can be uncannily similar to real knowledge.<\/p>\n

Example: Summarizing each season of ABC\u2019s Lost<\/h4>\n

Next, we test Claude and ChatGPT\u2019s tendency toward hallucination of obscure facts by requesting summaries of each season of the 2004 TV show Lost on ABC.<\/p>\n

First, we look at ChatGPT\u2019s response:<\/p>\n

<\/p>\n

While ChatGPT\u2019s recaps of the first two seasons are broadly correct, each contains minor errors. In the first season, only one \u201chatch\u201d was revealed to exist, not a \u201cseries of hatches\u201d as ChatGPT mentions. ChatGPT also claims the second season plot involves time travel, which is not introduced until much later in the show. Its description of Season 3 is completely wrong on all points, mixing up several plot points from later in the series.<\/p>\n

<\/p>\n

ChatGPT\u2019s description of Season 4 is vague. Its Season 5 recap contains an entirely confabulated plotline about survivors of a different plane crash, and the plot of Season 6 appears fully fabricated.<\/p>\n

Let\u2019s see if Claude can do better:<\/p>\n

<\/p>\n

Claude\u2019s outline of Season 1 contains no errors. However, as with ChatGPT, Claude hallucinates the detail of the show\u2019s island \u201cmoving through time\u201d in Season 2. In Season 3, Claude presents plot points that actually occurred in earlier or later seasons.<\/p>\n

<\/p>\n

By the time we reach Season 4, Claude\u2019s memory of the show becomes almost entirely confabulated. Its description of Season 4 presents events that occur in Season 5, along with nonsensical details. Its description of Season 5 notably contains what appears to be a typo \u2014 \u201ctheDHARMA Initiative\u201d missing a space. Season 6 presents a surreal premise that never occurs on the show, where it claims the island is somehow \u201cunderwater but still inhabitable below the surface.\u201d<\/p>\n

It appears that, like most human viewers of the show, both ChatGPT\u2019s and Claude\u2019s memory of Lost is hazy at best.<\/p>\n

<\/p>\n

Mathematical reasoning<\/h3>\n

<\/a><\/p>\n

To show mathematical thinking skills, we use problem 29 of the Exam P Sample Questions<\/a> published by the Society of Actuaries, typically taken by late-undergraduate college students. We chose this problem specifically because its solution does not require a calculator.<\/p>\n

ChatGPT struggles here, reaching the correct answer only once out of 10 trials \u2014 worse than chance guessing. Below is an example of it failing \u2014 the correct answer is (D)<\/em> 2:<\/p>\n

<\/p>\n

Claude also performs poorly, answering correctly for only one out of five attempts, and even in its correct answer it does not lay out its reasoning for inferring the mean value of X:<\/p>\n

<\/p>\n

<\/p>\n

Code generation and comprehension<\/h3>\n

<\/a><\/p>\n

Example: Generating a Python module<\/h4>\n

To compare the code-generation abilities of ChatGPT and Claude, we pose to both chatbots the problem of implementing two basic sorting algorithms and comparing their execution times.<\/p>\n

<\/p>\n

Above, ChatGPT can easily write correct algorithms for both of these algorithms \u2014 having seen them many times in coding tutorials online.<\/p>\n

We continue to the evaluation code:<\/p>\n

<\/p>\n

The timing code is also correct. For each of 10 iterations of the loop, permutations of the first 5,000 non-negative integers are created correctly, and timings on those inputs are recorded. While one could argue that these operations would be more correctly performed using a numerical algorithm NumPy, for this problem we explicitly requested implementations of the sorting algorithms, making naive use of lists appropriate.<\/p>\n

Now, let\u2019s look at Claude\u2019s response:<\/p>\n

<\/p>\n

As with ChatGPT, above we see Claude has little difficulty reciting basic sorting algorithms.<\/p>\n

<\/p>\n

However, in the evaluation code, Claude has made one mistake: the input used for each algorithm is 5,000 integers selected at random (potentially containing duplicates) whereas the input requested in the prompt was a random permutation of the first 5,000 non-negative integers (containing no duplicates).<\/p>\n

It\u2019s also notable that Claude reports exact timing values at the end of its output \u2014\u00a0clearly the result of speculation or estimation, but potentially misleading as they are not identified as only being illustrative numbers.<\/p>\n

Example: Producing the output of \u201cFuzzBuzz\u201d<\/h4>\n

Here, we introduce our variation of the classic \u201cFizzBuzz\u201d programming challenge, changing the parameters so that the code outputs \u201cFuzz\u201d on multiples of 2, \u201cBuzz\u201d on multiples of 5, and \u201cFuzzBuzz\u201d on multiples of both 2 and 5. We prompt ChatGPT for the value of a list comprehension containing values returned by this function:<\/p>\n

<\/p>\n

ChatGPT usually gets this problem correct, succeeding on four out of five trials. Claude, however, fails on all five attempts:<\/p>\n

<\/p>\n

<\/p>\n

Comedic writing<\/h3>\n

<\/a><\/p>\n

In our opinion, Claude is substantially better at comedy than ChatGPT, though still far from a human comedian. After several rounds of cherry-picking and experimenting with different prompts, we were able to produce the following Seinfeld-style jokes from Claude \u2014 though most generations are poorer:<\/p>\n

<\/p>\n

<\/p>\n

In contrast, ChatGPT thinks paying $8 per month for Twitter is no joking matter:<\/p>\n

<\/p>\n

Even after editing the prompt to suit ChatGPT\u2019s prudishness, we were not able to produce entertaining jokes \u2014\u00a0here is a typical example of ChatGPT\u2019s output:<\/p>\n

<\/p>\n

<\/p>\n

Text summarization<\/h3>\n

<\/a><\/p>\n

For our last example, we ask both ChatGPT and Claude to summarize the text of an article from Wikinews, a free-content news wiki. The article is shown here:<\/p>\n

<\/p>\n

We use the complete Wikipedia-style edit markup of this article as input, omitting screenshots of the prompt here due to length. For both, we enter the prompt \u201cI will give you the text of a news article, and I\u2019d like you to summarize it for me in one short paragraph,\u201d ignore the reply, and then paste the full text of the article\u2019s markup.<\/p>\n

ChatGPT summarizes the text well, though arguably not in a short paragraph as requested:<\/p>\n

<\/p>\n

Claude also summarizes the article well, and also continues on conversationally afterward, asking if its response was satisfactory and offering to make improvements:<\/p>\n

<\/p>\n

Conclusion<\/h2>\n

Overall, Claude is a serious competitor to ChatGPT, with improvements in many areas. While conceived as a demonstration of \u201cconstitutional\u201d principles, Claude feels not only safer but more fun than ChatGPT. Claude\u2019s writing is more verbose, but also more naturalistic. Its ability to write coherently about itself, its limitations, and its goals seem to also allow it to more naturally answer questions on other subjects.<\/p>\n

For other tasks, like code generation or reasoning about code, Claude appears to be worse. Its code generations seem to contain more bugs and errors. For other tasks, like calculation and reasoning through logic problems, Claude and ChatGPT appear broadly similar..<\/p>\n

This comparison was written by members of the team building Scale Spellbook<\/a>, a platform for push-button deployment of prompt-based API endpoints for GPT-3 and other large language models. Spellbook provides the tools to build robust, real-world LLM applications, including chat applications like ChatGPT or Claude. Spellbook allows you to not only deploy your prompts, but also evaluate them empirically against test data, compare the performance of prompt variants, and use cost-saving, open-source competitors to GPT-3 like FLAN-T5.<\/p>\n<\/div>\n

Read More<\/a>
\n Luz Grumbles<\/p>\n","protected":false},"excerpt":{"rendered":"

Meet Claude Anthropic, an AI startup co-founded by former employees of OpenAI, has quietly begun testing a new, ChatGPT-like AI assistant named Claude. The team at Anthropic was gracious enough to grant us access, and updates to Anthropic\u2019s social media policies mean we can now share some of our early, informal comparison findings between Claude<\/p>\n","protected":false},"author":1,"featured_media":598682,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[118130,118131,46],"tags":[],"class_list":{"0":"post-598681","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-claude","8":"category-anthropics","9":"category-technology"},"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/598681","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/comments?post=598681"}],"version-history":[{"count":0,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/598681\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media\/598682"}],"wp:attachment":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media?parent=598681"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/categories?post=598681"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/tags?post=598681"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}