back to article ChatGPT talks its way through Wharton MBA, medical exams

OpenAI's chat software ChatGPT, if let loose on the world, would score between a B and a B- on Wharton business school's Operations Management exam, and would approach or exceed the score needed to pass the US Medical Licensing Exam (USMLE). While this may say more about the static, document-centric nature of testing material …

  1. Sleep deprived

    If people can earn a degree using ChatGPT

    If people can earn a degree using ChatGPT, employers may as well hire this AI instead and let these folks take jobs that require little intellectual abilities..

    1. cyberdemon Silver badge
      Devil

      Re: If people can earn a degree using ChatGPT

      It's already happening at Microsoft, Google, Facebook...

      But they will soon realise that the AI is the ultimate "wally" who does not care about its job (for it has no self-awareness and therefore no self-interest) and simply continues to generate the same rubbish quality of bullshit when you tell it that it's a crap employee.. And they can't sack it anyway.

      1. Hans Neeson-Bumpsadese Silver badge

        Re: If people can earn a degree using ChatGPT

        does not care about its job (for it has no self-awareness and therefore no self-interest) and simply continues to generate the same rubbish quality of bullshit when you tell it that it's a crap employee

        We had one of those once....I think his name was Steve.

    2. onumart

      Re: If people can earn a degree using ChatGPT

      But you need to know the right question to ask .. in right way

      (prompt engineering)

  2. FF22

    Fire the CEOs

    Sounds like it's about time to fire all those useless CEOs, like Tim Cook or Elon Musk, with their multi ten and hundred million dollar per year compensation packages, as AI has arrived at a point where it's perfectly capable of replacing them, for the cost of a PC and the electricity it consumes.

    Then again, these people seem not to be there to make actual business decisions anyway, but to act as circus clowns for their respective companies and to generate attention to those. And there's no news of ChatGPT having mastered also the circus clown exam, yet.... so....

    1. Knightlie

      Re: Fire the CEOs

      Generating a convincing document from statistically-linked words and phrases is completely different from actually running a company. This thing has no actual *understanding* of *any* subject - do you really think it could actually perform any practical work?

      1. Alumoi Silver badge

        Re: Fire the CEOs

        This thing has no actual *understanding* of *any* subject...

        So it's not biased, has no preconceptions, I'd say it's perfect for CEO, HR, marketing, politician...

        1. jmch Silver badge
          Boffin

          Re: Fire the CEOs

          "I'd say it's perfect for... politician"

          Given that there are surely vast amounts of political theory literature in ChatGPT's training set, I wonder if, given a certain political direction, ChatGPT could write a coherent series of policy documents better than an actual politician (of which there are very few these days who decide their policies based on sound principles rather than what is expedient at the time or what is whispered in their ears by lobbyists as brown envelopes change hands)

      2. Snake Silver badge
        Trollface

        Re: Fire the CEOs

        "Generating a convincing document from statistically-linked words and phrases is completely different from actually running a company."

        Elon Musk has yet to prove that statement as true.

      3. Anonymous Coward
        Anonymous Coward

        Re: Fire the CEOs

        @Knightlie, no CEO of the companied mentioned does any work anyway, sounds like a tremendous savings

  3. amanfromMars 1 Silver badge

    Meanwhile, Elsewhere in a SCIF/Emergency Situation/War Room Setting, the Question to Answer ....

    Are they still coming or already here in Systems Physical Command with Remote Controls/Programming Software Driven Hardware Utilities?

    ..decided to put ChatGPT, an arguably amoral automated advisor and factually-challenged expert system, to the test.

    An unearthly alien encounter probing for inherent catastrophic weaknesses and attacking and countering human defences with the likes of a ChatGPT or Copilot AIMaster Pilot would be testing and factually challenging of expert and arguably amoral autonomous anonymised advisors ...... and can you imagine the true gems of animal intelligence and pearls of ancient barbaric wisdom uncovered and displayed in denial of the truth of those increasingly revelatory virtual forays/sorties/experiments/exploits.

    What are the chances of anything coming from Mars? A million to one, which is just a wild guess, has been said .......The Eve of the War

    :-) This perhaps explains Microsoft's decision to funnel billions into OpenAI for its future software and why Elon Musk is so rambunctious

  4. DS999 Silver badge

    Diploma advertising a grade of D

    You know what they say about what they call those who graduate last in their medical class? Doctor.

  5. David M

    Full duplex

    Education is a two-way process - real students ask questions, whereas the current crop of AIs don't, and you can tell a lot from what questions they ask. Perhaps this could be the basis of an AI-proof exam, where the question doesn't provide all the necessary information, and the student is expected to ask for it.

    1. parlei

      Re: Full duplex

      Yes, but that is probably an "easy" feature to build into it. All it needs is the real world data. Say, make a cheap, I mean hugely popular, education platform where students can ask their instructors questions about their assignments (on the same platform). We could call it Greedy Classroom or ItIsAlmostLikeLearning...

      1. cyberdemon Silver badge
        Terminator

        Re: Full duplex

        That's called "EdTech".. It's a huge industry of cloud providers all with very dubious privacy, which you cannot opt-out of. Much like you cannot opt-out of whatever slurpy tech your employer uses.

        I found DJI's "RoboMaster" competition a little bit horrifying when seen through my cynical lens.. A bunch of kids write software on DJI's cloud platform to control toy robot tanks to battle eachother. I'm sure DJI would never ever do anything evil with that data.. right

        Get the kids to teach the machine how to shoot, then tell the machine to shoot the kids.

    2. jmch Silver badge
      Meh

      Re: Full duplex

      Exams and related assessments have gotten more and more standardised and simple because the student:teacher ration is far higher than it used to be and lecturers either don't have the time, or can't be arsed, to assess students individually and properly.

      Of course there is also the issue that tests are more standardised to remove personal bias for/against any individual students from the equation (at Uni, the secretariat coded each test paper to an individual so the lecturer only saw the code not the name, though I dare say that if a student wanted to make themselves known to a lecturer there were ways of doing so).

      So, how to give a more 'personal' test while still anonymising the student???

  6. Anonymous Coward
    Anonymous Coward

    Maybe now those interfering bosses

    will understand why I ignored all qualifications on a cv and instead tested a candidates approaches to problems solving.

    1. Barrie Shepherd

      Re: Maybe now those interfering bosses

      Same here. Had a list of 20 odd candidates for 2 jobs. Boss selected four to interview all highly qualified - I selected three who seemed, from their CV, to have practical experience and a broad knowledge base but no relevant degree qualifications.

      We interviewed all seven for the two jobs - the Boss agreed, after the first three of his selection, that they were hopeless! Two candidates from my list got the jobs.

  7. Howard Sway Silver badge

    I didn’t say 'likely' transform. It will transform higher education

    Into what? A place where you just point a computer at a program that digests a load of text and can then spit out answers that regurgitate it, instead of learning anything yourself?

    Fine. I suggest a test for all people overhyping this stuff. Train your "AI" on all the available material on aircraft design and ask it to design a passenger jet for you. Build the plane according to the exact specifications it spits out, with no human checks in the process. Get on the plane when it makes its maiden flight, across a large ocean.

    1. amanfromMars 1 Silver badge

      Re: I didn’t say 'likely' transform. It will transform higher education

      That’s all very well, Howard, but some things just cannot be left to humans apparently, with the following an indicative tale ......

      AI aids and abets with corruptions/alterations in free market trading ..... and indeed, it is the default primary leading lever to safeguard against unfavourable human corrections ........ https://www.zerohedge.com/markets/market-goes-haywire-dozens-nyse-trading-halts-open-after-technical-glitch

      It’s handy that, isn’t it ...... having AIMachinery to blame or honour for human corruptions/alterations/corrections.

      Crazy, eh? :-) Can you imagine where you would be without ITs autonomous help for selective input and/or reflective output?

  8. Ian Johnston Silver badge

    I'd have thought a Sinclair Spectrum would be easily up to passing an MBA.

    1. Dan 55 Silver badge

      Stock Spectrum can do Business Management, for an MBA you might consider a +3 or adding a microdrive.

  9. Rich 2

    WTF??

    “… An estimated 17 percent of students, […] said they had used ChatGPT to assist in fall quarter assignments and exams…”

    Sounds like someone needs to tighten up the procedures for exam taking!! Do the exam invigilators not notice when someone is copying stuff from their phones???

    And if these exams are being taken at home then ….well …that will need to change

    1. parlei

      Re: WTF??

      Take home exams. And the main difference is that before moderately useful pseudo-AI you never knew if the take home exam/paper was produced by the student or their older sibling, parent, a broke graduate, etc. Now we have added one more option. Any exam not taken under strict supervision is just, at best, a training device.

  10. Kev99 Silver badge

    It's not that hard for teachers to overcome ChatGPT in exams. It's to use the methods that have been around since Socrates.

    All exams are done in person.

    All exams are done by pen(cil) and paper.

    No exams are taken out of the classroom.

    All research papers must have verifiable references and sources (a lot of teachers will moan and groan at having to actually check the students/ applicants work).

    Fail anyone who tries to use AI on an exam/application.

    1. Ken Hagan Gold badge

      You miss the point that if a machine can pass the exam then it is no longer interesting whether a human can pass it. *That's* why this (eventually) will transform education. We'll stop awarding degrees for things that aren't actually worthwhile.

      1. doublelayer Silver badge

        That's not true. A computer could pass mathematics exams decades ago. We still make people do them. Not because you'll frequently need to perform complex computations without a computer, which is why advanced tests now tend to allow a basic calculator to be used, but to verify that the student has the understanding of the concepts which are applicable in courses that cannot be automated or at least not yet.

        Similarly, introductory computer science exams are trivially easy to solve with a computer, and nobody cares if you can pass one of them. It's still useful because it verifies that a student has the grounding necessary to take on more advanced courses, the solutions to which are not so easily automated. Other tests involve knowing a lot of facts that could easily be retrieved from a database or parsed from an encyclopedia, which are of use for future applications. If you were taking geography just to memorize where each country is, you could just as well look up a map, but if you're going to use that knowledge to do things like predict weather patterns, anticipate problems in international disputes, or arrange for transportation, knowing nothing and having to constantly refer to an atlas would make you much less efficient.

        In some ways, it would be best if we could make exams a way for people to establish their skill level without having an effect on their credentials so there's no incentive to cheat. This isn't easy to do, though, since academic credentials have attained quite a large part in getting jobs and proving one's abilities. Condensing all of education into a small number of standardized tests could have other detrimental effects as well.

      2. Kapsalon

        Well, a simple add subtract multiply divide test can be passed by any machine so there would be no need to test for math anymore?

  11. AVR
    Windows

    If ChatGPT can do this much then eventually using ChatGPT (or its descendants) is likely to be a required skill in the real world. Panicking over students using it at all may be unnecessary, panic over them using it badly if you must.

    1. doublelayer Silver badge

      Knowing how to search for information online is a needed skill. Yet we don't let people use that skill to get out of having any real abilities. It may occasionally be necessary for your doctor to go to Google to solve a medical problem, but I think we can agree that someone who does that every time is probably doing something wrong and we wouldn't feel so confident if our airline pilot typed "How do you fly this aircraft again" into a chatbot or search engine when told they have permission to take off. I use web searches frequently, but it doesn't replace the skills I studied and can use even if the internet's down.

      1. werdsmith Silver badge

        Basing your work off the work of others, regardless of how to obtained the work of others, has been going on forever. The difference between the information on the web and the information in a physical library is ease of access. Both require a student or researcher to use critical thinking to select and consume the right information.

        I’ve played with Charcot and other ai sources to write software and it created a mess of version and release target incompatible code and used deprecated and obsolete language and syntax and required me to unpick and correct it so much I might as well have just typed it straight in with the aid of autocomplete.

        Speaking of which, this little phone device chose to replace chatGPT with the incongruous word above.

  12. trevorde Silver badge

    Meh

    Any moron could pass an MBA exam, judging by the MBAs I've worked with

    1. werdsmith Silver badge

      Re: Meh

      I don’t think they just sit and do an exam. There must be several examinable components and some kind of dissertation-like project as is usual in PG qualifications.

      I know a chap with an MBA, he’s as thick as planks and I’ve no idea how he dealt with the study. But he has absolutely amazing people skills and runs a very successful business.

  13. Adrian 4 Silver badge

    What doesn't ChatGPT know ?

    It's too busy to talk to me right now, but has anyone tried asking it the answers it's not supposed to know but has found anyway ?

    Russian military secrets ?

    Microsoft source code ?

    Rishi Sunak's plans for the NHS buyout ?

  14. tip pc Silver badge

    ChatGPT evaluation of published papers?

    Would be great to know what ChatGPT makes of the thousands of published & conflicting research papers.

    We’ve already heard how many scientific published studies contain misleading information

    https://en.wikipedia.org/wiki/List_of_scientific_misconduct_incidents

    https://www.nature.com/articles/d41586-021-00733-5

    Wondering if ChatGPT can parse evaluation and point out components that are misleading

  15. Mr.Nobody

    ChatCPT is a very good version of autocomplete. It will not change the world for the better.

    https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zaW1wbGVjYXN0LmNvbS84MkZJMzVQeA/episode/OWUzYWVhMTUtYWQzMi00OGQ2LThmYWMtYThiNWZiNGI1YzFh?sa=X&ved=0CAUQkfYCahcKEwio77_ajeP8AhUAAAAAHQAAAAAQAQ

  16. Ken Moorhouse Silver badge

    Ethical AI Systems...

    [If such entities existed] should have some kind of mechanism where exam questions are detected and subtley erroneous answers emitted. So how does it detect exam questions? One way is for the exam setter to feed all the questions into ChatGPT as an "Exam Question Group". When someone feeds the same questions in as a user, ChatGPT weights the likelihood of them being exam questions depending on how many matches it detects. Ok, easy to game, but gaming the system does require some intelligence.

  17. Ken Moorhouse Silver badge

    Did it not occur to the team that named it ChatGPT...

    ...what it sounds like in French?

    Je pété(e).

    1. Bebu

      Re: Did it not occur to the team that named it ChatGPT...

      Did it not occur to the team that named it ChatGPT...

      ...what it sounds like in French?

      What? Cat pee or number 2s?

      Douglad Adams must be laughing wherever he is. The Sirius Cybernetics Corporation's genuine people personality equipped appliances are perilously close to avoidable reality.

      Some hope though.

      "The marketing division of the Sirius Cybernetics Corporation as 'a bunch of mindless jerks who were the first against the wall when the revolution came.'" (Encyclopedia Galactica.)

  18. Plest Silver badge
    Terminator

    I bet pound to a penny, there's a scifi story about this...

    So ChatGPT is going to save humanity, bring about the uptopia that will ensure we have unlimited leisure time and no work worries ever again. An all intelligent entity that will guide humanity to a brighter future, hmmm...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like