He said the TTCT is protected proprietary material, so ChatGPT couldn't "cheat" by accessing information about the test on the internet or in a public database.
Of course no proprietary data at all could ever permeate through the sampling of LLM creators, well known for scrupulously following all licensing terms, and thoroughly checking that scrapped data might not be accidentally licensed.
-_____________________________________________________________________________-
The premise that a standardized test could truly measure originality or creative thinking seems dubious to me in the first place. Just because a response is common amongst humans does not suggest to me that it is uncreative or even unoriginal (strictly speaking as long as you come up with it independently it is still both).
That being said I am curious as how a Text Based Machine could have completed these portions of the test:
- Verbal tasks using verbal stimuli
- Verbal tasks using nonverbal stimuli (eg: visual input, if being extremely generous, they might have access to the "multi-modal" version of stochastic parrot 4, but this is not actually described as such in any source I could find)
- Non-verbal tasks (eg: completely visual input and output)
Until I see the methodology, I am not even sure I buy the result as advertised.