Models move so fast I question if papers like this are worth much. Out of curiosity I ran the clocks through Gemma-2-27B
it did ok. Pretty consistently it would mix up the hour/minute hand but was decent at actually saying what they were pointing at. It never acknowledged the second hand and sometimes mistook that for the hour hand
Interestingly it got the roman numeral one spot on. (minus the seconds) I wonder if it had more training data with that style clock.
Interestingly, one of the suggestions for people who want to have lucid dreams (and I have tested it) is to look at a clock. In a dream it will either be nonsense or unverifiable.
They have trouble keeping track of the day of the week as well. I chat with one every evening, going over my next day's schedule. It loses track of what day it is pretty much daily.
I found out fairly quickly. I had a little argument about what time it was in London. I had supposed that time-zone conversion would not be a problem. However, LLMs are knowledge-bases frozen in time, trained up on a certain corpus of material and then isolated, and it was beyond their competence to actually know the current time-of-day. Sure enough, various answers were given that did not really correlate to anyone's time of day, much less correct time zones.
Well, if you asked someone in a windowless room what date/time it is they wouldn't know either, but if they have access to an online calendar/clock, they'd be able to tell you. And as you figured out, the LLM you interacted with is "in a windowless room", but that's common knowledge.
AI wouldn't even need any AI powers to be able to repeat that it's currently 00:26 in London.
Models move so fast I question if papers like this are worth much. Out of curiosity I ran the clocks through Gemma-2-27B
it did ok. Pretty consistently it would mix up the hour/minute hand but was decent at actually saying what they were pointing at. It never acknowledged the second hand and sometimes mistook that for the hour hand
Interestingly it got the roman numeral one spot on. (minus the seconds) I wonder if it had more training data with that style clock.
Interestingly, one of the suggestions for people who want to have lucid dreams (and I have tested it) is to look at a clock. In a dream it will either be nonsense or unverifiable.
I think it would be better to link to the original study [0] instead of a blog post about it.
[0] https://arxiv.org/abs/2502.05092
They have trouble keeping track of the day of the week as well. I chat with one every evening, going over my next day's schedule. It loses track of what day it is pretty much daily.
I found out fairly quickly. I had a little argument about what time it was in London. I had supposed that time-zone conversion would not be a problem. However, LLMs are knowledge-bases frozen in time, trained up on a certain corpus of material and then isolated, and it was beyond their competence to actually know the current time-of-day. Sure enough, various answers were given that did not really correlate to anyone's time of day, much less correct time zones.
Well, if you asked someone in a windowless room what date/time it is they wouldn't know either, but if they have access to an online calendar/clock, they'd be able to tell you. And as you figured out, the LLM you interacted with is "in a windowless room", but that's common knowledge.
AI wouldn't even need any AI powers to be able to repeat that it's currently 00:26 in London.
You forgot about BST!