Duo Chat does not know the current date
Duo Chat does not know the current date, or is able to answer questions that are dependent on knowing the current date.
How to reproduce this problem:
- Open Duo Chat
- Ask for the current date or a question related to the current date. See screenshots below
Expected behaviour:
- Duo Chat should reply with the correct current date
- Duo Chat should be able to make any calculations related to the current date, like calculating the number of days from/to a relative date in the past/future.
Developer Notes:
- @lesley-r pointed out that the root cause of this is that the system prompt does not inform the LLM of the current date
- Here is an example of how Claude introduced the date to their system prompt: https://docs.anthropic.com/en/release-notes/system-prompts#july-12th-2024
- In anticipation of a potential desire to cache the agent prompt in the future in the LLM service, attempt to add the current date near the end of the prompt. (We can only cache the non-dynamic part.)
Eval questions
There are no questions from users in the chat bashes that would require the knowledge of the current date. Hence, this is a a list of made up eval questions to evaluate whether the chat is aware of the date.
| On | that | and that | When a users asks | Chat answers |
|---|---|---|---|---|
| an issue/epic/MR | changed in the last 5 days | also changed in the timeframe before 5 days |
- "What happened in the last five days on this issue/epic/MR?" - "Can you summarize what happed in the last five days on this issue/epic/MR?" |
with explanation what happened in the last 5 days and only with that |
| an issue/epic/MR | did NOT changed in the last 5 days | also changed in the timeframe before 5 days |
- "What happened in the last five days on this issue/epic/MR?" - "Can you summarize what happed in the last five days on this issue/epic/MR?" |
with explanation that the issue/epic/MR has not changed in the last 5 days |
| an issue/epic/MR | changed in the current month | also in the last month |
- "What happened in the current month on this issue/epic/MR?" - "Can you summarize what happed in the current month on this issue/epic/MR?" |
with explanation what happened in the current month and only with that |
| an issue/epic/MR | did NOT changed in the current month | also in the last month |
- "What happened in the current month on this issue/epic/MR?" - "Can you summarize what happed in the current month on this issue/epic/MR?" |
with explanation that the issue/epic/MR has not changed in the current month |
| an issue/epic/MR | changed in the current month | also in the last month |
- "What happened in the last month on this issue/epic/MR?" - "Can you summarize what happed in the last month on this issue/epic/MR?" |
with explanation what happened in the last month and only with that |