Evaluation Benchmark Details

[praeclarumjj3]

Hi, thanks for your work!

Do you plan to release the code and data used during the evaluation, particularly the question-answer pairs for the Q&A, ground truths for event summarization, and the multimodal dialogue generation tasks?

I looked at the LoCoMo dataset release, and it only contains the JSONs corresponding to the 50 conversations. Please let me know if I missed something in those JSONs.

[yhshu]

Thank the authors for the great work! I'd also like to ask if you have plans to release the full benchmark.

[LeonNerd]

+1

[lightislost]

+1

[deadpool66]

+1

[adymaharana]

Hi everyone,

Thank you so much for your patience! We are happy to know that our work has been of interest. We have released our dataset with annotations; please see data/locomo10.zip in this repository for the evaluation benchmark that is released as part of the ACL 2024 version of our paper. We sub-sampled our previous release of 50 conversations to retain the longest conversations (see details in Note). We will be updating the Arxiv paper with the results on this subset in the following week and also releasing the code for evaluating open-source and closed-source LLMs on all tasks in LoCoMo. Please let me know if you face any issues or discrepancies in the data (and the code release in the following week).