User research: Insights from our conversations with model users

Through conversations with model users, we are trying to answer the following questions about

1. Where language models are being used today

2. How effective are they? How are you measuring their efficiency?

3. How are you ranking or choosing the best model for a task?

Existing models are not capable enough for enterprise usage

Way forward

1. We need to identify the tasks that need to be evaluated

2. Evaluation should grow out of academic benchmarks to adopt more on group requirements- solved for by identifying the right metric for each task.

3. Evaluation quality is connected largely to the evaluation datasets and maintaining it.

Open Questions

1. In your work or industry, where are you currently using language models? What specific tasks are these models performing for you?

2. What aspects of the models' performance are your users responding well to, and what aspects are causing challenges? Have you encountered any recurring issues like inaccuracies, biases, or hallucinations?

3. When selecting a language model for your tasks, what key features or capabilities do you look for? Are there any particular strengths or weaknesses in current models that significantly influence your choice?

4. What factors are most important to you when choosing a model - for example, open-source availability, multilingual capabilities, or specific performance metrics? How do you prioritise these factors?

5. How do you define and measure "success" for the tasks where you're using language models? What metrics or outcomes are most critical for you?

6. When evaluating different language models, which performance metrics do you consider most important? How do you balance trade-offs between different metrics for various tasks?

7. Could you describe 3 specific tasks you use language models for, and outline what characteristics make a model particularly suitable or unsuitable for each of those tasks?

8. Based on your experience working with language models, what additional factors or considerations do you think are important when choosing between different models? Is there anything else you'd like to share that could help in developing better model selection criteria?

Academic research: A literature survey on current Indic models, existing solutions to evaluations and how we can be better?

Through conversations with model builders, we are trying to answer the following questions about

1. Where can we improve Indic models to make the most population scale impact in India?

2. What are the capabilities of Indic models today- what do we need to get better at? How do we get better performing models?

3. How are we measuring the performance of models today- how effective are they for Indian languages, multimodality and other needs of use cases?

4. How can we make these evaluations more application, use case and real world value driven?

We are looking for help to answer these questions!

[WG] White paper on the present and future of Indic models

Contribute: Volunteer and Open roles