Nice approach. But there is a problem with the auto-complete feature: I do not scroll to a particular item, especially at those insane small lists, but type in the first letters. And you finish a run immediately after the item matches. So in my test I didn't see the list item at all.

Before you start an empirical test you should draft a hypothesis. Comparing the medium sized lists with the smaller one I would argue with visual entities above and below the fixation point. Longer lists makes it hard to keep track of the fixated item. To evaluate this idea you could alternate row color, or add a horizontal ruler after 5 items. This should double the graspable item count. As theoretical background I have subitizing in mind.

And keep in mind that statistics makes no sense unless you compare average values over subjects. Run the test, report values only and let subjects upload it somewhere (e.g. in this thread). In this way the bulky python-matplotlib does not need to be installed .

Heiko Tietze wrote:Nice approach. But there is a problem with the auto-complete feature: I do not scroll to a particular item, especially at those insane small lists, but type in the first letters. And you finish a run immediately after the item matches. So in my test I didn't see the list item at all.

Yes, I think after all with sorted lists this test makes no sense if you use the type-to-search feature. You will find all items immediately regardless of the list size, even in reality, so you will end up measuring noise. What makes my diagram contain not only noise is that I only typed the first letter and then searched from there, so the size still matters. I guess since this is about menu-ish lists, it would make sense to test it while not using the type-to-search feature. I don't think that can be fixed in the program (how would you? Requiring to click the item after typing already selected it doesn't help).

Before you start an empirical test you should draft a hypothesis. Comparing the medium sized lists with the smaller one I would argue with visual entities above and below the fixation point. Longer lists makes it hard to keep track of the fixated item. To evaluate this idea you could alternate row color, or add a horizontal ruler after 5 items. This should double the graspable item count. As theoretical background I have subitizing in mind.

Feel free to test that, just add self.ui.view.setAlternatingRowColors(True) somewhere.
But I think the hypothesis that the optimal list size is the one which contains as many items as you can grasp at once is a good one.
My original hypothesis was that the longer the list the better, but I think we can already conclude that that is not the whole truth.

And keep in mind that statistics makes no sense unless you compare average values over subjects. Run the test, report values only and let subjects upload it somewhere (e.g. in this thread). In this way the bulky python-matplotlib does not need to be installed .

Not sure -- if this strongly depends on the subject then you might blur the data a lot by averaging it across individual users and that might obfuscate otherwise observable effects. But it certainly makes sense to collect data; here's a version which writes the data out: http://paste.kde.org/puve2f1vp/p7kolj/raw
Depending on how different this turns out to be across subjects it might be better to put the individual points into a historgram instead of averaging them (maybe that even needs to be done with the individual data points from users, since esp. for the long list I often looked through it, and mostly found what I was looking for quite quickly but sometimes I missed it on the first scan and had to do it again and then it took much longer).
Eventually we will even need to normalize the values with the overall time it took the user to complete the test?

Greetings!

This is soooo cool! Unless I forget, I will run a literature search on this topic next week and see if we're the first people to study that empirically. If we are, it would be awesome to work on the study until it's scientifically good enough and then publish a paper on it! The scientific HCI community really loves that kind of stuff.

First we should collect some more data in order to have error bars which are not three times larger than the observed effect

scummos wrote:First we should collect some more data in order to have error bars which are not three times larger than the observed effect

Well, that would certainly be one part of "work on the study until it's scientifically good enough"