One of the arguments we keep making about SurveyMan is that our approach allows us to debug surveys. We talk about the specific biases (order, wording, sampling) that skew the results, but one thing that we don’t really emphasize in the tech report is what it really means for a survey to be correct. Here I’d like to tease out various notions of correctness in the context of survey design.

Response Set Invariant

Over the past few months, I’ve been explaining the central concept of correctness as one where we’re trying to preserve an invariant: the set of answers returned by a particular respondent. The idea is that if this universe were to split and you were not permitted to break off, the you in Universe A would return the same set of answers as the you in Universe B. That is, your set of answers should be invariant with respect to ordering, branching, and sampling.

These guys should give the same responses to a survey, up to the whole timeline split thing.

Invalidation via biases Ordering and sampling can be flawed and when they are, they lead to our order bias and wording bias bugs. Since we randomize order, we want to be able to say that your answer to some question is invariant with respect to the questions that precede it. Since we sample possible wordings for variants, we want to be able to say that each question in an ALL block is essentially “the same.” We denote “sameness” by treating each question wording variant as exchangeable and aligning the answer options. The prototypicality survey best illustrates variants, both in the question wording and the option wording:

BLOCK	QUESTION	OPTIONS
_1._1	How good an example of an odd number is the number 3?	Somewhat bad
		Somewhat good
		Somewhat good
		Good
_1._1	How well does the number 3 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._1	How odd is the number 3?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._1	How odd is the number 3?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._2	How good an example of an odd number is the number 241?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._2	How well does the number 241 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._2	How odd is the number 241?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._2	How odd is the number 241?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._3	How good an example of an odd number is the number 2?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._3	How well does the number 2 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._3	How odd is the number 2?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._3	How odd is the number 2?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._4	How good an example of an odd number is the number 158?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._4	How well does the number 158 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._4	How odd is the number 158?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._4	How odd is the number 158?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._5	How good an example of an odd number is the number 7?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._5	How well does the number 7 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._5	How odd is the number 7?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._5	How odd is the number 7?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._6	How good an example of an odd number is the number 4?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._6	How well does the number 4 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._6	How odd is the number 4?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._6	How odd is the number 4?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._7	How good an example of an odd number is the number 465?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._7	How well does the number 465 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._7	How odd is the number 465?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._7	How odd is the number 465?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._8	How good an example of an odd number is the number 396?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._8	How well does the number 396 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._8	How odd is the number 396?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._8	How odd is the number 396?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._9	How good an example of an odd number is the number 1?	Bad
		Somewhat bad
_1._1	How good an example of an odd number is the number 3?	Bad
		Good
_1._9	How well does the number 1 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._9	How odd is the number 1?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._9	How odd is the number 1?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._10	How good an example of an odd number is the number 463?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._10	How well does the number 463 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._10	How odd is the number 463?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._10	How odd is the number 463?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._11	How good an example of an odd number is the number 6?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._11	How well does the number 6 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._11	How odd is the number 6?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._11	How odd is the number 6?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._12	How good an example of an odd number is the number 730?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._12	How well does the number 730 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._12	How odd is the number 730?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._12	How odd is the number 730?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._13	How good an example of an odd number is the number 9?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._13	How well does the number 9 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._13	How odd is the number 9?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._13	How odd is the number 9?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._14	How good an example of an odd number is the number 8?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._14	How well does the number 8 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._14	How odd is the number 8?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._14	How odd is the number 8?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._15	How good an example of an odd number is the number 827?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._15	How well does the number 827 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._15	How odd is the number 827?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._15	How odd is the number 827?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
_1._16	How good an example of an odd number is the number 532?	Bad
		Somewhat bad
		Somewhat good
		Good
_1._16	How well does the number 532 represent the category of odd numbers?	Poorly
		Somewhat poorly
		Somewhat well
		Well
_1._16	How odd is the number 532?	Not very odd
		Somewhat not odd
		Somewhat odd
		Very odd
_1._16	How odd is the number 532?	Not at all odd
		Somewhat not odd
		Somewhat odd
		Completely odd
2	What is your native tongue?	American English
		British English
		Indian English
		Other

Invalidation via runtime implementation Clearly ordering and sampling are typical, well-known biases that we are framing as survey bugs. These bugs violate our invariant. We have also put restrictions on the survey language in order to preserve this invariant. I discussed these restrictions previously, but didn’t delve into their relationship with correctness. In order to understand the correctness of a survey program, we will have to talk a little about semantics.

Generally when we talk about language design in the PL community, we have two sets of rules for understanding a language: the syntax and the semantics. As with natural language, the syntax describes the rules for how components may be combined, whereas the semantics describes what they “really mean”. The more information we have encoded in the syntax, the more we can check at compile time — that is, before actually executing the program. At its loosest, the syntax describes a valid surface string in the language, whereas the semantics describes the results of an execution.

When we want “safer” languages, we generally try to encode more and more information in the syntax. A common example of this is type annotation. We can think of this extra information baked into the language’s syntax as some additional structure to the program. Taking the example of types, if we say some variable foo has type string and then try to use it in a function that takes type number, we should fail before we actually load data into foo — typically before we begin executing the program.

In SurveyMan, these rules are encoded in the permitted values of the boolean columns, the syntax of the block notation, and the permitted branch destinations. Let’s first look at the syntax rules for SurveyMan.

SurveyMan Syntax : Preliminaries

Before we define the syntax formally, let us define two functions we’ll need to express our relaxed orderings over questions, blocks, and column headers:

$$ \sigma_n : \forall (n : int), \text{ list } A \times n \rightarrow \text{ list } A \times n$$

This is our random permutation. Actually, it isn’t *really* a random permutation, because we are not using the randomness as a resource. Instead, we will be using it more like a nondeterministic permutation (hence the subscript “n”). We will eventually want to say something along the lines of “this operator returns the appropriate permutation to match your input, so long as your input is a member of the set of valid outputs.” $$\sigma_n$$ takes a list and its length as arguments and returns another list of the same length. We can do some hand-waving and say that it’s equivalent to just having a variable length argument whose result has the same number of arguments, so that

$$ \sigma_n([\text{apples ; oranges ; pears}], 3) \equiv \sigma_n(\text{apples, oranges, pears})$$

and the set of all valid outputs for this function would be

$$ \lbrace \text{[apples ; oranges ; pears], [apples ; pears ; oranges], [oranges ; pears ; apples],}$$
$$\quad \text{[oranges ; apples ; pears], [pears ; apples ; oranges], [pears ; oranges ; apples]} \rbrace$$.

We are going to abuse notation a little bit more and say that in our context, the output is actually a string where each element is separated by a newline. So, if we have for example

$$\sigma_n(\text{apples, oranges, pears}) \mapsto [\text{oranges ; pears ; apples}]$$,

then we can rewrite this as

$$\sigma_n(\text{apples, oranges, pears}) \mapsto \text{oranges}\langle newline \rangle \text{pears}\langle newline \rangle \text{apples}$$,

where $$\langle newline \rangle$$ is the appropriate newline symbol for the program/operating system.

Why not encode this directly in the typing of $$\sigma_n$$? The initial definition makes the syntax confined and well-understood for the (PL-oriented) reader. Since we’re defining a specific kind of permutation operator, we want its type to be something easy to reason about. The “true” syntax is just some sugar, so we have some easy-to-understand rewrite rules to operate over, rather than having to define the function as taking variable arguments and returning a string whose value is restricted to some type. Note that I haven’t formalized anything like this before, so there might be a “right way” to express this kind of permutation operator in a syntax rule. If someone’s formalized spreadsheets or databases, I’m sure the answer lies in that literature.

The above permutation operator handles the case where we have a permutation where we can use some kind of equality operator to match up inputs and outputs that are actually equal. Consider this example: suppose there are two rules in our syntax:

$$\langle grocery\_list \rangle ::= \langle fruit\rangle \mid \sigma_n(\langle fruit\rangle,…,\langle fruit\rangle)$$
$$\langle fruit \rangle ::= \text{ apple } \mid \text{ orange } \mid \text{ pear } \mid \text{ banana } \mid \text{ grape }$$

Suppose store our grocery list in a spreadsheet, which we export as a simple csv:

apple
banana
orange

Then we just want to be able to match this with the $$grocery\_list$$ rule.

Next we need to introduce a projection function: a function to select one element from an ordered, fixed-sized list (i.e. a tuple). This will actually be a family of functions, parameterized by the natural numbers. The traditional projection function selects elements from a tuple and is denoted by $$\pi$$. If we have some tuple $$(3,2,5)$$, then we define a family of projection functions $$\pi_1$$, $$\pi_2$$, and $$\pi_3$$, where $$\pi_1((3,2,5))=3$$, $$\pi_2((3,2,5))=2$$, and $$\pi_3((3,2,5))=5$$. Our projection function will have to operator over list, rather than a tuple, since it will be used to extract values from columns and since we will not know the total number of columns until we read the first line of the input. The indices correspond to the column indices, but since columns may appear in any order, we must use the traditional permutation operator to denote the correct index:

$$\pi_{\sigma(\langle column \rangle)}(\langle col\_data \rangle_{(\sigma^{-1}(1))}, \langle col\_data \rangle_{(\sigma^{-1}(2))},…,\langle col\_data \rangle_{(\sigma^{-1}(n))})$$

Now we have some set of projection functions, $$\pi_{\sigma(\langle column \rangle)}$$, the size and values of which we do not know until we try to parse the first row of the csv. However, once we parse this row, the number of columns ($$n$$ above) and their values are fixed. Furthermore, we will also have to define a list of rules for the valid data in each column, but for column headers recognized by the SurveyMan parser, these are known ahead of time. All other columns are treated as string-valued. We parameterize the column data rules, using the subscripts to find the appropriate column name/rule for parsing.

SurveyMan Syntax : Proper

$$\langle survey \rangle ::= \langle block \rangle \mid \sigma_n(\langle block \rangle, …,\langle block \rangle)$$

$$\langle block \rangle ::= \langle question \rangle \mid \sigma_n(\langle block \rangle, …, \langle block \rangle)$$
$$\quad\quad\mid \; \sigma_n(\langle question \rangle,…,\langle question \rangle) \mid \sigma_n(\langle question \rangle, … ,\langle block \rangle,…)$$

$$\langle row \rangle ::= \langle col\_data \rangle_{(\sigma^{-1}(1))},…,\langle col\_data \rangle_{(\sigma^{-1}(n))} $$

$$\langle question \rangle ::= \langle other\_columns \rangle\; \pi_{\sigma(\mathbf{QUESTION})}(\langle row \rangle)$$
$$\quad\quad\quad\quad\quad\;\langle other\_columns\rangle\;\pi_{\sigma(\mathbf{OPTIONS})}(\langle row \rangle) \;\langle other\_columns \rangle$$
$$\quad\quad\mid\;\langle question \rangle \langle newline\rangle \langle option \rangle$$

$$\langle option \rangle ::= \langle empty\_question\rangle \; \langle other\_columns\rangle \; \pi_{\sigma(\mathbf{OPTIONS})}(\langle row \rangle) \; \langle other\_columns \rangle$$
$$\quad\quad\mid\;\langle other\_columns\rangle \;\langle empty\_question\rangle \; \pi_{\sigma(\mathbf{OPTIONS})}(\langle row \rangle) \; \langle other\_columns \rangle$$
$$\quad\quad\mid\;\langle other\_columns\rangle \; \pi_{\sigma(\mathbf{OPTIONS})}(\langle row \rangle) \; \langle empty\_question\rangle \;\langle other\_columns \rangle$$
$$\quad\quad\mid\;\langle other\_columns\rangle \; \pi_{\sigma(\mathbf{OPTIONS})}(\langle row \rangle) \; \langle other\_columns\; \rangle\langle empty\_question\rangle \;$$

$$\langle empty\_question \rangle ::= \langle null \rangle$$

$$\langle other\_columns \rangle ::= \langle null \rangle \mid \langle other\_column \rangle \; \langle other\_columns \rangle$$

$$\langle other\_column \rangle ::= \langle repeatable\_column \rangle$$
$$\quad\quad \mid\; \langle nonrepeatable\_column \rangle $$
$$\quad\quad \mid \;\langle changable\_column\rangle$$

$$\langle nonrepeatable\_column \rangle ::= \;, \;\pi_{\sigma(\mathbf{FREETEXT})}(\langle row \rangle) \mid \pi_{\sigma(\mathbf{FREETEXT})}(\langle row \rangle)$$

$$\langle repeatable\_column \rangle ::= \; , \;\pi_{\sigma(\mathbf{BLOCK})}(\langle row \rangle) \mid \pi_{\sigma(\mathbf{BLOCK})}(\langle row \rangle)$$
$$\quad\quad\mid\; , \;\pi_{\sigma(\mathbf{EXCLUSIVE})}(\langle row \rangle) \mid \pi_{\sigma(\mathbf{EXCLUSIVE})}(\langle row \rangle)$$
$$\quad\quad\mid\; , \;\pi_{\sigma(\mathbf{ORDERED})}(\langle row \rangle) \mid \pi_{\sigma(\mathbf{ORDERED})}(\langle row \rangle)$$
$$\quad\quad\mid\; , \;\pi_{\sigma(\mathbf{RANDOMIZE})}(\langle row \rangle) \mid \pi_{\sigma(\mathbf{RANDOMIZE})}(\langle row \rangle)$$
$$\quad\quad\mid\; , \;\pi_{\sigma(\mathbf{CORRELATION})}(\langle row \rangle) \mid \pi_{\sigma(\mathbf{CORRELATION})}(\langle row \rangle)$$

$$\langle changable\_column \rangle ::= , \;\pi_{\sigma(\mathbf{BRANCH})}(\langle row \rangle) \mid \pi_{\sigma(\mathbf{BRANCH})}(\langle row \rangle)$$
$$\quad\quad\mid\; , \;\pi_{\sigma(\langle user\_defined \rangle)}(\langle row \rangle) \mid \pi_{\sigma(\langle user\_defined \rangle)}(\langle row \rangle)$$

$$\langle col\_data \rangle_{(\mathbf{FREETEXT})} ::= \mathbf{TRUE} \mid \mathbf{FALSE} \mid \mathbf{YES} \mid \mathbf{NO} \mid \langle string \rangle \mid \#\;\lbrace\; \langle string \rangle \; \rbrace$$
$$\langle col\_data \rangle_{(\mathbf{BLOCK})} ::= (\_|[a-z])?[1-9][0-9]*(\setminus .\_?[1-9][0-9]*)*$$
$$\langle col\_data \rangle_{(\mathbf{BRANCH})} ::= \mathbf{NEXT} \mid [1-9][0-9]*$$
$$\langle col\_data \rangle_{(\mathbf{EXCLUSIVE})} ::= \mathbf{TRUE} \mid \mathbf{FALSE} \mid \mathbf{YES} \mid \mathbf{NO}$$
$$\langle col\_data \rangle_{(\mathbf{ORDERED})} ::= \mathbf{TRUE} \mid \mathbf{FALSE} \mid \mathbf{YES} \mid \mathbf{NO}$$
$$\langle col\_data \rangle_{(\mathbf{RANDOMIZE})} ::= \mathbf{TRUE} \mid \mathbf{FALSE} \mid \mathbf{YES} \mid \mathbf{NO}$$

Notes :

• The last match for the $$\langle block\rangle$$ rule should really be four rules — one for one question and one block, one for one question and many blocks, one for one block and many questions, and one for many blocks and many questions, but I thought that would be cumbersome to read, so I shortened it.
• All $$\langle col\_data \rangle$$ rules not explicitly listed are unconstrained strings.
• Remember, the subscripted $$n$$ in $$\sigma_n$$ is for nondeterminism; the $$n$$ indexing the column data denotes the number of input columns.
• We also accept json as input; the json parser assigns a canonical ordering to the columns, so generated ids and such are equivalent.

What’s worth noting here are the things that we intend to check on the first pass of the parser and compare them against those that require another iteration over the structure we’ve built. We would like to be able to push as much into the syntax as possible, since failing earlier is better than failing later. On the other hand, we want the syntax of the language to be clear to the survey writer; if the survey writer struggles to just get past the parser, we have a usability problem on our hands.

A good example of the limitations of what can be encoded where is the BRANCH column. Consider the following rules for branching:

1. Branch only to top-level blocks.
2. Branch only to static (i.e. non-floating) blocks.
3. Only branch forward.
4. Either branch to a specific block, or go to the next block (may be floating).

(1) and (2) are encoded in the grammar — note that the regular expression does not include the subblock notation used in the BLOCK column’s regular expression. Furthermore, the BRANCH rule will not match against floating blocks. Any survey that violates these rules will fail early.

(3) is not encoded directly in the grammar because we must build the question and top-level containing block first. Since it cannot be verified in the first pass, we perform this check after having parsed the survey into its internal representation.

(4) cannot be determined until runtime. This is an example of where our syntax and static checks reach their limits. $$\mathbf{NEXT}$$ is typically used when we want to select exactly one question from a block and continue as usual. A user could conceivably use $$\mathbf{NEXT}$$ as a branch destination in the usual branching context; the only effect this could have would be the usual introduction of constraints imposed by having that block contain a “true” branch question.

This leads into my next post, which will (finally!) be about the survey program semantics.

When we allow breakoff, we typically tell respondents that they will be paid a bonus commensurate with the amount and quality of work they do at the end of the study. We don’t typically inform them of when the study will end, since we may need to re-run surveys and we’d like to keep respondents from trying to game our algorithms. The experiments we did in the fall that awarded bonuses used a tiered system. The ones listed here are using respondent’s scores to identify the top 95% of scores and award those respondents two cents per question. I’d like to investigate more sophisticated pricing schemata with Sara in the future.

Wage Survey

It was my intention to only have one wage survey running on AMT. However, as I’ve been porting the Python analyses into Clojure, I found that I actually had three instances running. Given the expiration dates on the latter two, I’m pretty sure they were posted accidentally. I should probably consider asking the user if their sure they want to continue, or have a safe mode that asks a million times “are you sure you want to do X?” so future users don’t make this mistake. There’s also small possibility that when I extended the original HIT, it somehow spawned two new HITs instead. This isn’t documented anywhere, but it’s something I probably want to double check on sandbox.

So we had three surveys running. At the time of our OOPSLA submission, we had the wage survey running for about four days and only accrued 69 responses. I extended that HIT twice. It expired Mar 26 2014, 04:42 PM PDT. The other two HITs had expiration dates of Mar 28 2014, 04:44 PM PDT and Mar 30 2014, 07:22 PM PDT. Each HIT requested 150 assignments, and paid $0.10 base wage per survey. Between the three surveys, we collected 154 responses. Under normal circumstances, I wouldn’t have three of the same HIT running concurrently — a feature I might consider adding to SurveyMan is a check for whether a survey with similar parameters has been posted before. If I implement a “safe mode” version of SurveyMan, I could ask then ask the user if they really want to post this survey.

Anyway, the point is that because I had three versions running, I had repeaters. We had only 132 unique respondents. I typically exclude repeaters from the analyses, since we tell them to return the surveys if they haven’t taken them before. After running our new dynamic analyses report, I found that 98 respondents were classified as bad actors. I had a similarly high percentage in the Python analysis and wasn’t confident that it was correct. Since we hadn’t tested the effects of breakoff on bot classification in simulation, I was hesitant to make any strong assertions about these classifications, without investigating further. Furthermore, since we had so few data points at the time of the OOPSLA submission, we decided to simply report qualitative results.

Examining the larger data set, we found that the maximum number of questions answered by any one respondent was still 26. I’ll leave a more thorough analysis of our quality control to another blog post, but the results were quite interesting and corroborated some of my suspicions about the the original set of data. In any case, the number of questions bad actors answered had a high of 18 and a low of 2 (interestingly, those who only answered one question were confined to repeaters. Inspecting manually, I saw 10 repeaters in total, of whom only one appeared to be a legitimate bad actor.).

Using the federal minimum wage of $7.25 an hour and an estimated 10 seconds per question, we should award $0.02 per question. Since we already awarded a base pay of $0.10, we subtracted this quantity off the total payment calculated for honest respondents. The static analyses gave us an average path length of 41 questions and an expected payment of $0.825694 for each respondent who answered the survey to completion. Since we requested 150 responses, if every respondent were categorized as honest and answered the survey to completion, it would cost us $123.8541, not counting the AMT commission. AMT charges a 10% commission on both the base pay and the bonuses. This gives us a total expected cost of $136.23951.

Our actual costs were much lower (although the quality of the data was presumably also lower). Our 154 respondents cost us $22.53 in base pay, including AMT commission. We calculated $7.78 in bonuses to be awarded, costing us $8.558 for the commission. In total, the experiment under these conditions cost us $31.088.

Prototypicality

The next survey we’ll look at is Presley’s prototypicality survey. We had 149 responses total. The survey had an average path length of 17 questions. The estimated base price for honest respondents who answer the survey in full is $0.342361 per survey. Our expected cost for all honest respondents, plus AMT commission is $56.489565. We classified 65 respondents as bad actors, and 84 as valid responses. These results differ from our initial reported results due to how we calculate the frequencies for questions that are variants. In our Python code, we only compared questions that were exactly the same — that is, we didn’t unify the distributions of the variants. In the Clojure implementation, we first remove bots, and assume that there is no statistical difference between the questions, unifying all the variants. We’ll leave a discussion of the pros and cons of this approach to a future blog post.

The result of unifying variants is that significantly more responses are classified as bots, but none of the variants are flagged as being drawn from different distributions! I’ll have to double check to make sure that the variant code is running as expected, but since my unit test for flagging variants seems to work, I’m going to assume that the differences we detected were due to outliers. We can discuss what might be going on here in a later blog post, since this one is mostly about pricing. The bonuses to be paid amount to $20.32. With AMT commission this is $22.35. In total, this survey costs $38.742.

Choose Randomly

I ran a survey that wasn’t featured in our OOPSLA paper because it was one I made up entirely. The idea was to post a survey with two floating blocks, where both asked respondents to choose one of the responses randomly, but the one block had identical options, whereas the other had arbitrary categories of things. I wanted to see how random people could actually be. I also wanted to track timing information. One of the things I noticed about this survey, which I posted on a Wednesday morning, was that I was able to collect all of my responses within 90 minutes. The time per question here is clearly less than 10s. If it takes about 2 seconds per question, then the expected cost for a completed survey would be about $0.08. As a result, I decided to not award bonuses. The total cost was $16.50.

Phonology

Finally we have our classic phonology survey. With an average path length of 99 questions, we would expect to pay $1.993750 per survey. With a target of 150 responses, our total cost, including commission, would be $328.96875.

We ran this survey 3 times (not counting our preliminary run). The details of each run can be found in an earlier post. We collected 395 responses total and had 311 unique respondents. 22 respondents accounted for the 84 duplicate responses. This initial run cost us $43.45. 182 responses were classified as valid responses. The total bonuses to be paid were calculated to be $327.96. Factoring in AMT commission, this comes to $360.756. In all, this survey will have cost us $404.206.

Notes on timing

Some of the the timing data we returned was flawed, so we couldn’t use that information to improve our payment scheme. One possible use of this information would be to use it as a proxy for the difficulty the question and vary the payment in accordance with this information.