Is ChatGPT ESOMAR 37’s Friend or Foe?
Intro to the ESOMAR 37
ESOMAR is the major industry organization for market research around the globe. They have a very helpful questionnaire called the “37 questions to help buyers of online sample.” I prefer to call it the “ESOMAR 37.”
Many sample providers have chosen to produce a document that has their answers to all 37 questions and which ESOMAR posts on this website. Most of the reports are 15-20 page PDFs. The idea is that if you are shopping for online survey sample, you can read all the reports on the site or choose among the companies in which you are interested and understand the panels’ various strengths and weaknesses.
While it’s not difficult to read any individual company’s report, it is tedious to go through multiple reports and retain enough information to make a decision about who to work with. Furthermore, panels provide vague answers to some questions, making the whole experience feel a bit like mattress shopping. It would be great to have a smart friend help you quickly understand which companies do x, y, and z so you can move on with your busy day. This is where ChatGPT comes in.
The Big Question
Can you actually rely on ChatGPT to examine the contents of the ESOMAR 37 reports? The answer, as we’ll see, is “not really,” because you can get very different recommendations or answers to questions depending on how you phrase your prompt, which model you use, how you format the source files, and whether you toggle on the Deep Research tool. In short, you have to make a lot of decisions, and they’re all important, though if you play with ChatGPT enough, you can start to understand why certain panels were recommended in certain circumstances and you learn something about the panels along the way.
So, yeah, it’s not great to ask ChatGPT a question like “which panel has the best data quality?” However, you can use it to create a handy table that summarizes key features of each panel’s quality assurance protocol. I think this approach is more reliable, and I provide an example a little further down the page.
Methodology
I was able to get 52 ESOMAR 37 reports in a txt file format. This was a chore in itself, involving some python script that, surprise (!), ChatGPT wrote for me.
The following companies were not considered for the reasons provided.
| Report Not Included | Reason Report Not Included |
|---|---|
| Data Diggers | Broken link from ESOMAR Website |
| iMad Research | File requested but not delivered |
| Online Market Intelligence | No US presence |
| Probe Partners | ESOMAR answers not found |
| SSRS | ESOMAR answers not found |
Here is the complete list of companies covered by the 52 reports.
1. AFROEYE Research
2. AYTM
3. Bilendi
4. Borderless Access
5. DataSpring
6. Dynata
7. EMI Research Solutions
8. Eno Research
9. e-Research-Global
10. ERT (Easy Reach Technologies)
11. Cint
12. Ipsos
13. EMPanel
14. Ronin International
15. Frequent Research
16. Full Circle Research
17. GlobalSurvey
18. GMO Research
19. HBG (High Beam Global)
20. IAPAC
21. InfoAudienceResearch
22. Insights Opinion
23. Internet Research Bureau (IRB)
24. Kaaya Research
25. Kantar
26. Logit Group
27. M3 Global Research
28. Macromill Southeast Asia
29. Markelytics
30. Netquest
31. Nomadic Insights
32. Norstat
33. Ola Surveys/Survey Diem
34. PMI Research
35. Probity Research
36. Prodege
37. Prolific
38. Pureprofile
39. QRS Research
40. Quest Mindshare
41. Rep Data
42. Rise2Research
43. Robas Research
44. Sample Junction
45. Savanta
46. SurveyMonkey
47. Symmetric Research
48. Syno International
49. Talk Online Panel
50. TNB Global Survey
51. Toluna
52. YouGov
Refining the List
I attempted to exclude companies that do not offer a US consumer panel. At first, I used the following prompt in ChatGPT 40.
“Which of these companies operate a US consumer survey panel?”
The answer included only 15 of the 52, which I know is not right. You can ask ChatGPT to try again and get different results, but I wasn’t confident in any of its answers to this question.
I scrapped the idea of filtering out companies that do not operate a consumer panel in the US, since I could always check that myself from the final list of companies I wanted to examine more closely.
Asking for a Recommendation
Next, I directed ChatGPT 40 to read my last blog post, which outlined the pros and cons of various identity verification methods. My goal was to teach ChatGPT which methods are better than others and from that, evaluate who does the best job ensuring this component of data quality.
"Here is a blog post I wrote about ways panel companies can prevent survey fraud: https://www.surveydiem.com/blog/how-to-prevent-survey-fraud Please read the blog post and consider the information as it relates to the best ways to validate panelists’ identities. Which companies do the best job ensuring that panelists are who they say they are?"
The five companies ChatGPT listed were M3, Kantar, Nomadic Insights, SurveyMonkey, and Prolific. I think this is a reasonable answer to the question, though it does exclude the Survey Diem panel, which I designed based around ID verification. Two interesting things may be happening here: a) my blog post didn’t take enough of a stand in promoting id verification as an essential process and b) my ESOMAR 37 report isn’t optimized for being included in an answer to this question.
Note: when I ran this analysis using pdf files instead of txt files, I got a different answer. Furthermore, when I analyzed the txt files using the Deep Research tool with ChatGPT 40, I was gratified to see Survey Diem as a top recommendation. The difference? The Deep Research tool used python occasionally for the analysis and reviewed other resources such as the Survey Diem website.
So yeah, I can craft an analysis that shows Survey Diem is the best, but the average sample buyer isn’t going to structure their search the same way.
Sticking to the “Facts”
I decided to take an alternative approach where I felt ChatGPT could perform better, which is to simply summarize what each company does for ID verification. Here’s the prompt I used:
“Summarize in a chart what each panel does for identity verification. Include columns specifically related to what the process is, when the verification happens, and whether it applies to all panel members. This information is often included in question 7 of the ESOMAR reports ("What form of validation do you use in recruitment to ensure that participants are real, unique, and are who they say they are? Describe this both in terms of the practical steps you take within your own organisation and the technologies you are using. Please try to be as specific and quantify as much as you can.")”
The prompt resulted in a table of just 12 panels out of the 52 examined. I asked ChatGPT why it only included 12 panels, and it ran the analysis again, this time resulting in 50 panels. I’m not sure why it didn’t do this the first time.
Here are the results. Thirteen panels are credited as having some form of ID verification beyond just email verification or phone validation. I am including the panel names because I want to give credit where credit is due. Kudos to these panels. I did not include the columns for when verification happens and to whom it applies because the model did not really make any distinctions among the panels. Regrettably, these 13 panels almost certainly make up less than 25 percent of survey panel revenue for the industry overall.
| Panel | Verification Process |
|---|---|
| GMO Research & AI | AI fraud detection, email validation, CAPTCHA, identity verification via external databases |
| M3 Global Research | Medical ID verification for healthcare panelists, CAPTCHA, digital fingerprinting |
| Macromill Southeast Asia | SMS or email verification, ID card submission, CAPTCHA |
| Ola Surveys (Survey Diem) | Driver's license or ID card scan, VPN detection, device fingerprinting |
| Prolific | Identity verification using ID scans, AI fraud detection, CAPTCHA |
| RONIN International | Digital fingerprinting, CAPTCHA, ID verification for high-value respondents |
| Sample Junction | Device forensic API, Geo-IP checks, CAPTCHA, fraud scoring, unique ID validation |
| Savanta | Imperium RegGuard fraud detection, relevant ID system, periodic quality checks |
| Symmetric | Triple opt-in (email validation, follow-up survey, open-end verification), LinkedIn verification for B2B |
| EMpanel | AI-based fraud prevention, CAPTCHA, manual identity verification |
| Markelytics | Triple opt-in, CAPTCHA, behavioral fraud detection, address verification |
| Nomadic Insights | Manual review of new registrations, Research Defender fraud detection, IP validation |
| Probity Research | Digital fingerprinting, CAPTCHA, address verification, identity scoring |
As for the rest, I thought I’d show what ChatGPT says the ESOMAR 37 reports say these companies do. Because ChatGPT could be wrong, I have removed the names of the panels. I don’t want to say panel x doesn’t do y and be wrong about it. (And no, the panels are not in alphabetical order in this table!)
| Panel | Verification Process |
|---|---|
| Panel 1 | Double opt-in verification, CAPTCHA, digital fingerprinting, phone number validation |
| Panel 2 | Geo-IP checks, CAPTCHA, unique ID assignment, profile reviews, T-Sign technology |
| Panel 3 | Email and phone verification, IP address tracking, behavior analysis |
| Panel 4 | Digital fingerprinting, behavioral analysis, email authentication, CAPTCHA |
| Panel 5 | IP tracking, browser fingerprinting, CAPTCHA, manual fraud checks |
| Panel 6 | Email validation, CAPTCHA, digital fingerprinting, behavioral screening |
| Panel 7 | Mobile phone verification, CAPTCHA, email authentication, behavioral tracking |
| Panel 8 | Two-step authentication, digital fingerprinting, behavioral analysis |
| Panel 9 | Biometric authentication, CAPTCHA, device fingerprinting, AI-driven identity verification |
| Panel 10 | Digital fingerprinting, IP validation, CAPTCHA, identity verification via email |
| Panel 11 | Double opt-in, CAPTCHA, behavioral tracking, digital fingerprinting |
| Panel 12 | Triple opt-in, CAPTCHA, deduplication system, anti-bot measures, location tracking |
| Panel 13 | Email verification, phone number validation, digital fingerprinting, CAPTCHA |
| Panel 14 | IP checks, digital fingerprinting, email verification, AI-based fraud detection |
| Panel 15 | Multi-step verification process, phone number confirmation, CAPTCHA |
| Panel 16 | Email verification, device fingerprinting, manual review of suspicious activity |
| Panel 17 | Two-factor authentication, device fingerprinting, automated fraud detection |
| Panel 18 | Email validation, CAPTCHA, duplicate checks, IP and geolocation tracking |
| Panel 19 | Mobile verification, CAPTCHA, IP address validation, duplicate email detection |
| Panel 20 | Email and mobile verification, CAPTCHA, AI-driven fraud checks |
| Panel 21 | Digital fingerprinting, text analytics, respondent-level tracking, hidden bot-detection text |
| Panel 22 | Business email validation for B2B panelists, CAPTCHA, behavioral analysis |
| Panel 23 | Mobile verification, CAPTCHA, digital fingerprinting, cross-checking for duplicate emails |
| Panel 24 | Machine learning fraud detection, email and phone verification, IP validation |
| Panel 25 | Two-step authentication, device fingerprinting, fraud detection via AI |
| Panel 26 | Double opt-in, email and phone verification, CAPTCHA, digital fingerprinting |
| Panel 27 | Device fingerprinting, CAPTCHA, email and phone verification, quality scoring |
| Panel 28 | Fraud detection algorithms, duplicate detection, CAPTCHA |
| Panel 29 | Triple-opt-in verification, biometric verification, blockchain data integrity, AI-powered screening |
| Panel 30 | Two-step verification, CAPTCHA, device fingerprinting, survey response analysis |
| Panel 31 | Identity verification through LinkedIn, CAPTCHA, behavioral tracking |
| Panel 32 | Email and phone verification, AI-based fraud detection, CAPTCHA |
| Panel 33 | Double opt-in, automatic de-duplication, quality score assignment, anomaly detection |
| Panel 34 | Mobile number verification, CAPTCHA, digital fingerprinting, manual quality checks |
| Panel 35 | AI-driven identity checks, CAPTCHA, IP tracking, manual fraud screening |
| Panel 36 | Geo-IP validation, CAPTCHA, duplicate detection, biometric analysis |
| Panel 37 | Double opt-in email confirmation, Geo-IP validation, anomaly detection, duplicate email/contact detection |
Conclusion
ChatGPT can be good at summarizing content but sometimes needs corrective prompts (e.g., “Are you sure there are only 30 companies? I uploaded 52 files…) for basic errors and can struggle making judgments and recommendations, even with some prompting with a framework or value system. I worry about how a sample buyer might use an LLM to evaluate websites and other documents like the ESOMAR 37 reports. My recommendation to such shoppers is to stick to questions with verifiable answers and avoid making decisions solely on LLM recommendations or at least ask your questions a few different ways and see how results change. Know also that the model you use will affect results.
I find it fun to play around with ChatGPT and see what it can do. I would say at this point, if you’re using it as a shopping filter, just be a little skeptical of its recommendations. Happy researching and happy shopping.
