(Two-part series, written in November 2019: originally published on Medium: https://medium.com/@emmadaylaw/childrens-connected-toys-part-1-b231df9c6e82)
(Videography skills in state of evolution...)
The past five years has seen the first legal challenges to toy companies that produce artificial intelligence (AI) enabled toys for children. These toys, often sold as STEM educational tools, interact with children using voice and facial recognition biometric technology, and the latest generation of toys also provide opportunities for children to learn to code. As in many areas of life, the law has struggled to keep up with such dramatic changes in technology and to determine what regulatory measures are needed to be put in place to both protect children online, and also to allow children the freedom to engage with technology and learn the skills they will need to participate in the new tech-driven economy.
The first part of this paper analyses the applicability of the Children’s Online Privacy Protection Act (COPPA) of 1988, 15 U.S.C. 6501–6505 to AI toys in the US market as they have developed over the past five years, and finds that the Federal Trade Commission has been largely ineffective in holding toy companies accountable for protecting children’s rights to privacy and has done little to protect children from commercial exploitation. However promising new legislation from California may prove to be more effective in raising the bar for the realisation of children’s privacy rights across the US and beyond.
The second part of this paper looks at the significance of the US legal framework in the global market of children’s toys. As with many kinds of technology, the laws that apply to children’s use of robotic toys will depend largely on the age of the child, and where in the world they purchase and use the toy. Desk research for this paper concludes that AI enabled toys currently dominating the global market appear to be produced primarily by companies with teams who collaborate across the United States, Hong Kong, and China. Companies are choosing to carry out their product development in countries like Hong Kong which has little or no privacy protections for children, and once sufficient data has been collected to train their algorithms they ensure their products do not breach the rights of children in the United States under COPPA. It is concluded that more needs to be done to protect children in the “Global East” and “Global South” from commercial exploitation by Western companies.
Robotic toys in the U.S. market have gone through an evolution over the past five years. This started with toys that blatantly breached COPPA and contained serious security weaknesses; and then saw toys designed for the household also being used by children in the home, but argued by the private sector to fall outside of COPPA; and most recently moving towards the collection of biometric data from children.
Legislation in the US related to children’s online privacy dates back almost to the start of the Internet itself. COPPA was intended to “prohibit unfair or deceptive acts or practices in connection with the collection, use, or disclosure of personally identifiable information from and about children on the Internet.” and is enforced by the Federal Trade Commission (FTC). The Federal Trade Commission’s general mandate is to protect consumers and promote competition, but COPPA provides more specific consumer protection rights to children, which go far beyond those enjoyed by adults in the US. The FTC also has rule making powers under COPPA. However, COPPA only applies to children aged under 13, and adolescents aged over 13 are treated the same as adults under US privacy law.
Complaints have been made to the FTC raising privacy concerns related to AI toys, initiated by consumer action groups focused on privacy protection, but none of these have so far resulted in a formal FTC comment or decision. The FTC independently investigated and fined one AI toy company in 2018, and issued a blog putting the onus on parents to research the safety and privacy protections of toys before purchase.
‘Move fast and break things’: The first robotic toys to market
The first robotic toys for children came to market under the promise of innovation and ‘STEM education’, and seemingly without thinking through the implications for children’s rights to privacy, security and safety. Mark Zuckerberg famously coined the motto ‘move fast and break things’ to describe Facebook’s process for developing its platform. This motto reflects the ethos of the tech startup community’s approach which stresses agility, and celebrates frequent failures, fast learning, and adaptation. There is no time to dwell on the reasons for failure, or the possible human fallout along the way, and law and regulation of the tech space is viewed with contempt. This ethos has proven particularly problematic in the context of the children’s toy market.
Mattel’s Hello Barbie was amongst this first generation of robotic toys, which although capable of engaging in dialogue with children, also collected an abundance of children’s data, in blatant violation of COPPA. The doll recorded children’s conversations and these recordings could be accessed by both the child’s parents, and by ToyTalk, Mattel’s partner. The recordings were also allegedly shared with “unnamed third parties” and analysed using algorithms. The Hello Barbie doll was met with an advocacy campaign from the Campaign for a Commercial Free Childhood, who accused Mattel of ‘spying on kids’. There does not appear to have been any investigation of Hello Barbie by the FTC. However, a class action lawsuit was filed against Toytalk Inc. and Mattel in the Superior Court of the State of California, County of Los Angeles in 2015, on the grounds that the toy had failed to obtain effective verifiable parental consent as required by COPPA. COPPA requires companies to use a method for obtaining parental consent that is “reasonably designed in light of available technology to ensure that the person giving consent is the child’s parent”. It was alleged that the company should have known that there was great likelihood that other children aged under thirteen would play with the Hello Barbie doll along with the child owner, and that the voices of those other children would be recorded and transmitted to ToyTalk without their parents’ permission. This case does not appear to have proceeded to trial.
Genesis Toys was alleged to violate the deletion and data retention requirements of COPPA Rule. COPPA requires parents to be given the opportunity at any time to refuse to permit further use or collection of their children’s data and the opportunity to have the child’s personal information (PI) deleted. The complaint goes on to critique the company’s methods of obtaining verifiable parental consent for the collection of data from the child users, which is done through an in-app terms of service agreement, and there are no steps taken to verify the identity of the parent, which is alleged to be in violation of the direct parental notice, online notice, and parental consent requirements, of COPPA. COPPA requires “An operator must make reasonable efforts, taking into account available technology, to ensure that a parent of a child receives direct notice of the operator’s practices with regard to the collection, use, or disclosure of personal information from children, including notice of any material change in the collection, use, or disclosure practices to which the parent has previously consented.”. Finally, like in the Hello Barbie class action complaint above, the complaint against Genesis Toys alleges that the company’s representation to parents that it complies with COPPA requirements was deceptive. The complaint alleges that this amounted to deceptive misrepresentation in violation of Section 5 of the FTC Act, 15 U.S.C. § 45(a). The petitioners requested the FTC to investigate the two companies, halt the alleged COPPA and FTC Act violations, and investigate other companies engaged in similar practices. The FTC has not made any public response to this complaint. Far from suffering any kind of sanctions following these exposed privacy and security issues with Hello Barbie, the voice tech startup behind the doll, Pullstring, was reported to have been purchased by Apple for $30 million in February 2019.
As well as facing challenges through the legal system, both Hello Barbie and Cayla were successfully hacked by researchers, seeking to illustrate that as well as the privacy concerns already raised, both dolls presented significant security concerns, which can consequently also become privacy concerns. A researcher managed to hack into Cayla and use her to open the smart security device that controlled the owner’s front door. The doll also reportedly encouraged children to tell it about its home address, name of their school, and names of parents and siblings, all of which are PI under COPPA. Cayla was found to use a Bluetooth connection without the use of any authentication procedure, which made the device particularly insecure. Researchers also discovered that they were able to listen to conversations collected by Cayla by connecting their phones through the insecure Bluetooth connection, and calling that phone with a second phone. Hello Barbie was hacked by a researcher who managed to obtain enough information to access the home network of the child, and have the doll say whatever he wanted. He could also use the doll to access the phone used by the parent, which took him to the address of the parent and hence the child.
Although they are no longer available for direct purchase from their respective manufacturers, at the time of writing Hello Barbie can still be purchased on Amazon, and My Friend Cayla can still be purchased on eBay. However, neither of the apps for these dolls are available anymore on iTunes to connect the dolls to the Mattel software interface, which renders them virtually useless.
The U.S. Government response: one official regulatory decision, an FBI warning, and a cautionary blog
In May 2017, U.S. Senator Mark R. Warner, a Congressional leader on security in relation to the Internet of Things, sent an open letter to the FTC calling for increased efforts to protect privacy. He expressed concerns that protections for children were not “keeping pace with consumer and technology trends shaping the market for these products”, and asked the FTC to consider whether COPPA needed to be updated, and whether the FTC needed “additional authority from Congress to regulate the remote storage of data by operators or by third parties who store and handle children’s personal information”. Senator Warner also asked what the outcome had been of the complaints filed against Genesis Toys and another company, Spiral Toys. The Senator’s open letter was perhaps also seen by the FBI, who issued a warning in July 2017, on internet-connected toy privacy risks, and recommended that parents should be proactive and research where and how data is collected from their children through the toy, and ‘carefully read disclosures and privacy policies’ before purchase.
Following the VTech case, in December 2018, a blog was posted on the FTC website, entitled “Buying an Internet-connected smart toy? Read this.” The blog informs parents that smart toys could potentially be hacked by criminals, or could lead to the misuse of children’s data. It goes on to advise parents to do some research before purchasing a toy, such as “search online for the toy’s name, the company that makes it, plus the words “complaint,” “security,” and “privacy.”” , and it directs users to a page which lists safe harbour groups who sometimes make recommendations regarding the safety of smart toys. The blog goes on to advise parents to consider whether the toy they are looking to purchase contains a microphone or camera, and whether they are OK with a toy that connects to social media accounts. Parents are then advised to understand what information will be collected about their children, and how this will be used. It seems then, that parents are responsible for inspecting the toys they buy for their children, reading the privacy policies, searching online for the toy’s name and any legal action that may have been taken against them, and assessing the implications of the ways in which the company may collect different kinds of data from their child, and imagining what any negative consequences of this could be. This seems like a due diligence process involving quite advanced legal and technical literacy that should fall more on an industry regulatory body rather than on consumers themselves.
The second wave of robotic toys: the Alexa and Siri generation
As well as robotics that are expressly marketed as children’s toys, the internet of things market has developed enormously in the past five years, and many smart toys or connected household items aimed primarily at an adult audience, are now likely to be located in family homes. In a 2017 University of Washington study of “Parents, Children, and Internet-Connected Toys’, researchers found that children were frustrated with the limitations of smart toys designed for children, because they were used to interacting with Apple’s Siri and Amazon’s Alexa in the home, and expected a much more sophisticated response from the toys than the pre-prepared answers designed for children. The authors of the research recommend that the companies behind mainstream internet of things devices in the home recognise that it will be common for children to use these devices, and consider this point in their design process. It has been noted elsewhere that these kind of connected devices have become so common in the home that “by 2021 there will be almost as many assisted bots on the planet as people.” However, if companies were to acknowledge that children are likely to use their products in the home, then they would fall under COPPA because their products would be directed to children (SEC. 1302. Definitions (10)).
Factors that would lead the FTC to conclude that a website or online service is “directed to children under 13” include both factors related to audio or visual content, and “other reliable evidence about the age of the actual or intended audience.” If the University of Washington study was found by the FTC to be ‘reliable’ evidence that children are actually using Alexa, then the company is permitted to choose to apply COPPA protections only to users under age 13. This means they “must not collect personal information form any users without first collecting age information. For users who say they are under age 13, [companies must not] collect any personal information until [they] have obtained verifiable parental consent.” At the time of writing, it appears that Alexa only uses speech recognition technology, which converts speech to text, rather than voice identification technology, which collects biometric data from the user. However, the EFF highlighted in November 2018 that Amazon had filed a patent for a voice recognition technology which would be able to identify the user’s age, as well as other sensitive characteristics such as gender, ethnicity, and even emotional state. If Amazon decides to implement the broad use of voice recognition technology, it could risk obtaining actual knowledge that it has child users. It could be quite difficult in practice for devices such as Alexa to collect age information on each user that happens to walk into the room. For any given family this could be achieved by collecting voice biometric data from each family member, but for any visiting children this could be problematic. If a child enters the home without a parent should Alexa turn itself off due to the lack of available parental consent?
Amazon is currently facing potential legal action under State wiretapping legislation in relation to Alexa. In June 2019 it was reported that a class action lawsuit has been filed against Amazon by a mother on behalf of her ten year old child, alleging that Amazon is unlawfully recording children via Alexa, and is unlawfully holding on to their voice recordings. The case is not filed with reference to COPPA, but instead is filed under the State laws of Florida, Illinois, Michigan, Maryland, Massachusetts, New Hampshire, Pennsylvania, and Washington, which all prohibit “the recording of oral communications” without dual-party consent. The lawsuit compares Alexa negatively with Siri, because Apple deletes the voice recordings after they have been used to answer questions using Siri. It is too soon to know the outcome of this case at the time of writing.
It is not yet clear whether either of these cases will result in a formal decision from either the FTC or the courts.
The third wave of robotics: the ubiquity of biometrics
The third generation of robotic toys collect biometric data from children in the form of facial recognition features, giving children the sense that their toy recognises them and can form an emotional bond. At the time of writing one of the leading toys being sold under the category of STEM education is Cozmo the Robot, whose corresponding app includes games, and also a Code Lab where children can learn to code the robot to perform different tricks. Cozmo won two awards in 2018 for EdTech and educational learning.
Cozmo is in many ways a good example of robots in this genre of AI toys, because the biometric data taken from the child’s face is stored in the toy, rather than being communicated back to the cloud for use by the company, which reduces privacy concerns. Cozmo does still however raise some questions under COPPA. First, once the parent has downloaded the app and added in their own email address and date of birth, it is possible to keep adding additional users to the toy. This is done by adding the child’s name in the app, and inviting Cozmo to scan their face (and collect their biometric data), after which Cozmo will say the new child’s name. The app does not ask for any more parental permission to scan the face of a new child, and so assumes that the phone containing the app remains in the custody of the parent. It is necessary to keep the app open and move between the app and the robot in order to play games, and interact with the toy, so it seems very likely that the child would be left alone with the app and the toy, without constant supervision by an adult. If this toy is being used by a child, then they are likely to invite their siblings or friends to also engage with the toy. This again raises the question similar to that raised in the class action lawsuit against Mattel: Is the parent’s initial login sufficient to authorise the taking of biometric data from all children who may be playing with that toy, including anyone else’s children on a play date? Even though the biometric data is stored in the toy, it has still been collected, which triggers COPPA because it comes under “request, prompt, or encourage the submission of information, even if it’s optional”, and the requisite verifiable parental consent requirements appear to be somewhat lacking. COPPA requires that parents are given ‘direct notice’ of the company’s information practices. Here the parents of other people’s children are not given any notice.
Further, even though Cozmo stores the biometric data on the toy rather than in the cloud, which alleviates some privacy concerns, there are also separate cybersecurity risks to be considered. It could still be possible for someone to hack into Cozmo and access the child’s data, which raises the question of whether it is worth the risk of storing the data in the robot, when this is not a central feature of the toy. Additionally, for the purposes of research this author bought Cozmo second hand from Amazon, and discovered there were three names already stored in the ‘meet Cozmo’ section of the app; “Mom, Shay, and Carson”, presumably the previous family who owned Cozmo. Given that the facial recognition data is stored in the robot, it seems likely that the biometric data of that family remains in the toy and could be accessed. It is possible for parents to scrub their children’s data from Cozmo at any time, but they would need to know why that might be important and think to do so before they sold the toy or threw it away. Given the high price point of these kinds of robotic toys, it is highly likely that some parents will purchase them second hand, so this needs to be considered when it comes to regulation. An additional security risk exists in the rapidly evolving robotic toy sector because these toys seem to go off the market very quickly, which means that security patches and updates will no longer be provided for the supporting software, leaving it much more susceptible to hacking. Hello Barbie, i-Que, and My Friend Cayla are no longer on the market. Cozmo the robot is still being sold, but its parent company Anki Inc. announced in April this year that they have ceased manufacturing robots, and it has been reported that Anki Inc. is going out of business. 
I recommend that the collection of biometric data from children in the target age range of this toy, which is 6+, should be limited to true necessity, due to the potential security and privacy risks involved. I further recommend that retailers such as Amazon and e-bay should require the scrubbing of such toys before they are resold on their platform. Amazon and e-bay could also play a role in ensuring that robotic toys are no longer sold on their platforms where the parent company has gone or has declared that it is imminently going out of business.
A new era of internet regulation: bold new regulatory moves made by California
It is likely that these new California laws will improve the security and privacy protections for children using AI toys in general produced for the US market. Due to the size and value of the California market it would be in the business interest of most companies to comply with the State’s regulations in this regard, and it would be costly and complicated to produce different versions of toys or their supporting apps for use in different States.
 Children’s Online Privacy Protection Act (COPPA) of 1988, 15 U.S.C., §6502 (b)
 Drake Beer, Mark Zuckerberg Explains Wy Facebook Doesn’t ‘Move Fast And Break Things’ Anymore, Business Insider (May 2, 2014), https://www.businessinsider.com/mark-zuckerberg-on-facebooks-new-motto-2014-5
 Josh Golin, Advocates Say “Hell No Barbie” to stop Mattel from Spying on Kids, Campaign For a Commercial Free Childhood, (Nov. 9, 2015), https://commercialfreechildhood.org/advocates-say-hell-no-barbie-to-stop-mattel-from-spying-on-kids/
 Federal Trade Commission, Children’s Online Privacy Protection Rule: A Six-Step Compliance Plan for Your Business, (Dec. 9, 2019), https://www.ftc.gov/tips-advice/business-center/guidance/childrens-online-privacy-protection-rule-six-step-compliance
Ashley Archer-Hayes, on behalf of herself and C.H., a minor child, individually, Charity Johnson, on behalf of herself and A.P. a minor child, individually, and on behalf of all others similarly situated, v. TOYTALK, INC., MATTEL, INC., SAMET PRIVACY, LLC, dba KIDSAFE SEAL PROGRAM, and DOES 1–10, unreported Case No: VC603467, (2015), http://www.coppanow.com/wp-content/uploads/HelloBarbieComplaint.pdf
 Complaint and Request for Investigation, Injunction, and other Relief at 2, In re Genesis Toys & Nuance Communications, (FTC Dec. 6, 2016), available at https://epic.org/privacy/kids/EPIC-IPR-FTC-Genesis-Complaint.pdf.
 Id. Para. 13
 Id. paras 35–36
 COPPA Rule, 16 C.F.R. §§ 312.6 (a)(2)
 Id. paras 91–99, para 134; Rule, 16 C.F.R. §§ 312.4, 312.5
 COPPA Rule, 16 C.F.R. §§ 312.4 (b)
 Patrick Lion, Parents ordered to destroy talking My Friend Cayla Doll because hackers can connect and talk to children, Mirror, (Feb. 17, 2017), https://www.mirror.co.uk/news/uk-news/parents-ordered-destroy-talking-friend-9838125
 Mark R. Warner, Press Release: Sen. Warner Pushes FTC to Protect Children’s Data Security with Internet-connected “Smart Toys”, US Senator from the Commonwealth of Virginia, (May 22, 2017), https://www.warner.senate.gov/public/index.cfm/2017/5/warner-ftc-interntet-of-things-letter
 Federal Bureau of Investigation Public Service Announcement, Consumer Notice: Internet-Connected Toys Could Present Privacy and Contact Concerns for Children, Alert Number I-071717(Revised)-PSA, (Jul. 17, 2017), https://www.ic3.gov/media/2017/170717.aspx
 Federal Trade Commission, FTC Releases Annual Privacy and Data Security Update, (Jan. 18, 2018), https://www.ftc.gov/news-events/press-releases/2018/01/ftc-releases-annual-privacy-data-security-update
 Complaint filed by the United States, in United States v. VTECH, Case No: 1:18-cv-114, (Jan. 8, 2018), https://www.ftc.gov/system/files/documents/cases/vtech_file_stamped_complaint_w_exs_1-8-18.pdf
 Id. Para 22
 Id. Para 25; Section 312.8 of the Rule, 16 C.F.R. § 312.8
 Federal Trade Commission, Electronic Toy Maker VTech Settles FTC Allegations That it Violated Children’s Privacy Law and the FTC Act, (Jan. 8, 2018), https://www.ftc.gov/news-events/press-releases/2018/01/electronic-toy-maker-vtech-settles-ftc-allegations-it-violated
 Cristina Miranda, Buying and internet-connected smart toy? Read this., Federal Trade Commission Consumer Information blog, (Dec. 6, 2018), https://www.consumer.ftc.gov/blog/2018/12/buying-internet-connected-smart-toy-read
 Emily McReynolds et al., Toys that Listen: A Study of Parents, Children, and Internet-Connected Toys, Federal Trade Commission, https://www.ftc.gov/system/files/documents/public_comments/2017/11/00038-141895.pdf
 Id. P.5
 Federal Trade Commission, Children’s Online Privacy Protection Rule: A Six-Step Compliance Plan for Your Business, (Dec. 9, 2019), https://www.ftc.gov/tips-advice/business-center/guidance/childrens-online-privacy-protection-rule-six-step-compliance#step1
 Future of Privacy Forum, Kids & The Connected Home: Privacy in the Age of Connected Dolls, Talking Dinosaurs, and Battling Robots, at P.11, (Dec. 1, 2016), https://fpf.org/2016/12/01/kids-connected-home-privacy-age-connected-dolls-talking-dinosaurs-battling-robots/
 Chris Burt, Emotion and accent voice recognition capabilities patented by Amazon concern privacy experts, BiometricUpdate.Com, (Nov. 19, 2018), https://www.biometricupdate.com/201811/emotion-and-accent-voice-recognition-capabilities-patented-by-amazon-concern-privacy-experts
 Kaitlyn Tiffany, Amazon is being sued for recording children’s voices with Alexa, Vox, (Jun. 14, 2019) https://www.vox.com/the-goods/2019/6/14/18679360/amazon-alexa-federal-lawsuit-child-voice-recording
 Fla. Stat. ch. 934.03; 720 I.L.C.S. § 5/14–2(a) (Illinois Eavesdropping Law); Article 14; Mich. Comp. Laws § 750.539c; MD Cts & Jud Pro Code § 10–402 (2017); Mass. Gen. Laws ch. 272, § 99.; N.H. Rev. Stat. Ann. § 570-A:2(I-a); 18 Pa. Cons. Stat. § 5702 to § 5704; Wash. Rev. Code Ann. § 9.73.030
 Complaint in the matter of Request for Investigation of Amazon, Inc’s Echo Dot Kids Edition for Violating the Children’s Online Privacy Protection Act, Echo Kids Privacy, (May 9, 2019), https://www.echokidsprivacy.com/#readcomplaint
 Aaron Tang, Certification Tips for Alexa Kid Skills, Amazon Alexa Blogs, (Nov. 6, 2017), https://developer.amazon.com/blogs/alexa/post/7ffad993-15bf-42b2-9ffb-4e82a5d7cebe/kid-skills-alexa-skills-certification
 Complaint in the matter of Request for Investigation of Amazon, Supra at p. iv.
 Chaudron S., Di Gioia R., Gemo M., Holloway D., Marsh J., Mascheroni G., Peter
J., Yamada-Rice D. Kaleidoscope on the Internet of Toys — Safety, security, privacy and societal insights,
EUR 28397 EN, doi:10.2788/05383
 Federal Trade Commission, Children’s Online Privacy Protection Rule: A Six-Step Compliance Plan for Your Business, (Dec. 9, 2019), https://www.ftc.gov/tips-advice/business-center/guidance/childrens-online-privacy-protection-rule-six-step-compliance#step1
 Anki Inc. Website, (Dec. 9, 2019), supra.
 Cal. Civ Code § 1798.91.04(b) S.B. 327
 NBC News, California is bringing law and order to big data. It could change the internet in the U.S., (May 13, 2019), https://www.nbcnews.com/tech/tech-news/california-bringing-law-order-big-data-it-could-change-internet-n1005061