New York City’s first-of-its-kind law, known as the “Bias Audit Law,” designed to combat discrimination that may arise from the use of artificial intelligence when making employment decisions entered its enforcement phase on July 5, 2023. The law, which was originally passed in 2021 but did not become effective until January 1, 2023, prohibits employers from using automated employment decision tools (AEDTs) to evaluate applicants unless the employer meets several requirements prior to its use, including conducting an independent audit of the AEDT and providing notice of its use. Employers must update independent AEDT audits at least annually and, perhaps most significantly, publish the results.
What is an AEDT?
The law focuses primarily on the use of technological tools to evaluate applicants in place of human discretion. An AEDT is defined in the Local Law itself as “any computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence, which issues simplified output, including a score, classification, or recommendation that is used to substantially assist or replace discretionary decision making.” The Final Rule, which implements the Local Law, clarifies that “to substantially assist or replace human decision making” means: (i) to rely on a simplified output (score, tag, classification, ranking, etc.), with no other factors considered; (ii) to use a simplified output as one of a set of criteria where the simplified output is weighted more than any other criterion in the set; or (iii) to use a simplified output to overrule conclusions derived from other factors, including human decision-making.
Recently published FAQs, issued by the City’s Department of Consumer and Worker Protection, further clarify that a tool is an AEDT if it (i) generates a prediction or classification; and (ii) identifies the inputs, the relative importance of the identified inputs, and any other parameters to improve the accuracy of the generated prediction or classification.
When must employers comply with this law?
This law applies to the use of any AEDT “in the city.” Per the FAQS, an AEDT is used “in the city” if any of the following are true: (i) the job location is an office in NYC, at least part time; (ii) the job is fully remote but the location associated with it is an office in NYC; or (iii) the location of the employment agency using the AEDT is in NYC..
Employers should be mindful that the lack of any in-person presence at an office in New York City does not necessarily mean that the law does not apply. Indeed, even fully remote jobs are covered, per the FAQ, if the job is “associated with an office in NYC.” This analysis is inherently fact intensive, so each position must be evaluated on a case-by-case basis.
Notice requirements
When the law applies, employers must provide notice that an AEDT will be used. The notice must also include information about how to request a reasonable accommodation under other, potentially applicable laws. If the applicant lives in New York City, employers must provide the required notice, together with a description of the “job qualifications and characteristics” that the AEDT will be used to assess, at least 10 business days before it is used.
These requirements may, and often will, be met by including this information in the job posting. Alternatively, for job applicants, employers may provide the required notice via the employment section of their website. For candidates for promotion, the notice requirement can be satisfied via written employment policy or procedure.
Additionally, information about the type of data collected by the AEDT, the source of that data, and the applicable data retention policy must be “conspicuously posted” on the employer’s website. Alternatively, the website can provide clear instructions for how to make a written request for such information, which must be provided within 30 days after a request is received.
Bias audits
Before leveraging an AEDT, employers must conduct an audit of the tool to check for bias against protected groups—currently defined, under this law, to include only race/ethnicity and sex. The audit must check for bias against not only these groups, but also “intersectional categories of sex, ethnicity[,] and race,” meaning it must compare the rates at which, for example, African American males were selected against the rate of selection for Caucasian females, and so on across all pertinent intersectional categories. All categories must be analyzed unless one (or more) represents less than two percent of the data used for the audit.
The audit must be performed by an independent third-party with no financial interest in the employer. Audits must be performed, and subsequently updated, at least annually. An audit is only valid if it was performed no later than one year before the AEDT’s use.
The requirements for audits are complicated, but the focus is on two metrics: “selection rate” and “impact ratio.” The “selection rate” is the rate at which individuals in a protected category are either selected to move forward in the hiring process or assigned a threshold score (or similar classification) by an AEDT. The “impact ratio” focuses on comparing the selection rate(s) for protected groups to those for the most frequently selected, non-protected group.
Notably, employers must publish a summary of results of these bias audits. The summary must include the date that the employer began using the AEDT, the date of the most recent audit, the source of the data used to conduct the audit, an explanation of how the data was used, the number of individuals assessed who fall into unknown categories, the number of applicants/candidates, and the selection rate and impact ratios for all categories. Employers must keep the summary posted for at least six months after their most recent use of an AEDT.
Noncompliance—risks and penalties
Employers deemed to have violated the law are subject to civil penalties of “not more than $500 for a first violation and each additional violation occurring on the same day as the first violation, and not less than $500 nor more than $1,500 for each subsequent violation.” The risk, however, does not end there. Published bias audits, for example, may potentially be leveraged by current or former employees seeking to pursue a variety of discrimination claims under state or federal law.
Will other jurisdictions regulate AEDTs?
New York City’s Bias Audit Law may be the first of its kind, but it certainly will not be the last. Employers should anticipate that additional cities, states, and even the federal government will enact similar laws in the future. As a result, all employers should work with employment counsel to assess their use of technology to aid hiring and promotion decisions and determine whether additional steps need to be taken to audit those tools, provide compliant notices of their use, and otherwise come into compliance with this trailblazing legislation.