Whoops! AI tenant-screening tool SafeRent pays $2.28 million settlement for being massively racist
pivot-to-ai.com
SafeRent is a machine learning black box for landlords. It gives landlords a numerical rating of potential tenants and a yes/no result on whether to rent to them. In May 2022, Massachusetts housing…

SafeRent is a machine learning black box for landlords. It gives landlords a numerical rating of potential tenants and a yes/no result on whether to rent to them.

In May 2022, Massachusetts housing voucher recipients and the Community Action Agency of Somerville sued the company, claiming SafeRent gave Black and Hispanic rental applicants with housing vouchers disproportionately lower scores.

The tenants had no visibility into how the algorithm scored them. Appeals were rejected on the basis that this was what the computer output said.

  • @[email protected]English
    2•1 month ago

    -Edit:

    Adding Edit to the beginning to stop the replies from people who read the scenario for context and can’t fight their compulsion to reply by nitpicking my completely made up list of “unbiased” metrics. To these peeps I say, “Fucking no. Bad dog. No!” I don’t fucking care about your commentary to a quickly made up scenario. Whatever qualms you have, just fuckin change the imaginary scenario so it fits the purpose of what the purpose of the story is serving.

    -Preface of actual comment:

    Completely made up scenario to give context to my question. This is not me defending anything referenced to the article.

    -Actual scenario with read, write, edit permissions to all users:

    What if the court order the release of the AI code and training methods for this tenant analysis AI bot and found the metrics used were simply credit score, salary, employer and former rental references. No supplied data for race, name, background check or anything else that would tip the boy toward or away from any bias results. So this pure as it could be bot still produces the same results as seen in the article. Again, imaginary scenario that is likely no foundation of truth.

    -My questions for the provided context:

    1. Are there studies that compare methods of training LLMs with results showing differences in results ranging from less or no racist bias and more racist bias?

    2. Are there ways of training LLMs to perform without bias or is the problem with the LLM’s code and no matter how you train them there will always be a bias presence?

    3. In the exact imaginary scenario, would the pure, unbiased angel version of rhe AI bot but producing equally racist results as biased trained AI bots see different court rulings that the AI that shows it’s flawed design caused the biased results?

    -I’m using bias over racist to reach broader area beyond race related issues. My driving purposes is:

    1. To better understand how courts are handling AI related cases and if they give a fuck about the framework and design of the AI or if none of that matters and the courts are just looking at the results;

    2. Wondering if there are ways to make or already made LLMs that aren’t biased and what about their design makes them biased, is it the doing of the makers of the LLM or is it the training and implication of the LLM by the enduser/training party that is to blame?

    • @[email protected]English
      2•1 month ago

      The article is fake news. I suggest looking elsewhere for proper information.

      As for your questions: LLMs were certainly not involved here. I can’t guess what techniques were used.

      Racial discrimination is often hard to nail down. Race is implicit in any number of facts. Place of birth, current address, school, … You could infer race from such data. If you do not look at race at all but the end result still discriminates, then it’s probably still racial discrimination. I say probably because you are free to do what you like and discriminate based on any number of factors, as long as it isn’t race, sex, and the like. You certainly may discriminate based on education or wealth. Things being as they are, that will discriminate against minorities. They have systematically lower credit ratings, for example.

      In the case of generative AI, bias is often not clearly defined. For example, you type “US President” into an image generator. All US presidents so far were male, and all but one white. But half of all people who are eligible for the presidency are female and (I think) a little less than half non-white. So what’s the non-biased output?