back to article US politicos wake up to danger of black-box algorithms shaping all corners of American life

In Washington, DC, on Wednesday, academics and policy wonks warned US Congressional representatives about the perils of inscrutable algorithms, a red flag entangled by tangential worries about privacy, data collection, and net neutrality. As framed in a letter submitted by the Electronic Privacy Information Center, democracy …

  1. Number6
    Big Brother

    The US credit scoring system is not fit for purpose anyway. I don't care what algorithms they're using, I consider them to be wrong because they fail to account for all relevant factors. Worse, the credit system has wormed its way into almost everything - want a phone contract? Unless you're paying up-front they'll go check your credit score. Want a job? Yes, some employers want to know too.

    They seem to give greater weight to short-term things such as your current credit card balance, yet ignore the fact that this is a cyclical thing and that it's paid off in full every month, so you get a better rating just after you've paid off the card than just before, even though your overall spending/paying behaviour is the same (obviously if you don't pay it off then that's a different matter).

    Minority Report, anyone?

    1. joed

      Let me give an example of the bs of credit report over-reliance. My CC was on the auto-payment for utility bill for the account setup in the name of another family member. When I moved and setup the account with the same utility company - this time in my name - guess what they asked me for. A security deposit, as if they had not known me for years. The system is rigged to benefit credit report companies and even their customers (businesses) are screwed (in addition to consumers virtually prevented from opting out from this scam).

      1. tom dial Silver badge

        This would be more persuasive if you described how the credit reporting company benefited from the utility company requiring you to provide a security deposit.

        1. find users who cut cat tail

          tom dial: Does the credit reporting company provide their service (or disservice, but that is not the point here) for free?

    2. tom dial Silver badge

      Most phone contracts now come with a phone that bears a sticker price of several hundred dollars and a payoff period of around two years; there probably are some setup costs as well, although those probably are much less. The company never will see these again, and it is unsurprising that they would wish to check the ability and apparent willingness of the customer to honor the contract.

      Again, a decade ago we experienced a serious financial crisis caused, in large part, by the combined activities of dishonest or possibly misled borrowers, dishonest or possibly careless loan originators, gullible or possibly dishonest loan repackagers and risk analysts, and bankers who, by the time they bought the financial assets, could not reasonably tell whether they were any good or not, even if they had made an honest effort to find out. Secondary, but important, assistance and encouragement by government agencies whose mission was bound to promoting home ownership. Regular credit checking early in the loan process might have prevented that, and at the least would have mitigated it.

      It may be that there are reasonable arguments against whatever methods Fair Isaac uses to compute its magic number, and it may be desirable that those methods, and their inputs, be more public. However, it is long past the time when people went to their local bank or savings and loan or credit bureau and arranged a loan with someone they knew, and who knew them. Without the credit reporting and evaluation organizations, and their algorithms, economies would be much smaller, provide fewer opportunities, and goods and services would be, on average, considerably costlier than they are. Some might consider that a good thing, but I do not.

    3. A Non e-mouse Silver badge

      I once was applying to rent a flat. The letting agent said I had a bad credit score and was dubious about lettig the flat to me. The reason I had a bad credit score?

      I’d never had a credit card or personnal loan.

      It seems that not having borrowed money can be fatal to your credit score!

      1. Sgt_Oddball

        I know that one too

        It was too actually apply for a credit card for the first time. I enquired as you way in a branch and was told because of low credit score. After pointing out this was to be my first one ever... I was told they'd override the rejection and push it through.

        Sometimes common sense can be applied but it does still depend on the fleshy you're talking to.

      2. BobChip

        Not borrowing...

        This is indeed fatal to your "credit" score. If you never borrow money, there is no way lenders can ever make a profit from you - you are a bad credit proposition. If on the other hand you regularly borrow money, only pay back the minimum every month, but always clear the debt at the end of the day, your score will be almost 100%. Lenders will make a fortune from you, and you are the perfect credit proposition.

        A careful and prudent money manager is a dead loss to a lender. Sad but true.

        1. Anonymous Coward
          Anonymous Coward

          Re: Not borrowing...

          If on the other hand you regularly borrow money, only pay back the minimum every month, but always clear the debt at the end of the day, your score will be almost 100%.

          Well, no. If you only pay the minimum your score will drop. If you carry a balance, your score will drop. If you make purchases and pay off the statement balance every month, your score will drop.

          Lenders report based on the statement balance, and the percentage of that balance against the total credit limit is used in the calculation of score. Any utilization is a negative, so the only way to use a charge account and not have it be a negative is to pay the balance BEFORE the statement is created. At that, too many accounts can be a negative even if not being used.

          I will also add that special scores are returned for no credit history. The number is not low, it is not even in the score range -- it is recognized as a No Score by any requester that knows what the heck they are doing. If you have a low score it is because you have some form of a credit history. The history may be fine in regards to payments and such, just not a good combination of accounts. It takes account types of all kinds, and proper utilization (or lack of utilization) of those accounts to build a good score. A game and perhaps unfair for sure, but it is the way this stuff works.

      3. Fruit and Nutcase Silver badge

        Credit History - Victor Kiam

        @A Non e-mouse

        I’d never had a credit card or personnal loan.

        It seems that not having borrowed money can be fatal to your credit score!

        Once I heard Victor Kiam on a radio phone-in. Asked what advice he would give a young person hoping one day to have their own business. He advised the caller to wait until they were legally able to get a bank loan (the caller was 16 at the time), take one out for a small sum that they already have in savings or able to afford, for a minimum amount of time so that the cost is kept low. Once that's paid off, repeat it a couple more times, for higher amounts, but again, where you are able to and do pay it off without default.

        The purpose? Gets you a credit history. What Kiam had found when he started off was without a credit history, credit was difficult or expensive.

      4. Anonymous Coward
        Anonymous Coward

        "It seems that not having borrowed money can be fatal to your credit score!"

        When I left university and got a proper job, years ago, I was advised to take out a bank loan to buy something, and then pay it back quickly so I would get a credit rating. I did.

        Somewhat later I and the person I was then sharing a house with simultaneously applied for a credit card. I got a considerably bigger credit limit despite our earnings being similar.

    4. Anonymous Coward
      Anonymous Coward

      How is it a problem?

      To credit score for a *contract* phone or payment plan? That is literally what it is, credit. Now there may not be pay as you go or $10 handsets in the US as there is in the UK, but if there is, that is what you need if you don't have a credit score for a phone. Here in the UK, you just need to exist and not have defaulted/been declined to get an affordable (and yes they can decide to serve you as a customer or not depending on what you are paying) Phone.

      If you cannot afford a pay as you go and $10 top up, then how can you expect to afford a contract phone?

      But for everything else, home, car and other payments, then yes, some form of working and affordable system is needed. For those in the US I really feel for them, as most of those people will be exploited (see the sub prime mortgage bubble) and those are life dependant necessities.

      A phone though, while now becoming a necessities, is not an item priced out of reach, or an item I see them messing up the *checks* and balances, even in the murky world of "credit".

      (Credit checks for Jobs is a hard one to decide on. Some businesses are required to do background checks on criminal convictions. But money is not one of those. However, I'm happy for them to check me. I can understand others would not. And what do we do about people who are legitimately 5 months in arrears, about to be repossessed, and the employer asks politely if they can credit check before giving over the keys to the company car/safe/stock room/etc? I see it as a hard decision from everyone involved :( )

      1. Sgt_Oddball

        Re: How is it a problem?

        Usually companies that handle money will do a credit check to see if you'd be tempted to ' improve ' your situation or one where money can be used as a bribe to allow for unpleasentness all round.

      2. Hairy Scary

        Re: How is it a problem?

        Insurance companies have started doing credit checks when your policy comes up for renewal, the cost of the renewal is now affected by your credit score (there's a new entry in the terms and conditions to cover this) --- even though you pay cash.

        I very rarely use a credit card and if I do it's paid off at the end of the month so my credit score is probably not brilliant -- result my house insurance has gone up ---I phoned them and asked about the increase and was told that's why -- even though I have always paid by direct debit.

        Car insurance companies now do this as well.

        Why should a credit check affect a policy when you are paying by cash or DD ? I could see the reason if you were paying by installments over the year.

        I got quotes from other companies and there wasn't much difference (since they're all doing it now) so it wasn't worth changing to another insurer.

    5. Aqua Marina

      While they’re at it they can sort out the black box that is google search. Something is fishy when thousands of companies offering the same products are under the impression that their unique SEO practices have got them first page on google.

  2. JWLong

    The System of Legal Fraud

    The marketing system has now been joined at the hip to the credit system.

    Fuck them both.........

    Neither are secure or really necessary since they are both full of shit and/or garbage.

    And the entire financial system can go have sex with itself also as far as I'm concerned!

    I'm sorry, just so tired of this shit.............

    1. Charles 9

      Re: The System of Legal Fraud

      So now you gotta rent until you can actually save enough to buy a home all cash on hand (which the rent will eat into in the meantime)? Have to buy your car with cash on hand? Out of work and in the hole? Remember debtor's prisons?

      1. Sir Runcible Spoon

        Re: The System of Legal Fraud

        "Remember debtor's prisons?"

        1. Charles 9

          Re: The System of Legal Fraud

          Technically, that's more tax evasion than anything else. Think the days of Charles Dickens.

  3. A Non e-mouse Silver badge

    Not just credit scores...

    Isn’t America (in)famous for letting computers decide a convict’s jail sentence, rather than a judge?

    1. keithpeter Silver badge

      Re: Not just credit scores...

      Human judges can have their foibles as well apparently.

      I take the larger point and I am in favour of a simplification of things if possible. Remove some of the turtle layers.

      Coat: Battered copies of Naomi Klein's books and a couple of Ruskoff's in the pocket

    2. Palpy

      Re: Not just credit scores...prison sentences.

      Yes. Since the 1980s the percentage of the US population in prison has grown immensely. The US now imprisons more of its population than any documented nation on Earth, except a tiny island nation off the coast of east Africa, Seychelles.

      IIRC, it's mostly down to long sentences mandated by crime-and-punishment matrices. These are the result of state and federal mandatory-sentencing laws, not really the same as the black-box algorithms used in credit scoring and ad targeting.

      1. Charles 9

        Re: Not just credit scores...prison sentences.

        Ever considered many of those sentences are actually justified? Do we know what percent of the population got a serious, violent conviction such as for ADW, rape, kidnapping, or murder?

        1. a_yank_lurker

          Re: Not just credit scores...prison sentences.

          @Charles 9 - There are three major causes to the incarceration rates: long sentences, overcharging, and the 'war on drugs'. The first is obvious, long sentences mean people in prison for years if not decades for crimes that have much shorter sentences in other countries. Overcharging means there probably a trivial, semi-bogus charge you almost will be convicted of. Combine this under feral criminal law intent is not required and a minor misstep is now a potential felony. The 'war on drugs' criminalizes what is a common human behavior; seeking escape via drugs/alcohol and the consequent medical issues. A lot of very severe sentences are in this area. Also, in many cases the underlying issue is either psychological or medical not criminal in nature.

        2. Palpy

          Re: Ever consider that those prison sentences are justified?

          We're a bit off-topic, but I'm interested.

          First, I wish I could stick a graphic in here: I constructed a distribution curve plotting number of countries versus prisoners per 100,000. It is of course a modified bell curve. The US is not just out on the high tail of the bell curve, it is WAY out on the tail -- all by itself it defines an extension of over 10% on the curve. All other countries imprison fewer people. Except, as I mentioned, Seychelles -- which is probably a statistical anomaly, seeing as it does not even have a population of 100,000 people.

          Why? Here are two possible assumptions, Charles.

          1. The US imprisonment rate makes it the safest country on Earth, by a wide margin. This is patently false.

          2. The US population is by far the least law-abiding, most criminal, most debased and anti-social population existing anywhere on Earth.

          You may choose Door 2 if you like. But I don't believe it.


          3. The US criminal justice system has become unusually and unnecessarily punitive.

          If 3, then causes might include the corporate profits being made by private prison and prison-supply companies (who have lobbying groups to match their profits), political scare tactics used by "law-n'-order" candidates (often underwritten by said lobbying groups), and the desire in some states to remove as many minority citizens from the voting roles as possible (in many states, and all southern states, prisoners are denied the vote).

          Personally, I suspect the idea that US prison sentences are justified is not logically supportable, Charles9.

          1. Charles 9

            Re: Ever consider that those prison sentences are justified?

            "2. The US population is by far the least law-abiding, most criminal, most debased and anti-social population existing anywhere on Earth.

            You may choose Door 2 if you like. But I don't believe it."

            Oh, BELIEVE it! Don't believe me? Take a spin down South Central Los Angeles or some other crime hotbed. Or perhaps a few days in Pelican Bay State Prison will change your mind. Most crime in the US statistically is committed against other criminals. I don't think any other country has the kind of gang problems the US has (most of which are heterocultural in nature; something almost uniquely American).

  4. John Smith 19 Gold badge

    It's the 2nd decade of the 21st century. Isn't "Computer says no" something from the 70's?

    And should be left there.

    what you reall mean is

    "Is our profit optimizing algorithm has chewed through shedloads of data and found a bunch of parameters which maximized our profit (with that data set). We don't care what the parameters are and we don't care if they seem unfair."

    Say hello to "Social Darwinism" in the 21st century. *

    *Where privileged White males were declared as the "natural leaders" of society on the grounds that they were the leaders of society, with "evidence" from IQ tests written by other privileged White males

    1. Anonymous Coward
      Anonymous Coward

      Re: It's the 2nd decade of the 21st century. Isn't "Computer says no" something from the 70's?

      I know a few people at least know how to game the system. While I do not have the funds to, I do look out for supermarket or banking offers that are the rock bottom.

      However be warned, I have see a few times where the only reason they offer such good deals, is because they know they will be bankrupt tomorrow and they promised delivery/funds withdrawal next week!

  5. JLV

    >It is becoming increasingly clear that Congress must regulate AI to ensure accountability and transparency

    Wow, that sounds like a total clusterf*** in the making to me. How exactly do you regulate AI, esp of the neural network variety?

    This from a barely functional congress who has numerous others cats to skin - reigning in pharma corporations, being adult day cares, averting a government shutdown, fixing Obamacare, reforming taxes (though perhaps not strictly as per Trumpo's playbook), terminating unfit weapon procurement programs (hint: the one starting with 'F3').

    And generally fails at most of them.

    Regulating, where warranted and necessary, outcomes? Maybe. Algos? Please.

    Tell you what. Dis-enfranchise the Equifax bozos for gross negligence on critical data. i.e. pull out any authorization or government mandate that company has to allow them to access and store personal secrets, since they obviously can't be trusted with them. Then auction off the rights to be #3 in credit rating.

    Don't fall for regulatory overreach dreams.


    1. Charles 9

      The end result will be that #1 or #2 will be the only ones left to bid. Exclude them and there will likely be no bidders. End result is the same: a TERopoly becomes a DUopoly. Would you prefer that?

  6. jrchips

    AI advocates tell us the algorithms will "self-learn". On the surface, that sounds very impressive. But it begs the question, "Learn what?" How will developers know what algorithms their AI software is using if it has taught itself?

    The more decision making is delegated to AI, the more vulnerable companies will become, because untangling the code and deciphering the AI "self-taught" rules will be a nightmare. Setting aside the issue of frustrated customers who might be denied products or services, there are serious legal risks in the area of perceived discrimination and in product safety.

    For example, if a fully autonomous vehicle is involved in an accident, how will the manufacturer demonstrate that the vehicle behaved correctly? The more factors that are involved in the internal AI decision-making, the less transparent, and obvious they become. At some point, I can well foresee a company software engineer saying, in court, "we know what it did, but we don't know why it did it."

    Oops, "Bad answer!" Get your checkbook out.

  7. Anonymous Coward
    Anonymous Coward

    No score from no credit - red herring

    Not true.

    I learned at University that I could get any amount of credit cards and I did.

    I maxed them all and then defaulted due to the time between University and work.

    I ignored them and the debt collectors who came after the calls from the credit card companies ended.

    For at least 3 years I dodged them, meanwhile I worked in a well paid IT job.

    After feeling I had totally screwed my credit (to some degree deliberately) I finally called them and made a deal to settle immediately, paid about 30% of the debt which still ran to thousands.

    No more annoying credit card pre qualification letter I thought. No, within 6 months they were back to advertising at me. I had had no credit for 4 or 5 years preceded by total abuse of credit only to find that while I defaulted as extremely as possible, having paid it back and being in work I was now credit worthy.

    The systems are broken, I wasn't and am still not a good credit risk.

    To this day I have never had another loan or credit card since I saw just how callous they are in their lending and then in their debt collection. If you can't save up and buy it, you can't afford it. But you are free.

    Yes, even houses. I can put my hippy hat on and call all you "home owners" slaves to the man.

    But at least you have that nice big tv eh :)

    1. JLV

      Re: No score from no credit - red herring


      Reminds me of all the people that totally screw up under massive overspend and have to consider bankruptcy. Which they are against doing because it would scerw up their credit rating.

      Not realizing that perhaps that credit availability largely contributed to their position in many cases.

  8. Anonymous Coward
    Anonymous Coward

    Government, what is it good for? Absolutely nothing!

    "The recent hack of Equifax figured prominently, with reps expressing concern that the credit biz and its ilk operate without much regulatory oversight, perhaps forgetting what they were elected to do."

    You mean while we're being wined, dined and sixty-nine'd by lobbyists, we're supposed to remember who voted for us??? That's a good one!

  9. Michael Thibault

    So, algorithms should keep us...

    "safe from unfair, deceptive, and malicious practices". Whither propaganda? Or are we to suppose that the keepers of the algorithms never have, nor favour, a political agenda?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like