[CondorActual]

I am suddenly unable to log into my Ford.com account, or my FordPass account (via the mobile app). More specifically, I am unable to login due to being locked out of my account, rather than something like a password error. I noticed this after getting an email from my power company asking me to confirm some login details for my FordPass account. Some backstory - In parts of Massachusetts, our electricity is supplied by National Grid. National Grid has an application called Charge Smart that allows EV owners to charge vehicles at a reduced rate during off peak times. This application requires access to your Ford Pass account via some flavor of API access that uses the same credentials as your ford.com account, etc.

It seems that something that this application is doing is causing the Ford side of your account to decide that a security issue is happening, and they are automatically locking your account out. I don't know if it's an API call frequency issue, or something is wrong with the content of the API call(s) that this application is making. I've submitted a ticket with both Ford and National Grid, and will update here as I learn more. So far, Ford is telling me it's 3-5 days before someone will call me, and I have no idea how long it will be to actually become unlocked. I'm not sure how frequently this kind of issue happens, but perhaps Ford should have a form or some other expedited way for users to clear this lockout from whatever identity provider they are using to manage our accounts. If you get an error code: CSIAH0320E, then you are likely locked out as well and will need to contact Ford for help.

It's also not a great situation to have to be a customer who is sitting between two companies who are sharing data via our accounts, and to have to mediate these kind of issues that really should be B2B concerns. I get it, early adoption has a price, but these kinds of issues are not easy to manage for some folks.

 

[ChasingCoral]

Yes, the Cyber Security team at Ford to flagged some user accounts due to anomalous API use regarding Mach Es. This has affected Home Integration users as well as folks running custom apps for their phones. They should be able to clear it up soon.

 

[CondorActual]

I have a email conversation happening with the Charge Smart support people, and they are asking their developers to look into things from their end. No idea what kind of turnaround there will be for this, but hopefully they can identify some issue with their authentication process that might be the culprit here. I doubt Ford will stepping in to do any investigation here to work with the power company, so hopefully Charge Smart finds and fixes an issue on their side and this becomes clean again. In the meantime, I'll be trying to log into my FordPass account periodically to see if they unblocked me. Ford, it's not great being locked out of the connected features of my new truck and being told it will take days to resolve something that should take minutes to handle.

 

[CondorActual]

Update: 5+ days and @Ford Motor Company has not responded to my case on this issue yet. This security block on the SSO portion of my Ford account basically locks me out of my account at Ford.Com, and completely blocks me out of the connected features of my truck. My FordPass app is unable to connect, and my electric company can't poll my account for charging details needed to discount my overnight charging activities. A second call into Ford had the agent ask me to continue to wait 3-5 days for someone to reach out. :|

@Ford Motor Company - I get that this API stuff can be a challenge at times, but you need a way for customers to reach you and get this API security block lifted quickly. If your third party API customers are having issues that cause them to do things that get your mutual customers blocked, you need to address this with them via better API documentation, better response to developer inquires, and tools that help customers and integrators fix or address security lockouts.

I'm only like three weeks into this new Lightning, and the Ford connected management experience has turned sour. I get that we are early adopters and super willing to work with Ford on things like this, but customer response needs to be better.

 

[CondorActual]

Udate: Day 7 - Called into Ford today for a follow up. No progress so far, but the nice phone agent at Ford Pass did update my case ticket with the same information that I already submitted. Asking me to wait another 24-48 hours for my original case handler to reach out to me, and if at that point I'm not fixed up I could request that this case be 'expedited'. I also called for an update from my power company's app support team, with no added details provided from them other than they have an active ask with the Ford API folks.

 

[CondorActual]

Update - Day 9/10. Called into FordPass team, and got connected to someone who finally moved me up to an advanced agent of some kind. This person first had me confirm that I would delete my FordPass credentials from this 'third party application' that the terms and conditions of having a ford pass account seem to not permit using for some reason. This makes almost no sense as Ford has a developer API for doing exactly this kind of thing. IT's not even a case of stored credentials - the power company app calls into FordPass to get a token that is used for API communication. Why would that impact my account access?

I was then told that they can fix my lock out, and that if it happens again, the security lockout would be permanent for my Ford account. The only work around at that point would be to make a new Ford account with a new email address, and have any points or other items assigned to that account transferred over to the new one. This is more problematic for Ford EV drivers as we have things like Blue Oval, Blue Cruise, connected services, etc., all managed via this account. If you do any financing via Ford, this account is also tied to your leasing accounts. This headache is punishment for trying to access the data that your vehicle generates. This lockout rule is also completely arbitrary. Nothing in the tech stack makes this lockout a permanent thing.

It will take a day or two for them to complete the unlock. So, I'm still not connected back to my truck, and it seems like I now really can't connect my power company's app to my Ford account, for fear of them doing something wrong again and getting my account permanently banned. @Ford Motor Company - You really need to review your T&S for connected apps, especially where your EV customers are concerned. I understand API security is paramount, but locking your customers out of tools that your B2B partners develop using your own documented developer APIs doesn't make a lot of sense.

Now, because of this I'm considering selling my FCSP, and getting a ChargePoint home charger. The power company has an integration directly with ChargePoint as well, and they will report my charging details to Grid instead of the data that Ford collects on our behalf. I can't risk the power company having another or future bug in the communication with Ford and getting locked out, and having to move all of my account details over with another series of time consuming support phone calls.

@Ford Motor Company - This is not great customer service. Do better.

 

[btreece]

Update - Day 9/10. Called into FordPass team, and got connected to someone who finally moved me up to an advanced agent of some kind. This person first had me confirm that I would delete my FordPass credentials from this 'third party application' that the terms and conditions of having a ford pass account seem to not permit using for some reason. This makes almost no sense as Ford has a developer API for doing exactly this kind of thing. IT's not even a case of stored credentials - the power company app calls into FordPass to get a token that is used for API communication. Why would that impact my account access?

I was then told that they can fix my lock out, and that if it happens again, the security lockout would be permanent for my Ford account. The only work around at that point would be to make a new Ford account with a new email address, and have any points or other items assigned to that account transferred over to the new one. This is more problematic for Ford EV drivers as we have things like Blue Oval, Blue Cruise, connected services, etc., all managed via this account. If you do any financing via Ford, this account is also tied to your leasing accounts. This headache is punishment for trying to access the data that your vehicle generates. This lockout rule is also completely arbitrary. Nothing in the tech stack makes this lockout a permanent thing.

It will take a day or two for them to complete the unlock. So, I'm still not connected back to my truck, and it seems like I now really can't connect my power company's app to my Ford account, for fear of them doing something wrong again and getting my account permanently banned. @Ford Motor Company - You really need to review your T&S for connected apps, especially where your EV customers are concerned. I understand API security is paramount, but locking your customers out of tools that your B2B partners develop using your own documented developer APIs doesn't make a lot of sense.

Now, because of this I'm considering selling my FCSP, and getting a ChargePoint home charger. The power company has an integration directly with ChargePoint as well, and they will report my charging details to Grid instead of the data that Ford collects on our behalf. I can't risk the power company having another or future bug in the communication with Ford and getting locked out, and having to move all of my account details over with another series of time consuming support phone calls.

@Ford Motor Company - This is not great customer service. Do better.

Doesn't Recurrent also use the API to get information using FordPass credentials? Is it just a problem with the frequency that the utility company is using the API that is causing the locking?

 

[CondorActual]

Doesn't Recurrent also use the API to get information using FordPass credentials? Is it just a problem with the frequency that the utility company is using the API that is causing the locking?

That's a good question. They were not really forthcoming in what triggered the lockout. It could have been something in the connection logic, call frequency, overlapping logins, user agent mismatches, etc. My bigger concern is that this system we are using seems to have a T&S agreement that we all agreed to that has some language around not allowing 'Third Parties' to access our FordPass accounts. I really want Ford to adjust or clarify what exactly a third party is with respect to our accounts. Does Ford consider the partner companies that work with their developer teams to be third parties? Or are we all under threat of lock-out whenever a power company, or navigation tool, or data aggregation service touches our Ford accounts? Like, please present a list of 'approved' vendors and applications that Ford has blessed to interact with our EV data, and make sure that if any of those approved services make a mistake, that it's not a perma-ban on our Ford accounts that we use to manage our very expensive vehicles.

As far as I can tell, National Grid didn't store my credentials, they had me use my creds at Ford's SSO site to generate an encrypted token used for their API access. No idea what went wrong there. I hope both side can figure this out and assure me that I'm safe from getting locked out again.

 

[btreece]

That's a good question. They were not really forthcoming in what triggered the lockout. It could have been something in the connection logic, call frequency, overlapping logins, user agent mismatches, etc. My bigger concern is that this system we are using seems to have a T&S agreement that we all agreed to that has some language around not allowing 'Third Parties' to access our FordPass accounts. I really want Ford to adjust or clarify what exactly a third party is with respect to our accounts. Does Ford consider the partner companies that work with their developer teams to be third parties? Or are we all under threat of lock-out whenever a power company, or navigation tool, or data aggregation service touches our Ford accounts? Like, please present a list of 'approved' vendors and applications that Ford has blessed to interact with our EV data, and make sure that if any of those approved services make a mistake, that it's not a perma-ban on our Ford accounts that we use to manage our very expensive vehicles.

As far as I can tell, National Grid didn't store my credentials, they had me use my creds at Ford's SSO site to generate an encrypted token used for their API access. No idea what went wrong there. I hope both side can figure this out and assure me that I'm safe from getting locked out again.

OK, so that sounds exactly like what Recurrent does to get your charge and battery information. I bet the difference is in the frequency of the calls. When I signed up for Recurrent they told me it would take up to 2 weeks for them to load my history. That makes me think they know they cannot go out and hammer my account with a bunch of API calls at one time. I signed up on Recurrent on 10/19 and my FordPass account has had no issues.

 

[CondorActual]

Update: So, Ford called me back today and I had an interesting, if unproductive call with the FordPass rep. The rep claimed that my account lock was finally lifted, and that I should be good to go. Tried to log in, no good. I tried the Ford.com website, still nothing. Deleted and reinstalled FordPass on my phone - no joy. I need to call them back now and see what the deal is. It's not great knowing that the experts think they solved your issue, but they actually didn't.

The interesting part is that they told me that Ford is aware that this account lockout issue is a big problem, and that Ford is working on a way to make it so that the block locks out 3rd parties rather than the users like us who rely on some of these tools that our power companies provide. He didn't give dates or a timeline on how or when Ford was addressing this, but he was confident that Ford understands this is messed up that people get caught in the cross-fire of API usage between companies. So, that's a good thing that they get it and are working on it. The agent did mention that Ford 'pays for each call to the API' and that services that hammer the API impact Ford's endpoint performance and that it's a problem that 3rd parties need to address as well.

For those of us who use National Grid: he did suggest that I use this service: www.ford.com/grid/nationalgrid

This is a service that gives you a credit for letting Grid shut down your charging session during peak energy demand. Unfortunately, the agent seemed to think that this service was the equivalent to the reduced meter rate that the ChargeSmart app was providing, which it is not. SIgning up for that service is a good idea, but I can't sign up without a functioning Ford.Com account. 🤷‍♂️

Hey @Ford Motor Company - Side complaint here: When someone calls into your service line for Ford Pass, and they need someone from the 'advanced team' to look at a customer ticket, it's not great that the only person who can look at your ticket is the one person assigned to it. Now I need to wait for the one person in the entire staff of FMC to come back to work to review and restart my ticket. Amazing. Do better.

 

[CondorActual]

Last Update: Sometime over the weekend Ford discovered what was locking my account out. I received an email from Ford, that clearly wasn't a message that was meant for general consumers. It looked like a message directly generated by their identity management system. Basically a somewhat cryptic message saying my ID state was returned to 'Enabled'. Clearly someone reached out to some IT staff to reset my account state. No one from Ford has called me back yet to confirm the fix. I can report that all access to FordPass and Ford.com was restored by this change. So, about 14 days to flip a bit on an account end to end for this service case.

I will not be restarting the service with my National Grid discount app until I get a notification from Ford that this application is approved for use. I have no idea how that state would be reported or who can authorize apps, but until Ford gives a blessing on these third party apps, it's too much of a risk getting my Ford account locked out again, possibly permanently. Hopefully @Ford Motor Company can get some clarification on this issue sooner rather than later.

 

[ChasingCoral]

I was bitten by this issue on Friday. Yes @btreece, Recurrent can be a problem. In fact it was the problem.

Problem:
Friday morning I lost access to all my Ford accounts and my FordPass app died. So did my PaaK.

Step 1:
I contacted Ford who gave me a case number and escalated it. The FordPass Guide who called me back after the escalation was clearly guessing.

"Maybe it is some sort of home automation app. Please delete every home automation app to see if that works."
"It might be an external service. Please delete your Recurrent app as well."
"It also may be a service you have for getting a discount to charge the vehicle."
"By the way, if your account is reinstated and a violation is reported again your account will be locked out permanently."
"If you really want to use any home automation apps, maybe you just won't be able to use FordPass"

Sound familiar @CondorActual?

I ended my Recurrent connection to both vehicles and deleted the home automation apps (reluctantly). I was then awaiting word back from someone who knew what was going on.

Step 2:
Because of what I heard from the Ford Pass Guide, I did my own escalation and passed my concerns to some of my Ford contacts. I didn't know whether Ford had no solution or the FordPass Guide was clueless.

I heard back from of the top folks on their cloud services side. I'm back in my accounts now and all is fine again. The following is from those sources in Ford who know what is going on wrt this problem.

No surprise, it has NOTHING to do with those home automation apps. I'll be reinstalling those and rebuilding whatever broke by deleting them.

It also has nothing to do with the use of charging apps like Electrify America, ChargePoint, etc. They are independent log-ins with no access the Ford account or direct vehicle access other than through the charging ports via approved charging communications protocols.

It was all about Recurrent and their use of the Smartcar service they use to access information. I've discontinued that service and changed my Ford password. Don't use any service that accesses your account using Smartcar or you will almost certainly be locked out. That is probably the problem with National Grid. I have heard that Smartcar was spoofing the Ford servers and calling for data hundreds of times per minute!

The combination of actions had also killed the PaaK on my Mach E and my Lightning. I was able to re-activate PaaK in one try in the Lightning but had an issue in the Mach E. If the vehicle won't connect to Ford and stalls, delete PaaK in that vehicle:
Paak reset
From SYNC :
Tap vehicle icon
Tap Settings/General
Tap Reset
Tap FordPass Connection Reset
Tap Phone as a Key Reset

Then reactivate Paak key setup for all drivers on vehicle

One other insight: these uncontrolled accesses of the vehicle's information can cause battery drain. Some here have reported dead LVBs in Mach Es that are probably driven by this. I have been receiving these warnings for the last few weeks from my Mach E (since a few weeks after signing up with Recurrent):

Hopefully that power drain issue will go away now as well.

I've also told Recurrent why I disconnected and I won't reconnect until they have Ford's approval. I'll be Ford's guinea pig but not Recurrent's. Recurrent said they are checking into the problem as well.

 

[CondorActual]

"By the way, if your account is reinstated and a violation is reported again your account will be locked out permanently."

This part of Ford's response is what is making me cranky. These APIs you publish are meant for 3rd parties to access. Ford should be providing ways for 3rd parties like power companies and the like to access your charing data, authorized by you, without having your account access be in jeopardy.

Something I did to help protect myself from getting a perma-ban from Ford:
I made a 2nd Ford.Com account, and added my Lightning to that account. Once you do this, the Ford side asks if you want to share data between accounts. I assume so you spouse can share your vehicle with you, but maintain a separate Ford.Com account. I then gave the credentials for the 2nd account to ChargeSmart/NG. That way, if they run afoul of Ford's API access rules again, my main account is (hopefully) not impacted at the SSO layer. Using GMail as my primary account, it was easy to simply make a [email protected] address to use for the 2nd login. All the mail for that account goes to my primary gmail account. If that Ford account gets locked out for good, I just make ***[email protected], etc.

These tools like chargesmart don't even store creds, they just generate an access token and use that for API access.

I've also told Recurrent why I disconnected and I won't reconnect until they have Ford's approval. I'll be Ford's guinea pig but not Recurrent's. Recurrent said they are checking into the problem as well.

I wish there was a way for Ford to publish who is an approved app vendor, and who isn't. I don't mind debugging things as an early adopter, but don't cut my legs off as I try to help out between parties. I told Charge Smart the same thing, but decided to risk this 2nd account trick until I somehow hear they are on the safe list. Whatever that may be.

 

[CondorActual]

Oh, @ChasingCoral - This app that got me in trouble uses ev.energy on the backend to communicate and manage the data. They have been fixing bugs that I have been reporting in their app. So it's not just SmartCar that is getting folks burned.

 

[ChasingCoral]

Something I did to help protect myself from getting a perma-ban from Ford:
I made a 2nd Ford.Com account, and added my Lightning to that account. Once you do this, the Ford side asks if you want to share data between accounts. I assume so you spouse can share your vehicle with you, but maintain a separate Ford.Com account. I then gave the credentials for the 2nd account to ChargeSmart/NG. That way, if they run afoul of Ford's API access rules again, my main account is (hopefully) not impacted at the SSO layer. Using GMail as my primary account, it was easy to simply make a [email protected] address to use for the 2nd login. All the mail for that account goes to my primary gmail account. If that Ford account gets locked out for good, I just make ***[email protected], etc.

These tools like chargesmart don't even store creds, they just generate an access token and use that for API access.

I wish there was a way for Ford to publish who is an approved app vendor, and who isn't. I don't mind debugging things as an early adopter, but don't cut my legs off as I try to help out between parties. I told Charge Smart the same thing, but decided to risk this 2nd account trick until I somehow hear they are on the safe list. Whatever that may be.

Smart approach. I think this may keep you safe.

Ford certainly could publish and approved list.

Oh, @ChasingCoral - This app that got me in trouble uses ev.energy on the backend to communicate and manage the data. They have been fixing bugs that I have been reporting in their app. So it's not just SmartCar that is getting folks burned.

I realize Smartcar isn't the only one. However, it has been one of the worst offenders that have locked people out of their vehicles.

 

[ChasingCoral]

This was an announcement on Ford Pass in August and bears redistribution:

Use Caution When Sharing Your Login Credentials

Important Notice: We’ve become aware other companies are asking Ford customers for FordPass account login credentials and information from their vehicles to provide services. By sharing your FordPass account information with outside companies, you’re putting your personal information and vehicle functionality at risk. Before sharing information with any company, please read their privacy policies to understand what data is collected and how it is used.

 

[ChasingCoral]

The list of known offenders so far is: Optiwatt, Recurrent, ev.energy, ABRP (OTA live data), Tronity, Charge Smart MA, SmartCharge Rewards, HomeBridge, widgets. Use may be in the past but if you car is still on their servers it's an issue until fully removed.

Be sure to kill the service and change your password to your Ford account.

 

[PowercatNewt]

I lost connection to just my lightning about 13 hours ago and after trying to reload Ford Pass on both of my phones got this same account lock error. I haven't gave my password to any third party apps this is very frustrating @Ford Motor Company

 

[ChasingCoral]

I lost connection to just my lightning about 13 hours ago and after trying to reload Ford Pass on both of my phones got this same account lock error. I haven't gave my password to any third party apps this is very frustrating @Ford Motor Company

Are you using an external service like Recurrent to monitor the truck's battery performance? Have you signed up for anything to give you discounted rates through your power company?

 

[PowercatNewt]

No the only things on the account are the charge Station Pro I haven't used any third party service that required my Ford data