FIX ISSUED: Update as of 12:00pm EST, 5/21/2019:
Salesforce is now stating that a fix has been implemented and the incident is being declared over for users outside of NA53, 57 and 59. Anyone still experiencing issues should submit a support ticket if there are any left-over permission issues discovered.
Trust.salesforce.com states:
"All production instances are now out of service disruption and in a performance degradation state as service levels return to normal. During a performance degradation, end users are able to access the service, however, some functionality within the service may not be available or running at optimal performance. We are aware that some customers continue to experience issues, and Salesforce is working urgently to resolve them. For admins that need guidance on how to manually restore user permissions, please see the instructions in this Known Issue article http://sfdc.co/PermSetKI. Customers should continue to check Trust for updates."
The incident report is being continually updated by Salesforce here: https://status.salesforce.com/
Original Post/Older Updates:
If you are experiencing large-scale issues with Salesforce this morning, including “Something Went Wrong” or “Insufficient Privileges” errors, you should know that Salesforce is experiencing critical permission issues across multiple regions (EU6, EU8, NA41, NA49 at least). These issues are a result of a Pardot permission change that was rolled out overnight, and is reportedly affecting orgs who currently have Pardot as well as orgs that have had Pardot in the past. Premier Salesforce Support has confirmed that they are actively working on a solution.Status updates can be found at trust.pardot.com and status.salesforce.com/products/all.
How to know if you are affected:
-
Your users are receiving “Insufficient Privileges” errors
-
Users are unable to see or access objects (Accounts, Contacts, Leads etc)
-
Users are unable to log in to Salesforce
What you should do if your org is affected:
- Log a case with Salesforce immediately. This is a tedious process, so contact us if you need help interfacing with Salesforce.
- We do not recommend making any permission changes at this time, as Salesforce’s back-end fix will likely undo any changes made in individual orgs.
This is a rapidly evolving situation and permissions are being modified frequently on the back end at this time. If you have security concerns you may wish to freeze all users in Salesforce, which will prevent any unwanted data being shared.
Updates from the Salesforce Webinar as of May 17, 1:00pm EST:
At 9:56 AM UTC the Salesforce technology team became aware of a permission issue impacting NA and EU orgs, in both production and sandbox environments.
A permission change that should have only been applied to Pardot users was instead inadvertently applied to all users in current and former Pardot-connected orgs. This change gave every user the "modify all" permission, which is an admin level permission that allows users to see and change any and all data across the entire org.
Because this is a security issue, Salesforce made the decision to remove all permissions to all objects in any affected org. While Salesforce is aware of the major business impact this is causing, they felt it was important in order to protect the integrity of the data in affected orgs.
Currently this issue is not affecting AP (Japan), UM (United Kingdom) or Salesforce GovCloud instances.
Update as of May 17, 2:00PM EST from Salesforce Q&A:
Even if you do not have Pardot, and have never had Pardot, your org may be affected.
Highlights from the Q&A Portion of the webinar as of 2:00 PM EST (all answers provided by Salesforce support):
Q: Customers are reporting that they are unable to log in to their orgs even though they do not use Pardot. Is this really only affecting Pardot orgs?
A: The initial change that was rolled out which granted additional permissions only affected Pardot orgs. However as we work through the fix it was necessary to take down a subset of our NA and EU orgs and sandboxes, and so some customers who do not have Pardot are currently affected.
Q: Can you explain how you are deciding which orgs will have service disabled?
A: In our efforts to remediate the current issue, we ran the database change in question against the set of instances affected. We then provided another database change on top of that to remove user permissions that had been escalated. It was at that time that we decided to put all instances that did not have that second database change completed yet, into a service disruption status. As this database change completes, the orgs that are currently down will come back online.
Q: Has data been exposed to any outside parties?
A: Not that we know if, but as always, after the incident is over with we will do a full review.
Q: If we manually modify permissions for our users, is there a chance that those manual changes will be rolled back?
A: Yes, there is a chance that any manual updates made could be rolled back as we continue to make changes.
Q: Is creating a temporary permission set to re-grant object access to users a viable workaround?
A: Yes, once service to your org is restored, a temporary permission set is a viable workaround.
Q: Will permissions be automatically updated when the solution is in place?
A: Yes, that is a goal we are trying to achieve. Our goal is to restore all permissioning data.
Q: Will switching to a different server resolve the issue?
A: Unfortunately no, a server change will not resolve the current issue.
Q: Why can't these changes simply be rolled back?
A: The initial changes that caused the issue were made at the database level. As per our normal process, we performed the database changes in a staggered manner across all instances. We did perform these changes in a staggered manner as usual, but the changes were rolled out across both active sites as well as disaster recovery sites, so a roll back is not possible. Our current remediation efforts are focused on fixes to the live instances.
Q: Will this cause data loss in production orgs?
A: We are doing our best to ensure we do not lose any production data.
Q: How can this be prevented in the future?
A: As part of our standard process we provide a full root cause analysis. At that time we will provide our customers with the full details on what happened, and what we will do to prevent this issue from happening again.
Updates from 3:00 PM EST webinar:
Previously locked instances are beginning to come back up at this time. Administrators for these instances will be able to log in, but non-admin users will not be able to log in yet. There is no ETA on when non-admin users will regain access.
Immediately after instances come back online, user profile permissions will not be immediately restored. There are 2 paths forward to fix this issue:
- Salesforce is actively working on a remediation plan to restore all permissions.
- Salesforce is also encouraging administrators to manually restore prior permission settings while waiting for the remediation plan to be implemented.
There are currently no updates on whether this qualifies as a data breach that would need to be reported under GDPR, however Salesforce states they will address this question in an upcoming webinar.
During the most recent webinar, a user asked why the initial change was rolled out to disaster recovery databases at the same time as live instances. Salesforce responded that "Traditionally, when we run database DDL scripts, we only apply those to one side at a time. This change was not a DDL, it was a DNL — so it was updating records already in place on an existing table. When we do that, these changes are automatically logged into the disaster recovery environment. So this is why both environments now have a permissioning issue".
Updates to ongoing Salesforce service disruption as of 5/20/19:
Official documentation on the incident can be found here: https://status.
Official workarounds can be found here: https://success.
According to the most recent webinar given at 12:30 AM PDT today:
- Salesforce deployed a script intended to restore correct permissions on Sunday afternoon.
- The attempted fix did not function as intended and many instances have not had permissions restored.
- Salesforce is treating this incident as a top priority across all functions.
- There is work in progress to address the ongoing permission loss. There is still no ETA on a fix.
About Launch Team, Inc.
Our firm has expertise in creating scalable marketing and sales models for B2B companies.
Some of our sales and marketing technology offerings include:
- Data, reporting and system alignment
- Marketing and sales automation consulting
- Sales and marketing process and development
- Developing marketing and sales technology strategies that lead to growth
