Google Apps Assassinated My Domain – BEWARE!

I’ve been a long-time user of Google Apps and have recommended it to dozens of colleagues and associates. I currently use both the Standard (free) and Enterprise (paid) version of apps for several very high profile and long-standing domains. I’ve always considered it to be a reliable and cost-effective option for providing core email, calendar, and other services. However, I’ve recently fallen victim to activity by Google that demonstrates how an enterprise is placing their operations at risk if using Google Apps for their domain.

The first indication that there was a problem was an error message on my iPhone indicating that the password was incorrect for the domain in question. Since the password hadn’t been changed recently, I decided to investigate from my desktop by trying to log into the account. I was greeted with a message that stated “This account has been disabled” and a recommendation to change my password if I was having problems.

I followed the process for changing the password successfully and attempted to log in again. “This account has been disabled.” I consulted Google and found no helpful forum or blog postings on how to resolve this issue. I searched for a way contact Google associated with this issue. No luck. There is an option to obtain dial-in support, but that requires access to your account PIN and since I couldn’t log into the Apps admin panel, I couldn’t obtain the PIN. If you call Google Apps support, there is no way to proceed without the PIN.

I decided to post a message on the support forum for Google Apps. After ten days, there were no replies from other users or “advisors” from Google despite my attempt to bump the thread.

So let’s recap the impact to the domain after more than 7 days:

No email – not only could I not send and receive email, when folks tried to email me they got a hard bounce (account disabled). It is one thing to disable access to email, but bouncing email is catastrophic as communications completely fail instead of being queued in the inbox until the issue is resolved.

No App services – Calendar, Docs, Drive, and all Google App services are completely blocked. These are not critical for this domain in particular, but would be high impact if denied for some of my other domains.

Things were starting to get desperate. What could I do to restore email? Migrate the MX entries to another provider?

Ninety-six hours later, still no resolution. I’ve deliberately delayed migrating to another service provider so that I could provide an accurate perspective and timeline for this blog post, but it would appear that Google had successfully denied service to the business with no way to seek resolution.

I’d searched my email for this domain and the back-up email associated with the domain. There had been no correspondence from Google indicating an issue with the domain or an attempt to notify me of an impending or enacted disabling of the account. Google App services on the domain were killed with no explanation and no recourse.

In desperation, I decided to try and social engineer my way to support. I dialed in and entered the PIN from another one of my domains and once I got a human on the phone explained my issue. They promised they would look into it and call me in a few hours. I gave them my home and mobile numbers and made them promise to call back before hanging up. You guessed it, no call back. During that dial-in process, I was able to get a trouble-ticket issued so after a day I emailed Google Apps support in reference to that trouble ticket. Within a few days I had elevated it to a senior support person who suddenly emailed me a resolution:

“I have reviewed you case and can see that your account was incorrectly disabled due to an internal issue which our technical team has now resolved.

Therefore I have gone ahead and re-enabled your account and you should now be able to log in to use Google Apps.

I am sorry for the inconvenience caused. I understand that you were without access to the Google Apps services and I really appreciate your patience whist we have worked through this issue.”

The problem is now fixed, but not without significant impact to the domain in question and no explanation from GOogle as to how an “internal issue” can disable a business for over a week.

If you are currently using Google Apps for your domain, or considering the service, I would recommend you be aware that you are placing your livelihood in the hands of Google, which has no incentive to provide reliable services or support. Google can disable your domain services at their sole discretion with no recourse other than to move to an alternative service provide and social engineer a resolution that took over a week to resolve.

I’ll be looking at Rackspace and Microsoft as alternative service providers and appreciate any recommendations you may have regarding your experiences with those services.

UPDATE – NOVEMBER 25 2012

There has been a lot of great discussion regarding this article here on CTOVision and at HackerNews. As promised, here are some hints and recommendations on how to resolve this issue or at least plan for minimizing the impact to operations.

Things Google should do:

1) Alert the user at their secondary address if domain services are disabled with a time-expiring dynamic link to generate a trouble-ticket if this is in error. As a provider of services (even free) Google is obligated to at least try and ensure that services are not disabled due to an error on their end. Remember, even with free services, you are paying Google with your user data!

2) Allow a secondary mechanism for obtaining the support PIN in the event the admin panel is inaccessible. Perhaps they have you generate a TXT record in your domain DNS that once verified sends and email to the back-up domain email with your customer number and support PIN.

If you are the domain administrator:

1) Store your support PIN in a secure location outside of the Google ecosystem. I’m a big fan of using 1Password for this type of content, but a piece of paper in a safe would work just as well. You need to plan for not having access to your Admin panel.

2) Ensure your DNS is hosted outside of Google. Most registrars provide free DNS services, but I’m also a huge fan of Amazon Route 53. You need to be able to change your MX record to another email provider if the outage becomes unacceptable.

3) Establish a secondary provider in advance so you can hot-swap if you need to. This step may be cost prohibitive to some, but worth determining if the several hundred dollars per year is reasonable insurance in the event your Google services are disabled. For critical accounts, forward the email to the secondary provider account so you have a back-up of the email content as well.

4) Consider purchasing Google Apps through a reseller. Resellers have experience and access to support that normal “small” enterprise users don’t have. It might be worth having that reseller relationship in the event you have an issue like this.

5) There is a way to generate a Support Ticket in a circumstance like this, it is just hidden behind several layers. I was able to discover it through my conversations with Google, so if you are having this issue here is what I recommend.

- On your Google Apps login page, select the link that says “Can’t access your account.”

- Upon solving the Captcha on the next page look for a link that says “Reset your administrator password by domain verification”

- On the next page, enter a valid email address.

- On the following page, under item 3, you’ll see a link to contact Google support. That link is

https://support.google.com/a/bin/request.py?contact_type=admin_no_access

As of the date of this update, you can navigate directly to that link to generate a support ticket.

CTOvision Pro Special Technology Assessments

We produce special technology reviews continuously updated for CTOvision Pro members. Categories we cover include:

  • Analytical Tools - With a special focus on technologies that can make dramatic positive improvements for enterprise analysts.
  • Big Data - We cover the technologies that help organizations deal with massive quantities of data.
  • Cloud Computing - We curate information on the technologies enabling enterprise use of the cloud.
  • Communications - Advances in communications are revolutionizing how data gets moved.
  • GreenIT - A great and virtuous reason to modernize!
  • Infrastructure  - Modernizing Infrastructure can have dramatic benefits on functionality while reducing operating costs.
  • Mobile - This revolution is empowering the workforce in ways few of us ever dreamed of.
  • Security  -  There are real needs for enhancements to security systems.
  • Visualization  - Connecting computers with humans.
  • Hot Technologies - Firms we believe warrant special attention.

 

Recent Research

Tech Firms Seeking To Serve Federal Missions: Here is how to follow the money

Creating The New Cyber Warrior: Eight South Carolina Universities Compete

Mobile Gamers: Fun-Seeking but Fickle

Update from DIA CTO, CIO and Chief Engineer on ICITE and Enterprise Apps

Pew Report: Increasing Technology Use among Seniors

Finding The Elusive Data Scientist In The Federal Space

DoD Public And Private Cloud Mandates: And insights from a deployed communications professional on why it matters

Intel CEO Brian Krzanich and Cloudera CSO Mike Olson on Intel and Cloudera’s Technology Collaboration

Watch For More Product Feature Enhancements for Actifio Following $100M Funding Round

Navy Information Dominance Corps: IT still searching for the right governance model

DISA Provides A milCloud Overview: Looks like progress, but watch for two big risks

Innovators, Integrators and Tech Vendors: Here is what the government hopes they will buy from you in 2015

solid
About MattDevost

Matthew G. Devost is a technologist, entrepreneur, and international security expert specializing in counterterrorism, critical infrastructure protection, intelligence, risk management and cyber-security issues (cyberterrorism, information warfare, and network security). Find Matt online at: Devost.net, FusionX.com, and OODA.

  • Skitzzo

    @ircolle yeah, unfortunately there are still MANY areas of Google that lack any adequate support or offer customers any recourse.

  • Skitzzo

    @carlhancock the story wasnt written by @ircolle but if you read it, you have to have a PIN for phone support which they didnt have.

  • SQLSoldier

    Sorry to hear you’ve had such a bad experience. I use an Office365 Enterprise account and have not had any serious issues so far. It’s too early to declare it a grand success though. I’m keeping my fingers crossed.

    • http://ctovision.com/ ctovision

      SQLSoldier Thanks for the pointer to Office365. I am going to try that out.

    • tracker1

      SQLSoldier We had our email accounts unable to be accessed (the spooler for inbound mail worked, but couldn’t login to read/send) for a full day.. that’s about the worst we’ve had, in about 6 months since switching to the MS solution.  Honestly they all suck to varying degree, I think that google tends to be far more silent than most regarding their issues like this.  Which is a consistent problem in getting someone in email or on the phone.

  • graywolf

    See @Skitzzo I’m not so crazy … Ok maybe I am, but google really does mistreat their google apps customers cc:@mattcutts

  • http://secureemailplus.com/ FrankM

    As a forum TC (Top Contributor) for Google Apps and Postini services, I would be interested in seeing your post for help on the GApps Forum. We are also an authorized reseller and we are always looking for ways to help, not only for our own clients, but also the Google Apps community as a whole.
    Unfortunately, most do not write down their PIN for support, however I do feel there should be a way to login, if not only to gain access to the support PIN number. This is a feature we will recommend to Google for accounts not supported by a reseller, for additional support options.
    FrankM

  • primaryposition

    @jackschofield @mattdevost wow! We’re having massive problems too! They’ve become arrogant,don’t provide support, more court cases methinks

    • http://www.devost.net/ MattDevost

      primaryposition Support for this number of users is a challenge.  See my reply to flemingsean for ways Google might automate dealign with issues like this that are small in the grand scheme of things (e.g. the Google totality of services) but important to the end user.

  • flemingsean

    Ultimately, most of us are too small – in organisational terms – to be of real interest to Google, and far too small to support effectively.

    • http://secureemailplus.com/ FrankM

      That is why the Google reseller sales channel, has a greater edge over Google directly, for supporting organizations of all sizes. It costs nothing more to work with an authorized reseller and more often than not, you can gain additional value for the additional layer of support, that a reseller can bring to the table.

      • http://www.devost.net/ MattDevost

        FrankM That might be a valuable lesson here. If resellers have additional pull from a support perspective it might help mitigate an issue like this.  Have you ever had a client experience a situation like this?

        • http://secureemailplus.com/ FrankM

          As a reseller we do have an escalation avenue, should one of our clients run into this sort of issue. Clients can either contact Google directly or work through their reseller. In your case, if you were our client, we could have escalated the support case immediately and most likely had you back online in less time, depending on what the issue was.
          However, the fact that your Google forum post went unanswered is not thee norm. We have a very dedicated group of volunteer contributors, that Google invests in a great deal of resources, to help standard and premium accounts with help. In fact, last year in Sept 2011, Google had a world summit of forum contributors. some 350+ from all over the globe, get together in Mountain View, CA.

    • http://www.devost.net/ MattDevost

      flemingsean I agree that support is a challenge (as noted in my follow-on comment) but Google needs to try harder.  For example this could be automated to issue an email if an account goes into “DIsabled” status with a link to time-expiring support form to get a ticket in the system with higher priority.  The fact that the account admin was disabled and there were NO support options was an issue

  • Pingback: Google Apps Assassinated My Domain – BEWARE | My Daily Feeds

  • DannyD

    Shit happens. These systems aren’t perfect, and for the future make sure to store your PIN somewhere else.

  • Michael0000

    I don’t let google be my registrar, however love google apps.

    • http://www.devost.net/ MattDevost

      Michael0000 I love Google Apps too!!!  That is why this experience was so concerning.  Didn’t realize a scenario like this was even a possibility with Google Apps.

  • primaryposition

    @IvanODonoghue innit?

  • frosttt

    Did you have a Premium domain or you were using the crappy free edition? Because with Premium they tend to be quite fast resolving issues that might arise.

    • http://www.devost.net/ MattDevost

      frosttt Even with a Premium domain, unless you had the PIN stored outside the control panel you are stuck in this instance.  
      Even once I got the issue in the Google support system it was still several days until it was resolved, so having the PIN would have reduced the outage from over a week to a few days.  For most, a few days without email would be unacceptable.

  • Pingback: Google Apps Assassinated My Domain – BEWARE! | Helseo

  • http://ctovision.com/ ctovision

    Normally this time of year traffic at CTOvision slows to a halt.  But this post has caused a spike in traffic. Obviously it resonated with many people, and caused others to question if they have the right approach in trusting so much of their business to Google Apps.  For me, it is making me think I at least need an option to rapidly shift to another service.

  • http://www.devost.net/ MattDevost

    Thanks for all the interest in this article.  I agree that Google provides valuable services and that mistakes will happen.  The thing that was most disconcerting for me with this whole experience was that the Google Apps admin account was the one that was disabled.  It seems to me that should never happen.
    If for some reason, it needs to happen, Google needs to communicate with the domain administrator (at their back-up email address – which was set in this case) and let them know that the account was disabled, why it was disabled, and how to restore service.
    The issue would have been more tolerable had the disabling of the account not caused email to bounce back to the sender with an error. At that point, you don’t know what you are missing.  Had it simply collected in the inbox until the account was re-enabled, the user would have an ability to triage and reply to messages missed over the previous week.
    FrankM, feel free to DM me on twitter or contact me via the form at my site:  http://www.devost.net/contact/
    Again, I’m not sure that there are great alternatives to Google and I value the service they provide, but I felt like this experience was worth sharing as it highlights a significant risk if a business can’t go without their email for a week due to some unexplained error.  Providing support for services like this where companies are paying peanuts (or getting it for free) has to be an immense challenge, but I really feel like Google should try harder.

    • http://ctovision.com/ ctovision

      MattDevost Matt one thing I’m thinking of doing now is having my primary Google apps account, the one that all business e-mail goes to, auto forward 100% of all incoming e-mail to an account hosted elsewhere. Then if this happens at a minimum I’ll have a record of my stuff.

      • Jason26

        ctovision MattDevost If its your Google account that is doing the forwarding, and that gets disabled, or that domain gets disabled, your forwards may not work.
        While I do use Google Apps myself, I try to keep in a position where I have a forwarder prior to Google, or have a backup in place to flip my MXs in a hurry.

  • Gambawn

    Good post. Do follow up with how you are going to resolve this. As software engineers we need to make sure what we want to use for services are in line with out development paths.

    • http://www.devost.net/ MattDevost

      Gambawn Thanks.  As noted, the issue is resolved and I did follow-up on the trouble-ticket with Google to see if they can describe what happened in greater detail.  If they reply, I’ll post the follow-up here.
      For back-up, I’ll likely set up a domain on an alternative service and use the same account names, etc.  Then if I have an issue like this, I can immediately transition the MX records to the other host (and ad that domain to my account) as a back-up.  Downtime would be limited to a few hours, not days.
       Of course, I’m interested to see if any other folks have had the same issue in the past.  Perhaps this was just an isolated incident and 1/1million odds of happening.  If that is the case, then it probably isn’t worth an extensive contingency strategy.

      • Gambawn

        MattDevost Gambawn Nice, Isn’t it a shame that issues like this happen in the first place? 1/million are pretty obvious odds when it comes to the possible number of consumers of your services(infinity)

  • LX

    I couldn’t disagree with you more. Google doesn’t need to worry about your tiny startup. You think very high of yourself and I find it quite humorous to hear of your troubles. I sincerely hope this was not an accident and that the Google engineers were acting out of spite. I know I probably would if I were dealing with someone of such ego and rudeness. 
    Between forum spamming and not following service directions, I really can’t feel sorry for you at all.

    • http://ctovision.com/ ctovision

      @LX Pretty harsh there LX. I know for a fact that Matt was not seeking your or anyone else’s sympathy here. Just bringing some important factors up for consideration. 
      Anyway, I think the big lesson from the post is to ensure all dependancies are considered and if something goes wrong be ready to mitigate it.

    • OscarGoldman

      @LX
      “Google doesn’t need to worry about your tiny startup”
      What a spineless turd. Do you have no respect for yourself, that you would make excuses for a company that’s willing to screw you over? Grow a nut and stop living as a perpetual victim.
      Even worse is that you whine when someone else has the fortitude and takes the time to report a vendor’s offenses and how to deal with them. You’re obviously the rude one for not thanking this guy for taking his time. Pathetic ingrate.

  • Michael

    Try mailquatro.com

  • Matt

    This really is a broader issues with “cloud services” as a whole. In the service level agreements of your cloud providers most times you are at their mercy. In many instances you will not get the wording in the SLA whether a big corporation or a small start-up.

  • toetsrek025

    We use and sell Google Apps extensively. This is a big worry for us and many of our customers. There’s no real solution, as with any cloud provider all your data and business logic lives with them. As a part solution we use Syncdocs to backup all the data, which allows us to fall back to MS Office.
    The lack of support from Google is pretty much what you expect for such a low cost service. Their support forums of full of sad tales like yours.

    • http://secureemailplus.com/ FrankM

      Whether it’s email, docs or both, hosted or stored locally, backup and contingency (B&C) are always in play. It has been this way no matter the technology. Having been in the business since 1995, we have always stressed the importance of B&C to our clients who used our hosting and email services. Even now, as a GApps reseller we still suggest to our clients that B&C is just as important.
      Nonetheless, Matt’s incident and extended inability to access their account will need to be addressed by both parties. One, keep your PIN secure and updated and Google for providing an access option; to at least be directed to their default support page, when the account is disabled, for what ever reason, plus notification to the secondary account address when supplied. Both of these options have been requested by us to Google support.
      As for the forums, many of those posts are from the standard accounts, whereas the premium accounts have a support channel that comes with their service.

  • OscarGoldman

    “I searched for a way contact Google associated with this issue. No luck.”
    Disgusting, inexcusable, and sadly typical.

    • http://www.devost.net/ MattDevost

      OscarGoldman I’m going to update the post and show you how to get a support ticket for this issue generated based upon what I learned in a week of trying to get this resolved.

  • Ryan Ismert

    I’m baffled that the focus of this article and most of the comments is on the level of support provided – honestly, this is a $5/user/mo service with an SLA that only covers uptime.
    I’m a bit amazed that as “a technologist, entrepreneur, and international security expert” and the admin of a business domain, Matt, you didn’t have the PIN in a secure secondary location. Google is clear, and communicates in several places, that you’ll need the PIN if you ever want to talk to a support tech.
    I’m also horrified at Google, but not for their level of support. I’m horrified that they gave you back your account, based on a social engineering attack. Obviously I don’t have all of the details regarding secondary identify verification that you may have provided, but the whole purpose of the PIN is to function as an auth key that’s can’t be harvested through social attacks.
    As an “expert specializing in [...] cyber-security issues” you should know the value of a password vault for storing sensitive passwords / PINs. Let’s education folks on the right way to administer a domain, understand support options before buying, and choose the provider that provides the service they need (not just cheapest).
    Disclosure: As of a week ago, I work for YouTube, which is part of Google.

    • http://www.devost.net/ MattDevost

      Ryan Ismert Thanks for the comments.  I disagree with Google’s obligation to provide support.  While the $5.00/m cost is a great value, Google gets tremendous value from having users and getting us in the totality of their ecosystem and deriving value from that data to drive other revenue. They don’t provide these services out of the goodness of their hearts.
      Yes, shame on me for not having a PIN in a secure location, but there are many users that are using the free service that don’t have a PIN.  To be honest, I wasn’t even sure I had a PIN for this account as I use a combination of free and paid Google Apps.  If you read the Hacker News comments, there are lots of folks on there that didn’t even know a PIN was required or available.  I hand’t envisioned a scenario where I wouldn’t have access to the Admin panel unless Google was actually down.  Even if disabling services, the admin should always have access to the panel.  If you read my other comments, I’m not saying that Google should have a support team on hand for these issues, but they should automate alerting when a domain is being DISABLED and services are being denied.  Fire off an email to the secondary address noting that services have been disabled, state why, and provide a time-expiring link to open a ticket if this is in error.  That can all be automated by smart Google engineers in a day.
      With regards to Google support, I won’t speak to how they verified the domain ownership (as I dont’ want to give insight that could be used for malicious intent.  However, I would like to note that they did NOT provide any user data without verification and that they did not turn over information regarding administering the account.  In this instance, it really was just a simple matter of re-enabling a disabled account.  I don’t think they did anything wrong, except maybe express sympathy at tier one support to help resolve an issue that an identified paid user of their services was having.  I don’t see anything wrong with that and I think it is the only humanizing aspect of this whole experience.
      I’m more of a Google lover than a Google basher, but I wanted to drive awareness to the fact that this could happen to other users so they can plan accordingly like putting the PIN in a secure location, host DNS outside of Google so you can switch MX records, have a back-up provider.  In that regard, this post has done a lot to raise awareness that these services can fail and there are circumstances where services can be denied for extensive periods of time.

  • PlusRealTime

    You asked for a recommendation and mention Rackspace. If you hosted your mail there, you would have impeccable service, immediate recourse through people who understand the technology. However, it may not be at $50 per seat per year.
    I recommend using a DNS provider that allows alternative zone files in which case you could switch the MX in a few minutes and have mail back online within a day or less.

    • http://www.devost.net/ MattDevost

      PlusRealTime Thanks. I’ve had friends use Rackspace and they’ve found the support to be pretty responsive.
      On the DNS side, perhaps the article wasn’t clear, but DNS was hosted outside of Google.  Changing the MX records was always a potential option here, I just didn’t want to take that step before seeing how this played out.

      • PlusRealTime

        MattDevost yes, I got that you control DNS but was saying some DNS allows storing zones so one click would change it.
        I’ve seen how hard it us to find the PIN and contact info, too. Google has a lot to learn in that area.

  • http://www.devost.net/ MattDevost

    I’ve updated the post with hints on how to mitigate and resolve this issue.  I’ve also included some recommendations on things Google should do to minimize the impact to users in a scenario like this.  Welcome any additional thoughts or recommendations.

    • PlusRealTime

      MattDevost I sent feedback (heh, yeah, right) to say they should send a heads-up aleert tot he admin when they change the PINs, which apparently they do at random times! So saving the PIN, will not help if it’s been changed. I did just go and flash mine and will try to keep going back, but that won’t solve the problem.

      • http://secureemailplus.com/ FrankM

        We have already submitted a feature request about additional notifications to admins, when PIN’s change and when accounts are disabled and for what reason.
        Feature requests get more weighting, when support sees direct support cases sent in. Posting wish-lists to forums are helpful to a point, however they are not moderated by support personnel. When support cases are submitted, they are indexed and handled for use when updating or creating new features.

        • PlusRealTime

          FrankM Feedback = I sent it via their Feedback link.

        • PlusRealTime

          FrankM Incidentally, Google Apps is apparently not accepting any feature requests at the moment:
          Google Apps Feature Requests
          **NOTICE** We are temporarily hitting the ‘pause’ button so we can take some time to analyze all the great feedback and ideas we’ve received here. Our team will follow up in a few weeks with our thoughts, and we’ll be excited to gather your input on our thoughts as well!

        • http://secureemailplus.com/ FrankM

          I usually do not use the web based feature request, as I submit feature requests through our reseller support channel.

  • frosttt

    Sometimes they disable domains that violate copyrights or impersonate other entities which was the name of the domain used? Could have been confused with something else?

  • http://www.ardill.com/ Rob

    Matt,
    Do you have a link to the GA forum post you have referenced?
    Rob

  • http://gravatar.com/piquitodeoro piquitodeoro

    So, what answer did you received from Google? What was wrong with your app? I am in the same situation, suddenly my apps without notice.