Text Message Update

Please Note: This article is out of date. Please reference this post instead.

What went wrong?
Midnight on Aug 26th our text message provider stopped forwarding your text messages to us. It turns out that they preformed a software update that caused all sorts of problems that affected not just TSheets, but several other companies using their services. We received no advanced warning that this software update was going to take place, and now almost 24 hours and dozens of phone calls later, they still don’t have things completely fixed. We are very disappointed with our upstream providers (Tatango and 4Info) and couldn’t be more sorry for the inconvenience this is causing.

It’s working! (mostly)
One of the things changed by our provider’s software update is they are now intercepting all text messages with the word “start” or “stop”. Since 95% of the text messages sent to TSheets are “t start” or “t stop”, none of those messages are getting to us. To work around this we have setup the new words “in” & “out” to replace “start” & “stop” respectively. So if you would have previously texted “t start at 0800″, you will need to replace that with “t in at 0800″ instead.

Confirmations may not work
If you happened to text “t stop” to us during the last 24 hours, chances are that we won’t be able to send messages back to your phone. Our provider is intercepting the word “stop” and blocking your number so that we can’t send messages to you. We don’t have a work-around for this yet, but will keep you posted.

A backup plan
Be assured that we are completely committed to restoring a reliable system for tracking your time via text message as quickly as humanly possible. We are working through the night researching other text message gateway providers and alternatives to get a reliable system in place. Expect another announcement in very the near future with backup number you’ll be able to text to.

Please call or email if you’d like more details or just feel like venting.

Brandon

  • 2009-08-28: Status Update: Our provider will be applying a patch tonight. Read my update in the comments below.
  • 2009-08-31: Status Update: Things are back to 100%! Read my update in the comments below.

7 Responses to Text Message Update

  1. Well it’s frustrating, but thanks for keeping us in the loop. Good luck getting that backup number working.

    [Reply]

  2. Before I notify our employees to use T In and T Out, will you be changing back to T Start and T Stop once the problem is worked out? Just want to know if this is a temporary or permanent fix.

    Thanks.

    [Reply]

  3. @Rosemary: Great question. We will continue to support the in/out keywords, even if we can also re-enable the start/stop phrases.

    Quick update: I just looked at the logs and there have already been hundreds of you switch to the new in/out keywords. Thanks for being so flexible everyone!

    [Reply]

  4. Quick update:

    We just received word that our upstream provider will be applying a patch to their software later today that will supposedly make everything return to normal. We’re crossing our fingers that they don’t break things further, and will keep you posted!

    Brandon Zehm
    CTO, TSheets.com

    [Reply]

  5. Just curious, why did they start blocking “start” and “stop” in the first place?

    [Reply]

  6. Hey Tom,

    Apparently the software they upgraded to treats a text message with “stop” as a command to stop sending any messages from the company (TSheets) to the mobile phone user.

    We also got word that this outage also affected 10s of thousands of users of Yahoo and other very large companies. It feels (somewhat) good to know it wasn’t just us who was affected.

    [Reply]

  7. — Quick update —

    As of 4pm MT everything is back to 100% functional! All users should be receiving text message confirmations again if you are setup to receive them. Even the old “t start” and “t stop” commands are working again, although to save yourself a few extra keystrokes we’re recommending that you stick with the “t in” and “t out” syntax.

    We’re still working on a full backup solution or number for use if this ever happens again, subscribe our blog to make sure you get notified of the details when it’s in place.

    [Reply]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>