Triage! Recovering Scrivener Files From Dropbox or Google Drive Errors

There is nothing worse than writing tens, hundreds, or thousands of words and then losing all of that work.

A while back I did a comparison of Google Drive and Dropbox, and briefly discussed the issues you can run into with Scrivener, the writer's fancy-pants friend. I didn't really focus on Scrivener because I was comparing cloud storage solutions (and I really should take a look at SkyDrive sometime I suppose, now that I'm running Windows 8).

Anyway, reader Chaim recently got in touch with me to ask for help. He was using Scrivener with Google Drive, and somehow his work across two computers got out of sync. The result? Lost work.

I've had the same thing happen to me more than once working with Scrivener on Dropbox. It's a problem that afflicts anyone using Scrivener with either cloud product. And it sucks. Simply put, you must make sure that you've closed out of Scrivener and let your project sync to the cloud before you open it on any other computer. Period. If you don't, your project file will get screwed up and you will appear to lose work.

I say appear because, hey! All is not lost. And I'm here to walk you through getting your files back.

Step 1: First, copy your Scrivener project out of your cloud storage and onto another location on your local computer. This is like isolating a crime scene, or quarantining a patient: you don't want Dropbox or Google Drive to do any more futzing with your files while you're trying to recover your writing. Scrivener stores projects in a folder with the .scriv extension (on Macs this will look like a file, but trust me, it's a folder). This is what you need to copy somewhere - your Desktop will work just fine.

Step 2: Next you need to open the .scriv folder and take a look at its ooey-gooey contents. On Windows this just means opening the folder. On a Mac, you'll need to change the folder name and get rid of the .scriv extension, so the operating system will know to treat it as a folder instead of a file.

Either way, once you're inside the folder you want to look for the Files folder. Open that, then open the Docs folder inside it. You'll be greet with a big old list of numbered RTF files. (RTF is like a Word document's handicapped cousin.) These are where Scrivener stores all of your writing... and odds are that writing you "lost" is still there, even if Scrivener can't see it.*

Open up the RTFs, one by one, and look for your lost writing. If one of the RTFs is labeled "conflicted" or otherwise numbered oddly, that's probably the one with the missing text. Once you've found it, open up your original Scrivener project (the one that's still in your cloud storage folder) and paste the missing text in. Voila, done! This is what worked for Chaim, and hopefully it worked for you too. If not, move on to Step 3.

Step 3: Is your text not in the RTFs? Then it's time to check your patient history... I mean backups. One handy feature of Scrivener is that it automatically backs up the last five copies of your project file for you, by default. The problem is that it can be hard to find out where Scrivener is storing the backups if you just go looking for them. Fortunately, Scrivener has made this easy for you.

Open your project and select the "Tools" dropdown at the top menu, then "Options...". Click the "Backup" tab in the window that opens up. Along with some other options, you'll see your backup location and a button that says "Open backup folder...". Press the button and voila! Backups.

Now, with any luck you'll see a ZIP file named after your project. Copy the most recent one to a convenient location and unzip it. Then, open the unzipped project in Scrivener and look for your lost text. If all is well, you'll see it. If not, you'll want to go to Step 4.

Step 4: At this point your lost work isn't in your current project or your backups, and it's time to get serious. Get out your scalpel, fire up your web browser and browse to your cloud storage solution's website. We're going to check your version control system.

What's version control? Basically, when you save changes to a file to the cloud, Google Drive and Dropbox will both keep the previous version of the file for 30 days (by default; you can pay to extend this to life). If you make an error ("Whoops! I deleted 3,000 words and hit the Save button by accident!"), you can pull up the previous version and recover your work. The details vary between Google Drive and Dropbox, so look them up in their respective manuals.

What if you don't save that often? Well, Scrivener is your friend here too. It has an auto-save feature that fires off every two seconds, to keep your work up to date. If you're editing your project while you're hooked up to the cloud (i.e. you have an Internet connection), you're fine. If not, well... maybe go back to Step 2 and take another look?

Hopefully by this point you've got your work back (voila!), so let's consider leading a healthy lifestyle for a moment, and move back to...

Step 5: Backups. As nice as Scrivener's default backup configuration is, you can and should be able to do better, especially if you're using cloud storage already.

Open up the Backup window again and take a look at the options. First, make sure "Turn on automatic backup" is checked! Then, tell Scrivener when to back up your work. I use "Backup on project close", but if you're paranoid you can select "Backup with each manual save". I also like to compress my backups into ZIP files and add the date to backup file names, just in case I need an easy time stamp reference to some colossal revision effort.

Finally, there's the big two options I recommend. First, for the "Retain backup files:" option, I recommend keeping ALL of your backups in perpetuity. For modern computers the space cost isn't going to be a big deal, and you might be grateful for the option to look back at your work months later (say, if you want to harvest a discarded character for some new project).

Second, create a folder in your Dropbox or Google Drive for your Scrivener backups, and change your "Backup location:" option to point to that folder. You should only have to do this once, although if you work on both Mac and Windows computers, you will have to do it twice. I'd also recommend using a separate backup folder for Windows and Mac under those circumstances. I don't know that the different operating systems will clobber each other, but I also don't like to take chances.

Once more, voila! You now have a Scrivener configuration that will silently back up all of your work to a separate folder, time stamp every backup, and upload them to the cloud one by one. Your work should have a long and healthy life - but maybe hook an automated backup drive to one of your computers, too. Just in case.

Hopefully you've found this useful. If you've got any other tips on how to keep your writing safe, post a comment and share!

*The dirty details: Scrivener uses a .scrivx project file to tell it which RTFs contain what text. If your .scrivx folder gets screwed up during a cloud storage sync, Scrivener won't be able to "see" the file, even though the actual RTF is probably still there. As far as I can tell this is the most common file syncing error that Scrivener runs into.

Writer's Review: Dropbox vs Google Drive

If you're anything like me, you worry about backing up your writing. All sorts of things can burn you: hard drive failures, lost notebooks, actual fires, the dog ate my manuscript... anything, really.

Cloud storage is getting to be the backup solution of choice for a lot of people. Put your stuff on the Internet and you can download it anywhere! Never worry about losing your work because it's living on a gajillion servers!

I've been using Dropbox for months to backup my work. It's simple: You have a folder on your computer. You put your writing into it. If you're connected to the Internet, your work gets uploaded. Done. As long as you don't have more than 2 gigabytes of files to upload, you don't even have to pay for anything.

Then this week Google released Google Drive, which is Dropbox by Google. It works on the same principle (see folder: drop files in), but it integrates with Google Docs and comes with a whopping 5 gigabytes of storage, free. So if you're a writer, is there a good reason to switch?

Storage

It's hard to argue with 5 gigs of free storage compared to Dropbox's 2. Granted, it's very easy to get a lot of additional Dropbox storage, free, especially if you have a smartphone. And if you're just using these programs for writing, well, it's fairly hard to fill up 2 gigs of space with only text files and Word documents.

With that said, Google Drive unquestionably has more storage, cheaper, right out of the gate. And if you do get to the point where you need to pay for additional storage, Google sells it for about half the price Dropbox does.

Advantage: Google Drive

Syncing

Dropbox and Google Drive sync files in fairly different ways. The basic idea is the same - file changes are uploaded to your cloud storage from one computer, then downloaded from the cloud to any other computers you've got the program installed on. But Dropbox has a feature called LAN sync, which lets you sync files across Dropbox-equipped computers on your personal LAN (i.e. your home wireless network) without reaching out to the Internet. This means you get much faster syncing if you have multiple computers at home. Google Drive doesn't do this at all: everything has to upload to and download from Google directly.

Another issue comes with resolving conflicts, where you try to save two different copies of the same file to the cloud. Say you edit a document on one computer while you're offline, then edit that same document on a different computer while you're online. When you connect that first computer to the Internet, the two files will be different, and you'll need to resolve the conflicts to make sure you've got the right version of the file saved.

Dropbox handles this by renaming the conflicted file, identifying it as conflicted and adding the name of the problem computer to the filename. This makes it very easy to figure out what happened, and to clear up the conflict.

Google Drive, on the other hand, just uploads both files and saves them in the cloud with the same filename ( but different timestamps). The duplicate files get downloaded to all computers as copies, renamed using whatever native scheme the operating system uses (i.e. test.txt and test (1).txt). The problem with this is that you can easily end up losing track of which file is which. Here's an example:

- I create a file test.txt on my laptop while I'm offline. It says "Hello".

- I then create the same test.txt file, with different text in it, on my desktop while it's offline. It says "World".

- I connect both systems to the Internet and sync with Google Drive.

- I check my laptop. There's a file called test.txt that contains "Hello", and a file called test (1).txt that contains "World".

- I check my desktop. There's a file called test.txt that contains "World", and a file called test (1).txt that contains "Hello".

- This happens:

You can see how that could be a problem.

Advantage: Dropbox

Versioning

Both Google Drive and Dropbox have file versioning features. Basically if you edit a file, you can retrieve the previous version from your cloud storage for a fixed period of time.

Google Drive seems to keep your revisions pretty much forever, which is extremely handy. Dropbox can do the same thing, but only if you use a paid storage plan. If you only use the free storage, you only keep your revisions for 30 days. On the other hand, Google Drive only keeps revisions for files that you haven't deleted. If you delete a file and empty your trash, it's gone forever. Dropbox will let you reclaim your deleted files, but only for the times noted above.

Advantage: Google Drive - but only the free version

Writing Tools

I use Scrivener for most of my manuscripts these days, so it's important that my cloud backup system can cope with Scrivener's file format (really a bunch of folders containing text files and RTFs) well. On the surface, both Google Drive and Dropbox don't have any problem handling it. It's all files, after all.

Google Drive does have big one gotcha, however, which is that it wants to convert any RTF files you upload into Google Docs format, and store them like that. This is a huge no-no if you want Scrivener to keep working. A Google Doc is not an RTF, and you can't edit it like one, even if you have Google Docs offline set up. Fortunately it's very easy to disable this "feature". Dropbox doesn't have an equivalent cloud file format, so it's a non-issue.

There's also the syncing issue I mentioned above. A conflicted Scrivener file is a pain in the ass no matter what program you're using, but Google Drive has the potential to be a much bigger pain in the ass than Dropbox.

On the other hand, Google Drive does let you edit your files directly in the cloud using the well-developed Google Docs interface. If you're don't want to pay for Scrivener or Microsoft Office, this is a very good rich text editor (not as good as LibreOffice, mind, which is free). Also, Google Docs don't count towards your storage limits. Again, Dropbox has nothing comparable to this.

Advantage: Tie - it really depends on how you want to work

Collaboration

I haven't collaborated on my writing online, so I really can't speak to this from personal experience. However, Google Drive lets you collaborate on a per-file basis, where Dropbox limits you to collaborating by sharing entire folders with other people. This makes Google Drive easily a much more collaboration-friendly solution, giving you a greater amount of granularity in how you share files with much less hassle than Dropbox.

Advantage: Google Drive

Summary

Overall I'd say that either solution is very useful as a backup solution, if nothing else. Which one you choose is largely going to depend on how you like to work.

Google Drive gives you more space than Dropbox for less money. It has a robust collaborative editing system, and gives you the ability to create and edit files directly on the cloud. For somebody who wants to store a lot of files, or wants to work in a team environment frequently, Google Drive takes the crown.

On the other hand, Dropbox beats the pants off of Google Drive when it comes to syncing, both in speed and conflict resolution. It's also a bit nicer about recovering files you deleted, if only for 30 days, and Dropbox's storage space limitations can be overcome (and exceed Google Drive's limits by 11 gigabytes, if you work at it) without too much trouble. If you don't collaborate, and you work across multiple computers in your own house, Dropbox is the preferable solution.

I will probably end up sticking with Dropbox for now, if only because I'm already familiar with the program, and I value the syncing benefits Dropbox has over Google Drive's collaboration tools. If Google Drive improves over time, I might well consider switching later.

P.S. If you want to give Dropbox a try, please use this link to sign up. I could use the extra space.

The Dropbox TrueCrypt Paranoia Corollary

Well, it's a week now since I started using TrueCrypt to encrypt all of my files on Dropbox, as detailed in my last post. And, I'm sad to say, after one week I'm ready to drop the encryption and go back to using Dropbox, um... unprotected. I've got a few reasons why:

TrueCrypt adds extra steps. I'm annoyingly fragile when it comes to getting into a writing frame of mind. Most anything can distract me at a critical juncture, be it a Corgi jumping in my lap for attention or that pile of laundry I forgot to pull out of the washing machine three hours ago. So having to find and mount a TrueCrypt volume before I can find and open the Word document I want to work on can actually stop me from opening that Word document. And I can't have that, now can I?

TrueCrypt nerfs Dropbox's versioning system. An obvious point I think I mentioned before, but as long as my files are in a TrueCrypt volume, Dropbox can't version them individually. For the most part, I haven't had to take advantage of this feature. Still, I've done enough development work with Subversion to know I damn well want it.

TrueCrypt slows down my syncing. I noticed this right from the get-go: with TrueCrypt, it takes Dropbox about a minute to two minutes to sync any update I make to the files in it. It's not enough to be unusable, but it's just enough to get annoying after awhile, especially if I just want to turn my computer off and go to bed after a save.

TrueCrypt actually makes me more paranoid. This is the biggie. I don't keep anything on Dropbox I would mind people looking at. Mostly it's backup copies of eBooks, evidence of CISSP CPEs I've earned, and my manuscripts. Now, it's conceivable that Dropbox is going to steal all of my shit and do nefarious things with it, but I can't imagine what. The same goes for any random hacker who breaks into my account. At worst, I'd expect to get hit with some jerk deleting everything in my Dropbox, which is why I keep offline backups.

Now, with TrueCrypt, I get the added fear that my encrypted volume might get corrupted, either from bouncing it between operating systems or by forgetting to sync updates to the volume in the correct order and having Dropbox introduce a mess of file errors. I've made that kind of screw-up before, and I don't want to do it again and have it blow away all of my files.

So, no more TrueCrypt, at least not with Dropbox. I may go back to using it to encrypt some things down the line - an In-Case-Of-Emergency file, for example - but right now it's worth more to me to have an easy time using Dropbox.

I'd go into the annoying filename quirks I ran into when I copied everything out of my TrueCrypt0 volume, but the Corg0i just jum1ped int0o my lap and star1ted1 licking my keyboard. Stop that Lina!

The Dropbox TrueCrypt Paranoia Conundrum

I've been using Dropbox for months now to back up my important files to the Mystical Cloud that drifts through the Internet. Aside from one minor wrinkle of a file conflict (which I easily resolved), it's done sterling service. My files are backed up across multiple computers and their associated backup hard drives, not to mention the Dropbox servers themselves. Losing my work in a catastrophic incident should, theoretically, be impossible.*

But over the last couple of weeks Dropbox has gotten some bad press. Aside from the security breach (see "bad"), none of this is really a surprise; if you put your data on somebody else's computer, they are going to have to protect themselves legally in some fashion. And because copyright law is a hydra with infinite heads and a bad attitude, even an innocent company is going to look bad trying to comply with it.

That said...

The security breach did bother the heck out of me. I don't know of anyone who'd want to look at my files with malice in their heart, but I also didn't know anyone who'd want to run up a $300 bill on my Amazon account. Shit happens. And while all of my files are perfectly innocent**, I still feel less than clean knowing that someone could be looking at them right now with their filthy eyes...

Enter TrueCrypt. My files are now wrapped in one big, ambiguous blob of encrypted data, one that no one is liable to crack open in the next decade without the correct password. So I am, relatively speaking, secure.

But can I still be productive?

TrueCrypt bundles your data into what is effectively an encrypted hard drive. With the right password, you can mount it and edit everything on it just like any other filesystem. So what's in my Dropbox account now is one big file that is 1.99GB in size. There are some issues with this:

Syncing. The initial upload of this file took a good three hours. Fortunately Dropbox does bitwise syncing, so it only needs to resync the bits of the file that change during an edit. I opened up a Word document and added some text, and Dropbox updated it in about a minute.

Syncing again. The encryption works fine if I only edit the file on one computer at a time. Since that's what I do anyway, this is no big deal. But if I forget and let my systems get out of sync, I'm going to wind up with a 4GB conflict that could potentially corrupt my data. So be careful with those edits, m'kay?

Nerfed features. Dropbox allows you to access your files from the web, but not if they're in one big encrypted blob. Ditto for sharing files with other people, or versioning them. Happily I'm not using these features anyway and don't plan to start.

So this isn't a perfect solution. Still, I think it's a happy balance between ease-of-use and security, which is all I can ask for.

And if it turns out to be more annoying than I bargained for, I'll store my files in my data dog instead.

*This is tempting fate. I'm certain some alien intelligence with a global-scale EMP generator is reading this and giggling.

**Pay no mind to that donkey in the corner.