Possible to drop old ops from the oplog to save space?

[devongovett]

Is it possible to drop old operations from the oplog to free up database space after some specified timeout? We don't really need a full history of all operations after a few days (right?), and when operations are generated for every keypress pretty much, the database can grow very fast.

I tried using a Mongo ttl index on the ops but that doesn't work: We get a Missing oplog error from the redis driver when both redis and mongo have been cleared. Do we only need to keep around the latest operation (or 10 say), or is there a better solution? Thanks!

[dignifiedquire]

Take a look at this issue, as we had some testing and problems around this share/sharedb-mongo#7

[josephg]

Yes, as I alluded to in that pull request, I have no idea what will happen
if ops are deleted from the oplog right now. I wrote livedb expecting all
the ops to be there and nobody has written tests to see what happens if
that's not the case.

I'm starting to have the same problem in my production database too - I'll
do a testing pass soon to figure out what livedb does and how to handle all
this. I'm expecting to drop ops older than 2 weeks or something. ShareJS
should also be able to recover from the error - it'll lose local changes
(unavoidable) but we could bring the client document up to date at least.

[devongovett]

@josephg did you ever make any progress on this?

The ideal solution for me would allow combining operations together too, as described by josephg/ShareJS#318. That way, as you go farther back in the oplog, the changesets would get less granular per database record. Or maybe you could just store combine the last N operations together when writing to the database always if granular operations are not needed. Maybe a way to do this would be to store ranges of versions per mongo record or something. All of this seems like it would be rather hard to implement though.

Even if that's not possible, dropping ops older than a configurable time period would be good enough for most purposes, unless some sort of document revision history feature is needed.

I'm happy to work on this if we can figure out what the desired behavior should be. I'll need to work on it soon since we're going to start having this problem. Thanks!

[devongovett]

I just noticed this commit to livedb-mongo which avoids the error I described in the original issue. If there are no ops, the version returned is the version of the snapshot (if it exists) rather than 0.

I tested by deleting all ops for a document from mongo and redis and attempting to edit it, and it now seems to work properly: the oplog ends up with one op in it with the correct version number. I plan to write tests for what happens if old ops that have been deleted are requested (should be some sort of error). Not sure what ShareJS clients should do in that case. Maybe either let the app alert the user, or refetch the snapshot?

[nateps]

This should no longer be an issue as of the sharedb rewrite. It should work properly if no ops are present or only the latest ops are present and all submissions and fetches are based on versions that are recent enough.

If a client has an old doc version and the required ops are missing, an error will be returned as appropriate.