sqlite database growing large

[justafish]

The onepassword_connect_data docker volume has an ACTIVITY_EVENTS table in the sqlite database that never gets pruned - in our case it has grown to 14GB in the space of 3 weeks (and has previously filled up a disk where the volume had to be destroyed and re-made).

How should this be handled? I see there's a OP_LOG_LEVEL environment variable but it's not really documented what this does and what values it can be set to

[Marton6]

Thank you for reporting this issue. We will look into reproducing and fixing it.

OP_LOG_LEVEL only configures the amount of messages that get logged to stdout, so changing that would not have any impact on the number of activity events logged by Connect.

[justafish]

@Marton6 if at all helpful here is our docker compose file:

version: "3.4"

services:
  op-connect-api:
    image: 1password/connect-api:1.7.3
    ports:
      - "8080:8080"
    volumes:
      - "./1password-credentials.json:/home/opuser/.op/1password-credentials.json"
      - "data:/home/opuser/.op/data"
    environment:
      - OP_LOG_LEVEL=error
    restart: unless-stopped
  op-connect-sync:
    image: 1password/connect-sync:1.7.3
    restart: unless-stopped
    volumes:
      - "./1password-credentials.json:/home/opuser/.op/1password-credentials.json"
      - "data:/home/opuser/.op/data"

volumes:
  data:

and we're also using this puppet module https://forge.puppet.com/modules/bryxxit/onepassword_lookup

[rabbitlair]

Hello @Marton6, are there any news regarding this issue? We have the file /var/lib/docker/volumes/onepassword_connect_data/_data/1password.sqlite growing up to 26 GB as of today, and it keeps growing. It would be nice to have some way to reduce the size of this file. Thank you!

[q0rban]

Just ran into this, where onepassword_connect has gradually used all disk on the server. I stopped the onepassword_connect service and deleted the sqlite database and started the service again. It'd be nice if this service could have a way to prune itself.

[JillRegan]

This relates to this issue in the Kubernetes Operator: 1Password/onepassword-operator#147

A short polling interval (example 60 seconds) of 10-20 secrets will also add 10-20 entries in the activity_events table every minute which leads to database growing substantially.

I wonder if a good enhancement here would be to add some sort of configurable log retention policy on activity_events. Users could configure to keep logs in a certain windows of time. Those entries that are not in that window are deleted.

[fbouynot]

Hi, we hit the same issue, and the connect pod is growing really fast and has to be evicted pretty often.

a good enhancement here would be to add some sort of configurable log retention policy on activity_events

Would it be possible to implement this suggestion?