Add in-place deletion with flumelog-offset and aligned-block-file #4

christianbundy · 2019-01-29T00:29:15Z

WARNING: this ~~does~~ might not play nicely with the current Scuttlebutt stack.

This does in-place deletion with flumelog-offset and aligned-block-file. For example, {foo:42} would be replaced with {} and six spaces to retain the same item length without actually holding any metadata. Is there a better way to do this? Lots of code depends on msg.value.author and such, so I'm wondering whether there's a better way to go about this. Maybe we should be doing replacements with something like this?

{
  "key": "",
  "value": {
    "previous": "",
    "author": "",
    "sequence": 0,
    "timestamp": 0,
    "hash": "",
    "content": "",
    "signature":""
  }
}

christianbundy · 2019-01-29T04:10:40Z

It's working!

node test.js generates a file called .test-db with three messages and then does an in-place deletion of the second message
node ssb-blocking does an in-place deletion of all messages from each author blocked by you. I had to restart Patchbay for this to take effect, and had to run node delete-views to get rid of the views (although this is much less important IMO)

The only error I've seen:

{ message: 'Key not found in database [%PFtoLIvnRsKBbvIOx4Ga6CKLdbG6vg0Ghepo2bJidE0=.sha256]',
  name: 'NotFoundError',
  stack: 'NotFoundError: Key not found in database [%PFtoLIvnRsKBbvIOx4Ga6CKLdbG6vg0Ghepo2bJidE0=.sha256]\n    at /home/christianbundy/src/ssbc/patchbay/node_modules/ssb-unread/node_modules/levelup/lib/levelup.js:160:15\n    at /home/christianbundy/src/ssbc/patchbay/node_modules/ssb-unread/node_modules/encoding-down/index.js:51:21' }

I think this is because we have plugins that do get() messages to the db

Future

I'd really like to hook this up as an ssb-server plugin so that:

apps can hook into the deletion method
blocked feeds are deleted on startup
views are allowed to do deletion (or rebuilt if they don't support it), see: https://github.com/flumedb/flumedb/blob/del-only/index.js#L146-L152

Concern

The above "future" section works with a one-app setup, but you could cause a problem like this:

run the above code in Patchbay, running plugins A, B, and C
close Patchbay
open Patchwork, running plugins A, B, and D
block feed
restart Patchwork (deletes from log, and views A B and D)
close Patchwork
open Patchbay
plugin C catches up, but doesn't learn about the deletion

This makes me wonder whether delete/rebuild requests should be flumelog messages that flumeviews can choose to respect, but that sort of seems like a pain. I'm not sure what the solution here is, other than having one set of plugins per log.

christianbundy · 2019-01-29T04:12:04Z

Also, the skeleton message that we're using to replace deleted posts looks like this:

{
  "key": "%00000000000000000000000000000000000000000000.sha256",
  "value": {
    "previous": "%00000000000000000000000000000000000000000000.sha256",
    "author": "@00000000000000000000000000000000000000000000.ed25519",
    "sequence": 0,
    "timestamp": 0,
    "hash": "sha256",
    "content": {},
    "signature": "deleted"
  }
}

evbogue · 2019-02-02T17:58:30Z

Hey @christianbundy I just saw this because I haven't logged into Github in awhile.

Where are we with this? Let me know if there's a experiment you want me to try.

christianbundy · 2019-02-02T21:06:16Z

Hey @evbogue, check %OoBqCtaYm6ayBQqCVlHi66vsWfvaK5+t98aqsXlRyZU=.sha256 for the latest, I'll keep that thread up-to-date.

The only caveat is that two indexes, `last` and `contacts2`, depend on flumeview-reduce, which seems to require a restart after `flumeview.rebuild()` before indexing starts again. See flumedb/flumeview-reduce#10 Also, this throws a handful of "errors" when the rebuild request is started, but they seem to be harmless. I'll add the errors as a comment in the PR (#4).

christianbundy · 2019-02-08T23:35:09Z

Warning: still doesn't play nicely with regular clients that don't understand deletion.

FWIW, here's what console output looks like when running server.js:

$ node server.js
  ssb-prune loading block list, this requires sbot running with up-to-date indexes +0ms
Listening on :::8008 (multiserver net plugin)
ssb-friends: stream legacy api used
  ssb-prune block list: [ '@FJDMuATvh0a3qIcIPcstUANVIj362fWkYCG5bnkZ5nA=.ed25519',
  '@nN8L47n0avj6bWi1vjVMPXfXfxdMn9g7UAbgcKKpqEo=.ed25519',
  '@isFCNRNxl80IIy2ibgenfgEPqqFQ3vU5qyXCu+mSSeI=.ed25519',
  '@q68IdbW1X3mTYGyIp/Zt1NJtJLe61W5o6NFfmgvVMMc=.ed25519',
  '@yOtM80uIUMA4lx6NRa7jhFl1Q813cfGp8pXlJbT0U1Q=.ed25519',
  '@EdsF0/yRN3H0NcAnJAASuHpkgH3F879esRtGVqOWdV4=.ed25519',
  '@UXT5EtqMI2g6PMnYnyX71+1FaJaG0NrHlyt/Qog+gBU=.ed25519',
  '@n2/9k0xfb0TalKmq9YUVNcV1zpF+SWsZARAlA3YCljI=.ed25519',
  '@JZyUVcyR2cpjd9fGHyqiBrLy+jNnq97dSWMz7CZNbKE=.ed25519',
  '@N2DZllcyA+oJ72JpEJ6V6Jns/Te2DGe9YGZe6xy8iIM=.ed25519',
  '@+rMXLy1md42gvbBq+6l6rp95/drh6QyACO1ZZMMnWI0=.ed25519',
  '@URSb7GMTxPuEygB3956QXsaYELO2rMoSvj0GdVTnrIw=.ed25519',
  '@7DharwDpk0M2Gw8I1el/U3VsRmebEUhIwnpd0kG/p6c=.ed25519',
  '@zdvCHq9MZKX8zM8z9nXx5YTa/vvOOL4t3tmMwCh8MS0=.ed25519',
  '@1MCDaYSDpjsgtMkgB2CfqrouffoxvLiRu/QICE7LQPs=.ed25519',
  '@7xMrWP8708+LDvaJrRMRQJEixWYp4Oipa9ohqY7+NyQ=.ed25519',
  '@INwxdkSDE5mRB7IUq1/kp0EFi2+G1AIk1sCygmBfxKM=.ed25519' ] +7s
  ssb-prune starting... +2ms
  ssb-prune done draining messages +21s
  ssb-prune deleting 1716 messages... +0ms
  ssb-prune done! +7ms
Error: view stream error
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:183:35
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-write/index.js:72:17
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-stream/throughs/async-map.js:18:11
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-stream/throughs/filter.js:17:11
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-cursor/cursor.js:7:24
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-cursor/skip.js:8:25
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-looper/index.js:6:5
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/looper/index.js:11:9
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-looper/index.js:11:5
  Error: aborted
    at Function.reader.abort (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-write/index.js:67:19)
    at close (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/flumeview-level/index.js:35:30)
    at Object.destroy (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/flumeview-level/index.js:43:7)
    at rebuildView (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:48:8)
    at Promise (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:151:15)
    at new Promise (<anonymous>)
    at Promise.all.Object.values.map.view (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:135:11)
    at Array.map (<anonymous>)
    at log.del (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:134:42)
    at Promise.all.catch.then (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumelog-offset/inject.js:117:21)
Error: view stream error
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:183:35
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-write/index.js:72:17
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-stream/throughs/async-map.js:18:11
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-stream/throughs/filter.js:17:11
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-cursor/cursor.js:7:24
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-cursor/skip.js:8:25
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-looper/index.js:6:5
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/looper/index.js:11:9
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-looper/index.js:11:5
  Error: aborted
    at Function.reader.abort (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-write/index.js:67:19)
    at close (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/flumeview-level/index.js:35:30)
    at Object.destroy (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/flumeview-level/index.js:43:7)
    at rebuildView (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:48:8)
    at Promise (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:151:15)
    at new Promise (<anonymous>)
    at Promise.all.Object.values.map.view (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:135:11)
    at Array.map (<anonymous>)
    at log.del (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:134:42)
    at Promise.all.catch.then (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumelog-offset/inject.js:117:21)
Error: view stream error
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:183:35
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-write/index.js:72:17
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-stream/throughs/async-map.js:18:11
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-stream/throughs/filter.js:17:11
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-cursor/cursor.js:7:24
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-cursor/skip.js:8:25
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-looper/index.js:6:5
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/looper/index.js:11:9
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-looper/index.js:11:5
  Error: aborted
    at Function.reader.abort (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-write/index.js:67:19)
    at close (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/flumeview-level/index.js:35:30)
    at Object.destroy (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/flumeview-level/index.js:43:7)
    at rebuildView (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:48:8)
    at Promise (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:151:15)
    at new Promise (<anonymous>)
    at Promise.all.Object.values.map.view (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:135:11)
    at Array.map (<anonymous>)
    at log.del (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:134:42)
    at Promise.all.catch.then (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumelog-offset/inject.js:117:21)
Error: view stream error
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:183:35
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-write/index.js:72:17
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-stream/throughs/async-map.js:18:11
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-stream/throughs/filter.js:17:11
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-cursor/cursor.js:7:24
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-cursor/skip.js:8:25
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-looper/index.js:6:5
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/looper/index.js:11:9
    at /home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-looper/index.js:11:5
  Error: aborted
    at Function.reader.abort (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/pull-write/index.js:67:19)
    at close (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/flumeview-level/index.js:35:30)
    at Object.destroy (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/flumeview-level/index.js:43:7)
    at rebuildView (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:48:8)
    at Promise (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:151:15)
    at new Promise (<anonymous>)
    at Promise.all.Object.values.map.view (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:135:11)
    at Array.map (<anonymous>)
    at log.del (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumedb/index.js:134:42)
    at Promise.all.catch.then (/home/christianbundy/src/flumedb-delete/node_modules/ssb-server/node_modules/ssb-db/node_modules/flumelog-offset/inject.js:117:21)

As far as I understand these errors are harmless, but we have to Ctrl+C because flumedb/flumeview-reduce#10 is keeping two flumeviews from being reindexed until a restart happens. Running node server.js again will start the server without any problems, and the two remaining flumeviews should reindex painlessly.

The only issue is that this only works if you're running a deletion-respecting stack, so if you open up vanilla Patchwork with your log.offset it's going to throw a bunch of errors about those nasty \x00\x00\x00... messages. Time for some pull requests!

evbogue · 2019-07-10T23:01:40Z

Hey @christianbundy! I saw you closed this via the Microsoft Github dashboard.

I'm not sure if it got solved or not, but I'd like to share my unrelated experience implementing this from scratch in bogbook.

I opted to store individual feeds in separate database entries, and then build a global log based on all of the feeds that have synced into the client. All of this happens in the browser, so we're only using pubs to sync individual feeds.

This means if you want to delete a feed, you only need hit the "Delete Feed" button and then regenerate the global log from the feeds that are still present in your browser's indexedDB.

If you want, syncing and then deleting my feed at my bogbook feed.

It's early days, but this approach appears to make deleting feeds that you no longer want in your browser incredibly easy.

christianbundy added 2 commits January 28, 2019 16:07

Add flumelog-offset deletion (!)

90ffc07

Ensure that all utilities still work

077c9fc

christianbundy added the enhancement New feature or request label Jan 29, 2019

christianbundy self-assigned this Jan 29, 2019

christianbundy requested a review from evbogue January 29, 2019 00:29

Use flumelog-offset with skeleton message

f6b27a3

Add ssb-server plugin (!) to handle deletion

a1c4498

christianbundy mentioned this pull request Mar 27, 2019

Epic: Feed deletion ssbc/ssb-server#651

Closed

34 tasks

christianbundy closed this Jul 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add in-place deletion with flumelog-offset and aligned-block-file #4

Add in-place deletion with flumelog-offset and aligned-block-file #4

christianbundy commented Jan 29, 2019 •

edited

Loading

christianbundy commented Jan 29, 2019

christianbundy commented Jan 29, 2019

evbogue commented Feb 2, 2019

christianbundy commented Feb 2, 2019

christianbundy commented Feb 8, 2019

evbogue commented Jul 10, 2019

Add in-place deletion with flumelog-offset and aligned-block-file #4

Add in-place deletion with flumelog-offset and aligned-block-file #4

Conversation

christianbundy commented Jan 29, 2019 • edited Loading

christianbundy commented Jan 29, 2019

Future

Concern

christianbundy commented Jan 29, 2019

evbogue commented Feb 2, 2019

christianbundy commented Feb 2, 2019

christianbundy commented Feb 8, 2019

evbogue commented Jul 10, 2019

christianbundy commented Jan 29, 2019 •

edited

Loading