Uploaded image for project: 'ListenBrainz'
  1. ListenBrainz
  2. LB-50

use bulk kafka and cassandra read/insert methods

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Normal Normal
    • None
    • data store
    • None

      4:35 PM <russss> we're not batching the cassandra writes, nor the kafka reads (if that's possible)
      4:35 PM <russss> there is a listenstore.insert_batch(listens) method
      4:36 PM <russss> which does scatter-gather on the cassandra insert
      4:36 PM <russss> so ideally the process should be bulk-read 1000 listens from kafka (ideally with a timeout), then bulk-insert those into cassandra

            rob Robert Kaye
            alastairp Alastair Porter
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package