Notes from Redis Conference 2015

14 minute read

  • Redis Conference 2015 at the Innovation Hangar at the beautiful Palace of Fine Arts in San Francisco.
  • Thanks to sponsors Rackspace, Hulu, Redis Labs, Heroku, and others.
  • Talks list
  • Redis’ 6th birthday today!

Redis Keynote by Salvatore Sanfilippo (Pivotal) a.k.a. @antirez

  • Genesis
    • Big equivalence principle : Today’s startups can do everything with everything, but a subset of tools can be better suited for a given use case
  • Focus & Relevance
    • Cache, queue, shared state, database?
    • Redis is a small, networked, DSL interpreter, with optionally persistent state. Hence multipurpose.
  • Economically sustainable?
    • It is rare to get a lot of work from non paid developers
    • Pieter, Matt, Salvatore : VMware, Pivotal sponsorship
    • Most contributions to core and other areas often sponsored or with ROI in contributing.
  • Development model
    • 95% users happy with feature set, 5% create pull requests and want more; number of users grow, 5% grows bigger but core team size remains same, so number of github issues is not a good metric.
    • New branching model
      • Stable (2.8) <- Testing
      • Testing (3.0) <- Unstable
      • Unstable (3.2) : New features go here
  • Future optimizations (prefix “maybe” to every point below)
    • If you do not use pipelining, most time will be spent in network I/O
    • Memcached threaded I/O?
      • Threads for socket I/O, request parsing, etc.
      • Command execution via Global Lock
      • May require big core changes
      • Worth a try
    • Quicklists
      • 90% less memory - RDBs more compact
      • Faster to serialize
      • Same concept is applicable to Sets and Hashes at least
    • Cluster optimizations
      • Non-blocking migrate
      • Auto rebalancing
    • The Big Refactoring Project
      • Redesign in terms of granularity + additional layers of API
    • Operations optimizations
      • Diskless ops WIP
      • nosync branch for slaves to get only new data from master, not history
      • Sentinel changes
    • API changes
      • Bloom filters
        • API is not obvious, since Redis has no object creation rule
        • Can’t use fixed params like HyperLogLog
        • Design effort needed
      • Geo coding
        • Matt’s branch 90% ready to be merged
        • Useful for lot of users
      • More list ops
        • Lists don’t have obvious APIs
        • LJOIN & LSPLICE
      • Unification of types?
        • Hashes and Sorted sets unified?
        • Ordered maps
      • Frequency estimation
        • You can already use sorted sets for top N problems
        • Good companion to HyperLogLog
        • Binary counters
      • Specialized stripped down spin-offs
        • Disque / queues
        • Multimaster cache
      • Evolution of caching
        • Make it simple to mutate only existing state
        • More APIs to check / analyze data
  • Why Redis
    • Better system software -> Better technology -> Better world

Redis in the Real World by Bill Anderson (Rackspace)

  • How are users using Redis
    • Rackspace surveyed tens of thousands of instances of Redis hosted for customers
    • 600 billion commands
  • Top Commands
    • 23% : Get
    • 18% : Setnx
    • 13% : Llen
    • 7% : Lpop
    • 6% : Zrangebyscore
  • Operations by Data type
    • Lists > Strings > Sorted sets > Hash > Set
  • Is it coincidental that most used commands are the fastest commands?
  • Reads, Writes, Checks
    • 54% : Reads
    • 35% : Writes
    • 11% : Checks
  • List operations
    • 36% : LPOP (write)
    • 36% : LLEN (read)
    • So we use list data structure as a queue?
  • Sorted set
    • 84% : ZRANGEBYSCORE (reads)
  • Hash
    • 42% : HGETALL (all)
    • 28% : HMSET (subset)
    • But shouldn’t we just use a string key + JSON value to store hash instead?
    • Strange that HGET is not commonly used!
  • What does all this mean?
    • Top commands are indeed fastest
    • Since we know time complexity, we can predict performance of Redis commands, this is great
    • Favor Setnx > Set in general benchmarking
    • Favor list ops in general benchmarking, then sorted set
  • Benchmarking
    • Benchmarking makes you go bald ;-)
    • First rule of benchmarking : Don’t do it
    • Benchmarking is like choosing a car based on it’s top speed
    • Don’t benchmark Redis, benchmark your application!
    • Instrument your code, gather the data, analyze the data. That’s what matters.
    • If you still want to : redis-benchmark, memtier
    • redis-benchmark is by Salvatore
      • Good
        • Gold standard
      • Bad
        • Doesn’t support all commands
        • Can’t do complex scenarios
        • Terrible output format for analysing
    • memtier
      • Good
        • Can tune writes vs reads
        • Can adjust concurrency
      • Bad
        • GET/SET isn’t a Redis benchmark, it tests networking (99% of time in networking)
        • Terrible output format for analysing
    • Time of day for benchmarking is important because your network could be saturated
    • Let us benchmark by use case
      • Leaderboard
      • Object cache
      • Queue
      • Page cache
      • DB query cache
    • Introducing Commissar, to do exactly this
      • Both benchmarking as well as documentation on design patterns a.k.a use cases
      • Separate server time vs. network/client time
      • Example Leaderboard output - 99.91% time spent in network/client, 0.09% in Redis server
      • Will allow us to compare uses cases
      • Will allow us to compare and improve implementations for a use case
      • Will allow us to compare and improve client libraries
      • But there are many different ways to do a thing…
      • How can you help?
        • Write more implementations
        • Write a spec for a new scenario
        • Port to your favorite programming language

Monaco, Redis as a Service at Hulu by Keith Ainsworth

  • What is Monaco
    • Clustered service that provides Redis as a service
    • Supports sharding for large data sets
    • Addresses production Redis challenges (setting up clusters, failovers, etc.) that was faced by multiple teams
  • Production Redis issues
    • Clustering
      • Redis Cluster is great BUT one management layer for one cluster of redis servers
    • Persistence
      • FS based persistence has quirks
        • RDB can use huge amounts of RAM, fork speed
  • Clustering
    • Monaco provides one management layer for many redis server clusters
    • Per cluster
      • Monitor state of all servers
      • Maintain replication roles & failover
      • Expose servers transparently to clients (use existing client libraries)
  • Persistence
    • Rack-aware clustering + External backups
  • Web interface
    • Create a cluster, etc.
  • Failovers
    • Monaco nodes do health checks on redis servers
    • If a failure, promotes DB replicas for any lost servers in order of their replication sequence. After ensuring all user servers have a responsive master, Monaco creates replica servers as necessary
    • If redis-server failure (never seen in production), Monaco slaves notify Monaco master that will pick ideal redis slave to promote to redis master
  • Major components
    • Load balancer
      • Provides interface to redis-clusters in Monaco
    • Monaco masters
    • Monaco slaves
    • Hosted on internal Heroku-like service
  • (incomplete notes, could not capture many points because speaker had to run through the slides)
  • Monaco cluster node
    • Daemon : Maintains threads
    • LeaderPaxos : Maintains leader <- Quorum of Monaco nodes
      • ZeroMQ
      • Similar in behavior to redis-sentinel
    • Master : Responsible for MGMT DB <- Web API
    • Slave : Maintains user DB
    • Stat reporter : Continually polls local MGMT DB for redis-servers allocated to that node
    • You built Redis as a service on top of Redis!?
  • Demo
  • Results
    • Monaco eliminated a lot of inefficient hardware use
      • There were many single-purpose VMs and clusters
    • Redis can be guiltlessly used for a tiny cache or huge persistent DB, without a barrier of operations
    • Hardware : 12 x 24core, 128 GB RAM servers, 5 more this month
    • 110 production clusters of redis-server
    • 190 unique redis-servers
    • 468.5 GB total hosted space
    • 1.6 GB/s in, 1.1 GB/s out
    • ~1 million ops per sec
  • Beyond Hulu
    • Desire to open source
    • Hard part is load balancer configuration API which is currently internal Hulu API
  • Problems & Future Work
    • Beta support for Twemproxy as a proxy for sharding and multiple databases (did I understand that right?)
  • Questions
    • Latency issues because of load balancer? Not really, an extra hop in the network path
    • Redis-server version upgrades? You can mark a node in maintenance mode, upgrade and then continue
    • Paxos implementation used? Custom code following “Paxos made simple”

How to benchmark Redis against other databases by Itamar Haber (Redis Labs)

(Lunch discussions)

Redis seems to be usually an individual developer’s decision to use and then it grows and grows in scope of use, as opposed to centralized decisions of the main database, etc.

Redis as a data store for personalization engines by Ted Price (Knowd)

  • Web analytics
    • Redis changed everything, it made storing web analytics trivial
    • Use hashes, sorted sets
  • Use case : Tracking Links + primary category + secondary category
    • Started tracking users
    • Recommend content based on click-through rate (CTR)
    • Use HttpRedis2Module : where possible, hit Redis server directly from nginx
  • Use case : Classifiers
    • Gender classification of website, article (women, neutral, men)
      • Fact: Celebrity gossip visited almost equally by women and men
      • For returning users, filter widgets by classification, and this resulted in traffic increase
    • Geo classifier
      • Publishers wanted more North American traffic
      • (This also supports Salvatore earlier saying that geocoding would be a great API addition to Redis server)
    • Tier classifier
      • Tier based on website’s traffic levels
    • Use SVM (Support Vector Machine)
      • Use lists, hashes, sorted sets
      • Ruby gems aplenty
  • Other Redis use cases
    • Cache busting
    • Job queueing
  • Summary
    • Performance makes it an ideal choice for real-time personalization, clustering, segmentation of content
  • Q&A

Real-time Events-based Document Index Store by Siddarth Kothari (Appbase)

  • https://appbase.io : now open source
  • why?
    • relational data vs linked data
  • Pull a Mongo on top of Redis!
    • Collection has documents (JSON object)
    • Add references : { "/ref1" : "user/item#3" }
      • user is a collection
    • Paths
      • Network topology : a -> b -> z -> x becomes “/a/b/z/x”
    • Event subscriptions
    • All built on top of simple Redis data structures

Redis at Heroku (missed attending this since multiple tracks)

https://speakerdeck.com/hgmnz/redis-at-heroku

Building a massively scalable job queue on Redis by Jon Hyman (Appboy)

  • Appboy is marketing automation for mobile apps
  • Job queues
    • Hundreds of millions of jobs per day
    • 100K rps
    • “Redis is the only database that actually delivers what it promises”
  • Scheduled jobs
    • zadd payload to sorted set key with score of timestamp for job
    • zrangebyscore (O(1)) + zrem
    • Use Lua to overcome race conditions
  • (More details, hopefully Jon will put up the slides online)
  • Use jitter (random microseconds) added to the scheduled time so that it doesn’t load Redis zrangebyscore
  • Maintain counters on top of the queue using MULTI
  • Reliability
    • rpop means queue message can be lost in crash
    • Use rpoplpush to pop from one queue and push to another reliable queue
    • When job is finished, lrem from reliable queue
    • Check queue on process start for data recovery
  • Scaling
    • One instance for everything
    • Increased machine size
    • Dedicated Redis process for job queue
    • EC2 latency / noisy neighbours hurt performance, moved to physical machines via Rackspace OnMetal
    • Spikes in job queue peg individual CPU
    • At some point, you will hit your max rps and you may not know it
  • Application-level sharding
    • Round-robin lpush to multiple instances
  • Queue starvation
    • First thing everyone does is add simple priority queues or weights
    • Queue weights are not a bad idea, but static queues are
    • Queue per customer is hard to scale because a lot to manage
    • rpoplpush supports only one key at a time
    • Create a queue as-needed for work, remove it when work is done
      • Add new queue name to a “dynamic queue” set
      • When looking for work, smembers to find all dynamic queues
      • When rpoplpush-ing dynamic queue, if no job found, atomically srem using Lua to re-check queue to avoid race condition and losing jobs
      • Be conscious about rpoplpush, use brpoplpush where possible
  • Pause processing for maintenance of data
    • How to pause generic, non-dynamic queues?
    • Tag all jobs with customer identifier
    • Push customers under maintenance using PUBSUB
    • When jobs are picked up by worker server, check customer identifier, re-enqueue
    • Easily done with middleware around job processing
  • Here be dragons
    • Restoring from backup = hard / dangerous; definitely use slaves
  • Q&A
    • What if a job worker died, what happens to process in a reliable queue?
      • Cron job that checks entries in reliable queues and sends alerts

Running Twemproxy in production by Manju Rajashekhar

  • Proxy for memcache and redis, built at Twitter, open sourced in 2012, used now at pinterest, wikipedia, twitch, snapchat
  • Motivation: Lots of remote connections to Redis, so familiar computer science solution of indirection : twemproxy is deployed as a local proxy on each app server
  • Not recommended to deploy as a remote proxy
  • Features
    • Fast & lightweight
    • Persistent server connections
    • Implicit protocol pipelining (will work across multiple connections connected to one twemproxy)
    • Shards data automatically
    • Ketama a.k.a consistent hashing
  • Avoid double-copy problem (?) using mbuf (fixed-size memory buffers)
  • Fault Tolerance
    • Client retries on transient errors
    • Don’t do it on invalid argument errors, etc.
  • What happens if it is a slow redis server?
    • Configure timeout on twemproxy, it is infinite by default
  • Re-routing on failures
    • Remove host from ring after 3 consecutive server failures / timeouts
    • Use auto_eject_hosts: true when redis is being used as a cache
    • Use auto_eject_hosts: false and node names (instead of host names when doing sharding), when redis is being used as a database
    • Have number of client retries > number of twemproxy server retries if you want a query to have highest chance of success
  • Simultaneous failures
    • Have larger server_retry_timeout
      • Trade-off between transient failures and permanent failures
  • Use node names for hashing, instead of host names or IP addresses
    • This makes deployment easy
    • Use server1, server2, … instead of 127.0.0.1:6379:1, 127.0.0.1:6380:1, …
  • Check configuration syntax errors early
    • Use --test-conf flag
  • Logging
    • By default, disabled
    • Use at log level LOG_NOTICE (-v 5) in production
  • Graphing ejected servers
  • hash_tag: “{}”
    • What if you wanted to do SDIFF on set 1 on server 1 and set 2 on server 2?
    • Map two different keys to same server by suffixing a common tag to the keys, but this should be in your design early
  • Q&A
    • @antirez requests supporting redis cluster protocol
      • Manju is busy with his own startup and would love help and is willing to mentor volunteers

Scaling Video Views by Andres Rangel (Hulu)

  • Problem setting
    • High traffic (10K+ RPS) of data to be tracked per user, per video (even video progress!)
    • Low latency (99% < 20ms)
    • High Availability
    • Scalability
  • Architecture
    • Long ago, when there was no redis cluster, built masters and slaves via zookeeper
    • From MySQL to Redis + Cassandra
    • Redis as cache
      • Add job to queue to load data from Cassandra to Redis
    • Redis as job queue
    • Redis as database : video progress, video metadata, recommendations list, etc.
    • Redis as sync point : locking for cron jobs
  • Configuration
    • 64 shards
    • 12 physical boxes
    • Sharded by user id, simple % operation
    • 384 GB RAM boxes
    • Dual Hex core, 24 hyper-threads
    • 1 GB/s network
    • Redis 2.8.19 - recently upgraded
  • HA-Redis, similar to official Redis-Sentinel design
    • sentinel : monitoring nodes
    • client : drop-in replacement for standard python/java redis client
      • reads from zookeeper to determine who is master, slave, and then will make the actual connection
    • Why not Redis Sentinel? HA-Redis predates Sentinel
  • (Lots of implementation details of how they use Redis & Lua scripting)
  • Keep up to 1/2 of RAM free in case there’s a slave sync
  • Single-process architecture can be scaled by multiple instances per machine
  • Lua scripting is very powerful, helps reduce data in redis, helps reduce network trips
  • If you want low latency, do not save to disk
  • To save memory usage (50% reduction) in redis (tradeoff is CPU usage), set *-max-{ziplist,intset}-entries to 4096
  • http://blog.pivotal.io/pivotal/case-studies-2/case-study-how-hulu-scaled-serving-4-billion-videos-using-redis

The Secrets behind Redis Development by Matt Stancliff (Pivotal)

  • Freewheeling talk
  • Worked 16 hours per day for 6 weeks to implemented quicklist (mentioned by Salvatore that it reduced memory usage to one-tenth of what it was before)
    • Went dark on GitHub, did not look into there at all
  • Present
    • People build valuable, multi-billion dollar businesses on top of Redis
    • Redis: You use Redis? Great
    • Users: We use Redis, we expect features and bug fixes, but we won’t pay
    • So how to balance what everybody wants? Arrow’s Impossibility Theorem
  • Redis internals are no longer simple, so many different concepts
  • Secret: We have enough work for 50 people. One-person-does-everything does not scale!
  • Pull requests for spelling mistakes are ignored
    • Secret: Limited time
  • A Plan
    • Extract : make reusable C libraries, server, distributed systems
    • Refactor
      • Make it easy for new people to contribute
    • Improve
      • Fix many many holes
    • Grow
      • Add JSON support
      • Add geocoding support
      • Allow reading config from network config storage (etcd, consul, etc.)
  • Again, Redis development does not scale! More community participation and contributions needed.

Redis internals

Closing Comments by Matt Barker

  • All talks have been recorded and will be put online in the coming weeks

(PostScript)

Thanks to @vedang for letting me know about this conference!

Final note - Totally amazed that there is so much being built on top of Redis.


Comments

@harikt says:

Nice post

@erictyliaw says:

Thanks for sharing your notes!

@RedisLabsInc says:

good summary until the videos are online, thank you!

@appbaseio says:

A good summary!