Motivation
"dj_onlinefactor"
Tecblog from Netflix(03/27/2013)
Gap: offline/online processing
Mobile
Challenge
Data
Algorithm/Model
Software Architecture
Recommendation at Netflix Scale
Storm: Distributed and FT Realtime Computation
BackType/Twitter
Scalability
Fault-tolerance
Higher level abstraction than message passing
Companies & Projects Using Storm
Plato
Redis/hiredis(a c++ wrapper)
Beantalkd/beanstalkpp(internal version)
Paracel(communication to parameter servers)
Libraries & tools @douban
Offline
Process data
Async computation
Matrix factorization
- Collecter
- Batch processing(bias-sgd)
- Publish item factors
- Notify nearline backends after update
Nearline
Process events
Async computation
Regression
- Solve user-factors/status-factors
- Load offline result
- Realtime processing
- Reinit
Online
Process requests
Sync computation
Balltree
- Build
- Search with dot-product(user/status * item factor)
- Serve recommendations
Draft...
Python Interface
# Make socket
socket = TSocket.TSocket('beater7', 9090)
# Buffering is critical. Raw sockets are very slow
transport = TTransport.TFramedTransport(socket)
# Wrap in a protocol
protocol = TBinaryProtocol.TBinaryProtocol(transport)
# Create a client to use the protocol encoder
client = RorService.Client(protocol)
# Connect
transport.open()
rep = Request()
rep.rating = '2619286,9087,3.45'
rep.ktop = 10 rep.method = 'tree'
rep.whitelist = ['16685', '13984', '13925']
rep.blacklist = ['10294', '3223', '1']
# get result
req = client.search(rep)
print req.error, req.results
Demo
TODO
Bugfix
Unified interface for testing
Similarity/clustering/realtime recommendation framework
Reliability: fault-tolerance
Dj @FM, Recommender @Dongxi
Thank You!
contact info