info@chtd.ru

+7 (495) 118-29-36

UserDB – User Metadata Storage Service (for LG SmartBox cloud data storage)

Customer: LG Electronics


Kernel development for the LG Electronics SmartBox cloud data storage (analog of the iCloud remote data storage, synchronization and processing system).

Features

— File system functions, including various types of access (per-user access, per-device access – devices can access the system). Virtual folders are created in accordance with file types. Files of the same type are arranged by date, by subject etc.

— Recognition of metadata such as duration, dimensions etc. Image rotation, creation of thumbnails for image and video files.

— Transcoding. Recognition of formats for new files uploaded to the system and transcoding of the files into formats suitable for viewing on a specific user’s device. Viewing format is chosen in accordance with device type.

— Multipart uploading. The interface (based on the Amazon service) for the system’s users enables multipart uploading. UserDB signs (authenticates) requests of the client programs which can later access the corresponding Amazon services directly.

— Video streaming – playing online video by using any device.

— PVR – integrated video (TV program) recording service. Recorded video can be found in the home directory and played back immediately.

— Promo folders. Default files and folders are automatically copied to the newly created user’s home directory.

— Ability to use promo codes and invites.

Special Features
  • Two-tier storage
    • User data (files, content) are stored in the Amazon S3 cloud
    • File metadata and user settings are stored in a PostgreSQL distributed database which can be located both inside and outside of the cloud.

  • High loads
  • Massive uploading and downloading operations are conducted directly to and from the Amazon cloud storage. The system only signs the requests with a digital signature, and this enables high load operations.


  • Interaction of multiple services.
  • The services are implemented as different processes, interacting through Amazon SQS queue.


  • Scalable processing services.

  • Automatic scaling of the whole system. When the load grows, new servers are automatically connected to the system, and when the load decreases, the number of the servers is decreased.
Architecture

The system consists of three subsystems:

  1. Interface
  2. Data storage
  3. Asynchronous processes

The interface consists of multiple identical applications to which users’ requests are redirected by the load balancer (e.g. nginx) in a cyclic or random order.

Data storage is a sharding database. Distribution is done by using a single-byte key (256 values). Each database instance supports a certain subset of keys (for example, in case of 2 databases, two ranges are used: 0-127 and 128-256). The keys are calculated by using a user’s login name.

Asynchronous processes support additional operations that are not related to immediate response for requests. These operations include transcoding, metadata identification etc.

Each interface instance is connected to all data storage instances. The interface interacts with outside services such as Admin, Billing, Notification-gate, and with the Amazon/S3/SQS external data storage. Asynchronous processes support the data storage and the Amazon/SQS external queues.


+7 (495) 118-29-36

info@chtd.ru

г. Москва, Сущевский вал, 5с3