Scan the QR code with WeChat

Add your dedicated consultant

Favorite

Publication date: 2026-09-28 17:10:58 Views: 1788

Recommendation

Kuaishou, together with Beihang University, proposes AdaptCP to solve the Checkpoint bottleneck of TB-scale recommendation models; the work was accepted at the top conference EuroSys 2027. Core content: 1. Existing technical challenges and research background of Checkpoints in industrial-scale recommendation models 2. Core design ideas and key technical solutions of the AdaptCP system 3. Experimental performance of AdaptCP in real industrial scenarios

杨芳贤

Founder of 53AI / Tencent Cloud (TVP) Most Valuable Professional

In industrial-scale recommendation model training, the Embedding Table often contains billions or even tens of billions of Keys, with parameter sizes reaching tens of TB. Such enormous model states make Checkpointing a major performance bottleneck in distributed training: full saves incur huge storage I/O overhead and force training to pause to ensure consistency; meanwhile, existing incremental or differential Checkpoint schemes, although they reduce the overhead of each save, respectively suffer from overly long recovery chains or continuously growing checkpoints. Therefore, how to balance training throughput, storage overhead, and failure recovery efficiency has become a key challenge in fault-tolerant training of industrial-scale recommendation models.

I. Research Background: The Checkpoint Dilemma of Industrial-Scale Recommendation Models

To reduce the overhead of full saves, existing methods mainly adopt incremental Checkpoints and differential Checkpoints. The incremental approach saves a smaller amount per save, but recovery requires loading multiple checkpoints in sequence; the differential approach recovers more directly, but the amount of saved data keeps growing as training continues.

II. Method Overview: The Three Core Modules of AdaptCP

To address the above problems, AdaptCP performs systematic optimization at three levels: Checkpoint strategy, training pipeline, and underlying I/O:

  • Hot/Cold-Aware Adaptive Hybrid Checkpoint Strategy
  • SoU save-on-update mechanism
  • Parallel I/O engine based on SSD buffering

The three respectively address: which parameters to save, when to save, and how to save faster.

2.1 Hot/Cold-Aware Adaptive Hybrid Checkpoint

The first core innovation of AdaptCP is to dynamically select a more suitable Checkpoint strategy for different parameters based on the access hotness of Embeddings.

For Hot Embeddings that are frequently accessed and updated, when incremental Checkpoints are used, the same batch of high-frequency parameters is scattered across multiple incremental files, and failure recovery requires loading them in sequence, which tends to incur high recovery overhead; therefore, differential Checkpoints are more suitable. Conversely, for Cold Embeddings with lower access and update frequency, when differential Checkpoints are used, parameters that have changed are repeatedly saved in subsequent checkpoints, causing extra write and storage overhead; therefore, incremental Checkpoints are more suitable.

Based on this observation, AdaptCP adopts differential Checkpoints for Hot Embeddings and incremental Checkpoints for Cold Embeddings, thereby balancing save efficiency and recovery speed.

The paper further establishes a unified I/O cost model and theoretically proves that, as Embedding access frequency changes, there exists a unique optimal Hot/Cold partition threshold in the system that minimizes the total I/O cost of Checkpoint saving and failure recovery, and this threshold can be quickly solved via binary search. This theoretical derivation provides a clear basis for dynamically selecting the hot/cold parameter boundary during online training.

During online training, the access distribution of Embeddings is not fixed. As user behavior keeps changing, the original hot/cold relationships may shift, and a fixed threshold can hardly remain optimal over the long term. Therefore, AdaptCP further designs an online adaptive hot/cold partitioning mechanism, accomplished by three components working together:

  • Tracker: records the Embeddings updated within the current period;
  • Access Counter: counts the access frequency of different Embeddings in real time;
  • Threshold Solver: solves for the optimal hot/cold threshold online based on the current access distribution.

The system then dynamically partitions Embeddings into a Hot Set and a Cold Set according to the threshold, applying differential Checkpoints and incremental Checkpoints respectively, so that the Checkpoint strategy can continuously adjust with changes in runtime access patterns.

To this end, AdaptCP proposes the SoU mechanism, whose design idea is similar to Copy-on-Write in operating systems. Since each Batch in a recommendation model actually updates only a small number of Embeddings, SoU does not need to synchronously save all parameters immediately when a Checkpoint is triggered; instead, when an Embedding is about to be updated, it first retains the old value from before the update and then proceeds with the parameter update. Unchanged parameters can continue to be shared between the training process and the Checkpoint, thereby decomposing one large-scale synchronous save into fine-grained asynchronous save tasks.

Building on SoU, AdaptCP further combines Embedding Prefetch to design Prefetch-triggered SoU. Once parameters are prefetched, the system starts high-priority save tasks in advance, thereby using the computation window between Prefetch and parameter update to complete background Checkpoint I/O. By the time the parameters actually enter the update phase, most save tasks are already completed, which further reduces the waiting time of training threads and hides Checkpoint latency as much as possible within the normal training process.

III. Experimental Results

To comprehensively validate the effectiveness of AdaptCP, the team conducted experiments on public datasets and in a real industrial training environment respectively.

3.1 Overall Performance

In terms of training throughput, compared with Check-N-Run, AdaptCP achieves a 4.59× improvement in the Criteo Terabyte scenario and further reaches 13.87× in the real production data scenario. As the model and Checkpoint scale continue to grow, AdaptCP's performance advantage becomes even more pronounced, indicating better scalability for industrial-scale large Checkpoint scenarios.

In terms of failure recovery, AdaptCP also significantly reduces Checkpoint loading time. On the production dataset, the recovery times of Check-N-Run, Incr CKPT-N, and native incremental Checkpoint are 9.41×, 12.70×, and 13.62× those of AdaptCP, respectively. This advantage mainly comes from the parallel recovery mechanism, which can simultaneously load multiple Checkpoint shards from remote storage and fully utilize read bandwidth.

Checkpoint frequency directly affects the system's fault tolerance: the more frequently checkpoints are saved, the less data needs to be rolled back and recomputed when a failure occurs, but it also brings higher I/O pressure. The experiments gradually shortened the Checkpoint interval from every 10,000 Batches to every 100 Batches to test the performance of different schemes in high-frequency saving scenarios.

When the Checkpoint interval is raised to once every 100 batches, training throughput drops by 92.6% for Check-N-Run and 74.9% for native incremental Checkpoint, while AdaptCP drops by only 19.0% and maintains the highest throughput across all Checkpoint frequencies. This is because SoU overlaps the saving process with computation, while the parallel Writer Engine completes most data persistence within a limited time window, significantly reducing the impact of high-frequency Checkpoints on training.

IV. Summary and Outlook

The value of AdaptCP lies in combining the parameter access characteristics of recommendation models themselves, the training process, and the underlying storage system to redesign the Checkpoint mechanism in large-scale recommendation model training. Rather than treating Checkpointing as a periodic save task independent of training, AdaptCP dynamically selects save strategies based on parameter hotness, and through Save-on-Update and parallel I/O, integrates the saving process as much as possible into the normal training pipeline, providing a system solution for TB-scale recommendation models that balances training throughput and fault recovery efficiency.

For rapidly evolving large recommendation models, Checkpoint systems will continue to face new challenges. As model sizes keep growing, beyond massive Embedding parameters, the Dense parts are also scaling, and model states, optimizer states, and training cluster sizes will all grow further, placing higher demands on training fault tolerance and system stability. Going forward, the team will continue exploring efficient and stable training of large recommendation models, further reducing Checkpoint interference with the training process and improving fault recovery efficiency under larger parameter scales, more complex training architectures, and higher fault tolerance requirements, providing more reliable infrastructure support for the continued scaling of large recommendation models.

V. Team Introduction

The AI Platform team of Kuaishou's Algorithm Engine Department has long focused on building infrastructure for large-scale model training, supporting efficient training of core business models such as search, advertising, and recommendation, as well as in-house multimodal large models, while continuously optimizing the scheduling capabilities of large-scale training and inference resource pools. The team has supported the launch and full rollout of the OneRec end-to-end generative recommendation large model and the KeyeVL multimodal large model, and has driven these models to achieve notable business gains.

The team works across multiple technical directions. In recommendation training, it pioneered SKAI, an industry-leading software-hardware integrated supercomputing framework for large-scale generative recommendation models, raising MFU above 30%. On the unified training-inference resource scheduling platform, the team continuously improves resource utilization through industry-leading techniques such as mixed deployment, tidal scheduling, and unified scheduling. Over the past three years, the team has published multiple papers at top international academic conferences such as EuroSys, SoCC, and ASE, continuously advancing the research and industrial deployment of large-scale training systems.

53AI, the preferred service provider for enterprise adoption of large models

Products: scenario-based implementation consulting + large model application platform + industry solutions

Commitment: free POC validation; cooperation begins only after results meet the target. Zero-risk adoption of large models, delivered to 160+ medium and large enterprises

Loading...

Scan to consult

Scan to log in

By logging in, you agree to the "53AI Website Service Agreement"

Service Agreement

Welcome to the [53AI official website] (hereinafter referred to as "this website" or "we"). This "Membership Service Agreement" (hereinafter referred to as "this Agreement") is a legal agreement between you (hereinafter referred to as "member" or "user") and [Shenzhen Bosixiechuang Network Technology Co., Ltd.] regarding registration, login, and use of this website's membership services.

Before registering or logging in, please read and fully understand the terms carefully, especially the liability exemption or limitation clauses, intellectual property clauses, and dispute resolution clauses. Such clauses will be highlighted in bold for your attention. When you successfully log in to this website via WeChat official account authorization, SMS verification code, or other means, you are deemed to have fully understood and agreed to accept all contents of this Agreement.

I. Definitions

This website: refers to the website and related mobile pages operated by [Shenzhen Bosixiechuang Network Technology Co., Ltd.] with the domain name [53ai.com].

Membership services: refers to the knowledge base article access, content search, and other related value-added services provided by this website to registered members.

Knowledge base content: refers to the digital content resources published by this website, including but not limited to text, charts, data, research reports, and industry analyses.

II. Account Registration and Login

Login methods: This website supports the following login methods, which you may choose based on your actual situation:

WeChat official account authorized login: You agree to authorize your WeChat OpenID information to this website for creating or linking a member account.

SMS verification code login: You must provide a real and valid mobile phone number and complete identity verification and login/registration via an SMS verification code.

Account security: Your account is for your personal use only; gifting, lending, renting, transferring, or selling it is prohibited. Losses such as account theft or password leakage caused by your improper safekeeping shall be borne by you.

Real-name verification: As required by relevant laws and regulations, we may require you to complete real-name verification for certain features. If you refuse to provide it, you may be unable to use some or all services.

Protection of minors: If you are under 18, please read this Agreement accompanied by your legal guardian and use this service only with your guardian's consent.

III. Service Content and Rules

Knowledge base access rights: After logging in, members have the right to browse and search relevant articles and content in this website's knowledge base online within the scope of permissions corresponding to their membership level.

Service changes: We have the right to adjust, change, or terminate part of the service content as needed for business development, and will notify you in advance via website announcements, official account messages, and other means.

Prohibited conduct: When using the services, you must not engage in the following:

Using technical means to batch-crawl, download, or re-save knowledge base content;

Using knowledge base content for commercial purposes or disseminating it to third parties without authorization;

Interfering with the normal operation of this website or infringing the legitimate rights and interests of other users;

Publishing illegal or non-compliant information or engaging in activities contrary to public order and good morals.

IV. Intellectual Property Statement

Ownership of rights: The intellectual property rights of the layout design, software code, and other content in this website's knowledge base belong to [full company name] or the original rights holders, and are protected by laws such as the Copyright Law of the People's Republic of China.

Limited license: This website grants members a non-exclusive, non-transferable, non-sublicensable ordinary license, limited to online access to knowledge base content for personal study and research purposes only.

Infringement liability: Without written permission, no organization or individual may copy, reprint, excerpt, mirror, compile, or otherwise use the above content in any form. Once discovered, we reserve the right to pursue legal liability.

V. Personal Information Protection

We value the protection of your personal information. For how we collect, use, store, and protect your personal information, please read the "Privacy Policy" separately.

The information you provide through WeChat official account authorization or mobile phone verification will be handled strictly in accordance with the Personal Information Protection Law, and used only for necessary purposes such as identity verification, service provision, and security verification.

You may at any time exercise your rights to access, correct, and delete personal information and withdraw authorization consent through website settings or by contacting customer service.

VI. Disclaimer

Content accuracy: Knowledge base content is for reference only and does not constitute professional advice. We make no express or implied warranties regarding its completeness, accuracy, or timeliness; you should make your own judgment and bear the risks of use.

Force majeure: We are not liable for breach of contract for service interruptions or delays caused by force majeure such as natural disasters, changes in policies and regulations, network failures, or third-party platform interface anomalies (such as WeChat interface maintenance or carrier SMS channel failures).

Third-party links: This website may contain links to third-party websites, whose content and services are not under our control. Please assess the risks yourself.

VII. Liability for Breach

If you breach this Agreement, we may, depending on the severity, take measures such as warnings, feature restrictions, suspension of service, or account cancellation, and reserve the right to claim compensation for losses.

If your breach causes us to suffer administrative penalties, third-party claims, or damage to our goodwill, you shall bear full liability for compensation (including but not limited to fines, damages, attorney fees, notarization fees, etc.).

VIII. Governing Law and Dispute Resolution

The formation, performance, and interpretation of this Agreement shall be governed by the laws of Mainland China.

Any dispute arising from or in connection with this Agreement shall first be resolved through friendly negotiation between the parties; if negotiation fails, either party may file a lawsuit with a people's court of competent jurisdiction in [the company's location].

IX. Miscellaneous

This Agreement constitutes the entire agreement between the parties regarding this service and supersedes any prior oral or written agreements.

If any provision of this Agreement is found to be invalid or unenforceable, it shall not affect the validity of the other provisions.

We have the final right to interpret this Agreement and reserve the right to modify it at any time within the scope permitted by law. Any modified Agreement takes effect upon publication, and continued use of the service shall be deemed acceptance of the revised content.