Ad Delivery Strategy: The Potential-Ads
2020｜Type: Strategy｜Tag: Design for Effect｜Role: Product, UX Design
Background
During E-commerce promotions, ad optimizers filter out no-cost ads or cost slowly ads by cost and increase bids by 5% to 20% to gain more impression. In this situation, advertisers expect to quickly identify these ads and bulk raise their bids to ensure that quality ads get more impressions.
The learning phase of new ads
The learning phase is known as a cold start. The new ad has no training data or the sparsity of training data, which leads to inaccurate prediction, which further leads to high-conversion-rate-audience not being found, or not being impressed correctly, as well as low-conversion-rate-audience being overestimated resulting in over-cost, etc. As a result, new ads do not get enough impressions, and ML does not have enough data for training, thus the learning phase is interrupted.
For this reason, the learning phase aims to help new ads get more impressions, get enough data for modeling, and explore the best way to deliver your ads to the audience, in order to earn advertisers' budgets and improve long-term revenue.
The conditions of the learning phase are as follows:
- Boundary duration: 24 hours after the launch. Particularly 1-day retention ads require 48 hours.
- Optimization prerequisites: the conversions bottom line is reached within the boundary duration, and shallow-funnel events bottom line is 6 conversions.
- Cost achievement: conversions are more than 6 and cost deviation is within 30%.
- Shutdown trigger: very low cost, fingerprint filter, aggregation shutdown, etc.
- Very low cost: zero-conversion within online time over X and cost reaches Y times target_CPA; non-zero-conversion within online time over X, cost deviation greater than Z. X stands for boundary duration. Under this condition, Y and Z stand for the benchmarks within 3, 6, 9, and 12 hours of online time under different optimization goals.
- Fingerprint filter: the first retrieval reaches 3 hours, with more than 10,000 retrievals in total, and more than 90% fingerprint filter in the last 3 hours.
- Aggregate shutdown: as same as the fingerprint filter.
- Others: fail at one-click conversion improvement; no impression or click after a long time such as online time reaches 12 hours with more than 1,000 retrievals but 0 clicks, etc.
Value Proposition
How do I recognize these ads?
Ads that successfully exit the learning phase but have low cost and low eCPM.
Definition of Potential-Ads
When ads have exited the learning phase, based on the real-time ad auction, for ads that are currently Insufficient competitive due to low bids, and ads that their cost is within the expected but still insufficient. The delivery system predicts the ad auction after 1 hour by learning it before 1 hour. If the ratio of impression lift rate to bid lift rate is predicted to be greater than 2, the delivery system would set the ad as a Potential-Ad for the next 1-hour period. More opportunities for impressions are gained by increasing bids.
Selection of Potential-Ads feature
The features of Potential-Ads include, in addition to the basic features of ads set, the features built by the impression per 10min trend of this hour and an hour ago, the impression lift rate of 5%, 10%, 15%, 20% predicted by the ranking modeling, and the binary classification prediction of whether the next hour of impression lift rate is more than 2 times of bid lift rate.
Design Iteration
The Potential-Ads is a typical bid strategy product, and strategy products need to be robust. For this reason, we build MVPs to quickly run through the process from 0 to 1, and then get the optimization with the estimated metrics set in the MVPs.
MVP (Version 1.0)
Present
The user experience and its estimated metrics of the Potential-Ads are built from 0 to 1, and it supports 10% and 20% of bid lift rate in the system recommendation after the R&D discussion.
Initialization
For ads with predicted conversions greater than 1, the system will find the ads with the highest ratio of the impression lift rate to bid lift rate between the 10% and 20% selections.
Design Objectives
- Help advertisers understand what the Potential-Ads are and quickly identify and select them.
- Help ad delivery understand the usages and effects of Potential-Ads.
Design Strategy
Because the Potential-Ad strategy is one of the bid strategies, after ads are created, the main touchpoint for bidding is in the ad set level of the campaign menu. It will be triggered and displayed within certain conditions and periods. In the MVP, we can do the following:
- Adds the badge of new-coming Potential-Ads in the bid strategy column and deep bid strategy column in the ad set level of the campaign menu.
- Adds a new feature tour on the page of the campaign menu.
Solution
Pages
The Touchpoints of the Potential-Ads
Metrics
The Metric System
The measurement of the Potential-Ads includes scale, effect, user experience, evaluation, and performance.
- Potential-Ads Volume: The ads that marked as Potential-Ads
- Potential-Ads Impression Increase Rate: ( Post-bidding-lifted Impressions / Pre-bidding-lifted Impressions ) - 1
- Potential-Ads Cost Increase: Post-bidding-lifted Cost - Pre-bidding-lifted Cost
- Potential-Ads Cost Increase Rate: (Post-bidding-lifted Cost / Pre-bidding-lifted Cost) - 1
- Potential-Ads Usage Volume: The quantity of the recommended-bids-used ads + The quantity of the custom-bids-used ads + The quantity of the API-bids-used ads.
- Potential-Ads Usage Rate: ( The quantity of the recommended-bids-used ads + The quantity of the custom-bids-used ads + The quantity of the API-bids-used ads ) / The quantity of Potential-Ads
- Potential-Ads Usage Volume in Ads Delivery Platform: The quantity of the recommended-bids-used ads + The quantity of the custom-bids-used ads
- Potential-Ads Usage Rate in Ads Delivery Platform: ( The quantity of the recommended-bids-used ads + The quantity of the custom-bids-used ads ) / The quantity of Potential-Ads
- Potential-Ads Adoption Volume in Ads Delivery Platform: The quantity of the recommended-bids-used ads + The quantity of the recommended-bids-used ads with the bids equal to recommended bids
- Potential-Ads Adoption Rate in Ads Delivery Platform: ( The quantity of the recommended-bids-used ads + The quantity of the recommended-bids-used ads with the bids equal to recommended bids ) / The quantity of Potential-Ads
- Potential-Ads Precision Rate: The quantity of Potential-Ads with more than 2 times lift of impression / The quantity of Potential-Ads
- Potential-Ads Recall Rate: The quantity of Potential-Ads with more than 2 times the lift of impression / The quantity of all Ads with more than 2 times the lift of impression
- API Call Success Rate
- API Latency
- SLA
Version 1.1
Problem
- Insufficient usage of the Potential-Ads, which is not perceived by advertisers.
10% and 20% of bid lift rates being too aggressive, leads to advertisers dare take the risk.
Zero or unchanged conversions after adopting the recommended bids.
Potential-Ads Setup
In version 1.0, the condition that ads with predicted conversions greater than 1 might not result in good conversions after adoption. In version 1.1, we tried to label potential ads by measuring cost achievement within a controllable budget and cost. Also, the10% and 20% of bid lift rates were expanded to four rates of 5%, 10%, 15%, and 20%.
The Definition of the Potential-Ads
Key Results
- Improve advertisers' perception of Potential-Ads.
- Expand to four rates of bids.
- Improve the accuracy rate of potential ads and increases the weight of potential ads for conversion estimation.
Design Objectives
Make every touchpoint works.
Design Strategy
- Add the Potential-Ads quantity to the overview on the dashboard.
- Add the bulk editing in the ad set level.
- Label the new Potential-Ads badge in both the ad set level and the ad level.
Solution
- When all the selected items are Potential-Ads, recommended would be checked; if not custom would be checked.
- When using recommended, the optimized Potential-Ads will be removed, leaving only the Potential-Ads to be optimized.
Modify Bid Strategy
Modify Deep Bid Strategy
Bulk Actions
Refine the touchpoint within the ad set level and ad level.
Modifying Bid Strategy in the Detail of Ad Set
Modifying Bid Strategy in the Detail of Ad
Version 1.2
Effects
Base on the education industry effects of the optimized CPM ads on November 16, 2021 to November 19, 2021, we can see that:
The Pilot Phase of the Potential-Ads
The current Potential-Ads precision rate is around 75%, which could be improved. Ads with no conversions for 8 hours after the bid change require experiments to optimize the features of modeling. In terms of the raised bid range, we can see that from the table below:
The Cases under the Different Raised Range of Bids
In the education industry, currently, optimizers prefer aggressive bid lifts to gain volume.
Potential-Ads Setup
Maintain version 1.1.
Problems
The Potential-Ads are optimized CPM Ads which means both bid strategy and deep bid strategy could be set in ads.
The Potential-Ads Usage Volume with both bid strategy and deep bid strategy is underperformed.
Through online user interviews, we’ve realized that advertisers didn’t turn on the deep bid strategy column in the ad set level, and 40% of education industry accounts didn’t have these two columns turned on.
Key Results
Improve the impression of this kind of strategy product.
Design Objectives
Guide users to turn on the bid strategy and deep bid strategy columns. Optimizes the rules of the columns’ arrangement.
Design Strategy
- Make the bid strategy column and deep bid strategy column in front of the queue in default.
Add the Potential-Ads badge in the dropdown menu of the customized column panel.
Solution
Front-end optimization only, can be modified directly.
Version 1.3
Effects
The average daily cost of Potential-Ads has exceeded 12 million, and the adoption rate of the Potential-Ads in Ad Delivery Platform is close to 70%.
Potential-Ads Setup
Maintain version 1.1.
Problem
The accuracy of the prediction model on non-WeChat-Ad traffic was affected by editing the deep bid strategy without editing the bid strategy simultaneously. WeChat-Ad traffic is optimized in phases for the shallow funnel events and deep funnel events, while non-WeChat-Ads traffic is optimized for both at the same time.
Key Results
- For ads with bid strategy but without deep bid strategy: Maintain the same.
- For ads with bid strategy and deep bid strategy except for 1-day retention and goals of ROI-related: bidirectional support for both synchronous and asynchronous editing, as well as editing one of them at once.
- For ads with bid strategy and deep bid strategy set for 1-day retention or goals of ROI-related: only bid editing is supported.
Design Objectives
Help advertisers to edit bids quickly and synchronously.
Design Strategy
- Add synchronized bid editing on bids, and deep bids on the badges of the Potential-Ads.
Add synchronized bid editing on bids, and deep bids in the ads editor.
Solution
The badges of the Potential-Ads within both bid strategy column and deep bid strategy column. So did the editor panels.
Modify Bid Strategy through the Badge
Version 1.4
Effects
The Cost on the Potential-Ads are currently close to 15 million per day on average.
Potential-Ads Setup
Maintain version 1.1.
Problem
Among the four kinds of raised bids range, the 5% has the highest adoption rate but is may not the best bid point recommended by the system.
Conflict between the Best Adoption Rate and the Best Bid Strategy
Key Results
Supported by scenarios-oriented for the Potential-Ads with different gradient bids.
Design Objectives
Clearly inform advertisers of the benefits of every bid, so that advertisers could know that the recommended bid is the current best bid.
Design Strategy
Clearly disclose the recommended bid and the best bid.
Solution
Disclose the best bid within the bid and the deep bid in the dialog by hovering over the badges of the Potential-Ads. So did the editor panels.
Disclosure of the Best Bid Strategy through the Badge
What I’ve learned
With the crossover of projects’ manpower, Potential-Ads released 5 versions in 2 months. We conducted a nitty-gritty trial, research, and optimization in the education industry. The Potential-Ads was designed to provide a reference for subsequent product delivery overall.
- Aimed direction. The positioning of Potential-Ads is to quickly identify the ads that could win more impressions with a little bid lift for advertisers. Subsequent versions have evolved in this direction.
- Propelled delivery. MVP and metrics are established in the first phase, providing insight and data to support the subsequent product delivery.
- Evidence-based focus. In the metrics system, we evaluate the Potential-Ads by scale, performance, user experience, model evaluation, and network service. From the very beginning to solve the usage issue of the touch point, solve the precision rate issue of the dual-goal advertising, to improve the user experience to enhance user trust. In each version of the decision is to do the right thing.
Some things didn't go so well. For instance, when it came to bulk actions, the R&D workload was underestimated, leading to incomplete implementation of bulk actions. Also, when the metrics system was initially built, the metrics for machine resources and performance weren't evaluated. Both issues were discovered retrospectively. The memory requirements for Potential-Ads aren't significant, but it's still necessary to implement memory monitoring for rigor. Additionally, new fields increased performance risks. In incremental updates, the system processed an average of 60,000 rows of data to execute the diff command, taking about 27 seconds; in full updates, the system processed about 90,000 rows of data, taking about 37 seconds; the full query of the index took 7 to 9 seconds; the duration of all traffics was about 45 seconds in total. All these findings offer new insights for constructing a metric system for future strategy product design.
© 2024 Xiang PENG. All Rights Reserved.