Ad Data Strategy: ROAS lift-based Strategy

Ad Data Strategy: ROAS lift-based Strategy

2021｜Type: Strategy｜Tag: Design for Effect｜Role: Product, UX Design

Industry Background

Finance and Marketplace industries fail to send their deep-funnel data back to the media due to advertisers' concerns about data security (or for other reasons). There are inconsistencies between optimization and assessment goals during ad delivery, and the existing machine learning models can only optimize based on the shallow-funnel data (such as activation, order, etc.) sent by advertisers, resulting in models that cannot optimize advertisers' deep goals well.

Industries such as real estate, home furnishing, auto, investment promotion, wedding photography, etc., have two obvious characteristics of the low frequency of high average transaction value and long decision chain due to their promotion commodity features, which led to unsatisfactory deep-funnel events in ad delivery.

What is the ROAS lift-based Ad?

The ROAS lift-based Ads are optimized for a specific ROAS goal. Increase ROAS by optimizing eCPM to bid for your highest value customers and improving pLTV accuracy. Advertisers need to choose a shallow-funnel event as optimization for Ad Delivery (e.g. activation, order, etc.) and set the target ROAS.

ROAS = Lifetime Value / Cost
ROAS Achieving Rate = Realistic ROAS / Target ROAS
LTV: Life Time Value is an estimate of the average revenue that a customer will generate throughout their lifespan as a customer. Only present the revenue per customer in this situation.
Cost: Known as ad cost, only presents the cost the advertiser is willing to spend to get the user for a specific conversion.

Shallow-funnel conversion data and deep-funnel conversion data of advertisers, such as purchase events, are required for ROAS lift-based Ads. Based on these data, the system would calculate a pLTV model for this regression problem to predict paid amount per user within the first day - a metric which means the total amount within the first day after activation as the conversion window, and likely, paid an amount per user within 3 days, 7 days, etc.

eCPM of ROAS lift-based Ad = 1000 * pCTR * pCVR * ( pLTV / Target ROAS )

In this formula, predicted LTV / Target ROAS is considered as a conversion-optimized bid, and predicted LTV is the predicted paid amount within different periods, as same as windows chosen in ad delivery, after the conversion event occurs. Besides, the window set should be consistent between predicted LTV and predicted CVR. Target ROAS is the target ROAS set for the ad.

For example, the ROAS of the paid amount per user within the first day is a ratio of 0.1. For a user with a predicted paid amount of 100 CNY, the advertiser’s willingness to spend for this conversion is 1,000 CNY (equal to 100 / 0.1) when the target ROAS of the ad is a ratio of 0.1.

User

Advertiser teams, mainly data analysts and ad optimizers, have inconsistencies between the optimization and assessment goals. The inconsistency is reflected in, for example, the assessment goals are order, purchase, LTV, 1-day retention, and so on, and the optimization goal is the cost of activation; or the assessment goal is the effectiveness of form, and the optimization goal is the cost of form.

Risk & Opportunity

Value Proposition

The ROAS lift-based strategy which uses audience differentiation provides data analysts and ad optimizers with three bidding strategies that work directly on the ad's ranking and reranking. Securely uploading advertisers’ own deep-funnel data while ensuring that costs under optimization goals are controlled allows advertisers to customize eCPM based on the deep-funnel data value of different audiences. Improve advertisers' effects and get more high-value traffic.

The essence of the three strategies is label-based, file-based, and ML-examples-based. Among them, machine-learning-examples-based relies on selecting data sets and their classification.

Label-based: Enhanced-by-Label-Mode
File-based: Enhanced-by-File-Mode
ML-examples-based: Enhanced-by-Machine-Learning-Mode

Design Analysis

The ROAS lift-based strategy leverages the XGBoost algorithm to enhance advertising performance. This technology powers not only the Ad Data team's Data Management Platform (DMP) but also the Ad Delivery Platform, as well as supporting the traffic and industry service teams. The traffic team, known as WeChat-Ad, operates through programmatic advertising, while the industry service teams utilize the Landing Page Configuration Platform. Both platforms provide advanced modeling capabilities directly to customers. Additionally, the Data Management Platform and Ad Delivery Platform feature user-friendly interfaces that enable advertisers to manage campaigns efficiently.

Data Management Platform: Provides data analysts with the capability to create and manage strategies. It currently serves all advertisers who have activated the professional version of the Data Management Platform, offering a full range of three strategies.
Ad Delivery Platform: Equips ad optimizers with the ability to create and manage ads. Currently, it only serves gaming advertisers with Enhanced-by-Label-Mode and Enhanced-by-File-Mode.

How Strategies Work on Online Systems

Below, we will analyze the workflow of the ROAS lift-based strategy in the Data Management Platform.

Creating a ROAS lift-based strategy requires meeting certain criteria:

Basic Information: clarify the type of strategies and fill in the name.
Configuration: how the three strategies are set up. Particularly in ML-examples-based mode, the scale limitations of examples must meet the machine learning requirements.
Application Range: the areas where the strategy takes effect, such as ad set and account, with the ad set taking precedence.

A ROAS lift-based strategy works successfully when it meets the following criteria:

Basic Information: must not be empty.
Configuration: must not be empty.
Application Range: must not be empty. Application ranges can also override each other within their own accounts, potentially resulting in an empty range instead of the previous one.
Machine Resources: the quota must not be fully occupied. Enhanced-by-Machine-Learning-Mode has specific restrictions to prevent resource wastage.
Dataflow Status: must not be restricted. The strategy will face several limitations from creation until it goes online.
Effects: must show positive outcomes over time.

To verify that the positive effect results from a ROAS lift-based strategy, we conduct standard traffic experiments to compare the ROAS-achieving rates of advertisers. We carry out a long-term experiment involving WeChat-Ad traffic and non-WeChat-Ad traffic. The sampling method is based on Unique Visitors (UV), with 50% of the traffic being effective and 50% being ineffective.

Control group: 50% of traffic (0–4999)
Experiment group: 50% of traffic (5000–9999)

An ad's current request parameter is 1, indicating it is an example of an experiment group. Consequently, all factors of the ROAS lift-based strategy are applied to ad ranking. Later, the data for each ROAS lift-based strategy are obtained by parsing page view logs.

Problems

Weak intention in basic information

Users have difficulty understanding which strategy to choose in what scenario and cannot tell the difference between these strategies.

Weak Intention in Basic Information

Complex actions in multi-tiers of configuration

Configuration currently includes four tiers, with cumbersome information architecture and complex flows. Concepts such as rules, enhancement, high-value files, and the point of enhancement increase the cognitive burden on users. This flow needs to be simpler and more intuitive.

Complex Actions in Multi-Tiers of Configuration

Unclarified selection in the application scope

In both account and ad set of application range sections, there are problems with counterintuitive selection, disabled selection without reasons, and also hard-to-locate options. After accounts and ad sets are applied to the previous strategies, the present strategies cannot be used, and both of them can only be deleted from the previous strategies. However, the strategy must contain the application range, the accounts, and ad sets that cannot be deleted in full. The actual working application range is more ambiguous than ever.

Unclarified Selection in the Application Range

The hardcore in Machine Learning

When machine learning is involved, additional examples of datasets and machine resources need to be met. For example, the quantity of quota for Enhanced-by-Machine-Learning-Mode is currently only 1,000, and it is bound to be difficult to meet the requirement after opening the ROAS lift-based strategy fully to the professional version.

The High Entry Barrier in Machine Learning

Restrictive status in dataflow

There is a waiting time before putting it online and there may be an “obstacle” within function return values failing. Strategies that are normally online cannot be edited and need to be taken offline first. The reason is that editing the examples in ML-examples-based Mode will cause the recalculation of modeling. To avoid the situation where the strategy is not available or the calculation fails, the restriction that normal online strategies cannot be edited has reached an agreement, and the restriction is taking charge of all three strategies. At the same time, the online but not positive significantly strategies within Enhanced-by-Machine-Learning-Mode are less than 30% utilized, which is also a waste of machine resources.

Restrictive Status in Dataflow

Difficulty in checking data of effect

The path of download is long, the entrance is scattered, and only experimental data is supported. There is a nearly 70% gap due to the difference between WeChat-Ad traffic and non-WeChat-Ad traffic in the parent-ad-ID and child-ad-ID, the definition of windows, the intermediate tables, and the different statistical objects.

Difficulty in Checking Data of Effect

High interference in experiment setup

The ad contrast mode is not scientific. On top of that, the ROAS lift-based strategy is working for the ad sets or accounts, and the test and control groups are very uneven in the view of the account. And the sampling happens in the request phase. This unevenness is not caused by sampling but by the uneven cost in ad auctions caused by sharing the same experimental tier. The difference between the AA groups is relatively large. Setting the ROAS lift-based strategy's bid factor to 1 is equivalent to the experimental group not taking effect, but simply grouping and the cost difference can even exceed 10%.

High Interference in Experiment Setup

Design Objectives

Intuitional information
Concise configuration
Clear application range
Predictable machine learning
Flexible status in dataflow
Unified strategy effects
Scientific experiment tiers

Solution

Intuitional information: Delete unnecessary sections such as goal-setting and directly disclose introductions of these strategies to help users determine which strategy to use.

Intuitional information

Concise configuration: Reduce tiers of information architecture and streamline the steps from four to two. And reduce the textual concepts.

Concise configuration

Clear application scope: Merge dimensions in the selection, add fields to make the selection more explainable, support overrides, and show range with order directly in the list.

Clear application range

Predictable machine learning: Disclosure of related information early helps clarify requirements for machine learning, and quotas are set for Enhanced-by-Machine-Learning-Mode to avoid wasting resources.

Predictable machine learning

Flexible status in dataflow: Add new states of action, improve states of configuration and separate editing of these strategies. To avoid wasting resources, strategies that are not positive significantly for 10 days will be automatically taken offline.

Flexible status in dataflow

Unified strategy effects: Merge strategy details, strategy effect, and download features into one page, add new logs and shorten the path of download.

Unified Strategy Effects

Scientific experiment tiers: Remove ad comparison experiments. Create experimental tiers for different strategies under the same account, which means each strategy is an experimental tier, and reassign randomization to the whole traffic in each experimental tier. This approach is more scientific, but machine resources need to be re-evaluated.

Scientific experiment tiers

Evaluation

Intuitional information: User interview shows that users who have already used it are more willing to create and try Enhanced-by-Machine-Learning-Mode.
Concise configuration: Tiers of information architecture and steps are cut in half, from 4 to 2 tiers. Duration of configuration is reduced by almost 60%.
Clear application scope: N/A
Predictable machine learning: Quotas are more evenly distributed.
Flexible status in dataflow: Enhanced-by-Machine-Learning-Mode increases the adoption rate from 30% to 70%.
Unified strategy effects: Less steps spent on navigating to the download feature are reduced by nearly 50%, and the gap between data is reduced to 10%.
Scientific experiment tiers: N/A

What's more

Neither the application range nor the experiment setup is complete due to short-staffed issues. As well as labels, alignment is currently not possible among Label-Square, Insights-of-Label, Enhanced-by-Label-Mode of Data Management Platform as well as internal marketing tools and Ad Delivery Platform. It is proposed to gradually standardize the entire process of the user profile from mining to application in data governance in the future.