Background: I read about this ad project at Pinterest.
Goal: The goal is to deliver the relevant ads to the Pinners by sending ads to “Lookalike audiences (AAL)”. Their idea is to send an ads to potential new users.
Problem: This is a user recommendation problem.
Previous Method:
There are two current methods:
- Regression-based
- Similarity-based
The regression-based method trains a linear classifier on a list of users (seed-list). Each user is represented as a sparse vector. This approach works well on large seed lists but performs poorly when the seed-list is too small.
The similarity-based method uses a user-embedding and then perform a K-NN search. The number of nearest users or the distance from the centroid must be specified. The works well on smaller seed-list because there are less noises but it does not work well on a large seed-list due to too many similar users.
Solution
They combine the aforementioned methods so that the combined model will be robust on small and large seed lists. The cost of this approach is the cost of inferencing as well as the cost of serving neural network models for each advertisers.
- The user embeddings are trained by a different model [2]
- a positive sampler is a pair of ads and a user from the seed list.
- a negative sampler is a pair of ads and randomly sampled users from the given target country.
- Train a classifier using a binary cross-entropy loss.
They applies weighting on each user in the seed-list. The idea is a user with higher engagement should have a higher weight. This approach seems to be helpful to increase the revenue. I think this weighting scheme will force the model to focus more on super-user, which can be good and bad because the model will be less bias to the new users.
Results
- The offline evaluation using precision and recall looks promising.
- The A/B testing reports the gains on users with ads impressions and revenue.
- They also reported CTR on the ads which drops significantly. They mentioned that this happens due to the quality of engagement has improved.
- The combined model has simplified the infrastructure. Instead of deploying two separated models, deploying one model is much simpler.
Reference:
- https://medium.com/pinterest-engineering/the-machine-learning-behind-delivering-relevant-ads-8987fc5ba1c0
- deWet, Stephanie & Ou, Jiafan. (2019). Finding Users Who Act Alike: Transfer Learning for Expanding Advertiser Audiences
[Sage]Blog 