Offline Causal Reinforcement Learning for Dynamic Marketing Uplift Optimization with Uplift-Decomposed Q-Learning
DOI:
https://doi.org/10.66525/ijictt144Keywords:
uplift modeling, offline reinforcement learning, causal forest, heterogeneous treatment effects, customer relationship management, pessimismAbstract
Marketing teams own large logs of past campaigns but can rarely experiment with new contact policies online. Uplift models estimate the incremental effect of a single contact, yet repeated contacts interact through fatigue, carryover and unsubscription, so one-shot uplift targeting is myopic. We study offline causal reinforcement learning for this setting and propose Uplift-Decomposed Q-learning (UDQ). Under sequential ignorability, the advantage of contacting a customer equals the immediate causal uplift plus a discounted long-term uplift; UDQ estimates the first term with an honest causal forest, learns only the second by temporal-difference updates with shrinkage, and acts on a lower confidence bound built from the forest's standard errors. On the real Hillstrom randomized e-mail experiment (64,000 customers), one-shot targeting policies earn no more than blanket mailing, which motivates the dynamic view. On a 12-period benchmark calibrated on the same experiment, UDQ reaches a net value of 0.180 per customer under uniform logging, 13% above myopic causal-forest targeting, 26% above conservative Q-learning, 75% above a reimplemented causal-forest-plus-DQN framework trained in a learned simulator, and 90% of an oracle agent. Under rule-based logging it ties the best baselines while sending 27% fewer e-mails. Ablations show that the gains come from anchoring the Q-function on the causal-forest uplift and from uncertainty-based pessimism, whereas the learned long-term term must be strongly shrunk and adds little at this data scale.
Downloads
References
N. J. Radcliffe and P. D. Surry, “Real-world uplift modelling with significance-based uplift trees,” Stochastic Solutions, Edinburgh, U.K., White Paper TR-2011-1, 2011.
P. Gutierrez and J.-Y. Gérardy, “Causal inference and uplift modelling: A review of the literature,” in Proc. Int. Conf. Predictive Applications and APIs, ser. Proc. Mach. Learn. Res., vol. 67, 2017, pp. 1–13.
F. Devriendt, D. Moldovan, and W. Verbeke, “A literature survey and experimental evaluation of the state-of-the-art in uplift modeling: A stepping stone toward the development of prescriptive analytics,” Big Data, vol. 6, no. 1, pp. 13–41, 2018, doi: 10.1089/big.2017.0104.
S. Wager and S. Athey, “Estimation and inference of heterogeneous treatment effects using random forests,” J. Amer. Statist. Assoc., vol. 113, no. 523, pp. 1228–1242, 2018, doi: 10.1080/01621459.2017.1319839.
S. Athey, J. Tibshirani, and S. Wager, “Generalized random forests,” Ann. Statist., vol. 47, no. 2, pp. 1148–1178, 2019, doi: 10.1214/18-AOS1709.
S. R. Künzel, J. S. Sekhon, P. J. Bickel, and B. Yu, “Metalearners for estimating heterogeneous treatment effects using machine learning,” Proc. Natl. Acad. Sci. USA, vol. 116, no. 10, pp. 4156–4165, 2019, doi: 10.1073/pnas.1804597116.
J. Wang, J. Liu, Y. Ge, and B. Jiang, “Risk-aware ensemble learning for credit default prediction under economic uncertainty,” World Journal of Economics and Business Research, vol. 1, no. 1, pp. 18–32, 2023.
E. Pednault, N. Abe, and B. Zadrozny, “Sequential cost-sensitive decision making with reinforcement learning,” in Proc. 8th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2002, pp. 259–268, doi: 10.1145/775047.775086.
G. Theocharous, P. S. Thomas, and M. Ghavamzadeh, “Personalized ad recommendation systems for life-time value optimization with guarantees,” in Proc. 24th Int. Joint Conf. Artif. Intell. (IJCAI), 2015, pp. 1806–1812.
J. Wang, Y. Tan, B. Jiang, B. Wu, and W. Liu, “Dynamic marketing uplift modeling: A symmetry-preserving framework integrating causal forests with deep reinforcement learning for personalized intervention strategies,” Symmetry, vol. 17, no. 4, Art. no. 610, 2025, doi: 10.3390/sym17040610.
T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma, “MOPO: Model-based offline policy optimization,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 14129–14142.
S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv:2005.01643, 2020.
A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative Q-learning for offline reinforcement learning,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1179–1191.
S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in Proc. 36th Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res., vol. 97, 2019, pp. 2052–2062.
Y. Jin, Z. Yang, and Z. Wang, “Is pessimism provably efficient for offline RL?” in Proc. 38th Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res., vol. 139, 2021, pp. 5084–5096.
F. Zhou, “Moment-constrained side-force allocation for wind-assisted merchant ships: Hydrodynamic performance of retractable anti-leeway fins and flap rudders,” Innovation and Technology Studies, vol. 3, no. 1, pp. 27–41, 2026.
Y. Zhao, X. Fang, and D. Simchi-Levi, “Uplift modeling with multiple treatments and general response types,” in Proc. SIAM Int. Conf. Data Mining (SDM), 2017, pp. 588–596, doi: 10.1137/1.9781611974973.66.
X. Nie and S. Wager, “Quasi-oracle estimation of heterogeneous treatment effects,” Biometrika, vol. 108, no. 2, pp. 299–319, 2021, doi: 10.1093/biomet/asaa076.
V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins, “Double/debiased machine learning for treatment and structural parameters,” Econometrics J., vol. 21, no. 1, pp. C1–C68, 2018, doi: 10.1111/ectj.12097.
B. Chakraborty and S. A. Murphy, “Dynamic treatment regimes,” Annu. Rev. Statist. Appl., vol. 1, pp. 447–464, 2014, doi: 10.1146/annurev-statistics-022513-115553.
X. Zhang, K. Wang, Z. Wang, B. Du, S. Zhao, R. Wu, X. Shen, T. Lv, and C. Fan, “Temporal uplift modeling for online marketing,” in Proc. 30th ACM SIGKDD Conf. Knowl. Discovery Data Mining, 2024, pp. 6247–6256, doi: 10.1145/3637528.3671560.
J. Chen, X. Han, X. Zhang, and Y. Yang, “Exact concept-level explanations for early detection of financial misstatements,” Eurasia Journal of Science and Technology, vol. 5, no. 2, pp. 34–48, 2023.
D. Ernst, P. Geurts, and L. Wehenkel, “Tree-based batch mode reinforcement learning,” J. Mach. Learn. Res., vol. 6, pp. 503–556, 2005.
S. Fujimoto, E. Conti, M. Ghavamzadeh, and J. Pineau, “Benchmarking batch deep reinforcement learning algorithms,” arXiv:1910.01708, 2019.
Z. Deng, J. Jiang, G. Long, and C. Zhang, “Causal reinforcement learning: A survey,” arXiv:2307.01452, 2023.
K. Battocchi, E. Dillon, M. Hei, G. Lewis, P. Oka, M. Oprescu, and V. Syrgkanis, “EconML: A Python package for ML-based heterogeneous treatment effects estimation,” 2019. [Online]. Available: https://github.com/py-why/EconML
H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proc. 30th AAAI Conf. Artif. Intell., 2016, pp. 2094–2100, doi: 10.1609/aaai.v30i1.10295.
K. Hillstrom, “The MineThatData e-mail analytics and data mining challenge,” MineThatData blog, Mar. 2008. [Online]. Available: https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html
D. Teng, “TRIAD: Deadline- and reliability-aware scheduling for post-earthquake structural assessment workflows,” World Journal of Engineering Research, vol. 3, no. 6, pp. 55–66, 2025.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2016.
G. Wang, P. Li, E. Feng, A. Shen, and M. Rhee, “A Dual-Process Memory-Augmented Large Language Model Agent for Explainable Sequential Recommendation,” IEEE Access, 2026, doi: 10.1109/ACCESS.2026.3741894.
P. Li, M. Bian, and Y. Lin, “Carbon-Aware Offline Reinforcement Learning for Joint Inventory Replenishment and Transportation Scheduling in Multi-Echelon Supply Chains,” Highlights in Business, Economics and Management, vol. 69, pp. 251–262, 2026.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Gabriel Hayes, Grace Tan (Author)

This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the International Journal of Information and Communication Technology Trends (IJICTT) are distributed under a Creative Commons license. This allows readers to freely access, share, and adapt the content for non-commercial purposes, provided proper attribution is given to the original authors. Authors retain copyright of their work.
