AI-Assisted Market Sensing: How Tanqory Reads Demand Signals Automatically
Research blog — deep dive on architecture, methods, and empirical grounding
Executive Summary
Most companies still treat market sensing as episodic research: quarterly surveys, lagging sales reports, and ad hoc agency studies. By the time insight arrives, opportunity windows have narrowed and competitors have moved. Tanqory operates differently. Market sensing is a continuous, AI-assisted system that ingests heterogeneous signals, infers intent and risk, and connects insights to action engines across merchandising, pricing, localization, and supply chain. This post details Tanqory's Market Sensing Fabric (MSF): an event-driven, causal, and governance-aware pipeline that automates how we read demand signals globally.
Key ideas:
- Event-first data foundation spans behavioral telemetry, macro indicators, search and social trends, catalog signals, logistics constraints, and payment risk.
- Causal inference and synthetic controls distinguish signal from noise and avoid mistaking correlation for demand.
- Multi-region adaptation handles data sparsity, regulatory variance, and cultural nuance.
- Decision policies connect sensing to action: assortment bets, localization, pricing experiments, and promise shaping.
- Governance, fairness, and privacy are first-class, ensuring trust with merchants, customers, and regulators.
1. Why Market Sensing Must Be Continuous
1.1 Limits of Traditional Research Cadence
Traditional market research cycles rely on periodic surveys, panels, or consultant reports. Literature shows that static views underperform in volatile environments (Davenport and Harris, 2017). Static reports miss:
- Rapid shifts in consumer intent during promotions or shocks.
- Cross-channel feedback loops (search → social → marketplace).
- Latent demand that surfaces only in granular behavioral traces.
- Supply and payment frictions that suppress observable demand.
1.2 Consequences of Lag
- Mis-timed expansion: entering markets after early adopters already picked alternatives.
- Overstock and markdowns: inventory bets made on stale signals.
- Pricing misalignment: elasticity misread because of sample bias or seasonality.
- Poor localization: content and payments remain misfit, reducing trust and conversion.
1.3 Tanqory's Approach
Market sensing is an always-on capability that runs in the background of every region and category. It combines streaming data, causal models, and action hooks so that sensing flows into decisions within hours, not quarters.
2. Design Principles
- Event-first: meaningful state changes become events with contracts; batch is a supplement.
- Causal over correlational: prefer designs that isolate treatment effects (Athey and Imbens, 2016; Pearl, 2009).
- Multi-horizon: nowcasting for immediate actions; forecasting for capacity and assortment.
- Multi-resolution: signals at SKU, category, corridor, region, and macro levels.
- Policy-aware: decisions respect fairness, privacy, cost, and latency budgets (Kohavi et al., 2020).
- Human-in-the-loop: analysts validate high-impact insights; explanations accompany recommendations (Lundberg and Lee, 2017).
- Governed and auditable: lineage from raw signals to decisions; exportable for regulators and partners.
3. Architecture Overview: Market Sensing Fabric (MSF)
3.1 Layered View
- Ingest and Contracts: streaming pipelines for behavioral, operational, and macro signals; schema and consent enforcement.
- Canonical Signal Lake: conformed entities (user, session, SKU, merchant, corridor, payment instrument) with slowly changing dimensions.
- Feature and Signal Store: online/offline parity; freshness SLAs; quality monitors.
- Causal and Predictive Services: uplift models, synthetic controls, forecasting, anomaly detection.
- Decision Layer: bandits and planners that drive experiments in pricing, assortment, and localization.
- Action Connectors: hooks into merchandising tools, pricing engines, CMS, and supply chain orchestration.
- Observability and Governance: tracing, fairness, privacy, latency, and cost dashboards; audit trails.
3.2 Signal Types
- Behavioral: search queries, clickstreams, scroll depth, cart edits, wishlists, returns, and cancellations.
- Commerce state: stock levels, substitutions, promise accuracy, delivery fees, and shipping times.
- Payments: authorization rates, retries, decline codes, fraud scores.
- Macro and external: exchange rates, inflation, holidays, local events, weather, mobility trends.
- Social and trend data: search interest, social buzz for brands/categories (aligned with privacy policies).
- Support and trust: ticket topics, CSAT, refund reasons, dispute rates.
3.3 Data Quality and Contracts
- Schema registry enforces types, enums, and null thresholds.
- Freshness targets per signal: critical behavioral signals <15 minutes; macro indicators daily; social trend summaries hourly.
- Deduplication and watermarking to handle late events (Kleppmann, 2017).
- Consent/residency flags baked into events; regional routing enforced.
4. Causal and Predictive Modeling
4.1 Causal Inference
- Uplift modeling: doubly robust learners estimate treatment effects of promos or placements across cohorts (Athey and Imbens, 2016).
- Synthetic controls: estimate counterfactual demand for regions without experiments (Abadie et al., 2010).
- Instrumental variables: when natural experiments exist (e.g., carrier outages), isolate causal impact on conversion and cancellations.
- Do-calculus reasoning: causal graphs to identify confounders; avoid spurious correlations (Pearl, 2009).
4.2 Forecasting and Nowcasting
- Time-series: SARIMA and Prophet-style models for medium-horizon demand (Box et al., 2015).
- Sequence and transformer models: for high-frequency behavioral streams with latency caps (Vaswani et al., 2017; Chen and Guestrin, 2016 for tree-based fast paths).
- Hierarchical models: borrow strength across regions/categories when data is sparse.
- Anomaly detection: seasonal hybrid ESD and autoencoders for sudden shifts.
4.3 Explainability and Trust
- SHAP values for tree-based models (Lundberg and Lee, 2017).
- Counterfactual prompts: what if price drops 5%? what if delivery improves by one day?
- Operator dashboards display feature influence shifts; alerts on unexplained jumps.
5. Multi-Region Adaptation
- Localization of signals: language-specific tokenization; locale-aware search features.
- Heterogeneous data availability: some regions have sparse payment data; others have rich social signals. Hierarchical priors and transfer learning mitigate sparsity.
- Regulatory variance: privacy and residency constraints differ; MSF enforces per-region processing.
- Cultural nuance: holiday calendars, sentiment lexicons, and price sensitivity differ; region-specific features capture this.
- Latency budgets: mobile-first regions run lightweight models; desktop-heavy regions can serve richer policies.
6. From Sensing to Action
