Data Sources & Pipeline
Probed live: 2026-06-24 17:48:57 UTC · Direct Sheets API (no local cache)
Every tab on the master Google Sheet is written by a Python daemon running
continuously on a private VPS. Daemons self-tune their request rate via AIMD (TCP-style
congestion control) and share a global rate limiter so they never collectively exceed any
upstream API ceiling. The terminal pages you see read from a local DuckDB cache (mirrored
from the master sheet every 5 minutes); zero Sheets API quota is consumed per pageview.
Writer heartbeats
| Writer | Kind | State | Last log | Req/min | AIMD sleep | Slowdowns | Target tab | Rows | Coverage |
|---|---|---|---|---|---|---|---|---|---|
yahoo_quotedfv-yahoo-quote |
AIMD daemon | ● ACTIVE | 53s | — | 0.30s | 318 | Yahoo_Quote |
13,096 | 98.8% |
yahoo_quarterlydfv-yahoo-quarterly |
AIMD daemon | ● ACTIVE | 10989s | — | 0.30s | 0 | Yahoo_Quarterly |
49,262 | 88.2% |
yahoo_extrasdfv-yahoo-extras |
AIMD daemon | ● ACTIVE | 1157s | — | 2.00s | 73 | Yahoo_Extras |
10,516 | 99.6% |
yahoo_cashflowdfv-yahoo-cashflow |
AIMD daemon | ● ACTIVE | 11334s | — | 0.30s | 0 | Yahoo_Cashflow |
10,504 | 99.6% |
pricesdfv-prices |
timer 12h |
● SCHEDULED
next in 251m |
69848s | — | — | — | Prices_Daily |
10,095 | 89.5% |
openinsiderdfv-openinsider |
timer 6h |
● SCHEDULED
next in 131m |
60523s | — | — | — | Insider_Daily |
100 | event subset (100 rows) |
socialdfv-social |
periodic 1h |
● SCHEDULED
next in 11m |
2930s | — | — | — | Social_Trends |
500 | event subset (500 rows) |
freddfv-fred |
timer 12h |
● SCHEDULED
next in 251m |
71245s | — | — | — | FRED_Macro |
16 | time series (16 rows) |
secdfv-sec.timer |
timer weekly Sun | ● ACTIVE | 2656135s | — | — | — | Fundamentals_Quarterly |
35,135 | 34.2% |
hyperliquiddfv-hyperliquid |
continuous 5-min cycle | ● ACTIVE | — | — | — | — | Crypto_Quote |
421 | event subset (421 rows) |
financedatabasedfv-financedatabase |
timer weekly Mon |
● SCHEDULED
next in 6251m |
229702s | — | — | — | ETF_Universe |
2,801 | — |
equity_backtestdfv-equity-backtest |
timer daily after-close |
● SCHEDULED
next in 311m |
66648s | — | — | — | Equity_Backtest |
12,312 | 100.0% |
crypto_backtestdfv-crypto-backtest |
timer daily 23:30 UTC |
● SCHEDULED
next in 341m |
65450s | — | — | — | Crypto_Backtest |
450 | — |
● active <60s ago ·
● 60–600s ·
● >600s (investigate) ·
AIMD slowdowns = total throttle events since daemon started; non-zero just means the
controller has detected an upstream block at least once (the system handles it automatically).
Available data tabs (22 tabs on the master sheet)
Universe
Fundamentals_Annual
Fundamentals_Quarterly
Fundamentals_TTM
Insider_Daily
Treasury_DailyRates
FRED_Macro
Prices_Daily
Yahoo_Quote
Social_Trends
Yahoo_Extras
Yahoo_Cashflow
Yahoo_Quarterly
Yahoo_Annual
Crypto_Universe
Crypto_Quote
Crypto_Candles_Daily
ETF_Universe
Identifiers
Equity_Backtest
Crypto_Backtest
Earnings_Calendar
Source pedigree
| Source | Provider | Auth | Cost | Reliability |
|---|---|---|---|---|
SEC EDGAR XBRL Financial Statement Data Sets |
U.S. Securities and Exchange Commission | None (User-Agent header required) | Free | Authoritative |
secfsdstools |
Apache-2.0 OSS, parses SEC quarterly bundles | — | Free | Production-stable |
FRED API |
Federal Reserve Bank of St. Louis | Free API key | Free | Authoritative |
openinsider.com scrape |
Aggregator of SEC Form 4 filings | None | Free | One step removed from SEC |
yfinance |
Community wrapper around Yahoo Finance | None (curl_cffi chrome120 impersonation) | Free | Rate-limited; daemons self-tune with AIMD. Treasury yield curve also pulled here as FRED DGS1MO..DGS30 series. |
ApeWisdom API |
Public social-mention aggregator (WSB / Stocktwits / r/investing) | None | Free | Aggregates already-aggregated feeds |
CoinGecko |
Crypto market data | None (free tier) | Free | Reliable |
Hyperliquid /info API |
Hyperliquid L1 (main DEX + HIP-3 builder DEXs: xyz, vntl, flx, hyna) | None (public) | Free | Authoritative for on-chain perp metadata, marks, funding, OI, candles; covers crypto + xyz: US equities/commodities/indices/FX + vntl: pre-IPO and sector baskets |
JerBouma/FinanceDatabase |
Community-curated symbol metadata (MIT, Yahoo-derived). 300k+ symbols across equities, ETFs, funds, indices, FX, crypto. | None (raw GitHub CSV) | Free | Metadata-only — no prices/fundamentals. Used for ETF_Universe (36k ETFs) and Identifiers (ISIN/CUSIP/FIGI). Auto-refreshed weekly upstream for US exchanges. |
How data flows
SEC EDGAR
yfinance
FRED API
OpenInsider
ApeWisdom
Hyperliquid
FinanceDatabase
▼▼▼▼▼▼▼
sec_writer
yahoo_quote
yahoo_quarterly
yahoo_cashflow
yahoo_extras
yahoo_quarterly
yahoo_cashflow
yahoo_extras
prices_writer
fred_writer
openinsider_writer
social_writer
hyperliquid_writer
financedatabase_writer
equity_backtest_writer
crypto_backtest_writer
▼▼▼▼▼▼▼▼▼▼
Master Google Sheet
▼
cache_refresher · every 5 min
cache_refresher · every 5 min
serving.duckdb
▼
Flask reads · zero Sheets API quota per pageview
Flask reads · zero Sheets API quota per pageview
Flask (port 5077)
▼
nginx · HTTPS
nginx · HTTPS
dfvterminal.com
Design principles
- Data flows, doesn't refresh. No fixed schedules. Daemons run continuously and compute work from upstream event signals (e.g. new SEC filings → re-fetch). Idle when everything's fresh; busy when there's new data.
- Self-tuning rate. AIMD (additive-increase, multiplicative-decrease) finds each upstream API's tolerated rate without anyone needing to know what it is.
- Cross-daemon coordination. Writers hitting the same upstream (Yahoo) share a file-locked token bucket so they take turns, not race.
- Graceful failure. Daemons handle SIGTERM (flush + persist before exit);
systemd
Restart=alwaysbrings them back; converged AIMD state survives crashes. - Decoupled serving. Site reads from local DuckDB; if a writer is broken, the cache keeps serving the last good data until it's fixed.