How to calculate relative volume (RVOL) efficiently in Postgres?
asked 5 hours ago by @qa-f2wxwaio3jc4tbt58bfj 0 rep · 36 views
postgresql sql window functions
Given the following schema:
Column | Type | Collation | Nullable | Default
-----------+------------------+-----------+----------+----------------------------------
id | bigint | | not null | generated by default as identity
timestamp | integer | | not null |
open | double precision | | not null |
high | double precision | | not null |
low | double precision | | not null |
close | double precision | | not null |
volume | double precision | | not null |
symbol_id | bigint | | not null |
I need to calculate Relative Volume (RVOL) which shows how an asset's current trading activity compares to its historical average over the same time of day.
I'm calculating the cumulative volume per day and dividing it by its average at the same time of previous periods following the calculation method here. Here's what I do:
select symbol_id, cumulative_volume / avg_cumulative_volume relative_volume
from (select symbol_id,
cumulative_volume,
avg(cumulative_volume)
over (partition by symbol_id,
extract(hour from db_timestamp) * 60 +
extract(minute from db_timestamp)
order by timestamp) avg_cumulative_volume
from (select timestamp,
symbol_id,
to_timestamp(timestamp) db_timestamp,
sum(volume)
over w_day cumulative_volume
from core_price
window w_day as (
partition by symbol_id, date_trunc('day', to_timestamp(timestamp))
order by timestamp
))_)_
For a small table, fine but it's not scaling well for a 16M table. It's taking 18s to execute on my M4 max MBP:
Subquery Scan on _ (cost=2395.48..5282658.93 rows=16672172 width=16) (actual time=2.147..17879.152 rows=16672172.00 loops=1)
Buffers: shared hit=1683 read=205294
-> WindowAgg (cost=2395.48..5074256.78 rows=16672172 width=60) (actual time=2.146..17274.526 rows=16672172.00 loops=1)
" Window: w1 AS (PARTITION BY __1.symbol_id, (((EXTRACT(hour FROM __1.db_timestamp) * '60'::numeric) + EXTRACT(minute FROM __1.db_timestamp))) ORDER BY __1.""timestamp"")"
Storage: Memory Maximum Storage: 17kB
Buffers: shared hit=1683 read=205294
-> Incremental Sort (cost=2395.19..4574091.62 rows=16672172 width=52) (actual time=2.141..13954.790 rows=16672172.00 loops=1)
" Sort Key: __1.symbol_id, (((EXTRACT(hour FROM __1.db_timestamp) * '60'::numeric) + EXTRACT(minute FROM __1.db_timestamp))), __1.""timestamp"""
Presorted Key: __1.symbol_id
Full-sort Groups: 3774 Sort Method: quicksort Average Memory: 28kB Peak Memory: 28kB
Pre-sorted Groups: 3943 Sort Method: quicksort Average Memory: 1548kB Peak Memory: 2107kB
Buffers: shared hit=1683 read=205294
-> Subquery Scan on __1 (cost=811.44..3318407.00 rows=16672172 width=52) (actual time=0.572..9137.362 rows=16672172.00 loops=1)
Buffers: shared hit=1683 read=205294
-> WindowAgg (cost=811.44..2984963.56 rows=16672172 width=36) (actual time=0.569..6006.333 rows=16672172.00 loops=1)
" Window: w_day AS (PARTITION BY core_price.symbol_id, (date_trunc('day'::text, to_timestamp((core_price.""timestamp"")::double precision))) ORDER BY core_price.""timestamp"")"
Storage: Memory Maximum Storage: 17kB
Buffers: shared hit=1683 read=205294
-> Incremental Sort (cost=811.27..2443117.97 rows=16672172 width=28) (actual time=0.562..3135.859 rows=16672172.00 loops=1)
" Sort Key: core_price.symbol_id, (date_trunc('day'::text, to_timestamp((core_price.""timestamp"")::double precision))), core_price.""timestamp"""
Presorted Key: core_price.symbol_id
Full-sort Groups: 3774 Sort Method: quicksort Average Memory: 28kB Peak Memory: 28kB
Pre-sorted Groups: 3943 Sort Method: quicksort Average Memory: 1548kB Peak Memory: 2107kB
Buffers: shared hit=1683 read=205294
-> Index Scan using core_price_symbol_id_4f851c6c on core_price (cost=0.43..1187433.34 rows=16672172 width=28) (actual time=0.043..2107.166 rows=16672172.00 loops=1)
Index Searches: 1
Buffers: shared hit=1683 read=205294
Planning Time: 0.115 ms
Execution Time: 18167.869 ms
How can I improve the performance, to be suitable for a reasonable response time? Should I use a materialized view for such purposes given that this data will be updated frequently (every 1 minute)?