Task ID: task_e_682fd4139be48323a97511b1cb54ca87
Configuring language runtimes...
+ rm -- /tmp/JAbv31-setup_script.sh
+ source ./codex_setup.sh
++ uv sync --frozen
Using CPython 3.12.3 interpreter at: /usr/bin/python3.12
Creating virtual environment at: .venv
Building scubaduck @ file:///workspace/scubaduck
Downloading playwright (43.1MiB)
Downloading pyright (5.3MiB)
Downloading duckdb (19.3MiB)
Downloading ruff (11.0MiB)
Downloading ruff
Built scubaduck @ file:///workspace/scubaduck
Downloading duckdb
Downloading playwright
Downloading pyright
Prepared 33 packages in 1.65s
Installed 33 packages in 79ms
+ blinker==1.9.0
+ certifi==2025.4.26
+ charset-normalizer==3.4.2
+ click==8.2.0
+ duckdb==1.2.2
+ execnet==2.1.1
+ flask==3.1.1
+ greenlet==3.2.2
+ idna==3.10
+ iniconfig==2.1.0
+ itsdangerous==2.2.0
+ jinja2==3.1.6
+ markupsafe==3.0.2
+ nodeenv==1.9.1
+ packaging==25.0
+ playwright==1.52.0
+ pluggy==1.6.0
+ pyee==13.0.0
+ pyright==1.1.400
+ pytest==8.3.5
+ pytest-base-url==2.1.0
+ pytest-playwright==0.7.0
+ pytest-xdist==3.6.1
+ python-dateutil==2.9.0.post0
+ python-slugify==8.0.4
+ requests==2.32.3
+ ruff==0.11.10
+ scubaduck==0.1.0 (from file:///workspace/scubaduck)
+ six==1.17.0
+ text-unidecode==1.3
+ typing-extensions==4.13.2
+ urllib3==2.4.0
+ werkzeug==3.1.3
++ source .venv/bin/activate
+++ '[' -z '' ']'
+++ '[' -n x ']'
+++ SCRIPT_PATH=.venv/bin/activate
+++ '[' .venv/bin/activate = /tmp/JAbv31-setup_script.sh ']'
+++ deactivate nondestructive
+++ unset -f pydoc
+++ '[' -z '' ']'
+++ '[' -z '' ']'
+++ hash -r
+++ '[' -z '' ']'
+++ unset VIRTUAL_ENV
+++ unset VIRTUAL_ENV_PROMPT
+++ '[' '!' nondestructive = nondestructive ']'
+++ VIRTUAL_ENV=/workspace/scubaduck/.venv
+++ '[' linux-gnu = cygwin ']'
+++ '[' linux-gnu = msys ']'
+++ export VIRTUAL_ENV
+++ '[' -z '' ']'
+++ unset SCRIPT_PATH
+++ _OLD_VIRTUAL_PATH=/root/.cargo/bin:/root/.rbenv/shims:/root/.rbenv/bin:/root/.rbenv/shims:/root/.local/share/swiftly/bin:/root/.bun/bin:/root/.nvm/versions/node/v22.16.0/bin:/root/.pyenv/shims:3441PYENV_ROOT/shims:/root/.pyenv/bin:/usr/local/go/bin:/root/go/bin:/root/.rbenv/bin:/root/.rbenv/shims:/root/.bun/bin:/root/.local/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+++ PATH=/workspace/scubaduck/.venv/bin:/root/.cargo/bin:/root/.rbenv/shims:/root/.rbenv/bin:/root/.rbenv/shims:/root/.local/share/swiftly/bin:/root/.bun/bin:/root/.nvm/versions/node/v22.16.0/bin:/root/.pyenv/shims:3441PYENV_ROOT/shims:/root/.pyenv/bin:/usr/local/go/bin:/root/go/bin:/root/.rbenv/bin:/root/.rbenv/shims:/root/.bun/bin:/root/.local/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+++ export PATH
+++ '[' xscubaduck '!=' x ']'
+++ VIRTUAL_ENV_PROMPT=scubaduck
+++ export VIRTUAL_ENV_PROMPT
+++ '[' -z '' ']'
+++ '[' -z '' ']'
+++ _OLD_VIRTUAL_PS1=
+++ PS1='(scubaduck) '
+++ export PS1
+++ alias pydoc
+++ true
+++ hash -r
++ python -c 'import os; import duckdb; con = duckdb.connect(); con.execute(f"SET http_proxy = '\''{os.getenv("HTTP_PROXY")}'\''"); con.execute("INSTALL '\''sqlite'\'';")'
++ playwright install chromium
Downloading Chromium 136.0.7103.25 (playwright build v1169) from https://cdn.playwright.dev/dbazure/download/playwright/builds/chromium/1169/chromium-linux.zip
167.7 MiB [] 0% 0.0s167.7 MiB [] 0% 31.7s167.7 MiB [] 0% 19.5s167.7 MiB [] 0% 14.4s167.7 MiB [] 0% 9.9s167.7 MiB [] 1% 6.4s167.7 MiB [] 2% 4.6s167.7 MiB [] 3% 3.2s167.7 MiB [] 4% 2.6s167.7 MiB [] 5% 2.4s167.7 MiB [] 6% 2.3s167.7 MiB [] 8% 2.0s167.7 MiB [] 10% 1.8s167.7 MiB [] 11% 1.6s167.7 MiB [] 13% 1.5s167.7 MiB [] 14% 1.4s167.7 MiB [] 15% 1.4s167.7 MiB [] 17% 1.3s167.7 MiB [] 20% 1.2s167.7 MiB [] 21% 1.1s167.7 MiB [] 24% 1.0s167.7 MiB [] 26% 1.0s167.7 MiB [] 29% 0.9s167.7 MiB [] 31% 0.8s167.7 MiB [] 33% 0.8s167.7 MiB [] 34% 0.8s167.7 MiB [] 36% 0.8s167.7 MiB [] 38% 0.7s167.7 MiB [] 40% 0.7s167.7 MiB [] 41% 0.7s167.7 MiB [] 43% 0.7s167.7 MiB [] 45% 0.6s167.7 MiB [] 47% 0.6s167.7 MiB [] 49% 0.6s167.7 MiB [] 50% 0.6s167.7 MiB [] 52% 0.5s167.7 MiB [] 54% 0.5s167.7 MiB [] 55% 0.5s167.7 MiB [] 57% 0.5s167.7 MiB [] 59% 0.5s167.7 MiB [] 61% 0.4s167.7 MiB [] 63% 0.4s167.7 MiB [] 64% 0.4s167.7 MiB [] 67% 0.4s167.7 MiB [] 70% 0.3s167.7 MiB [] 71% 0.3s167.7 MiB [] 73% 0.3s167.7 MiB [] 75% 0.3s167.7 MiB [] 77% 0.2s167.7 MiB [] 79% 0.2s167.7 MiB [] 80% 0.2s167.7 MiB [] 82% 0.2s167.7 MiB [] 85% 0.2s167.7 MiB [] 87% 0.1s167.7 MiB [] 89% 0.1s167.7 MiB [] 91% 0.1s167.7 MiB [] 92% 0.1s167.7 MiB [] 95% 0.0s167.7 MiB [] 97% 0.0s167.7 MiB [] 99% 0.0s167.7 MiB [] 100% 0.0s
Chromium 136.0.7103.25 (playwright build v1169) downloaded to /root/.cache/ms-playwright/chromium-1169
Downloading FFMPEG playwright build v1011 from https://cdn.playwright.dev/dbazure/download/playwright/builds/ffmpeg/1011/ffmpeg-linux.zip
2.3 MiB [] 0% 0.0s2.3 MiB [] 5% 0.3s2.3 MiB [] 17% 0.2s2.3 MiB [] 53% 0.0s2.3 MiB [] 100% 0.0s
FFMPEG playwright build v1011 downloaded to /root/.cache/ms-playwright/ffmpeg-1011
Downloading Chromium Headless Shell 136.0.7103.25 (playwright build v1169) from https://cdn.playwright.dev/dbazure/download/playwright/builds/chromium/1169/chromium-headless-shell-linux.zip
101.4 MiB [] 0% 0.0s101.4 MiB [] 0% 20.0s101.4 MiB [] 0% 13.6s101.4 MiB [] 0% 8.8s101.4 MiB [] 1% 5.5s101.4 MiB [] 2% 3.1s101.4 MiB [] 4% 2.3s101.4 MiB [] 6% 1.7s101.4 MiB [] 9% 1.3s101.4 MiB [] 11% 1.2s101.4 MiB [] 13% 1.1s101.4 MiB [] 16% 1.0s101.4 MiB [] 19% 0.8s101.4 MiB [] 22% 0.7s101.4 MiB [] 26% 0.6s101.4 MiB [] 30% 0.6s101.4 MiB [] 35% 0.5s101.4 MiB [] 38% 0.5s101.4 MiB [] 41% 0.4s101.4 MiB [] 43% 0.4s101.4 MiB [] 47% 0.4s101.4 MiB [] 50% 0.3s101.4 MiB [] 54% 0.3s101.4 MiB [] 57% 0.3s101.4 MiB [] 61% 0.2s101.4 MiB [] 62% 0.2s101.4 MiB [] 65% 0.2s101.4 MiB [] 69% 0.2s101.4 MiB [] 73% 0.2s101.4 MiB [] 76% 0.1s101.4 MiB [] 79% 0.1s101.4 MiB [] 83% 0.1s101.4 MiB [] 87% 0.1s101.4 MiB [] 90% 0.1s101.4 MiB [] 93% 0.0s101.4 MiB [] 96% 0.0s101.4 MiB [] 99% 0.0s101.4 MiB [] 100% 0.0s
Chromium Headless Shell 136.0.7103.25 (playwright build v1169) downloaded to /root/.cache/ms-playwright/chromium_headless_shell-1169
++ echo 'source .venv/bin/activate'I'll check the root directory for an AGENTS.md file.
bash: .venv/bin/activate: No such file or directory
root@df946f6027ab:/workspace# ls
scubaduckroot@df946f6027ab:/workspace# cd scubaduck
root@df946f6027ab:/workspace/scubaduck# ls
AGENTS.md codex_setup.sh scubaduck tests
README.md pyproject.toml stubs uv.lockroot@df946f6027ab:/workspace/scubaduck# cat AGENTS.md
- Use "source .venv/bin/activate" to activate the virtual environment before
running any commands
- ONLY use non-interactive commands like cat, sed, apply_patch to do edits.
Do NOT use interactive editors.
- Do NOT attempt to install packages. Only the packages specified in
pyproject.toml are available. You cannot add new packages. If you
desperately want another package, make a note of it in the final PR
description.
- Use conventional commits to format PR title
- There are no nested AGENTS.md files, this is the only agents file
- When using Playwright, ONLY use chromium browser. This browser is already
installed.
- Use "ruff check" to check lint, "ruff format" to autoformat files and
"pyright" to typecheck.
- When writing the PR description, include the original prompt supplied by the
user.
- When you add functionality to the server, add server tests to
tests/test_server.py. When you add features to the frontend, add frontend
tests to tests/test_web.pyThe AGENTS file has clear instructions. I'll examine codex_setup.sh next for setup details.
root@df946f6027ab:/workspace/scubaduck# cat codex_setup.sh
uv sync --frozen
source .venv/bin/activate
python -c "import os; import duckdb; con = duckdb.connect(); con.execute(f\"SET
http_proxy = '{os.getenv(\"HTTP_PROXY\")}'\"); con.execute(\"INSTALL 'sqlite';\"
)"
playwright install chromium
echo "source .venv/bin/activate" >> ~/.bashrcI'll check the README for any specific instructions regarding the environment.
root@df946f6027ab:/workspace/scubaduck# cat README.md | head -n 20
I really like Scuba (Meta's internal real-time database system). The distributed
, real-time database part of Scuba is quite difficult (and expensive) to replica
te, but I also really like Scuba's UI for doing queries, and I have found myself
wishing that I have access to it even for "small" databases, e.g., I have a sql
ite dataset I want to explore.
Pivotal ideas:
* Time series by default. In the dedicated "time series" view, there are many fe
atures specifically oriented towards working towards tables that represent event
s that occurred over time: the start, end, compare, aggregate and granularity fi
elds all specially privilege the timestamp field. In fact, you can't log events
to Scuba's backing data store without a timestamp, they always come with one. (S
cuba also supports other views that don't presuppose a time series, but the time
series is the most beloved and well used view.) This is in contrast to typical
software which tries to generalize to arbitrary data first, with time series bei
ng added on later.
* It's all about exploration. Scuba is predicated on the idea that you don't kno
w what you're looking for, that you are going to spend time tweaking queries and
changing filters/grouping as part of an investigation to figure out why a syste
m behaves the way it is. So the filters/comparisons/groupings you want to edit a
re always visible on the left sidebar, with the expectation that you're going to
tweak the query to look at something else. Similarly, all the parameters of you
r query get saved into your URL, so your browser history can double up as a quer
y history / you can easily share a query with someone else. This is contrast to
typical software which is often oriented to making pretty dashboards and reports
. (This function is important too, but it's not what I want in exploration mode!
)
* You can fix data problems in the query editor. It's pretty common to have mess
ed up and ended up with a database that doesn't have exactly the columns you nee
d, or some columns that are corrupted in some way. Scuba has pretty robust suppo
rt for defining custom columns with arbitrary SQL functions, grouping over them
as if they were native functions, and doing so with minimal runtime cost (Scuba
aims to turn around your query in milliseconds!) Having to go and run a huge dat
a pipeline to fix your data is a big impediment to exploration; quick and easy c
ustom columns means you can patch over problems when you're investigating and fi
x them for real later.
We're going to build a exploratory data analysis tool like Scuba for time series
database (i.e., a database with a mandatory timestamp representing the time an
event occurred). We'll use DuckDB as the underlying SQL engine served from a Py
thon server, and render the GUI/results as a webpage with vanilla HTML and JS. W
e'll use choices.js to support token inputs. We define a token input to mean a
text input element where as you type a dropdown displays with valid values, and
if you select one or press enter, the selection turns into a token/chip that can
only be deleted as one unit.
To start, we are going to support one views: samples. The samples view only all
ows you to view individual samples from the database, subject to a filter. Our m
ain UI concept is that there is a left sidebar that is the query editor, and the
right side that shows the view. The sidebar is always visible and defaults to
the query parameters of the current view. After you make changes to the query,
clicking the "Dive" button updates the view. The URL of the page encodes all of
the values of the query (and gets updated when you Dive), so the browser's back
button lets you view previous queries.
The query editor's job is to generate a SQL query, which then is applied on the
database, and then the result visualized according to the view.
Here are the settings you can apply to the query. The help text should show up w
hen you mouse over the field name:
* Start/End - Help text: "Sets the start/end of the time range to query. Can be
any kind of datetime string. For example: 'April 23, 2014' or 'yesterday'." The
UI for this selector supports both relative selections (now, -1 hour, -3 hours,
-12 hours, -1 day, -3 days, -1 week, -1 fortnight, -30 days, -90 days) as well a
s specifying an absolute date. The way this field is rendered is there is a fre
e form text box, a drop down arrow (for the relative selectors), and then a cale
ndar button (for date selection).
* Order By - Help text: "Choose a column to sort results by." There is an ASC/D
ESC toggle next to it.I’m going to check what tests are already in place.
root@df946f6027ab:/workspace/scubaduck# ls tests
conftest.py test_server_time.py test_web_graph.py
test_multi_table_web.py test_server_timeseries.py test_web_table.py
test_server.py test_web.py test_web_timeseries.py
test_server_basic.py test_web_basics.py web_utils.py
test_server_db_types.py test_web_chips.py
test_server_errors.py test_web_columns.pyI'll check the tests and see how the style is set up.
root@df946f6027ab:/workspace/scubaduck# sed -n '1,40p' tests/test_server.py
"""Legacy test module.
Tests have been split into multiple files:
- test_server_basic.py
- test_server_db_types.py
- test_server_time.py
- test_server_errors.py
- test_server_timeseries.py
"""I’m taking a look at tests/test_server_time.py now.
root@df946f6027ab:/workspace/scubaduck# sed -n '1,160p' tests/test_server_time.p
y
from __future__ import annotations
import json
from pathlib import Path
import pytest
from scubaduck import server
def test_integer_time_column(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text("created,event\n1704067200,login\n1704070800,logout\n")
app = server.create_app(csv_file)
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-01 01:00:00",
"order_by": "created",
"columns": ["created", "event"],
"time_column": "created",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 2
def test_integer_time_unit_ms(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text("created,event\n1704067200000,login\n1704070800000,logou
t\n")
app = server.create_app(csv_file)
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-01 01:00:00",
"order_by": "created",
"columns": ["created", "event"],
"time_column": "created",
"time_unit": "ms",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 2
def test_timeseries_default_xaxis_uses_time_column(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text("created,event\n1704067200000,login\n1704070800000,logou
t\n")
app = server.create_app(csv_file)
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-01 01:00:00",
"graph_type": "timeseries",
"granularity": "1 hour",
"columns": ["event"],
"aggregate": "Count",
"time_column": "created",
"time_unit": "ms",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 2
def test_integer_time_unit_us_default_start_end(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text(
"created,event\n1704067200000000,login\n1704070800000000,logout\n"
)
app = server.create_app(csv_file)
client = app.test_client()
payload = {
"table": "events",
"order_by": "created",
"columns": ["created", "event"],
"time_column": "created",
"time_unit": "us",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["start"] == "2024-01-01 00:00:00"
assert data["end"] == "2024-01-01 01:00:00"
assert len(data["rows"]) == 2
def test_sqlite_integer_time_unit_us(tmp_path: Path) -> None:
sqlite_file = tmp_path / "events.sqlite"
import sqlite3
conn = sqlite3.connect(sqlite_file)
conn.execute("CREATE TABLE visits (visit_time INTEGER, event TEXT)")
big_ts = 13384551652000000
conn.execute("INSERT INTO visits VALUES (?, ?)", (big_ts, "foo"))
conn.commit()
conn.close() # pyright: ignore[reportUnknownMemberType, reportAttributeAcce
ssIssue]
app = server.create_app(sqlite_file)
client = app.test_client()
payload = {
"table": "visits",
"start": "2394-02-20 00:00:00",
"end": "2394-02-21 00:00:00",
"order_by": "visit_time",
"columns": ["visit_time", "event"],
"time_column": "visit_time",
"time_unit": "us",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 1
def test_relative_time_query(monkeypatch: pytest.MonkeyPatch) -> None:
app = server.app
client = app.test_client()
from datetime import datetime
fixed_now = datetime(2024, 1, 2, 4, 0, 0)
class FixedDateTime(datetime):
@classmethod
def now(cls, tz=None): # type: ignore[override]
return fixed_now if tz is None else fixed_now.astimezone(tz)
monkeypatch.setattr(server, "datetime", FixedDateTime)
payload = {
"table": "events",
"start": "-1 hour",
"end": "now",
"order_by": "timestamp",
"limit": 100,
"columns": ["timestamp", "event", "value", "user"],
"filters": [],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()root@df946f6027ab:/workspace/scubaduck# sed -n '160,320p' tests/test_server_time
.py
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 1
assert data["rows"][0][3] == "charlie"
def test_default_start_end_returned() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"order_by": "timestamp",
"limit": 5,
"columns": ["timestamp"],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["start"] == "2024-01-01 00:00:00"
assert data["end"] == "2024-01-02 03:00:00"
def test_time_column_none_no_time_filter() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"columns": ["timestamp", "event"],
"time_column": "",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert "start" not in data and "end" not in data
assert len(data["rows"]) == 4I'll take a look at tests/test_server_timeseries.py to see what's inside.
root@df946f6027ab:/workspace/scubaduck# sed -n '1,160p' tests/test_server_timese
ries.py
from __future__ import annotations
import json
from typing import Any, cast
from scubaduck import server
def test_group_by_table() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "table",
"order_by": "user",
"limit": 10,
"columns": ["value"],
"group_by": ["user"],
"aggregate": "Sum",
"show_hits": True,
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
rows = rv.get_json()["rows"]
assert rows[0][0] == "alice"
assert rows[0][1] == 2
assert rows[0][2] == 40
def test_table_avg_with_timestamp() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "table",
"order_by": "user",
"limit": 100,
"columns": ["user", "timestamp", "value"],
"group_by": ["user"],
"aggregate": "Avg",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert "error" not in data
rows = data["rows"]
assert rows[0][0] == "alice"
from dateutil import parser
ts = parser.parse(rows[0][1]).replace(tzinfo=None)
assert ts == parser.parse("2024-01-01 12:00:00")
def test_timeseries_basic() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "timeseries",
"limit": 100,
"group_by": ["user"],
"aggregate": "Count",
"columns": ["value"],
"x_axis": "timestamp",
"granularity": "1 day",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 4
def test_timeseries_orders_by_xaxis() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "timeseries",
"limit": 100,
"columns": ["value"],
"x_axis": "timestamp",
"granularity": "1 day",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
rows = data["rows"]
from dateutil import parser
timestamps = [parser.parse(r[0]).replace(tzinfo=None) for r in rows]
assert timestamps == sorted(timestamps)
def test_timeseries_count_no_columns() -> None:
app = server.app
client = app.test_client()
payload: dict[str, Any] = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "timeseries",
"granularity": "1 day",
"columns": [],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
rows = data["rows"]
assert len(rows) == 2
assert rows[0][1] == 2
assert rows[1][1] == 2
def test_timeseries_limit_applies_to_series() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "timeseries",
"limit": 1,
"order_by": "user",
"group_by": ["user"],
"aggregate": "Count",
"columns": ["value"],
"x_axis": "timestamp",
"granularity": "1 day",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 2
assert all(r[1] == "alice" for r in data["rows"])
def test_timeseries_auto_and_fine_buckets() -> None:
app = server.app
client = app.test_client()
def run(gran: str) -> None:root@df946f6027ab:/workspace/scubaduck# sed -n '160,320p' tests/test_server_time
series.py
def run(gran: str) -> None:
payload = {
"start": "2024-01-01 00:00:00",
"end": "2024-01-02 03:00:00",
"graph_type": "timeseries",
"columns": ["value"],
"x_axis": "timestamp",
"granularity": gran,
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/js
on"
)
data = rv.get_json()
assert rv.status_code == 200
from dateutil import parser
start = parser.parse(cast(str, payload["start"])).replace(tzinfo=None)
buckets = [
parser.parse(cast(str, r[0])).replace(tzinfo=None) for r in data["ro
ws"]
]
assert buckets[0] == start
if len(buckets) > 1:
step = (buckets[1] - buckets[0]).total_seconds()
assert step % data["bucket_size"] == 0
assert any(r[1] != 0 for r in data["rows"])
run("Auto")
run("Fine")
def test_timeseries_string_column_error() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "timeseries",
"limit": 7,
"columns": ["timestamp", "event", "value", "user"],
"x_axis": "timestamp",
"granularity": "1 hour",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
assert rv.status_code == 200
def test_derived_column_basic() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"limit": 10,
"columns": ["timestamp"],
"derived_columns": {"val2": "value * 2"},
"filters": [],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["rows"][0][1] == 20
def test_timeseries_derived_column() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "timeseries",
"granularity": "1 hour",
"limit": 7,
"columns": ["value"],
"derived_columns": {"derived_1": "value * 2"},
"aggregate": "Avg",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
rows = data["rows"]
assert all(r[2] == r[1] * 2 for r in rows)
def test_reserved_word_column() -> None:
app = server.create_app("TEST")
client = app.test_client()
payload = {
"table": "extra",
"columns": ["ts", "desc"],
"order_by": "ts",
"time_column": "",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 2
assert data["rows"][0][1] == "x"I'll add tests to test_server_time.py or a new file to handle these bugs.
root@df946f6027ab:/workspace/scubaduck# ls scubaduck
__init__.py sample.csv server.py staticroot@df946f6027ab:/workspace/scubaduck# sed -n '1,160p' scubaduck/server.py
from __future__ import annotations
from dataclasses import dataclass, field, replace
from typing import Any, Dict, List, Tuple, cast
import re
from datetime import datetime, timedelta, timezone
import time
from pathlib import Path
import os
import traceback
import math
import duckdb
from dateutil import parser as dtparser
from flask import Flask, jsonify, request, send_from_directory
def _quote(ident: str) -> str:
"""Return identifier quoted for SQL."""
return f'"{ident.replace('"', '""')}"'
@dataclass
class Filter:
column: str
op: str
value: str | int | float | list[str] | None
@dataclass
class QueryParams:
start: str | None = None
end: str | None = None
order_by: str | None = None
order_dir: str = "ASC"
limit: int | None = None
columns: list[str] = field(default_factory=lambda: [])
filters: list[Filter] = field(default_factory=lambda: [])
derived_columns: dict[str, str] = field(default_factory=lambda: {})
graph_type: str = "samples"
group_by: list[str] = field(default_factory=lambda: [])
aggregate: str | None = None
show_hits: bool = False
x_axis: str | None = None
granularity: str = "Auto"
fill: str = "0"
table: str = "events"
time_column: str | None = "timestamp"
time_unit: str = "s"
def _load_database(path: Path) -> duckdb.DuckDBPyConnection:
if not path.exists():
raise FileNotFoundError(path)
ext = path.suffix.lower()
if ext == ".csv":
con = duckdb.connect()
con.execute(
f"CREATE TABLE events AS SELECT * FROM read_csv_auto('{path.as_posix
()}')"
)
elif ext in {".db", ".sqlite"}:
con = duckdb.connect()
con.execute("LOAD sqlite")
con.execute(f"ATTACH '{path.as_posix()}' AS db (TYPE SQLITE)")
tables = [
r[0]
for r in con.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()
]
for t in tables:
con.execute(f'CREATE VIEW "{t}" AS SELECT * FROM db."{t}"')
else:
con = duckdb.connect(path)
return con
def _create_test_database() -> duckdb.DuckDBPyConnection:
"""Return a DuckDB connection with a small multi-table dataset."""
con = duckdb.connect()
con.execute(
"CREATE TABLE events (id INTEGER PRIMARY KEY, ts INTEGER, val REAL, name
TEXT, flag BOOLEAN)"
)
con.execute("INSERT INTO events VALUES (1, 1704067200, 1.5, 'alice', 1)")
con.execute("INSERT INTO events VALUES (2, 1704070800, 2.0, 'bob', 0)")
con.execute('CREATE TABLE extra (ts INTEGER, "desc" TEXT, num INTEGER)')
con.execute("INSERT INTO extra VALUES (1704067200, 'x', 1)")
con.execute("INSERT INTO extra VALUES (1704070800, 'y', 2)")
return con
_REL_RE = re.compile(
r"([+-]?\d+(?:\.\d*)?)\s*(hour|hours|day|days|week|weeks|fortnight|fortnight
s)",
re.IGNORECASE,
)
def parse_time(val: str | None) -> str | None:
"""Parse an absolute or relative time string into ``YYYY-MM-DD HH:MM:SS``.""
"
if val is None or val == "":
return None
s = val.strip()
if s.lower() == "now":
dt = datetime.now(timezone.utc)
return dt.replace(microsecond=0).strftime("%Y-%m-%d %H:%M:%S")
m = _REL_RE.fullmatch(s)
if m:
qty = float(m.group(1))
unit = m.group(2).lower()
delta: timedelta
if unit.startswith("hour"):
delta = timedelta(hours=qty)
elif unit.startswith("day"):
delta = timedelta(days=qty)
elif unit.startswith("week"):
...
graph_type: str = "samples"
group_by: list[str] = field(default_factory=lambda: [])
aggregate: str | None = None
show_hits: bool = False
x_axis: str | None = None
granularity: str = "Auto"
fill: str = "0"
table: str = "events"
time_column: str | None = "timestamp"
time_unit: str = "s"
def _load_database(path: Path) -> duckdb.DuckDBPyConnection:
if not path.exists():
raise FileNotFoundError(path)
ext = path.suffix.lower()
if ext == ".csv":
con = duckdb.connect()
con.execute(
f"CREATE TABLE events AS SELECT * FROM read_csv_auto('{path.as_posix
()}')"
)
elif ext in {".db", ".sqlite"}:
con = duckdb.connect()
con.execute("LOAD sqlite")
con.execute(f"ATTACH '{path.as_posix()}' AS db (TYPE SQLITE)")
tables = [
r[0]
for r in con.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()
]
for t in tables:
con.execute(f'CREATE VIEW "{t}" AS SELECT * FROM db."{t}"')
else:
con = duckdb.connect(path)
return con
def _create_test_database() -> duckdb.DuckDBPyConnection:
"""Return a DuckDB connection with a small multi-table dataset."""
con = duckdb.connect()
con.execute(
"CREATE TABLE events (id INTEGER PRIMARY KEY, ts INTEGER, val REAL, name
TEXT, flag BOOLEAN)"
)
con.execute("INSERT INTO events VALUES (1, 1704067200, 1.5, 'alice', 1)")
con.execute("INSERT INTO events VALUES (2, 1704070800, 2.0, 'bob', 0)")
con.execute('CREATE TABLE extra (ts INTEGER, "desc" TEXT, num INTEGER)')
con.execute("INSERT INTO extra VALUES (1704067200, 'x', 1)")
con.execute("INSERT INTO extra VALUES (1704070800, 'y', 2)")
return con
_REL_RE = re.compile(
r"([+-]?\d+(?:\.\d*)?)\s*(hour|hours|day|days|week|weeks|fortnight|fortnight
s)",
re.IGNORECASE,
)
def parse_time(val: str | None) -> str | None:
"""Parse an absolute or relative time string into ``YYYY-MM-DD HH:MM:SS``.""
"
if val is None or val == "":
return None
s = val.strip()
if s.lower() == "now":
dt = datetime.now(timezone.utc)
return dt.replace(microsecond=0).strftime("%Y-%m-%d %H:%M:%S")
m = _REL_RE.fullmatch(s)
if m:
qty = float(m.group(1))
unit = m.group(2).lower()
delta: timedelta
if unit.startswith("hour"):
delta = timedelta(hours=qty)
elif unit.startswith("day"):
delta = timedelta(days=qty)
elif unit.startswith("week"):
delta = timedelta(weeks=qty)
elif unit.startswith("fortnight"):
delta = timedelta(weeks=2 * qty)
else: # pragma: no cover - defensive
raise ValueError(f"Unsupported unit: {unit}")
dt = datetime.now(timezone.utc) + delta
return dt.replace(microsecond=0).strftime("%Y-%m-%d %H:%M:%S")
dt = dtparser.parse(s)
return dt.replace(microsecond=0, tzinfo=None).strftime("%Y-%m-%d %H:%M:%S")
def _granularity_seconds(granularity: str, start: str | None, end: str | None) -
> int:
gran = granularity.lower()
mapping = {
"1 second": 1,
"5 seconds": 5,
"10 seconds": 10,
"30 seconds": 30,
"1 minute": 60,
"4 minutes": 240,
"5 minutes": 300,
"10 minutes": 600,
"15 minutes": 900,
"30 minutes": 1800,
"1 hour": 3600,
"3 hours": 10800,
"6 hours": 21600,
"1 day": 86400,
"1 week": 604800,
"30 days": 2592000,
}
if gran in mapping:
return mapping[gran]
if gran in {"auto", "fine"} and start and end:
try:
s = dtparser.parse(start)
e = dtparser.parse(end)
except Exception:
return 3600
total = max((e - s).total_seconds(), 1)Keep going to capture more details.
root@df946f6027ab:/workspace/scubaduck# sed -n '160,320p' scubaduck/server.py
total = max((e - s).total_seconds(), 1)
buckets = 100 if gran == "auto" else 500
return max(int(total // buckets), 1)
return 3600
def _time_expr(col: str, column_types: Dict[str, str] | None, unit: str) -> str:
"""Return SQL expression for column interpreted as timestamp."""
qcol = _quote(col)
if column_types is None:
return qcol
ctype = column_types.get(col, "").upper()
if not any(t in ctype for t in ["TIMESTAMP", "DATE", "TIME"]):
if any(
t in ctype
for t in [
"INT",
"DECIMAL",
"REAL",
"DOUBLE",
"FLOAT",
"NUMERIC",
"HUGEINT",
]
):
if unit == "ns":
expr = f"CAST({qcol} AS BIGINT)"
return f"make_timestamp_ns({expr})"
multiplier = {
"s": 1_000_000,
"ms": 1_000,
"us": 1,
}.get(unit, 1_000_000)
expr = (
f"CAST({qcol} * {multiplier} AS BIGINT)"
if multiplier != 1
else f"CAST({qcol} AS BIGINT)"
)
return f"make_timestamp({expr})"
return qcol
def build_query(params: QueryParams, column_types: Dict[str, str] | None = None)
-> str:
select_parts: list[str] = []
group_cols = params.group_by[:]
selected_for_order = set(params.columns) | set(params.derived_columns.keys()
)
if params.graph_type == "timeseries":
sec = _granularity_seconds(params.granularity, params.start, params.end)
x_axis = params.x_axis or params.time_column
if x_axis is None:
raise ValueError("x_axis required for timeseries")
xexpr = _time_expr(x_axis, column_types, params.time_unit)
if params.start:
bucket_expr = (
f"TIMESTAMP '{params.start}' + INTERVAL '{sec} second' * "
f"CAST(floor((epoch({xexpr}) - epoch(TIMESTAMP '{params.start}')
)/{sec}) AS BIGINT)"
)
else:
bucket_expr = (
f"TIMESTAMP 'epoch' + INTERVAL '{sec} second' * "
f"CAST(floor(epoch({xexpr})/{sec}) AS BIGINT)"
)
select_parts.append(f"{bucket_expr} AS bucket")
group_cols = ["bucket"] + group_cols
selected_for_order.add("bucket")
has_agg = bool(group_cols) or params.aggregate is not None
if has_agg:
select_cols = (
group_cols[1:] if params.graph_type == "timeseries" else group_cols
)
select_parts.extend(_quote(c) for c in select_cols)
agg = (params.aggregate or "count").lower()
selected_for_order.update(group_cols)
def agg_expr(col: str) -> str:
expr = _quote(col)
ctype = column_types.get(col, "").upper() if column_types else ""
if "BOOL" in ctype:
expr = f"CAST({_quote(col)} AS BIGINT)"
if agg.startswith("p"):
quant = float(agg[1:]) / 100
return f"quantile({expr}, {quant})"
if agg == "count distinct":
return f"count(DISTINCT {expr})"
if agg == "avg" and column_types is not None:
if "TIMESTAMP" in ctype or "DATE" in ctype or "TIME" in ctype:
return (
"TIMESTAMP 'epoch' + INTERVAL '1 second' * "
f"CAST(avg(epoch({_quote(col)})) AS BIGINT)"
)
return f"{agg}({expr})"
if agg == "count":
select_parts.append("count(*) AS Count")
selected_for_order.add("Count")
else:
for col in params.columns:
if col in group_cols:
continue
select_parts.append(f"{agg_expr(col)} AS {_quote(col)}")
selected_for_order.add(col)
if params.show_hits:
select_parts.insert(len(group_cols), "count(*) AS Hits")
selected_for_order.add("Hits")
else:
select_parts.extend(_quote(c) for c in params.columns)
selected_for_order.update(params.columns)
order_by = params.order_by if params.order_by in selected_for_order else Non
e
if has_agg and params.derived_columns:
inner_params = replace(
params,
derived_columns={},
order_by=None,
limit=None,
)
inner_sql = build_query(inner_params, column_types)
outer_select = ["t.*"] + [
f"{expr} AS {name}" for name, expr in params.derived_columns.items()
]
indented_inner = "\n".join(" " + line for line in inner_sql.splitline
s())
lines = [
f"SELECT {', '.join(outer_select)}",
"FROM (",
indented_inner,
") t",
]
if order_by:
lines.append(f"ORDER BY {_quote(order_by)} {params.order_dir}")
elif params.graph_type == "timeseries":
lines.append("ORDER BY bucket")
if params.limit is not None:
lines.append(f"LIMIT {params.limit}")
return "\n".join(lines)
for name, expr in params.derived_columns.items():
select_parts.append(f"{expr} AS {name}")
selected_for_order.add(name)
select_clause = ", ".join(select_parts) if select_parts else "*"
lines = [f"SELECT {select_clause}", f'FROM "{params.table}"']
where_parts: list[str] = []
if params.time_column:
time_expr = _time_expr(params.time_column, column_types, params.time_uni
t)
else:
time_expr = None
if time_expr and params.start:
where_parts.append(f"{time_expr} >= '{params.start}'")
if time_expr and params.end:
where_parts.append(f"{time_expr} <= '{params.end}'")
for f in params.filters:
op = f.op
if op in {"empty", "!empty"}:
val = "''"
else:
if f.value is None:
continue
if isinstance(f.value, list):
if not f.value:
continuecontinue
if op == "=":
qcol = _quote(f.column)
vals = " OR ".join(f"{qcol} = '{v}'" for v in f.value)
where_parts.append(f"({vals})")
continue
val = f"'{f.value}'" if isinstance(f.value, str) else str(f.value)
qcol = _quote(f.column)
if op == "contains":
where_parts.append(f"{qcol} ILIKE '%' || {val} || '%'")
elif op == "!contains":
where_parts.append(f"{qcol} NOT ILIKE '%' || {val} || '%'")
elif op == "empty":
where_parts.append(f"{qcol} = {val}")
elif op == "!empty":
where_parts.append(f"{qcol} != {val}")
else:
where_parts.append(f"{qcol} {op} {val}")
if where_parts:
lines.append("WHERE " + " AND ".join(where_parts))
if group_cols:
lines.append("GROUP BY " + ", ".join(_quote(c) for c in group_cols))
if order_by:
lines.append(f"ORDER BY {_quote(order_by)} {params.order_dir}")
elif params.graph_type == "timeseries":
lines.append("ORDER BY bucket")
if params.limit is not None:
lines.append(f"LIMIT {params.limit}")
return "\n".join(lines)
def create_app(db_file: str | Path | None = None) -> Flask:
app = Flask(__name__, static_folder="static")
if db_file is None:
env_db = os.environ.get("SCUBADUCK_DB")
if env_db:
db_file = env_db
if isinstance(db_file, str) and db_file.upper() == "TEST":
con = _create_test_database()
else:
db_path = Path(db_file or Path(__file__).with_name("sample.csv")).resolv
e()
con = _load_database(db_path)
tables = [r[0] for r in con.execute("SHOW TABLES").fetchall()]
if not tables:
raise ValueError("No tables found in database")
default_table = tables[0]
columns_cache: Dict[str, Dict[str, str]] = {}
def get_columns(table: str) -> Dict[str, str]:
if table not in columns_cache:
rows = con.execute(f'PRAGMA table_info("{table}")').fetchall()
if not rows:
raise ValueError(f"Unknown table: {table}")
columns_cache[table] = {r[1]: r[2] for r in rows}
return columns_cache[table]
sample_cache: Dict[Tuple[str, str, str], Tuple[List[str], float]] = {}
CACHE_TTL = 60.0
CACHE_LIMIT = 200
@app.route("/")
def index() -> Any: # pyright: ignore[reportUnusedFunction]
assert app.static_folder is not None
return send_from_directory(app.static_folder, "index.html")
@app.route("/js/<path:filename>")
def js(filename: str) -> Any: # pyright: ignore[reportUnusedFunction]
assert app.static_folder is not None
folder = Path(app.static_folder) / "js"
return send_from_directory(folder, filename)
@app.route("/api/tables")
def tables_endpoint() -> Any: # pyright: ignore[reportUnusedFunction]
return jsonify(tables)
@app.route("/api/columns")
def columns() -> Any: # pyright: ignore[reportUnusedFunction]
table = request.args.get("table", default_table)
rows = con.execute(f'PRAGMA table_info("{table}")').fetchall()
return jsonify([{"name": r[1], "type": r[2]} for r in rows])
def _cache_get(key: Tuple[str, str, str]) -> List[str] | None:
item = sample_cache.get(key)
if item is None:
return None
vals, ts = item
if time.time() - ts > CACHE_TTL:
del sample_cache[key]
return None
sample_cache[key] = (vals, time.time())
return vals
def _cache_set(key: Tuple[str, str, str], vals: List[str]) -> None:
sample_cache[key] = (vals, time.time())
if len(sample_cache) > CACHE_LIMIT:
oldest = min(sample_cache.items(), key=lambda kv: kv[1][1])[0]
del sample_cache[oldest]
@app.route("/api/samples")
def sample_values() -> Any: # pyright: ignore[reportUnusedFunction]
table = request.args.get("table", default_table)
column = request.args.get("column")
substr = request.args.get("q", "")
column_types = get_columns(table)
if not column or column not in column_types:
return jsonify([])
ctype = column_types[column].upper()
if "CHAR" not in ctype and "STRING" not in ctype and "VARCHAR" not in ct
ype:
return jsonify([])
key = (table, column, substr)
cached = _cache_get(key)
if cached is not None:
return jsonify(cached)
qcol = _quote(column)
rows = con.execute(
f"SELECT DISTINCT {qcol} FROM \"{table}\" WHERE CAST({qcol} AS VARCH
AR) ILIKE '%' || ? || '%' LIMIT 20",
[substr],
).fetchall()
values = [r[0] for r in rows]
_cache_set(key, values)
...
return jsonify({"error": f"Unknown column: {col}"}), 400
if params.order_by and params.order_by not in valid_cols:
return jsonify({"error": f"Unknown column: {params.order_by}"}), 400
if params.group_by or params.graph_type == "timeseries":
agg = (params.aggregate or "count").lower()
if agg.startswith("p") or agg == "sum":
need_numeric = True
allow_time = False
elif agg == "avg" or agg in {"min", "max"}:
need_numeric = False
allow_time = True
else:
need_numeric = False
allow_time = False
if need_numeric or allow_time:
for c in params.columns:
if c in params.group_by or c == params.x_axis:
continue
if c not in column_types:
continue
ctype = column_types.get(c, "").upper()
is_numeric = any(
t in ctype
for t in [
"INT",
"DECIMAL",
"REAL",
"DOUBLE",
"FLOAT",
"NUMERIC",
"HUGEINT",
]
)
is_time = "TIMESTAMP" in ctype or "DATE" in ctype or "TIME"
in ctype
if need_numeric and not is_numeric:
return (
jsonify(
{
"error": f"Aggregate {agg} cannot be applied
to column {c}",
}
),
400,
)
if allow_time and not (is_numeric or is_time):
return (
jsonify(
{
"error": f"Aggregate {agg} cannot be applied
to column {c}",
}
),
400,
)
if (params.start is None or params.end is None) and (
params.x_axis or params.time_column
):
axis = params.x_axis or params.time_column
assert axis is not None
row = cast(
tuple[datetime | None, datetime | None],
con.execute(
f'SELECT min({_quote(axis)}), max({_quote(axis)}) FROM "{par
ams.table}"'
).fetchall()[0],
)
mn, mx = row
divisor = {
"s": 1,
"ms": 1000,
"us": 1_000_000,
"ns": 1_000_000_000,
}.get(params.time_unit, 1)
if isinstance(mn, (int, float)):
try:
mn = datetime.fromtimestamp(int(mn) / divisor, tz=timezone.u
tc)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mn} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if isinstance(mx, (int, float)):
try:
mx = datetime.fromtimestamp(int(mx) / divisor, tz=timezone.u
tc)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mx} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if params.start is None and mn is not None:
params.start = (
mn.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mn, str)
else mn
)
if params.end is None and mx is not None:
params.end = (
mx.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mx, str)
else mx
)
bucket_size: int | None = None
if params.graph_type == "timeseries":
bucket_size = _granularity_seconds(
params.granularity,
params.start if isinstance(params.start, str) else None,
params.end if isinstance(params.end, str) else None,root@df946f6027ab:/workspace/scubaduck# sed -n '640,960p' scubaduck/server.py
params.end if isinstance(params.end, str) else None,
)
if (
params.limit is not None
and params.start is not None
and params.end is not None
):
try:
start_dt = dtparser.parse(params.start)
end_dt = dtparser.parse(params.end)
buckets = math.ceil(
(end_dt - start_dt).total_seconds() / bucket_size
)
if buckets > 1:
params.limit *= buckets
except Exception:
pass
sql = build_query(params, column_types)
try:
rows = con.execute(sql).fetchall()
except Exception as exc:
tb = traceback.format_exc()
print(f"Query failed:\n{sql}\n{tb}")
return (
jsonify({"sql": sql, "error": str(exc), "traceback": tb}),
400,
)
result: Dict[str, Any] = {"sql": sql, "rows": rows}
if params.start is not None:
result["start"] = str(params.start)
if params.end is not None:
result["end"] = str(params.end)
if bucket_size is not None:
result["bucket_size"] = bucket_size
return jsonify(result)
return app
app = create_app()
if __name__ == "__main__":
app.run(debug=True)I’ll open up sample.csv to understand the structure and investigate the time-related issues.
root@df946f6027ab:/workspace/scubaduck# cat scubaduck/sample.csv | head -n 5
timestamp,event,value,user
2024-01-01 00:00:00,login,10,alice
2024-01-01 01:00:00,logout,20,bob
2024-01-02 00:00:00,login,30,alice
2024-01-02 03:00:00,login,40,charlieThe dataset might be larger, but I'll examine the first few lines in init.py.
root@df946f6027ab:/workspace/scubaduck# sed -n '1,40p' scubaduck/__init__.py
def hello() -> str:
return "Hello from scubaduck!"To avoid overflow, I'll cast to BIGINT before multiplication. Then I'll adjust the bug reproduction logic for handling string-type time columns.
I’ll check if "TEST dataset" is linked to server.app and _create_test_database.
root@df946f6027ab:/workspace/scubaduck# grep -n "_create_test_database" -n scuba
duck/server.py
81:def _create_test_database() -> duckdb.DuckDBPyConnection:
359: con = _create_test_database())
elif ext in {".db", ".sqlite"}:
con = duckdb.connect()
con.execute("LOAD sqlite")
con.execute(f"ATTACH '{path.as_posix()}' AS db (TYPE SQLITE)")
tables = [
r[0]
for r in con.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()
]
for t in tables:
con.execute(f'CREATE VIEW "{t}" AS SELECT * FROM db."{t}"')
else:
con = duckdb.connect(path)
return con
def _create_test_database() -> duckdb.DuckDBPyConnection:
"""Return a DuckDB connection with a small multi-table dataset."""
con = duckdb.connect()
con.execute(
"CREATE TABLE events (id INTEGER PRIMARY KEY, ts INTEGER, val REAL, name
TEXT, flag BOOLEAN)"
)
con.execute("INSERT INTO events VALUES (1, 1704067200, 1.5, 'alice', 1)")
con.execute("INSERT INTO events VALUES (2, 1704070800, 2.0, 'bob', 0)")
con.execute('CREATE TABLE extra (ts INTEGER, "desc" TEXT, num INTEGER)')
con.execute("INSERT INTO extra VALUES (1704067200, 'x', 1)")
con.execute("INSERT INTO extra VALUES (1704070800, 'y', 2)")
return con
_REL_RE = re.compile(
r"([+-]?\d+(?:\.\d*)?)\s*(hour|hours|day|days|week|weeks|fortnight|fortnight
s)",
re.IGNORECASE,
)
def parse_time(val: str | None) -> str | None:
"""Parse an absolute or relative time string into ``YYYY-MM-DD HH:MM:SS``.""
"
if val is None or val == "":
return None
s = val.strip()
if s.lower() == "now":
dt = datetime.now(timezone.utc)
return dt.replace(microsecond=0).strftime("%Y-%m-%d %H:%M:%S")
m = _REL_RE.fullmatch(s)
if m:
qty = float(m.group(1))
unit = m.group(2).lower()
delta: timedelta
if unit.startswith("hour"):
delta = timedelta(hours=qty)
elif unit.startswith("day"):
delta = timedelta(days=qty)
elif unit.startswith("week"):
delta = timedelta(weeks=qty)
elif unit.startswith("fortnight"):
delta = timedelta(weeks=2 * qty)
else: # pragma: no cover - defensive
raise ValueError(f"Unsupported unit: {unit}")
dt = datetime.now(timezone.utc) + delta
return dt.replace(microsecond=0).strftime("%Y-%m-%d %H:%M:%S")
dt = dtparser.parse(s)
return dt.replace(microsecond=0, tzinfo=None).strftime("%Y-%m-%d %H:%M:%S")
def _granularity_seconds(granularity: str, start: str | None, end: str | None) -
> int:
gran = granularity.lower()
mapping = {
"1 second": 1,
"5 seconds": 5,
"10 seconds": 10,
"30 seconds": 30,
"1 minute": 60,
"4 minutes": 240,I'll check tests/test_server_basic for an example, particularly how the TEST dataset is used.
def test_basic_query() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-02 00:00:00",
"order_by": "timestamp",
"order_dir": "ASC",
"limit": 10,
"columns": ["timestamp", "event", "value", "user"],
"filters": [],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert data
rows = data["rows"]
# We expect first three rows (until 2024-01-02 00:00:00)
assert len(rows) == 3
assert rows[0][1] == "login"
assert rows[1][1] == "logout"
def test_js_served() -> None:
app = server.app
client = app.test_client()
rv = client.get("/js/chip_input.js")
assert rv.status_code == 200
assert b"initChipInput" in rv.data
def test_filter_multi_token() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-02 03:00:00",
"order_by": "timestamp",
"limit": 10,
"columns": ["timestamp", "event", "value", "user"],
"filters": [{"column": "user", "op": "=", "value": ["alice", "charlie"]}
],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert data
rows = data["rows"]
# Should only return rows for alice and charlie
assert len(rows) == 3
assert rows[0][3] == "alice"
assert rows[-1][3] == "charlie"
def test_empty_filter_is_noop() -> None:
app = server.app
client = app.test_client()
base_payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"limit": 100,
"columns": ["timestamp", "event", "value", "user"],
}
no_filter = {**base_payload, "filters": []}
empty_filter = {
**base_payload,
"filters": [{"column": "user", "op": "=", "value": None}],
}
rv1 = client.post(
"/api/query", data=json.dumps(no_filter), content_type="application/json
"
)
rv2 = client.post(
"/api/query", data=json.dumps(empty_filter), content_type="application/j
son"
)
rows1 = rv1.get_json()["rows"]
rows2 = rv2.get_json()["rows"]
assert rows1 == rows2
def test_select_columns() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"order_by": "timestamp",
"limit": 10,
"columns": ["timestamp", "user"],
"filters": [],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert data
rows = data["rows"]
assert len(rows[0]) == 2
assert rows[0][1] == "alice"
def test_string_filter_ops() -> None:
app = server.app
client = app.test_client()
base = {
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"order_by": "timestamp",
"limit": 100,
"columns": ["timestamp", "event", "value", "user"],
}
contains = {
**base,
"filters": [{"column": "user", "op": "contains", "value": "ali"}],
}
rv = client.post(
"/api/query", data=json.dumps(contains), content_type="application/json"
)
rows = rv.get_json()["rows"]
assert all("ali" in r[3] for r in rows)
regex = {
**base,
"filters": [{"column": "user", "op": "~", "value": "^a.*"}],
}
rv = client.post(
"/api/query", data=json.dumps(regex), content_type="application/json"
)
rows = rv.get_json()["rows"]
assert all(r[3].startswith("a") for r in rows)
assert len(rows) == 2
not_empty = {**base, "filters": [{"column": "user", "op": "!empty"}]}
rv = client.post(
"/api/query", data=json.dumps(not_empty), content_type="application/json
"
)
assert len(rv.get_json()["rows"]) == 4
def test_order_by_ignored_when_not_selected() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"order_by": "value",
"columns": ["timestamp"],
}root@df946f6027ab:/workspace/scubaduck# sed -n '160,320p' tests/test_server_basi
c.py
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert "ORDER BY" not in data["sql"]I see tests for time column, but not for this particular bug. I'll check other files.
import json
from scubaduck import server
def test_invalid_time_error() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "nonsense",
"end": "now",
"order_by": "timestamp",
"limit": 10,
"columns": ["timestamp"],
"filters": [],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 400
assert "error" in data
def test_query_error_returns_sql_and_traceback() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"columns": ["event"],
"group_by": ["user"],
"aggregate": "avg",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 400
assert "error" in data
def test_table_unknown_column_error() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-03 00:00:00",
"graph_type": "table",
"order_by": "timestamp",
"limit": 100,
"columns": ["user", "Hits", "value"],
"group_by": ["user"],
"aggregate": "Count",
"show_hits": True,
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 400
assert "Unknown column" in data["error"]
def test_samples_view_rejects_group_by() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"graph_type": "samples",
"group_by": ["user"],
"columns": ["timestamp"],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 400
assert "only valid" in data["error"]I think the issue happens when our server doesn't parse time correctly for integers. I'll check the parse_time function and also review build_query for _time_expr. There may also be something with the default calculation for Start/End when time unit is 'us'. I'll dive into the test and server.py to confirm.
root@df946f6027ab:/workspace/scubaduck# grep -n "start is None" -n scubaduck/ser
ver.py
578: if (params.start is None or params.end is None) and (
626: if params.start is None and mn is not None:if need_numeric and not is_numeric:
return (
jsonify(
{
"error": f"Aggregate {agg} cannot be applied
to column {c}",
}
),
400,
)
if allow_time and not (is_numeric or is_time):
return (
jsonify(
{
"error": f"Aggregate {agg} cannot be applied
to column {c}",
}
),
400,
)
if (params.start is None or params.end is None) and (
params.x_axis or params.time_column
):
axis = params.x_axis or params.time_column
assert axis is not None
row = cast(
tuple[datetime | None, datetime | None],
con.execute(
f'SELECT min({_quote(axis)}), max({_quote(axis)}) FROM "{par
ams.table}"'
).fetchall()[0],
)
mn, mx = row
divisor = {
"s": 1,
"ms": 1000,
"us": 1_000_000,
"ns": 1_000_000_000,
}.get(params.time_unit, 1)
if isinstance(mn, (int, float)):
try:
mn = datetime.fromtimestamp(int(mn) / divisor, tz=timezone.u
tc)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mn} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if isinstance(mx, (int, float)):
try:
mx = datetime.fromtimestamp(int(mx) / divisor, tz=timezone.u
tc)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mx} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if params.start is None and mn is not None:
params.start = (
mn.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mn, str)
else mn
)
if params.end is None and mx is not None:
params.end = (
mx.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mx, str)
else mx
)
bucket_size: int | None = None
if params.graph_type == "timeseries":
bucket_size = _granularity_seconds(
params.granularity,
params.start if isinstance(params.start, str) else None,
params.end if isinstance(params.end, str) else None,import json
from pathlib import Path
import pytest
from scubaduck import server
def test_integer_time_column(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text("created,event\n1704067200,login\n1704070800,logout\n")
app = server.create_app(csv_file)
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-01 01:00:00",
"order_by": "created",
"columns": ["created", "event"],
"time_column": "created",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 2
def test_integer_time_unit_ms(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text("created,event\n1704067200000,login\n1704070800000,logou
t\n")
app = server.create_app(csv_file)
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-01 01:00:00",
"order_by": "created",
"columns": ["created", "event"],
"time_column": "created",
"time_unit": "ms",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()def test_timeseries_default_xaxis_uses_time_column(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text("created,event\n1704067200000,login\n1704070800000,logou
t\n")
app = server.create_app(csv_file)
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-01 01:00:00",
"graph_type": "timeseries",
"granularity": "1 hour",
"columns": ["event"],
"aggregate": "Count",
"time_column": "created",
"time_unit": "ms",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 2
def test_integer_time_unit_us_default_start_end(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text(
"created,event\n1704067200000000,login\n1704070800000000,logout\n"
)
app = server.create_app(csv_file)
client = app.test_client()
payload = {
"table": "events",
"order_by": "created",
"columns": ["created", "event"],
"time_column": "created",
"time_unit": "us",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["start"] == "2024-01-01 00:00:00"
assert data["end"] == "2024-01-01 01:00:00"
assert len(data["rows"]) == 2
def test_sqlite_integer_time_unit_us(tmp_path: Path) -> None:
sqlite_file = tmp_path / "events.sqlite"
import sqlite3
conn = sqlite3.connect(sqlite_file)
conn.execute("CREATE TABLE visits (visit_time INTEGER, event TEXT)")
big_ts = 13384551652000000
conn.execute("INSERT INTO visits VALUES (?, ?)", (big_ts, "foo"))
conn.commit()
conn.close() # pyright: ignore[reportUnknownMemberType, reportAttributeAcce
ssIssue]
app = server.create_app(sqlite_file)
client = app.test_client()
payload = {
"table": "visits",
"start": "2394-02-20 00:00:00",
"end": "2394-02-21 00:00:00",
"order_by": "visit_time",
"columns": ["visit_time", "event"],
"time_column": "visit_time",
"time_unit": "us",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 1
def test_relative_time_query(monkeypatch: pytest.MonkeyPatch) -> None:
app = server.app
client = app.test_client()
from datetime import datetime
fixed_now = datetime(2024, 1, 2, 4, 0, 0)
class FixedDateTime(datetime):
@classmethod
def now(cls, tz=None): # type: ignore[override]
return fixed_now if tz is None else fixed_now.astimezone(tz)
monkeypatch.setattr(server, "datetime", FixedDateTime)
payload = {
"table": "events",
"start": "-1 hour",
"end": "now",
"order_by": "timestamp",
"limit": 100,
"columns": ["timestamp", "event", "value", "user"],
"filters": [],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json().py
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 1
assert data["rows"][0][3] == "charlie"
def test_default_start_end_returned() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"order_by": "timestamp",
"limit": 5,
"columns": ["timestamp"],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["start"] == "2024-01-01 00:00:00"
assert data["end"] == "2024-01-02 03:00:00"
def test_time_column_none_no_time_filter() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"columns": ["timestamp", "event"],
"time_column": "",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert "start" not in data and "end" not in data
assert len(data["rows"]) == 4The test passes, but there’s overflow risk with integer seconds and default time_unit 's'.
I'll use DuckDB to see if the dataset infers "created" as BIGINT or not.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# python - <<'PY'
> import duckdb, pandas
> import io
> csv='created,event\n1704067200,login\n1704070800,logout\n'
> con=duckdb.connect()
> # read_csv_auto: load from string
> con.execute("CREATE TABLE events AS SELECT * FROM read_csv_auto('data.csv', de
lim=',')")I realized I need a file path for read_csv_auto—time to create one!
> PYpython - <<'PY'
> import duckdb, io
> csv='created,event\n1704067200,login\n1704070800,logout\n'
> open('tmp.csv','w').write(csv)
> con=duckdb.connect()
> con.execute("CREATE TABLE events AS SELECT * FROM read_csv_auto('tmp.csv')")
> print(con.execute("PRAGMA table_info('events')").fetchall())I think I missed an "EOF" after the script. I'll run it again to check.
> PYpython - <<'PY'
> import duckdb, io
> csv='created,event\n1704067200,login\n1704070800,logout\n'
> open('tmp.csv','w').write(csv)
> con=duckdb.connect()
> con.execute("CREATE TABLE events AS SELECT * FROM read_csv_auto('tmp.csv')")
> print(con.execute("PRAGMA table_info('events')").fetchall())
> PY
File "<stdin>", line 7
PYpython - <<'PY'
^^
SyntaxError: invalid syntax
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I forgot to add quotes. I'll try again with the correct syntax this time.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# python - <<'PY'
> import duckdb, io
> csv='created,event\n1704067200,login\n1704070800,logout\n'
> open('tmp.csv','w').write(csv)
> con=duckdb.connect()
> con.execute("CREATE TABLE events AS SELECT * FROM read_csv_auto('tmp.csv')")
> print(con.execute("PRAGMA table_info('events')").fetchall())
> PY
[(0, 'created', 'BIGINT', False, None, False), (1, 'event', 'VARCHAR', False, No
ne, False)]
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The bug occurs with INT32, not BIGINT. I’ll check tests using INT32 columns.
from pathlib import Path
import duckdb
import pytest
from scubaduck import server
def _make_payload() -> dict[str, object]:
return {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-02 00:00:00",
"order_by": "timestamp",
"order_dir": "ASC",
"limit": 10,
"columns": ["timestamp", "event", "value", "user"],
"filters": [],
}
def test_database_types(tmp_path: Path) -> None:
csv_file = tmp_path / "events.csv"
csv_file.write_text(Path("scubaduck/sample.csv").read_text())
sqlite_file = tmp_path / "events.sqlite"
import sqlite3
conn = sqlite3.connect(sqlite_file)
conn.execute(
"CREATE TABLE events (timestamp TEXT, event TEXT, value INTEGER, user TE
XT)"
)
with open(csv_file) as f:
next(f)
for line in f:
ts, ev, val, user = line.strip().split(",")
conn.execute(
"INSERT INTO events VALUES (?, ?, ?, ?)", (ts, ev, int(val), use
r)
)
conn.commit()
conn.close() # pyright: ignore[reportUnknownMemberType, reportAttributeAcce
ssIssue]
duckdb_file = tmp_path / "events.duckdb"
con = duckdb.connect(duckdb_file)
con.execute(
f"CREATE TABLE events AS SELECT * FROM read_csv_auto('{csv_file.as_posix
()}')"
)
con.close() # pyright: ignore[reportUnknownMemberType, reportAttributeAcces
sIssue]
for db in (csv_file, sqlite_file, duckdb_file):
app = server.create_app(db)
client = app.test_client()
payload = _make_payload()
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/js
on"
)
rows = rv.get_json()["rows"]
assert len(rows) == 3
def test_sqlite_longvarchar(tmp_path: Path) -> None:
sqlite_file = tmp_path / "events.sqlite"
import sqlite3
conn = sqlite3.connect(sqlite_file)
conn.execute(
"CREATE TABLE events (timestamp TEXT, url LONGVARCHAR, title VARCHAR(10)
)"
)
conn.execute(
"INSERT INTO events VALUES ('2024-01-01 00:00:00', 'https://a.com', 'Hom
e')"
)
conn.commit()
conn.close() # pyright: ignore[reportUnknownMemberType, reportAttributeAcce
ssIssue]
app = server.create_app(sqlite_file)
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-01 01:00:00",
"order_by": "timestamp",
"columns": ["timestamp", "url", "title"],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["rows"][0][1] == "https://a.com"
def test_sqlite_bigint(tmp_path: Path) -> None:
sqlite_file = tmp_path / "big.sqlite"
import sqlite3
conn = sqlite3.connect(sqlite_file)
conn.execute("CREATE TABLE events (timestamp TEXT, value INTEGER)")
big_value = 13385262862605259
conn.execute(
"INSERT INTO events VALUES ('2024-01-01 00:00:00', ?)",
(big_value,),
)
conn.commit()
conn.close() # pyright: ignore[reportUnknownMemberType, reportAttributeAcce
ssIssue]
app = server.create_app(sqlite_file)
client = app.test_client()
payload = {
"table": "events",
"order_by": "timestamp",
"columns": ["timestamp", "value"],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["rows"][0][1] == big_value
def test_sqlite_boolean_aggregation(tmp_path: Path) -> None:
sqlite_file = tmp_path / "bool.sqlite"
import sqlite3
conn = sqlite3.connect(sqlite_file)
conn.execute("CREATE TABLE events (timestamp TEXT, flag BOOLEAN)")
conn.execute("INSERT INTO events VALUES ('2024-01-01 00:00:00', 1)")
conn.execute("INSERT INTO events VALUES ('2024-01-01 00:30:00', 0)")
conn.commit()
conn.close() # pyright: ignore[reportUnknownMemberType, reportAttributeAcce
ssIssue]
app = server.create_app(sqlite_file)
client = app.test_client()
payload = {
"table": "events",
"start": "2024-01-01 00:00:00",
"end": "2024-01-02 00:00:00",
"graph_type": "table",
"aggregate": "Avg",
"columns": ["flag"],
"show_hits": True,
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["rows"][0][0] == 2
assert data["rows"][0][1] == 0.5
def test_envvar_db(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
csv_file = tmp_path / "custom.csv"
csv_file.write_text("timestamp,event,value,user\n2024-01-01 00:00:00,login,5
,bob\n")
monkeypatch.setenv("SCUBADUCK_DB", str(csv_file))
app = server.create_app()
client = app.test_client()
payload = _make_payload()
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
rows = rv.get_json()["rows"]
assert len(rows) == 1
def test_envvar_db_missing(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> N
one:
missing = tmp_path / "missing.sqlite"
monkeypatch.setenv("SCUBADUCK_DB", str(missing))
with pytest.raises(FileNotFoundError):
server.create_app()> from scubaduck import server
> app = server.create_app("TEST")
> client = app.test_client()
> payload = {
> "table":"events",
> "time_column":"ts",
> "time_unit":"us",
> }
> rv = client.post("/api/query", data=json.dumps(payload), content_type="applica
tion/json")
> print(rv.status_code)
> print(rv.get_json())
> PY
200
{'end': '1970-01-01 00:28:24', 'rows': [], 'sql': 'SELECT *\nFROM "events"\nWHER
E make_timestamp(CAST("ts" AS BIGINT)) >= \'1970-01-01 00:28:24\' AND make_times
tamp(CAST("ts" AS BIGINT)) <= \'1970-01-01 00:28:24\'', 'start': '1970-01-01 00:
28:24'}
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The server uses a wrong method to compute start/end, interpreting seconds as microseconds.
I’ll check if timestamps under 1e12 should be treated as seconds, above as microseconds.
The issue might be from missing start/end values, not time unit misinterpretation.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# python - <<'PY'
> import json
> from scubaduck import server
> app = server.create_app("TEST")
> client = app.test_client()
> payload = {
> "table":"events",
> "time_column":"ts",
> }
> rv = client.post("/api/query", data=json.dumps(payload), content_type="applica
tion/json")
> print(rv.status_code)
> print(rv.get_json())
> PY
Query failed:
SELECT *
FROM "events"
WHERE make_timestamp(CAST("ts" * 1000000 AS BIGINT)) >= '2024-01-01 00:00:00' AN
D make_timestamp(CAST("ts" * 1000000 AS BIGINT)) <= '2024-01-01 01:00:00'
Traceback (most recent call last):
File "/workspace/scubaduck/scubaduck/server.py", line 660, in query
rows = con.execute(sql).fetchall()
^^^^^^^^^^^^^^^^
duckdb.duckdb.OutOfRangeException: Out of Range Error: Overflow in multiplicatio
n of INT32 (1704067200 * 1000000)!
400
{'error': 'Out of Range Error: Overflow in multiplication of INT32 (1704067200 *
1000000)!', 'sql': 'SELECT *\nFROM "events"\nWHERE make_timestamp(CAST("ts" * 1
000000 AS BIGINT)) >= \'2024-01-01 00:00:00\' AND make_timestamp(CAST("ts" * 100
0000 AS BIGINT)) <= \'2024-01-01 01:00:00\'', 'traceback': 'Traceback (most rece
nt call last):\n File "/workspace/scubaduck/scubaduck/server.py", line 660, in
query\n rows = con.execute(sql).fetchall()\n ^^^^^^^^^^^^^^^^\nduck
db.duckdb.OutOfRangeException: Out of Range Error: Overflow in multiplication of
INT32 (1704067200 * 1000000)!\n'}
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The server overflows with time_column ts when start/end are unspecified, defaulting to 2024. I’ll add tests to check both time_unit scenarios.
I’ll adjust the heuristics for detecting time units, especially for small timestamps, to handle bugs.
I’ll adjust time unit handling to check the timestamp year and multiply if too small.
I’ll add a function to handle edge cases and avoid mis-adjusting timestamps for small units.
I’ll add tests for time_column with units 's' and 'us' to validate the fix.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# python - <<'PY'
> import json
> from scubaduck import server
> app = server.create_app("TEST")
> client = app.test_client()
> payload = {
> "table":"events",
> "time_column":"ts",
> "time_unit":"us",
> }
> rv = client.post("/api/query", data=json.dumps(payload), content_type="applica
tion/json")
> print(rv.get_json())
> PY
{'end': '1970-01-01 00:28:24', 'rows': [], 'sql': 'SELECT *\nFROM "events"\nWHER
E make_timestamp(CAST("ts" AS BIGINT)) >= \'1970-01-01 00:28:24\' AND make_times
tamp(CAST("ts" AS BIGINT)) <= \'1970-01-01 00:28:24\'', 'start': '1970-01-01 00:
28:24'}(scubaduck) root@df946f6027ab:/workspace/scubaduck# python - <<'PY'
> import json
> from scubaduck import server
> app = server.create_app("TEST")
> client = app.test_client()
> payload = {
> "table":"events",
> "time_column":"ts",
> "time_unit":"s",
> }
> rv = client.post("/api/query", data=json.dumps(payload), content_type="applica
tion/json")
> print(rv.status_code, rv.get_json())
> PY
Query failed:
SELECT *
FROM "events"
WHERE make_timestamp(CAST("ts" * 1000000 AS BIGINT)) >= '2024-01-01 00:00:00' AN
D make_timestamp(CAST("ts" * 1000000 AS BIGINT)) <= '2024-01-01 01:00:00'
Traceback (most recent call last):
File "/workspace/scubaduck/scubaduck/server.py", line 660, in query
rows = con.execute(sql).fetchall()
^^^^^^^^^^^^^^^^
duckdb.duckdb.OutOfRangeException: Out of Range Error: Overflow in multiplicatio
n of INT32 (1704067200 * 1000000)!
400 {'error': 'Out of Range Error: Overflow in multiplication of INT32 (17040672
00 * 1000000)!', 'sql': 'SELECT *\nFROM "events"\nWHERE make_timestamp(CAST("ts"
* 1000000 AS BIGINT)) >= \'2024-01-01 00:00:00\' AND make_timestamp(CAST("ts" *
1000000 AS BIGINT)) <= \'2024-01-01 01:00:00\'', 'traceback': 'Traceback (most
recent call last):\n File "/workspace/scubaduck/scubaduck/server.py", line 660,
in query\n rows = con.execute(sql).fetchall()\n ^^^^^^^^^^^^^^^^\n
duckdb.duckdb.OutOfRangeException: Out of Range Error: Overflow in multiplicatio
n of INT32 (1704067200 * 1000000)!\n'}
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The overflow problem occurs due to calling _time_expr before casting. I'll fix that and account for microseconds by adjusting the start/end time calculation. Let’s patch server.py to include the new _parse_numeric_time function. After retrieving min/max, I'll apply this conversion.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# sed -n '560,640p' scubaduck/
server.py
if need_numeric and not is_numeric:
return (
jsonify(
{
"error": f"Aggregate {agg} cannot be applied
to column {c}",
}
),
400,
)
if allow_time and not (is_numeric or is_time):
return (
jsonify(
{
"error": f"Aggregate {agg} cannot be applied
to column {c}",
}
),
400,
)
if (params.start is None or params.end is None) and (
params.x_axis or params.time_column
):
axis = params.x_axis or params.time_column
assert axis is not None
row = cast(
tuple[datetime | None, datetime | None],
con.execute(
f'SELECT min({_quote(axis)}), max({_quote(axis)}) FROM "{par
ams.table}"'
).fetchall()[0],
)
mn, mx = row
divisor = {
"s": 1,
"ms": 1000,
"us": 1_000_000,
"ns": 1_000_000_000,
}.get(params.time_unit, 1)
if isinstance(mn, (int, float)):
try:
mn = datetime.fromtimestamp(int(mn) / divisor, tz=timezone.u
tc)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mn} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if isinstance(mx, (int, float)):
try:
mx = datetime.fromtimestamp(int(mx) / divisor, tz=timezone.u
tc)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mx} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if params.start is None and mn is not None:
params.start = (
mn.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mn, str)
else mn
)
if params.end is None and mx is not None:
params.end = (
mx.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mx, str)
else mx
)
bucket_size: int | None = None
if params.graph_type == "timeseries":
bucket_size = _granularity_seconds(
params.granularity,
params.start if isinstance(params.start, str) else None,
params.end if isinstance(params.end, str) else None,
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll begin by patching the _time_expr section earlier in the file. Let's adjust that.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# grep -n "_time_expr" -n scub
aduck/server.py
166:def _time_expr(col: str, column_types: Dict[str, str] | None, unit: str) ->
str:
212: xexpr = _time_expr(x_axis, column_types, params.time_unit)
304: time_expr = _time_expr(params.time_column, column_types, params.time
_unit)server.py
total = max((e - s).total_seconds(), 1)
buckets = 100 if gran == "auto" else 500
return max(int(total // buckets), 1)
return 3600
def _time_expr(col: str, column_types: Dict[str, str] | None, unit: str) -> str:
"""Return SQL expression for column interpreted as timestamp."""
qcol = _quote(col)
if column_types is None:
return qcol
ctype = column_types.get(col, "").upper()
if not any(t in ctype for t in ["TIMESTAMP", "DATE", "TIME"]):
if any(
t in ctype
for t in [
"INT",
"DECIMAL",
"REAL",
"DOUBLE",
"FLOAT",
"NUMERIC",
"HUGEINT",
]
):
if unit == "ns":
expr = f"CAST({qcol} AS BIGINT)"
return f"make_timestamp_ns({expr})"
multiplier = {
"s": 1_000_000,
"ms": 1_000,
"us": 1,
}.get(unit, 1_000_000)
expr = (
f"CAST({qcol} * {multiplier} AS BIGINT)"
if multiplier != 1
else f"CAST({qcol} AS BIGINT)"
)
return f"make_timestamp({expr})"
return qcol
def build_query(params: QueryParams, column_types: Dict[str, str] | None = None)
-> str:
select_parts: list[str] = []
group_cols = params.group_by[:]
selected_for_order = set(params.columns) | set(params.derived_columns.keys()
)
if params.graph_type == "timeseries":
sec = _granularity_seconds(params.granularity, params.start, params.end)
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I’ll modify _time_expr to cast as BIGINT before multiplication and add a normalization helper.
> *** Begin Patch
> *** Update File: scubaduck/server.py
> @@
> - multiplier = {
> - "s": 1_000_000,
> - "ms": 1_000,
> - "us": 1,
> - }.get(unit, 1_000_000)
> - expr = (
> - f"CAST({qcol} * {multiplier} AS BIGINT)"
> - if multiplier != 1
> - else f"CAST({qcol} AS BIGINT)"
> - )
> - return f"make_timestamp({expr})"
> + multiplier = {
> + "s": 1_000_000,
> + "ms": 1_000,
> + "us": 1,
> + }.get(unit, 1_000_000)
> + base = f"CAST({qcol} AS BIGINT)"
> + expr = (
> + f"CAST({base} * {multiplier} AS BIGINT)" if multiplier != 1 e
lse base
> + )
> + return f"make_timestamp({expr})"
> *** End Patch
> PATCH
Success. Updated the following files:
M scubaduck/server.py
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The patch worked; now I'll focus on adjusting the start and end calculations.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# sed -n '560,640p' scubaduck/
server.py
return (
jsonify(
{
"error": f"Aggregate {agg} cannot be applied
to column {c}",
}
),
400,
)
if allow_time and not (is_numeric or is_time):
return (
jsonify(
{
"error": f"Aggregate {agg} cannot be applied
to column {c}",
}
),
400,
)
if (params.start is None or params.end is None) and (
params.x_axis or params.time_column
):
axis = params.x_axis or params.time_column
assert axis is not None
row = cast(
tuple[datetime | None, datetime | None],
con.execute(
f'SELECT min({_quote(axis)}), max({_quote(axis)}) FROM "{par
ams.table}"'
).fetchall()[0],
)
mn, mx = row
divisor = {
"s": 1,
"ms": 1000,
"us": 1_000_000,
"ns": 1_000_000_000,
}.get(params.time_unit, 1)
if isinstance(mn, (int, float)):
try:
mn = datetime.fromtimestamp(int(mn) / divisor, tz=timezone.u
tc)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mn} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if isinstance(mx, (int, float)):
try:
mx = datetime.fromtimestamp(int(mx) / divisor, tz=timezone.u
tc)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mx} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if params.start is None and mn is not None:
params.start = (
mn.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mn, str)
else mn
)
if params.end is None and mx is not None:
params.end = (
mx.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mx, str)
else mx
)
bucket_size: int | None = None
if params.graph_type == "timeseries":
bucket_size = _granularity_seconds(
params.granularity,
params.start if isinstance(params.start, str) else None,
params.end if isinstance(params.end, str) else None,
)
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll replace the numeric computation of mn and mx with the new helper function.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# apply_patch<<'PATCH'
> *** Begin Patch
> *** Update File: scubaduck/server.py
> @@
> ):
> axis = params.x_axis or params.time_column
> assert axis is not None
> row = cast(
> tuple[datetime | None, datetime | None],
> con.execute(
> f'SELECT min({_quote(axis)}), max({_quote(axis)}) FROM "{
params.table}"'
> ).fetchall()[0],
> )
> mn, mx = row
> - divisor = {
> - "s": 1,
> - "ms": 1000,
> - "us": 1_000_000,
> - "ns": 1_000_000_000,
> - }.get(params.time_unit, 1)
> - if isinstance(mn, (int, float)):
> - try:
> - mn = datetime.fromtimestamp(int(mn) / divisor, tz=timezon
e.utc)
> - except Exception:
> - return (
> - jsonify(
> - {
> - "error": (
> - f"Invalid time value {mn} for column {axi
s}"
> - f" with time_unit {params.time_unit}"
> - )
> - }
> - ),
> - 400,
> - )
> - if isinstance(mx, (int, float)):
> - try:
> - mx = datetime.fromtimestamp(int(mx) / divisor, tz=timezon
e.utc)
> - except Exception:
> - return (
> - jsonify(
> - {
> - "error": (
> - f"Invalid time value {mx} for column {axi
s}"
> - f" with time_unit {params.time_unit}"
> - )
> - }
> - ),
> - 400,
> - )
> + if isinstance(mn, (int, float)):
> + try:
> + mn = _numeric_to_datetime(mn, params.time_unit)
> + except Exception:
> + return (
> + jsonify(
> + {
> + "error": (
> + f"Invalid time value {mn} for column {axi
s}"
> + f" with time_unit {params.time_unit}"
> + )
> + }
> + ),
> + 400,
> + )
> + if isinstance(mx, (int, float)):
> + try:
> + mx = _numeric_to_datetime(mx, params.time_unit)
> + except Exception:
> + return (
> + jsonify(
> + {
> + "error": (
> + f"Invalid time value {mx} for column {axi
s}"
> + f" with time_unit {params.time_unit}"
> + )
> + }
> + ),
> + 400,
> + )
> *** End Patch
> PATCH
Success. Updated the following files:
M scubaduck/server.py
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I need to add the _numeric_to_datetime function near parse_time. I'll search for parse_time next.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# grep -n "parse_time(" -n scu
baduck/server.py
101:def parse_time(val: str | None) -> str | None:
446: start = parse_time(payload.get("start"))
447: end = parse_time(payload.get("end"))con.execute("INSERT INTO extra VALUES (1704067200, 'x', 1)")
con.execute("INSERT INTO extra VALUES (1704070800, 'y', 2)")
return con
_REL_RE = re.compile(
r"([+-]?\d+(?:\.\d*)?)\s*(hour|hours|day|days|week|weeks|fortnight|fortnight
s)",
re.IGNORECASE,
)
def parse_time(val: str | None) -> str | None:
"""Parse an absolute or relative time string into ``YYYY-MM-DD HH:MM:SS``.""
"
if val is None or val == "":
return None
s = val.strip()
if s.lower() == "now":
dt = datetime.now(timezone.utc)
return dt.replace(microsecond=0).strftime("%Y-%m-%d %H:%M:%S")
m = _REL_RE.fullmatch(s)
if m:
qty = float(m.group(1))
unit = m.group(2).lower()
delta: timedelta
if unit.startswith("hour"):
delta = timedelta(hours=qty)
elif unit.startswith("day"):
delta = timedelta(days=qty)
elif unit.startswith("week"):
delta = timedelta(weeks=qty)
elif unit.startswith("fortnight"):
delta = timedelta(weeks=2 * qty)
else: # pragma: no cover - defensive
raise ValueError(f"Unsupported unit: {unit}")
dt = datetime.now(timezone.utc) + delta
return dt.replace(microsecond=0).strftime("%Y-%m-%d %H:%M:%S")
dt = dtparser.parse(s)
return dt.replace(microsecond=0, tzinfo=None).strftime("%Y-%m-%d %H:%M:%S")
def _granularity_seconds(granularity: str, start: str | None, end: str | None) -
> int:
gran = granularity.lower()
mapping = {
"1 second": 1,
"5 seconds": 5,
"10 seconds": 10,
"30 seconds": 30,
"1 minute": 60,
"4 minutes": 240,
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I think adding a _numeric_to_datetime helper after parse_time could be useful.
> *** Update File: scubaduck/server.py
> @@
> def parse_time(val: str | None) -> str | None:
> @@
> dt = dtparser.parse(s)
> return dt.replace(microsecond=0, tzinfo=None).strftime("%Y-%m-%d %H:%M:%S
")
> +
> +
> +def _numeric_to_datetime(value: int | float, unit: str) -> datetime:
> + """Convert a numeric timestamp ``value`` with unit ``unit`` to ``datetime
``.
> +
> + Heuristically fall back to seconds when the converted value is before 199
0
> + but the seconds interpretation is in a reasonable range. This handles
> + integer columns stored in seconds even when ``unit`` is mistakenly set to
a
> + finer granularity.
> + """
> +
> + divisor = {
> + "s": 1,
> + "ms": 1000,
> + "us": 1_000_000,
> + "ns": 1_000_000_000,
> + }.get(unit, 1)
> +
> + dt = datetime.fromtimestamp(int(value) / divisor, tz=timezone.utc)
> + if unit != "s" and dt.year < 1990:
> + alt = datetime.fromtimestamp(int(value), tz=timezone.utc)
> + if alt.year >= 1990:
> + dt = alt
> + return dt
> *** End Patch
> PATCH
Success. Updated the following files:
M scubaduck/server.py
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I confirmed the imports are correct. Now, I'll update the start/end patch and check formatting.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# grep -n "_numeric_to_datetim
e" -n scubaduck/server.py
132:def _numeric_to_datetime(value: int | float, unit: str) -> datetime:
615: mn = _numeric_to_datetime(mn, params.time_unit)
630: mx = _numeric_to_datetime(mx, params.time_unit)
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I've replaced the dividing code and removed the 'divisor' variable—checking around the patch now.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# sed -n '602,640p' scubaduck/
server.py
params.x_axis or params.time_column
):
axis = params.x_axis or params.time_column
assert axis is not None
row = cast(
tuple[datetime | None, datetime | None],
con.execute(
f'SELECT min({_quote(axis)}), max({_quote(axis)}) FROM "{par
ams.table}"'
).fetchall()[0],
)
mn, mx = row
if isinstance(mn, (int, float)):
try:
mn = _numeric_to_datetime(mn, params.time_unit)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mn} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),
400,
)
if isinstance(mx, (int, float)):
try:
mx = _numeric_to_datetime(mx, params.time_unit)
except Exception:
return (
jsonify(
{
"error": (
f"Invalid time value {mx} for column {axis}"
f" with time_unit {params.time_unit}"
)
}
),(scubaduck) root@df946f6027ab:/workspace/scubaduck# sed -n '640,680p' scubaduck/
server.py
),
400,
)
if params.start is None and mn is not None:
params.start = (
mn.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mn, str)
else mn
)
if params.end is None and mx is not None:
params.end = (
mx.strftime("%Y-%m-%d %H:%M:%S") if not isinstance(mx, str)
else mx
)
bucket_size: int | None = None
if params.graph_type == "timeseries":
bucket_size = _granularity_seconds(
params.granularity,
params.start if isinstance(params.start, str) else None,
params.end if isinstance(params.end, str) else None,
)
if (
params.limit is not None
and params.start is not None
and params.end is not None
):
try:
start_dt = dtparser.parse(params.start)
end_dt = dtparser.parse(params.end)
buckets = math.ceil(
(end_dt - start_dt).total_seconds() / bucket_size
)
if buckets > 1:
params.limit *= buckets
except Exception:
pass
sql = build_query(params, column_types)
try:
rows = con.execute(sql).fetchall()
except Exception as exc:
tb = traceback.format_exc()
print(f"Query failed:\n{sql}\n{tb}")
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll add new tests to test_server_time.py to confirm start and end times.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# sed -n '280,340p' tests/test
_server_time.py
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The file ends around line 260 or 280, but I’ll check near line 240.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# wc -n tests/test_server_time
.py
wc: invalid option -- 'n'
Try 'wc --help' for more information.(scubaduck) root@df946f6027ab:/workspace/scubaduck# wc -l tests/test_server_time
.py
198 tests/test_server_time.py
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The file is 198 lines long; I'll open it toward the bottom to append.
_time.py
)
data = rv.get_json()
assert rv.status_code == 200
assert len(data["rows"]) == 1
assert data["rows"][0][3] == "charlie"
def test_default_start_end_returned() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"order_by": "timestamp",
"limit": 5,
"columns": ["timestamp"],
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert data["start"] == "2024-01-01 00:00:00"
assert data["end"] == "2024-01-02 03:00:00"
def test_time_column_none_no_time_filter() -> None:
app = server.app
client = app.test_client()
payload = {
"table": "events",
"columns": ["timestamp", "event"],
"time_column": "",
}
rv = client.post(
"/api/query", data=json.dumps(payload), content_type="application/json"
)
data = rv.get_json()
assert rv.status_code == 200
assert "start" not in data and "end" not in data
assert len(data["rows"]) == 4
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The file ends at test_time_column_none_no_time_filter. I'll add tests right after that.
> *** Update File: tests/test_server_time.py
> @@
> def test_time_column_none_no_time_filter() -> None:
> @@
> assert len(data["rows"]) == 4
> +
> +
> +def test_test_dataset_int32_time_s() -> None:
> + app = server.create_app("TEST")
> + client = app.test_client()
> + payload = {
> + "table": "events",
> + "time_column": "ts",
> + "time_unit": "s",
> + }
> + rv = client.post(
> + "/api/query", data=json.dumps(payload), content_type="application/jso
n"
> + )
> + data = rv.get_json()
> + assert rv.status_code == 200
> + assert data["start"] == "2024-01-01 00:00:00"
> + assert data["end"] == "2024-01-01 01:00:00"
> + assert len(data["rows"]) == 2
> +
> +
> +def test_test_dataset_int32_time_us() -> None:
> + app = server.create_app("TEST")
> + client = app.test_client()
> + payload = {
> + "table": "events",
> + "time_column": "ts",
> + "time_unit": "us",
> + }
> + rv = client.post(
> + "/api/query", data=json.dumps(payload), content_type="application/jso
n"
> + )
> + data = rv.get_json()
> + assert rv.status_code == 200
> + assert data["start"] == "2024-01-01 00:00:00"
> + assert data["end"] == "2024-01-01 01:00:00"
> + assert len(data["rows"]) == 2
> *** End Patch
> PATCH
Success. Updated the following files:
M tests/test_server_time.py
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I’ll go ahead and run ruff format now that I’ve updated the tests.
.py tests/test_server_time.py >/tmp/ruff_format.log && tail -n 20 /tmp/ruff_form
at.log
1 file reformatted, 1 file left unchanged
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'm running the ruff check now to inspect the code.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# ruff check scubaduck/server.
py tests/test_server_time.py >/tmp/ruff_check.log && tail -n 20 /tmp/ruff_check.
log
All checks passed!(scubaduck) root@df946f6027ab:/workspace/scubaduck# pyright >/tmp/pyright.log &&
tail -n 20 /tmp/pyright.log0 errors, 0 warnings, 0 informations(scubaduck) root@df946f6027ab:/workspace/scubaduck# cat /tmp/pyright.log
0 errors, 0 warnings, 0 informations
(scubaduck) root@df946f6027ab:/workspace/scubaduck#It's time to run pytest and see if all tests pass. Let's go!
(scubaduck) root@df946f6027ab:/workspace/scubaduck# pytest -q >/tmp/pytest.log &
& tail -n 20 /tmp/pytest.logbringing up nodes...
bringing up nodes...
........................................................................ [ 63%]
..................................F...... [100%]
=================================== FAILURES ===================================
_______________________ test_test_dataset_int32_time_us ________________________
[gw1] linux -- Python 3.12.3 /workspace/scubaduck/.venv/bin/python
Traceback (most recent call last):
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/runner.p
y", line 341, in from_call
result: TResult | None = func()
^^^^^^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/runner.p
y", line 242, in <lambda>
lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_hooks.py
", line 512, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult
)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_manager.
py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 167, in _multicall
raise exception
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 139, in _multicall
teardown.throw(exception)
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/threadex
ception.py", line 92, in pytest_runtest_call
yield from thread_exception_runtest_hook()
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/threadex
ception.py", line 68, in thread_exception_runtest_hook
yield
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 139, in _multicall
teardown.throw(exception)
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/unraisab
leexception.py", line 95, in pytest_runtest_call
yield from unraisable_exception_runtest_hook()
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/unraisab
leexception.py", line 70, in unraisable_exception_runtest_hook
yield
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 139, in _multicall
teardown.throw(exception)
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/logging.
py", line 846, in pytest_runtest_call
yield from self._runtest_for(item, "call")
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/logging.
py", line 829, in _runtest_for
yield
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 139, in _multicall
teardown.throw(exception)
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/capture.
py", line 898, in pytest_runtest_call
return (yield)
^^^^^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 139, in _multicall
teardown.throw(exception)
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/skipping
.py", line 257, in pytest_runtest_call
return (yield)
^^^^^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 121, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/runner.p
y", line 174, in pytest_runtest_call
item.runtest()
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/python.p
y", line 1627, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_hooks.py
", line 512, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult
)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_manager.
py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 167, in _multicall
raise exception
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/pluggy/_callers.
py", line 121, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/scubaduck/.venv/lib/python3.12/site-packages/_pytest/python.p
y", line 159, in pytest_pyfunc_call
result = testfunction(**testargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/scubaduck/tests/test_server_time.py", line 234, in test_test_
dataset_int32_time_us
assert len(data["rows"]) == 2
AssertionError: assert 0 == 2
+ where 0 = len([])
=========================== short test summary info ============================
FAILED tests/test_server_time.py::test_test_dataset_int32_time_us - assert 0 ...
1 failed, 112 passed in 15.94s
(scubaduck) root@df946f6027ab:/workspace/scubaduck#Pytest failed since the fix didn't compute start/end as expected. I'll check the function.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# python - <<'PY'
> import json
> from scubaduck import server
> app=server.create_app("TEST")
> client=app.test_client()
> payload={"table":"events","time_column":"ts","time_unit":"us"}
> rv=client.post("/api/query",data=json.dumps(payload),content_type="application
/json")
> print(rv.get_json())
> PY
{'end': '2024-01-01 01:00:00', 'rows': [], 'sql': 'SELECT *\nFROM "events"\nWHER
E make_timestamp(CAST("ts" AS BIGINT)) >= \'2024-01-01 00:00:00\' AND make_times
tamp(CAST("ts" AS BIGINT)) <= \'2024-01-01 01:00:00\'', 'start': '2024-01-01 00:
00:00'}
(scubaduck) root@df946f6027ab:/workspace/scubaduck#The solution now adjusts _time_expr to handle integer columns like INT32 correctly, treating them as seconds.
I'll adjust _time_expr to treat 32-bit integers as seconds when unit is "ms" or "us".
I'll treat 'ns' as seconds if column is 32-bit, using a different conversion.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# sed -n '164,208p' scubaduck/
server.py
"4 minutes": 240,
"5 minutes": 300,
"10 minutes": 600,
"15 minutes": 900,
"30 minutes": 1800,
"1 hour": 3600,
"3 hours": 10800,
"6 hours": 21600,
"1 day": 86400,
"1 week": 604800,
"30 days": 2592000,
}
if gran in mapping:
return mapping[gran]
if gran in {"auto", "fine"} and start and end:
try:
s = dtparser.parse(start)
e = dtparser.parse(end)
except Exception:
return 3600
total = max((e - s).total_seconds(), 1)
buckets = 100 if gran == "auto" else 500
return max(int(total // buckets), 1)
return 3600
def _time_expr(col: str, column_types: Dict[str, str] | None, unit: str) -> str:
"""Return SQL expression for column interpreted as timestamp."""
qcol = _quote(col)
if column_types is None:
return qcol
ctype = column_types.get(col, "").upper()
if not any(t in ctype for t in ["TIMESTAMP", "DATE", "TIME"]):
if any(
t in ctype
for t in [
"INT",
"DECIMAL",
"REAL",
"DOUBLE",
"FLOAT",
"NUMERIC",
"HUGEINT",
]
):):
if unit == "ns":
expr = f"CAST({qcol} AS BIGINT)"
return f"make_timestamp_ns({expr})"
multiplier = {
"s": 1_000_000,
"ms": 1_000,
"us": 1,
}.get(unit, 1_000_000)
base = f"CAST({qcol} AS BIGINT)"
expr = f"CAST({base} * {multiplier} AS BIGINT)" if multiplier != 1 e
lse base
return f"make_timestamp({expr})"
return qcol
def build_query(params: QueryParams, column_types: Dict[str, str] | None = None)
-> str:
select_parts: list[str] = []
group_cols = params.group_by[:]
selected_for_order = set(params.columns) | set(params.derived_columns.keys()
)
if params.graph_type == "timeseries":
sec = _granularity_seconds(params.granularity, params.start, params.end)
x_axis = params.x_axis or params.time_column
if x_axis is None:
raise ValueError("x_axis required for timeseries")
xexpr = _time_expr(x_axis, column_types, params.time_unit)
if params.start:
bucket_expr = (
f"TIMESTAMP '{params.start}' + INTERVAL '{sec} second' * "
f"CAST(floor((epoch({xexpr}) - epoch(TIMESTAMP '{params.start}')
)/{sec}) AS BIGINT)"
)
else:
bucket_expr = (
f"TIMESTAMP 'epoch' + INTERVAL '{sec} second' * "
f"CAST(floor(epoch({xexpr})/{sec}) AS BIGINT)"
)
select_parts.append(f"{bucket_expr} AS bucket")
group_cols = ["bucket"] + group_cols
selected_for_order.add("bucket")
has_agg = bool(group_cols) or params.aggregate is not None
if has_agg:
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'm working on patching the _time_expr function now.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# apply_patch<<'PATCH'
> *** Begin Patch
> *** Update File: scubaduck/server.py
> @@
> - if any(
> - t in ctype
> - for t in [
> - "INT",
> - "DECIMAL",
> - "REAL",
> - "DOUBLE",
> - "FLOAT",
> - "NUMERIC",
> - "HUGEINT",
> - ]
> - ):
> - if unit == "ns":
> - expr = f"CAST({qcol} AS BIGINT)"
> - return f"make_timestamp_ns({expr})"
> -
> - multiplier = {
> - "s": 1_000_000,
> - "ms": 1_000,
> - "us": 1,
> - }.get(unit, 1_000_000)
> - base = f"CAST({qcol} AS BIGINT)"
> - expr = f"CAST({base} * {multiplier} AS BIGINT)" if multiplier !=
1 else base
> - return f"make_timestamp({expr})"
> + if any(
> + t in ctype
> + for t in [
> + "INT",
> + "DECIMAL",
> + "REAL",
> + "DOUBLE",
> + "FLOAT",
> + "NUMERIC",
> + "HUGEINT",
> + ]
> + ):
> + if unit == "ns":
> + # Use nanosecond helper unless column cannot represent such l
arge values
> + if "INT" in ctype and "BIGINT" not in ctype and "HUGEINT" not
in ctype:
> + unit = "s"
> + else:
> + expr = f"CAST({qcol} AS BIGINT)"
> + return f"make_timestamp_ns({expr})"
> +
> + if unit != "s" and "INT" in ctype and "BIGINT" not in ctype and "
HUGEINT" not in ctype:
> + # 32-bit integers cannot store sub-second precision for moder
n dates
> + unit = "s"
> +
> + multiplier = {
> + "s": 1_000_000,
> + "ms": 1_000,
> + "us": 1,
> + }.get(unit, 1_000_000)
> + base = f"CAST({qcol} AS BIGINT)"
> + expr = f"CAST({base} * {multiplier} AS BIGINT)" if multiplier !=
1 else base
> + return f"make_timestamp({expr})"
> *** End Patch
> PATCH
Success. Updated the following files:
M scubaduck/server.py
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll run ruff format once more to ensure everything is in order.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# ruff format scubaduck/server
.py tests/test_server_time.py >/tmp/ruff_format.log && tail -n 20 /tmp/ruff_form
at.log
1 file reformatted, 1 file left unchanged
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll rerun the ruff check to ensure everything looks good.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# ruff check scubaduck/server.
py tests/test_server_time.py >/tmp/ruff_check.log && tail -n 20 /tmp/ruff_check.
log
All checks passed!
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I’ll run Pyright again to check for any issues.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# pyright >/tmp/pyright.log &&
tail -n 20 /tmp/pyright.log0 errors, 0 warnings, 0 informations(scubaduck) root@df946f6027ab:/workspace/scubaduck# cat /tmp/pyright.log
0 errors, 0 warnings, 0 informations
(scubaduck) root@df946f6027ab:/workspace/scubaduck#Time to rerun the pytest test suite to check the results.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# pytest -q >/tmp/pytest.log &
& tail -n 20 /tmp/pytest.logbringing up nodes...
bringing up nodes...
........................................................................ [ 63%]
......................................... [100%]
113 passed in 14.87s
(scubaduck) root@df946f6027ab:/workspace/scubaduck#All tests passed, so it's time to commit the changes!
(scubaduck) root@df946f6027ab:/workspace/scubaduck# git status --short
M scubaduck/server.py
M tests/test_server_time.py
?? tmp.csv
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll clean up by deleting the tmp.csv file left from earlier.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# rm tmp.csv(scubaduck) root@df946f6027ab:/workspace/scubaduck# git status --short
M scubaduck/server.py
M tests/test_server_time.py
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'm going ahead and committing the changes now.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# git add scubaduck/server.py
tests/test_server_time.py(scubaduck) root@df946f6027ab:/workspace/scubaduck# git commit -m "fix(server):
handle int time columns" && git status --short
[work 950be3b] fix(server): handle int time columns
2 files changed, 79 insertions(+), 15 deletions(-)(scubaduck) root@df946f6027ab:/workspace/scubaduck# git status --short
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll gather the lines for modifications in scubaduck/server.py, focusing on those functions.
sed -n '120,150p'
120 delta = timedelta(weeks=qty)
121 elif unit.startswith("fortnight"):
122 delta = timedelta(weeks=2 * qty)
123 else: # pragma: no cover - defensive
124 raise ValueError(f"Unsupported unit: {unit}")
125 dt = datetime.now(timezone.utc) + delta
126 return dt.replace(microsecond=0).strftime("%Y-%m-%d %H:%M:%S")
127
128 dt = dtparser.parse(s)
129 return dt.replace(microsecond=0, tzinfo=None).strftime("%Y-%m-%d %H:
%M:%S")
130
131
132 def _numeric_to_datetime(value: int | float, unit: str) -> datetime:
133 """Convert a numeric timestamp ``value`` with unit ``unit`` to ``dat
etime``.
134
135 Heuristically fall back to seconds when the converted value is befor
e 1990
136 but the seconds interpretation is in a reasonable range. This handl
es
137 integer columns stored in seconds even when ``unit`` is mistakenly s
et to a
138 finer granularity.
139 """
140
141 divisor = {
142 "s": 1,
143 "ms": 1000,
144 "us": 1_000_000,
145 "ns": 1_000_000_000,
146 }.get(unit, 1)
147
148 dt = datetime.fromtimestamp(int(value) / divisor, tz=timezone.utc)
149 if unit != "s" and dt.year < 1990:
150 alt = datetime.fromtimestamp(int(value), tz=timezone.utc)(scubaduck) root@df946f6027ab:/workspace/scubaduck# nl -ba scubaduck/server.py |
sed -n '150,175p'
150 alt = datetime.fromtimestamp(int(value), tz=timezone.utc)
151 if alt.year >= 1990:
152 dt = alt
153 return dt
154
155
156 def _granularity_seconds(granularity: str, start: str | None, end: str |
None) -> int:
157 gran = granularity.lower()
158 mapping = {
159 "1 second": 1,
160 "5 seconds": 5,
161 "10 seconds": 10,
162 "30 seconds": 30,
163 "1 minute": 60,
164 "4 minutes": 240,
165 "5 minutes": 300,
166 "10 minutes": 600,
167 "15 minutes": 900,
168 "30 minutes": 1800,
169 "1 hour": 3600,
170 "3 hours": 10800,
171 "6 hours": 21600,
172 "1 day": 86400,
173 "1 week": 604800,
174 "30 days": 2592000,
175 }
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll focus on getting the lines around the _time_expr patch.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# nl -ba scubaduck/server.py |
sed -n '200,240p'
200 "INT",
201 "DECIMAL",
202 "REAL",
203 "DOUBLE",
204 "FLOAT",
205 "NUMERIC",
206 "HUGEINT",
207 ]
208 ):
209 if unit == "ns":
210 # Use nanosecond helper unless column cannot represent s
uch large values
211 if "INT" in ctype and "BIGINT" not in ctype and "HUGEINT
" not in ctype:
212 unit = "s"
213 else:
214 expr = f"CAST({qcol} AS BIGINT)"
215 return f"make_timestamp_ns({expr})"
216
217 if (
218 unit != "s"
219 and "INT" in ctype
220 and "BIGINT" not in ctype
221 and "HUGEINT" not in ctype
222 ):
223 # 32-bit integers cannot store sub-second precision for
modern dates
224 unit = "s"
225
226 multiplier = {
227 "s": 1_000_000,
228 "ms": 1_000,
229 "us": 1,
230 }.get(unit, 1_000_000)
231 base = f"CAST({qcol} AS BIGINT)"
232 expr = f"CAST({base} * {multiplier} AS BIGINT)" if multiplie
r != 1 else base
233 return f"make_timestamp({expr})"
234 return qcol
235
236
237 def build_query(params: QueryParams, column_types: Dict[str, str] | None
= None) -> str:
238 select_parts: list[str] = []
239 group_cols = params.group_by[:]
240 selected_for_order = set(params.columns) | set(params.derived_column
s.keys())
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'll get the relevant line ranges around the patch in tests/test_server_time.py.
(scubaduck) root@df946f6027ab:/workspace/scubaduck# nl -ba tests/test_server_tim
e.py | sed -n '208,266p'
208 }
209 rv = client.post(
210 "/api/query", data=json.dumps(payload), content_type="applicatio
n/json"
211 )
212 data = rv.get_json()
213 assert rv.status_code == 200
214 assert data["start"] == "2024-01-01 00:00:00"
215 assert data["end"] == "2024-01-01 01:00:00"
216 assert len(data["rows"]) == 2
217
218
219 def test_test_dataset_int32_time_us() -> None:
220 app = server.create_app("TEST")
221 client = app.test_client()
222 payload = {
223 "table": "events",
224 "time_column": "ts",
225 "time_unit": "us",
226 }
227 rv = client.post(
228 "/api/query", data=json.dumps(payload), content_type="applicatio
n/json"
229 )
230 data = rv.get_json()
231 assert rv.status_code == 200
232 assert data["start"] == "2024-01-01 00:00:00"
233 assert data["end"] == "2024-01-01 01:00:00"
234 assert len(data["rows"]) == 2
(scubaduck) root@df946f6027ab:/workspace/scubaduck#I'm drafting the PR message to summarize the changes now.