Tuning Home Assistant Voice: Sensor Fixes and Dynamic Weather with Ollama
Tuning Home Assistant Voice: Sensor Fixes and Dynamic Weather with Ollama
Fixing sensor lookup failures in Home Assistant Voice with Ollama prompt tuning, and building a dynamic weather script that resolves any city to a forecast with no API keys and no per-location setup.
This post builds on Local AI Voice Assistant with Home Assistant, Ollama, and Nvidia GPU on Proxmox — the Ollama LXC and voice pipeline are assumed to already be running.
Architecture Overview
+---------------------+ voice query +---------------------+ tool call +---------------------+
| | | | | |
| HA Voice PE | -------------------> | Ollama | -----------------> | get_weather |
| (ESP32 device) | | qwen3:8b | script action | script |
| | | | | |
+---------------------+ +---------------------+ +----------+----------+
| |
| sensor tool calls v
v +----------+----------+
+--------+--------+ | |
| | | weather.py |
| HA Entities | | |
| (sensors, | | Nominatim (OSM) |
| switches, | | + Open-Meteo API |
| lights) | | (no API key) |
+-----------------+ +---------------------+
Background
After getting Ollama running with Home Assistant Voice, two problems surfaced quickly in daily use.
Sensor lookups fail on natural language. Asking “what is the temperature in the bedroom?” returns nothing. Asking “what is the bedroom temperature sensor reading?” works. The model does not connect room-condition questions to sensor entities — it looks for a thermostat or climate device, finds none, and gives up.
Weather is locked to a single location. The built-in weather blueprints and integrations require a zone and weather entity per location. Asking about a different city is not possible without manual setup for each one. With a second location needed — a spouse’s workplace in a different town — the manual approach doesn’t scale.
Both problems are solvable: the sensor issue with prompt tuning, the weather issue with a small Python script and two free APIs.
Problem 1 — Sensor Lookups
Root cause
Small models (4–8B) treat device type literally. When asked about bedroom temperature, qwen3:4b-instruct looks for a climate domain entity in the bedroom. The temperature sensor is in the sensor domain, not climate — so the model finds nothing and falls back to its default failure response.
The fix is in the system prompt: explicitly tell the model that room-condition questions map to sensor lookups.
Prompt tuning
Before:
When asked about sensors, devices, or anything in the home, always use the
available tools to look up the current state before answering.
Never guess the state of a device or sensor.
After:
When asked about anything in the home — devices, sensors, lights, temperature,
humidity, or any smart home state — you MUST call the available tools to get
the current data before answering. Do not rely on memory or prior context.
When asked about temperature, humidity, CO2, or air quality in any room,
treat this as a sensor query and use the tools to look it up.
"What is the temperature in the bedroom" means the same as
"What is the bedroom temperature sensor reading."
If the tool returns data, read the exact value and unit from the result.
Do not round, estimate, or paraphrase sensor values.
If the tool returns no data for the requested device or area, say:
"I could not find that information in your smart home."
Never guess the state of a device or sensor.
Key changes:
MUST callinstead ofalways use— imperative phrasing improves tool-call compliance in smaller models- Explicit mapping of room-condition questions to sensor domain lookups
Do not rely on memory or prior context— prevents the model from reusing a stale reading from earlier in the conversationDo not round, estimate, or paraphrase sensor values— stops the model from returning “about 70 degrees” when the sensor reads 68.6°F
Problem 2 — Dynamic Weather for Any Location
Approach
Rather than adding a weather integration per location, a Python script handles the full lookup chain:
- Accept any city name or city + state string as input
- Resolve it to coordinates via OpenStreetMap Nominatim (free, no key)
- Fetch the daily forecast from Open-Meteo (free, no key)
- Print a plain spoken-word result to stdout
Home Assistant exposes this as a shell_command, wrapped in a script with a location field, exposed to the Ollama conversation agent as a callable tool.
File structure
config/
├── configuration.yaml
├── scripts.yaml
└── python_scripts/
└── weather.py
configuration.yaml
Add to the bottom of the existing file:
1python_script:
2
3shell_command:
4 get_weather: "python3 /config/python_scripts/weather.py '{{ location }}'"
python_scripts/weather.py
1import urllib.request
2import urllib.parse
3import json
4import sys
5
6STATE_MAP = {
7 "AL": "Alabama", "AK": "Alaska", "AZ": "Arizona", "AR": "Arkansas",
8 "CA": "California", "CO": "Colorado", "CT": "Connecticut", "DE": "Delaware",
9 "FL": "Florida", "GA": "Georgia", "HI": "Hawaii", "ID": "Idaho",
10 "IL": "Illinois", "IN": "Indiana", "IA": "Iowa", "KS": "Kansas",
11 "KY": "Kentucky", "LA": "Louisiana", "ME": "Maine", "MD": "Maryland",
12 "MA": "Massachusetts", "MI": "Michigan", "MN": "Minnesota", "MS": "Mississippi",
13 "MO": "Missouri", "MT": "Montana", "NE": "Nebraska", "NV": "Nevada",
14 "NH": "New Hampshire", "NJ": "New Jersey", "NM": "New Mexico", "NY": "New York",
15 "NC": "North Carolina", "ND": "North Dakota", "OH": "Ohio", "OK": "Oklahoma",
16 "OR": "Oregon", "PA": "Pennsylvania", "RI": "Rhode Island", "SC": "South Carolina",
17 "SD": "South Dakota", "TN": "Tennessee", "TX": "Texas", "UT": "Utah",
18 "VT": "Vermont", "VA": "Virginia", "WA": "Washington", "WV": "West Virginia",
19 "WI": "Wisconsin", "WY": "Wyoming"
20}
21
22# Small towns that Nominatim returns the wrong state for when using abbreviations.
23# Add entries here as needed — full state name in the query fixes the ambiguity.
24LOCATION_FIXES = {
25 "springfield, or": "Springfield, Oregon",
26 "springfield, il": "Springfield, Illinois",
27 "springfield, mo": "Springfield, Missouri",
28}
29
30def get_weather(location):
31 # Apply hardcoded fixes before any processing
32 location = LOCATION_FIXES.get(location.lower(), location)
33
34 # Expand state abbreviations before querying — Nominatim returns wrong
35 # state when abbreviations are ambiguous between common city names
36 if "," in location:
37 city, state = location.split(",", 1)
38 state = state.strip().upper()
39 state = STATE_MAP.get(state, state)
40 location = f"{city.strip()}, {state}"
41
42 # Step 1: Geocode via OpenStreetMap Nominatim
43 headers = {"User-Agent": "HomeAssistantWeather/1.0"}
44 nom_url = (
45 f"https://nominatim.openstreetmap.org/search"
46 f"?q={urllib.parse.quote(location)}"
47 f"&format=json&limit=1&countrycodes=us&addressdetails=1"
48 )
49 req = urllib.request.Request(nom_url, headers=headers)
50 with urllib.request.urlopen(req) as r:
51 results = json.loads(r.read())
52
53 if not results:
54 print(f"I couldn't find a location called {location}.")
55 return
56
57 matched = results[0]
58 lat = float(matched["lat"])
59 lon = float(matched["lon"])
60 addr = matched.get("address", {})
61 city = (
62 addr.get("town")
63 or addr.get("city")
64 or addr.get("village")
65 or location.split(",")[0]
66 )
67 state = addr.get("state", "")
68
69 # Step 2: Fetch forecast from Open-Meteo
70 wx_url = (
71 f"https://api.open-meteo.com/v1/forecast"
72 f"?latitude={lat}&longitude={lon}"
73 f"&daily=temperature_2m_max,temperature_2m_min,precipitation_probability_max"
74 f"&temperature_unit=fahrenheit"
75 f"&timezone=America/New_York"
76 f"&forecast_days=2"
77 )
78 with urllib.request.urlopen(wx_url) as r:
79 wx = json.loads(r.read())
80
81 daily = wx["daily"]
82 high = daily["temperature_2m_max"][0]
83 low = daily["temperature_2m_min"][0]
84 precip = daily["precipitation_probability_max"][0]
85
86 print(
87 f"In {city}, {state}: high of {high} degrees, low of {low} degrees, "
88 f"{precip} percent chance of precipitation."
89 )
90
91get_weather(sys.argv[1])
scripts.yaml
1get_weather:
2 alias: Get Weather
3 description: Get the current weather forecast for any location
4 fields:
5 location:
6 description: The city and state to get weather for, e.g. Springfield, Oregon
7 required: true
8 example: Springfield, Oregon
9 sequence:
10 - action: shell_command.get_weather
11 data:
12 location: "{{ location }}"
13 response_variable: weather_result
14 - stop: "Done"
15 response_variable: weather_result
The stop action passes the full shell command response object back to Ollama, which includes stdout, stderr, and returncode. The system prompt instructs the model to read the stdout field.
Geocoding Notes
Two issues came up during development worth documenting.
State abbreviation ambiguity. For town names shared across multiple states, Nominatim may return the wrong state when an abbreviation is used — ranking results by dataset density rather than the intended location. Expanding the abbreviation to the full state name in the query string before sending it resolves this consistently.
Small towns missing from Open-Meteo geocoding. The Open-Meteo geocoding API has incomplete coverage for small towns and may return a different city with the same name in another state. OpenStreetMap Nominatim covers small towns more reliably. Nominatim is used exclusively for geocoding; Open-Meteo is used only for the forecast data after coordinates are resolved.
Hardcoded fixes for persistent mismatches. For locations where the abbreviation + full state expansion still returns the wrong result, a LOCATION_FIXES dict at the top of the script maps the raw input directly to the correct full-name query. This is a pragmatic override rather than a general solution — add entries as they come up.
Removing a Conflicting Blueprint
During testing, weather queries were returning “Unable to get forecasts for…” in under 10 milliseconds — fast enough to confirm that Ollama was never running at all. The pipeline trace showed intent-start and intent-end with a 6ms gap, which means a built-in HA intent intercepted the query before it reached the conversation agent.
The cause was a previously installed weather blueprint that registers a weather intent at startup regardless of whether it is exposed to the voice assistant. Removing the blueprint from scripts.yaml entirely and doing a full restart cleared the conflict.
If weather queries return instantly with a canned error message, check scripts.yaml for any weather-related blueprints and remove them before troubleshooting the conversation agent.
Updated System Prompt
Full system prompt after both fixes:
You are a voice assistant for Home Assistant.
Answer questions about the world truthfully.
Answer in plain text only. No markdown, no bullet points, no asterisks.
Keep answers short and conversational, as your response will be spoken aloud.
The current time is {{ now() }}.
The location is [Your City], [Your State].
When asked about anything in the home — devices, sensors, lights, temperature,
humidity, or any smart home state — you MUST call the available tools to get
the current data before answering. Do not rely on memory or prior context.
When asked about temperature, humidity, CO2, or air quality in any room,
treat this as a sensor query and use the tools to look it up.
"What is the temperature in the bedroom" means the same as
"What is the bedroom temperature sensor reading."
When asked about weather for any location, use the get_weather script tool
and pass the city and state as the location parameter.
If no location is specified, use [Your City], [Your State].
If asked about my wife's work, use [Spouse's City], [State].
When the get_weather tool returns a result, read the stdout field and
speak it directly.
If the tool returns data, read the exact value and unit from the result.
Do not round, estimate, or paraphrase sensor values.
If the tool returns no data, say: "I could not find that information."
Never guess the state of a device, sensor, or weather condition.
Model Notes
The GTX 1080 Ti (11GB VRAM) was running qwen3:4b-instruct at roughly 30% VRAM utilization. Upgrading to qwen3:8b at Q5_K_M quantization uses about 75% of available VRAM and noticeably improves tool-call reliability with no meaningful speed penalty for short voice responses.
| Model | Quantization | VRAM | Tool call reliability |
|---|---|---|---|
| qwen3:4b-instruct | default | ~3GB | Moderate |
| qwen3:8b | Q5_K_M | ~8GB | Good |
| mistral:7b | Q5_K_M | ~7GB | Good — strong function calling |
Avoid Q8 for 8B models on 11GB cards — it fits but leaves no headroom and may spill to system RAM under load.
Outcomes
| Sensor lookups | Natural room-condition questions correctly resolve to sensor entities |
| Weather | Any city resolvable by name — no zones, no integrations, no API keys |
| APIs used | OpenStreetMap Nominatim (geocoding), Open-Meteo (forecast) — both free |
| Model | qwen3:8b Q5_K_M — better tool reliability than 4b, fits in 11GB VRAM |
| Response time | 5–8 seconds end-to-end including geocoding and forecast fetch |