Wikidata:Report a technical problem/WDQS and Search

SpBot archives all sections tagged with {{Section resolved|1=~~~~}} after 1 day.

Report a problemHow to report a problemHelp with PhabricatorGet involvedWDQS and Search

On this page, old discussions are archived after 60 days. An overview of all archives can be found at this page's archive index. The current archive is located at 2026/08.

Start a new discussion

Chained OPTIONALs: NULL in first OPTIONAL causes unrelated DB IDs (all Q and P numbers) to be returned instead of NULL

Summary

When two dependent OPTIONAL patterns are used (the second depends on a variable bound by the first) and the first OPTIONAL yields NULL, the engine returns many or all database IDs (e.g., Q1…, P1…) as bindings for that variable instead of NULL. In the example ?PartOF contains every ID found in the dataset. Running this query on query.wikidata.org can cause high load or crash the browser.

Reproduction steps

Open a SPARQL endpoint (e.g., query.wikidata.org).

Run:

SELECT DISTINCT ?Person  ?PartOF ?PartOFFilterLabel
WHERE{
   ?Person wdt:P31 wd:Q825 .
   OPTIONAL { ?Person wdt:P361 ?PartOF . }
   OPTIONAL { ?PartOF rdfs:label ?PartOFFilterLabel . }
 }

Observed behavior

For persons without a P361 value (i.e., ?PartOF should be NULL), the result nonetheless includes many or all IDs from the database as values for ?PartOF, and corresponding labels for ?PartOFFilterLabel. On query.wikidata.org this can cause excessive load or a browser crash.

Expected behavior

If ?PartOF is unbound (NULL) for a given ?Person, both ?PartOF and ?PartOFFilterLabel should remain NULL. The second OPTIONAL must not produce bindings when the variable from the first OPTIONAL is unbound.

Verified workaround

Placing both triple patterns inside a single OPTIONAL produces the correct result:

It works as intended when you do

 SELECT DISTINCT ?Person  ?PartOF ?PartOFFilterLabel
 WHERE{
   ?Person wdt:P31 wd:Q825 .
   OPTIONAL { 
     ?Person wdt:P361 ?PartOF .
     ?PartOF rdfs:label ?PartOFFilterLabel .
   }
 } ~2026-18859-94 (talk) 09:37, 26 March 2026 (UTC)

Better solution that keeps the label optional even if the P361 statement exists:

SELECT DISTINCT ?Person  ?PartOF ?PartOFFilterLabel
WHERE{
  ?Person wdt:P31 wd:Q825 .
  OPTIONAL { 
    ?Person wdt:P361 ?PartOF .
    OPTIONAL {
      ?PartOF rdfs:label ?PartOFFilterLabel .
    }
  }
}

Try it!

--Lucas Werkmeister (WMDE) (talk) 12:20, 26 March 2026 (UTC)

This issue is expected behavior AFAIK. A completely unbound variable would match any value. With a left join that has undefined values on the join key, the behavior is that the undefined value matches any value on the opposite side of the join that fits the basic graph pattern, so for that one row it matches all the labels in the graph. It is an interesting corner case, catches people by surprise I would imagine. Infrastruktur (talk) 17:29, 26 March 2026 (UTC)

Thanks to you both.

If that's the case, something like:

OPTIONAL { FILTER(BOUND(?PartOF)) ?PartOF rdfs:label ?PartOFFilterLabel . }

FILTER(BOUND(?PartOF) && BOUND(?PartOFFilterLabel))

should work, right? But it doesn't. ~2026-19142-54 (talk) 13:47, 27 March 2026 (UTC)

SELECT * WHERE {
  # Intersection join on ?a. 1 is in both sets, UNDEF matches all rows in other set
  { SELECT * WHERE {
    VALUES (?a ?b) { (1 4) (2 5) (UNDEF 6) }
  } }
  VALUES (?a ?c) { (1 10) (8 11) (9 12) }
}

Try it! (QLever)

Q.E.D. Infrastruktur (talk) 09:08, 28 March 2026 (UTC)

Not really, no. As I understand it, OPTIONAL blocks are semantically evaluated “inside out”: the contents of the OPTIONAL block are first evaluated on their own, then joined with the solutions of the outer graph pattern. The SPARQL engine can often optimize this (when you write ?item wdt:P1559 "Douglas Adams"@en. OPTIONAL { ?item wdt:P18 ?image. }, Blazegraph doesn’t actually collect six million wdt:P18 triples first before joining them with the single ?item result from the outside pattern), but if you try to do something like FILTER(BOUND(…)) inside the OPTIONAL block using a variable from outside the block, you won’t get the expected result. Lucas Werkmeister (WMDE) (talk) 10:06, 1 April 2026 (UTC)

requests.exceptions.JSONDecodeError, maybe truncated responses?

A few times per week, I get a requests.exceptions.JSONDecodeError exception from my python code that uses WDQS. The code is https://github.com/dseomn/rock-paper-sand/blob/9eb47940b46659317a57082dfd546fb452f8f477/rock_paper_sand/wikidata.py#L231-L235 which translates to the query below, where {class_ref.id} is a class Q-id. I don't know which class(es) are triggering the issue, but judging from the large line numbers in the errors, it's probably classes with many transitive subclasses.

SELECT REDUCED ?class WHERE {
  ?class (wdt:P279|owl:sameAs)* wd:{class_ref.id}.
  ?class wikibase:sitelinks [].
}

Here are some of the errors I've seen from that query:

• requests.exceptions.JSONDecodeError: Unterminated string starting at: line 721849 column 19 (char 17465326)
• requests.exceptions.JSONDecodeError: Expecting value: line 115291 column 7 (char 2785280)
• requests.exceptions.JSONDecodeError: Expecting ',' delimiter: line 2741 column 3 (char 65536)

I'm not too familiar with the json parser, but at a glance those errors all look like they could be caused by a truncated response. The errors are sporadic though, not every time. Is there a maximum response size with some randomness, or something? dseomn (talk) 17:50, 23 April 2026 (UTC)

Hey @Dseomn,

It would be very useful if you could log the {class_ref.id} that is triggering this issue, as well as the full response payload.

> I'm not too familiar with the json parser, but at a glance those errors all look like they could be caused by a truncated response. The errors are sporadic though, not every time. Is there a maximum response size with some randomness, or something?

WDQS streams results as they are produced rather than materializing the full result set first, and then sending it. This means the http response starts with a 200 and begins writing JSON bindings incrementally. If something goes wrong (a server-side timeout, memory pressure, the reverse proxy cutting the connection, or some network transient error happened) the response stops mid-stream. Your client will see a 200, but the response body is incomplete (and fails to parse). This can happen with queries that return large result sets, like queries over transitive subclasses of a large class.

One mitigation that comes to mind is wrapping the .json() call and retry the call to WDQS on JSONDecodeError (ideally with backoff).

Unfortunately WDQS's Blazegraph backend does not report Content-Length of chunk encoding transfer headers. This is an area were we (Wikidata Platform) could improve, I'll file a ticket for future WDQS v2 work. GModena (WMF) (talk) 20:03, 23 April 2026 (UTC)

Typo:

> Unfortunately WDQS's Blazegraph backend does not report Content-Length of or chunk encoding transfer headers GModena (WMF) (talk) 20:16, 23 April 2026 (UTC)

> I'll file a ticket for future WDQS v2 work.

https://phabricator.wikimedia.org/T424336 GModena (WMF) (talk) 12:35, 24 April 2026 (UTC)

I added a line to include the full query in the traceback. I didn't include the full response though since I wasn't sure if that would cause the error message to get truncated before I saw it. Hopefully once I see what class(es) are causing it, I can try those queries a few times with the local cache disabled, and get the full response that way.

Once I've got that, I'll post here and look into retrying on JSONDecodeError, thanks! dseomn (talk) 19:37, 24 April 2026 (UTC)

I got the error requests.exceptions.JSONDecodeError: Unterminated string starting at: line 721848 column 18 (char 17465341) with sculpture (Q860861): SELECT REDUCED ?class WHERE { ?class (wdt:P279|owl:sameAs)* wd:Q860861. ?class wikibase:sitelinks []. }. I tried to get the full response, but didn't get the error again with that query after a bunch of tries over two days.

P.S. I tried to put the code that I used to try to get the error here, but I couldn't figure out how to put a pre-formatted block of code in a reply. The colons kept messing up the formatting. dseomn (talk) 19:07, 29 April 2026 (UTC)

unexpected side effect

Please compare the results of these 2 queries:

SELECT DISTINCT ?id ?idLabel (sample(?geo) as ?geo)
( concat( ?article, ?to, '[[File:Wikidata-logo S.svg|16px|link=d:', substr(str(?id),32,13), ']]',  coalesce(SAMPLE(?idBild),'') ) as ?description )
('water' as ?marker_symbol) ('small' as ?marker_size) 
?article 
#?osmLink 
WITH
{ SELECT ?id WHERE 
         { bind ('none' as ?cacheInvalidator) ?id wdt:P403? ?target. values ?target { wd:Q3411 }. ?id wdt:P31/wdt:P279* ?obj. 
                         hint:Prior hint:gearing 'reverse'. 
                         values ?obj { wd:Q355304 }. ?id wdt:P625 []. 
          values ?id {wd:Q22581036}. # <--- to select a faulty case
         }
} AS %sub 
WHERE { INCLUDE %sub. ?id wdt:P625 ?geo. 
       OPTIONAL { ?id wdt:P18 ?img. bind(concat('[[File:', substr(str(?img), 52, 400), '|250px]]') as ?idBild) } 
       OPTIONAL { ?link schema:about ?id ; schema:isPartOf <https://de.wikipedia.org/> ; schema:name ?article_name . } 
       SERVICE wikibase:label { bd:serviceParam wikibase:language 'de,mul'. ?id rdfs:label ?idLabel. } 
       
       bind( coalesce( concat('[[:de:', ?article_name, '|', ?idLabel, ']]'), concat('[[:de:', ?article_name, '|', ?article_name, ']]'), ?idLabel, '' ) as ?article ) ?id wdt:P403 ?toRiver. 
       OPTIONAL { ?triverlink schema:about ?toRiver ; schema:isPartOf <https://de.wikipedia.org/> ; schema:name ?toRiver_name . } 
       SERVICE wikibase:label { bd:serviceParam wikibase:language 'de,mul'. ?toRiver rdfs:label ?toRiverLabel.} 
       
      OPTIONAL { ?id wdt:P402 ?osmRel. } #bind( coalesce( concat('[[File:Openstreetmap logo.svg|16px|link=https://www.openstreetmap.org/relation/', ?osmRel, '|auf OSM]]'), ' ' ) as ?osmLink ) 
       bind( coalesce( concat(' ↝ [[:de:', ?toRiver_name, '|', ?toRiverLabel, ']]'), concat(' ↝ [[:de:', ?toRiver_name, ']]'), concat(' ↝ ', ?toRiverLabel), ' ' ) as ?to ) 
      }
GROUP BY ?id ?idLabel ?geo ?article ?idBild ?to #?osmLink

Try it!

and

SELECT DISTINCT ?id ?idLabel (sample(?geo) as ?geo)
( concat( ?article, ?to, '[[File:Wikidata-logo S.svg|16px|link=d:', substr(str(?id),32,13), ']]',  coalesce(SAMPLE(?idBild),'') ) as ?description )
('water' as ?marker_symbol) ('small' as ?marker_size) 
?article 
#?osmLink 
WITH
{ SELECT ?id WHERE 
         { bind ('none' as ?cacheInvalidator) ?id wdt:P403? ?target. values ?target { wd:Q3411 }. ?id wdt:P31/wdt:P279* ?obj. 
                         hint:Prior hint:gearing 'reverse'. 
                         values ?obj { wd:Q355304 }. ?id wdt:P625 []. 
          values ?id {wd:Q22581036}. # <--- to select a faulty case
         }
} AS %sub 
WHERE { INCLUDE %sub. ?id wdt:P625 ?geo. 
       OPTIONAL { ?id wdt:P18 ?img. bind(concat('[[File:', substr(str(?img), 52, 400), '|250px]]') as ?idBild) } 
       OPTIONAL { ?link schema:about ?id ; schema:isPartOf <https://de.wikipedia.org/> ; schema:name ?article_name . } 
       SERVICE wikibase:label { bd:serviceParam wikibase:language 'de,mul'. ?id rdfs:label ?idLabel. } 
       
       bind( coalesce( concat('[[:de:', ?article_name, '|', ?idLabel, ']]'), concat('[[:de:', ?article_name, '|', ?article_name, ']]'), ?idLabel, '' ) as ?article ) ?id wdt:P403 ?toRiver. 
       OPTIONAL { ?triverlink schema:about ?toRiver ; schema:isPartOf <https://de.wikipedia.org/> ; schema:name ?toRiver_name . } 
       SERVICE wikibase:label { bd:serviceParam wikibase:language 'de,mul'. ?toRiver rdfs:label ?toRiverLabel.} 
       
      #OPTIONAL { ?id wdt:P402 ?osmRel. } #bind( coalesce( concat('[[File:Openstreetmap logo.svg|16px|link=https://www.openstreetmap.org/relation/', ?osmRel, '|auf OSM]]'), ' ' ) as ?osmLink ) 
       bind( coalesce( concat(' ↝ [[:de:', ?toRiver_name, '|', ?toRiverLabel, ']]'), concat(' ↝ [[:de:', ?toRiver_name, ']]'), concat(' ↝ ', ?toRiverLabel), ' ' ) as ?to ) 
      }
GROUP BY ?id ?idLabel ?geo ?article ?idBild ?to #?osmLink

Try it!

The only difference is in the statement OPTIONAL { ?id wdt:P402 ?osmRel. }. The difference in result is in cols description and article. Don’t understand how retrieving P402 changes the value of article (and thus of description). (Hope the example is prettyprinted enough, tried to condense the query to less, but failed). best –Herzi Pinki (talk) 12:08, 2 May 2026 (UTC)

SELECT DISTINCT ?id ?idLabel (sample(?geo) as ?geo)
( concat( ?article, ?to, '[[File:Wikidata-logo S.svg|16px|link=d:', substr(str(?id),32,13), ']]',  coalesce(SAMPLE(?idBild),'') ) as ?description )
('water' as ?marker_symbol) ('small' as ?marker_size) 
?article 
#?osmLink 
WITH
{ SELECT ?id WHERE 
         { bind ('none' as ?cacheInvalidator) ?id wdt:P403? ?target. values ?target { wd:Q3411 }. ?id wdt:P31/wdt:P279* ?obj. 
                         hint:Prior hint:gearing 'reverse'. 
                         values ?obj { wd:Q355304 }. ?id wdt:P625 []. 
          values ?id {wd:Q22581036}. # <--- to select a faulty case
         }
} AS %sub 
WHERE { INCLUDE %sub. ?id wdt:P625 ?geo. 
       OPTIONAL { ?id wdt:P402 ?osmRel. } # <----- moved up
       OPTIONAL { ?id wdt:P18 ?img. bind(concat('[[File:', substr(str(?img), 52, 400), '|250px]]') as ?idBild) } 
       OPTIONAL { ?link schema:about ?id ; schema:isPartOf <https://de.wikipedia.org/> ; schema:name ?article_name . } 
       SERVICE wikibase:label { bd:serviceParam wikibase:language 'de,mul'. ?id rdfs:label ?idLabel. } 
       
       bind( coalesce( concat('[[:de:', ?article_name, '|', ?idLabel, ']]'), concat('[[:de:', ?article_name, '|', ?article_name, ']]'), ?idLabel, '' ) as ?article ) ?id wdt:P403 ?toRiver. 
       OPTIONAL { ?triverlink schema:about ?toRiver ; schema:isPartOf <https://de.wikipedia.org/> ; schema:name ?toRiver_name . } 
       SERVICE wikibase:label { bd:serviceParam wikibase:language 'de,mul'. ?toRiver rdfs:label ?toRiverLabel.} 
       
      #bind( coalesce( concat('[[File:Openstreetmap logo.svg|16px|link=https://www.openstreetmap.org/relation/', ?osmRel, '|auf OSM]]'), ' ' ) as ?osmLink ) 
       bind( coalesce( concat(' ↝ [[:de:', ?toRiver_name, '|', ?toRiverLabel, ']]'), concat(' ↝ [[:de:', ?toRiver_name, ']]'), concat(' ↝ ', ?toRiverLabel), ' ' ) as ?to ) 
      }
GROUP BY ?id ?idLabel ?geo ?article ?idBild ?to #?osmLink

Try it!

Moving the statement OPTIONAL { ?id wdt:P402 ?osmRel. } up renders expected results. I would expect that the order of (unrelated) triples does not effect the result. best --Herzi Pinki (talk) 16:57, 3 May 2026 (UTC)

Suchplattform-Team der WMF? lg --Herzi Pinki (talk) 15:00, 14 May 2026 (UTC)

The problem is that the query builds ?article text out of ?idLabel (the item's label). And ?idLabel doesn't come from the data directly — it comes from the label service. The label service (usually) binds its variables after everything else in the processing block runs. So, "SERVICE wikibase:label { bd:serviceParam wikibase:language 'de,mul'. ?id rdfs:label ?idLabel. } " used ?idLabel before it had a value. Whether it ever worked always depended on luck. Adding the OPTIONAL changed the query plan - so that ?idLabel didn't have a value.

Change the SERVICE line to:

OPTIONAL { ?id rdfs:label ?idLabel_de FILTER(lang(?idLabel_de) = 'de') }

OPTIONAL { ?id rdfs:label ?idLabel_mul FILTER(lang(?idLabel_mul) = 'mul') }

bind( coalesce(?idLabel_de, ?idLabel_mul) as ?idLabel )

And it should work.

BTW, the ?toRiverLabel has the same potential problem. AWesterinen-WMF (talk) 14:59, 22 May 2026 (UTC)

Thanks @AWesterinen-WMF:, although I just accept and do not fully understand your explanation (I do not see how my naive approach can be avoided in the wild by others), the change you proposed works well. --Herzi Pinki (talk) 12:01, 24 May 2026 (UTC)

@AWesterinen-WMF:, to mimic the behaviour of the label service also for cases where neither de nor mul is given, we have to use:

  OPTIONAL { ?id rdfs:label ?idLabel_de FILTER(lang(?idLabel_de) = 'de') }
  OPTIONAL { ?id rdfs:label ?idLabel_mul FILTER(lang(?idLabel_mul) = 'mul') }
  bind( coalesce(?idLabel_de, ?idLabel_mul, STRAFTER(STR(?id), 'entity/')) as ?idLabel )

--Herzi Pinki (talk) 12:42, 24 May 2026 (UTC)

Aku mau liat kejujurannya, selama ini di tipu tipu ~2026-39617-37 (talk) 20:57, 12 July 2026 (UTC)

@Herzi Pinki Yes, that is included in the rewrite strategy. AWesterinen-WMF (talk) 19:27, 24 July 2026 (UTC)

Scholarly chapters on wrong side of graph split

I recently encountered an issue caused by instances of scholarly chapter (Q21481766) being in the main subgraph rather than the scholarly subgraph. It turns out that this class isn't part of Wikidata:SPARQL query service/WDQS graph split/Rules § Scholarly Articles. Could it be added? jlwoodwa (talk) 04:00, 4 May 2026 (UTC)

Slowdowns in WDQS today

Tracked in Phabricator
Task T425861

Experiencing slowdowns in WDQS queries today. The queries were submitted from the normal Web UI. It shouldn't take 65 seconds to fetch a list of cats. I'm also occasionally getting HTTP 502 "Bad gateway" errors from nginx. ~2026-28094-94 (talk) 18:16, 9 May 2026 (UTC)

High maxlag 2026-07-15

High maxlag today (~5 mins) preventing queries, bots, etc. from working: https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts?viewPanel=panel-12 Yirba (talk) 16:01, 15 July 2026 (UTC)

It's been very high again over the last 24 hours. When lag is queried through the API[1] it's showing values that stop compliant bots from working. William Avery (talk) 12:56, 28 July 2026 (UTC)