Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/source/architecture/events.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ OSHConnect has two pub/sub layers and they're easy to confuse:
`:data/<format>` topics on the OSH server's MQTT broker (e.g.
`…/observations:data/swe-binary`); ControlStreams publish commands to
the matching `…/commands:data/<format>` topic and receive status on
`…/status:data/json`. The hyphen-token format subtopic per CS API
`…/commands/{cmdId}/status:data`. Topics are nested under
`systems/{sysId}` by default. The hyphen-token format subtopic per CS API
Part 3 §"Resource Data Messages Content Negotiation" — see the
tutorial's *MQTT topic conventions* section for the full mapping.
Implemented via `paho-mqtt` in `csapi4py/mqtt.py`.
Expand Down
5 changes: 3 additions & 2 deletions docs/source/architecture/insertion.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,9 @@ above. Differences:
with `params_schema`.
- The endpoint is `/systems/{system_id}/controlstreams` instead of
`/datastreams`.
- The wrapper class produced is `ControlStream`, with a `_status_topic`
computed alongside the regular command topic during construction.
- The wrapper class produced is `ControlStream`. Its status topic is
per command with the default topic scheme
(`get_mqtt_status_topic(command_id)`).

Otherwise the dump → POST → `Location` header → ID-capture chain is
identical.
Expand Down
18 changes: 13 additions & 5 deletions docs/source/guides/commanding.rst
Original file line number Diff line number Diff line change
Expand Up @@ -131,17 +131,25 @@ section) to see whether the system accepted and executed it.

Subscribing to Command Status
-----------------------------
Each control stream exposes two MQTT topics: ``/commands:data/<format>``
(input — the operator publishes here) and ``/status:data/json`` (output —
the system reports execution results here). See the format-token table
in :doc:`consuming`. Subscribe to status updates:
Each control stream has a command topic, ``…/commands:data/<format>``
(input — the operator publishes here), and a status topic per command,
``…/commands/{cmdId}/status:data`` (output — the system reports execution
results here). See the format-token table in :doc:`consuming`. Subscribe to
status updates with the id of a command you sent (from the ``Location``
header of the HTTP response):

.. code-block:: python

def on_status(client, userdata, msg):
print(f"Status on {msg.topic}: {msg.payload}")

control_stream.subscribe(topic='status', callback=on_status)
control_stream.subscribe(topic='Status', callback=on_status,
command_id=command_id)

The server currently delivers the status of every command on the control
stream on that topic, so check ``command@id`` in the payload. With the
``"part3-flat"`` and ``"legacy"`` topic schemes (see :doc:`connecting`) there
is one status topic per control stream and ``command_id`` is not needed.

Inbound status reports are also pushed onto an internal deque — drain it
exactly like a datastream's inbound queue:
Expand Down
50 changes: 43 additions & 7 deletions docs/source/guides/connecting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,20 +49,56 @@ To connect a node with MQTT support for streaming:
enable_mqtt=True, mqtt_port=1883)
app.add_node(node)

To talk to an **older OSH server** that predates the CS API Part 3 topic
scheme, enable legacy topics. This reverts MQTT topic construction to the
pre-Part-3 form — a leading slash and no ``:data`` suffix or format
subtopic (e.g. ``/api/datastreams/{id}/observations`` instead of
``api/datastreams/{id}/observations:data/<token>``):
MQTT topic schemes
~~~~~~~~~~~~~~~~~~
``mqtt_topic_scheme`` selects how MQTT topics are built. Pick the one your
server's MQTT module uses:

.. list-table::
:header-rows: 1
:widths: 16 50 34

* - Scheme
- Observation topic
- OSH module
* - ``"part3"`` (default)
- ``api/systems/{sysId}/datastreams/{dsId}/observations:data/<token>``
- ``sensorhub-service-csapi-mqtt``
* - ``"part3-flat"``
- ``api/datastreams/{dsId}/observations:data/<token>``
- builds that predate the system path
* - ``"legacy"``
- ``/api/datastreams/{dsId}/observations``
- ``sensorhub-service-consys-mqtt``

.. code-block:: python

node = Node(protocol='http', address='localhost', port=8585,
username='test', password='test',
enable_mqtt=True, mqtt_legacy_topics=True)
enable_mqtt=True, mqtt_topic_scheme='legacy')
app.add_node(node)

Legacy mode affects MQTT topics only; NATS subjects are unaffected.
``mqtt_legacy_topics=True`` still works and means ``mqtt_topic_scheme='legacy'``.
The first topic level is ``api_root`` unless you set ``mqtt_topic_root`` to the
server's ``nodeId``. The scheme is saved with the node configuration and
affects MQTT topics only; NATS subjects are unaffected.

.. admonition:: Upgrading from 0.5.5 or earlier

The default changed from the flat form to ``"part3"``.

- If your server still uses flat ``:data`` topics, pass
``mqtt_topic_scheme='part3-flat'`` to keep the old behaviour.
- With ``"part3"``, command status is per command: call
``control_stream.subscribe(topic='Status', command_id=...)``.
There is no per-control-stream status topic.
- With ``"part3"``, status cannot be published over MQTT
(``publish_status`` raises ``ValueError``); report it over HTTP at
``/controlstreams/{id}/status``.
- A configuration saved by an earlier version has no scheme and loads as
``"part3"``.
- A datastream or control stream needs its parent system id. Streams from
discovery have it; for hand-built ones call ``set_parent_resource_id``.

To stream over **NATS.io** instead (the corporate-bus transport, served by
OSH's ``sensorhub-service-consys-nats`` binding), enable it the same way —
Expand Down
39 changes: 29 additions & 10 deletions docs/source/guides/consuming.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,22 @@ MQTT Topic Conventions
----------------------
OSHConnect speaks the CS API Part 3 pub/sub conventions, including the
optional **format subtopic** that selects the wire format for each
Resource Data Topic. A subscription path looks like::
Resource Data Topic. Datastream and control stream topics are nested under
their system. A subscription path looks like::

{mqtt_root}/datastreams/{ds_id}/observations:data/<format-token>
{mqtt_root}/controlstreams/{cs_id}/commands:data/<format-token>
{mqtt_root}/controlstreams/{cs_id}/status:data/json
{mqtt_root}/systems/{sys_id}/datastreams/{ds_id}/observations:data/<format-token>
{mqtt_root}/systems/{sys_id}/controlstreams/{cs_id}/commands:data/<format-token>
{mqtt_root}/systems/{sys_id}/controlstreams/{cs_id}/commands/{cmd_id}/status:data

``{mqtt_root}`` is the server's ``nodeId`` (default ``api``); set it with
``Node(mqtt_topic_root=...)``. The system id comes from discovery: streams
found through ``System.discover_datastreams`` / ``discover_controlstreams``
already know their parent. A stream built by hand needs
``set_parent_resource_id(system_id)`` before ``initialize()``, otherwise
topic construction raises ``ValueError``.

Servers that use other topic forms are selected with
``Node(mqtt_topic_scheme=...)`` — see :doc:`connecting`.

The trailing ``<format-token>`` is the hyphen-substituted MIME subtype
(``+`` is reserved as an MQTT wildcard and is disallowed in Kafka topic
Expand All @@ -65,19 +76,27 @@ Content-type Topic token
``application/swe+csv`` ``swe-csv``
``application/om+json`` ``om-json``
``application/sml+json`` ``sml-json``
``application/sml+xml`` ``sml-xml``
``application/geo+json`` ``geo-json``
``application/swe+xml`` ``swe-xml``
============================ ======================

Any other type, such as a custom format a node lists in a stream's
``formats``, gets its token by the same rule: drop the parameters and the
leading ``application/``, replace ``+`` with ``-``, lowercase
(``application/x-custom+json`` → ``x-custom-json``). A type whose name would
still contain ``/`` (``text/csv``) has no subtopic.

The Python client picks the right token for you. ``Datastream.init_mqtt``
reads the discovered ``record_schema.obs_format`` (e.g.
``application/swe+binary`` for video datastreams) and appends
``/swe-binary`` to the data topic. ``ControlStream.init_mqtt`` does the
same with ``command_schema.command_format``, and the status topic is
always suffixed with ``/json`` since status payloads are JSON by
convention. If you build a topic manually via
same with ``command_schema.command_format``. The status topic has no
format subtopic (status is JSON). If you build a topic manually via
``APIHelper.get_mqtt_topic`` you can pass ``format=...`` explicitly; an
unknown MIME type raises ``ValueError`` from
``oshconnect.csapi4py.mqtt.mqtt_topic_format_token`` so the client never
sends a token the server can't parse.
MIME type with no valid token raises ``ValueError`` from
``oshconnect.csapi4py.mqtt.mqtt_topic_format_token``. The server rejects a
token for a format the stream does not offer.

Older servers that don't recognise the format subtopic still accept the
bare ``:data`` form — that's what ``init_mqtt`` produces when a
Expand Down
2 changes: 2 additions & 0 deletions examples/AXIS_MQTT_STREAM_README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@ The demo is read-mostly: it subscribes to a video datastream over MQTT and
3. The server must support the **`:data/<format>` format subtopic** added in
CS API Part 3 — the demo subscribes to
`…/datastreams/<id>/observations:data/swe-binary`, not bare `:data`.
The demo pins `mqtt_topic_scheme="part3-flat"` (topics without the system
path), because it relies on the per-control-stream status topic.
4. The datastream must actually be **producing observations** — i.e. the
camera/RTP feed is connected and frames are buffered. An idle datastream
discovers fine but the panel stays on "waiting for first frame".
Expand Down
3 changes: 3 additions & 0 deletions examples/axis_video_mqtt_stream.py
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,9 @@ def connect_and_discover() -> tuple[OSHConnect, Node]:
password=PASS,
enable_mqtt=True,
mqtt_port=MQTT_PORT,
# This demo uses the per-control-stream status topic and flat
# ``:data`` topics, which the default "part3" scheme does not have.
mqtt_topic_scheme="part3-flat",
)
osh.add_node(node)
osh.discover_systems()
Expand Down
5 changes: 4 additions & 1 deletion examples/swe_proto_node_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -703,7 +703,10 @@ def main() -> int:
try:
node = Node(protocol="http", address=HOST, port=HTTP_PORT,
username=USER, password=PASS,
enable_mqtt=MQTT_SECS > 0, mqtt_port=MQTT_PORT)
enable_mqtt=MQTT_SECS > 0, mqtt_port=MQTT_PORT,
# The swe+proto nodes this check targets use flat
# ``:data`` topics (no system path).
mqtt_topic_scheme="part3-flat")
except OSError as exc:
if MQTT_SECS <= 0:
raise
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "oshconnect"
version = "0.5.5a2"
version = "0.5.6a0"
description = "Library for interfacing with OSH, helping guide visualization efforts, and providing a place to store configurations. Implements OGC CS API Part 3 (Pub/Sub) MQTT topic conventions including :data topics and resource event topics."
readme = "README.md"
authors = [
Expand Down
93 changes: 88 additions & 5 deletions src/oshconnect/csapi4py/default_api_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@

from .con_sys_api import DeleteRequest, GetRequest, PostRequest, PutRequest
from .constants import APIResourceTypes, ContentTypes, APITerms
from .mqtt import mqtt_topic_format_token
from .mqtt import MQTT_TOPIC_SCHEMES, TOPIC_SCHEME_LEGACY, TOPIC_SCHEME_PART3, mqtt_topic_format_token
from ..exceptions import ConfigurationError


# TODO: rework to make the first resource in the endpoint the primary key for URL construction, currently, the implementation is a bit on the confusing side with what is being generated and why.
Expand Down Expand Up @@ -100,7 +101,32 @@ class APIHelper(ABC):
# When True, build the pre-Part-3 ("legacy") MQTT topic form: a leading
# slash, no ``:data`` suffix, and no format subtopic — for talking to
# older OSH servers that predate the CS API Part 3 topic scheme.
# Kept for backwards compatibility; equivalent to ``topic_scheme="legacy"``.
legacy_topics: bool = False
# One of ``MQTT_TOPIC_SCHEMES``. ``None`` means ``"legacy"`` when
# ``legacy_topics`` is set, else ``"part3"``.
topic_scheme: str = None

def __post_init__(self):
self.get_topic_scheme()

def get_topic_scheme(self) -> str:
"""Return the MQTT topic scheme in effect: ``"part3"`` (default),
``"part3-flat"`` or ``"legacy"``.

:raises ConfigurationError: if ``topic_scheme`` is not a known scheme,
or if it contradicts ``legacy_topics=True``.
"""
if self.topic_scheme is None:
return TOPIC_SCHEME_LEGACY if self.legacy_topics else TOPIC_SCHEME_PART3
if self.topic_scheme not in MQTT_TOPIC_SCHEMES:
raise ConfigurationError(
f"Unknown MQTT topic scheme {self.topic_scheme!r}; expected one of {MQTT_TOPIC_SCHEMES}.")
if self.legacy_topics and self.topic_scheme != TOPIC_SCHEME_LEGACY:
raise ConfigurationError(
f"legacy_topics=True conflicts with topic_scheme={self.topic_scheme!r}; "
f"set only one of them.")
return self.topic_scheme

def get_mqtt_root(self) -> str:
"""
Expand Down Expand Up @@ -296,9 +322,15 @@ def set_protocol(self, protocol: str):

# TODO: add validity checking for resource type combinations
def get_mqtt_topic(self, resource_type, subresource_type, resource_id: str, subresource_id: str = None,
data_topic: bool = True, format: str | None = None, legacy: bool | None = None):
data_topic: bool = True, format: str | None = None, legacy: bool | None = None,
system_path: str | list[str] | None = None):
"""
Returns the MQTT topic for the resource type, does not check for validity of the resource type combination
Returns the MQTT topic for the resource type, does not check for validity of the resource type combination.

With the default ``"part3"`` scheme, datastream and control stream topics are nested under the system
path, e.g. ``api/systems/{sysId}/datastreams/{dsId}/observations:data``, and the command status topic is
``…/controlstreams/{csId}/commands/{cmdId}/status:data`` (``subresource_id`` is the command id).
With ``"part3-flat"`` the system path is left out; with ``"legacy"`` the topic mirrors the HTTP path.
:param resource_type: The API resource type of the resource that comes first in the URL, cannot be None
:param subresource_type: The API resource type of the sub-resource that comes second in the URL, optional if there
is no sub-resource.
Expand All @@ -314,10 +346,24 @@ def get_mqtt_topic(self, resource_type, subresource_type, resource_id: str, subr
:func:`oshconnect.csapi4py.mqtt.mqtt_topic_format_token`.
:param legacy: Force the pre-Part-3 topic form (leading slash, no ``:data`` suffix, no format subtopic).
``None`` (default) uses this helper's ``legacy_topics`` setting; pass ``True``/``False`` to override per-call.
In legacy mode ``data_topic`` and ``format`` are ignored.
In legacy mode ``data_topic`` and ``format`` are ignored. ``legacy=False`` on a helper whose scheme is
``"legacy"`` builds the ``"part3"`` form.
:param system_path: Id of the parent system of the datastream or control stream, or the list of ids from
the top-level system down to the owning subsystem. Required by the ``"part3"`` scheme when
``resource_type`` is a datastream or control stream; ignored by the other schemes.
:raises ValueError: in the ``"part3"`` scheme, if ``system_path`` is missing for a datastream or control
stream topic, or the command id is missing for a status topic.
:return:
"""
use_legacy = self.legacy_topics if legacy is None else legacy
scheme = self.get_topic_scheme()
if legacy is True:
scheme = TOPIC_SCHEME_LEGACY
elif legacy is False and scheme == TOPIC_SCHEME_LEGACY:
scheme = TOPIC_SCHEME_PART3
use_legacy = scheme == TOPIC_SCHEME_LEGACY
if scheme == TOPIC_SCHEME_PART3:
return self._part3_mqtt_topic(resource_type, subresource_type, resource_id, subresource_id,
data_topic, format, system_path)
subresource_endpoint = f'/{resource_type_to_endpoint(subresource_type)}'
resource_endpoint = "" if resource_type is None else f'/{resource_type_to_endpoint(resource_type)}'
resource_ident = "" if resource_id is None else f'/{resource_id}'
Expand All @@ -331,6 +377,43 @@ def get_mqtt_topic(self, resource_type, subresource_type, resource_id: str, subr
topic_locator = f'{self.get_mqtt_root()}{resource_endpoint}{resource_ident}{subresource_endpoint}{data_suffix}{subresource_ident}'
return topic_locator

def _part3_mqtt_topic(self, resource_type, subresource_type, resource_id, subresource_id,
data_topic, format, system_path) -> str:
"""Build a CS API Part 3 topic: ``{root}/{systemPath}/…[:data[/{token}]]``."""
parts = [self.get_mqtt_root()]
if resource_type in (APIResourceTypes.DATASTREAM, APIResourceTypes.CONTROL_CHANNEL):
path = [system_path] if isinstance(system_path, str) else list(system_path or [])
if not path or not all(path):
raise ValueError(
f"Cannot build the MQTT topic for {resource_type.value} {resource_id!r}: the parent system id "
f"is unknown. Part 3 topics are nested under 'systems/{{id}}'. Discover the resource through "
f"its System, or call set_parent_resource_id().")
parts += [resource_type_to_endpoint(APIResourceTypes.SYSTEM), path[0]]
for subsystem_id in path[1:]:
parts += ['subsystems', subsystem_id]
if resource_type is not None:
parts.append(resource_type_to_endpoint(resource_type))
if resource_id is not None:
parts.append(resource_id)
if resource_type is APIResourceTypes.CONTROL_CHANNEL and subresource_type is APIResourceTypes.STATUS:
# Status is published per command: …/commands/{cmdId}/status
if subresource_id is None:
raise ValueError(
"Cannot build the command status topic: Part 3 status topics are per command "
"('…/commands/{cmdId}/status:data'). Pass the command id.")
parts += [resource_type_to_endpoint(APIResourceTypes.COMMAND), subresource_id,
resource_type_to_endpoint(APIResourceTypes.STATUS)]
else:
parts.append(resource_type_to_endpoint(subresource_type))
if subresource_id is not None:
parts.append(subresource_id)
topic = '/'.join(str(p) for p in parts)
if data_topic:
topic += ':data'
if format is not None:
topic += f'/{mqtt_topic_format_token(format)}'
return topic


@dataclass(kw_only=True)
class ResponseParserHelper:
Expand Down
Loading
Loading