diff --git a/docs/source/architecture/events.md b/docs/source/architecture/events.md index 33518c1..1029c1f 100644 --- a/docs/source/architecture/events.md +++ b/docs/source/architecture/events.md @@ -6,7 +6,8 @@ OSHConnect has two pub/sub layers and they're easy to confuse: `:data/` topics on the OSH server's MQTT broker (e.g. `…/observations:data/swe-binary`); ControlStreams publish commands to the matching `…/commands:data/` topic and receive status on - `…/status:data/json`. The hyphen-token format subtopic per CS API + `…/commands/{cmdId}/status:data`. Topics are nested under + `systems/{sysId}` by default. The hyphen-token format subtopic per CS API Part 3 §"Resource Data Messages Content Negotiation" — see the tutorial's *MQTT topic conventions* section for the full mapping. Implemented via `paho-mqtt` in `csapi4py/mqtt.py`. diff --git a/docs/source/architecture/insertion.md b/docs/source/architecture/insertion.md index 246ad94..a5cd67b 100644 --- a/docs/source/architecture/insertion.md +++ b/docs/source/architecture/insertion.md @@ -91,8 +91,9 @@ above. Differences: with `params_schema`. - The endpoint is `/systems/{system_id}/controlstreams` instead of `/datastreams`. -- The wrapper class produced is `ControlStream`, with a `_status_topic` - computed alongside the regular command topic during construction. +- The wrapper class produced is `ControlStream`. Its status topic is + per command with the default topic scheme + (`get_mqtt_status_topic(command_id)`). Otherwise the dump → POST → `Location` header → ID-capture chain is identical. diff --git a/docs/source/guides/commanding.rst b/docs/source/guides/commanding.rst index 9fc5894..428ff06 100644 --- a/docs/source/guides/commanding.rst +++ b/docs/source/guides/commanding.rst @@ -131,17 +131,25 @@ section) to see whether the system accepted and executed it. Subscribing to Command Status ----------------------------- -Each control stream exposes two MQTT topics: ``/commands:data/`` -(input — the operator publishes here) and ``/status:data/json`` (output — -the system reports execution results here). See the format-token table -in :doc:`consuming`. Subscribe to status updates: +Each control stream has a command topic, ``…/commands:data/`` +(input — the operator publishes here), and a status topic per command, +``…/commands/{cmdId}/status:data`` (output — the system reports execution +results here). See the format-token table in :doc:`consuming`. Subscribe to +status updates with the id of a command you sent (from the ``Location`` +header of the HTTP response): .. code-block:: python def on_status(client, userdata, msg): print(f"Status on {msg.topic}: {msg.payload}") - control_stream.subscribe(topic='status', callback=on_status) + control_stream.subscribe(topic='Status', callback=on_status, + command_id=command_id) + +The server currently delivers the status of every command on the control +stream on that topic, so check ``command@id`` in the payload. With the +``"part3-flat"`` and ``"legacy"`` topic schemes (see :doc:`connecting`) there +is one status topic per control stream and ``command_id`` is not needed. Inbound status reports are also pushed onto an internal deque — drain it exactly like a datastream's inbound queue: diff --git a/docs/source/guides/connecting.rst b/docs/source/guides/connecting.rst index deab810..8a9a544 100644 --- a/docs/source/guides/connecting.rst +++ b/docs/source/guides/connecting.rst @@ -49,20 +49,56 @@ To connect a node with MQTT support for streaming: enable_mqtt=True, mqtt_port=1883) app.add_node(node) -To talk to an **older OSH server** that predates the CS API Part 3 topic -scheme, enable legacy topics. This reverts MQTT topic construction to the -pre-Part-3 form — a leading slash and no ``:data`` suffix or format -subtopic (e.g. ``/api/datastreams/{id}/observations`` instead of -``api/datastreams/{id}/observations:data/``): +MQTT topic schemes +~~~~~~~~~~~~~~~~~~ +``mqtt_topic_scheme`` selects how MQTT topics are built. Pick the one your +server's MQTT module uses: + +.. list-table:: + :header-rows: 1 + :widths: 16 50 34 + + * - Scheme + - Observation topic + - OSH module + * - ``"part3"`` (default) + - ``api/systems/{sysId}/datastreams/{dsId}/observations:data/`` + - ``sensorhub-service-csapi-mqtt`` + * - ``"part3-flat"`` + - ``api/datastreams/{dsId}/observations:data/`` + - builds that predate the system path + * - ``"legacy"`` + - ``/api/datastreams/{dsId}/observations`` + - ``sensorhub-service-consys-mqtt`` .. code-block:: python node = Node(protocol='http', address='localhost', port=8585, username='test', password='test', - enable_mqtt=True, mqtt_legacy_topics=True) + enable_mqtt=True, mqtt_topic_scheme='legacy') app.add_node(node) -Legacy mode affects MQTT topics only; NATS subjects are unaffected. +``mqtt_legacy_topics=True`` still works and means ``mqtt_topic_scheme='legacy'``. +The first topic level is ``api_root`` unless you set ``mqtt_topic_root`` to the +server's ``nodeId``. The scheme is saved with the node configuration and +affects MQTT topics only; NATS subjects are unaffected. + +.. admonition:: Upgrading from 0.5.5 or earlier + + The default changed from the flat form to ``"part3"``. + + - If your server still uses flat ``:data`` topics, pass + ``mqtt_topic_scheme='part3-flat'`` to keep the old behaviour. + - With ``"part3"``, command status is per command: call + ``control_stream.subscribe(topic='Status', command_id=...)``. + There is no per-control-stream status topic. + - With ``"part3"``, status cannot be published over MQTT + (``publish_status`` raises ``ValueError``); report it over HTTP at + ``/controlstreams/{id}/status``. + - A configuration saved by an earlier version has no scheme and loads as + ``"part3"``. + - A datastream or control stream needs its parent system id. Streams from + discovery have it; for hand-built ones call ``set_parent_resource_id``. To stream over **NATS.io** instead (the corporate-bus transport, served by OSH's ``sensorhub-service-consys-nats`` binding), enable it the same way — diff --git a/docs/source/guides/consuming.rst b/docs/source/guides/consuming.rst index 31813bd..c45cebf 100644 --- a/docs/source/guides/consuming.rst +++ b/docs/source/guides/consuming.rst @@ -46,11 +46,22 @@ MQTT Topic Conventions ---------------------- OSHConnect speaks the CS API Part 3 pub/sub conventions, including the optional **format subtopic** that selects the wire format for each -Resource Data Topic. A subscription path looks like:: +Resource Data Topic. Datastream and control stream topics are nested under +their system. A subscription path looks like:: - {mqtt_root}/datastreams/{ds_id}/observations:data/ - {mqtt_root}/controlstreams/{cs_id}/commands:data/ - {mqtt_root}/controlstreams/{cs_id}/status:data/json + {mqtt_root}/systems/{sys_id}/datastreams/{ds_id}/observations:data/ + {mqtt_root}/systems/{sys_id}/controlstreams/{cs_id}/commands:data/ + {mqtt_root}/systems/{sys_id}/controlstreams/{cs_id}/commands/{cmd_id}/status:data + +``{mqtt_root}`` is the server's ``nodeId`` (default ``api``); set it with +``Node(mqtt_topic_root=...)``. The system id comes from discovery: streams +found through ``System.discover_datastreams`` / ``discover_controlstreams`` +already know their parent. A stream built by hand needs +``set_parent_resource_id(system_id)`` before ``initialize()``, otherwise +topic construction raises ``ValueError``. + +Servers that use other topic forms are selected with +``Node(mqtt_topic_scheme=...)`` — see :doc:`connecting`. The trailing ```` is the hyphen-substituted MIME subtype (``+`` is reserved as an MQTT wildcard and is disallowed in Kafka topic @@ -65,19 +76,27 @@ Content-type Topic token ``application/swe+csv`` ``swe-csv`` ``application/om+json`` ``om-json`` ``application/sml+json`` ``sml-json`` +``application/sml+xml`` ``sml-xml`` +``application/geo+json`` ``geo-json`` +``application/swe+xml`` ``swe-xml`` ============================ ====================== +Any other type, such as a custom format a node lists in a stream's +``formats``, gets its token by the same rule: drop the parameters and the +leading ``application/``, replace ``+`` with ``-``, lowercase +(``application/x-custom+json`` → ``x-custom-json``). A type whose name would +still contain ``/`` (``text/csv``) has no subtopic. + The Python client picks the right token for you. ``Datastream.init_mqtt`` reads the discovered ``record_schema.obs_format`` (e.g. ``application/swe+binary`` for video datastreams) and appends ``/swe-binary`` to the data topic. ``ControlStream.init_mqtt`` does the -same with ``command_schema.command_format``, and the status topic is -always suffixed with ``/json`` since status payloads are JSON by -convention. If you build a topic manually via +same with ``command_schema.command_format``. The status topic has no +format subtopic (status is JSON). If you build a topic manually via ``APIHelper.get_mqtt_topic`` you can pass ``format=...`` explicitly; an -unknown MIME type raises ``ValueError`` from -``oshconnect.csapi4py.mqtt.mqtt_topic_format_token`` so the client never -sends a token the server can't parse. +MIME type with no valid token raises ``ValueError`` from +``oshconnect.csapi4py.mqtt.mqtt_topic_format_token``. The server rejects a +token for a format the stream does not offer. Older servers that don't recognise the format subtopic still accept the bare ``:data`` form — that's what ``init_mqtt`` produces when a diff --git a/examples/AXIS_MQTT_STREAM_README.md b/examples/AXIS_MQTT_STREAM_README.md index 165cf29..068a6c5 100644 --- a/examples/AXIS_MQTT_STREAM_README.md +++ b/examples/AXIS_MQTT_STREAM_README.md @@ -78,6 +78,8 @@ The demo is read-mostly: it subscribes to a video datastream over MQTT and 3. The server must support the **`:data/` format subtopic** added in CS API Part 3 — the demo subscribes to `…/datastreams//observations:data/swe-binary`, not bare `:data`. + The demo pins `mqtt_topic_scheme="part3-flat"` (topics without the system + path), because it relies on the per-control-stream status topic. 4. The datastream must actually be **producing observations** — i.e. the camera/RTP feed is connected and frames are buffered. An idle datastream discovers fine but the panel stays on "waiting for first frame". diff --git a/examples/axis_video_mqtt_stream.py b/examples/axis_video_mqtt_stream.py index 2c85ff1..fd7524b 100644 --- a/examples/axis_video_mqtt_stream.py +++ b/examples/axis_video_mqtt_stream.py @@ -276,6 +276,9 @@ def connect_and_discover() -> tuple[OSHConnect, Node]: password=PASS, enable_mqtt=True, mqtt_port=MQTT_PORT, + # This demo uses the per-control-stream status topic and flat + # ``:data`` topics, which the default "part3" scheme does not have. + mqtt_topic_scheme="part3-flat", ) osh.add_node(node) osh.discover_systems() diff --git a/examples/swe_proto_node_check.py b/examples/swe_proto_node_check.py index 01e7b7f..2eed3e2 100644 --- a/examples/swe_proto_node_check.py +++ b/examples/swe_proto_node_check.py @@ -703,7 +703,10 @@ def main() -> int: try: node = Node(protocol="http", address=HOST, port=HTTP_PORT, username=USER, password=PASS, - enable_mqtt=MQTT_SECS > 0, mqtt_port=MQTT_PORT) + enable_mqtt=MQTT_SECS > 0, mqtt_port=MQTT_PORT, + # The swe+proto nodes this check targets use flat + # ``:data`` topics (no system path). + mqtt_topic_scheme="part3-flat") except OSError as exc: if MQTT_SECS <= 0: raise diff --git a/pyproject.toml b/pyproject.toml index 1991817..d4cb094 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "oshconnect" -version = "0.5.5a2" +version = "0.5.6a0" description = "Library for interfacing with OSH, helping guide visualization efforts, and providing a place to store configurations. Implements OGC CS API Part 3 (Pub/Sub) MQTT topic conventions including :data topics and resource event topics." readme = "README.md" authors = [ diff --git a/src/oshconnect/csapi4py/default_api_helpers.py b/src/oshconnect/csapi4py/default_api_helpers.py index f839112..a718e0b 100644 --- a/src/oshconnect/csapi4py/default_api_helpers.py +++ b/src/oshconnect/csapi4py/default_api_helpers.py @@ -14,7 +14,8 @@ from .con_sys_api import DeleteRequest, GetRequest, PostRequest, PutRequest from .constants import APIResourceTypes, ContentTypes, APITerms -from .mqtt import mqtt_topic_format_token +from .mqtt import MQTT_TOPIC_SCHEMES, TOPIC_SCHEME_LEGACY, TOPIC_SCHEME_PART3, mqtt_topic_format_token +from ..exceptions import ConfigurationError # TODO: rework to make the first resource in the endpoint the primary key for URL construction, currently, the implementation is a bit on the confusing side with what is being generated and why. @@ -100,7 +101,32 @@ class APIHelper(ABC): # When True, build the pre-Part-3 ("legacy") MQTT topic form: a leading # slash, no ``:data`` suffix, and no format subtopic — for talking to # older OSH servers that predate the CS API Part 3 topic scheme. + # Kept for backwards compatibility; equivalent to ``topic_scheme="legacy"``. legacy_topics: bool = False + # One of ``MQTT_TOPIC_SCHEMES``. ``None`` means ``"legacy"`` when + # ``legacy_topics`` is set, else ``"part3"``. + topic_scheme: str = None + + def __post_init__(self): + self.get_topic_scheme() + + def get_topic_scheme(self) -> str: + """Return the MQTT topic scheme in effect: ``"part3"`` (default), + ``"part3-flat"`` or ``"legacy"``. + + :raises ConfigurationError: if ``topic_scheme`` is not a known scheme, + or if it contradicts ``legacy_topics=True``. + """ + if self.topic_scheme is None: + return TOPIC_SCHEME_LEGACY if self.legacy_topics else TOPIC_SCHEME_PART3 + if self.topic_scheme not in MQTT_TOPIC_SCHEMES: + raise ConfigurationError( + f"Unknown MQTT topic scheme {self.topic_scheme!r}; expected one of {MQTT_TOPIC_SCHEMES}.") + if self.legacy_topics and self.topic_scheme != TOPIC_SCHEME_LEGACY: + raise ConfigurationError( + f"legacy_topics=True conflicts with topic_scheme={self.topic_scheme!r}; " + f"set only one of them.") + return self.topic_scheme def get_mqtt_root(self) -> str: """ @@ -296,9 +322,15 @@ def set_protocol(self, protocol: str): # TODO: add validity checking for resource type combinations def get_mqtt_topic(self, resource_type, subresource_type, resource_id: str, subresource_id: str = None, - data_topic: bool = True, format: str | None = None, legacy: bool | None = None): + data_topic: bool = True, format: str | None = None, legacy: bool | None = None, + system_path: str | list[str] | None = None): """ - Returns the MQTT topic for the resource type, does not check for validity of the resource type combination + Returns the MQTT topic for the resource type, does not check for validity of the resource type combination. + + With the default ``"part3"`` scheme, datastream and control stream topics are nested under the system + path, e.g. ``api/systems/{sysId}/datastreams/{dsId}/observations:data``, and the command status topic is + ``…/controlstreams/{csId}/commands/{cmdId}/status:data`` (``subresource_id`` is the command id). + With ``"part3-flat"`` the system path is left out; with ``"legacy"`` the topic mirrors the HTTP path. :param resource_type: The API resource type of the resource that comes first in the URL, cannot be None :param subresource_type: The API resource type of the sub-resource that comes second in the URL, optional if there is no sub-resource. @@ -314,10 +346,24 @@ def get_mqtt_topic(self, resource_type, subresource_type, resource_id: str, subr :func:`oshconnect.csapi4py.mqtt.mqtt_topic_format_token`. :param legacy: Force the pre-Part-3 topic form (leading slash, no ``:data`` suffix, no format subtopic). ``None`` (default) uses this helper's ``legacy_topics`` setting; pass ``True``/``False`` to override per-call. - In legacy mode ``data_topic`` and ``format`` are ignored. + In legacy mode ``data_topic`` and ``format`` are ignored. ``legacy=False`` on a helper whose scheme is + ``"legacy"`` builds the ``"part3"`` form. + :param system_path: Id of the parent system of the datastream or control stream, or the list of ids from + the top-level system down to the owning subsystem. Required by the ``"part3"`` scheme when + ``resource_type`` is a datastream or control stream; ignored by the other schemes. + :raises ValueError: in the ``"part3"`` scheme, if ``system_path`` is missing for a datastream or control + stream topic, or the command id is missing for a status topic. :return: """ - use_legacy = self.legacy_topics if legacy is None else legacy + scheme = self.get_topic_scheme() + if legacy is True: + scheme = TOPIC_SCHEME_LEGACY + elif legacy is False and scheme == TOPIC_SCHEME_LEGACY: + scheme = TOPIC_SCHEME_PART3 + use_legacy = scheme == TOPIC_SCHEME_LEGACY + if scheme == TOPIC_SCHEME_PART3: + return self._part3_mqtt_topic(resource_type, subresource_type, resource_id, subresource_id, + data_topic, format, system_path) subresource_endpoint = f'/{resource_type_to_endpoint(subresource_type)}' resource_endpoint = "" if resource_type is None else f'/{resource_type_to_endpoint(resource_type)}' resource_ident = "" if resource_id is None else f'/{resource_id}' @@ -331,6 +377,43 @@ def get_mqtt_topic(self, resource_type, subresource_type, resource_id: str, subr topic_locator = f'{self.get_mqtt_root()}{resource_endpoint}{resource_ident}{subresource_endpoint}{data_suffix}{subresource_ident}' return topic_locator + def _part3_mqtt_topic(self, resource_type, subresource_type, resource_id, subresource_id, + data_topic, format, system_path) -> str: + """Build a CS API Part 3 topic: ``{root}/{systemPath}/…[:data[/{token}]]``.""" + parts = [self.get_mqtt_root()] + if resource_type in (APIResourceTypes.DATASTREAM, APIResourceTypes.CONTROL_CHANNEL): + path = [system_path] if isinstance(system_path, str) else list(system_path or []) + if not path or not all(path): + raise ValueError( + f"Cannot build the MQTT topic for {resource_type.value} {resource_id!r}: the parent system id " + f"is unknown. Part 3 topics are nested under 'systems/{{id}}'. Discover the resource through " + f"its System, or call set_parent_resource_id().") + parts += [resource_type_to_endpoint(APIResourceTypes.SYSTEM), path[0]] + for subsystem_id in path[1:]: + parts += ['subsystems', subsystem_id] + if resource_type is not None: + parts.append(resource_type_to_endpoint(resource_type)) + if resource_id is not None: + parts.append(resource_id) + if resource_type is APIResourceTypes.CONTROL_CHANNEL and subresource_type is APIResourceTypes.STATUS: + # Status is published per command: …/commands/{cmdId}/status + if subresource_id is None: + raise ValueError( + "Cannot build the command status topic: Part 3 status topics are per command " + "('…/commands/{cmdId}/status:data'). Pass the command id.") + parts += [resource_type_to_endpoint(APIResourceTypes.COMMAND), subresource_id, + resource_type_to_endpoint(APIResourceTypes.STATUS)] + else: + parts.append(resource_type_to_endpoint(subresource_type)) + if subresource_id is not None: + parts.append(subresource_id) + topic = '/'.join(str(p) for p in parts) + if data_topic: + topic += ':data' + if format is not None: + topic += f'/{mqtt_topic_format_token(format)}' + return topic + @dataclass(kw_only=True) class ResponseParserHelper: diff --git a/src/oshconnect/csapi4py/mqtt.py b/src/oshconnect/csapi4py/mqtt.py index 88305a5..4adf3aa 100644 --- a/src/oshconnect/csapi4py/mqtt.py +++ b/src/oshconnect/csapi4py/mqtt.py @@ -20,6 +20,8 @@ # Tokens use '-' instead of '+' because MQTT reserves '+' as a single-level # wildcard and Kafka disallows '+' in topic names. A topic of the form # `…:data/` selects the wire format for both subscribe and publish. +# Types that are not listed here (custom formats) get a token by the same +# rule; see `mqtt_topic_format_token`. MQTT_TOPIC_FORMAT_TOKENS = { "application/json": "json", "application/swe+json": "swe-json", @@ -29,26 +31,53 @@ "application/swe+flatbuffers": "swe-flatbuffers", "application/om+json": "om-json", "application/sml+json": "sml-json", + "application/sml+xml": "sml-xml", + "application/geo+json": "geo-json", + "application/swe+xml": "swe-xml", } +# MQTT topic schemes. See ``APIHelper.get_mqtt_topic``. +# +# - ``part3``: CS API Part 3 topics, nested under the system path, e.g. +# ``api/systems/{sysId}/datastreams/{dsId}/observations:data``. Matches +# ``sensorhub-service-csapi-mqtt``. This is the default. +# - ``part3-flat``: ``:data`` topics without the system path, e.g. +# ``api/datastreams/{dsId}/observations:data``. The default before 0.5.6. +# - ``legacy``: pre-Part-3 topics that mirror the HTTP path, e.g. +# ``/api/datastreams/{dsId}/observations``. Matches +# ``sensorhub-service-consys-mqtt``. +TOPIC_SCHEME_PART3 = "part3" +TOPIC_SCHEME_PART3_FLAT = "part3-flat" +TOPIC_SCHEME_LEGACY = "legacy" +MQTT_TOPIC_SCHEMES = (TOPIC_SCHEME_PART3, TOPIC_SCHEME_PART3_FLAT, TOPIC_SCHEME_LEGACY) + + def mqtt_topic_format_token(content_type: str) -> str: - """Return the hyphen-token for a CS API Part 3 ``:data/`` subtopic. + """Return the token for a CS API Part 3 ``:data/`` format subtopic. + + Known types come from :data:`MQTT_TOPIC_FORMAT_TOKENS`. Any other type, + such as a custom format a node offers in a stream's ``formats`` list, is + normalized with the Part 3 rule the server applies: drop the media-type + parameters, drop a leading ``application/``, replace ``+`` with ``-`` and + lowercase. ``application/x-thing+json; v=2`` becomes ``x-thing-json``. :param content_type: MIME type string, e.g. ``"application/swe+binary"``. - :raises ValueError: if ``content_type`` is not in - :data:`MQTT_TOPIC_FORMAT_TOKENS`. Callers must register a token for - every format they intend to stream — the server raises - ``InvalidTopicException`` on unknown subtopic tokens. + :raises ValueError: if the type has no valid token — the normalized name + is empty or contains ``/`` (for example ``text/csv``) or ``#``. Such + a format cannot have a subtopic. """ - try: - return MQTT_TOPIC_FORMAT_TOKENS[content_type] - except KeyError: + token = MQTT_TOPIC_FORMAT_TOKENS.get(content_type) + if token is not None: + return token + token = str(content_type or "").split(";", 1)[0].strip().lower() + token = token.removeprefix("application/").replace("+", "-") + if not token or "/" in token or "#" in token: raise ValueError( - f"No MQTT topic-format token registered for content-type " - f"{content_type!r}. Known content-types: " - f"{sorted(MQTT_TOPIC_FORMAT_TOKENS)}" - ) + f"No MQTT topic-format token for content-type {content_type!r}: " + f"a format subtopic is one topic level made from the media type " + f"without 'application/' and with '+' replaced by '-'.") + return token class MQTTCommClient: diff --git a/src/oshconnect/node.py b/src/oshconnect/node.py index 6e545c3..881083b 100644 --- a/src/oshconnect/node.py +++ b/src/oshconnect/node.py @@ -164,8 +164,14 @@ class Node: ``"sensorhub"``). :param api_root: Second path segment under ``server_root`` (default ``"api"``). - :param mqtt_topic_root: Override for the MQTT topic root if it diverges - from the HTTP api root (CS API Part 3 § A.1). + :param mqtt_topic_root: First level of every MQTT topic — the server's + ``nodeId``. Defaults to ``api_root``. + :param mqtt_topic_scheme: ``"part3"`` (default) nests datastream and + control stream topics under their system + (``api/systems/{sysId}/datastreams/{dsId}/observations:data``); + ``"part3-flat"`` leaves the system out (the default before 0.5.6); + ``"legacy"`` is the pre-Part-3 form (``/api/datastreams/{dsId}/observations``). + :param mqtt_legacy_topics: Same as ``mqtt_topic_scheme="legacy"``. :param session_manager: Optional `SessionManager`; if given the node registers itself and gets a fresh `OSHClientSession`. :param enable_mqtt: If True, connects an MQTT client to ``address``. @@ -191,7 +197,8 @@ def __init__(self, protocol: str, address: str, port: int, username: str = None, server_root: str = 'sensorhub', api_root: str = 'api', mqtt_topic_root: str = None, session_manager: SessionManager = None, enable_mqtt: bool = False, mqtt_port: int = 1883, enable_nats: bool = False, nats_host: str = None, nats_port: int = 4222, - nats_token: str = None, mqtt_legacy_topics: bool = False): + nats_token: str = None, mqtt_legacy_topics: bool = False, + mqtt_topic_scheme: str = None): self._id = f'node-{uuid.uuid4()}' self.protocol = protocol self.address = address @@ -216,6 +223,7 @@ def __init__(self, protocol: str, address: str, port: int, username: str = None, server_url=self.address, protocol=self.protocol, port=self.port, server_root=self.server_root, api_root=api_root, mqtt_topic_root=mqtt_topic_root, username=username, password=password, legacy_topics=mqtt_legacy_topics, + topic_scheme=mqtt_topic_scheme, ) if self.is_secure: self._api_helper.user_auth = True @@ -442,6 +450,7 @@ def to_storage_dict(self) -> dict: "server_root": self.server_root, "api_root": getattr(self._api_helper, "api_root", "api"), "mqtt_topic_root": getattr(self._api_helper, "mqtt_topic_root", None), + "mqtt_topic_scheme": self._api_helper.get_topic_scheme(), "is_secure": self.is_secure, "username": getattr(self._api_helper, "username", None), "password": getattr(self._api_helper, "password", None), @@ -500,6 +509,7 @@ def from_storage_dict(cls, data: dict, session_manager: 'SessionManager' = None) server_root=data.get("server_root", "sensorhub"), api_root=data.get("api_root", "api"), mqtt_topic_root=data.get("mqtt_topic_root"), + mqtt_topic_scheme=data.get("mqtt_topic_scheme"), ) node._id = data["_id"] node.is_secure = data.get("is_secure", False) diff --git a/src/oshconnect/resources/base.py b/src/oshconnect/resources/base.py index 774e5c5..450505a 100644 --- a/src/oshconnect/resources/base.py +++ b/src/oshconnect/resources/base.py @@ -41,7 +41,7 @@ from ..csapi4py.constants import APIResourceTypes from ..csapi4py.default_api_helpers import resource_type_to_endpoint -from ..csapi4py.mqtt import MQTTCommClient, mqtt_topic_format_token +from ..csapi4py.mqtt import MQTTCommClient, TOPIC_SCHEME_PART3, mqtt_topic_format_token from ..csapi4py.nats import NatsCommClient from ..exceptions import MissingLocationHeaderError, ResourceInsertError from ..resource_datamodels import ControlStreamResource @@ -270,13 +270,42 @@ def init_mqtt(self): self._mqtt_client.set_on_subscribe(self._default_on_subscribe) - # self.get_mqtt_topic() + def _topic_unavailable_without_client(self, error: ValueError) -> bool: + """Decide what a failed topic build means during ``init_mqtt``. + + Without a pub/sub client the topics are never used, so a resource + whose parent system is unknown must still initialize (HTTP-only use). + Returns True in that case; re-raises ``error`` when a client exists. + """ + if self._mqtt_client is not None: + raise error + logger.debug("Topics not built for %s (no pub/sub client): %s", self._id, error) + return True def _default_on_subscribe(self, client, userdata, mid, granted_qos, properties): logger.debug("OSH Subscribed: mid=%s granted_qos=%s", mid, granted_qos) + def _system_path(self) -> list[str] | None: + """Ids of the systems that own this datastream or control stream, + top-level system first. ``None`` when the parent system is unknown. + + Only one level is tracked today, so a stream of a subsystem needs the + full path passed to ``APIHelper.get_mqtt_topic`` directly. + """ + sys_id = None + if isinstance(self._underlying_resource, DatastreamResource): + # The server's own ``system@id`` wins, as in get_nats_subject. + sys_id = getattr(self._underlying_resource, "system_id", None) + sys_id = sys_id or self._parent_resource_id + return [sys_id] if sys_id else None + + def _uses_part3_mqtt_topics(self) -> bool: + """True when topics are built with the nested ``"part3"`` MQTT scheme.""" + return (not self.uses_nats() + and self._parent_node.get_api_helper().get_topic_scheme() == TOPIC_SCHEME_PART3) + def get_mqtt_topic(self, subresource: APIResourceTypes | None = None, data_topic: bool = True, - format: str | None = None): + format: str | None = None, command_id: str | None = None): """ Retrieves the MQTT topic for this streamable resource based on its underlying resource type. By default, returns a Resource Data Topic (`:data` suffix per CS API Part 3). @@ -285,10 +314,16 @@ def get_mqtt_topic(self, subresource: APIResourceTypes | None = None, data_topic Resource Event Topics. :param format: Optional MIME content-type for the ``:data/`` format subtopic. ``None`` (default) emits bare ``:data`` so the server's default format applies. Ignored when ``data_topic=False``. + :param command_id: Command id for a control stream's status topic. Required by the ``"part3"`` scheme, + where status is per command (``…/commands/{cmdId}/status:data``); ignored otherwise. + :raises ValueError: in the ``"part3"`` scheme, if the parent system id of a datastream or control stream + is unknown, or ``command_id`` is missing for a status topic. """ resource_type = None parent_res_type = None parent_id = None + subresource_id = None + api_helper = self._parent_node.get_api_helper() if isinstance(self._underlying_resource, ControlStreamResource): parent_res_type = APIResourceTypes.CONTROL_CHANNEL @@ -299,6 +334,8 @@ def get_mqtt_topic(self, subresource: APIResourceTypes | None = None, data_topic resource_type = APIResourceTypes.COMMAND case APIResourceTypes.STATUS: resource_type = APIResourceTypes.STATUS + if api_helper.get_topic_scheme() == TOPIC_SCHEME_PART3: + subresource_id = command_id elif isinstance(self._underlying_resource, DatastreamResource): parent_res_type = APIResourceTypes.DATASTREAM @@ -322,17 +359,17 @@ def get_mqtt_topic(self, subresource: APIResourceTypes | None = None, data_topic case _: raise ValueError(f"Unsupported subresource type {subresource} for SystemResource.") - topic = self._parent_node.get_api_helper().get_mqtt_topic(subresource_type=resource_type, resource_id=parent_id, - resource_type=parent_res_type, data_topic=data_topic, - format=format) + topic = api_helper.get_mqtt_topic(subresource_type=resource_type, resource_id=parent_id, + resource_type=parent_res_type, data_topic=data_topic, + format=format, subresource_id=subresource_id, + system_path=self._system_path()) return topic def uses_nats(self) -> bool: """True when this resource's active transport is NATS. - Drives the flat-vs-nested topic choice: NATS data subjects nest under - ``systems`` (and need the parent system id) where the MQTT topics are - flat. See `get_nats_subject`. + NATS subjects are built by `get_nats_subject`; the MQTT topic scheme + setting does not apply to them. """ return isinstance(self._mqtt_client, NatsCommClient) @@ -340,8 +377,8 @@ def get_stream_topic(self, subresource: APIResourceTypes | None = None, data_top format: str | None = None) -> str: """Return the data topic/subject for the *active* transport. - Dispatches to `get_nats_subject` (nested-under-systems, dot-delimited) - when NATS is active, else `get_mqtt_topic` (flat, slash-delimited). + Dispatches to `get_nats_subject` (dot-delimited) when NATS is active, + else `get_mqtt_topic` (slash-delimited, per the node's topic scheme). Subclasses call this from ``init_mqtt`` so they stay transport-agnostic. """ if self.uses_nats(): @@ -368,7 +405,7 @@ def get_nats_subject(self, subresource: APIResourceTypes | None = None, data_top format: str | None = None, format_wildcard: bool = False) -> str: """Build the CS API Part 3 NATS data subject for this resource. - Unlike the flat MQTT topics, NATS data subjects are the canonical + NATS data subjects are the canonical *nested-under-systems* resource path, dot-delimited, with a ``:data[.]`` suffix — e.g. ``api.systems.{sysId}.datastreams.{dsId}.observations:data.swe-proto``. @@ -447,14 +484,21 @@ def get_event_topic(self) -> str: (create/update/delete) published by the server. For Datastream/ControlStream, includes the parent system path when a parent resource ID is available. + + :raises ValueError: with the ``"part3"`` MQTT topic scheme, if the parent system id of a datastream or + control stream is unknown (the server only accepts the nested form). """ mqtt_root = self._parent_node.get_api_helper().get_mqtt_root() + if (self._uses_part3_mqtt_topics() and not self._system_path() + and isinstance(self._underlying_resource, (DatastreamResource, ControlStreamResource))): + raise ValueError( + f"Cannot build the event topic for {self._resource_id!r}: the parent system id is unknown. " + f"Part 3 topics are nested under 'systems/{{id}}'. Discover the resource through its System, " + f"or call set_parent_resource_id().") if isinstance(self._underlying_resource, DatastreamResource): - # Prefer the nested-under-system path (required by the NATS event - # subjects, and valid for MQTT too). Fall back to the datastream's - # own ``system@id`` when no parent id was assigned locally. - sys_id = self._parent_resource_id or getattr(self._underlying_resource, "system_id", None) + # Nested under the system, resolved as for the data topics. + sys_id = (self._system_path() or [None])[0] if sys_id: return f'{mqtt_root}/systems/{sys_id}/datastreams/{self._resource_id}' return f'{mqtt_root}/datastreams/{self._resource_id}' diff --git a/src/oshconnect/resources/controlstream.py b/src/oshconnect/resources/controlstream.py index 5f19a1d..40bd320 100644 --- a/src/oshconnect/resources/controlstream.py +++ b/src/oshconnect/resources/controlstream.py @@ -84,23 +84,49 @@ def init_mqtt(self): super().init_mqtt() schema = getattr(self._underlying_resource, "command_schema", None) cmd_format = getattr(schema, "command_format", None) if schema is not None else None - self._topic = self.get_stream_topic(subresource=APIResourceTypes.COMMAND, - data_topic=True, format=cmd_format) - # Parent system id is resolved by now (via discovery or - # set_parent_resource_id), so a NATS status subject can be built. - self._status_topic = self.get_mqtt_status_topic() - # Publish uses the exact-format topics above; subscribe uses - # format-wildcard subjects over NATS (server picks the proactive format). - self._subscribe_topic = self.get_subscribe_topic( - subresource=APIResourceTypes.COMMAND, format=cmd_format) - self._status_subscribe_topic = self.get_subscribe_topic( - subresource=APIResourceTypes.STATUS, format="application/json") - - def get_mqtt_status_topic(self) -> str: - """Return the topic/subject for command status updates. Status payloads - are always ``application/json``, so it is suffixed with the ``json`` - format subtopic (``…/status:data/json`` for MQTT, or the nested - ``…commands... status:data.json`` equivalent for NATS).""" + try: + self._topic = self.get_stream_topic(subresource=APIResourceTypes.COMMAND, + data_topic=True, format=cmd_format) + # Publish uses the exact-format topics above; subscribe uses + # format-wildcard subjects over NATS (server picks the proactive format). + self._subscribe_topic = self.get_subscribe_topic( + subresource=APIResourceTypes.COMMAND, format=cmd_format) + if self._uses_part3_mqtt_topics(): + # Status topics are per command; see get_mqtt_status_topic(command_id). + self._status_topic = None + self._status_subscribe_topic = None + else: + # Parent system id is resolved by now (via discovery or + # set_parent_resource_id), so a NATS status subject can be built. + self._status_topic = self.get_mqtt_status_topic() + self._status_subscribe_topic = self.get_subscribe_topic( + subresource=APIResourceTypes.STATUS, format="application/json") + except ValueError as e: + if self._topic_unavailable_without_client(e): + self._topic = None + self._subscribe_topic = None + self._status_topic = None + self._status_subscribe_topic = None + + def get_mqtt_status_topic(self, command_id: str | None = None) -> str: + """Return the topic/subject for command status updates. + + With the ``"part3"`` MQTT topic scheme the status topic is per + command: ``…/controlstreams/{csId}/commands/{cmdId}/status:data``, so + ``command_id`` is required. The server currently delivers the status + of every command on the control stream on that topic. + + With the other MQTT schemes and over NATS there is one status topic + per control stream and ``command_id`` is ignored. Status payloads are + ``application/json``, so it carries the ``json`` format subtopic + (``…/status:data/json`` for MQTT, ``…status:data.json`` for NATS). + + :raises ValueError: with the ``"part3"`` scheme, if ``command_id`` is + missing or the parent system id is unknown. + """ + if self._uses_part3_mqtt_topics(): + return self.get_mqtt_topic(subresource=APIResourceTypes.STATUS, data_topic=True, + command_id=command_id) return self.get_stream_topic(subresource=APIResourceTypes.STATUS, data_topic=True, format="application/json") @@ -153,7 +179,11 @@ def publish_command(self, payload): def publish_status(self, payload): """Publish ``payload`` to the status MQTT topic. Convenience wrapper - for ``publish(payload, APIResourceTypes.STATUS.value)``.""" + for ``publish(payload, APIResourceTypes.STATUS.value)``. + + :raises ValueError: with the ``"part3"`` MQTT topic scheme, where + status cannot be published over MQTT. + """ self.publish(payload, topic=APIResourceTypes.STATUS.value) def publish(self, payload, topic: str = APIResourceTypes.COMMAND.value): @@ -171,6 +201,11 @@ def publish(self, payload, topic: str = APIResourceTypes.COMMAND.value): if topic == APIResourceTypes.COMMAND.value: self._publish_mqtt(self._topic, payload) elif topic == APIResourceTypes.STATUS.value: + if self._uses_part3_mqtt_topics(): + raise ValueError( + "Command status cannot be published over MQTT with the 'part3' topic scheme: " + "the status topic is subscribe-only. Report status over HTTP " + "(POST /controlstreams/{id}/status).") self._publish_mqtt(self._status_topic, payload) else: raise ValueError( @@ -179,7 +214,7 @@ def publish(self, payload, topic: str = APIResourceTypes.COMMAND.value): f"{APIResourceTypes.STATUS.value!r}." ) - def subscribe(self, topic=None, callback=None, qos=0): + def subscribe(self, topic=None, callback=None, qos=0, command_id: str | None = None): """ Subscribes to the MQTT topic associated with this control stream resource. @@ -189,6 +224,11 @@ def subscribe(self, topic=None, callback=None, qos=0): case-sensitive against the canonical CS API resource-type strings. :param callback: Optional callback function to handle incoming messages, if None the default handler is used. :param qos: Quality of Service level for the subscription, default is 0. + :param command_id: Id of the command whose status to subscribe to. + Required for a status subscription with the ``"part3"`` MQTT topic + scheme; ignored otherwise. + :raises ValueError: for an unknown ``topic``, or a ``"part3"`` status + subscription without ``command_id``. """ t = None @@ -196,7 +236,10 @@ def subscribe(self, topic=None, callback=None, qos=0): if topic is None or topic == APIResourceTypes.COMMAND.value: t = self._subscribe_topic elif topic == APIResourceTypes.STATUS.value: - t = self._status_subscribe_topic + if self._uses_part3_mqtt_topics(): + t = self.get_mqtt_status_topic(command_id) + else: + t = self._status_subscribe_topic else: raise ValueError( f"Invalid topic {topic!r}; must be None, " diff --git a/src/oshconnect/resources/datastream.py b/src/oshconnect/resources/datastream.py index a779098..d78375d 100644 --- a/src/oshconnect/resources/datastream.py +++ b/src/oshconnect/resources/datastream.py @@ -146,12 +146,17 @@ def init_mqtt(self): super().init_mqtt() schema = getattr(self._underlying_resource, "record_schema", None) obs_format = getattr(schema, "obs_format", None) if schema is not None else None - self._topic = self.get_stream_topic(subresource=APIResourceTypes.OBSERVATION, - data_topic=True, format=obs_format) - # Publish uses the exact-format topic above; subscribe uses a - # format-wildcard subject over NATS (server picks the proactive format). - self._subscribe_topic = self.get_subscribe_topic(subresource=APIResourceTypes.OBSERVATION, - format=obs_format) + try: + self._topic = self.get_stream_topic(subresource=APIResourceTypes.OBSERVATION, + data_topic=True, format=obs_format) + # Publish uses the exact-format topic above; subscribe uses a + # format-wildcard subject over NATS (server picks the proactive format). + self._subscribe_topic = self.get_subscribe_topic(subresource=APIResourceTypes.OBSERVATION, + format=obs_format) + except ValueError as e: + if self._topic_unavailable_without_client(e): + self._topic = None + self._subscribe_topic = None def _emit_inbound_event(self, msg): evt = (EventBuilder().with_type(DefaultEventTypes.NEW_OBSERVATION).with_topic(msg.topic).with_data( diff --git a/src/oshconnect/resources/system.py b/src/oshconnect/resources/system.py index 28a11b3..3006c11 100644 --- a/src/oshconnect/resources/system.py +++ b/src/oshconnect/resources/system.py @@ -189,6 +189,7 @@ def discover_datastreams(self) -> list[Datastream]: for ds in datastream_json: datastream_objs = DatastreamResource.model_validate(ds, by_alias=True) new_ds = Datastream(self._parent_node, datastream_objs) + new_ds.set_parent_resource_id(self._resource_id) obs_format, parser = self._pick_datastream_schema_format( datastream_objs.formats) if obs_format is None: @@ -261,6 +262,7 @@ def discover_controlstreams(self) -> list[ControlStream]: for cs_json in controlstream_json: controlstream_objs = ControlStreamResource.model_validate(cs_json) new_cs = ControlStream(self._parent_node, controlstream_objs) + new_cs.set_parent_resource_id(self._resource_id) try: schema_resp = api.get_resource( APIResourceTypes.CONTROL_CHANNEL, controlstream_objs.cs_id, @@ -648,6 +650,12 @@ def from_storage_dict(cls, data: dict, node: 'Node') -> 'System': obj._id = uuid.UUID(data["id"]) obj.datastreams = [Datastream.from_storage_dict(ds, node) for ds in data.get("datastreams", [])] obj.control_channels = [ControlStream.from_storage_dict(cc, node) for cc in data.get("control_channels", [])] + # Streams saved before the parent system id was recorded: fill it in, + # since Part 3 MQTT topics are nested under the system. + if obj._resource_id: + for stream in (*obj.datastreams, *obj.control_channels): + if not stream.get_parent_resource_id(): + stream.set_parent_resource_id(obj._resource_id) underlying = data.get("underlying_resource") obj._underlying_resource = SystemResource.model_validate(underlying) if underlying else None return obj diff --git a/tests/test_default_api_helpers.py b/tests/test_default_api_helpers.py index dbc83fe..706518c 100644 --- a/tests/test_default_api_helpers.py +++ b/tests/test_default_api_helpers.py @@ -273,7 +273,7 @@ def test_collection_membership_uses_items_endpoint(self): class TestGetMQTTTopic: def test_data_topic_for_datastream_observations(self): - helper = _make_helper() + helper = _make_helper(topic_scheme="part3-flat") topic = helper.get_mqtt_topic( resource_type=APIResourceTypes.DATASTREAM, subresource_type=APIResourceTypes.OBSERVATION, @@ -282,6 +282,17 @@ def test_data_topic_for_datastream_observations(self): ) assert topic == "api/datastreams/ds-1/observations:data" + def test_data_topic_is_nested_under_the_system_by_default(self): + helper = _make_helper() + topic = helper.get_mqtt_topic( + resource_type=APIResourceTypes.DATASTREAM, + subresource_type=APIResourceTypes.OBSERVATION, + resource_id="ds-1", + data_topic=True, + system_path="sys-1", + ) + assert topic == "api/systems/sys-1/datastreams/ds-1/observations:data" + def test_event_topic_omits_data_suffix(self): helper = _make_helper() topic = helper.get_mqtt_topic( @@ -293,7 +304,7 @@ def test_event_topic_omits_data_suffix(self): assert topic == "api/systems/sys-1/datastreams" def test_topic_uses_mqtt_topic_root_when_set(self): - helper = _make_helper(mqtt_topic_root="osh/mqtt") + helper = _make_helper(mqtt_topic_root="osh/mqtt", topic_scheme="part3-flat") topic = helper.get_mqtt_topic( resource_type=APIResourceTypes.DATASTREAM, subresource_type=APIResourceTypes.OBSERVATION, @@ -303,7 +314,7 @@ def test_topic_uses_mqtt_topic_root_when_set(self): assert topic.startswith("osh/mqtt/") def test_topic_with_subresource_id_appends_after_data_suffix(self): - helper = _make_helper() + helper = _make_helper(topic_scheme="part3-flat") topic = helper.get_mqtt_topic( resource_type=APIResourceTypes.DATASTREAM, subresource_type=APIResourceTypes.OBSERVATION, diff --git a/tests/test_discovery.py b/tests/test_discovery.py index a842ac3..d659a59 100644 --- a/tests/test_discovery.py +++ b/tests/test_discovery.py @@ -21,6 +21,7 @@ import pytest from oshconnect import Node, System +from oshconnect.csapi4py.constants import APIResourceTypes from oshconnect.resource_datamodels import DatastreamResource from oshconnect.schema_datamodels import SWEDatastreamRecordSchema from oshconnect.streamableresource import SchemaFetchWarning @@ -145,6 +146,9 @@ def test_discover_datastreams_populates_record_schema(node, monkeypatch): discovered = sys.discover_datastreams() assert len(discovered) == 1 + # Part 3 MQTT topics are nested under the system, so the stream must + # know its parent. + assert discovered[0].get_parent_resource_id() == "sys-1" populated = discovered[0]._underlying_resource.record_schema assert isinstance(populated, SWEDatastreamRecordSchema) assert populated.obs_format == "application/swe+json" @@ -227,6 +231,28 @@ def schema_handler(ds_id): ) +def test_discover_controlstreams_records_parent_system(node, monkeypatch): + """A control stream resource has no ``system@id``, so discovery must + record the parent system: Part 3 MQTT topics are nested under it.""" + def mock_get(url, params=None, headers=None, auth=None, **kwargs): + if str(url).endswith("/schema"): + return MockResponse(status=404) + return MockResponse({"items": [{"id": "cs-1", "name": "ptz"}]}) + + monkeypatch.setattr( + "oshconnect.csapi4py.request_wrappers.requests.get", mock_get, + ) + sys = System(label="S", urn="urn:test:s", + parent_node=node, resource_id="sys-1") + with pytest.warns(SchemaFetchWarning): + discovered = sys.discover_controlstreams() + + assert len(discovered) == 1 + assert discovered[0].get_parent_resource_id() == "sys-1" + assert (discovered[0].get_mqtt_topic(subresource=APIResourceTypes.COMMAND) + == "api/systems/sys-1/controlstreams/cs-1/commands:data") + + # --------------------------------------------------------------------------- # Node.discover_systems: parsed SystemResource must be bound to the wrapper # --------------------------------------------------------------------------- diff --git a/tests/test_mqtt_topics.py b/tests/test_mqtt_topics.py index dd24f86..3ddc2a9 100644 --- a/tests/test_mqtt_topics.py +++ b/tests/test_mqtt_topics.py @@ -1,19 +1,26 @@ """ Tests verifying that each resource type produces the correct MQTT topic strings -per the CS API Part 3 pub/sub conventions. +for each topic scheme. -Topic format (Resource Data Topic): - /{api_root}/{resource_type}/{resource_id}/{subresource_type}:data +``part3`` (default), CS API Part 3 nested under the system path: + {root}/systems/{sysId}/datastreams/{dsId}/observations:data[/{format}] + {root}/systems/{sysId}/controlstreams/{csId}/commands:data[/{format}] + {root}/systems/{sysId}/controlstreams/{csId}/commands/{cmdId}/status:data + {root}/systems/{sysId}/datastreams/{dsId} (Resource Event) -Event topic format (Resource Event Topic): - /{api_root}/{resource_type}/{resource_id} - /{api_root}/{parent_type}/{parent_id}/{resource_type}/{resource_id} (with parent) +``part3-flat``, the same without the system path: + {root}/datastreams/{dsId}/observations:data[/{format}] + +``legacy``, pre-Part-3: + /{root}/datastreams/{dsId}/observations """ import pytest from unittest.mock import MagicMock +from oshconnect import ConfigurationError, Datastream, Node, System from oshconnect.csapi4py.constants import APIResourceTypes from oshconnect.csapi4py.default_api_helpers import APIHelper +from oshconnect.resource_datamodels import DatastreamResource from tests import helpers DS_ID = "ds_test_001" @@ -22,8 +29,16 @@ PARENT_SYS_ID = "sys_parent_001" -def make_mock_node(api_root="api", mqtt_topic_root=None, legacy_topics=False): - """Returns a mock Node backed by a real APIHelper so topic construction is exercised.""" +def make_mock_node(api_root="api", mqtt_topic_root=None, legacy_topics=False, + topic_scheme="part3-flat"): + """Returns a mock Node backed by a real APIHelper so topic construction is exercised. + + Defaults to the ``part3-flat`` scheme: the classes below pin the topics + that scheme (and ``legacy``) must keep producing. The default ``part3`` + scheme is covered by ``TestPart3NestedTopics``. + """ + if legacy_topics: + topic_scheme = None api_helper = APIHelper( server_url="localhost", port=8282, @@ -32,6 +47,7 @@ def make_mock_node(api_root="api", mqtt_topic_root=None, legacy_topics=False): api_root=api_root, mqtt_topic_root=mqtt_topic_root, legacy_topics=legacy_topics, + topic_scheme=topic_scheme, ) node = MagicMock() node.get_api_helper.return_value = api_helper @@ -325,7 +341,29 @@ def test_format_token_mapping(self, content_type, token): def test_unknown_format_raises_value_error(self): from oshconnect.csapi4py.mqtt import mqtt_topic_format_token with pytest.raises(ValueError, match="No MQTT topic-format token"): - mqtt_topic_format_token("application/swe+protobuf") + mqtt_topic_format_token("text/csv") + + @pytest.mark.parametrize("content_type,token", [ + ("application/geo+json", "geo-json"), + ("application/sml+xml", "sml-xml"), + ("application/swe+xml", "swe-xml"), + # custom formats follow the Part 3 normalization rule + ("application/x-custom+json", "x-custom-json"), + ("application/X-Custom+JSON; version=2", "x-custom-json"), + ("application/swe+protobuf", "swe-protobuf"), + ("application/vnd.acme.thing+json", "vnd.acme.thing-json"), + ]) + def test_format_token_follows_part3_rule(self, content_type, token): + from oshconnect.csapi4py.mqtt import mqtt_topic_format_token + assert mqtt_topic_format_token(content_type) == token + + @pytest.mark.parametrize("content_type", [ + "text/csv", "image/png", "", None, "application/a#b", + ]) + def test_format_without_a_valid_token_raises(self, content_type): + from oshconnect.csapi4py.mqtt import mqtt_topic_format_token + with pytest.raises(ValueError, match="No MQTT topic-format token"): + mqtt_topic_format_token(content_type) def test_get_mqtt_topic_omits_format_when_none(self): """``format=None`` (default) emits bare ``:data`` so the server's @@ -351,7 +389,7 @@ def test_get_mqtt_topic_appends_format_when_provided(self): ) assert topic == f"api/datastreams/{DS_ID}/observations:data/swe-binary" - def test_get_mqtt_topic_raises_for_unknown_format(self): + def test_get_mqtt_topic_raises_for_format_without_token(self): helper = make_mock_node().get_api_helper() with pytest.raises(ValueError, match="No MQTT topic-format token"): helper.get_mqtt_topic( @@ -359,7 +397,7 @@ def test_get_mqtt_topic_raises_for_unknown_format(self): subresource_type=APIResourceTypes.OBSERVATION, resource_id=DS_ID, data_topic=True, - format="application/swe+protobuf", + format="text/csv", ) def test_get_mqtt_topic_ignores_format_on_event_topic(self): @@ -507,14 +545,14 @@ def test_controlstream_init_mqtt_sets_legacy_topics(self): assert cs._topic == f"/api/controlstreams/{CS_ID}/commands" assert cs._status_topic == f"/api/controlstreams/{CS_ID}/status" - def test_legacy_off_by_default(self): + def test_flat_scheme_is_not_legacy(self): ds = make_datastream(make_mock_node()) topic = ds.get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION, data_topic=True) assert topic == f"api/datastreams/{DS_ID}/observations:data" def test_per_call_legacy_override(self): - """Callers can force legacy per-call even when the helper defaults to Part 3.""" - node = make_mock_node() # legacy_topics=False + """Callers can force legacy per-call whatever the helper's scheme.""" + node = make_mock_node() # part3-flat helper = node.get_api_helper() topic = helper.get_mqtt_topic( resource_type=APIResourceTypes.DATASTREAM, @@ -522,3 +560,288 @@ def test_per_call_legacy_override(self): resource_id=DS_ID, data_topic=True, legacy=True, ) assert topic == f"/api/datastreams/{DS_ID}/observations" + + +# --------------------------------------------------------------------------- +# Default ``part3`` scheme: topics nested under the system path. +# +# The expected strings are the ones the server accepts or rejects in +# osh-addons ``sensorhub-service-csapi-mqtt`` (``TestTopicMapper``), so a +# mismatch here means the client and the server disagree. +# --------------------------------------------------------------------------- + +def part3_node(mqtt_topic_root="node1"): + return make_mock_node(mqtt_topic_root=mqtt_topic_root, topic_scheme="part3") + + +def part3_datastream(system_id="134", node=None): + ds = helpers.make_datastream(node or part3_node(), ds_id="12") + if system_id is not None: + ds.set_parent_resource_id(system_id) + return ds + + +def part3_controlstream(system_id="134", node=None): + cs = helpers.make_controlstream(node or part3_node(), cs_id="3") + if system_id is not None: + cs.set_parent_resource_id(system_id) + return cs + + +class TestTopicScheme: + def test_default_scheme_is_part3(self): + assert APIHelper().get_topic_scheme() == "part3" + + def test_node_default_scheme_is_part3(self): + node = Node(protocol="http", address="localhost", port=8282) + assert node.get_api_helper().get_topic_scheme() == "part3" + + def test_legacy_flag_selects_legacy_scheme(self): + node = Node(protocol="http", address="localhost", port=8282, mqtt_legacy_topics=True) + assert node.get_api_helper().get_topic_scheme() == "legacy" + + @pytest.mark.parametrize("scheme", ["part3", "part3-flat", "legacy"]) + def test_node_accepts_each_scheme(self, scheme): + node = Node(protocol="http", address="localhost", port=8282, mqtt_topic_scheme=scheme) + assert node.get_api_helper().get_topic_scheme() == scheme + + def test_unknown_scheme_raises(self): + with pytest.raises(ConfigurationError, match="Unknown MQTT topic scheme"): + Node(protocol="http", address="localhost", port=8282, mqtt_topic_scheme="flat") + + def test_legacy_flag_conflicting_with_scheme_raises(self): + with pytest.raises(ConfigurationError, match="conflicts"): + Node(protocol="http", address="localhost", port=8282, + mqtt_legacy_topics=True, mqtt_topic_scheme="part3") + + def test_legacy_flag_with_legacy_scheme_is_allowed(self): + node = Node(protocol="http", address="localhost", port=8282, + mqtt_legacy_topics=True, mqtt_topic_scheme="legacy") + assert node.get_api_helper().get_topic_scheme() == "legacy" + + @pytest.mark.parametrize("scheme", ["part3", "part3-flat", "legacy"]) + def test_scheme_survives_storage_round_trip(self, scheme): + node = Node(protocol="http", address="localhost", port=8282, mqtt_topic_scheme=scheme) + stored = node.to_storage_dict() + assert stored["mqtt_topic_scheme"] == scheme + assert Node.from_storage_dict(stored).get_api_helper().get_topic_scheme() == scheme + + def test_config_without_scheme_loads_as_part3(self): + stored = Node(protocol="http", address="localhost", port=8282).to_storage_dict() + del stored["mqtt_topic_scheme"] + assert Node.from_storage_dict(stored).get_api_helper().get_topic_scheme() == "part3" + + +class TestPart3NestedTopics: + def test_observation_data_topic(self): + ds = part3_datastream() + assert (ds.get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION) + == "node1/systems/134/datastreams/12/observations:data") + + def test_observation_data_topic_with_format(self): + ds = part3_datastream() + assert (ds.get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION, format="application/swe+csv") + == "node1/systems/134/datastreams/12/observations:data/swe-csv") + + def test_datastream_parent_from_system_id_field(self): + """A datastream read from the server carries ``system@id``.""" + res = DatastreamResource.model_validate({ + "id": "12", "name": "d", "system@id": "134", + "validTime": ["2024-01-01T00:00:00Z", "2025-01-01T00:00:00Z"]}) + ds = Datastream(parent_node=part3_node(), datastream_resource=res) + assert (ds.get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION) + == "node1/systems/134/datastreams/12/observations:data") + + def test_datastream_system_id_field_wins_over_parent_id(self): + res = DatastreamResource.model_validate({ + "id": "12", "name": "d", "system@id": "134", + "validTime": ["2024-01-01T00:00:00Z", "2025-01-01T00:00:00Z"]}) + ds = Datastream(parent_node=part3_node(), datastream_resource=res) + ds.set_parent_resource_id("other") + assert (ds.get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION) + == "node1/systems/134/datastreams/12/observations:data") + + # data and event topics resolve the system the same way + assert ds.get_event_topic() == "node1/systems/134/datastreams/12" + + def test_command_data_topic(self): + cs = part3_controlstream() + assert (cs.get_mqtt_topic(subresource=APIResourceTypes.COMMAND) + == "node1/systems/134/controlstreams/3/commands:data") + + def test_command_status_topic_is_per_command(self): + cs = part3_controlstream() + assert (cs.get_mqtt_status_topic("99") + == "node1/systems/134/controlstreams/3/commands/99/status:data") + + def test_subsystem_path(self): + helper = part3_node().get_api_helper() + topic = helper.get_mqtt_topic( + resource_type=APIResourceTypes.DATASTREAM, subresource_type=APIResourceTypes.OBSERVATION, + resource_id="12", system_path=["134", "7", "8"]) + assert topic == "node1/systems/134/subsystems/7/subsystems/8/datastreams/12/observations:data" + + def test_subsystem_path_for_command_status(self): + helper = part3_node().get_api_helper() + topic = helper.get_mqtt_topic( + resource_type=APIResourceTypes.CONTROL_CHANNEL, subresource_type=APIResourceTypes.STATUS, + resource_id="3", subresource_id="99", system_path=["134", "7"]) + assert topic == "node1/systems/134/subsystems/7/controlstreams/3/commands/99/status:data" + + def test_system_path_accepts_a_single_id(self): + helper = part3_node().get_api_helper() + topic = helper.get_mqtt_topic( + resource_type=APIResourceTypes.DATASTREAM, subresource_type=APIResourceTypes.OBSERVATION, + resource_id="2", system_path="1", format="application/swe+json") + assert topic == "node1/systems/1/datastreams/2/observations:data/swe-json" + + def test_system_collection_topics(self): + sysres = helpers.make_system(part3_node(), resource_id="134") + assert sysres.get_mqtt_topic() == "node1/systems:data" + assert (sysres.get_mqtt_topic(subresource=APIResourceTypes.DATASTREAM) + == "node1/systems/134/datastreams:data") + assert (sysres.get_mqtt_topic(subresource=APIResourceTypes.CONTROL_CHANNEL) + == "node1/systems/134/controlstreams:data") + + def test_event_topics(self): + assert part3_datastream().get_event_topic() == "node1/systems/134/datastreams/12" + assert part3_controlstream().get_event_topic() == "node1/systems/134/controlstreams/3" + assert helpers.make_system(part3_node(), resource_id="134").get_event_topic() == "node1/systems/134" + + def test_default_root_is_api(self): + ds = part3_datastream(node=part3_node(mqtt_topic_root=None)) + assert (ds.get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION) + == "api/systems/134/datastreams/12/observations:data") + + # -- forms the server rejects are never built --------------------------- + + def test_datastream_without_parent_raises(self): + """The server rejects ``node1/datastreams/2/observations:data``.""" + ds = part3_datastream(system_id=None) + with pytest.raises(ValueError, match="parent system id"): + ds.get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION) + + def test_controlstream_without_parent_raises(self): + cs = part3_controlstream(system_id=None) + with pytest.raises(ValueError, match="parent system id"): + cs.get_mqtt_topic(subresource=APIResourceTypes.COMMAND) + + def test_event_topic_without_parent_raises(self): + """The server rejects ``node1/datastreams/2``.""" + with pytest.raises(ValueError, match="parent system id"): + part3_datastream(system_id=None).get_event_topic() + with pytest.raises(ValueError, match="parent system id"): + part3_controlstream(system_id=None).get_event_topic() + + def test_status_topic_without_command_id_raises(self): + with pytest.raises(ValueError, match="per command"): + part3_controlstream().get_mqtt_status_topic() + + def test_unknown_format_raises(self): + with pytest.raises(ValueError, match="No MQTT topic-format token"): + part3_datastream().get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION, + format="text/csv") + + def test_custom_format_subtopic(self): + ds = part3_datastream() + assert (ds.get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION, + format="application/x-custom+json") + == "node1/systems/134/datastreams/12/observations:data/x-custom-json") + + def test_command_format_subtopic(self): + cs = part3_controlstream() + assert (cs.get_mqtt_topic(subresource=APIResourceTypes.COMMAND, format="application/swe+json") + == "node1/systems/134/controlstreams/3/commands:data/swe-json") + + def test_init_mqtt_without_parent_raises_when_a_client_exists(self): + node = part3_node() + node.get_mqtt_client.return_value = MagicMock() + with pytest.raises(ValueError, match="parent system id"): + part3_datastream(system_id=None, node=node).init_mqtt() + with pytest.raises(ValueError, match="parent system id"): + part3_controlstream(system_id=None, node=node).init_mqtt() + + def test_init_mqtt_without_parent_or_client_does_not_raise(self): + """HTTP-only use: no pub/sub client, so the topics are never needed.""" + ds = part3_datastream(system_id=None) + ds.init_mqtt() + assert ds._topic is None and ds._subscribe_topic is None + cs = part3_controlstream(system_id=None) + cs.init_mqtt() + assert cs._topic is None and cs._status_topic is None + + def test_constructing_without_parent_does_not_raise(self): + """The parent id may be set after construction; only topic building needs it.""" + cs = part3_controlstream(system_id=None) + assert cs._status_topic is None + + +class TestPart3ResourceWiring: + def test_datastream_init_mqtt_sets_nested_topics(self): + ds = part3_datastream() + ds.init_mqtt() + assert ds._topic == "node1/systems/134/datastreams/12/observations:data" + assert ds._subscribe_topic == "node1/systems/134/datastreams/12/observations:data" + + def test_controlstream_init_mqtt_sets_command_topic_only(self): + cs = part3_controlstream() + cs.init_mqtt() + assert cs._topic == "node1/systems/134/controlstreams/3/commands:data" + assert cs._subscribe_topic == "node1/systems/134/controlstreams/3/commands:data" + assert cs._status_topic is None + assert cs._status_subscribe_topic is None + + def test_publish_command_uses_nested_topic(self): + node = part3_node() + client = MagicMock() + node.get_mqtt_client.return_value = client + cs = part3_controlstream(node=node) + cs.init_mqtt() + cs.publish_command("payload") + client.publish.assert_called_once_with( + "node1/systems/134/controlstreams/3/commands:data", "payload", qos=0) + + def test_publish_status_raises_and_sends_nothing(self): + node = part3_node() + client = MagicMock() + node.get_mqtt_client.return_value = client + cs = part3_controlstream(node=node) + cs.init_mqtt() + with pytest.raises(ValueError, match="subscribe-only"): + cs.publish_status("payload") + client.publish.assert_not_called() + + def test_subscribe_status_needs_command_id(self): + node = part3_node() + client = MagicMock() + node.get_mqtt_client.return_value = client + cs = part3_controlstream(node=node) + cs.init_mqtt() + with pytest.raises(ValueError, match="per command"): + cs.subscribe(topic=APIResourceTypes.STATUS.value) + client.subscribe.assert_not_called() + + def test_subscribe_status_with_command_id(self): + node = part3_node() + client = MagicMock() + node.get_mqtt_client.return_value = client + cs = part3_controlstream(node=node) + cs.init_mqtt() + cs.subscribe(topic=APIResourceTypes.STATUS.value, command_id="99") + assert (client.subscribe.call_args.args[0] + == "node1/systems/134/controlstreams/3/commands/99/status:data") + + def test_reloaded_streams_get_the_parent_system_id(self): + """Configs saved before the parent id was recorded still build nested topics.""" + node = part3_node() + system = helpers.make_system(node, resource_id="134") + system.datastreams = [part3_datastream(system_id=None, node=node)] + system.control_channels = [part3_controlstream(system_id=None, node=node)] + stored = system.to_storage_dict() + assert stored["datastreams"][0]["parent_resource_id"] is None + + reloaded = System.from_storage_dict(stored, node) + assert (reloaded.datastreams[0].get_mqtt_topic(subresource=APIResourceTypes.OBSERVATION) + == "node1/systems/134/datastreams/12/observations:data") + assert (reloaded.control_channels[0].get_mqtt_topic(subresource=APIResourceTypes.COMMAND) + == "node1/systems/134/controlstreams/3/commands:data") diff --git a/tests/test_nats_subjects.py b/tests/test_nats_subjects.py index 3ec0932..05e35f8 100644 --- a/tests/test_nats_subjects.py +++ b/tests/test_nats_subjects.py @@ -214,11 +214,11 @@ def test_get_stream_topic_nested_for_nats(self, nats_client): assert (ds.get_stream_topic(subresource=APIResourceTypes.OBSERVATION) == "api.systems.sys-1.datastreams.ds-1.observations:data") - def test_get_stream_topic_flat_for_mqtt(self): + def test_get_stream_topic_slash_delimited_for_mqtt(self): ds = make_ds(make_mock_node(comm_client=None)) - # Flat MQTT topic: datastream at top level, slash-delimited. + # MQTT topic: same nesting as NATS, slash-delimited. assert (ds.get_stream_topic(subresource=APIResourceTypes.OBSERVATION) - == "api/datastreams/ds-1/observations:data") + == "api/systems/sys-1/datastreams/ds-1/observations:data") def test_nats_subscribe_topic_is_format_wildcard(self, nats_client): ds = make_ds(make_mock_node(comm_client=nats_client)) @@ -229,7 +229,7 @@ def test_mqtt_subscribe_topic_is_exact_format(self): ds = make_ds(make_mock_node(comm_client=None)) assert (ds.get_subscribe_topic(subresource=APIResourceTypes.OBSERVATION, format="application/swe+json") - == "api/datastreams/ds-1/observations:data/swe-json") + == "api/systems/sys-1/datastreams/ds-1/observations:data/swe-json") def test_datastream_init_mqtt_sets_nested_topic(self, nats_client): ds = make_ds(make_mock_node(comm_client=nats_client)) diff --git a/uv.lock b/uv.lock index 25fc699..25555f6 100644 --- a/uv.lock +++ b/uv.lock @@ -642,7 +642,7 @@ wheels = [ [[package]] name = "oshconnect" -version = "0.5.5a2" +version = "0.5.6a0" source = { virtual = "." } dependencies = [ { name = "certifi" },