Bug in AIT-GUI's script console

[nttoole]

Email (ppartially redacted):

OCO-3 ran into an error and we have a work around right now. took a look but I think he needs a bit more time [...] to look into this further and provide a fix. How do you think we should proceed?

Description of Error:

OCO3 recently made the upgrade from python2 to python3, in the process also updating our version of AIT to one of the most recent releases. While running the AIT-GUI v2.4.1 plugin, the OCO3 team attempted to execute a python script in-gui as we had in prior versions of AIT. (Below is a paired-down test.py script for debugging purposes)

The script ran nominally, but upon executing the script again, or executing another python script in-gui, the gui crashed. Logging information and error traceback can be found in the sections below. Fortunately for the team, the python scripts did execute completely; they did not fail mid-script, but the gui crashed after the script completed. Our current solution is to restart the gui after every script execution. Notably, running the python scripts from the command line does not cause any errors; they run nominally. It is only when the scripts are run from the gui that we see this issue.

Logging information upon boot up and script execution:

(oco3) [oco3ops@oco3-gds1 latest]$ ait-server
2025-03-04T15:09:28.192 | INFO     | Replacing ait.core.cmd.Cmd with custom extension: oco3.cmd.OCO3Cmd
2025-03-04T15:09:28.244 | INFO     | Attempting to load leapseconds.dat
2025-03-04T15:09:28.244 | INFO     | Loaded leapseconds config file successfully
2025-03-04T15:09:28.244 | INFO     | Leapseconds data valid until 06/28/2025
2025-03-04T15:09:28.277 | INFO     | Current pickle file loaded: tlm.pkl
2025-03-04T15:09:28.277 | INFO     | Created handler PacketHandler for stream telem_stream
2025-03-04T15:09:28.278 | INFO     | Added inbound stream <PortInputStream address=127.0.0.1:3076>
2025-03-04T15:09:28.278 | WARNING  | No handlers specified for stream command_stream
2025-03-04T15:09:28.278 | INFO     | Added outbound stream <PortOutputStream name=command_stream>
2025-03-04T15:09:28.278 | WARNING  | No configuration found for API Streams. Attempting to determine valid streams to use as default.
2025-03-04T15:09:28.279 | INFO     | Located potentially valid streams. ['telem_stream'] uses a compatible handler (['PacketHandler', 'CCSDSPacketHandler']).
2025-03-04T15:09:28.279 | WARNING  | No handlers specified for stream __tlmpkts__
2025-03-04T15:09:28.291 | WARNING  | [GUI Playback Configuration]Unable to locate DataArchive plugin configuration for historical data queries. Historical telemetry playback will be disabled in monitoring UI and server endpoints.
2025-03-04T15:09:28.300 | INFO     | Current pickle file loaded: cmd.pkl
2025-03-04T15:09:29.302 | INFO     | Running AIT GUI v2.4.1
2025-03-04T15:09:29.304 | INFO     | [GUI Plugin Configuration] Static file directory is set to /gds/ops/sw/oco3-gds1/released/latest/oco3mos/gui
2025-03-04T15:09:29.304 | INFO     | Added plugin <Plugin name=AITGUIPlugin>
2025-03-04T15:09:29.304 | INFO     | Starting <Broker at 0x7fd79850e0e0: _run> greenlet...
2025-03-04T15:09:29.304 | INFO     | Starting <Plugin name=AITGUIPlugin> greenlet...
2025-03-04T15:09:29.304 | INFO     | Starting <ZMQStream name=__tlmpkts__> greenlet...
2025-03-04T15:09:29.304 | INFO     | Starting <PortOutputStream name=command_stream> greenlet...
2025-03-04T15:09:29.304 | INFO     | Starting <PortInputStream address=127.0.0.1:3076> greenlet...
2025-03-04T15:09:29.310 | INFO     | Subscribing <ZMQStream name=__tlmpkts__> to topic telem_stream
2025-03-04T15:09:29.310 | INFO     | Subscribing <Plugin name=AITGUIPlugin> to topic telem_stream
2025-03-04T15:09:29.311 | INFO     | Subscribing <PortOutputStream name=command_stream> to topic AITGUIPlugin
2025-03-04T15:09:29.311 | WARNING  | No output stream was designated as the command subscriber, command_stream was selected as the default.
2025-03-04T15:09:29.311 | INFO     | Subscribing <PortOutputStream name=command_stream> to topic __commands__
2025-03-04T15:09:29.311 | INFO     | Starting broker...
2025-03-04T15:09:29.828 | INFO     | Current pickle file loaded: evr.pkl
2025-03-04T15:09:31.235 | INFO     | Current pickle file loaded: ioc_limits.pkl
2025-03-04T15:10:04.675 | INFO     | OCO3Payload: Waiting at least 5s for two 1553 EHS packets.
2025-03-04T15:10:04.928 | INFO     | OCO3Payload: Received two 1553 EHS packets.
2025-03-04T15:10:04.928 | PROGRAM  | /export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/bin/ait-server
2025-03-04T15:10:04.928 | INFO     | This is a test!
2025-03-04T15:10:04.929 | PROGRAM  | done.
2025-03-04T15:10:04.929 | ERROR    | Exception raised in <Plugin name=AITGUIPlugin> while receiving messages:

Error traceback:

--- Logging error ---
Traceback (most recent call last):
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/server/client.py", line 114, in _run
    self.process(message, topic=topic)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/gui/__init__.py", line 340, in process
    self.process_telem_msg(input_data)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/gui/__init__.py", line 355, in process_telem_msg
    Sessions.add_telemetry(msg[0], msg[1])
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/gui/__init__.py", line 140, in add_telemetry
    delta, dntoeus = get_packet_delta(pkt_defn, packet)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/gui/__init__.py", line 838, in get_packet_delta
    dntoeu_val = getattr(ait_pkt, field.name)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/tlm.py", line 420, in __getattr__
    return self._getattr(fieldname)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/tlm.py", line 490, in _getattr
    value = defn.dntoeu.eval(self)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/tlm.py", line 99, in eval
    result = self._equation.eval(packet)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/tlm.py", line 786, in eval
    result = eval(self._code, packet._defn.globals, context)
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "/usr/local/python-3.10.13/lib/python3.10/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/local/python-3.10.13/lib/python3.10/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/local/python-3.10.13/lib/python3.10/logging/handlers.py", line 993, in emit
    self.socket.sendto(msg, self.address)
OSError: [Errno 9] Bad file descriptor
Call stack:
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/server/client.py", line 117, in _run
    log.error(
Message: 'Exception raised in <Plugin name=AITGUIPlugin> while receiving messages: '
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/server/client.py", line 114, in _run
    self.process(message, topic=topic)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/gui/__init__.py", line 340, in process
    self.process_telem_msg(input_data)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/gui/__init__.py", line 355, in process_telem_msg
    Sessions.add_telemetry(msg[0], msg[1])
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/gui/__init__.py", line 140, in add_telemetry
    delta, dntoeus = get_packet_delta(pkt_defn, packet)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/gui/__init__.py", line 838, in get_packet_delta
    dntoeu_val = getattr(ait_pkt, field.name)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/tlm.py", line 420, in __getattr__
    return self._getattr(fieldname)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/tlm.py", line 490, in _getattr
    value = defn.dntoeu.eval(self)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/tlm.py", line 99, in eval
    result = self._equation.eval(packet)
  File "/export/gds/ops/sw/oco3-gds1/released/1.13.2/.virtualenv/oco3/lib/python3.10/site-packages/ait/core/tlm.py", line 786, in eval
    result = eval(self._code, packet._defn.globals, context)
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "/usr/local/python-3.10.13/lib/python3.10/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/local/python-3.10.13/lib/python3.10/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit

[nttoole]

Based on Slack-based convo between AIT lead and MGSS-IDS rep:

• Consider dropping support for the GUI script stuff. There is technically lost functionality if you run it in the CLI, but may not be an issue for users.
• CHARM may be a better option, so it not worth the effort to fix and support the GUI stuff
• Look into CHARM, If we can cut out functionality and have an integration point it’s a better strategy
• CHARM: https://github.jpl.nasa.gov/bornstei/charm