Skip to content

Xiaoya refactor publisher#48

Open
xiaoyachong wants to merge 3 commits into
als-computing:mainfrom
xiaoyachong:xiaoya-refactor-publisher
Open

Xiaoya refactor publisher#48
xiaoyachong wants to merge 3 commits into
als-computing:mainfrom
xiaoyachong:xiaoya-refactor-publisher

Conversation

@xiaoyachong
Copy link
Copy Markdown
Contributor

@xiaoyachong xiaoyachong commented Apr 29, 2026

The main purpose of this PR is to move the Tiled publisher from LSE to arroyoXPS, and to have arroyoXPS send results to LSE via ZMQ instead of using a WebSocket connection.

  1. Results are written to Tiled and a ZMQ message is sent to LSE using the same publisher, XPSTiledResultPublisher.
  2. In the XPSTiledResultPublisher, we import:
    from arroyosas.schemas import RawFrameEvent, SASStart, SASStop, SerializableNumpyArrayModel
    Since both arroyosas and arroyoxps send the same message types (RawFrameEvent, SASStart, SASStop) to LSE, it may make sense to move these schemas into arroyopy and rename them (e.g., remove the SAS prefix).
  3. To handle mid-scan issues, I avoid writing to Tiled unless a start message has been received. For example, if splash_timepix starts before arroyoxps, the Tiled publisher logs the following message:
    Received scan 'None' frame 49 but no start message has been received yet — discarding frame to avoid Tiled index desync.
  4. The Tiled structure:
{tiled_base_uri}/{tiled_prefix}/
│
├── {root_segments1:lse_live_results}/
│   └── {YYYY}/
│       └── {MM}/
│           └── {DD}/
│               └── {experiment_name}/
│                   └── {UUID}/
│                       └── feature_vectors  
│                           (table storing feature vector results)
│
└── {root_segments2:xps_processed_images}/
    └── {YYYY}/
        └── {MM}/
            └── {DD}/
                └── {UUID}/
                    └── xps_averaged_heatmaps  
                        (3D array storing shot-mean heatmaps)

Companion PRs:
LSE: mlexchange/mlex_latent_explorer#79
arroyosas: als-computing/arroyosas#22

Comment thread src/tr_ap_xps/xps_tiled_zmq_publisher.py Outdated
Copy link
Copy Markdown
Contributor

@dylanmcreynolds dylanmcreynolds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can make this simpler. See commend.

Copy link
Copy Markdown
Contributor

@dylanmcreynolds dylanmcreynolds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approved. Is this mergable?

Copy link
Copy Markdown

@taxe10 taxe10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good. I left some very minor comments.

parts.extend(root_segments)
parts.append(date_path)
parts.append(uuid)
parts.append(ARRAY_KEY)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ARRAY_KEY is referenced before it's defined

self.poll_interval_sec = poll_interval_sec
self.frame_poll_interval_sec = frame_poll_interval_sec
self.unchanged_cycles_threshold = unchanged_cycles_threshold
self.seen_scans_file = seen_scans_file
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.seen_scans_file = seen_scans_file
self.seen_scans_file = Path(seen_scans_file)

poll_interval_sec=poll_interval_sec,
frame_poll_interval_sec=frame_poll_interval_sec,
unchanged_cycles_threshold=unchanged_cycles_threshold,
seen_scans_file=seen_scans_file,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
seen_scans_file=seen_scans_file,
self.seen_scans_file = Path(seen_scans_file)

return
if isinstance(message, RawFrameEvent):
message = message.model_dump()
# message["image"] = SerializableNumpyArrayModel.serialize_array(
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove this comment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants