Items with no label
3335 Discussions

Question about Frame Sync Index [D435][HW sync]

sliu52
Beginner
2,663 Views

Hello guys and administers,

I am using two D435 and connected their HW sync pin as the whitepaper instructed:

(https://realsense.intel.com/wp-content/uploads/sites/63/Multiple_Camera_WhitePaper04.pdf https://realsense.intel.com/wp-content/uploads/sites/63/Multiple_Camera_WhitePaper04.pdf)

The SW seems working, both camera supports RS2_OPTION_INTER_CAM_SYNC_MODE and I can setup one camera as Master and the other as Slave:

bool support_hw_sync = depth_sensor.supports(RS2_OPTION_INTER_CAM_SYNC_MODE);

if (support_hw_sync) {

try {

float sync_mode = found_master ? (float)1.0 : (float)2.0 ;// [0:default/1:sync master/2:sync slave]

depth_sensor.set_option( RS2_OPTION_INTER_CAM_SYNC_MODE, sync_mode );

Here comes the problem when I want to verified the HW sync result.

The way is using some information as "frame sync index", with same "frame sync index" , the contents captured by different camera should be the same.

I have tried the following metadatas as "frame sync index",:

[1] RS2_FRAME_METADATA_FRAME_TIMESTAMP or RS2_FRAME_METADATA_SENSOR_TIMESTAMP

According to it is stated here ( https://github.com/IntelRealSense/librealsense/issues/2188)

These timestamp are counted by individual device clock, which means they are not aligned at all.

By my experiments, it is true that when HW sync mode is on, the framesets polled at the same time from different camera,

their FRAME_TIMESTAMP and SENSOR_TIMESTAMP are not aligned

=> it can not be used as "frame sync index"

[2] RS2_FRAME_METADATA_FRAME_COUNTER

The whitepaper says that with hw sync, the counters are aligned between cameras;

However, according to my experiment, they are not, here is a snippet of log:

// camera 1, serial number 825312070233, polled at 30:813, frame counter is 172

[10/17/18 16:35:30:813][8152 018664][D] [825312070233] polled frame[172] timestamp (1539765330757.000000)

// camera 2, serial number 819312070204, polled at 30:814, frame counter is 215

[10/17/18 16:35:30:814][8152 018664][D] [819312070204] polled frame[215] timestamp (1539765330756.000000)

=>the frame counters are NOT aligned and not useful

[3] RS2_FRAME_METADATA_BACKEND_TIMESTAMP

So I turn to BACKEND_TIMESTAMP.

At first glance, it seems doable if I use this timestamp as reference, the frameset acquired from rs2::pipelines are perfectly aligned (by checking the IR image shooting a same stopwatch)

<img __jive_ID="522275" class="image-3 jive-image" src="pastedImage_19.png" style="max-width: 1200px; max-height: 90...

0 Kudos
15 Replies
idata
Employee
881 Views

Hello shane7226107,

 

 

Thank you for your interest in the Intel RealSense Technology.

 

 

We have received your query and are currently investigating.

 

 

I will get back to your as soon as possible.

 

 

Best regards,

 

Casandra
0 Kudos
Anders_G_Intel
Employee
881 Views

shane7226107

Regarding this statement

[2] RS2_FRAME_METADATA_FRAME_COUNTER

The whitepaper says that with hw sync, the counters are aligned between cameras;

However, according to my experiment, they are not, here is a snippet of log:

// camera 1, serial number 825312070233, polled at 30:813, frame counter is 172

[10/17/18 16:35:30:813][8152 018664][D] [825312070233] polled frame[172] timestamp (1539765330757.000000)

// camera 2, serial number 819312070204, polled at 30:814, frame counter is 215

[10/17/18 16:35:30:814][8152 018664][D] [819312070204] polled frame[215] timestamp (1539765330756.000000)

=>the frame counters are NOT aligned and not useful

They counters will stay aligned but they will have an offset. In software you can "tar" this and subtract a fixed number from each. So if you subtract 172 from counter 1 and 215 from counter 2, then from then on they should both progress aligned as 1, 2, 3, 4, 5 etc.

idata
Employee
881 Views

Hello shane7226107,

 

 

Thank you very much for your patience in receiving an answer.

 

We noticed that you are using an outdated version of the Muticam Sync Whitepaper - you can find the updated one here: https://www.intel.com/content/www/us/en/support/articles/000028140/emerging-technologies/intel-realsense-technology.html

 

 

Please have a look at the following sections:

F. System vs HW time

G. HW sync validation

H. Latency

3. Multi-Camera Programming

Let me know if you have any questions.

 

 

Best regards,

 

Casandra
sliu52
Beginner
881 Views

Hi Casandra and Anders,

Thanks for your reply!

I have read through the whitepaper again, however there is still some thing I can not clearly understand.

The question is about boundary case as described below:

Assume that there is an offset of 3 between master and slave's frame counter (i.e. the sync frame pair from master is "N", slave's "N+3")

[Normal Case]

in this normal case, I can subtract slave's frame counter by an offset of 3, after that the frame counter is sw aligned and can be used as "frame sync index

[Boundary Case]

Since the polling threads are independent, it is not guaranteed to be perfectly executed at the same moment.

There is some chance boundary case happens like this, master polled a frame with frame counter "0", slave polled frame counter is "4"

If I subtract slave by a constant offset "4" afterward, it will obviously causing problems.

In this case, what can I do to handle it properly ?

Thank you!

0 Kudos
Anders_G_Intel
Employee
881 Views

You are absolutely correct.

There will be the boundary condition you mention, so best practice would be to read a few frames and select the median offset.

0 Kudos
sliu52
Beginner
881 Views

Hi AndersGJ,

Thanks! I will use the method you have suggested.

0 Kudos
sliu52
Beginner
881 Views

Hi AndersGJ,

I tried the method you have suggested, however the result is not what I expected(there should be one significant offset appears most of the time).

It is often that the offsets "diverges", for example, If I took first 60 frames as reference,

one of the statistic result is like below:

offset (slave's frame count minus master)

number of appearance

-48

3

-47

2

-46

6

-45

5

-44

5

-43

4

-42

4

-41

1

Is it reasonable?

0 Kudos
Anders_G_Intel
Employee
881 Views

The queue has to be set to 1 (or two if using depth and RGB). By default it is 10. This means that if there is any latency in the host system you may get frames arriving at different times.

Also once you do this, you may drop a few frames (depending on your system/number of streams/resolutions/frame rates)

0 Kudos
sliu52
Beginner
881 Views

Hi AndersGJ,

Thanks for replying!

The whitepaper also has mentioned about the frame queue size, but it did not state clearly how to setup frame queue size.

(In page 9, it said about rs2_create_frame_queue...)

Since I am using rs2::pipeline, I've check the source code of pipeline.cpp then find out that the queue size is by default set to "1"

https://github.com/IntelRealSense/librealsense/blob/master/src/pipeline.cpp# L15 librealsense/pipeline.cpp at master · IntelRealSense/librealsense · GitHub

pipeline_processing_block::pipeline_processing_block(const std::vector& streams_to_aggregate) :

_queue(new single_consumer_frame_queue(1)),

I am not sure if we still need to do it manually?

Or can you provide some sample code that describes how to setup frame queue size properly?

Thank you very much!

0 Kudos
idata
Employee
881 Views

Hello Shane,

 

 

I apologize for the long delay in getting back to you.

 

 

I have discussed this with RealSense engineers and they have confirmed that the queue size is indeed 1 by default.

 

 

The RS2_FRAME_METADATA_BACKEND_TIMESTAMP may be your best bet. The phenomenon you reported in your first post where three different frames have the same timestamp should not happen.

 

 

One thing to verify is whether there are three different frames or three instances of the same frame ( by checking the librealsense frame number and Host timestamp for instance).

 

The origin of the frame indexes frame[174]… is not clear from the log above.

 

 

I see that the three polls occurred within 60msec, and it may possible that "wait_for_frame()" will provide the same frame for more than one call.

 

 

Please tell us what PC/platform you are using as your host.

 

 

Regards,

 

Jesus G.

 

Intel Customer Support

 

0 Kudos
sliu52
Beginner
881 Views

Hello Jesus,

Thanks for your reply!

Here are some of my host information:

Camera ModelD435Firmware Version5.10.3Operating System & VersionWin 10 64 bits ,version 1803, build 17134.407PlatformPCSDK Version2.16.1CPUIntel Xeon E5-2667 v2 @3.30GHzRAM112 GBUSB interfaceASMedia 2-port USB3.1 gen II, TypeA PCIe Host Card

If there is anything else you need, please let me know

thanks,

Shane

0 Kudos
Anders_G_Intel
Employee
881 Views

I am going to double down on my previous answer, so please bear with me.

Assuming you have the HW sync enabled, the best approach is to look at the FRAME COUNTERS.

Here is the reasoning:

  1. Frame counters are generated by the ASIC at time of capture.
  2. Inside a single camera all sensors images (left, right, depth) get the same count.
  3. With frame counters you will know if you are dropping frames because there will be gaps in sequences. (This may also be an explanation for issues you are seeing)
  4. If the units are HW synced the difference in frame count between different cameras should NEVER drift.

So when you are seeing some inconsistencies and are trying the System Time Stamp, that is the time when frames are received by the host (and not when they are captured). There may be some buffers in the host or some other strange driver issues regarding when the stamps is applied. Plus you wont know for sure which frame it is or if one or more frames were dropped.

Are there some complications with using frame counters? Yes. At the start, you do need to make sure you are "aligned" by subtracting a different offset from each camera. In principle you only need to do this once, as there is some issue of capturing frames near boundaries. For that you should be able to use time stamps to confirm they were all nominally aligned.

There is one other issue. If an ESD/EMI event causes a reset on a camera, this will reset the frame counter of that camera. So you will need to have a check all the time to make sure that there is not suddenly a large jump in counter, which will indicate a reset.

So to summarize, my recommendation is that it is best to rely on the counters that are being generated by the ASIC. If you set frame buffers to minimum, and OS has good driver, and you dont lose frames, then everything should be fine. In fact, if frame counter is truly set to 1 and host is not buffering, then you should not have an issue with aligning, because all frames received are the latest ones.

0 Kudos
MartyG
Honored Contributor III
881 Views

Whilst I do not have an answer to your specific question (one of the Intel support staff on this forum can likely assist with that), you can test whether hardware sync is working by observing the timestamps. If hardware sync is inactive, the timestamps will remain perfectly aligned. If hardware sync is working, the timestamps will actually drift apart over time.

The workings of this effect are described in section 3 of the white paper, headed 'Multi-camera programming'.

sliu52
Beginner
881 Views

Hi Marty,

Yes I have read through the whitepaper and I understand the way to verify whether HW sync is working.

I am just hoping to resolve the "SW" part of the problem.

I am still appreciate your support!

Thanks,

Shane

0 Kudos
MartyG
Honored Contributor III
881 Views

You're very welcome. The forum did not show me that Jesus had posted a reply before mine when I was writing my answer. So I'm glad you were able to get Intel assistance!

0 Kudos
Reply