9 Replies Latest reply on Oct 31, 2018 9:57 PM by shane7226107

    Question about Frame Sync Index [D435][HW sync]

    shane7226107

      Hello guys and administers,

       

      I am using two D435 and connected their HW sync pin as the whitepaper instructed:

      (https://realsense.intel.com/wp-content/uploads/sites/63/Multiple_Camera_WhitePaper04.pdf)

      dual_camera_setup.png

      The SW seems working, both camera supports RS2_OPTION_INTER_CAM_SYNC_MODE and I can setup one camera as Master and the other as Slave:

      bool support_hw_sync = depth_sensor.supports(RS2_OPTION_INTER_CAM_SYNC_MODE);

      if (support_hw_sync) {

           try {

                float sync_mode = found_master ? (float)1.0 : (float)2.0 ;// [0:default/1:sync master/2:sync slave]

                depth_sensor.set_option( RS2_OPTION_INTER_CAM_SYNC_MODE, sync_mode );

       

      Here comes the problem when I want to verified the HW sync result.

       

      The way is using some information as "frame sync index", with same "frame sync index" , the contents captured by different camera should be the same.

      I have tried the following metadatas as "frame sync index",:

       

      [1] RS2_FRAME_METADATA_FRAME_TIMESTAMP or RS2_FRAME_METADATA_SENSOR_TIMESTAMP

          According to it is stated here (https://github.com/IntelRealSense/librealsense/issues/2188)

          These timestamp are counted by individual device clock, which means they are not aligned at all.

          By my experiments, it is true that when HW sync mode is on, the framesets polled at the same time from different camera,

          their FRAME_TIMESTAMP and SENSOR_TIMESTAMP are not aligned

          => it can not be used as "frame sync index"

       

      [2] RS2_FRAME_METADATA_FRAME_COUNTER

          The whitepaper says that with hw sync, the counters are aligned between cameras;

          However, according to my experiment, they are not, here is a snippet of log:

         // camera 1, serial number 825312070233, polled at 30:813, frame counter is 172

         [10/17/18 16:35:30:813][8152 018664][D] [825312070233] polled frame[172] timestamp (1539765330757.000000)

         // camera 2, serial number 819312070204, polled at 30:814, frame counter is  215

         [10/17/18 16:35:30:814][8152 018664][D] [819312070204] polled frame[215] timestamp (1539765330756.000000)

         =>the frame counters are NOT aligned and not useful

       

      [3] RS2_FRAME_METADATA_BACKEND_TIMESTAMP

          So I turn to BACKEND_TIMESTAMP.

          At first glance, it seems doable if I use this timestamp as reference, the frameset acquired from rs2::pipelines are perfectly aligned (by checking the IR image shooting a same stopwatch)

      Unfortnuately, with more test, I have noticed that sometimes the BACKEND_TIMESTAMP are same even the frames are polled at diffetent time:

       

      // camera 1, serial number 825312070233, polled at 30:877, backend timestamp is 1539765330821

      [10/17/18 16:35:30:877][8152 018664][D] [825312070233] polled frame[174] timestamp (1539765330821.000000)

      // camera 1, serial number 825312070233, polled at 30:937, backend timestamp is 1539765330821

      [10/17/18 16:35:30:937][8152 018664][D] [825312070233] polled frame[175] timestamp (1539765330821.000000)

      // camera 1, serial number 825312070233, polled at 30:940, backend timestamp is 1539765330821

      [10/17/18 16:35:30:940][8152 018664][D] [825312070233] polled frame[176] timestamp (1539765330821.000000)

       

      =>There are 3 consecutive frames with the same BACKEND_TIMESTAMP!

       

      My question is, is there any information that I can used as "frame sync index"?

      Or is there any official way to make sure the frameset acquired at the same time are captured exactly at the same time?

      Thank you!

        • 1. Re: Question about Frame Sync Index [D435][HW sync]
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hello shane7226107, 

          Thank you for your interest in the Intel RealSense Technology. 

          We have received your query and are currently investigating. 

          I will get back to your as soon as possible. 

          Best regards, 
          Casandra 

          • 2. Re: Question about Frame Sync Index [D435][HW sync]
            AndersGJ

            shane7226107

             

            Regarding this statement

            [2] RS2_FRAME_METADATA_FRAME_COUNTER

                The whitepaper says that with hw sync, the counters are aligned between cameras;

                However, according to my experiment, they are not, here is a snippet of log:

               // camera 1, serial number 825312070233, polled at 30:813, frame counter is 172

               [10/17/18 16:35:30:813][8152 018664][D] [825312070233] polled frame[172] timestamp (1539765330757.000000)

               // camera 2, serial number 819312070204, polled at 30:814, frame counter is  215

               [10/17/18 16:35:30:814][8152 018664][D] [819312070204] polled frame[215] timestamp (1539765330756.000000)

               =>the frame counters are NOT aligned and not useful

             

            They counters will stay aligned but they will have an offset. In software you can "tar" this and subtract a fixed number from each. So if you subtract 172 from counter 1 and 215 from counter 2, then from then on they should both progress aligned as 1, 2, 3, 4, 5 etc.

            1 of 1 people found this helpful
            • 3. Re: Question about Frame Sync Index [D435][HW sync]
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hello shane7226107,

              Thank you very much for your patience in receiving an answer. 
              We noticed that you are using an outdated version of the Muticam Sync Whitepaper - you can find the updated one here: https://www.intel.com/content/www/us/en/support/articles/000028140/emerging-technologies/intel-realsense-technology.html 

              Please have a look at the following sections:

              F. System vs HW time

              G. HW sync validation

              H. Latency

              3. Multi-Camera Programming

              Let me know if you have any questions. 

              Best regards, 
              Casandra 
              • 4. Re: Question about Frame Sync Index [D435][HW sync]
                shane7226107

                Hi Casandra and Anders,

                 

                Thanks for your reply!

                 

                I have read through the whitepaper again, however there is still some thing I can not clearly understand.

                 

                The question is about boundary case as described below:

                 

                Assume that there is an offset of 3 between master and slave's frame counter (i.e. the sync frame pair from master is "N", slave's "N+3")

                 

                [Normal Case]

                normal_case.png

                in this normal case, I can subtract slave's frame counter by an offset of 3, after that the frame counter is sw aligned and can be used  as "frame sync index

                 

                [Boundary Case]

                boundary_case.png

                Since the polling threads are independent, it is not guaranteed to be perfectly executed at the same moment.

                 

                There is some chance boundary case happens like this, master polled a frame with frame counter "0", slave polled frame counter is "4"

                 

                If I subtract slave by a constant offset "4" afterward, it will obviously causing problems.

                 

                In this case, what can I do to handle it properly ?

                 

                Thank you!

                • 5. Re: Question about Frame Sync Index [D435][HW sync]
                  AndersGJ

                  You are absolutely correct.

                  There will be the boundary condition you mention, so best practice would be to read a few frames and select the median offset.

                  • 6. Re: Question about Frame Sync Index [D435][HW sync]
                    shane7226107

                    Hi AndersGJ,

                     

                    Thanks! I will use the method you have suggested.

                    • 7. Re: Question about Frame Sync Index [D435][HW sync]
                      shane7226107

                      Hi AndersGJ,

                       

                      I tried the method you have suggested, however the result is not what I expected(there should be one significant offset appears most of the time).

                       

                      It is often that the offsets "diverges", for example, If I took first 60 frames as reference,

                       

                      one of the statistic result is like below:

                       

                      offset (slave's frame count minus master)

                      number of appearance

                      -48

                      3

                      -47

                      2

                      -46

                      6

                      -45

                      5

                      -44

                      5

                      -43

                      4

                      -42

                      4

                      -41

                      1

                       

                      Is it reasonable?

                      • 8. Re: Question about Frame Sync Index [D435][HW sync]
                        AndersGJ

                        The queue has to be set to 1 (or two if using depth and RGB). By default it is 10. This means that if there is any latency in the host system you may get frames arriving at different times.

                        Also once you do this, you may drop a few frames (depending on your system/number of streams/resolutions/frame rates)

                        • 9. Re: Question about Frame Sync Index [D435][HW sync]
                          shane7226107

                          Hi AndersGJ,

                           

                          Thanks for replying!

                           

                          The whitepaper also has mentioned about the frame queue size, but it did not state clearly how to setup frame queue size.

                          (In page 9, it said about rs2_create_frame_queue...)

                           

                          Since I am using rs2::pipeline, I've check the source code of pipeline.cpp then find out that the queue size is by default set to "1"

                          librealsense/pipeline.cpp at master · IntelRealSense/librealsense · GitHub

                           

                          pipeline_processing_block::pipeline_processing_block(const std::vector<int>& streams_to_aggregate) :

                                  _queue(new single_consumer_frame_queue<frame_holder>(1)),

                           

                          I am not sure if we still need to do it manually?

                           

                          Or can you provide some sample code that describes how to setup frame queue size properly?

                           

                          Thank you very much!