Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.424782892s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:54:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424811957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.72909453s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:54:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.729183088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.02644071s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:54:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 6.026584022s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.505915156s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:53:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505944331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.859343077s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:53:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.859480048s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.451556598s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:52:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.451588498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.775778697s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:51:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.775809376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.741072664s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:51:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.741099506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.320439883s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:51:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.320469218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.817593341s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:50:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.817625221s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.546568248s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:50:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546596842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.507389966s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:49:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507409592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 7.264539564s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 7.264570011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.333810594s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:49:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.333837914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 8.639153756s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:49:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 8.639185777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.93510936s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:49:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.935132893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.774489873s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.774508639s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.422567895s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:48:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.422595167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.039074818s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:47:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 6.039096969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.679554545s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:47:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.679574043s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.54763645s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:47:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54766215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.797327431s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:46:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.797394189s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 4.501223669s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:46:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.501272542s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001034125s trace: https://jaeger.dev.storj.io/trace/24af5fa3f4d6cd8c FAIL (2025-11-29 16:45:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001073691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 9.627815919s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:45:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 9.627869061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.99316756s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:44:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.993193801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.796765756s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:44:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.796801394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.792334421s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:43:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.792362293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.847167468s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.847359172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.981569768s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:41:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.981603001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.948551792s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:40:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.948626175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.820363978s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:40:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.820397271s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.851601411s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:39:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.851640185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.759769176s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:38:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.759786639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.79542939s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:37:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.795457363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X................................X..........................] duration: 5.900778806s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:36:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.900807261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X................................X.........................] duration: 5.941903768s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:35:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.9419426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................................X........................] duration: 5.76702502s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:35:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.767069122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X................................X.......................] duration: 5.824356182s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:33:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.824383142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X................................X......................] duration: 6.264766219s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:32:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 6.264818349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X....X................................X.....................] duration: 6.002246259s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:31:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 6.002299902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....X................................X....................] duration: 8.822733334s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:31:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 8.822766197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....X................................X...................] duration: 5.927738361s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:30:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.927771323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....X................................X..................] duration: 5.973250385s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.973271737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....X................................X.................] duration: 5.737622173s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:29:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.737655327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X....X................................X................] duration: 5.858675135s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:28:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.858706204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X....X................................X...............] duration: 5.856340607s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:27:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.856370784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X....X................................X..............] duration: 5.816265105s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:26:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.816297367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X....X................................X.............] duration: 5.785933348s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:26:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.785964306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X....X................................X............] duration: 5.877471935s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:25:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.877503976s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)