Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....X................................X..................] duration: 5.973250385s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.973271737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....X................................X.................] duration: 5.737622173s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:29:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.737655327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X....X................................X................] duration: 5.858675135s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:28:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.858706204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X....X................................X...............] duration: 5.856340607s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:27:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.856370784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X....X................................X..............] duration: 5.816265105s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:26:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.816297367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X....X................................X.............] duration: 5.785933348s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:26:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.785964306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X....X................................X............] duration: 5.877471935s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:25:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.877503976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X....X................................X...........] duration: 5.847606805s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:24:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.847636812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X................................X..........] duration: 5.779560732s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:23:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.77964841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X................................X.........] duration: 5.782170853s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:23:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.782199478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X................................X........] duration: 5.864055186s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:22:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.864084561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X....X................................X.......] duration: 5.860002032s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:21:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.860131879s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X....X................................X......] duration: 5.840303788s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:20:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.840330099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X....X................................X.....] duration: 6.041160802s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:19:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 6.041202401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X....X................................X....] duration: 5.852419975s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:19:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.852457676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X....X................................X...] duration: 5.920669218s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:18:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.920711618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X....X................................X..] duration: 5.79051239s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:17:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.790541284s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X....X................................X.] duration: 5.826856324s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:17:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 5.826889518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X....X................................X] duration: 30.00362071s trace: https://jaeger.dev.storj.io/trace/3be417f27a24e0e1 FAIL (2025-11-29 16:16:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 16:15:48.306068573 (488h11m46.857539034s) [upload=429888]: Starting upload Nov 29 16:15:48.306133106 (488h11m46.857603557s) [upload=429888]: Done starting upload Nov 29 16:15:48.306334500 (488h11m46.857804961s) [upload=429888]: Uploading segments... Nov 29 16:15:48.306468054 (488h11m46.857938515s) [upload=429888]: Got next segment. Inline: false Nov 29 16:15:48.307065482 (488h11m46.858535943s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 16:15:48.307075881 (488h11m46.858546342s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 16:15:48.572138553 (488h11m47.123609014s) [seg_pos={0 0}, upload=429888]: Begin upload segment... Nov 29 16:15:48.572199558 (488h11m47.123670009s) [seg_pos={0 0}, upload=429888]: Done begin upload segment. Nov 29 16:15:48.572252149 (488h11m47.12372261s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572265003 (488h11m47.123735474s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.572735740 (488h11m47.124206191s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573157172 (488h11m47.124627633s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.573550541 (488h11m47.125021002s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.574066824 (488h11m47.125537285s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.575887261 (488h11m47.127357722s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.577526573 (488h11m47.128997024s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.578901570 (488h11m47.130372031s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.580825444 (488h11m47.132295906s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.586024163 (488h11m47.137494624s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.587129989 (488h11m47.13860045s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.589171576 (488h11m47.140642037s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.591247800 (488h11m47.142718251s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593224494 (488h11m47.144694955s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:48.593989230 (488h11m47.145459692s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Uploading piece... Nov 29 16:15:50.278965670 (488h11m48.830436132s) [upload=429888]: Next returned nil segment Nov 29 16:15:50.278984326 (488h11m48.830454788s) [upload=429888]: Waiting for error group managing segments... Nov 29 16:15:50.608656298 (488h11m49.160126769s) [noise=true , address=77.90.31.11:4555 , piece_id=BDKVFV2SAKRSIEWQP6GZDZ477H5BNN5O6SH5RHZIH32FCJAJPX2A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.659307709 (488h11m49.21077817s) [noise=true , address=77.90.31.12:4589 , piece_id=CBIIUFN2QJMG7BWMTBWZHYOHWXERY5NMEL3OWJND774MGIHBPWAA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.704201807 (488h11m49.255672268s) [noise=true , address=77.90.31.15:4699 , piece_id=M75SNAMC7WDW3PV2HAPAIJC2IL565R3VEFRTSA5DVHE77KIERQDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.722355675 (488h11m49.273826136s) [noise=true , address=145.79.227.10:4401 , piece_id=VKVHEE6R7JVWS3BBR5A2E5TPRHF7G5SXLLISTZ53PADOCDMKWPDA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.736386273 (488h11m49.287856724s) [noise=true , address=45.91.116.4:4937 , piece_id=LFT2TNJII64LMXYQGDG646XYL3RW2FL4RYTVECKQMH72TVR57VMA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750237959 (488h11m49.30170842s) [noise=true , address=145.79.227.14:4839 , piece_id=AJCCONIRTNPLN2U7AW4ERB3UNLZX3UA3BFZ6DUK4JOCRDQB75QPA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.750499367 (488h11m49.301969818s) [noise=true , address=145.79.227.39:4013 , piece_id=67EIO4VJDPHGT6BDFLZJ3NGEUDCMCVVI2UNUO7FOZNW5E3AUFNJA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.760354173 (488h11m49.311824644s) [noise=true , address=145.79.227.13:4803 , piece_id=MPTH7T5TX7JOCF4M5QJN774RY72IGRANWJFEJWFFUNKJW24DQW3A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.764807903 (488h11m49.316278374s) [noise=true , address=145.79.227.19:4079 , piece_id=CKSZGZ74AKQ36LP4CAT6KHO6QTMRIB5TJS56DFD5MM43MEQDEAXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.786597866 (488h11m49.338068327s) [noise=true , address=145.79.227.2:4114 , piece_id=RG6M2HF64XMML6IPVYMBOKFB5E36LALKZNJIEFUL4ONXWRTJ2HSQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.794547924 (488h11m49.346018385s) [noise=true , address=145.79.227.41:4085 , piece_id=5SJSJM7R2K423SNPSTHLZVERF26HQCLCGAKEDWZWK2QC6GCXRK7Q, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:50.797931226 (488h11m49.349401677s) [noise=true , address=145.79.227.22:4510 , piece_id=ETES3FF65TTA5PSYMQEYNSDULQ4XWKR36CG3FJC53LRPGBI7XBTQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461524433 (488h11m50.012994894s) [noise=true , address=145.79.227.16:4912 , piece_id=KLSNSA5VIJIZSSVXLHBLLWDYFLRMYJDUDHQMEPWIZHVHK77XFYAQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: <nil> Nov 29 16:15:51.461551976 (488h11m50.013022437s) [seg_pos={0 0}, upload=429888]: Segment reached optimal threshold of 13 pieces. Nov 29 16:15:51.461977686 (488h11m50.013448147s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.461995471 (488h11m50.013465932s) [noise=true , address=145.79.227.32:4834 , piece_id=LE3V2GUPFHJFXC6DZYXDFM6EQUDOL4YCYDBFDSWUGWLXNR6KVJIQ, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462148732 (488h11m50.013619193s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462160144 (488h11m50.013630615s) [noise=true , address=145.79.227.11:4729 , piece_id=BDIBOEECACPAUSKTIVPF2FIPZVQMYNP4GBSQTUKPH7JSQ7MQ4VXA, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462181164 (488h11m50.013651625s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Nov 29 16:15:51.462194249 (488h11m50.01366471s) [noise=true , address=145.79.227.8:4331 , piece_id=HZVG46ZTVD6HDR5UHEFZUBNYONBVOQITDNEDKRN34R4D6LGHU75A, seg_pos={0 0}, upload=429888]: Long tail cancellation due to optimal threshold Nov 29 16:15:51.462224877 (488h11m50.013695338s) [seg_pos={0 0}, upload=429888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 16:15:51.462325980 (488h11m50.013796441s) [upload=429888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 16:15:51.462332522 (488h11m50.013802983s) [upload=429888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 16:15:51.783425640 (488h11m50.334896121s) [upload=429888]: Done uploading segments... Nov 29 16:15:51.783497116 (488h11m50.334967577s) [upload=429888]: Upload finished. err: <nil> Duration: 30.003646009s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.682156766s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:16:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.682189787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.470422091s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:15:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470447711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.015848965s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:15:05 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.015876839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....X................................] duration: 5.888133506s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:15:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.888162963s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.41146582s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:14:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.41159687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.356660329s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:14:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.356678094s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration: 5.834404648s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:14:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.834436948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.06314166s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:14:14 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.063173491s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.097330021s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:13:30 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.097362073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 6.105170317s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:13:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.105206775s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.502831197s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:13:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502850185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.996436661s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:12:20 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.996466557s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.436538816s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:12:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.436569684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 5.750319755s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:12:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.750351325s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 4.369302137s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:12:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.369330831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.08219548s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:11:54 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.082226279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.99352773s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:11:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.993557327s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....X............................] duration: 5.719415587s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:11:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.71944841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.004201977s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:11:27 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.004234909s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.566245776s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:11:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566266355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration: 5.921389613s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.921420472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.673565127s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:10:59 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.673597398s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.42053002s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:10:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420559247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.991776156s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:10:26 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.991807235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.946986036s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:10:23 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.94701486s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 5.849332182s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:10:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.849365666s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 5.856672844s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:09:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.856700485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.026145081s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:09:36 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.026171311s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.366308244s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:09:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.366338712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.068013495s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:09:21 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.068044235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.759924099s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:09:10 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.759958154s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)