Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..........] duration: 6.623137713s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:49:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 6.623156809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X.........] duration: 28.584272516s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:48:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 28.584288236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 28.021266664s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:48:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 28.021296742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X........] duration: 6.17259892s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 6.172623778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 15.346258351s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:47:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 15.346285292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X.......] duration: 5.972261642s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:47:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 5.972289165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 24.107725424s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:47:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 24.107751975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X......] duration: 8.322946404s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:47:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 8.322968979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X.....] duration: 6.305248107s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:46:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 6.305271423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 15.39182576s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:46:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 15.391859323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X....] duration: 6.075657226s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:45:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 6.075678607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 19.042730105s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:45:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 19.042762896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X...] duration: 6.131311782s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:44:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 6.131329507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 24.843957932s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:44:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 24.843970945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..] duration: 7.113055425s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:44:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 7.11309073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 53.344257132s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:44:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 53.344281989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X.] duration: 13.290486962s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:43:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 13.290511308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.004025823s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:43:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 1m0.004047184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X] duration: 30.002082576s trace: https://jaeger.dev.storj.io/trace/5030a360a68c7b58 FAIL (2025-12-26 19:42:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA) context deadline exceeded ======LOGS====== :42:14.690438849 (508h13m14.561521894s) [upload=445975]: Starting upload Dec 26 19:42:14.690541144 (508h13m14.561624179s) [upload=445975]: Done starting upload Dec 26 19:42:14.690774407 (508h13m14.561857462s) [upload=445975]: Uploading segments... Dec 26 19:42:14.690923582 (508h13m14.562006617s) [upload=445975]: Got next segment. Inline: false Dec 26 19:42:14.690980991 (508h13m14.562064036s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:42:14.690987333 (508h13m14.562070368s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:42:14.942766084 (508h13m14.813849129s) [seg_pos={0 0}, upload=445975]: Begin upload segment... Dec 26 19:42:14.942828693 (508h13m14.813911748s) [seg_pos={0 0}, upload=445975]: Done begin upload segment. Dec 26 19:42:14.942877907 (508h13m14.813960952s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.942927852 (508h13m14.814010897s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943368180 (508h13m14.814451225s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.943786156 (508h13m14.814869201s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944206767 (508h13m14.815289812s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.944649330 (508h13m14.815732365s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945095580 (508h13m14.816178625s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.945536769 (508h13m14.816619824s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.946943318 (508h13m14.818026353s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.947140994 (508h13m14.818224039s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.948177227 (508h13m14.819260262s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.949649390 (508h13m14.820732506s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.952008092 (508h13m14.823091137s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.953650950 (508h13m14.824733995s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.954926569 (508h13m14.826009605s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:14.956133618 (508h13m14.827216653s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Uploading piece... Dec 26 19:42:16.038436069 (508h13m15.909519114s) [noise=true , address=77.90.31.12:4589 , piece_id=VYQSBI6MNWYBF3ZD3OV7I3YMWXD7OZXFQNEFIKCGWSSFZ5WXIKIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 19:42:16.671757773 (508h13m16.542840808s) [upload=445975]: Next returned nil segment Dec 26 19:42:16.671795266 (508h13m16.542878311s) [upload=445975]: Waiting for error group managing segments... Dec 26 19:42:16.945422991 (508h13m16.816506026s) [noise=true , address=145.79.227.31:4799 , piece_id=SNTRRSMITATJMIMX4ZBWZMAHNXHHMS7IE3ERQK6WQDJLIPPURDGA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.953393567 (508h13m16.824476602s) [noise=true , address=145.79.227.42:4122 , piece_id=SWSNK7ZX527KFGI3KWAITZXEAYMYSOYGHKCVV2WMJV62XUSWHCHQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.959801991 (508h13m16.830885036s) [noise=true , address=145.79.227.45:4229 , piece_id=6AH7ZKNYF7ULUH2XSUDPYTE7TVZPMWVMM7QZQLNX3B4RWZDAJ4MQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.972763061 (508h13m16.843846106s) [noise=true , address=145.79.227.39:4015 , piece_id=BXD4KVQXUNVRBQQWR2KC5L2HMF4O44SNDSSY6LZTPSYRC4MQ3VQA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.978419741 (508h13m16.849502786s) [noise=true , address=145.79.227.49:4338 , piece_id=5DORCGT7OUV5U2JDFKPZSE6LCAARAHYQF6ZMNWIQJYMF2ID7U3IA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:16.986205257 (508h13m16.857288302s) [noise=true , address=145.79.227.37:4566 , piece_id=VZPYRXBJ364PWUGPJ6Z4MYVUNF7XGAAWY4O465AXOPSYKFJPMUKQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.103484857 (508h13m16.974567892s) [noise=true , address=145.79.227.16:4912 , piece_id=NBTRNJLNULJBFI7L7BKEKGOTNSPVRFXFD3IRA6Y4N4HWEJHVBJVQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.177267022 (508h13m17.048350077s) [noise=true , address=77.90.31.19:4844 , piece_id=ATJ6TVBZ4LRF4RHEAF6BH7N264BUX5G7IXJPMNZ7IXQK4ECQEAPA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.315631081 (508h13m17.186714116s) [noise=true , address=45.91.116.9:4264 , piece_id=OK7HWZHDZDNIOKFTQNXI74ZWUMMS6I4LUZTGIYXDLNXL5DHBW24Q, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.331967127 (508h13m17.203050162s) [noise=true , address=45.91.116.11:4117 , piece_id=5GKE342LWLYBQZK7HWY2TABJMGLR44SKE5KPUE4ODHW2RR2PDKEA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.373993253 (508h13m17.245076288s) [noise=true , address=45.91.116.5:4298 , piece_id=T7GCGLE6DYTP3NQ64QZNGSRIYSVNVIUB2ILQ244VBNOQQXOGF7XQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:17.912637535 (508h13m17.78372058s) [noise=true , address=45.91.116.14:4192 , piece_id=KWTM2ROCDHDOAURHAEFR34WQOWBHE6Q4JEN7ENBGFLM23QMCOUMA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911466433 (508h13m19.782549468s) [noise=true , address=145.79.227.4:4186 , piece_id=P5I7RQ7TV7D7YZ6YEXPAFJ2EONGNUGKK7J572SEUVEYJ2GL3DSFA, seg_pos={0 0}, upload=445975]: Done uploading piece. err: <nil> Dec 26 19:42:19.911488685 (508h13m19.78257173s) [seg_pos={0 0}, upload=445975]: Segment reached optimal threshold of 13 pieces. Dec 26 19:42:19.911777876 (508h13m19.782860911s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41684->145.79.227.15:4873: use of closed network connection Dec 26 19:42:19.911788126 (508h13m19.782871161s) [noise=true , address=145.79.227.15:4873 , piece_id=QD7EDQQ4EO4KX5KS7Q3HHWLSCIGKNFQH3WRQGY6PBQ3KMTUXUROQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911899418 (508h13m19.782982473s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55800->145.79.227.5:4223: use of closed network connection Dec 26 19:42:19.911918804 (508h13m19.783001839s) [noise=true , address=145.79.227.5:4223 , piece_id=M4MHZ66LJE4WYVTUCBHRO56QHY57ZIPXINTFKCS6XXP4YMOEKJIQ, seg_pos={0 0}, upload=445975]: Long tail cancellation due to optimal threshold Dec 26 19:42:19.911952929 (508h13m19.783035974s) [seg_pos={0 0}, upload=445975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:42:19.912086614 (508h13m19.783169649s) [upload=445975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:19.912095792 (508h13m19.783178837s) [upload=445975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:20.202519429 (508h13m20.073602464s) [upload=445975]: Done uploading segments... Dec 26 19:42:20.202537142 (508h13m20.073620177s) [upload=445975]: Upload finished. err: <nil> Duration: 30.002109698s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.] duration: 6.20380862s trace: https://jaeger.dev.storj.io/trace/4454d09632c3e462 FAIL (2025-12-26 19:41:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CSRMEJ2IHZV7OXGK35YLAU2GWNAXL2FYVMXFFPU4GOQJDEZW27YA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: NTBR3HRILEHFGAXYC4DIGGKCKORMRGDZDDQZ6W2R25IAPL4POFKQ) context deadline exceeded; piecestore: (Node ID: 12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, Piece ID: LRVVAD24EXN2T2LASF3BDUYVESWVYYAB5VKRD4MG5LQQFTVUS5TQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: F3THXHQB7TQ3FCEG22L47UO75FFVN2FIHAEQC77YTBFMLAE3UBXQ) context deadline exceeded ======LOGS====== s... Dec 26 19:40:12.453431516 (508h11m12.324514551s) [upload=445942]: Got next segment. Inline: false Dec 26 19:40:12.453469258 (508h11m12.324552303s) [upload=445942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:40:12.453473796 (508h11m12.324556841s) [upload=445942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:40:12.723821886 (508h11m12.594904932s) [seg_pos={0 0}, upload=445942]: Begin upload segment... Dec 26 19:40:12.723885728 (508h11m12.594968773s) [seg_pos={0 0}, upload=445942]: Done begin upload segment. Dec 26 19:40:12.723965460 (508h11m12.595048505s) [noise=true , address=145.79.227.17:4972 , piece_id=LRVVAD24EXN2T2LASF3BDUYVESWVYYAB5VKRD4MG5LQQFTVUS5TQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.723981590 (508h11m12.595064635s) [noise=true , address=77.90.31.12:4590 , piece_id=J2I54RC5Z34O3QRIEO334OKIDJQ5R6KXE7HOFMYSHMTLLMYHFQCQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.724727992 (508h11m12.595811037s) [noise=true , address=145.79.227.8:4330 , piece_id=7BGDGECQBT3J3KYWPA4UI2GIGHGLYSJTSTJEIB5URTRFLCYZBULQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.724739954 (508h11m12.595822999s) [noise=true , address=145.79.227.47:4266 , piece_id=3WHKDAXLEIXOFWOQUDIKPXKXE43LZMEOGTWLQIPEEVJDJT47WNLA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725135638 (508h11m12.596218683s) [noise=true , address=145.79.227.51:4409 , piece_id=3F64FBYR3ZZZDGSHXEYWKYKK5YQXX6HLJ3E5EPB6AXLKBCYKANSA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725217374 (508h11m12.596300419s) [noise=true , address=145.79.227.15:4874 , piece_id=B32MKQQDWDDXCNLVFYLD2PTYYLVYNIGSKJKKRND4KXAHCXHXPCUA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725501765 (508h11m12.59658481s) [noise=true , address=45.91.116.9:4261 , piece_id=CSRMEJ2IHZV7OXGK35YLAU2GWNAXL2FYVMXFFPU4GOQJDEZW27YA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725668713 (508h11m12.596751748s) [noise=true , address=145.79.227.48:4303 , piece_id=FS4MW576XGMYN3NEHJQB4AVTROHNBEF2H6GVBTY26373AGAOQWQA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725955920 (508h11m12.597038965s) [noise=true , address=145.79.227.52:4447 , piece_id=FTXPQ6JRWTX7EDKDSXPZZZHN56WWWDL5U7VN4D7YD2FQWVYVTX6A, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.726354790 (508h11m12.597437835s) [noise=true , address=45.91.116.13:4156 , piece_id=U2SKC4THXFFQE3QPONJ5GB2I4VUD5EV2TIUHX72PYYOM7TY32VLQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.726856455 (508h11m12.5979395s) [noise=true , address=45.91.116.7:4009 , piece_id=NTBR3HRILEHFGAXYC4DIGGKCKORMRGDZDDQZ6W2R25IAPL4POFKQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.727348091 (508h11m12.598431136s) [noise=true , address=145.79.227.16:4909 , piece_id=RHPCBUHV3NFE454BMCVD5BOO3NBNYATWXKDUYX5LOCRRB4N347YA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.727760406 (508h11m12.598843451s) [noise=true , address=145.79.227.42:4123 , piece_id=IXYHOV6UJD5MWW3XEJYEAPWP5QWBL3EH35UL7AISKLANSSUAMXRA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.728168283 (508h11m12.599251328s) [noise=true , address=45.91.116.10:4082 , piece_id=F3THXHQB7TQ3FCEG22L47UO75FFVN2FIHAEQC77YTBFMLAE3UBXQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.730907117 (508h11m12.601990193s) [noise=true , address=145.79.227.36:4518 , piece_id=7BZ2SU4HMSW5DSYFIR7YQUKHMQIREMKEF5YDQ6OYYEY4ECHJWIBQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.732132351 (508h11m12.603215396s) [noise=true , address=145.79.227.5:4223 , piece_id=TJP5KMH2KKZCFBA73MDCKMNYJNLND5INS25SK222W4MFXPLFMAFQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:14.440577007 (508h11m14.311660042s) [upload=445942]: Next returned nil segment Dec 26 19:40:14.440593539 (508h11m14.311676574s) [upload=445942]: Waiting for error group managing segments... Dec 26 19:40:14.729047165 (508h11m14.60013021s) [noise=true , address=145.79.227.47:4266 , piece_id=3WHKDAXLEIXOFWOQUDIKPXKXE43LZMEOGTWLQIPEEVJDJT47WNLA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.730436661 (508h11m14.601519706s) [noise=true , address=145.79.227.42:4123 , piece_id=IXYHOV6UJD5MWW3XEJYEAPWP5QWBL3EH35UL7AISKLANSSUAMXRA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.732146236 (508h11m14.603229271s) [noise=true , address=145.79.227.36:4518 , piece_id=7BZ2SU4HMSW5DSYFIR7YQUKHMQIREMKEF5YDQ6OYYEY4ECHJWIBQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.748136504 (508h11m14.619219539s) [noise=true , address=145.79.227.48:4303 , piece_id=FS4MW576XGMYN3NEHJQB4AVTROHNBEF2H6GVBTY26373AGAOQWQA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.759337182 (508h11m14.630420227s) [noise=true , address=145.79.227.52:4447 , piece_id=FTXPQ6JRWTX7EDKDSXPZZZHN56WWWDL5U7VN4D7YD2FQWVYVTX6A, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.762211204 (508h11m14.633294249s) [noise=true , address=145.79.227.15:4874 , piece_id=B32MKQQDWDDXCNLVFYLD2PTYYLVYNIGSKJKKRND4KXAHCXHXPCUA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.767878376 (508h11m14.638961411s) [noise=true , address=145.79.227.8:4330 , piece_id=7BGDGECQBT3J3KYWPA4UI2GIGHGLYSJTSTJEIB5URTRFLCYZBULQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.774795467 (508h11m14.645878512s) [noise=true , address=145.79.227.16:4909 , piece_id=RHPCBUHV3NFE454BMCVD5BOO3NBNYATWXKDUYX5LOCRRB4N347YA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.789083313 (508h11m14.660166359s) [noise=true , address=145.79.227.17:4972 , piece_id=LRVVAD24EXN2T2LASF3BDUYVESWVYYAB5VKRD4MG5LQQFTVUS5TQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.978539702 (508h11m14.849622737s) [noise=true , address=45.91.116.7:4009 , piece_id=NTBR3HRILEHFGAXYC4DIGGKCKORMRGDZDDQZ6W2R25IAPL4POFKQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:15.030121352 (508h11m14.901204387s) [noise=true , address=45.91.116.9:4261 , piece_id=CSRMEJ2IHZV7OXGK35YLAU2GWNAXL2FYVMXFFPU4GOQJDEZW27YA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:15.031543190 (508h11m14.902626225s) [noise=true , address=45.91.116.10:4082 , piece_id=F3THXHQB7TQ3FCEG22L47UO75FFVN2FIHAEQC77YTBFMLAE3UBXQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:15.099602837 (508h11m14.970685912s) [noise=true , address=77.90.31.12:4590 , piece_id=J2I54RC5Z34O3QRIEO334OKIDJQ5R6KXE7HOFMYSHMTLLMYHFQCQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:15.099632985 (508h11m14.97071602s) [seg_pos={0 0}, upload=445942]: Segment reached optimal threshold of 13 pieces. Dec 26 19:40:15.099933887 (508h11m14.971016922s) [noise=true , address=45.91.116.13:4156 , piece_id=U2SKC4THXFFQE3QPONJ5GB2I4VUD5EV2TIUHX72PYYOM7TY32VLQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 19:40:15.099941833 (508h11m14.971024868s) [noise=true , address=45.91.116.13:4156 , piece_id=U2SKC4THXFFQE3QPONJ5GB2I4VUD5EV2TIUHX72PYYOM7TY32VLQ, seg_pos={0 0}, upload=445942]: Long tail cancellation due to optimal threshold Dec 26 19:40:15.100020923 (508h11m14.971103958s) [noise=true , address=145.79.227.51:4409 , piece_id=3F64FBYR3ZZZDGSHXEYWKYKK5YQXX6HLJ3E5EPB6AXLKBCYKANSA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:53674->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:53674->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53674->145.79.227.51:4409: use of closed network connection Dec 26 19:40:15.100037093 (508h11m14.971120139s) [noise=true , address=145.79.227.51:4409 , piece_id=3F64FBYR3ZZZDGSHXEYWKYKK5YQXX6HLJ3E5EPB6AXLKBCYKANSA, seg_pos={0 0}, upload=445942]: Long tail cancellation due to optimal threshold Dec 26 19:40:15.100085957 (508h11m14.971169002s) [noise=true , address=145.79.227.5:4223 , piece_id=TJP5KMH2KKZCFBA73MDCKMNYJNLND5INS25SK222W4MFXPLFMAFQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:38712->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:38712->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38712->145.79.227.5:4223: use of closed network connection Dec 26 19:40:15.100095074 (508h11m14.971178109s) [noise=true , address=145.79.227.5:4223 , piece_id=TJP5KMH2KKZCFBA73MDCKMNYJNLND5INS25SK222W4MFXPLFMAFQ, seg_pos={0 0}, upload=445942]: Long tail cancellation due to optimal threshold Dec 26 19:40:15.100115504 (508h11m14.971198549s) [seg_pos={0 0}, upload=445942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:40:15.100201417 (508h11m14.971284462s) [upload=445942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:40:15.100210093 (508h11m14.971293138s) [upload=445942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:40:15.405734425 (508h11m15.27681749s) [upload=445942]: Done uploading segments... Dec 26 19:40:15.405774892 (508h11m15.276857937s) [upload=445942]: Upload finished. err: <nil> Duration: 6.203833967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X] duration: 30.002639939s trace: https://jaeger.dev.storj.io/trace/4454d09632c3e462 FAIL (2025-12-26 19:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CSRMEJ2IHZV7OXGK35YLAU2GWNAXL2FYVMXFFPU4GOQJDEZW27YA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: NTBR3HRILEHFGAXYC4DIGGKCKORMRGDZDDQZ6W2R25IAPL4POFKQ) context deadline exceeded; piecestore: (Node ID: 12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, Piece ID: LRVVAD24EXN2T2LASF3BDUYVESWVYYAB5VKRD4MG5LQQFTVUS5TQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: F3THXHQB7TQ3FCEG22L47UO75FFVN2FIHAEQC77YTBFMLAE3UBXQ) context deadline exceeded ======LOGS====== s... Dec 26 19:40:12.453431516 (508h11m12.324514551s) [upload=445942]: Got next segment. Inline: false Dec 26 19:40:12.453469258 (508h11m12.324552303s) [upload=445942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:40:12.453473796 (508h11m12.324556841s) [upload=445942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:40:12.723821886 (508h11m12.594904932s) [seg_pos={0 0}, upload=445942]: Begin upload segment... Dec 26 19:40:12.723885728 (508h11m12.594968773s) [seg_pos={0 0}, upload=445942]: Done begin upload segment. Dec 26 19:40:12.723965460 (508h11m12.595048505s) [noise=true , address=145.79.227.17:4972 , piece_id=LRVVAD24EXN2T2LASF3BDUYVESWVYYAB5VKRD4MG5LQQFTVUS5TQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.723981590 (508h11m12.595064635s) [noise=true , address=77.90.31.12:4590 , piece_id=J2I54RC5Z34O3QRIEO334OKIDJQ5R6KXE7HOFMYSHMTLLMYHFQCQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.724727992 (508h11m12.595811037s) [noise=true , address=145.79.227.8:4330 , piece_id=7BGDGECQBT3J3KYWPA4UI2GIGHGLYSJTSTJEIB5URTRFLCYZBULQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.724739954 (508h11m12.595822999s) [noise=true , address=145.79.227.47:4266 , piece_id=3WHKDAXLEIXOFWOQUDIKPXKXE43LZMEOGTWLQIPEEVJDJT47WNLA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725135638 (508h11m12.596218683s) [noise=true , address=145.79.227.51:4409 , piece_id=3F64FBYR3ZZZDGSHXEYWKYKK5YQXX6HLJ3E5EPB6AXLKBCYKANSA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725217374 (508h11m12.596300419s) [noise=true , address=145.79.227.15:4874 , piece_id=B32MKQQDWDDXCNLVFYLD2PTYYLVYNIGSKJKKRND4KXAHCXHXPCUA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725501765 (508h11m12.59658481s) [noise=true , address=45.91.116.9:4261 , piece_id=CSRMEJ2IHZV7OXGK35YLAU2GWNAXL2FYVMXFFPU4GOQJDEZW27YA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725668713 (508h11m12.596751748s) [noise=true , address=145.79.227.48:4303 , piece_id=FS4MW576XGMYN3NEHJQB4AVTROHNBEF2H6GVBTY26373AGAOQWQA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.725955920 (508h11m12.597038965s) [noise=true , address=145.79.227.52:4447 , piece_id=FTXPQ6JRWTX7EDKDSXPZZZHN56WWWDL5U7VN4D7YD2FQWVYVTX6A, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.726354790 (508h11m12.597437835s) [noise=true , address=45.91.116.13:4156 , piece_id=U2SKC4THXFFQE3QPONJ5GB2I4VUD5EV2TIUHX72PYYOM7TY32VLQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.726856455 (508h11m12.5979395s) [noise=true , address=45.91.116.7:4009 , piece_id=NTBR3HRILEHFGAXYC4DIGGKCKORMRGDZDDQZ6W2R25IAPL4POFKQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.727348091 (508h11m12.598431136s) [noise=true , address=145.79.227.16:4909 , piece_id=RHPCBUHV3NFE454BMCVD5BOO3NBNYATWXKDUYX5LOCRRB4N347YA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.727760406 (508h11m12.598843451s) [noise=true , address=145.79.227.42:4123 , piece_id=IXYHOV6UJD5MWW3XEJYEAPWP5QWBL3EH35UL7AISKLANSSUAMXRA, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.728168283 (508h11m12.599251328s) [noise=true , address=45.91.116.10:4082 , piece_id=F3THXHQB7TQ3FCEG22L47UO75FFVN2FIHAEQC77YTBFMLAE3UBXQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.730907117 (508h11m12.601990193s) [noise=true , address=145.79.227.36:4518 , piece_id=7BZ2SU4HMSW5DSYFIR7YQUKHMQIREMKEF5YDQ6OYYEY4ECHJWIBQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:12.732132351 (508h11m12.603215396s) [noise=true , address=145.79.227.5:4223 , piece_id=TJP5KMH2KKZCFBA73MDCKMNYJNLND5INS25SK222W4MFXPLFMAFQ, seg_pos={0 0}, upload=445942]: Uploading piece... Dec 26 19:40:14.440577007 (508h11m14.311660042s) [upload=445942]: Next returned nil segment Dec 26 19:40:14.440593539 (508h11m14.311676574s) [upload=445942]: Waiting for error group managing segments... Dec 26 19:40:14.729047165 (508h11m14.60013021s) [noise=true , address=145.79.227.47:4266 , piece_id=3WHKDAXLEIXOFWOQUDIKPXKXE43LZMEOGTWLQIPEEVJDJT47WNLA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.730436661 (508h11m14.601519706s) [noise=true , address=145.79.227.42:4123 , piece_id=IXYHOV6UJD5MWW3XEJYEAPWP5QWBL3EH35UL7AISKLANSSUAMXRA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.732146236 (508h11m14.603229271s) [noise=true , address=145.79.227.36:4518 , piece_id=7BZ2SU4HMSW5DSYFIR7YQUKHMQIREMKEF5YDQ6OYYEY4ECHJWIBQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.748136504 (508h11m14.619219539s) [noise=true , address=145.79.227.48:4303 , piece_id=FS4MW576XGMYN3NEHJQB4AVTROHNBEF2H6GVBTY26373AGAOQWQA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.759337182 (508h11m14.630420227s) [noise=true , address=145.79.227.52:4447 , piece_id=FTXPQ6JRWTX7EDKDSXPZZZHN56WWWDL5U7VN4D7YD2FQWVYVTX6A, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.762211204 (508h11m14.633294249s) [noise=true , address=145.79.227.15:4874 , piece_id=B32MKQQDWDDXCNLVFYLD2PTYYLVYNIGSKJKKRND4KXAHCXHXPCUA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.767878376 (508h11m14.638961411s) [noise=true , address=145.79.227.8:4330 , piece_id=7BGDGECQBT3J3KYWPA4UI2GIGHGLYSJTSTJEIB5URTRFLCYZBULQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.774795467 (508h11m14.645878512s) [noise=true , address=145.79.227.16:4909 , piece_id=RHPCBUHV3NFE454BMCVD5BOO3NBNYATWXKDUYX5LOCRRB4N347YA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.789083313 (508h11m14.660166359s) [noise=true , address=145.79.227.17:4972 , piece_id=LRVVAD24EXN2T2LASF3BDUYVESWVYYAB5VKRD4MG5LQQFTVUS5TQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:14.978539702 (508h11m14.849622737s) [noise=true , address=45.91.116.7:4009 , piece_id=NTBR3HRILEHFGAXYC4DIGGKCKORMRGDZDDQZ6W2R25IAPL4POFKQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:15.030121352 (508h11m14.901204387s) [noise=true , address=45.91.116.9:4261 , piece_id=CSRMEJ2IHZV7OXGK35YLAU2GWNAXL2FYVMXFFPU4GOQJDEZW27YA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:15.031543190 (508h11m14.902626225s) [noise=true , address=45.91.116.10:4082 , piece_id=F3THXHQB7TQ3FCEG22L47UO75FFVN2FIHAEQC77YTBFMLAE3UBXQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:15.099602837 (508h11m14.970685912s) [noise=true , address=77.90.31.12:4590 , piece_id=J2I54RC5Z34O3QRIEO334OKIDJQ5R6KXE7HOFMYSHMTLLMYHFQCQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: <nil> Dec 26 19:40:15.099632985 (508h11m14.97071602s) [seg_pos={0 0}, upload=445942]: Segment reached optimal threshold of 13 pieces. Dec 26 19:40:15.099933887 (508h11m14.971016922s) [noise=true , address=45.91.116.13:4156 , piece_id=U2SKC4THXFFQE3QPONJ5GB2I4VUD5EV2TIUHX72PYYOM7TY32VLQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 19:40:15.099941833 (508h11m14.971024868s) [noise=true , address=45.91.116.13:4156 , piece_id=U2SKC4THXFFQE3QPONJ5GB2I4VUD5EV2TIUHX72PYYOM7TY32VLQ, seg_pos={0 0}, upload=445942]: Long tail cancellation due to optimal threshold Dec 26 19:40:15.100020923 (508h11m14.971103958s) [noise=true , address=145.79.227.51:4409 , piece_id=3F64FBYR3ZZZDGSHXEYWKYKK5YQXX6HLJ3E5EPB6AXLKBCYKANSA, seg_pos={0 0}, upload=445942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:53674->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:53674->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53674->145.79.227.51:4409: use of closed network connection Dec 26 19:40:15.100037093 (508h11m14.971120139s) [noise=true , address=145.79.227.51:4409 , piece_id=3F64FBYR3ZZZDGSHXEYWKYKK5YQXX6HLJ3E5EPB6AXLKBCYKANSA, seg_pos={0 0}, upload=445942]: Long tail cancellation due to optimal threshold Dec 26 19:40:15.100085957 (508h11m14.971169002s) [noise=true , address=145.79.227.5:4223 , piece_id=TJP5KMH2KKZCFBA73MDCKMNYJNLND5INS25SK222W4MFXPLFMAFQ, seg_pos={0 0}, upload=445942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:38712->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:38712->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38712->145.79.227.5:4223: use of closed network connection Dec 26 19:40:15.100095074 (508h11m14.971178109s) [noise=true , address=145.79.227.5:4223 , piece_id=TJP5KMH2KKZCFBA73MDCKMNYJNLND5INS25SK222W4MFXPLFMAFQ, seg_pos={0 0}, upload=445942]: Long tail cancellation due to optimal threshold Dec 26 19:40:15.100115504 (508h11m14.971198549s) [seg_pos={0 0}, upload=445942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:40:15.100201417 (508h11m14.971284462s) [upload=445942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:40:15.100210093 (508h11m14.971293138s) [upload=445942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:40:15.405734425 (508h11m15.27681749s) [upload=445942]: Done uploading segments... Dec 26 19:40:15.405774892 (508h11m15.276857937s) [upload=445942]: Upload finished. err: <nil> Duration: 30.002664916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.] duration: 9.100038544s trace: https://jaeger.dev.storj.io/trace/1aa177e6ebb6bf4c FAIL (2025-12-26 19:40:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (508h10m21.768427653s) [upload=445930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:39:22.139166048 (508h10m22.010249093s) [seg_pos={0 0}, upload=445930]: Begin upload segment... Dec 26 19:39:22.139216585 (508h10m22.01029962s) [seg_pos={0 0}, upload=445930]: Done begin upload segment. Dec 26 19:39:22.139237585 (508h10m22.01032063s) [noise=true , address=45.91.116.10:4082 , piece_id=352FDNDNSH2MHX2ST3S2JOBLG4LIZ65OW6F6BYXQ25DHIRBDQCEQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.139250239 (508h10m22.010333274s) [noise=true , address=145.79.227.49:4337 , piece_id=FWAQ4WCFISA5SZA52W3DUMOV7HTSAVOJE7GEZOMAXVBWAGWXIGQQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.139329930 (508h10m22.010412975s) [noise=true , address=145.79.227.52:4447 , piece_id=2LYG3HPVFSR3GAVEXLNH2MN4Q5CID5PYFZ3ELOCYNF7D5S2BTMMA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.140393816 (508h10m22.011476851s) [noise=true , address=77.90.31.10:4520 , piece_id=UWSP7FFYKNSYDTAWQ7MTU7UYCLETA5WIY75R7Q3HDAY6R7YVJLGQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.140723884 (508h10m22.01180692s) [noise=true , address=45.91.116.11:4120 , piece_id=IGQECIJPS6CNFG67YYQ7FXICZQEAVRE57TXXPCYSAAHF6WESZ53Q, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.141081615 (508h10m22.01216465s) [noise=true , address=145.79.227.46:4603 , piece_id=PJKC22D4PBWBPCAWTL2AJZO3CFRGPDP7QSHM2I3W6QGLIH25JVTQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.141457541 (508h10m22.012540576s) [noise=true , address=145.79.227.45:4231 , piece_id=HEF7ND4PYBOPBHKFNOUTEBGJI4GRYBNHAXNOTOL35FFROF4S3PRA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.141532143 (508h10m22.012615178s) [noise=true , address=45.91.116.8:4046 , piece_id=AXCGJD4TN5KLW7Y5KPX6I36T7AG2JCIE2GFUUR4CYGWOYDCQPF4Q, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.142744081 (508h10m22.013827116s) [noise=true , address=145.79.227.11:4731 , piece_id=VV5E3DSDX3KW4NUJHNDT2UIGETTUG5CDRGM4LSU6JPHAPZXSQB5Q, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.142945826 (508h10m22.014028871s) [noise=true , address=45.91.116.7:4010 , piece_id=X4GUO3T4IEUGHBGSHOY5VIQI3L5SU6UGAKJ3Y5PUCEIST4KUDPNA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.143362950 (508h10m22.014445995s) [noise=true , address=145.79.227.36:4517 , piece_id=6TKKAJGLD6FSCF6VSPF6X5PVRICWALBKQG5SYSLJ7LJLODZH235Q, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.143754455 (508h10m22.01483749s) [noise=true , address=77.90.31.17:4773 , piece_id=6SQWHSXYLOB2DF42A24GZETDY7NN5ILXXTACYXOETJWMVEPUGPMA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.144257123 (508h10m22.015340158s) [noise=true , address=145.79.227.15:4873 , piece_id=JYLTCRAX6IE22DXIPKUCCCWGCWDQVLRTHDBRQPFBNT2SJ753EUCA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.144552134 (508h10m22.015635169s) [noise=true , address=145.79.227.13:4803 , piece_id=DGGG77IF54A4TYKJCV356HWDHASWIJGGJI7KXZZCD2TKQI2X6EGA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.145075411 (508h10m22.016158446s) [noise=true , address=145.79.227.43:4158 , piece_id=42ALENTRHVD4KZYGWJAEYL437QSCBACQ6YQKZXMGUGHG6UNPMPNQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.145387695 (508h10m22.01647073s) [noise=true , address=145.79.227.51:4410 , piece_id=U5DE42Y7N3APTAHVC57AYTPIK57Q7XJ2OA7AYLEQRKOBRFUX26TA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:23.857494574 (508h10m23.728577609s) [upload=445930]: Next returned nil segment Dec 26 19:39:23.857509413 (508h10m23.728592448s) [upload=445930]: Waiting for error group managing segments... Dec 26 19:39:24.141130116 (508h10m24.012213151s) [noise=true , address=145.79.227.46:4603 , piece_id=PJKC22D4PBWBPCAWTL2AJZO3CFRGPDP7QSHM2I3W6QGLIH25JVTQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.142114731 (508h10m24.013197766s) [noise=true , address=145.79.227.36:4517 , piece_id=6TKKAJGLD6FSCF6VSPF6X5PVRICWALBKQG5SYSLJ7LJLODZH235Q, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.147973206 (508h10m24.019056241s) [noise=true , address=145.79.227.43:4158 , piece_id=42ALENTRHVD4KZYGWJAEYL437QSCBACQ6YQKZXMGUGHG6UNPMPNQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.155235285 (508h10m24.02631832s) [noise=true , address=145.79.227.49:4337 , piece_id=FWAQ4WCFISA5SZA52W3DUMOV7HTSAVOJE7GEZOMAXVBWAGWXIGQQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.159856544 (508h10m24.03093958s) [noise=true , address=145.79.227.52:4447 , piece_id=2LYG3HPVFSR3GAVEXLNH2MN4Q5CID5PYFZ3ELOCYNF7D5S2BTMMA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.163228505 (508h10m24.03431154s) [noise=true , address=145.79.227.45:4231 , piece_id=HEF7ND4PYBOPBHKFNOUTEBGJI4GRYBNHAXNOTOL35FFROF4S3PRA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.166987081 (508h10m24.038070126s) [noise=true , address=145.79.227.15:4873 , piece_id=JYLTCRAX6IE22DXIPKUCCCWGCWDQVLRTHDBRQPFBNT2SJ753EUCA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.209726194 (508h10m24.080809229s) [noise=true , address=145.79.227.11:4731 , piece_id=VV5E3DSDX3KW4NUJHNDT2UIGETTUG5CDRGM4LSU6JPHAPZXSQB5Q, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.383497926 (508h10m24.254580961s) [noise=true , address=45.91.116.8:4046 , piece_id=AXCGJD4TN5KLW7Y5KPX6I36T7AG2JCIE2GFUUR4CYGWOYDCQPF4Q, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.386033664 (508h10m24.257116699s) [noise=true , address=77.90.31.10:4520 , piece_id=UWSP7FFYKNSYDTAWQ7MTU7UYCLETA5WIY75R7Q3HDAY6R7YVJLGQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.393221170 (508h10m24.264304215s) [noise=true , address=77.90.31.17:4773 , piece_id=6SQWHSXYLOB2DF42A24GZETDY7NN5ILXXTACYXOETJWMVEPUGPMA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.421870163 (508h10m24.292953208s) [noise=true , address=45.91.116.7:4010 , piece_id=X4GUO3T4IEUGHBGSHOY5VIQI3L5SU6UGAKJ3Y5PUCEIST4KUDPNA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.494780056 (508h10m24.365863091s) [noise=true , address=45.91.116.10:4082 , piece_id=352FDNDNSH2MHX2ST3S2JOBLG4LIZ65OW6F6BYXQ25DHIRBDQCEQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.494799213 (508h10m24.365882258s) [seg_pos={0 0}, upload=445930]: Segment reached optimal threshold of 13 pieces. Dec 26 19:39:24.494972223 (508h10m24.366055258s) [noise=true , address=145.79.227.51:4410 , piece_id=U5DE42Y7N3APTAHVC57AYTPIK57Q7XJ2OA7AYLEQRKOBRFUX26TA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:58552->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:58552->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58552->145.79.227.51:4410: use of closed network connection Dec 26 19:39:24.494978044 (508h10m24.366061079s) [noise=true , address=145.79.227.51:4410 , piece_id=U5DE42Y7N3APTAHVC57AYTPIK57Q7XJ2OA7AYLEQRKOBRFUX26TA, seg_pos={0 0}, upload=445930]: Long tail cancellation due to optimal threshold Dec 26 19:39:24.495150392 (508h10m24.366233427s) [noise=true , address=45.91.116.11:4120 , piece_id=IGQECIJPS6CNFG67YYQ7FXICZQEAVRE57TXXPCYSAAHF6WESZ53Q, seg_pos={0 0}, upload=445930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43812->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43812->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->45.91.116.11:4120: use of closed network connection Dec 26 19:39:24.495167053 (508h10m24.366250088s) [noise=true , address=45.91.116.11:4120 , piece_id=IGQECIJPS6CNFG67YYQ7FXICZQEAVRE57TXXPCYSAAHF6WESZ53Q, seg_pos={0 0}, upload=445930]: Long tail cancellation due to optimal threshold Dec 26 19:39:24.495195658 (508h10m24.366278693s) [noise=true , address=145.79.227.13:4803 , piece_id=DGGG77IF54A4TYKJCV356HWDHASWIJGGJI7KXZZCD2TKQI2X6EGA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 188.166.225.170:58428->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58428->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58428->145.79.227.13:4803: use of closed network connection Dec 26 19:39:24.495209213 (508h10m24.366292278s) [noise=true , address=145.79.227.13:4803 , piece_id=DGGG77IF54A4TYKJCV356HWDHASWIJGGJI7KXZZCD2TKQI2X6EGA, seg_pos={0 0}, upload=445930]: Long tail cancellation due to optimal threshold Dec 26 19:39:24.495238700 (508h10m24.366321745s) [seg_pos={0 0}, upload=445930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:39:24.495311087 (508h10m24.366394122s) [upload=445930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:39:24.495316067 (508h10m24.366399102s) [upload=445930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:39:24.784973986 (508h10m24.656057021s) [upload=445930]: Done uploading segments... Dec 26 19:39:24.784996860 (508h10m24.656079895s) [upload=445930]: Upload finished. err: <nil> Duration:* 9.10005768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.X] duration: 30.003039367s trace: https://jaeger.dev.storj.io/trace/1aa177e6ebb6bf4c FAIL (2025-12-26 19:39:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (508h10m21.768427653s) [upload=445930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:39:22.139166048 (508h10m22.010249093s) [seg_pos={0 0}, upload=445930]: Begin upload segment... Dec 26 19:39:22.139216585 (508h10m22.01029962s) [seg_pos={0 0}, upload=445930]: Done begin upload segment. Dec 26 19:39:22.139237585 (508h10m22.01032063s) [noise=true , address=45.91.116.10:4082 , piece_id=352FDNDNSH2MHX2ST3S2JOBLG4LIZ65OW6F6BYXQ25DHIRBDQCEQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.139250239 (508h10m22.010333274s) [noise=true , address=145.79.227.49:4337 , piece_id=FWAQ4WCFISA5SZA52W3DUMOV7HTSAVOJE7GEZOMAXVBWAGWXIGQQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.139329930 (508h10m22.010412975s) [noise=true , address=145.79.227.52:4447 , piece_id=2LYG3HPVFSR3GAVEXLNH2MN4Q5CID5PYFZ3ELOCYNF7D5S2BTMMA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.140393816 (508h10m22.011476851s) [noise=true , address=77.90.31.10:4520 , piece_id=UWSP7FFYKNSYDTAWQ7MTU7UYCLETA5WIY75R7Q3HDAY6R7YVJLGQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.140723884 (508h10m22.01180692s) [noise=true , address=45.91.116.11:4120 , piece_id=IGQECIJPS6CNFG67YYQ7FXICZQEAVRE57TXXPCYSAAHF6WESZ53Q, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.141081615 (508h10m22.01216465s) [noise=true , address=145.79.227.46:4603 , piece_id=PJKC22D4PBWBPCAWTL2AJZO3CFRGPDP7QSHM2I3W6QGLIH25JVTQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.141457541 (508h10m22.012540576s) [noise=true , address=145.79.227.45:4231 , piece_id=HEF7ND4PYBOPBHKFNOUTEBGJI4GRYBNHAXNOTOL35FFROF4S3PRA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.141532143 (508h10m22.012615178s) [noise=true , address=45.91.116.8:4046 , piece_id=AXCGJD4TN5KLW7Y5KPX6I36T7AG2JCIE2GFUUR4CYGWOYDCQPF4Q, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.142744081 (508h10m22.013827116s) [noise=true , address=145.79.227.11:4731 , piece_id=VV5E3DSDX3KW4NUJHNDT2UIGETTUG5CDRGM4LSU6JPHAPZXSQB5Q, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.142945826 (508h10m22.014028871s) [noise=true , address=45.91.116.7:4010 , piece_id=X4GUO3T4IEUGHBGSHOY5VIQI3L5SU6UGAKJ3Y5PUCEIST4KUDPNA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.143362950 (508h10m22.014445995s) [noise=true , address=145.79.227.36:4517 , piece_id=6TKKAJGLD6FSCF6VSPF6X5PVRICWALBKQG5SYSLJ7LJLODZH235Q, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.143754455 (508h10m22.01483749s) [noise=true , address=77.90.31.17:4773 , piece_id=6SQWHSXYLOB2DF42A24GZETDY7NN5ILXXTACYXOETJWMVEPUGPMA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.144257123 (508h10m22.015340158s) [noise=true , address=145.79.227.15:4873 , piece_id=JYLTCRAX6IE22DXIPKUCCCWGCWDQVLRTHDBRQPFBNT2SJ753EUCA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.144552134 (508h10m22.015635169s) [noise=true , address=145.79.227.13:4803 , piece_id=DGGG77IF54A4TYKJCV356HWDHASWIJGGJI7KXZZCD2TKQI2X6EGA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.145075411 (508h10m22.016158446s) [noise=true , address=145.79.227.43:4158 , piece_id=42ALENTRHVD4KZYGWJAEYL437QSCBACQ6YQKZXMGUGHG6UNPMPNQ, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:22.145387695 (508h10m22.01647073s) [noise=true , address=145.79.227.51:4410 , piece_id=U5DE42Y7N3APTAHVC57AYTPIK57Q7XJ2OA7AYLEQRKOBRFUX26TA, seg_pos={0 0}, upload=445930]: Uploading piece... Dec 26 19:39:23.857494574 (508h10m23.728577609s) [upload=445930]: Next returned nil segment Dec 26 19:39:23.857509413 (508h10m23.728592448s) [upload=445930]: Waiting for error group managing segments... Dec 26 19:39:24.141130116 (508h10m24.012213151s) [noise=true , address=145.79.227.46:4603 , piece_id=PJKC22D4PBWBPCAWTL2AJZO3CFRGPDP7QSHM2I3W6QGLIH25JVTQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.142114731 (508h10m24.013197766s) [noise=true , address=145.79.227.36:4517 , piece_id=6TKKAJGLD6FSCF6VSPF6X5PVRICWALBKQG5SYSLJ7LJLODZH235Q, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.147973206 (508h10m24.019056241s) [noise=true , address=145.79.227.43:4158 , piece_id=42ALENTRHVD4KZYGWJAEYL437QSCBACQ6YQKZXMGUGHG6UNPMPNQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.155235285 (508h10m24.02631832s) [noise=true , address=145.79.227.49:4337 , piece_id=FWAQ4WCFISA5SZA52W3DUMOV7HTSAVOJE7GEZOMAXVBWAGWXIGQQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.159856544 (508h10m24.03093958s) [noise=true , address=145.79.227.52:4447 , piece_id=2LYG3HPVFSR3GAVEXLNH2MN4Q5CID5PYFZ3ELOCYNF7D5S2BTMMA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.163228505 (508h10m24.03431154s) [noise=true , address=145.79.227.45:4231 , piece_id=HEF7ND4PYBOPBHKFNOUTEBGJI4GRYBNHAXNOTOL35FFROF4S3PRA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.166987081 (508h10m24.038070126s) [noise=true , address=145.79.227.15:4873 , piece_id=JYLTCRAX6IE22DXIPKUCCCWGCWDQVLRTHDBRQPFBNT2SJ753EUCA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.209726194 (508h10m24.080809229s) [noise=true , address=145.79.227.11:4731 , piece_id=VV5E3DSDX3KW4NUJHNDT2UIGETTUG5CDRGM4LSU6JPHAPZXSQB5Q, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.383497926 (508h10m24.254580961s) [noise=true , address=45.91.116.8:4046 , piece_id=AXCGJD4TN5KLW7Y5KPX6I36T7AG2JCIE2GFUUR4CYGWOYDCQPF4Q, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.386033664 (508h10m24.257116699s) [noise=true , address=77.90.31.10:4520 , piece_id=UWSP7FFYKNSYDTAWQ7MTU7UYCLETA5WIY75R7Q3HDAY6R7YVJLGQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.393221170 (508h10m24.264304215s) [noise=true , address=77.90.31.17:4773 , piece_id=6SQWHSXYLOB2DF42A24GZETDY7NN5ILXXTACYXOETJWMVEPUGPMA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.421870163 (508h10m24.292953208s) [noise=true , address=45.91.116.7:4010 , piece_id=X4GUO3T4IEUGHBGSHOY5VIQI3L5SU6UGAKJ3Y5PUCEIST4KUDPNA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.494780056 (508h10m24.365863091s) [noise=true , address=45.91.116.10:4082 , piece_id=352FDNDNSH2MHX2ST3S2JOBLG4LIZ65OW6F6BYXQ25DHIRBDQCEQ, seg_pos={0 0}, upload=445930]: Done uploading piece. err: <nil> Dec 26 19:39:24.494799213 (508h10m24.365882258s) [seg_pos={0 0}, upload=445930]: Segment reached optimal threshold of 13 pieces. Dec 26 19:39:24.494972223 (508h10m24.366055258s) [noise=true , address=145.79.227.51:4410 , piece_id=U5DE42Y7N3APTAHVC57AYTPIK57Q7XJ2OA7AYLEQRKOBRFUX26TA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:58552->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:58552->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58552->145.79.227.51:4410: use of closed network connection Dec 26 19:39:24.494978044 (508h10m24.366061079s) [noise=true , address=145.79.227.51:4410 , piece_id=U5DE42Y7N3APTAHVC57AYTPIK57Q7XJ2OA7AYLEQRKOBRFUX26TA, seg_pos={0 0}, upload=445930]: Long tail cancellation due to optimal threshold Dec 26 19:39:24.495150392 (508h10m24.366233427s) [noise=true , address=45.91.116.11:4120 , piece_id=IGQECIJPS6CNFG67YYQ7FXICZQEAVRE57TXXPCYSAAHF6WESZ53Q, seg_pos={0 0}, upload=445930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43812->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43812->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->45.91.116.11:4120: use of closed network connection Dec 26 19:39:24.495167053 (508h10m24.366250088s) [noise=true , address=45.91.116.11:4120 , piece_id=IGQECIJPS6CNFG67YYQ7FXICZQEAVRE57TXXPCYSAAHF6WESZ53Q, seg_pos={0 0}, upload=445930]: Long tail cancellation due to optimal threshold Dec 26 19:39:24.495195658 (508h10m24.366278693s) [noise=true , address=145.79.227.13:4803 , piece_id=DGGG77IF54A4TYKJCV356HWDHASWIJGGJI7KXZZCD2TKQI2X6EGA, seg_pos={0 0}, upload=445930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 188.166.225.170:58428->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58428->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58428->145.79.227.13:4803: use of closed network connection Dec 26 19:39:24.495209213 (508h10m24.366292278s) [noise=true , address=145.79.227.13:4803 , piece_id=DGGG77IF54A4TYKJCV356HWDHASWIJGGJI7KXZZCD2TKQI2X6EGA, seg_pos={0 0}, upload=445930]: Long tail cancellation due to optimal threshold Dec 26 19:39:24.495238700 (508h10m24.366321745s) [seg_pos={0 0}, upload=445930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:39:24.495311087 (508h10m24.366394122s) [upload=445930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:39:24.495316067 (508h10m24.366399102s) [upload=445930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:39:24.784973986 (508h10m24.656057021s) [upload=445930]: Done uploading segments... Dec 26 19:39:24.784996860 (508h10m24.656079895s) [upload=445930]: Upload finished. err: <nil> Duration:* 30.003062612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X.] duration: 6.894318741s trace: https://jaeger.dev.storj.io/trace/5b7a480cd6bb8af1 FAIL (2025-12-26 19:38:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .623861158s) [upload=445903]: Uploading segments... Dec 26 19:37:39.752851042 (508h8m39.623934087s) [upload=445903]: Got next segment. Inline: false Dec 26 19:37:39.752891318 (508h8m39.623974353s) [upload=445903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:37:39.752897740 (508h8m39.623980775s) [upload=445903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:37:40.005319916 (508h8m39.876402951s) [seg_pos={0 0}, upload=445903]: Begin upload segment... Dec 26 19:37:40.005364471 (508h8m39.876447516s) [seg_pos={0 0}, upload=445903]: Done begin upload segment. Dec 26 19:37:40.005388848 (508h8m39.876471883s) [noise=true , address=77.90.31.16:4736 , piece_id=WUAPUSXFLAXGV64YLO57RBJX54J5GBVX5LH2UB7QT2OJJVBADKEA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.005622383 (508h8m39.876705418s) [noise=true , address=145.79.227.5:4222 , piece_id=NCN23QH5JVGIQYYW4Z4F3OCB62MBE2NZPGDYF42YPTTFOFBJU45Q, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.005789891 (508h8m39.876872926s) [noise=true , address=145.79.227.31:4799 , piece_id=SBUGEIDVJPPJEWOKMS72257NKLSK6X5JBM4XOC3LHXTWDJ2F5V7A, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.006304872 (508h8m39.877387907s) [noise=true , address=145.79.227.42:4122 , piece_id=PUJHHLGSFT7AVGZ7GXB7X3BT2MVRUACQGEGLQ44VXQ2SNVMK47GA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.006792540 (508h8m39.877875575s) [noise=true , address=45.91.116.5:4297 , piece_id=BBTCWXEEWY3MIPXO3EWXYBG7XMAJE427ALXCBPDDKO5ZOE2VMS5A, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.007190629 (508h8m39.878273664s) [noise=true , address=145.79.227.47:4265 , piece_id=PQQVGT5KBXCHPH7G5PFFUYHHFFCX6MZPVKHWB3CIJPL7TN6VRMHA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.007593495 (508h8m39.87867653s) [noise=true , address=145.79.227.15:4875 , piece_id=KXMPAWFTLKH6XGRS55IWULYBSQVGLUQWAP6NGMTHSVLBZAYK3ZJQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.008013425 (508h8m39.87909647s) [noise=true , address=145.79.227.11:4732 , piece_id=SD7QPIPSF5WWYRARU74UYC4FWWQIBYEASHJAVVBRINZPIBPEIR7A, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.009016064 (508h8m39.880099099s) [noise=true , address=45.91.116.13:4156 , piece_id=I4IHGYTSZYPVD2PJ7YNWHPZOL35JCI5IG44BXWXCY4YFES2XMX2Q, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.009532578 (508h8m39.880615613s) [noise=true , address=145.79.227.29:4941 , piece_id=UCYRAJ2SICCFCKM3XNPOWEFBGCIRBBQ4NRRKNCD2Y4MP6AUGSPWA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.010078398 (508h8m39.881161433s) [noise=true , address=145.79.227.41:4087 , piece_id=5R6VFL7J4STVQD7AG46BAJSDTVPVGEI5GJQ2VLNALNY247TIJ7OQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.011050719 (508h8m39.882133774s) [noise=true , address=45.91.116.7:4010 , piece_id=VVLXQOC6JIQBWCJIBNUPRYPYGZA22FGJY6LB4RESDPUODN273XHA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.011464747 (508h8m39.882547782s) [noise=true , address=45.91.116.9:4263 , piece_id=XY46IWFY6VXO73JC4X3QYDPQOVDM4GIQ3WRQACKFHD6SMBJ56Y6Q, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.012521680 (508h8m39.883604725s) [noise=true , address=145.79.227.45:4229 , piece_id=CMKF4GDOFS5QUKCKEKFYADAMTVCPMCIEFTHZAYV6CD6YOA4YDXDQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.015728346 (508h8m39.886811381s) [noise=true , address=45.91.116.11:4119 , piece_id=SH4LY4E2IAYAOPORQPKHDAW6PUMQRDQXPNGXSY5OUAMWPZD6G3UQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.018577942 (508h8m39.889660977s) [noise=true , address=77.90.31.15:4700 , piece_id=N5UTWVQFLJZR4TYG7HULQ3QHY42AQ646CQ7554SFKJ7F7CHJLMEQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:41.714774546 (508h8m41.585857591s) [upload=445903]: Next returned nil segment Dec 26 19:37:41.714816987 (508h8m41.585900042s) [upload=445903]: Waiting for error group managing segments... Dec 26 19:37:42.017494728 (508h8m41.888577763s) [noise=true , address=145.79.227.42:4122 , piece_id=PUJHHLGSFT7AVGZ7GXB7X3BT2MVRUACQGEGLQ44VXQ2SNVMK47GA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.018913048 (508h8m41.889996083s) [noise=true , address=145.79.227.15:4875 , piece_id=KXMPAWFTLKH6XGRS55IWULYBSQVGLUQWAP6NGMTHSVLBZAYK3ZJQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.019712981 (508h8m41.890796017s) [noise=true , address=145.79.227.45:4229 , piece_id=CMKF4GDOFS5QUKCKEKFYADAMTVCPMCIEFTHZAYV6CD6YOA4YDXDQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.026365539 (508h8m41.897448574s) [noise=true , address=145.79.227.31:4799 , piece_id=SBUGEIDVJPPJEWOKMS72257NKLSK6X5JBM4XOC3LHXTWDJ2F5V7A, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.036039099 (508h8m41.907122144s) [noise=true , address=145.79.227.47:4265 , piece_id=PQQVGT5KBXCHPH7G5PFFUYHHFFCX6MZPVKHWB3CIJPL7TN6VRMHA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.048379849 (508h8m41.919462884s) [noise=true , address=145.79.227.11:4732 , piece_id=SD7QPIPSF5WWYRARU74UYC4FWWQIBYEASHJAVVBRINZPIBPEIR7A, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.223712192 (508h8m42.094795227s) [noise=true , address=77.90.31.15:4700 , piece_id=N5UTWVQFLJZR4TYG7HULQ3QHY42AQ646CQ7554SFKJ7F7CHJLMEQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.223810299 (508h8m42.094893334s) [noise=true , address=77.90.31.16:4736 , piece_id=WUAPUSXFLAXGV64YLO57RBJX54J5GBVX5LH2UB7QT2OJJVBADKEA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.239245380 (508h8m42.110328425s) [noise=true , address=45.91.116.11:4119 , piece_id=SH4LY4E2IAYAOPORQPKHDAW6PUMQRDQXPNGXSY5OUAMWPZD6G3UQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.353612856 (508h8m42.224695891s) [noise=true , address=45.91.116.5:4297 , piece_id=BBTCWXEEWY3MIPXO3EWXYBG7XMAJE427ALXCBPDDKO5ZOE2VMS5A, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.355563630 (508h8m42.226646665s) [noise=true , address=45.91.116.9:4263 , piece_id=XY46IWFY6VXO73JC4X3QYDPQOVDM4GIQ3WRQACKFHD6SMBJ56Y6Q, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.370028243 (508h8m42.241111278s) [noise=true , address=45.91.116.7:4010 , piece_id=VVLXQOC6JIQBWCJIBNUPRYPYGZA22FGJY6LB4RESDPUODN273XHA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.399040808 (508h8m42.270123853s) [noise=true , address=45.91.116.13:4156 , piece_id=I4IHGYTSZYPVD2PJ7YNWHPZOL35JCI5IG44BXWXCY4YFES2XMX2Q, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.399062759 (508h8m42.270145805s) [seg_pos={0 0}, upload=445903]: Segment reached optimal threshold of 13 pieces. Dec 26 19:37:42.399281276 (508h8m42.270364321s) [noise=true , address=145.79.227.41:4087 , piece_id=5R6VFL7J4STVQD7AG46BAJSDTVPVGEI5GJQ2VLNALNY247TIJ7OQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:53244->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:53244->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53244->145.79.227.41:4087: use of closed network connection Dec 26 19:37:42.399289932 (508h8m42.270372978s) [noise=true , address=145.79.227.41:4087 , piece_id=5R6VFL7J4STVQD7AG46BAJSDTVPVGEI5GJQ2VLNALNY247TIJ7OQ, seg_pos={0 0}, upload=445903]: Long tail cancellation due to optimal threshold Dec 26 19:37:42.399414310 (508h8m42.270497355s) [noise=true , address=145.79.227.29:4941 , piece_id=UCYRAJ2SICCFCKM3XNPOWEFBGCIRBBQ4NRRKNCD2Y4MP6AUGSPWA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 19:37:42.399433145 (508h8m42.27051619s) [noise=true , address=145.79.227.29:4941 , piece_id=UCYRAJ2SICCFCKM3XNPOWEFBGCIRBBQ4NRRKNCD2Y4MP6AUGSPWA, seg_pos={0 0}, upload=445903]: Long tail cancellation due to optimal threshold Dec 26 19:37:42.399600424 (508h8m42.270683469s) [noise=true , address=145.79.227.5:4222 , piece_id=NCN23QH5JVGIQYYW4Z4F3OCB62MBE2NZPGDYF42YPTTFOFBJU45Q, seg_pos={0 0}, upload=445903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:34904->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:34904->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34904->145.79.227.5:4222: use of closed network connection Dec 26 19:37:42.399607537 (508h8m42.270690572s) [noise=true , address=145.79.227.5:4222 , piece_id=NCN23QH5JVGIQYYW4Z4F3OCB62MBE2NZPGDYF42YPTTFOFBJU45Q, seg_pos={0 0}, upload=445903]: Long tail cancellation due to optimal threshold Dec 26 19:37:42.399624590 (508h8m42.270707635s) [seg_pos={0 0}, upload=445903]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:37:42.399714300 (508h8m42.270797335s) [upload=445903]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:37:42.399719911 (508h8m42.270802946s) [upload=445903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:37:42.702630335 (508h8m42.57371339s) [upload=445903]: Done uploading segments... Dec 26 19:37:42.702708874 (508h8m42.573791919s) [upload=445903]: Upload finished. err: <nil> Duration: 6.894333128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.X] duration: 30.003452577s trace: https://jaeger.dev.storj.io/trace/5b7a480cd6bb8af1 FAIL (2025-12-26 19:38:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .623861158s) [upload=445903]: Uploading segments... Dec 26 19:37:39.752851042 (508h8m39.623934087s) [upload=445903]: Got next segment. Inline: false Dec 26 19:37:39.752891318 (508h8m39.623974353s) [upload=445903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:37:39.752897740 (508h8m39.623980775s) [upload=445903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:37:40.005319916 (508h8m39.876402951s) [seg_pos={0 0}, upload=445903]: Begin upload segment... Dec 26 19:37:40.005364471 (508h8m39.876447516s) [seg_pos={0 0}, upload=445903]: Done begin upload segment. Dec 26 19:37:40.005388848 (508h8m39.876471883s) [noise=true , address=77.90.31.16:4736 , piece_id=WUAPUSXFLAXGV64YLO57RBJX54J5GBVX5LH2UB7QT2OJJVBADKEA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.005622383 (508h8m39.876705418s) [noise=true , address=145.79.227.5:4222 , piece_id=NCN23QH5JVGIQYYW4Z4F3OCB62MBE2NZPGDYF42YPTTFOFBJU45Q, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.005789891 (508h8m39.876872926s) [noise=true , address=145.79.227.31:4799 , piece_id=SBUGEIDVJPPJEWOKMS72257NKLSK6X5JBM4XOC3LHXTWDJ2F5V7A, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.006304872 (508h8m39.877387907s) [noise=true , address=145.79.227.42:4122 , piece_id=PUJHHLGSFT7AVGZ7GXB7X3BT2MVRUACQGEGLQ44VXQ2SNVMK47GA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.006792540 (508h8m39.877875575s) [noise=true , address=45.91.116.5:4297 , piece_id=BBTCWXEEWY3MIPXO3EWXYBG7XMAJE427ALXCBPDDKO5ZOE2VMS5A, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.007190629 (508h8m39.878273664s) [noise=true , address=145.79.227.47:4265 , piece_id=PQQVGT5KBXCHPH7G5PFFUYHHFFCX6MZPVKHWB3CIJPL7TN6VRMHA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.007593495 (508h8m39.87867653s) [noise=true , address=145.79.227.15:4875 , piece_id=KXMPAWFTLKH6XGRS55IWULYBSQVGLUQWAP6NGMTHSVLBZAYK3ZJQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.008013425 (508h8m39.87909647s) [noise=true , address=145.79.227.11:4732 , piece_id=SD7QPIPSF5WWYRARU74UYC4FWWQIBYEASHJAVVBRINZPIBPEIR7A, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.009016064 (508h8m39.880099099s) [noise=true , address=45.91.116.13:4156 , piece_id=I4IHGYTSZYPVD2PJ7YNWHPZOL35JCI5IG44BXWXCY4YFES2XMX2Q, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.009532578 (508h8m39.880615613s) [noise=true , address=145.79.227.29:4941 , piece_id=UCYRAJ2SICCFCKM3XNPOWEFBGCIRBBQ4NRRKNCD2Y4MP6AUGSPWA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.010078398 (508h8m39.881161433s) [noise=true , address=145.79.227.41:4087 , piece_id=5R6VFL7J4STVQD7AG46BAJSDTVPVGEI5GJQ2VLNALNY247TIJ7OQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.011050719 (508h8m39.882133774s) [noise=true , address=45.91.116.7:4010 , piece_id=VVLXQOC6JIQBWCJIBNUPRYPYGZA22FGJY6LB4RESDPUODN273XHA, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.011464747 (508h8m39.882547782s) [noise=true , address=45.91.116.9:4263 , piece_id=XY46IWFY6VXO73JC4X3QYDPQOVDM4GIQ3WRQACKFHD6SMBJ56Y6Q, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.012521680 (508h8m39.883604725s) [noise=true , address=145.79.227.45:4229 , piece_id=CMKF4GDOFS5QUKCKEKFYADAMTVCPMCIEFTHZAYV6CD6YOA4YDXDQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.015728346 (508h8m39.886811381s) [noise=true , address=45.91.116.11:4119 , piece_id=SH4LY4E2IAYAOPORQPKHDAW6PUMQRDQXPNGXSY5OUAMWPZD6G3UQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:40.018577942 (508h8m39.889660977s) [noise=true , address=77.90.31.15:4700 , piece_id=N5UTWVQFLJZR4TYG7HULQ3QHY42AQ646CQ7554SFKJ7F7CHJLMEQ, seg_pos={0 0}, upload=445903]: Uploading piece... Dec 26 19:37:41.714774546 (508h8m41.585857591s) [upload=445903]: Next returned nil segment Dec 26 19:37:41.714816987 (508h8m41.585900042s) [upload=445903]: Waiting for error group managing segments... Dec 26 19:37:42.017494728 (508h8m41.888577763s) [noise=true , address=145.79.227.42:4122 , piece_id=PUJHHLGSFT7AVGZ7GXB7X3BT2MVRUACQGEGLQ44VXQ2SNVMK47GA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.018913048 (508h8m41.889996083s) [noise=true , address=145.79.227.15:4875 , piece_id=KXMPAWFTLKH6XGRS55IWULYBSQVGLUQWAP6NGMTHSVLBZAYK3ZJQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.019712981 (508h8m41.890796017s) [noise=true , address=145.79.227.45:4229 , piece_id=CMKF4GDOFS5QUKCKEKFYADAMTVCPMCIEFTHZAYV6CD6YOA4YDXDQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.026365539 (508h8m41.897448574s) [noise=true , address=145.79.227.31:4799 , piece_id=SBUGEIDVJPPJEWOKMS72257NKLSK6X5JBM4XOC3LHXTWDJ2F5V7A, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.036039099 (508h8m41.907122144s) [noise=true , address=145.79.227.47:4265 , piece_id=PQQVGT5KBXCHPH7G5PFFUYHHFFCX6MZPVKHWB3CIJPL7TN6VRMHA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.048379849 (508h8m41.919462884s) [noise=true , address=145.79.227.11:4732 , piece_id=SD7QPIPSF5WWYRARU74UYC4FWWQIBYEASHJAVVBRINZPIBPEIR7A, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.223712192 (508h8m42.094795227s) [noise=true , address=77.90.31.15:4700 , piece_id=N5UTWVQFLJZR4TYG7HULQ3QHY42AQ646CQ7554SFKJ7F7CHJLMEQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.223810299 (508h8m42.094893334s) [noise=true , address=77.90.31.16:4736 , piece_id=WUAPUSXFLAXGV64YLO57RBJX54J5GBVX5LH2UB7QT2OJJVBADKEA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.239245380 (508h8m42.110328425s) [noise=true , address=45.91.116.11:4119 , piece_id=SH4LY4E2IAYAOPORQPKHDAW6PUMQRDQXPNGXSY5OUAMWPZD6G3UQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.353612856 (508h8m42.224695891s) [noise=true , address=45.91.116.5:4297 , piece_id=BBTCWXEEWY3MIPXO3EWXYBG7XMAJE427ALXCBPDDKO5ZOE2VMS5A, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.355563630 (508h8m42.226646665s) [noise=true , address=45.91.116.9:4263 , piece_id=XY46IWFY6VXO73JC4X3QYDPQOVDM4GIQ3WRQACKFHD6SMBJ56Y6Q, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.370028243 (508h8m42.241111278s) [noise=true , address=45.91.116.7:4010 , piece_id=VVLXQOC6JIQBWCJIBNUPRYPYGZA22FGJY6LB4RESDPUODN273XHA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.399040808 (508h8m42.270123853s) [noise=true , address=45.91.116.13:4156 , piece_id=I4IHGYTSZYPVD2PJ7YNWHPZOL35JCI5IG44BXWXCY4YFES2XMX2Q, seg_pos={0 0}, upload=445903]: Done uploading piece. err: <nil> Dec 26 19:37:42.399062759 (508h8m42.270145805s) [seg_pos={0 0}, upload=445903]: Segment reached optimal threshold of 13 pieces. Dec 26 19:37:42.399281276 (508h8m42.270364321s) [noise=true , address=145.79.227.41:4087 , piece_id=5R6VFL7J4STVQD7AG46BAJSDTVPVGEI5GJQ2VLNALNY247TIJ7OQ, seg_pos={0 0}, upload=445903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:53244->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:53244->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53244->145.79.227.41:4087: use of closed network connection Dec 26 19:37:42.399289932 (508h8m42.270372978s) [noise=true , address=145.79.227.41:4087 , piece_id=5R6VFL7J4STVQD7AG46BAJSDTVPVGEI5GJQ2VLNALNY247TIJ7OQ, seg_pos={0 0}, upload=445903]: Long tail cancellation due to optimal threshold Dec 26 19:37:42.399414310 (508h8m42.270497355s) [noise=true , address=145.79.227.29:4941 , piece_id=UCYRAJ2SICCFCKM3XNPOWEFBGCIRBBQ4NRRKNCD2Y4MP6AUGSPWA, seg_pos={0 0}, upload=445903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 19:37:42.399433145 (508h8m42.27051619s) [noise=true , address=145.79.227.29:4941 , piece_id=UCYRAJ2SICCFCKM3XNPOWEFBGCIRBBQ4NRRKNCD2Y4MP6AUGSPWA, seg_pos={0 0}, upload=445903]: Long tail cancellation due to optimal threshold Dec 26 19:37:42.399600424 (508h8m42.270683469s) [noise=true , address=145.79.227.5:4222 , piece_id=NCN23QH5JVGIQYYW4Z4F3OCB62MBE2NZPGDYF42YPTTFOFBJU45Q, seg_pos={0 0}, upload=445903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:34904->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:34904->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34904->145.79.227.5:4222: use of closed network connection Dec 26 19:37:42.399607537 (508h8m42.270690572s) [noise=true , address=145.79.227.5:4222 , piece_id=NCN23QH5JVGIQYYW4Z4F3OCB62MBE2NZPGDYF42YPTTFOFBJU45Q, seg_pos={0 0}, upload=445903]: Long tail cancellation due to optimal threshold Dec 26 19:37:42.399624590 (508h8m42.270707635s) [seg_pos={0 0}, upload=445903]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:37:42.399714300 (508h8m42.270797335s) [upload=445903]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:37:42.399719911 (508h8m42.270802946s) [upload=445903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:37:42.702630335 (508h8m42.57371339s) [upload=445903]: Done uploading segments... Dec 26 19:37:42.702708874 (508h8m42.573791919s) [upload=445903]: Upload finished. err: <nil> Duration: 30.003476281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.] duration: 5.820168372s trace: https://jaeger.dev.storj.io/trace/725257198532534 FAIL (2025-12-26 19:36:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA) context deadline exceeded ======LOGS====== .BatchRequestItem_ObjectBegin Dec 26 19:35:32.416073714 (508h6m32.287156749s) [upload=445873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:35:32.664653082 (508h6m32.535736117s) [seg_pos={0 0}, upload=445873]: Begin upload segment... Dec 26 19:35:32.664701694 (508h6m32.535784729s) [seg_pos={0 0}, upload=445873]: Done begin upload segment. Dec 26 19:35:32.664731571 (508h6m32.535814616s) [noise=true , address=145.79.227.36:4519 , piece_id=BYK3O6GOUHCVRS4DPOICNMYX7DLVMWY4G6RRPBFFEZUXOHXL6H4A, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.664920772 (508h6m32.536003817s) [noise=true , address=145.79.227.42:4121 , piece_id=6K7CZHGNWTECDT2LN4TPJQNHXD7C74T2VFIYZR35PICZU35D5PEA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.665689175 (508h6m32.53677222s) [noise=true , address=145.79.227.37:4567 , piece_id=GEATQYGSNS55Q6KNBTTW36PMZHCGB4FYMQMJJ7AWFIZNVZAZBH5A, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.666294708 (508h6m32.537377753s) [noise=true , address=145.79.227.45:4231 , piece_id=DTL5UZIOTCGIGIGJIE77M2J2USY4CRUGV4B4I5IVE2JCJHOPHEZQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.666986245 (508h6m32.53806928s) [noise=true , address=145.79.227.49:4338 , piece_id=NLBLJ5SWF67MRFTMMSH3EUMHFIGWZIHNKU364CIMPNLVQISHDKAA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.667200192 (508h6m32.538283237s) [noise=true , address=145.79.227.3:4151 , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.667324149 (508h6m32.538407184s) [noise=true , address=45.91.116.14:4191 , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.667694814 (508h6m32.538777859s) [noise=true , address=77.90.31.10:4520 , piece_id=E26JISLDLMD6PP2AAGIOOWOMELCOLTC4464QFDE4V42XACJITQTQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.667802980 (508h6m32.538886025s) [noise=true , address=77.90.31.21:4880 , piece_id=LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.668091610 (508h6m32.539174655s) [noise=true , address=145.79.227.46:4603 , piece_id=TNZB76INU3PMB6SWG5DBJMR47Q2P2J7DF2XXDN4IGSFATIQXTIJQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.668459841 (508h6m32.539542896s) [noise=true , address=45.91.116.5:4297 , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.668469298 (508h6m32.539552343s) [noise=true , address=45.91.116.7:4009 , piece_id=7GWILCYXB3ACUFBECP5QHOXLGLTMNPWHMBAHTAUM44QAJ4Z5WNIQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.668790591 (508h6m32.539873626s) [noise=true , address=45.91.116.6:4973 , piece_id=YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.669122512 (508h6m32.540205547s) [noise=true , address=145.79.227.39:4015 , piece_id=LBMAA4YAAMH532P6YGYPNQGHXOPNURFN3FA2I4CORCD77QNSK2AQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.669954907 (508h6m32.541037952s) [noise=true , address=45.91.116.12:4226 , piece_id=BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.670094884 (508h6m32.541177919s) [noise=true , address=145.79.227.19:4078 , piece_id=NLOFZDPLKEVLD53Q4ILQM7AX3NCI7IE6N2X5M2HWCKRN5GEN3AAA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:34.369947019 (508h6m34.241030054s) [upload=445873]: Next returned nil segment Dec 26 19:35:34.369990361 (508h6m34.241073416s) [upload=445873]: Waiting for error group managing segments... Dec 26 19:35:34.658250368 (508h6m34.529333403s) [noise=true , address=145.79.227.46:4603 , piece_id=TNZB76INU3PMB6SWG5DBJMR47Q2P2J7DF2XXDN4IGSFATIQXTIJQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.665658745 (508h6m34.53674178s) [noise=true , address=145.79.227.49:4338 , piece_id=NLBLJ5SWF67MRFTMMSH3EUMHFIGWZIHNKU364CIMPNLVQISHDKAA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.665700594 (508h6m34.536783639s) [noise=true , address=145.79.227.42:4121 , piece_id=6K7CZHGNWTECDT2LN4TPJQNHXD7C74T2VFIYZR35PICZU35D5PEA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.670607338 (508h6m34.541690383s) [noise=true , address=145.79.227.39:4015 , piece_id=LBMAA4YAAMH532P6YGYPNQGHXOPNURFN3FA2I4CORCD77QNSK2AQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.692231203 (508h6m34.563314248s) [noise=true , address=145.79.227.37:4567 , piece_id=GEATQYGSNS55Q6KNBTTW36PMZHCGB4FYMQMJJ7AWFIZNVZAZBH5A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.697176921 (508h6m34.568259967s) [noise=true , address=145.79.227.36:4519 , piece_id=BYK3O6GOUHCVRS4DPOICNMYX7DLVMWY4G6RRPBFFEZUXOHXL6H4A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.705259562 (508h6m34.576342607s) [noise=true , address=145.79.227.45:4231 , piece_id=DTL5UZIOTCGIGIGJIE77M2J2USY4CRUGV4B4I5IVE2JCJHOPHEZQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.718464647 (508h6m34.589547692s) [noise=true , address=145.79.227.19:4078 , piece_id=NLOFZDPLKEVLD53Q4ILQM7AX3NCI7IE6N2X5M2HWCKRN5GEN3AAA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.899524947 (508h6m34.770607982s) [noise=true , address=77.90.31.10:4520 , piece_id=E26JISLDLMD6PP2AAGIOOWOMELCOLTC4464QFDE4V42XACJITQTQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.939433770 (508h6m34.810516805s) [noise=true , address=45.91.116.6:4973 , piece_id=YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.957276556 (508h6m34.828359601s) [noise=true , address=45.91.116.12:4226 , piece_id=BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.966613435 (508h6m34.83769647s) [noise=true , address=45.91.116.7:4009 , piece_id=7GWILCYXB3ACUFBECP5QHOXLGLTMNPWHMBAHTAUM44QAJ4Z5WNIQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.970246202 (508h6m34.841329247s) [noise=true , address=77.90.31.21:4880 , piece_id=LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.970260069 (508h6m34.841343104s) [seg_pos={0 0}, upload=445873]: Segment reached optimal threshold of 13 pieces. Dec 26 19:35:34.970541395 (508h6m34.84162443s) [noise=true , address=45.91.116.5:4297 , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection Dec 26 19:35:34.970557565 (508h6m34.8416406s) [noise=true , address=45.91.116.5:4297 , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold Dec 26 19:35:34.970570791 (508h6m34.841653826s) [noise=true , address=45.91.116.14:4191 , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection Dec 26 19:35:34.970577363 (508h6m34.841660398s) [noise=true , address=45.91.116.14:4191 , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold Dec 26 19:35:34.970743881 (508h6m34.841826956s) [noise=true , address=145.79.227.3:4151 , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection Dec 26 19:35:34.970755472 (508h6m34.841838517s) [noise=true , address=145.79.227.3:4151 , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold Dec 26 19:35:34.970837137 (508h6m34.841920173s) [seg_pos={0 0}, upload=445873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:35:34.970917782 (508h6m34.842000817s) [upload=445873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:35:34.970922801 (508h6m34.842005836s) [upload=445873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:35:35.274411846 (508h6m35.145494891s) [upload=445873]: Done uploading segments... Dec 26 19:35:35.274430442 (508h6m35.145513487s) [upload=445873]: Upload finished. err: <nil> Duration:* 5.820194131s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X] duration: 30.004382056s trace: https://jaeger.dev.storj.io/trace/725257198532534 FAIL (2025-12-26 19:36:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA) context deadline exceeded ======LOGS====== .BatchRequestItem_ObjectBegin Dec 26 19:35:32.416073714 (508h6m32.287156749s) [upload=445873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:35:32.664653082 (508h6m32.535736117s) [seg_pos={0 0}, upload=445873]: Begin upload segment... Dec 26 19:35:32.664701694 (508h6m32.535784729s) [seg_pos={0 0}, upload=445873]: Done begin upload segment. Dec 26 19:35:32.664731571 (508h6m32.535814616s) [noise=true , address=145.79.227.36:4519 , piece_id=BYK3O6GOUHCVRS4DPOICNMYX7DLVMWY4G6RRPBFFEZUXOHXL6H4A, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.664920772 (508h6m32.536003817s) [noise=true , address=145.79.227.42:4121 , piece_id=6K7CZHGNWTECDT2LN4TPJQNHXD7C74T2VFIYZR35PICZU35D5PEA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.665689175 (508h6m32.53677222s) [noise=true , address=145.79.227.37:4567 , piece_id=GEATQYGSNS55Q6KNBTTW36PMZHCGB4FYMQMJJ7AWFIZNVZAZBH5A, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.666294708 (508h6m32.537377753s) [noise=true , address=145.79.227.45:4231 , piece_id=DTL5UZIOTCGIGIGJIE77M2J2USY4CRUGV4B4I5IVE2JCJHOPHEZQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.666986245 (508h6m32.53806928s) [noise=true , address=145.79.227.49:4338 , piece_id=NLBLJ5SWF67MRFTMMSH3EUMHFIGWZIHNKU364CIMPNLVQISHDKAA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.667200192 (508h6m32.538283237s) [noise=true , address=145.79.227.3:4151 , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.667324149 (508h6m32.538407184s) [noise=true , address=45.91.116.14:4191 , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.667694814 (508h6m32.538777859s) [noise=true , address=77.90.31.10:4520 , piece_id=E26JISLDLMD6PP2AAGIOOWOMELCOLTC4464QFDE4V42XACJITQTQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.667802980 (508h6m32.538886025s) [noise=true , address=77.90.31.21:4880 , piece_id=LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.668091610 (508h6m32.539174655s) [noise=true , address=145.79.227.46:4603 , piece_id=TNZB76INU3PMB6SWG5DBJMR47Q2P2J7DF2XXDN4IGSFATIQXTIJQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.668459841 (508h6m32.539542896s) [noise=true , address=45.91.116.5:4297 , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.668469298 (508h6m32.539552343s) [noise=true , address=45.91.116.7:4009 , piece_id=7GWILCYXB3ACUFBECP5QHOXLGLTMNPWHMBAHTAUM44QAJ4Z5WNIQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.668790591 (508h6m32.539873626s) [noise=true , address=45.91.116.6:4973 , piece_id=YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.669122512 (508h6m32.540205547s) [noise=true , address=145.79.227.39:4015 , piece_id=LBMAA4YAAMH532P6YGYPNQGHXOPNURFN3FA2I4CORCD77QNSK2AQ, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.669954907 (508h6m32.541037952s) [noise=true , address=45.91.116.12:4226 , piece_id=BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:32.670094884 (508h6m32.541177919s) [noise=true , address=145.79.227.19:4078 , piece_id=NLOFZDPLKEVLD53Q4ILQM7AX3NCI7IE6N2X5M2HWCKRN5GEN3AAA, seg_pos={0 0}, upload=445873]: Uploading piece... Dec 26 19:35:34.369947019 (508h6m34.241030054s) [upload=445873]: Next returned nil segment Dec 26 19:35:34.369990361 (508h6m34.241073416s) [upload=445873]: Waiting for error group managing segments... Dec 26 19:35:34.658250368 (508h6m34.529333403s) [noise=true , address=145.79.227.46:4603 , piece_id=TNZB76INU3PMB6SWG5DBJMR47Q2P2J7DF2XXDN4IGSFATIQXTIJQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.665658745 (508h6m34.53674178s) [noise=true , address=145.79.227.49:4338 , piece_id=NLBLJ5SWF67MRFTMMSH3EUMHFIGWZIHNKU364CIMPNLVQISHDKAA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.665700594 (508h6m34.536783639s) [noise=true , address=145.79.227.42:4121 , piece_id=6K7CZHGNWTECDT2LN4TPJQNHXD7C74T2VFIYZR35PICZU35D5PEA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.670607338 (508h6m34.541690383s) [noise=true , address=145.79.227.39:4015 , piece_id=LBMAA4YAAMH532P6YGYPNQGHXOPNURFN3FA2I4CORCD77QNSK2AQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.692231203 (508h6m34.563314248s) [noise=true , address=145.79.227.37:4567 , piece_id=GEATQYGSNS55Q6KNBTTW36PMZHCGB4FYMQMJJ7AWFIZNVZAZBH5A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.697176921 (508h6m34.568259967s) [noise=true , address=145.79.227.36:4519 , piece_id=BYK3O6GOUHCVRS4DPOICNMYX7DLVMWY4G6RRPBFFEZUXOHXL6H4A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.705259562 (508h6m34.576342607s) [noise=true , address=145.79.227.45:4231 , piece_id=DTL5UZIOTCGIGIGJIE77M2J2USY4CRUGV4B4I5IVE2JCJHOPHEZQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.718464647 (508h6m34.589547692s) [noise=true , address=145.79.227.19:4078 , piece_id=NLOFZDPLKEVLD53Q4ILQM7AX3NCI7IE6N2X5M2HWCKRN5GEN3AAA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.899524947 (508h6m34.770607982s) [noise=true , address=77.90.31.10:4520 , piece_id=E26JISLDLMD6PP2AAGIOOWOMELCOLTC4464QFDE4V42XACJITQTQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.939433770 (508h6m34.810516805s) [noise=true , address=45.91.116.6:4973 , piece_id=YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.957276556 (508h6m34.828359601s) [noise=true , address=45.91.116.12:4226 , piece_id=BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.966613435 (508h6m34.83769647s) [noise=true , address=45.91.116.7:4009 , piece_id=7GWILCYXB3ACUFBECP5QHOXLGLTMNPWHMBAHTAUM44QAJ4Z5WNIQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.970246202 (508h6m34.841329247s) [noise=true , address=77.90.31.21:4880 , piece_id=LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil> Dec 26 19:35:34.970260069 (508h6m34.841343104s) [seg_pos={0 0}, upload=445873]: Segment reached optimal threshold of 13 pieces. Dec 26 19:35:34.970541395 (508h6m34.84162443s) [noise=true , address=45.91.116.5:4297 , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection Dec 26 19:35:34.970557565 (508h6m34.8416406s) [noise=true , address=45.91.116.5:4297 , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold Dec 26 19:35:34.970570791 (508h6m34.841653826s) [noise=true , address=45.91.116.14:4191 , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection Dec 26 19:35:34.970577363 (508h6m34.841660398s) [noise=true , address=45.91.116.14:4191 , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold Dec 26 19:35:34.970743881 (508h6m34.841826956s) [noise=true , address=145.79.227.3:4151 , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection Dec 26 19:35:34.970755472 (508h6m34.841838517s) [noise=true , address=145.79.227.3:4151 , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold Dec 26 19:35:34.970837137 (508h6m34.841920173s) [seg_pos={0 0}, upload=445873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:35:34.970917782 (508h6m34.842000817s) [upload=445873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:35:34.970922801 (508h6m34.842005836s) [upload=445873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:35:35.274411846 (508h6m35.145494891s) [upload=445873]: Done uploading segments... Dec 26 19:35:35.274430442 (508h6m35.145513487s) [upload=445873]: Upload finished. err: <nil> Duration:* 30.004409448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......] duration: 7.306531825s trace: https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b FAIL (2025-12-26 19:34:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded ======LOGS====== Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments... Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment... Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment. Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments... Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces. Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments... Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil> Duration: 7.306563886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX.....] duration: 7.206037947s trace: https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b FAIL (2025-12-26 19:33:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded ======LOGS====== Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments... Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment... Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment. Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments... Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces. Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments... Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil> Duration: 7.206065239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX....] duration: 6.433736853s trace: https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b FAIL (2025-12-26 19:32:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded ======LOGS====== Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments... Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment... Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment. Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments... Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces. Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments... Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil> Duration: 6.433753385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX...] duration: 6.296305188s trace: https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b FAIL (2025-12-26 19:32:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded ======LOGS====== Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments... Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment... Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment. Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments... Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces. Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments... Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil> Duration: 6.296335246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX..] duration: 12.866996317s trace: https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b FAIL (2025-12-26 19:31:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded ======LOGS====== Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments... Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment... Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment. Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments... Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces. Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments... Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil> Duration: 12.867014661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX.] duration: 7.687812027s trace: https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b FAIL (2025-12-26 19:30:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded ======LOGS====== Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments... Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment... Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment. Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments... Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces. Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments... Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil> Duration: 7.687832938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX] duration: 30.002706126s trace: https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b FAIL (2025-12-26 19:30:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded ======LOGS====== Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments... Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment... Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment. Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece... Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments... Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601 , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447 , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730 , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229 , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518 , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265 , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301 , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338 , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122 , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565 , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880 , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937 , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046 , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil> Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces. Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975 , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972 , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085 , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments... Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil> Duration: 30.002732987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXX] duration: 30.005725155s trace: https://jaeger.dev.storj.io/trace/3daee636559e1cec FAIL (2025-12-26 19:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: DJYANR5BB6EYTKHHZGJMJF3XZ6RTVYSDPXL2TAOEP7BRFHHCLTDA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: HZTOKQFGOC7PZ6PNE2AEI4B7ZPLV4KWZVQRTR7L5TRA4UHP75HBQ) context deadline exceeded ======LOGS====== Dec 26 19:29:23.427370966 (508h0m23.298454001s) [upload=445785]: Starting upload Dec 26 19:29:23.427429568 (508h0m23.298512603s) [upload=445785]: Done starting upload Dec 26 19:29:23.427605463 (508h0m23.298688508s) [upload=445785]: Uploading segments... Dec 26 19:29:23.427641962 (508h0m23.298724997s) [upload=445785]: Got next segment. Inline: false Dec 26 19:29:23.427668303 (508h0m23.298751338s) [upload=445785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:29:23.427672671 (508h0m23.298755716s) [upload=445785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:29:23.685240285 (508h0m23.55632333s) [seg_pos={0 0}, upload=445785]: Begin upload segment... Dec 26 19:29:23.685284329 (508h0m23.556367364s) [seg_pos={0 0}, upload=445785]: Done begin upload segment. Dec 26 19:29:23.685309527 (508h0m23.556392562s) [noise=true , address=145.79.227.4:4187 , piece_id=D3LTSZXDLLQEJ2JUMK4TTXMUAKHCHZDJYGZY6ZO6CTMG7U5SFWEQ, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.689883055 (508h0m23.56096612s) [noise=true , address=145.79.227.21:4475 , piece_id=ABYAXLZPRMVDYX2PZRW7XBGPFXLFKFFXMZ4TLNQWWHEYYJ4AXJQA, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.691775148 (508h0m23.562858193s) [noise=true , address=145.79.227.45:4230 , piece_id=TUHJXV5ZHD34IFRW7EH7HCCKOCFDHG7TQMGKAKNHAPWYVYVXVCGQ, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.693130779 (508h0m23.564213824s) [noise=true , address=145.79.227.11:4730 , piece_id=ZTS3PEQA46YHDBVAHLSEQMIPDDBOIZBJVW4HCA3WJW5F5BZTO5YA, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.694416928 (508h0m23.565499963s) [noise=true , address=77.90.31.18:4808 , piece_id=DJYANR5BB6EYTKHHZGJMJF3XZ6RTVYSDPXL2TAOEP7BRFHHCLTDA, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.695628095 (508h0m23.56671114s) [noise=true , address=145.79.227.36:4519 , piece_id=72W22DPNORECKX3GRYHK5ZS4FJD4HMJ724OSU4KK43WSKTAVHUFQ, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.696697942 (508h0m23.567780987s) [noise=true , address=145.79.227.49:4337 , piece_id=RNZAWUNENGDCF33JXS3CIP33X25LRCLX5E2MWRFA6H4SQIEAIVPA, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.697948624 (508h0m23.569031659s) [noise=true , address=45.91.116.5:4299 , piece_id=HZTOKQFGOC7PZ6PNE2AEI4B7ZPLV4KWZVQRTR7L5TRA4UHP75HBQ, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.699233470 (508h0m23.570316506s) [noise=true , address=145.79.227.39:4014 , piece_id=3IA5BCBNUHGPCDJ6HWNCCUNWQQ4TUTGIPC6LVPAWI3Y3RMFA33GQ, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.700512255 (508h0m23.57159529s) [noise=true , address=145.79.227.38:4978 , piece_id=CAWZ72QZ34FFTX3WXJJB4EAWN4UWGE7QUNHRRRW2PGV26FIRZU2A, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.701856595 (508h0m23.57293963s) [noise=true , address=45.91.116.9:4264 , piece_id=5LBGA57TPBIL7S43JI2YAKQP65Y5WFUIZC7N23PDGTXPI3M6ATXA, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.703068703 (508h0m23.574151738s) [noise=true , address=145.79.227.50:4374 , piece_id=F2TUD7D625NQKKFORBV2CR3CROWBEM5NACK4BTR3Z5NXZMMEU5RQ, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.704257246 (508h0m23.575340281s) [noise=true , address=145.79.227.46:4602 , piece_id=TA4PJFFXWHUJJMAR4OGTW3F5FI7VDFZL2TRZINWEDNBCHCXIHY3Q, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.705582339 (508h0m23.576665395s) [noise=true , address=145.79.227.52:4447 , piece_id=UBGKPT6WYNL7RSD4FFAJIALSTQQW5CB46U2YEVWPIQJNWQFYXAGA, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.706972316 (508h0m23.578055351s) [noise=true , address=77.90.31.10:4520 , piece_id=LKQSJOYBOJDHMIS3CP6LE3XIX2AL5ZCJQDO37SIP5A54I7XCOWOQ, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:23.708149258 (508h0m23.579232293s) [noise=true , address=45.91.116.14:4192 , piece_id=3SZRRHSKLACT6BU6D7XHMEUB5IE44LBZEYBGZSLXIN5QIMJSC2ZA, seg_pos={0 0}, upload=445785]: Uploading piece... Dec 26 19:29:25.411769017 (508h0m25.282852052s) [upload=445785]: Next returned nil segment Dec 26 19:29:25.411811608 (508h0m25.282894643s) [upload=445785]: Waiting for error group managing segments... Dec 26 19:29:25.698202356 (508h0m25.569285391s) [noise=true , address=145.79.227.21:4475 , piece_id=ABYAXLZPRMVDYX2PZRW7XBGPFXLFKFFXMZ4TLNQWWHEYYJ4AXJQA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.698434057 (508h0m25.569517092s) [noise=true , address=145.79.227.52:4447 , piece_id=UBGKPT6WYNL7RSD4FFAJIALSTQQW5CB46U2YEVWPIQJNWQFYXAGA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.699301128 (508h0m25.570384163s) [noise=true , address=145.79.227.45:4230 , piece_id=TUHJXV5ZHD34IFRW7EH7HCCKOCFDHG7TQMGKAKNHAPWYVYVXVCGQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.714752430 (508h0m25.585835465s) [noise=true , address=145.79.227.46:4602 , piece_id=TA4PJFFXWHUJJMAR4OGTW3F5FI7VDFZL2TRZINWEDNBCHCXIHY3Q, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.720797712 (508h0m25.591880757s) [noise=true , address=145.79.227.50:4374 , piece_id=F2TUD7D625NQKKFORBV2CR3CROWBEM5NACK4BTR3Z5NXZMMEU5RQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.732212037 (508h0m25.603295072s) [noise=true , address=145.79.227.39:4014 , piece_id=3IA5BCBNUHGPCDJ6HWNCCUNWQQ4TUTGIPC6LVPAWI3Y3RMFA33GQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.732631846 (508h0m25.603714891s) [noise=true , address=145.79.227.36:4519 , piece_id=72W22DPNORECKX3GRYHK5ZS4FJD4HMJ724OSU4KK43WSKTAVHUFQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.735263757 (508h0m25.606346802s) [noise=true , address=145.79.227.38:4978 , piece_id=CAWZ72QZ34FFTX3WXJJB4EAWN4UWGE7QUNHRRRW2PGV26FIRZU2A, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.743327212 (508h0m25.614410257s) [noise=true , address=145.79.227.49:4337 , piece_id=RNZAWUNENGDCF33JXS3CIP33X25LRCLX5E2MWRFA6H4SQIEAIVPA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.763319370 (508h0m25.634402415s) [noise=true , address=145.79.227.4:4187 , piece_id=D3LTSZXDLLQEJ2JUMK4TTXMUAKHCHZDJYGZY6ZO6CTMG7U5SFWEQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:25.964663325 (508h0m25.83574636s) [noise=true , address=145.79.227.11:4730 , piece_id=ZTS3PEQA46YHDBVAHLSEQMIPDDBOIZBJVW4HCA3WJW5F5BZTO5YA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:26.007126412 (508h0m25.878209447s) [noise=true , address=77.90.31.18:4808 , piece_id=DJYANR5BB6EYTKHHZGJMJF3XZ6RTVYSDPXL2TAOEP7BRFHHCLTDA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:26.101332597 (508h0m25.972415632s) [noise=true , address=45.91.116.5:4299 , piece_id=HZTOKQFGOC7PZ6PNE2AEI4B7ZPLV4KWZVQRTR7L5TRA4UHP75HBQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil> Dec 26 19:29:26.101357664 (508h0m25.972440709s) [seg_pos={0 0}, upload=445785]: Segment reached optimal threshold of 13 pieces. Dec 26 19:29:26.101591310 (508h0m25.972674345s) [noise=true , address=45.91.116.14:4192 , piece_id=3SZRRHSKLACT6BU6D7XHMEUB5IE44LBZEYBGZSLXIN5QIMJSC2ZA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 19:29:26.101599435 (508h0m25.97268247s) [noise=true , address=45.91.116.14:4192 , piece_id=3SZRRHSKLACT6BU6D7XHMEUB5IE44LBZEYBGZSLXIN5QIMJSC2ZA, seg_pos={0 0}, upload=445785]: Long tail cancellation due to optimal threshold Dec 26 19:29:26.101772885 (508h0m25.97285593s) [noise=true , address=45.91.116.9:4264 , piece_id=5LBGA57TPBIL7S43JI2YAKQP65Y5WFUIZC7N23PDGTXPI3M6ATXA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:47768->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:47768->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47768->45.91.116.9:4264: use of closed network connection Dec 26 19:29:26.101780570 (508h0m25.972863615s) [noise=true , address=45.91.116.9:4264 , piece_id=5LBGA57TPBIL7S43JI2YAKQP65Y5WFUIZC7N23PDGTXPI3M6ATXA, seg_pos={0 0}, upload=445785]: Long tail cancellation due to optimal threshold Dec 26 19:29:26.101975291 (508h0m25.973058346s) [noise=true , address=77.90.31.10:4520 , piece_id=LKQSJOYBOJDHMIS3CP6LE3XIX2AL5ZCJQDO37SIP5A54I7XCOWOQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 19:29:26.101982624 (508h0m25.97306567s) [noise=true , address=77.90.31.10:4520 , piece_id=LKQSJOYBOJDHMIS3CP6LE3XIX2AL5ZCJQDO37SIP5A54I7XCOWOQ, seg_pos={0 0}, upload=445785]: Long tail cancellation due to optimal threshold Dec 26 19:29:26.102003565 (508h0m25.97308661s) [seg_pos={0 0}, upload=445785]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:29:26.102101672 (508h0m25.973184717s) [upload=445785]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:29:26.102108585 (508h0m25.97319163s) [upload=445785]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:29:26.397196714 (508h0m26.268279749s) [upload=445785]: Done uploading segments... Dec 26 19:29:26.397215139 (508h0m26.268298184s) [upload=445785]: Upload finished. err: <nil> Duration: 30.005742919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXX] duration: 30.00440889s trace: https://jaeger.dev.storj.io/trace/24c8fc16133b06c FAIL (2025-12-26 19:28:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: KW5CPEMPKGPC4OGKTRSUQT6FNY3JPY4COBMNJIWI7RJI4CW3KUUQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: UOBAGMEZ5AU7JTV53WPRDF543MO4ZC6L6RU4VCLBPZOUEDUIW5MA) context deadline exceeded ======LOGS====== Dec 26 19:28:20.086996090 (507h59m19.958079135s) [upload=445770]: Starting upload Dec 26 19:28:20.087073647 (507h59m19.958156683s) [upload=445770]: Done starting upload Dec 26 19:28:20.087268749 (507h59m19.958351784s) [upload=445770]: Uploading segments... Dec 26 19:28:20.087304517 (507h59m19.958387553s) [upload=445770]: Got next segment. Inline: false Dec 26 19:28:20.087344824 (507h59m19.958427859s) [upload=445770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:28:20.087349112 (507h59m19.958432147s) [upload=445770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:28:20.347294434 (507h59m20.21837747s) [seg_pos={0 0}, upload=445770]: Begin upload segment... Dec 26 19:28:20.347343347 (507h59m20.218426382s) [seg_pos={0 0}, upload=445770]: Done begin upload segment. Dec 26 19:28:20.347376040 (507h59m20.218459075s) [noise=true , address=145.79.227.52:4447 , piece_id=A4KHWZ6SNPW5UABWXV224H6FJQQIKQEMNXGLAQIR2TCVRNP7D55A, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.348306672 (507h59m20.219389717s) [noise=true , address=145.79.227.41:4087 , piece_id=OT3PLCULCUEUMBXALNGP4JVIENRKUTZ76A3S3YH6WPUSSTSSSIMQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.348759534 (507h59m20.219842569s) [noise=true , address=45.91.116.7:4011 , piece_id=3JCPE5GG6XH4P4NTSUYTMMZYPOS4RX2FOD3KL2XRY6GD2UU7UCLA, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.349123998 (507h59m20.220207043s) [noise=true , address=145.79.227.36:4517 , piece_id=IK6HZP4TGLKUF45T3D33RB3IZO65RVGX55I5Y5CCCQUPYKD4EYMQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.349411435 (507h59m20.22049448s) [noise=true , address=45.91.116.10:4084 , piece_id=PJRZRVZFSM63OKSAQKNV2VHR5RBVGF2VWLJ7KL46ONP2CZQRVLSA, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.349477581 (507h59m20.220560616s) [noise=true , address=45.91.116.4:4939 , piece_id=2W6P7RHOX3ZI2ANYDXCGB44WKHA7C7KE5WPMPF6GRCPCESCFZBAQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.349885448 (507h59m20.220968483s) [noise=true , address=77.90.31.17:4773 , piece_id=XYKHLFRJOEVB2T654G6EODXUN5NX7WGLJZ4RZDFUDF2FRK526FQQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.350258659 (507h59m20.221341704s) [noise=true , address=45.91.116.11:4117 , piece_id=KW5CPEMPKGPC4OGKTRSUQT6FNY3JPY4COBMNJIWI7RJI4CW3KUUQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.350301380 (507h59m20.221384425s) [noise=true , address=77.90.31.19:4844 , piece_id=A5ALTTDV6T2EIOBVDV2ASHS2ZIE6B6BNKM6VLZXCCE54DI5JW2RQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.350618474 (507h59m20.221701519s) [noise=true , address=45.91.116.12:4226 , piece_id=DKKVVEMPD6WNDOS76EED4BPMVZMIDQTU7I5CH5453PJQXLOBZ2MQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.351026180 (507h59m20.222109215s) [noise=true , address=145.79.227.51:4411 , piece_id=GGDIWGBFN67LYUPYWOI6TVR3J7TRKWESLWO5IX4AWTGWG2WJQ4EA, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.351403048 (507h59m20.222486083s) [noise=true , address=145.79.227.5:4222 , piece_id=72TPYCQG7FS62LI5TL2DPNUPAJFV7M7LTOT5I2FWMZ7U5P7KNKBQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.351463422 (507h59m20.222546457s) [noise=true , address=45.91.116.6:4976 , piece_id=UOBAGMEZ5AU7JTV53WPRDF543MO4ZC6L6RU4VCLBPZOUEDUIW5MA, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.351836302 (507h59m20.222919338s) [noise=true , address=145.79.227.37:4567 , piece_id=4OLF5PS43TR3D2O2ZWAIQTG5RGPTI5M4HCHQ5R4HPTTYTDHLTHIQ, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.352027998 (507h59m20.223111033s) [noise=true , address=77.90.31.21:4878 , piece_id=27P3UY6S545J4I7DFOWZWSJPFXDPTPFSC3VBJJCV43IMHEBNJQGA, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:20.352277964 (507h59m20.223360999s) [noise=true , address=77.90.31.10:4517 , piece_id=NP6QYUDVAFNBEKYF7VAWMHPWC7GXWXBUY6ESYQED7OASRDFAXE5Q, seg_pos={0 0}, upload=445770]: Uploading piece... Dec 26 19:28:22.063140836 (507h59m21.934223871s) [upload=445770]: Next returned nil segment Dec 26 19:28:22.063156916 (507h59m21.934239951s) [upload=445770]: Waiting for error group managing segments... Dec 26 19:28:22.353113845 (507h59m22.22419688s) [noise=true , address=145.79.227.37:4567 , piece_id=4OLF5PS43TR3D2O2ZWAIQTG5RGPTI5M4HCHQ5R4HPTTYTDHLTHIQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.360348070 (507h59m22.231431105s) [noise=true , address=145.79.227.41:4087 , piece_id=OT3PLCULCUEUMBXALNGP4JVIENRKUTZ76A3S3YH6WPUSSTSSSIMQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.363007053 (507h59m22.234090098s) [noise=true , address=145.79.227.52:4447 , piece_id=A4KHWZ6SNPW5UABWXV224H6FJQQIKQEMNXGLAQIR2TCVRNP7D55A, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.375569715 (507h59m22.24665275s) [noise=true , address=145.79.227.36:4517 , piece_id=IK6HZP4TGLKUF45T3D33RB3IZO65RVGX55I5Y5CCCQUPYKD4EYMQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.375954317 (507h59m22.247037352s) [noise=true , address=145.79.227.51:4411 , piece_id=GGDIWGBFN67LYUPYWOI6TVR3J7TRKWESLWO5IX4AWTGWG2WJQ4EA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.464190494 (507h59m22.33527354s) [noise=true , address=145.79.227.5:4222 , piece_id=72TPYCQG7FS62LI5TL2DPNUPAJFV7M7LTOT5I2FWMZ7U5P7KNKBQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.590783859 (507h59m22.461866894s) [noise=true , address=45.91.116.10:4084 , piece_id=PJRZRVZFSM63OKSAQKNV2VHR5RBVGF2VWLJ7KL46ONP2CZQRVLSA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.596918882 (507h59m22.468001927s) [noise=true , address=45.91.116.11:4117 , piece_id=KW5CPEMPKGPC4OGKTRSUQT6FNY3JPY4COBMNJIWI7RJI4CW3KUUQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.598764727 (507h59m22.469847772s) [noise=true , address=45.91.116.6:4976 , piece_id=UOBAGMEZ5AU7JTV53WPRDF543MO4ZC6L6RU4VCLBPZOUEDUIW5MA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.661889257 (507h59m22.532972292s) [noise=true , address=45.91.116.7:4011 , piece_id=3JCPE5GG6XH4P4NTSUYTMMZYPOS4RX2FOD3KL2XRY6GD2UU7UCLA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.695616008 (507h59m22.566699044s) [noise=true , address=77.90.31.10:4517 , piece_id=NP6QYUDVAFNBEKYF7VAWMHPWC7GXWXBUY6ESYQED7OASRDFAXE5Q, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.700244672 (507h59m22.571327717s) [noise=true , address=45.91.116.4:4939 , piece_id=2W6P7RHOX3ZI2ANYDXCGB44WKHA7C7KE5WPMPF6GRCPCESCFZBAQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.707794629 (507h59m22.578877674s) [noise=true , address=45.91.116.12:4226 , piece_id=DKKVVEMPD6WNDOS76EED4BPMVZMIDQTU7I5CH5453PJQXLOBZ2MQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil> Dec 26 19:28:22.707819486 (507h59m22.578902531s) [seg_pos={0 0}, upload=445770]: Segment reached optimal threshold of 13 pieces. Dec 26 19:28:22.708029075 (507h59m22.579112111s) [noise=true , address=77.90.31.17:4773 , piece_id=XYKHLFRJOEVB2T654G6EODXUN5NX7WGLJZ4RZDFUDF2FRK526FQQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 19:28:22.708037171 (507h59m22.579120206s) [noise=true , address=77.90.31.17:4773 , piece_id=XYKHLFRJOEVB2T654G6EODXUN5NX7WGLJZ4RZDFUDF2FRK526FQQ, seg_pos={0 0}, upload=445770]: Long tail cancellation due to optimal threshold Dec 26 19:28:22.708141860 (507h59m22.579224925s) [noise=true , address=77.90.31.19:4844 , piece_id=A5ALTTDV6T2EIOBVDV2ASHS2ZIE6B6BNKM6VLZXCCE54DI5JW2RQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 19:28:22.708152390 (507h59m22.579235425s) [noise=true , address=77.90.31.19:4844 , piece_id=A5ALTTDV6T2EIOBVDV2ASHS2ZIE6B6BNKM6VLZXCCE54DI5JW2RQ, seg_pos={0 0}, upload=445770]: Long tail cancellation due to optimal threshold Dec 26 19:28:22.708283950 (507h59m22.579366996s) [noise=true , address=77.90.31.21:4878 , piece_id=27P3UY6S545J4I7DFOWZWSJPFXDPTPFSC3VBJJCV43IMHEBNJQGA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 19:28:22.708292557 (507h59m22.579375602s) [noise=true , address=77.90.31.21:4878 , piece_id=27P3UY6S545J4I7DFOWZWSJPFXDPTPFSC3VBJJCV43IMHEBNJQGA, seg_pos={0 0}, upload=445770]: Long tail cancellation due to optimal threshold Dec 26 19:28:22.708311754 (507h59m22.579394799s) [seg_pos={0 0}, upload=445770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:28:22.708452983 (507h59m22.579536018s) [upload=445770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:28:22.708459495 (507h59m22.57954256s) [upload=445770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:28:23.007946681 (507h59m22.879029726s) [upload=445770]: Done uploading segments... Dec 26 19:28:23.007965386 (507h59m22.879048421s) [upload=445770]: Upload finished. err: <nil> Duration: 30.004432645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XX] duration: 30.001734416s trace: https://jaeger.dev.storj.io/trace/57629f9909823f FAIL (2025-12-26 19:28:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:27:41.059559220 (507h58m40.930642255s) [upload=445758]: Starting upload Dec 26 19:27:41.059621878 (507h58m40.930704944s) [upload=445758]: Done starting upload Dec 26 19:27:41.059724604 (507h58m40.930807639s) [upload=445758]: Uploading segments... Dec 26 19:27:41.059836047 (507h58m40.930919092s) [upload=445758]: Got next segment. Inline: false Dec 26 19:27:41.059867397 (507h58m40.930950432s) [upload=445758]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:27:41.059871795 (507h58m40.93095483s) [upload=445758]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:27:41.308684257 (507h58m41.179767293s) [seg_pos={0 0}, upload=445758]: Begin upload segment... Dec 26 19:27:41.308727480 (507h58m41.179810515s) [seg_pos={0 0}, upload=445758]: Done begin upload segment. Dec 26 19:27:41.308777845 (507h58m41.17986088s) [noise=true , address=77.90.31.21:4877 , piece_id=O2P4YQO6LWZCB7AGAEQZWHUGD5KFGSMFLGQ6Q45EEEDHTBCYGXLA, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.308784939 (507h58m41.179867984s) [noise=true , address=45.91.116.7:4012 , piece_id=2HTHDBMFBNK57TG7HLF5WSV3SOPLZPAAYN4FURH4RSQ2TGEPTJWA, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.309341018 (507h58m41.180424063s) [noise=true , address=77.90.31.14:4663 , piece_id=6R3DSFZ7R5DNWU2WLKMXIYFQLKLDAO25L5UUSFNNULNIA3G2C5DQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.309500321 (507h58m41.180583356s) [noise=true , address=45.91.116.14:4189 , piece_id=FYPN5P3UX26IN3WWH32H7EKIY573XOXNYVPEC66G233C6WGESKQQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.309897037 (507h58m41.180980072s) [noise=true , address=77.90.31.10:4520 , piece_id=UJQNPXGGVTJM4BV35NCEVOUNPKLEEK4HXSZDSBVWSALQWZ2DX4SQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.309939598 (507h58m41.181022633s) [noise=true , address=145.79.227.49:4339 , piece_id=5S3X2XLFEP6ZCTFNTW2TS4RRP2K6QEKYIGKVJQJJWIHUPCTTXXKQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.310456903 (507h58m41.181539948s) [noise=true , address=45.91.116.5:4297 , piece_id=ZWU2RXIHKPM7WC2EXRCGCAJGGY25VOLWZ6XBER7C25V76WTL7JRQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.311146315 (507h58m41.18222936s) [noise=true , address=45.91.116.13:4156 , piece_id=YFMC5JXF4NE7FXLLPLZTPM37XC3PRPIBHCN23UCNSYUP62BRWDHQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.311898197 (507h58m41.182981232s) [noise=true , address=45.91.116.6:4976 , piece_id=3QSWQQUVFPENRSPFFLT67JISMOWMVBWUPUBEPKUVKEUWDIV6TDZA, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.312501115 (507h58m41.18358415s) [noise=true , address=145.79.227.46:4601 , piece_id=M5Q3IEZNDGYULCWTSIZC64JDMD7O5A2BTB6B7ZJHKWLIYGNLNMEQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.313163576 (507h58m41.184246611s) [noise=true , address=145.79.227.45:4229 , piece_id=BAGK7P6P4RE6IYZRSR3CUBFMM2735KFOKTSDECYRIUVOD4WMFOIQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.315011475 (507h58m41.18609452s) [noise=true , address=145.79.227.41:4085 , piece_id=6H6OPJTA36IGAA44MEB7TRUM57FXLHUTKAHAFVO7TUYCHSZUFZDQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.320588425 (507h58m41.19167147s) [noise=true , address=145.79.227.39:4013 , piece_id=4RBG7SCJ4LPWC7Z7WVBCSZSREMKEGXGF24LQWIF4YTLY4NXLQ4YA, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.322580969 (507h58m41.193664004s) [noise=true , address=145.79.227.38:4978 , piece_id=K54VTN3PCPRU25YRPZY6BMOKLNPXAI7KLTXJZ5A3FU5G2TSPY73Q, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.325330164 (507h58m41.196413209s) [noise=true , address=145.79.227.31:4799 , piece_id=HFLZ3NWOGQUGW652C2K4F7AO62JO33WPECV6XOMEAGMMLIFNL3VA, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:41.329506136 (507h58m41.200589181s) [noise=true , address=145.79.227.52:4445 , piece_id=GEY5C2RN52JCFMQYFRJNYZUWOFR3B2OMSAXLFJNYRHALWOM2NEIQ, seg_pos={0 0}, upload=445758]: Uploading piece... Dec 26 19:27:43.036143702 (507h58m42.907226747s) [upload=445758]: Next returned nil segment Dec 26 19:27:43.036155795 (507h58m42.90723884s) [upload=445758]: Waiting for error group managing segments... Dec 26 19:27:43.311910360 (507h58m43.182993395s) [noise=true , address=145.79.227.49:4339 , piece_id=5S3X2XLFEP6ZCTFNTW2TS4RRP2K6QEKYIGKVJQJJWIHUPCTTXXKQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.327633199 (507h58m43.198716234s) [noise=true , address=145.79.227.39:4013 , piece_id=4RBG7SCJ4LPWC7Z7WVBCSZSREMKEGXGF24LQWIF4YTLY4NXLQ4YA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.329777191 (507h58m43.200860226s) [noise=true , address=145.79.227.31:4799 , piece_id=HFLZ3NWOGQUGW652C2K4F7AO62JO33WPECV6XOMEAGMMLIFNL3VA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.340871687 (507h58m43.211954722s) [noise=true , address=145.79.227.38:4978 , piece_id=K54VTN3PCPRU25YRPZY6BMOKLNPXAI7KLTXJZ5A3FU5G2TSPY73Q, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.341378092 (507h58m43.212461127s) [noise=true , address=145.79.227.45:4229 , piece_id=BAGK7P6P4RE6IYZRSR3CUBFMM2735KFOKTSDECYRIUVOD4WMFOIQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.347898938 (507h58m43.218981973s) [noise=true , address=145.79.227.52:4445 , piece_id=GEY5C2RN52JCFMQYFRJNYZUWOFR3B2OMSAXLFJNYRHALWOM2NEIQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.357660586 (507h58m43.228743621s) [noise=true , address=145.79.227.46:4601 , piece_id=M5Q3IEZNDGYULCWTSIZC64JDMD7O5A2BTB6B7ZJHKWLIYGNLNMEQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.419216029 (507h58m43.290299064s) [noise=true , address=145.79.227.41:4085 , piece_id=6H6OPJTA36IGAA44MEB7TRUM57FXLHUTKAHAFVO7TUYCHSZUFZDQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.503482356 (507h58m43.374565402s) [noise=true , address=77.90.31.14:4663 , piece_id=6R3DSFZ7R5DNWU2WLKMXIYFQLKLDAO25L5UUSFNNULNIA3G2C5DQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.554993453 (507h58m43.426076488s) [noise=true , address=45.91.116.13:4156 , piece_id=YFMC5JXF4NE7FXLLPLZTPM37XC3PRPIBHCN23UCNSYUP62BRWDHQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.581941526 (507h58m43.453024561s) [noise=true , address=77.90.31.21:4877 , piece_id=O2P4YQO6LWZCB7AGAEQZWHUGD5KFGSMFLGQ6Q45EEEDHTBCYGXLA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.642103044 (507h58m43.513186079s) [noise=true , address=45.91.116.5:4297 , piece_id=ZWU2RXIHKPM7WC2EXRCGCAJGGY25VOLWZ6XBER7C25V76WTL7JRQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.714436521 (507h58m43.585519556s) [noise=true , address=45.91.116.6:4976 , piece_id=3QSWQQUVFPENRSPFFLT67JISMOWMVBWUPUBEPKUVKEUWDIV6TDZA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.714459804 (507h58m43.585542859s) [seg_pos={0 0}, upload=445758]: Segment reached optimal threshold of 13 pieces. Dec 26 19:27:43.714815792 (507h58m43.585898827s) [noise=true , address=45.91.116.14:4189 , piece_id=FYPN5P3UX26IN3WWH32H7EKIY573XOXNYVPEC66G233C6WGESKQQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 19:27:43.714843725 (507h58m43.58592677s) [noise=true , address=45.91.116.14:4189 , piece_id=FYPN5P3UX26IN3WWH32H7EKIY573XOXNYVPEC66G233C6WGESKQQ, seg_pos={0 0}, upload=445758]: Long tail cancellation due to optimal threshold Dec 26 19:27:43.715168605 (507h58m43.58625164s) [noise=true , address=45.91.116.7:4012 , piece_id=2HTHDBMFBNK57TG7HLF5WSV3SOPLZPAAYN4FURH4RSQ2TGEPTJWA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil> Dec 26 19:27:43.715246663 (507h58m43.586329718s) [noise=true , address=77.90.31.10:4520 , piece_id=UJQNPXGGVTJM4BV35NCEVOUNPKLEEK4HXSZDSBVWSALQWZ2DX4SQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 19:27:43.715255109 (507h58m43.586338144s) [noise=true , address=77.90.31.10:4520 , piece_id=UJQNPXGGVTJM4BV35NCEVOUNPKLEEK4HXSZDSBVWSALQWZ2DX4SQ, seg_pos={0 0}, upload=445758]: Long tail cancellation due to optimal threshold Dec 26 19:27:43.715274667 (507h58m43.586357702s) [seg_pos={0 0}, upload=445758]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 26 19:27:43.715349048 (507h58m43.586432083s) [upload=445758]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:27:43.715355821 (507h58m43.586438856s) [upload=445758]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:27:44.015517451 (507h58m43.886600496s) [upload=445758]: Done uploading segments... Dec 26 19:27:44.015545825 (507h58m43.88662886s) [upload=445758]: Upload finished. err: <nil> Duration: 30.0017577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.X] duration: 30.002821637s trace: https://jaeger.dev.storj.io/trace/5cf14e31711fa112 FAIL (2025-12-26 19:27:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07h57m46.181626598s) [upload=445745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:26:46.558397721 (507h57m46.429480756s) [seg_pos={0 0}, upload=445745]: Begin upload segment... Dec 26 19:26:46.558461483 (507h57m46.429544528s) [seg_pos={0 0}, upload=445745]: Done begin upload segment. Dec 26 19:26:46.558512139 (507h57m46.429595174s) [noise=true , address=145.79.227.31:4799 , piece_id=D23KY7Z3EWTZRBQVRCGKSROEJU5FHQOQK3YOTJRMH2G73AYHVAMA, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.558584446 (507h57m46.429667491s) [noise=true , address=145.79.227.19:4078 , piece_id=DSMGTXQHOOWHQZK5TMIKCRRVURMRZGW4ITQLWGCI5VPJZJYGDQ4A, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.559344594 (507h57m46.430427629s) [noise=true , address=145.79.227.37:4566 , piece_id=TO7ZWTW5D7RU4M74USQNIEKL6CK4WKVXNZX5K6WJXIQBOSHXEEGA, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.559836811 (507h57m46.430919846s) [noise=true , address=145.79.227.5:4221 , piece_id=F7ZH3PGVVDT6UFOMNCYSBGXRTMRHWBNFYDA4JSXJLNNCJ2YX3OCQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.560341412 (507h57m46.431424477s) [noise=true , address=145.79.227.38:4977 , piece_id=PPIG3XW3TXUVVBIS5J65IZ4WM3ASAB4J34XLZ53YPWVBKQQFXKRQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.560862735 (507h57m46.43194578s) [noise=true , address=45.91.116.14:4190 , piece_id=2K3353VUW2I2KBOCLLLWDGMERHFNLDSSEBNFR7EGY52VOIHLWDZQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.561407251 (507h57m46.432490296s) [noise=true , address=45.91.116.12:4227 , piece_id=FQ7KM3GU45ETPFADASPRBXHV535AIFPSQQJSALMMELX5WNO5ZMCQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.561934767 (507h57m46.433017812s) [noise=true , address=145.79.227.49:4338 , piece_id=X3JQGLVGVDUYYHPWERSGBGJK74PH2KSLLVENV5ODWTGFJ6LW4IOA, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.562337653 (507h57m46.433420688s) [noise=true , address=145.79.227.6:4257 , piece_id=EYHP7CQELXFTSYYEG6T6GVHX7WAXI325YN7BFQ3C4CWTBXCTZQVQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.562904202 (507h57m46.433987237s) [noise=true , address=145.79.227.50:4374 , piece_id=54QIJJQDG4XS3NHCBWSHLQVF33KFG3B7OZF2LRXFPLQGGOVR7SYA, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.564251387 (507h57m46.435334432s) [noise=true , address=77.90.31.17:4773 , piece_id=YA5COEY7NLE77SXXZGVTSA32BQQYDDH7IAMSC6EXEP6V2B4L74FQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.565609073 (507h57m46.436692108s) [noise=true , address=145.79.227.39:4015 , piece_id=SJEJOLCVIXHD6KQ2NF3QWWT5YJEP4AMCMCW2BEXRAJ6XD4WHHXIA, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.568821780 (507h57m46.439904815s) [noise=true , address=145.79.227.3:4151 , piece_id=BKD7F2XBZC5SYGOX2NVTS6UITKJ24MOQ6Z2PZHWBRDB7IFYZO3QQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.572671070 (507h57m46.443754125s) [noise=true , address=45.91.116.11:4118 , piece_id=QWEXSHDSNKB2JFCWH2X3ZOAO3LMPIN3EXFO4NPQGEC45E7TDRXLA, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.575938442 (507h57m46.447021477s) [noise=true , address=145.79.227.52:4445 , piece_id=TWEHRAHXCZ5T6SM7PH2RTPUHKQNT5O27C4AVLUP6US6VQBTYBVBQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:46.577463897 (507h57m46.448546942s) [noise=true , address=45.91.116.8:4045 , piece_id=XNFUL4XOEPIAJSHUL6ZSTEHOUJGBSGM5LRCV6JGCTWUCU72U7NTQ, seg_pos={0 0}, upload=445745]: Uploading piece... Dec 26 19:26:48.278009493 (507h57m48.149092528s) [upload=445745]: Next returned nil segment Dec 26 19:26:48.278029751 (507h57m48.149112796s) [upload=445745]: Waiting for error group managing segments... Dec 26 19:26:48.565447053 (507h57m48.436530088s) [noise=true , address=145.79.227.31:4799 , piece_id=D23KY7Z3EWTZRBQVRCGKSROEJU5FHQOQK3YOTJRMH2G73AYHVAMA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.568483436 (507h57m48.439566481s) [noise=true , address=145.79.227.38:4977 , piece_id=PPIG3XW3TXUVVBIS5J65IZ4WM3ASAB4J34XLZ53YPWVBKQQFXKRQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.578588057 (507h57m48.449671102s) [noise=true , address=145.79.227.37:4566 , piece_id=TO7ZWTW5D7RU4M74USQNIEKL6CK4WKVXNZX5K6WJXIQBOSHXEEGA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.578873851 (507h57m48.449956886s) [noise=true , address=145.79.227.52:4445 , piece_id=TWEHRAHXCZ5T6SM7PH2RTPUHKQNT5O27C4AVLUP6US6VQBTYBVBQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.584278612 (507h57m48.455361647s) [noise=true , address=145.79.227.50:4374 , piece_id=54QIJJQDG4XS3NHCBWSHLQVF33KFG3B7OZF2LRXFPLQGGOVR7SYA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.595013924 (507h57m48.46609697s) [noise=true , address=145.79.227.49:4338 , piece_id=X3JQGLVGVDUYYHPWERSGBGJK74PH2KSLLVENV5ODWTGFJ6LW4IOA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.598795354 (507h57m48.469878399s) [noise=true , address=145.79.227.39:4015 , piece_id=SJEJOLCVIXHD6KQ2NF3QWWT5YJEP4AMCMCW2BEXRAJ6XD4WHHXIA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.608734831 (507h57m48.479817866s) [noise=true , address=145.79.227.3:4151 , piece_id=BKD7F2XBZC5SYGOX2NVTS6UITKJ24MOQ6Z2PZHWBRDB7IFYZO3QQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.790524801 (507h57m48.661607846s) [noise=true , address=45.91.116.12:4227 , piece_id=FQ7KM3GU45ETPFADASPRBXHV535AIFPSQQJSALMMELX5WNO5ZMCQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.791413293 (507h57m48.662496328s) [noise=true , address=77.90.31.17:4773 , piece_id=YA5COEY7NLE77SXXZGVTSA32BQQYDDH7IAMSC6EXEP6V2B4L74FQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.850543698 (507h57m48.721626743s) [noise=true , address=45.91.116.14:4190 , piece_id=2K3353VUW2I2KBOCLLLWDGMERHFNLDSSEBNFR7EGY52VOIHLWDZQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:48.926229337 (507h57m48.797312382s) [noise=true , address=45.91.116.8:4045 , piece_id=XNFUL4XOEPIAJSHUL6ZSTEHOUJGBSGM5LRCV6JGCTWUCU72U7NTQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:49.689915969 (507h57m49.560999014s) [noise=true , address=45.91.116.11:4118 , piece_id=QWEXSHDSNKB2JFCWH2X3ZOAO3LMPIN3EXFO4NPQGEC45E7TDRXLA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil> Dec 26 19:26:49.689942380 (507h57m49.561025415s) [seg_pos={0 0}, upload=445745]: Segment reached optimal threshold of 13 pieces. Dec 26 19:26:49.690159282 (507h57m49.561242317s) [noise=true , address=145.79.227.19:4078 , piece_id=DSMGTXQHOOWHQZK5TMIKCRRVURMRZGW4ITQLWGCI5VPJZJYGDQ4A, seg_pos={0 0}, upload=445745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:59256->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:59256->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59256->145.79.227.19:4078: use of closed network connection Dec 26 19:26:49.690167197 (507h57m49.561250252s) [noise=true , address=145.79.227.19:4078 , piece_id=DSMGTXQHOOWHQZK5TMIKCRRVURMRZGW4ITQLWGCI5VPJZJYGDQ4A, seg_pos={0 0}, upload=445745]: Long tail cancellation due to optimal threshold Dec 26 19:26:49.690198667 (507h57m49.561281702s) [noise=true , address=145.79.227.6:4257 , piece_id=EYHP7CQELXFTSYYEG6T6GVHX7WAXI325YN7BFQ3C4CWTBXCTZQVQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:46082->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:46082->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46082->145.79.227.6:4257: use of closed network connection Dec 26 19:26:49.690209038 (507h57m49.561292073s) [noise=true , address=145.79.227.6:4257 , piece_id=EYHP7CQELXFTSYYEG6T6GVHX7WAXI325YN7BFQ3C4CWTBXCTZQVQ, seg_pos={0 0}, upload=445745]: Long tail cancellation due to optimal threshold Dec 26 19:26:49.690348513 (507h57m49.561431558s) [noise=true , address=145.79.227.5:4221 , piece_id=F7ZH3PGVVDT6UFOMNCYSBGXRTMRHWBNFYDA4JSXJLNNCJ2YX3OCQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:49458->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:49458->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->145.79.227.5:4221: use of closed network connection Dec 26 19:26:49.690357179 (507h57m49.561440225s) [noise=true , address=145.79.227.5:4221 , piece_id=F7ZH3PGVVDT6UFOMNCYSBGXRTMRHWBNFYDA4JSXJLNNCJ2YX3OCQ, seg_pos={0 0}, upload=445745]: Long tail cancellation due to optimal threshold Dec 26 19:26:49.690377147 (507h57m49.561460192s) [seg_pos={0 0}, upload=445745]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:26:49.690473090 (507h57m49.561556135s) [upload=445745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:26:49.690479582 (507h57m49.561562617s) [upload=445745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:26:49.990591157 (507h57m49.861674212s) [upload=445745]: Done uploading segments... Dec 26 19:26:49.990616936 (507h57m49.861699971s) [upload=445745]: Upload finished. err: <nil> Duration:* 30.00290669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.] duration: 6.145797684s trace: https://jaeger.dev.storj.io/trace/360f40575f4edd2b FAIL (2025-12-26 19:25:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h56m1.278588598s) [upload=445722]: Uploading segments... Dec 26 19:25:01.407644558 (507h56m1.278727603s) [upload=445722]: Got next segment. Inline: false Dec 26 19:25:01.407693922 (507h56m1.278776967s) [upload=445722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:25:01.407700344 (507h56m1.278783389s) [upload=445722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:25:01.657179045 (507h56m1.52826209s) [seg_pos={0 0}, upload=445722]: Begin upload segment... Dec 26 19:25:01.657242936 (507h56m1.528325981s) [seg_pos={0 0}, upload=445722]: Done begin upload segment. Dec 26 19:25:01.657299795 (507h56m1.52838284s) [noise=true , address=145.79.227.47:4267 , piece_id=FETXS2MBOLWO5CUY7BOIDV4VAEEKVXZPHRW3AO56UUVNLEPIP67A, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.657308621 (507h56m1.528391666s) [noise=true , address=45.91.116.11:4119 , piece_id=ZUNSEPXAHQXOV7BIGDSWZ6GGEBCGREMCRICWV2NVMPODXK7UUFHA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.657990390 (507h56m1.529073435s) [noise=true , address=145.79.227.15:4873 , piece_id=OOXL562IEKFINV33FNW3264QF6NEETQWLM74YOPIRAKKL6UGLNOA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.658037429 (507h56m1.529120504s) [noise=true , address=145.79.227.37:4565 , piece_id=6AHTF6RQ76A2MR2IAG7V6AJ5MYZPPR367327D6NHB3NJ26SGWE6Q, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.658388147 (507h56m1.529471182s) [noise=true , address=145.79.227.46:4603 , piece_id=2HOS7RDUD73RVXYS3Z2SNWR5RH6YD5AN5VSDWI2R6JACNJCGFFBA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.658746029 (507h56m1.529829104s) [noise=true , address=77.90.31.10:4520 , piece_id=BSQDSDIMCDH7OHIAS37ZNL6GJ4AQ6RE25C2ANYIXDDO5WCTZVYEA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.659188842 (507h56m1.530271887s) [noise=true , address=145.79.227.6:4257 , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.659525764 (507h56m1.530608799s) [noise=true , address=77.90.31.12:4590 , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.659840793 (507h56m1.530923838s) [noise=true , address=145.79.227.4:4185 , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.660203183 (507h56m1.531286228s) [noise=true , address=145.79.227.42:4121 , piece_id=PXOPHAFX7HVIZAAG7IJLCBMH47KHNOA7TWW3T23Q4N2AYTTAPB2Q, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.660577847 (507h56m1.531660892s) [noise=true , address=45.91.116.8:4045 , piece_id=FHKV65OZ4R7FMFVKAVYPVLEHZLQRIYHUXC7AGMLQCSPAQWTZRI2Q, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.660938073 (507h56m1.532021118s) [noise=true , address=145.79.227.51:4410 , piece_id=DNDEWSY5AW5BHM4M6Z6E4ODHA6WSNGHIA5GNQKYPYL6VVYVZXYGA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.661266889 (507h56m1.532349934s) [noise=true , address=145.79.227.38:4978 , piece_id=PZOZALTDLSEFO2SJNP54TVCI4JHBAVVQI4NKR7VN2BKSZBPEL62Q, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.661551861 (507h56m1.532634906s) [noise=true , address=45.91.116.7:4011 , piece_id=WW65UELB7XJGUWMW5EZXPSY7GPVVGJMMVHUJT55V5BNBMU5KTU4A, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.662362496 (507h56m1.533445531s) [noise=true , address=145.79.227.8:4330 , piece_id=U4LFLTQDT7SGRO5IIP3PCLSSWNA42CUPKPDDYFAZP4UAX5PTOG2A, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.663593539 (507h56m1.534676584s) [noise=true , address=77.90.31.18:4808 , piece_id=UV7CAJIXLA5LJWSN4LKSP2376KY3CV6DX7I7MXJT4NPB26LBOBGA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:03.289076833 (507h56m3.160159878s) [upload=445722]: Next returned nil segment Dec 26 19:25:03.289095970 (507h56m3.160179005s) [upload=445722]: Waiting for error group managing segments... Dec 26 19:25:03.685430911 (507h56m3.556513956s) [noise=true , address=145.79.227.38:4978 , piece_id=PZOZALTDLSEFO2SJNP54TVCI4JHBAVVQI4NKR7VN2BKSZBPEL62Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.687006622 (507h56m3.558089667s) [noise=true , address=145.79.227.8:4330 , piece_id=U4LFLTQDT7SGRO5IIP3PCLSSWNA42CUPKPDDYFAZP4UAX5PTOG2A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.691721880 (507h56m3.562804925s) [noise=true , address=145.79.227.37:4565 , piece_id=6AHTF6RQ76A2MR2IAG7V6AJ5MYZPPR367327D6NHB3NJ26SGWE6Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.692105290 (507h56m3.563188335s) [noise=true , address=145.79.227.46:4603 , piece_id=2HOS7RDUD73RVXYS3Z2SNWR5RH6YD5AN5VSDWI2R6JACNJCGFFBA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.692772531 (507h56m3.563855576s) [noise=true , address=145.79.227.47:4267 , piece_id=FETXS2MBOLWO5CUY7BOIDV4VAEEKVXZPHRW3AO56UUVNLEPIP67A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.696193235 (507h56m3.56727629s) [noise=true , address=145.79.227.42:4121 , piece_id=PXOPHAFX7HVIZAAG7IJLCBMH47KHNOA7TWW3T23Q4N2AYTTAPB2Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.698467325 (507h56m3.56955037s) [noise=true , address=145.79.227.51:4410 , piece_id=DNDEWSY5AW5BHM4M6Z6E4ODHA6WSNGHIA5GNQKYPYL6VVYVZXYGA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.731878005 (507h56m3.60296105s) [noise=true , address=145.79.227.15:4873 , piece_id=OOXL562IEKFINV33FNW3264QF6NEETQWLM74YOPIRAKKL6UGLNOA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.855972482 (507h56m3.727055528s) [noise=true , address=77.90.31.18:4808 , piece_id=UV7CAJIXLA5LJWSN4LKSP2376KY3CV6DX7I7MXJT4NPB26LBOBGA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.942103340 (507h56m3.813186385s) [noise=true , address=45.91.116.7:4011 , piece_id=WW65UELB7XJGUWMW5EZXPSY7GPVVGJMMVHUJT55V5BNBMU5KTU4A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.950017160 (507h56m3.821100205s) [noise=true , address=45.91.116.11:4119 , piece_id=ZUNSEPXAHQXOV7BIGDSWZ6GGEBCGREMCRICWV2NVMPODXK7UUFHA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:04.053891054 (507h56m3.924974099s) [noise=true , address=45.91.116.8:4045 , piece_id=FHKV65OZ4R7FMFVKAVYPVLEHZLQRIYHUXC7AGMLQCSPAQWTZRI2Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:04.085425060 (507h56m3.956508105s) [noise=true , address=77.90.31.10:4520 , piece_id=BSQDSDIMCDH7OHIAS37ZNL6GJ4AQ6RE25C2ANYIXDDO5WCTZVYEA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:04.085451440 (507h56m3.956534485s) [seg_pos={0 0}, upload=445722]: Segment reached optimal threshold of 13 pieces. Dec 26 19:25:04.085663595 (507h56m3.95674664s) [noise=true , address=145.79.227.4:4185 , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection Dec 26 19:25:04.085670157 (507h56m3.956753202s) [noise=true , address=145.79.227.4:4185 , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold Dec 26 19:25:04.085854888 (507h56m3.956937923s) [noise=true , address=77.90.31.12:4590 , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 19:25:04.085873264 (507h56m3.956956299s) [noise=true , address=77.90.31.12:4590 , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold Dec 26 19:25:04.086049079 (507h56m3.957132144s) [noise=true , address=145.79.227.6:4257 , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection Dec 26 19:25:04.086060330 (507h56m3.957143375s) [noise=true , address=145.79.227.6:4257 , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold Dec 26 19:25:04.086087442 (507h56m3.957170487s) [seg_pos={0 0}, upload=445722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:25:04.086189787 (507h56m3.957272832s) [upload=445722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:25:04.086195668 (507h56m3.957278713s) [upload=445722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:25:04.388933212 (507h56m4.260016268s) [upload=445722]: Done uploading segments... Dec 26 19:25:04.388956417 (507h56m4.260039462s) [upload=445722]: Upload finished. err: <nil> Duration: 6.145815126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X] duration: 30.002326734s trace: https://jaeger.dev.storj.io/trace/360f40575f4edd2b FAIL (2025-12-26 19:25:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h56m1.278588598s) [upload=445722]: Uploading segments... Dec 26 19:25:01.407644558 (507h56m1.278727603s) [upload=445722]: Got next segment. Inline: false Dec 26 19:25:01.407693922 (507h56m1.278776967s) [upload=445722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:25:01.407700344 (507h56m1.278783389s) [upload=445722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:25:01.657179045 (507h56m1.52826209s) [seg_pos={0 0}, upload=445722]: Begin upload segment... Dec 26 19:25:01.657242936 (507h56m1.528325981s) [seg_pos={0 0}, upload=445722]: Done begin upload segment. Dec 26 19:25:01.657299795 (507h56m1.52838284s) [noise=true , address=145.79.227.47:4267 , piece_id=FETXS2MBOLWO5CUY7BOIDV4VAEEKVXZPHRW3AO56UUVNLEPIP67A, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.657308621 (507h56m1.528391666s) [noise=true , address=45.91.116.11:4119 , piece_id=ZUNSEPXAHQXOV7BIGDSWZ6GGEBCGREMCRICWV2NVMPODXK7UUFHA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.657990390 (507h56m1.529073435s) [noise=true , address=145.79.227.15:4873 , piece_id=OOXL562IEKFINV33FNW3264QF6NEETQWLM74YOPIRAKKL6UGLNOA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.658037429 (507h56m1.529120504s) [noise=true , address=145.79.227.37:4565 , piece_id=6AHTF6RQ76A2MR2IAG7V6AJ5MYZPPR367327D6NHB3NJ26SGWE6Q, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.658388147 (507h56m1.529471182s) [noise=true , address=145.79.227.46:4603 , piece_id=2HOS7RDUD73RVXYS3Z2SNWR5RH6YD5AN5VSDWI2R6JACNJCGFFBA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.658746029 (507h56m1.529829104s) [noise=true , address=77.90.31.10:4520 , piece_id=BSQDSDIMCDH7OHIAS37ZNL6GJ4AQ6RE25C2ANYIXDDO5WCTZVYEA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.659188842 (507h56m1.530271887s) [noise=true , address=145.79.227.6:4257 , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.659525764 (507h56m1.530608799s) [noise=true , address=77.90.31.12:4590 , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.659840793 (507h56m1.530923838s) [noise=true , address=145.79.227.4:4185 , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.660203183 (507h56m1.531286228s) [noise=true , address=145.79.227.42:4121 , piece_id=PXOPHAFX7HVIZAAG7IJLCBMH47KHNOA7TWW3T23Q4N2AYTTAPB2Q, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.660577847 (507h56m1.531660892s) [noise=true , address=45.91.116.8:4045 , piece_id=FHKV65OZ4R7FMFVKAVYPVLEHZLQRIYHUXC7AGMLQCSPAQWTZRI2Q, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.660938073 (507h56m1.532021118s) [noise=true , address=145.79.227.51:4410 , piece_id=DNDEWSY5AW5BHM4M6Z6E4ODHA6WSNGHIA5GNQKYPYL6VVYVZXYGA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.661266889 (507h56m1.532349934s) [noise=true , address=145.79.227.38:4978 , piece_id=PZOZALTDLSEFO2SJNP54TVCI4JHBAVVQI4NKR7VN2BKSZBPEL62Q, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.661551861 (507h56m1.532634906s) [noise=true , address=45.91.116.7:4011 , piece_id=WW65UELB7XJGUWMW5EZXPSY7GPVVGJMMVHUJT55V5BNBMU5KTU4A, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.662362496 (507h56m1.533445531s) [noise=true , address=145.79.227.8:4330 , piece_id=U4LFLTQDT7SGRO5IIP3PCLSSWNA42CUPKPDDYFAZP4UAX5PTOG2A, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:01.663593539 (507h56m1.534676584s) [noise=true , address=77.90.31.18:4808 , piece_id=UV7CAJIXLA5LJWSN4LKSP2376KY3CV6DX7I7MXJT4NPB26LBOBGA, seg_pos={0 0}, upload=445722]: Uploading piece... Dec 26 19:25:03.289076833 (507h56m3.160159878s) [upload=445722]: Next returned nil segment Dec 26 19:25:03.289095970 (507h56m3.160179005s) [upload=445722]: Waiting for error group managing segments... Dec 26 19:25:03.685430911 (507h56m3.556513956s) [noise=true , address=145.79.227.38:4978 , piece_id=PZOZALTDLSEFO2SJNP54TVCI4JHBAVVQI4NKR7VN2BKSZBPEL62Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.687006622 (507h56m3.558089667s) [noise=true , address=145.79.227.8:4330 , piece_id=U4LFLTQDT7SGRO5IIP3PCLSSWNA42CUPKPDDYFAZP4UAX5PTOG2A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.691721880 (507h56m3.562804925s) [noise=true , address=145.79.227.37:4565 , piece_id=6AHTF6RQ76A2MR2IAG7V6AJ5MYZPPR367327D6NHB3NJ26SGWE6Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.692105290 (507h56m3.563188335s) [noise=true , address=145.79.227.46:4603 , piece_id=2HOS7RDUD73RVXYS3Z2SNWR5RH6YD5AN5VSDWI2R6JACNJCGFFBA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.692772531 (507h56m3.563855576s) [noise=true , address=145.79.227.47:4267 , piece_id=FETXS2MBOLWO5CUY7BOIDV4VAEEKVXZPHRW3AO56UUVNLEPIP67A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.696193235 (507h56m3.56727629s) [noise=true , address=145.79.227.42:4121 , piece_id=PXOPHAFX7HVIZAAG7IJLCBMH47KHNOA7TWW3T23Q4N2AYTTAPB2Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.698467325 (507h56m3.56955037s) [noise=true , address=145.79.227.51:4410 , piece_id=DNDEWSY5AW5BHM4M6Z6E4ODHA6WSNGHIA5GNQKYPYL6VVYVZXYGA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.731878005 (507h56m3.60296105s) [noise=true , address=145.79.227.15:4873 , piece_id=OOXL562IEKFINV33FNW3264QF6NEETQWLM74YOPIRAKKL6UGLNOA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.855972482 (507h56m3.727055528s) [noise=true , address=77.90.31.18:4808 , piece_id=UV7CAJIXLA5LJWSN4LKSP2376KY3CV6DX7I7MXJT4NPB26LBOBGA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.942103340 (507h56m3.813186385s) [noise=true , address=45.91.116.7:4011 , piece_id=WW65UELB7XJGUWMW5EZXPSY7GPVVGJMMVHUJT55V5BNBMU5KTU4A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:03.950017160 (507h56m3.821100205s) [noise=true , address=45.91.116.11:4119 , piece_id=ZUNSEPXAHQXOV7BIGDSWZ6GGEBCGREMCRICWV2NVMPODXK7UUFHA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:04.053891054 (507h56m3.924974099s) [noise=true , address=45.91.116.8:4045 , piece_id=FHKV65OZ4R7FMFVKAVYPVLEHZLQRIYHUXC7AGMLQCSPAQWTZRI2Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:04.085425060 (507h56m3.956508105s) [noise=true , address=77.90.31.10:4520 , piece_id=BSQDSDIMCDH7OHIAS37ZNL6GJ4AQ6RE25C2ANYIXDDO5WCTZVYEA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil> Dec 26 19:25:04.085451440 (507h56m3.956534485s) [seg_pos={0 0}, upload=445722]: Segment reached optimal threshold of 13 pieces. Dec 26 19:25:04.085663595 (507h56m3.95674664s) [noise=true , address=145.79.227.4:4185 , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection Dec 26 19:25:04.085670157 (507h56m3.956753202s) [noise=true , address=145.79.227.4:4185 , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold Dec 26 19:25:04.085854888 (507h56m3.956937923s) [noise=true , address=77.90.31.12:4590 , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 19:25:04.085873264 (507h56m3.956956299s) [noise=true , address=77.90.31.12:4590 , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold Dec 26 19:25:04.086049079 (507h56m3.957132144s) [noise=true , address=145.79.227.6:4257 , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection Dec 26 19:25:04.086060330 (507h56m3.957143375s) [noise=true , address=145.79.227.6:4257 , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold Dec 26 19:25:04.086087442 (507h56m3.957170487s) [seg_pos={0 0}, upload=445722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:25:04.086189787 (507h56m3.957272832s) [upload=445722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:25:04.086195668 (507h56m3.957278713s) [upload=445722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:25:04.388933212 (507h56m4.260016268s) [upload=445722]: Done uploading segments... Dec 26 19:25:04.388956417 (507h56m4.260039462s) [upload=445722]: Upload finished. err: <nil> Duration: 30.002350218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....] duration: 6.097242214s trace: https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19 FAIL (2025-12-26 19:24:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment... Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment. Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments... Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces. Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments... Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil> Duration: 6.097268906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X...] duration: 10.602774485s trace: https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19 FAIL (2025-12-26 19:24:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment... Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment. Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments... Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces. Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments... Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil> Duration: 10.602811286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X..] duration: 6.150134131s trace: https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19 FAIL (2025-12-26 19:23:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment... Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment. Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments... Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces. Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments... Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil> Duration: 6.150166372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X.] duration: 6.983247294s trace: https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19 FAIL (2025-12-26 19:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment... Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment. Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments... Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces. Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments... Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil> Duration: 6.983279655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X] duration: 30.003050232s trace: https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19 FAIL (2025-12-26 19:22:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment... Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment. Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece... Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments... Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229 , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979 , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447 , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566 , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373 , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475 , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601 , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509 , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873 , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014 , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736 , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519 , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973 , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil> Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces. Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086 , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115 , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078 , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments... Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil> Duration: 30.003079167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.] duration: 6.138537759s trace: https://jaeger.dev.storj.io/trace/568ddfc5b1abce90 FAIL (2025-12-26 19:20:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ) context deadline exceeded; piecestore: (Node ID: 12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, Piece ID: INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ) context deadline exceeded ======LOGS====== ec 26 19:19:30.227211085 (507h50m30.09829413s) [upload=445644]: Starting upload Dec 26 19:19:30.227295465 (507h50m30.09837852s) [upload=445644]: Done starting upload Dec 26 19:19:30.227427417 (507h50m30.098510452s) [upload=445644]: Uploading segments... Dec 26 19:19:30.227716587 (507h50m30.098799622s) [upload=445644]: Got next segment. Inline: false Dec 26 19:19:30.227772574 (507h50m30.098855619s) [upload=445644]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:19:30.227778886 (507h50m30.098861941s) [upload=445644]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:19:30.476589825 (507h50m30.34767286s) [seg_pos={0 0}, upload=445644]: Begin upload segment... Dec 26 19:19:30.476641673 (507h50m30.347724708s) [seg_pos={0 0}, upload=445644]: Done begin upload segment. Dec 26 19:19:30.476666711 (507h50m30.347749746s) [noise=true , address=145.79.227.40:4050 , piece_id=ZWXNCWZULTXE5LCG4QF7AQWWR34JCFTTZQVMJWEBDPHR3FJS2RUQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.476722157 (507h50m30.347805192s) [noise=true , address=77.90.31.14:4663 , piece_id=FL6ENXYOQXN43CDISST24MCBT3F6HUMCPNL5ODU32K3VEC4NFEZA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477203974 (507h50m30.348287009s) [noise=true , address=145.79.227.36:4518 , piece_id=BXZPJQI3CAXRQEYGIMGSRYBTGAK4LU437WUHXOP6WIQ3NBQCSXYQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477215627 (507h50m30.348298662s) [noise=true , address=45.91.116.11:4119 , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477565563 (507h50m30.348648608s) [noise=true , address=77.90.31.10:4518 , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477769281 (507h50m30.348852316s) [noise=true , address=145.79.227.49:4337 , piece_id=BB7ALN4BNYPJILNJ7FNXMC6ITAO7SMPP5R3HHMDRIQJTEFDGVR5A, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477949394 (507h50m30.349032429s) [noise=true , address=45.91.116.9:4264 , piece_id=7DWE6EUWCORGIZMSQ4ZSI3ITDM5JXPGUHKRVVW4FAPYNBTEDADSQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478139976 (507h50m30.349223011s) [noise=true , address=145.79.227.38:4977 , piece_id=LWI4RXWMFJK2CROE7CSPOEVEB4TP2FKXPY3NWYLJDSJMLPXUWWFQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478320461 (507h50m30.349403506s) [noise=true , address=145.79.227.11:4732 , piece_id=JYLU2BPQ255J4YVWS4XXPSE6FZBNQFBOT5EYE43AFSWPK7IFBHPQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478486897 (507h50m30.349569932s) [noise=true , address=145.79.227.46:4601 , piece_id=XW4ED5LYEKC6CCETSJQXM2NTOTBGPGOTKZZKSCKFFEX43X3VZBDA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478634619 (507h50m30.349717654s) [noise=true , address=77.90.31.16:4736 , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478860288 (507h50m30.349943333s) [noise=true , address=145.79.227.50:4373 , piece_id=STLHC2XIAWAJUF5B2QSR22LOTNMPL2DVPKV6IEOVCVVRM7OOCGTA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478986719 (507h50m30.350069754s) [noise=true , address=45.91.116.13:4156 , piece_id=XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.479264238 (507h50m30.350347273s) [noise=true , address=145.79.227.45:4231 , piece_id=ECT3RFR36JKZ3OEDUIYAC3ZBC6N7TH5TBKZJKU4VNOLNZHNV7JFA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.479288273 (507h50m30.350371318s) [noise=true , address=145.79.227.48:4301 , piece_id=BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.479655703 (507h50m30.350738738s) [noise=true , address=145.79.227.15:4874 , piece_id=INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:32.181356449 (507h50m32.052439484s) [upload=445644]: Next returned nil segment Dec 26 19:19:32.181390153 (507h50m32.052473198s) [upload=445644]: Waiting for error group managing segments... Dec 26 19:19:32.474459137 (507h50m32.345542182s) [noise=true , address=145.79.227.50:4373 , piece_id=STLHC2XIAWAJUF5B2QSR22LOTNMPL2DVPKV6IEOVCVVRM7OOCGTA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.479753769 (507h50m32.350836804s) [noise=true , address=145.79.227.38:4977 , piece_id=LWI4RXWMFJK2CROE7CSPOEVEB4TP2FKXPY3NWYLJDSJMLPXUWWFQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.485159292 (507h50m32.356242327s) [noise=true , address=145.79.227.48:4301 , piece_id=BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.487314757 (507h50m32.358397792s) [noise=true , address=145.79.227.40:4050 , piece_id=ZWXNCWZULTXE5LCG4QF7AQWWR34JCFTTZQVMJWEBDPHR3FJS2RUQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.490154434 (507h50m32.361237469s) [noise=true , address=145.79.227.11:4732 , piece_id=JYLU2BPQ255J4YVWS4XXPSE6FZBNQFBOT5EYE43AFSWPK7IFBHPQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.493673595 (507h50m32.36475664s) [noise=true , address=145.79.227.46:4601 , piece_id=XW4ED5LYEKC6CCETSJQXM2NTOTBGPGOTKZZKSCKFFEX43X3VZBDA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.495333936 (507h50m32.366416971s) [noise=true , address=145.79.227.36:4518 , piece_id=BXZPJQI3CAXRQEYGIMGSRYBTGAK4LU437WUHXOP6WIQ3NBQCSXYQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.496214773 (507h50m32.367297808s) [noise=true , address=145.79.227.15:4874 , piece_id=INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.497151808 (507h50m32.368234843s) [noise=true , address=145.79.227.45:4231 , piece_id=ECT3RFR36JKZ3OEDUIYAC3ZBC6N7TH5TBKZJKU4VNOLNZHNV7JFA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.507448064 (507h50m32.378531099s) [noise=true , address=145.79.227.49:4337 , piece_id=BB7ALN4BNYPJILNJ7FNXMC6ITAO7SMPP5R3HHMDRIQJTEFDGVR5A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.729908207 (507h50m32.600991242s) [noise=true , address=45.91.116.13:4156 , piece_id=XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.814922039 (507h50m32.686005084s) [noise=true , address=45.91.116.9:4264 , piece_id=7DWE6EUWCORGIZMSQ4ZSI3ITDM5JXPGUHKRVVW4FAPYNBTEDADSQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.880215529 (507h50m32.751298574s) [noise=true , address=77.90.31.14:4663 , piece_id=FL6ENXYOQXN43CDISST24MCBT3F6HUMCPNL5ODU32K3VEC4NFEZA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.880240045 (507h50m32.75132308s) [seg_pos={0 0}, upload=445644]: Segment reached optimal threshold of 13 pieces. Dec 26 19:19:32.880432512 (507h50m32.751515547s) [noise=true , address=77.90.31.10:4518 , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 19:19:32.880440587 (507h50m32.751523632s) [noise=true , address=77.90.31.10:4518 , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold Dec 26 19:19:32.880449193 (507h50m32.751532238s) [noise=true , address=45.91.116.11:4119 , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 19:19:32.880457460 (507h50m32.751540495s) [noise=true , address=45.91.116.11:4119 , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold Dec 26 19:19:32.880586004 (507h50m32.751669039s) [noise=true , address=77.90.31.16:4736 , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection Dec 26 19:19:32.880592556 (507h50m32.751675601s) [noise=true , address=77.90.31.16:4736 , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold Dec 26 19:19:32.880618526 (507h50m32.751701561s) [seg_pos={0 0}, upload=445644]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:19:32.880688279 (507h50m32.751771314s) [upload=445644]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:19:32.880694290 (507h50m32.751777325s) [upload=445644]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:19:33.182498829 (507h50m33.053581874s) [upload=445644]: Done uploading segments... Dec 26 19:19:33.182520670 (507h50m33.053603705s) [upload=445644]: Upload finished. err: <nil> Duration: 6.138565352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX] duration: 30.00262386s trace: https://jaeger.dev.storj.io/trace/568ddfc5b1abce90 FAIL (2025-12-26 19:19:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ) context deadline exceeded; piecestore: (Node ID: 12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, Piece ID: INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ) context deadline exceeded ======LOGS====== ec 26 19:19:30.227211085 (507h50m30.09829413s) [upload=445644]: Starting upload Dec 26 19:19:30.227295465 (507h50m30.09837852s) [upload=445644]: Done starting upload Dec 26 19:19:30.227427417 (507h50m30.098510452s) [upload=445644]: Uploading segments... Dec 26 19:19:30.227716587 (507h50m30.098799622s) [upload=445644]: Got next segment. Inline: false Dec 26 19:19:30.227772574 (507h50m30.098855619s) [upload=445644]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:19:30.227778886 (507h50m30.098861941s) [upload=445644]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:19:30.476589825 (507h50m30.34767286s) [seg_pos={0 0}, upload=445644]: Begin upload segment... Dec 26 19:19:30.476641673 (507h50m30.347724708s) [seg_pos={0 0}, upload=445644]: Done begin upload segment. Dec 26 19:19:30.476666711 (507h50m30.347749746s) [noise=true , address=145.79.227.40:4050 , piece_id=ZWXNCWZULTXE5LCG4QF7AQWWR34JCFTTZQVMJWEBDPHR3FJS2RUQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.476722157 (507h50m30.347805192s) [noise=true , address=77.90.31.14:4663 , piece_id=FL6ENXYOQXN43CDISST24MCBT3F6HUMCPNL5ODU32K3VEC4NFEZA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477203974 (507h50m30.348287009s) [noise=true , address=145.79.227.36:4518 , piece_id=BXZPJQI3CAXRQEYGIMGSRYBTGAK4LU437WUHXOP6WIQ3NBQCSXYQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477215627 (507h50m30.348298662s) [noise=true , address=45.91.116.11:4119 , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477565563 (507h50m30.348648608s) [noise=true , address=77.90.31.10:4518 , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477769281 (507h50m30.348852316s) [noise=true , address=145.79.227.49:4337 , piece_id=BB7ALN4BNYPJILNJ7FNXMC6ITAO7SMPP5R3HHMDRIQJTEFDGVR5A, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.477949394 (507h50m30.349032429s) [noise=true , address=45.91.116.9:4264 , piece_id=7DWE6EUWCORGIZMSQ4ZSI3ITDM5JXPGUHKRVVW4FAPYNBTEDADSQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478139976 (507h50m30.349223011s) [noise=true , address=145.79.227.38:4977 , piece_id=LWI4RXWMFJK2CROE7CSPOEVEB4TP2FKXPY3NWYLJDSJMLPXUWWFQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478320461 (507h50m30.349403506s) [noise=true , address=145.79.227.11:4732 , piece_id=JYLU2BPQ255J4YVWS4XXPSE6FZBNQFBOT5EYE43AFSWPK7IFBHPQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478486897 (507h50m30.349569932s) [noise=true , address=145.79.227.46:4601 , piece_id=XW4ED5LYEKC6CCETSJQXM2NTOTBGPGOTKZZKSCKFFEX43X3VZBDA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478634619 (507h50m30.349717654s) [noise=true , address=77.90.31.16:4736 , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478860288 (507h50m30.349943333s) [noise=true , address=145.79.227.50:4373 , piece_id=STLHC2XIAWAJUF5B2QSR22LOTNMPL2DVPKV6IEOVCVVRM7OOCGTA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.478986719 (507h50m30.350069754s) [noise=true , address=45.91.116.13:4156 , piece_id=XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.479264238 (507h50m30.350347273s) [noise=true , address=145.79.227.45:4231 , piece_id=ECT3RFR36JKZ3OEDUIYAC3ZBC6N7TH5TBKZJKU4VNOLNZHNV7JFA, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.479288273 (507h50m30.350371318s) [noise=true , address=145.79.227.48:4301 , piece_id=BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:30.479655703 (507h50m30.350738738s) [noise=true , address=145.79.227.15:4874 , piece_id=INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ, seg_pos={0 0}, upload=445644]: Uploading piece... Dec 26 19:19:32.181356449 (507h50m32.052439484s) [upload=445644]: Next returned nil segment Dec 26 19:19:32.181390153 (507h50m32.052473198s) [upload=445644]: Waiting for error group managing segments... Dec 26 19:19:32.474459137 (507h50m32.345542182s) [noise=true , address=145.79.227.50:4373 , piece_id=STLHC2XIAWAJUF5B2QSR22LOTNMPL2DVPKV6IEOVCVVRM7OOCGTA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.479753769 (507h50m32.350836804s) [noise=true , address=145.79.227.38:4977 , piece_id=LWI4RXWMFJK2CROE7CSPOEVEB4TP2FKXPY3NWYLJDSJMLPXUWWFQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.485159292 (507h50m32.356242327s) [noise=true , address=145.79.227.48:4301 , piece_id=BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.487314757 (507h50m32.358397792s) [noise=true , address=145.79.227.40:4050 , piece_id=ZWXNCWZULTXE5LCG4QF7AQWWR34JCFTTZQVMJWEBDPHR3FJS2RUQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.490154434 (507h50m32.361237469s) [noise=true , address=145.79.227.11:4732 , piece_id=JYLU2BPQ255J4YVWS4XXPSE6FZBNQFBOT5EYE43AFSWPK7IFBHPQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.493673595 (507h50m32.36475664s) [noise=true , address=145.79.227.46:4601 , piece_id=XW4ED5LYEKC6CCETSJQXM2NTOTBGPGOTKZZKSCKFFEX43X3VZBDA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.495333936 (507h50m32.366416971s) [noise=true , address=145.79.227.36:4518 , piece_id=BXZPJQI3CAXRQEYGIMGSRYBTGAK4LU437WUHXOP6WIQ3NBQCSXYQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.496214773 (507h50m32.367297808s) [noise=true , address=145.79.227.15:4874 , piece_id=INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.497151808 (507h50m32.368234843s) [noise=true , address=145.79.227.45:4231 , piece_id=ECT3RFR36JKZ3OEDUIYAC3ZBC6N7TH5TBKZJKU4VNOLNZHNV7JFA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.507448064 (507h50m32.378531099s) [noise=true , address=145.79.227.49:4337 , piece_id=BB7ALN4BNYPJILNJ7FNXMC6ITAO7SMPP5R3HHMDRIQJTEFDGVR5A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.729908207 (507h50m32.600991242s) [noise=true , address=45.91.116.13:4156 , piece_id=XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.814922039 (507h50m32.686005084s) [noise=true , address=45.91.116.9:4264 , piece_id=7DWE6EUWCORGIZMSQ4ZSI3ITDM5JXPGUHKRVVW4FAPYNBTEDADSQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.880215529 (507h50m32.751298574s) [noise=true , address=77.90.31.14:4663 , piece_id=FL6ENXYOQXN43CDISST24MCBT3F6HUMCPNL5ODU32K3VEC4NFEZA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil> Dec 26 19:19:32.880240045 (507h50m32.75132308s) [seg_pos={0 0}, upload=445644]: Segment reached optimal threshold of 13 pieces. Dec 26 19:19:32.880432512 (507h50m32.751515547s) [noise=true , address=77.90.31.10:4518 , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 19:19:32.880440587 (507h50m32.751523632s) [noise=true , address=77.90.31.10:4518 , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold Dec 26 19:19:32.880449193 (507h50m32.751532238s) [noise=true , address=45.91.116.11:4119 , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 19:19:32.880457460 (507h50m32.751540495s) [noise=true , address=45.91.116.11:4119 , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold Dec 26 19:19:32.880586004 (507h50m32.751669039s) [noise=true , address=77.90.31.16:4736 , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection Dec 26 19:19:32.880592556 (507h50m32.751675601s) [noise=true , address=77.90.31.16:4736 , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold Dec 26 19:19:32.880618526 (507h50m32.751701561s) [seg_pos={0 0}, upload=445644]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:19:32.880688279 (507h50m32.751771314s) [upload=445644]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:19:32.880694290 (507h50m32.751777325s) [upload=445644]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:19:33.182498829 (507h50m33.053581874s) [upload=445644]: Done uploading segments... Dec 26 19:19:33.182520670 (507h50m33.053603705s) [upload=445644]: Upload finished. err: <nil> Duration: 30.002641323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XX] duration: 30.002375608s trace: https://jaeger.dev.storj.io/trace/65ca9b941f274d1a FAIL (2025-12-26 19:18:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 19:18:26.318350402 (507h49m26.189433437s) [upload=445627]: Starting upload Dec 26 19:18:26.318422900 (507h49m26.189505935s) [upload=445627]: Done starting upload Dec 26 19:18:26.318537800 (507h49m26.189620845s) [upload=445627]: Uploading segments... Dec 26 19:18:26.318653029 (507h49m26.189736074s) [upload=445627]: Got next segment. Inline: false Dec 26 19:18:26.318703836 (507h49m26.189786881s) [upload=445627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:18:26.318710338 (507h49m26.189793373s) [upload=445627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:18:26.572049420 (507h49m26.443132455s) [seg_pos={0 0}, upload=445627]: Begin upload segment... Dec 26 19:18:26.572135163 (507h49m26.443218198s) [seg_pos={0 0}, upload=445627]: Done begin upload segment. Dec 26 19:18:26.572203884 (507h49m26.443286929s) [noise=true , address=77.90.31.12:4590 , piece_id=VMELCTDYY7FCQQXJH2NPPRAZ64YEIY2B3PIVOES4AHPPSQ2TUAWA, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.572237227 (507h49m26.443320262s) [noise=true , address=145.79.227.47:4266 , piece_id=SVGPB5X3YLU2UIKUGJS4GUNMLKL6BXZPJRQ3G66T6JZT6MI6BSZQ, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.572921571 (507h49m26.444004616s) [noise=true , address=145.79.227.52:4445 , piece_id=4KJEI3BIXC7EUXNDWDYFGU2NLHPKOJUWI77MCVYZDE6QP3NPLF3A, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.573405502 (507h49m26.444488537s) [noise=true , address=145.79.227.45:4230 , piece_id=7RUBFVPQDHIUQ5HUXGGH2BA3BPQWHGN2SMAANXCP3UO5EVYGVIEQ, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.573428435 (507h49m26.444511481s) [noise=true , address=145.79.227.50:4375 , piece_id=EZPMQA525ESJKSW6M365LF7GPW67RGM3NYWCJE3JQNVCQIRH433Q, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.573887591 (507h49m26.444970636s) [noise=true , address=145.79.227.30:4761 , piece_id=UWOPYRA6WTI75GRCAIKSYDPGZ7YEQPZAQP6ZM7ZIXPXQTDDE6YMA, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.573911566 (507h49m26.444994611s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7SJVXPFSLK6RZZWRTKSIMTKEVSBT2N5ZI57GUCVMOVX4XMH7FIA, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.574211357 (507h49m26.445294392s) [noise=true , address=145.79.227.15:4875 , piece_id=EXHATTOQREEYUXA4ODVDSWQFDSYJUSPTTBT35DQYIARNSLFKDNFA, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.574259498 (507h49m26.445342533s) [noise=true , address=145.79.227.49:4338 , piece_id=BQWZOZ3AOMVB7VOZ3C3TYEGIAFPCLEN64HEVSYV2RFS5EGH5A5PA, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.574587613 (507h49m26.445670648s) [noise=true , address=45.91.116.5:4300 , piece_id=MWVUKWESPIC2T2UM5HAP3OQJQ6NOIOIZK6VCZ5TJHDADO63BCVCA, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.574955584 (507h49m26.446038629s) [noise=true , address=145.79.227.39:4014 , piece_id=JN7CWFCIFS3IUFS7PWMVVMPCK7BOXLXHNVSNNVEFZN6CT3BDVKOA, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.575284771 (507h49m26.446367806s) [noise=true , address=145.79.227.21:4475 , piece_id=5YKRLXVU7XSY3EC5PTQ4R3JLXAMOV2IIKAT65NYTLPJ4NUFVWTYA, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.575534045 (507h49m26.44661708s) [noise=true , address=45.91.116.14:4189 , piece_id=EVWJBUTXLTGWSTY3KMZOIEIYIL6H4KFDYLON26DKINZIPSQ7UA3Q, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.575582898 (507h49m26.446665933s) [noise=true , address=45.91.116.13:4156 , piece_id=PMTLLDWJ3SSDLD6JUW55M6MQ667LCXWJMWQYRDBQGA5ZIBZOTL5Q, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.575953935 (507h49m26.44703697s) [noise=true , address=77.90.31.14:4663 , piece_id=ENBFWB654DDFQFAS7CEYJUWBBCZW64OGGUZDRDCEP5MRNNDE7X3A, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:26.576256882 (507h49m26.447339917s) [noise=true , address=145.79.227.8:4330 , piece_id=GPZGJAV7FGQTP23ITFSSTIZWPC4ZMES6T74QZX452EIEYV4WTG7Q, seg_pos={0 0}, upload=445627]: Uploading piece... Dec 26 19:18:28.285067696 (507h49m28.156150731s) [upload=445627]: Next returned nil segment Dec 26 19:18:28.285079248 (507h49m28.156162283s) [upload=445627]: Waiting for error group managing segments... Dec 26 19:18:28.575725339 (507h49m28.446808374s) [noise=true , address=145.79.227.49:4338 , piece_id=BQWZOZ3AOMVB7VOZ3C3TYEGIAFPCLEN64HEVSYV2RFS5EGH5A5PA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.577689810 (507h49m28.448772845s) [noise=true , address=145.79.227.39:4014 , piece_id=JN7CWFCIFS3IUFS7PWMVVMPCK7BOXLXHNVSNNVEFZN6CT3BDVKOA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.579691752 (507h49m28.450774797s) [noise=true , address=145.79.227.52:4445 , piece_id=4KJEI3BIXC7EUXNDWDYFGU2NLHPKOJUWI77MCVYZDE6QP3NPLF3A, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.590362772 (507h49m28.461445807s) [noise=true , address=145.79.227.47:4266 , piece_id=SVGPB5X3YLU2UIKUGJS4GUNMLKL6BXZPJRQ3G66T6JZT6MI6BSZQ, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.596782185 (507h49m28.46786523s) [noise=true , address=145.79.227.21:4475 , piece_id=5YKRLXVU7XSY3EC5PTQ4R3JLXAMOV2IIKAT65NYTLPJ4NUFVWTYA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.598067502 (507h49m28.469150537s) [noise=true , address=145.79.227.8:4330 , piece_id=GPZGJAV7FGQTP23ITFSSTIZWPC4ZMES6T74QZX452EIEYV4WTG7Q, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.600425994 (507h49m28.471509039s) [noise=true , address=145.79.227.50:4375 , piece_id=EZPMQA525ESJKSW6M365LF7GPW67RGM3NYWCJE3JQNVCQIRH433Q, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.604239915 (507h49m28.47532296s) [noise=true , address=145.79.227.45:4230 , piece_id=7RUBFVPQDHIUQ5HUXGGH2BA3BPQWHGN2SMAANXCP3UO5EVYGVIEQ, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.648156541 (507h49m28.519239576s) [noise=true , address=145.79.227.15:4875 , piece_id=EXHATTOQREEYUXA4ODVDSWQFDSYJUSPTTBT35DQYIARNSLFKDNFA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.850404066 (507h49m28.721487111s) [noise=true , address=77.90.31.14:4663 , piece_id=ENBFWB654DDFQFAS7CEYJUWBBCZW64OGGUZDRDCEP5MRNNDE7X3A, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.915554142 (507h49m28.786637177s) [noise=true , address=45.91.116.14:4189 , piece_id=EVWJBUTXLTGWSTY3KMZOIEIYIL6H4KFDYLON26DKINZIPSQ7UA3Q, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.941022299 (507h49m28.812105334s) [noise=true , address=45.91.116.13:4156 , piece_id=PMTLLDWJ3SSDLD6JUW55M6MQ667LCXWJMWQYRDBQGA5ZIBZOTL5Q, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.955655843 (507h49m28.826738878s) [noise=true , address=45.91.116.5:4300 , piece_id=MWVUKWESPIC2T2UM5HAP3OQJQ6NOIOIZK6VCZ5TJHDADO63BCVCA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil> Dec 26 19:18:28.955679017 (507h49m28.826762063s) [seg_pos={0 0}, upload=445627]: Segment reached optimal threshold of 13 pieces. Dec 26 19:18:28.955910448 (507h49m28.826993504s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7SJVXPFSLK6RZZWRTKSIMTKEVSBT2N5ZI57GUCVMOVX4XMH7FIA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:42040->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:42040->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42040->145.79.227.4:4185: use of closed network connection Dec 26 19:18:28.955918734 (507h49m28.827001779s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7SJVXPFSLK6RZZWRTKSIMTKEVSBT2N5ZI57GUCVMOVX4XMH7FIA, seg_pos={0 0}, upload=445627]: Long tail cancellation due to optimal threshold Dec 26 19:18:28.956092866 (507h49m28.827175901s) [noise=true , address=145.79.227.30:4761 , piece_id=UWOPYRA6WTI75GRCAIKSYDPGZ7YEQPZAQP6ZM7ZIXPXQTDDE6YMA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 19:18:28.956130106 (507h49m28.827213151s) [noise=true , address=145.79.227.30:4761 , piece_id=UWOPYRA6WTI75GRCAIKSYDPGZ7YEQPZAQP6ZM7ZIXPXQTDDE6YMA, seg_pos={0 0}, upload=445627]: Long tail cancellation due to optimal threshold Dec 26 19:18:28.956297095 (507h49m28.82738015s) [noise=true , address=77.90.31.12:4590 , piece_id=VMELCTDYY7FCQQXJH2NPPRAZ64YEIY2B3PIVOES4AHPPSQ2TUAWA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 19:18:28.956307264 (507h49m28.827390319s) [noise=true , address=77.90.31.12:4590 , piece_id=VMELCTDYY7FCQQXJH2NPPRAZ64YEIY2B3PIVOES4AHPPSQ2TUAWA, seg_pos={0 0}, upload=445627]: Long tail cancellation due to optimal threshold Dec 26 19:18:28.956330138 (507h49m28.827413183s) [seg_pos={0 0}, upload=445627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:18:28.956424948 (507h49m28.827507993s) [upload=445627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:18:28.956431601 (507h49m28.827514666s) [upload=445627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:18:29.259717791 (507h49m29.130800836s) [upload=445627]: Done uploading segments... Dec 26 19:18:29.259737017 (507h49m29.130820052s) [upload=445627]: Upload finished. err: <nil> Duration: 30.002396988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...X] duration: 30.003087512s trace: https://jaeger.dev.storj.io/trace/5d7569c1fff810c5 FAIL (2025-12-26 19:17:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:17:22.580110613 (507h48m22.451193648s) [upload=445613]: Starting upload Dec 26 19:17:22.580186407 (507h48m22.451269472s) [upload=445613]: Done starting upload Dec 26 19:17:22.580272612 (507h48m22.451355647s) [upload=445613]: Uploading segments... Dec 26 19:17:22.580470548 (507h48m22.451553593s) [upload=445613]: Got next segment. Inline: false Dec 26 19:17:22.580511406 (507h48m22.451594451s) [upload=445613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:17:22.580517598 (507h48m22.451600644s) [upload=445613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:17:22.831904463 (507h48m22.702987508s) [seg_pos={0 0}, upload=445613]: Begin upload segment... Dec 26 19:17:22.831965889 (507h48m22.703048924s) [seg_pos={0 0}, upload=445613]: Done begin upload segment. Dec 26 19:17:22.832049128 (507h48m22.703132193s) [noise=true , address=145.79.227.17:4972 , piece_id=Y7546SOCADPAHMIHCJWX2QXZHNBQS5KXPDGEM6MHF5SRW3O4EU2Q, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.832068595 (507h48m22.70315164s) [noise=true , address=45.91.116.6:4973 , piece_id=T2QIAYEWQSDMRBVYP6NBJVHFV5X6WHISDULVSVLG6Q5IFEN75CLA, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.832150822 (507h48m22.703233867s) [noise=true , address=77.90.31.16:4736 , piece_id=5HZHYZEDX4QGUH5TTFEJD7WQ5YZFCCCMIWIVUADOTUTPMOA5CSOQ, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.832603193 (507h48m22.703686228s) [noise=true , address=145.79.227.29:4943 , piece_id=OXTFVJQ66JRUDU4TBZRACAS7K3Q57KKDLNVB3RQXETLRMY3J6XMA, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.832979991 (507h48m22.704063026s) [noise=true , address=145.79.227.50:4373 , piece_id=HTZVFRXDIOK2THILPRWPF227OSF5HQLKP3HAVM4BBNIPKX3VZTOQ, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.833328565 (507h48m22.70441161s) [noise=true , address=145.79.227.36:4519 , piece_id=4OTC4ECO3CSN6UPBCDMBHQ6GYUIKRQLIB53RRYUEQUOKDHZYAODA, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.833902036 (507h48m22.704985081s) [noise=true , address=145.79.227.21:4475 , piece_id=IM2NITU6YUO5JISV7DHC5FWL4N7PFCAHZVGTJPLPG2LHIWLN274A, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.834288552 (507h48m22.705371597s) [noise=true , address=45.91.116.5:4300 , piece_id=XBHVJEAOYLMHHES6HYJW7FI2THGPDQQUVRU3INJAXE75FIVXPAEQ, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.835366365 (507h48m22.7064494s) [noise=true , address=45.91.116.7:4010 , piece_id=E2UELKH2MJHIZA4MAUW6CVYEPF5ACTONI3Y4EAPYTOE34QHUDRRQ, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.836070876 (507h48m22.707153921s) [noise=true , address=145.79.227.37:4566 , piece_id=DECHKQLTGNJP6PPTMGVJHVINZU2ZUP3N5NESZBUKMWU4E3N2RRLQ, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.837665983 (507h48m22.708749018s) [noise=true , address=45.91.116.13:4156 , piece_id=4OTRZKFLXZ5QOVE6C5TYAS4X5NP46LAIE2YHBPB52TLQHMC6GNXQ, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.838905163 (507h48m22.709988208s) [noise=true , address=145.79.227.34:4906 , piece_id=K6XDZVFP5WOVIJBHJ2I66MSR63LVIPBBMFJIM4PDM66P7PQS3GEA, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.839975181 (507h48m22.711058216s) [noise=true , address=145.79.227.42:4123 , piece_id=WSG7OFS5ZG4FEXF32IN7TMGTPEE5NYMD7FMI2H25QRF22O7ZIV3Q, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.843199160 (507h48m22.714282205s) [noise=true , address=145.79.227.52:4446 , piece_id=XN5S723EXYOHRCKYMD3RVCJWTCNOZRKPJUXPTTEMSWRIHZ6UWO6A, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.845970467 (507h48m22.717053512s) [noise=true , address=45.91.116.14:4190 , piece_id=KPD22ZZYZYKKN3IFDBMSUZDEUJWFK6PBPLNK7LDBQJNTAGIAWK3Q, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:22.856966315 (507h48m22.72804938s) [noise=true , address=77.90.31.21:4880 , piece_id=EJLIXDTVJVRWNTQSWVF44BUVBKRHJY5GCBMCPNYBLNO7FWJEIV7Q, seg_pos={0 0}, upload=445613]: Uploading piece... Dec 26 19:17:24.451893797 (507h48m24.322976842s) [upload=445613]: Next returned nil segment Dec 26 19:17:24.451906852 (507h48m24.322989887s) [upload=445613]: Waiting for error group managing segments... Dec 26 19:17:24.840151067 (507h48m24.711234112s) [noise=true , address=145.79.227.29:4943 , piece_id=OXTFVJQ66JRUDU4TBZRACAS7K3Q57KKDLNVB3RQXETLRMY3J6XMA, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:24.850410993 (507h48m24.721494038s) [noise=true , address=145.79.227.21:4475 , piece_id=IM2NITU6YUO5JISV7DHC5FWL4N7PFCAHZVGTJPLPG2LHIWLN274A, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:24.857887349 (507h48m24.728970384s) [noise=true , address=145.79.227.36:4519 , piece_id=4OTC4ECO3CSN6UPBCDMBHQ6GYUIKRQLIB53RRYUEQUOKDHZYAODA, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:24.861534053 (507h48m24.732617088s) [noise=true , address=145.79.227.42:4123 , piece_id=WSG7OFS5ZG4FEXF32IN7TMGTPEE5NYMD7FMI2H25QRF22O7ZIV3Q, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:24.863606520 (507h48m24.734689555s) [noise=true , address=145.79.227.50:4373 , piece_id=HTZVFRXDIOK2THILPRWPF227OSF5HQLKP3HAVM4BBNIPKX3VZTOQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:24.866588588 (507h48m24.737671633s) [noise=true , address=145.79.227.52:4446 , piece_id=XN5S723EXYOHRCKYMD3RVCJWTCNOZRKPJUXPTTEMSWRIHZ6UWO6A, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:24.873162546 (507h48m24.744245601s) [noise=true , address=145.79.227.37:4566 , piece_id=DECHKQLTGNJP6PPTMGVJHVINZU2ZUP3N5NESZBUKMWU4E3N2RRLQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:25.115439072 (507h48m24.986522108s) [noise=true , address=77.90.31.16:4736 , piece_id=5HZHYZEDX4QGUH5TTFEJD7WQ5YZFCCCMIWIVUADOTUTPMOA5CSOQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:25.116803822 (507h48m24.987886867s) [noise=true , address=45.91.116.13:4156 , piece_id=4OTRZKFLXZ5QOVE6C5TYAS4X5NP46LAIE2YHBPB52TLQHMC6GNXQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:25.136568887 (507h48m25.007651922s) [noise=true , address=45.91.116.7:4010 , piece_id=E2UELKH2MJHIZA4MAUW6CVYEPF5ACTONI3Y4EAPYTOE34QHUDRRQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:25.178459335 (507h48m25.04954237s) [noise=true , address=77.90.31.21:4880 , piece_id=EJLIXDTVJVRWNTQSWVF44BUVBKRHJY5GCBMCPNYBLNO7FWJEIV7Q, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:25.199009766 (507h48m25.070092801s) [noise=true , address=45.91.116.5:4300 , piece_id=XBHVJEAOYLMHHES6HYJW7FI2THGPDQQUVRU3INJAXE75FIVXPAEQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:25.219002777 (507h48m25.090085852s) [noise=true , address=45.91.116.14:4190 , piece_id=KPD22ZZYZYKKN3IFDBMSUZDEUJWFK6PBPLNK7LDBQJNTAGIAWK3Q, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil> Dec 26 19:17:25.219028695 (507h48m25.09011173s) [seg_pos={0 0}, upload=445613]: Segment reached optimal threshold of 13 pieces. Dec 26 19:17:25.219257652 (507h48m25.090340737s) [noise=true , address=145.79.227.17:4972 , piece_id=Y7546SOCADPAHMIHCJWX2QXZHNBQS5KXPDGEM6MHF5SRW3O4EU2Q, seg_pos={0 0}, upload=445613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 26 19:17:25.219268973 (507h48m25.090352028s) [noise=true , address=145.79.227.17:4972 , piece_id=Y7546SOCADPAHMIHCJWX2QXZHNBQS5KXPDGEM6MHF5SRW3O4EU2Q, seg_pos={0 0}, upload=445613]: Long tail cancellation due to optimal threshold Dec 26 19:17:25.219347252 (507h48m25.090430287s) [noise=true , address=145.79.227.34:4906 , piece_id=K6XDZVFP5WOVIJBHJ2I66MSR63LVIPBBMFJIM4PDM66P7PQS3GEA, seg_pos={0 0}, upload=445613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 19:17:25.219357322 (507h48m25.090440357s) [noise=true , address=145.79.227.34:4906 , piece_id=K6XDZVFP5WOVIJBHJ2I66MSR63LVIPBBMFJIM4PDM66P7PQS3GEA, seg_pos={0 0}, upload=445613]: Long tail cancellation due to optimal threshold Dec 26 19:17:25.219587790 (507h48m25.090670845s) [noise=true , address=45.91.116.6:4973 , piece_id=T2QIAYEWQSDMRBVYP6NBJVHFV5X6WHISDULVSVLG6Q5IFEN75CLA, seg_pos={0 0}, upload=445613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 19:17:25.219611165 (507h48m25.09069421s) [noise=true , address=45.91.116.6:4973 , piece_id=T2QIAYEWQSDMRBVYP6NBJVHFV5X6WHISDULVSVLG6Q5IFEN75CLA, seg_pos={0 0}, upload=445613]: Long tail cancellation due to optimal threshold Dec 26 19:17:25.219634419 (507h48m25.090717454s) [seg_pos={0 0}, upload=445613]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:17:25.219720092 (507h48m25.090803127s) [upload=445613]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:17:25.219727967 (507h48m25.090811002s) [upload=445613]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:17:25.516821405 (507h48m25.38790445s) [upload=445613]: Done uploading segments... Dec 26 19:17:25.516848556 (507h48m25.387931631s) [upload=445613]: Upload finished. err: <nil> Duration: 30.003100208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...] duration: 26.053880201s trace: https://jaeger.dev.storj.io/trace/8cbaebf4bc4ac12 FAIL (2025-12-26 19:17:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q) context deadline exceeded ======LOGS====== (507h45m39.539934276s) [upload=445569]: Done starting upload Dec 26 19:14:39.668953175 (507h45m39.54003622s) [upload=445569]: Uploading segments... Dec 26 19:14:39.669098882 (507h45m39.540181917s) [upload=445569]: Got next segment. Inline: false Dec 26 19:14:39.669131354 (507h45m39.540214389s) [upload=445569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:14:39.669138347 (507h45m39.540221382s) [upload=445569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:14:39.919054782 (507h45m39.790137817s) [seg_pos={0 0}, upload=445569]: Begin upload segment... Dec 26 19:14:39.919108674 (507h45m39.790191709s) [seg_pos={0 0}, upload=445569]: Done begin upload segment. Dec 26 19:14:39.919135595 (507h45m39.79021863s) [noise=true , address=145.79.227.39:4013 , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.919154241 (507h45m39.790237286s) [noise=true , address=145.79.227.8:4330 , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.919204817 (507h45m39.790287852s) [noise=true , address=145.79.227.51:4411 , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.919472367 (507h45m39.790555412s) [noise=true , address=145.79.227.45:4229 , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.919769612 (507h45m39.790852648s) [noise=true , address=145.79.227.36:4518 , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.920059435 (507h45m39.79114247s) [noise=true , address=145.79.227.38:4977 , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.920325922 (507h45m39.791408957s) [noise=true , address=145.79.227.16:4910 , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.920744308 (507h45m39.791827344s) [noise=true , address=77.90.31.21:4878 , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.921114644 (507h45m39.792197689s) [noise=true , address=145.79.227.3:4149 , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.921466324 (507h45m39.792549369s) [noise=true , address=77.90.31.12:4590 , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.921896022 (507h45m39.792979068s) [noise=true , address=145.79.227.14:4839 , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.923066100 (507h45m39.794149145s) [noise=true , address=45.91.116.4:4937 , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.923152836 (507h45m39.794235881s) [noise=true , address=145.79.227.22:4509 , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.924242731 (507h45m39.795325766s) [noise=true , address=145.79.227.50:4374 , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.925507038 (507h45m39.796590083s) [noise=true , address=45.91.116.10:4082 , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:39.927865880 (507h45m39.798948915s) [noise=true , address=45.91.116.14:4191 , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Uploading piece... Dec 26 19:14:41.639443473 (507h45m41.510526508s) [upload=445569]: Next returned nil segment Dec 26 19:14:41.639464642 (507h45m41.510547677s) [upload=445569]: Waiting for error group managing segments... Dec 26 19:14:41.926143951 (507h45m41.797226986s) [noise=true , address=145.79.227.38:4977 , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.939547924 (507h45m41.810630959s) [noise=true , address=145.79.227.36:4518 , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.940037276 (507h45m41.811120321s) [noise=true , address=145.79.227.8:4330 , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.940433921 (507h45m41.811516956s) [noise=true , address=145.79.227.16:4910 , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.941067779 (507h45m41.812150814s) [noise=true , address=145.79.227.45:4229 , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.942830555 (507h45m41.81391359s) [noise=true , address=145.79.227.22:4509 , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.957594908 (507h45m41.828677944s) [noise=true , address=145.79.227.50:4374 , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.958160777 (507h45m41.829243812s) [noise=true , address=145.79.227.3:4149 , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.958603559 (507h45m41.829686604s) [noise=true , address=145.79.227.39:4013 , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:41.963562983 (507h45m41.834646018s) [noise=true , address=145.79.227.51:4411 , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:42.038967997 (507h45m41.910051042s) [noise=true , address=145.79.227.14:4839 , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:42.167135100 (507h45m42.038218135s) [noise=true , address=45.91.116.14:4191 , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:42.205466509 (507h45m42.076549544s) [noise=true , address=77.90.31.21:4878 , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil> Dec 26 19:14:42.205483141 (507h45m42.076566177s) [seg_pos={0 0}, upload=445569]: Segment reached optimal threshold of 13 pieces. Dec 26 19:14:42.205645099 (507h45m42.076728134s) [noise=true , address=77.90.31.12:4590 , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 19:14:42.205650480 (507h45m42.076733515s) [noise=true , address=77.90.31.12:4590 , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold Dec 26 19:14:42.205763836 (507h45m42.076846871s) [noise=true , address=45.91.116.4:4937 , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection Dec 26 19:14:42.205769326 (507h45m42.076852361s) [noise=true , address=45.91.116.4:4937 , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold Dec 26 19:14:42.206093253 (507h45m42.077176298s) [noise=true , address=45.91.116.10:4082 , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:51406->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 26 19:14:42.206110065 (507h45m42.0771931s) [noise=true , address=45.91.116.10:4082 , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold Dec 26 19:14:42.206152786 (507h45m42.077235821s) [seg_pos={0 0}, upload=445569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:14:42.206317099 (507h45m42.077400135s) [upload=445569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:14:42.206322900 (507h45m42.077405935s) [upload=445569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:14:42.505831747 (507h45m42.376914782s) [upload=445569]: Done uploading segments... Dec 26 19:14:42.505853869 (507h45m42.376936905s) [upload=445569]: Upload finished. err: <nil> Duration: 26.053905788s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)