Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [.XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000723378s trace:
https://jaeger.dev.storj.io/trace/32f15219426b3d21
FAIL (2026-01-04 04:08:43 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000753596s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [X.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..XXXXXX.XX] duration:
1m0.002977105s trace:
https://jaeger.dev.storj.io/trace/49fa0590239761c8
FAIL (2026-01-04 04:08:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, Piece ID: LAOS5HYKJR55FEA7OF3CERRA6RXD7O76FQ2FZDNPE7US4SXKXYAA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: UK3YRG4N7EPGTFZSSQDBUSKDX3YI222SBZNQ7MCXD6UVHLE4FMBA) context deadline exceeded; piecestore: (Node ID: 1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, Piece ID: WAP7AXDQ3TNNO5GZ7EC3CSEYEJCY4SF4N7S5PCK6SX3IWZFYS2EQ) context deadline exceeded
======LOGS======
=true , address=145.79.227.33:4898 , piece_id=CYH4HDJOQDXODZHAMD7IAB2ZX7RJDUG4GXJUX52SV7CGJPEKA4CA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.248206752 (708h38m34.119289787s) [noise=true , address=145.79.227.40:4053 , piece_id=F3MYAA5SZRDUU3LZ627TVFMDDDKLX2SAS6KK2QEYCP6HBQJG37LA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.248698648 (708h38m34.119781683s) [noise=true , address=145.79.227.18:4027 , piece_id=SY7FUYBVZXYUDTQHZQGKIARXR53PFRMFTJKTMU6QK3UQQACKIRUA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.248720100 (708h38m34.119803135s) [noise=true , address=77.90.31.12:4616 , piece_id=UK3YRG4N7EPGTFZSSQDBUSKDX3YI222SBZNQ7MCXD6UVHLE4FMBA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.249183221 (708h38m34.120266256s) [noise=true , address=145.79.227.29:4963 , piece_id=M3TC3AAZW7SFDTLJ7NYU6IDX3IZR75GBDRMDLIUPCZMECWPN7CIA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.249212066 (708h38m34.120295111s) [noise=true , address=45.91.116.13:4169 , piece_id=WAP7AXDQ3TNNO5GZ7EC3CSEYEJCY4SF4N7S5PCK6SX3IWZFYS2EQ, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.249294974 (708h38m34.120378009s) [noise=true , address=64.49.13.27:4966 , piece_id=NXUY6FVP43PVGZIJN4GOJQVQVEONXX6N2T6K55DUXB6GQX3FEJFQ, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.249722989 (708h38m34.120806035s) [noise=true , address=45.91.116.5:4310 , piece_id=VTFVKDOATFWFB3MBKKKAVZPRDRHLAEAM3G7TDTHGOUV3GV4QLDTA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.249900578 (708h38m34.120983623s) [noise=true , address=64.49.13.169:4708 , piece_id=DF32GWURGD47KVEPHJK65QH5EHHVZD64VJ6CE3JG33MZK7FAILJQ, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.250412703 (708h38m34.121495748s) [noise=true , address=145.79.227.35:4508 , piece_id=HIVQO5VD2B6NRKFASLA4ZK7IOJADVS2Y5F23N5PTOF3SM5LGP2UA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.250422712 (708h38m34.121505757s) [noise=true , address=45.91.116.9:4278 , piece_id=7QHGRJREZMBHWXXD24SXORPC7HHWIOUKBJUYON3FXJICZPVCBSWA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.250922834 (708h38m34.122005879s) [noise=true , address=145.79.227.40:4055 , piece_id=HNGVZLFALUZW565UM67UA55VARO5TOLZY3EGZQEKKAVUE4SAPWMA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.250952671 (708h38m34.122035736s) [noise=true , address=77.90.31.21:4897 , piece_id=443KTCY3N7NTVGSDPIJ5OMGFCPSHIIURGGNJ3OCJCMWHATBB4QKA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.251449567 (708h38m34.122532602s) [noise=true , address=45.91.116.4:4962 , piece_id=LAOS5HYKJR55FEA7OF3CERRA6RXD7O76FQ2FZDNPE7US4SXKXYAA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.251944450 (708h38m34.123027485s) [noise=true , address=145.79.227.30:4771 , piece_id=4WH4M2Z2ITRY4FWSWTYWCMLQ2Z57LV4JUZ2JU6E5ZUMCLSAWQNHA, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:34.254059597 (708h38m34.125142642s) [noise=true , address=193.149.164.143:4702 , piece_id=UNWGJCMTFD64BCOGEFI2ZJQPT6Z5JBPRQYVMV3EAYG5JTCEK73FQ, seg_pos={0 0}, upload=620407]: Uploading piece...
Jan 4 04:07:35.699515992 (708h38m35.570599037s) [upload=620407]: Next returned nil segment
Jan 4 04:07:35.699529328 (708h38m35.570612363s) [upload=620407]: Waiting for error group managing segments...
Jan 4 04:07:36.078582289 (708h38m35.949665324s) [noise=true , address=64.49.13.27:4966 , piece_id=NXUY6FVP43PVGZIJN4GOJQVQVEONXX6N2T6K55DUXB6GQX3FEJFQ, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:36.102786588 (708h38m35.973869623s) [noise=true , address=145.79.227.18:4027 , piece_id=SY7FUYBVZXYUDTQHZQGKIARXR53PFRMFTJKTMU6QK3UQQACKIRUA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:36.124095364 (708h38m35.995178399s) [noise=true , address=145.79.227.16:4935 , piece_id=3QSPAPCNED5LYUJBME5VXYYRT3WMUJGYVXHFL4ZG5TP65CUX2P7A, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:36.583215987 (708h38m36.454299032s) [noise=true , address=45.91.116.13:4169 , piece_id=WAP7AXDQ3TNNO5GZ7EC3CSEYEJCY4SF4N7S5PCK6SX3IWZFYS2EQ, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.084064809 (708h38m36.955147854s) [noise=true , address=145.79.227.35:4508 , piece_id=HIVQO5VD2B6NRKFASLA4ZK7IOJADVS2Y5F23N5PTOF3SM5LGP2UA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.120340344 (708h38m36.991423389s) [noise=true , address=145.79.227.40:4053 , piece_id=F3MYAA5SZRDUU3LZ627TVFMDDDKLX2SAS6KK2QEYCP6HBQJG37LA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.128728717 (708h38m36.999811762s) [noise=true , address=145.79.227.40:4055 , piece_id=HNGVZLFALUZW565UM67UA55VARO5TOLZY3EGZQEKKAVUE4SAPWMA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.151588576 (708h38m37.022671611s) [noise=true , address=145.79.227.30:4771 , piece_id=4WH4M2Z2ITRY4FWSWTYWCMLQ2Z57LV4JUZ2JU6E5ZUMCLSAWQNHA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.154559483 (708h38m37.025642518s) [noise=true , address=145.79.227.29:4963 , piece_id=M3TC3AAZW7SFDTLJ7NYU6IDX3IZR75GBDRMDLIUPCZMECWPN7CIA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.178961779 (708h38m37.050044814s) [noise=true , address=145.79.227.33:4898 , piece_id=CYH4HDJOQDXODZHAMD7IAB2ZX7RJDUG4GXJUX52SV7CGJPEKA4CA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.217037832 (708h38m37.088120867s) [noise=true , address=145.79.227.44:4196 , piece_id=X5WUBSPGTP3AMM57BO3L5GHMKINFBHICLR3QKJOVWDLRIYSLID3A, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.439390670 (708h38m37.310473715s) [noise=true , address=45.91.116.4:4962 , piece_id=LAOS5HYKJR55FEA7OF3CERRA6RXD7O76FQ2FZDNPE7US4SXKXYAA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:37.608900937 (708h38m37.479983972s) [noise=true , address=77.90.31.12:4616 , piece_id=UK3YRG4N7EPGTFZSSQDBUSKDX3YI222SBZNQ7MCXD6UVHLE4FMBA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:38.079852529 (708h38m37.950935574s) [noise=true , address=193.149.164.143:4702 , piece_id=UNWGJCMTFD64BCOGEFI2ZJQPT6Z5JBPRQYVMV3EAYG5JTCEK73FQ, seg_pos={0 0}, upload=620407]: Done uploading piece. err: <nil>
Jan 4 04:07:38.079877357 (708h38m37.950960402s) [seg_pos={0 0}, upload=620407]: Segment reached optimal threshold of 14 pieces.
Jan 4 04:07:38.080133965 (708h38m37.95121701s) [noise=true , address=45.91.116.9:4278 , piece_id=7QHGRJREZMBHWXXD24SXORPC7HHWIOUKBJUYON3FXJICZPVCBSWA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12htsH8bL33XpUeEstKHb2yeL6TNLusbdRmms9bwzmVCHkej2Xo): protocol: expected piece hash; context canceled; EOF
Jan 4 04:07:38.080142832 (708h38m37.951225877s) [noise=true , address=45.91.116.9:4278 , piece_id=7QHGRJREZMBHWXXD24SXORPC7HHWIOUKBJUYON3FXJICZPVCBSWA, seg_pos={0 0}, upload=620407]: Long tail cancellation due to optimal threshold
Jan 4 04:07:38.080202546 (708h38m37.951285601s) [noise=true , address=77.90.31.21:4897 , piece_id=443KTCY3N7NTVGSDPIJ5OMGFCPSHIIURGGNJ3OCJCMWHATBB4QKA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: expected piece hash; context canceled; EOF
Jan 4 04:07:38.080218816 (708h38m37.951301851s) [noise=true , address=77.90.31.21:4897 , piece_id=443KTCY3N7NTVGSDPIJ5OMGFCPSHIIURGGNJ3OCJCMWHATBB4QKA, seg_pos={0 0}, upload=620407]: Long tail cancellation due to optimal threshold
Jan 4 04:07:38.080295884 (708h38m37.951378929s) [noise=true , address=45.91.116.5:4310 , piece_id=VTFVKDOATFWFB3MBKKKAVZPRDRHLAEAM3G7TDTHGOUV3GV4QLDTA, seg_pos={0 0}, upload=620407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: expected piece hash; context canceled; EOF
Jan 4 04:07:38.080304299 (708h38m37.951387334s) [noise=true , address=45.91.116.5:4310 , piece_id=VTFVKDOATFWFB3MBKKKAVZPRDRHLAEAM3G7TDTHGOUV3GV4QLDTA, seg_pos={0 0}, upload=620407]: Long tail cancellation due to optimal threshold
Jan 4 04:07:38.080556319 (708h38m37.951639374s) [noise=true , address=64.49.13.169:4708 , piece_id=DF32GWURGD47KVEPHJK65QH5EHHVZD64VJ6CE3JG33MZK7FAILJQ, seg_pos={0 0}, upload=620407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:47308->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:47308->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47308->64.49.13.169:4708: use of closed network connection
Jan 4 04:07:38.080565546 (708h38m37.951648591s) [noise=true , address=64.49.13.169:4708 , piece_id=DF32GWURGD47KVEPHJK65QH5EHHVZD64VJ6CE3JG33MZK7FAILJQ, seg_pos={0 0}, upload=620407]: Long tail cancellation due to optimal threshold
Jan 4 04:07:38.080663062 (708h38m37.951746107s) [seg_pos={0 0}, upload=620407]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 04:07:38.080743857 (708h38m37.951826902s) [upload=620407]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:07:38.080749457 (708h38m37.951832502s) [upload=620407]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:07:38.370038161 (708h38m38.241121206s) [upload=620407]: Done uploading segments...
Jan 4 04:07:38.370060934 (708h38m38.241143969s) [upload=620407]: Upload finished. err: <nil>
Duration: 1m0.003001482s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..XX..XXX.] duration:
7.127952906s trace:
https://jaeger.dev.storj.io/trace/61cba965454e0d81
FAIL (2026-01-04 04:07:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2K5JEP5QWL5DH6D4O4LL23BOA3AXJKZVCNFBWUNXQUA2JWFC2NIA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 7AWYBAD2H7BC66E4PFYUW2ZYCLB3PDQ2HB5GK56ZUSPZEE4AI67A) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: ESJ74PGR6QPE55FZ766CRIOMXIJGMMS4KV6DNZXXE6VW2S5QWDTA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: LAOD6ESCYAJMJMJSKLFBTBMBD66W5HEYYRS5US4U7JXBZMBDU2KQ) context deadline exceeded
======LOGS======
Jan 4 04:06:38.331893281 (708h37m38.202976326s) [upload=620390]: Starting upload
Jan 4 04:06:38.331979856 (708h37m38.203062891s) [upload=620390]: Done starting upload
Jan 4 04:06:38.332071790 (708h37m38.203154825s) [upload=620390]: Uploading segments...
Jan 4 04:06:38.332220244 (708h37m38.203303279s) [upload=620390]: Got next segment. Inline: false
Jan 4 04:06:38.332253627 (708h37m38.203336662s) [upload=620390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 04:06:38.332260079 (708h37m38.203343125s) [upload=620390]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 04:06:38.578582479 (708h37m38.449665514s) [seg_pos={0 0}, upload=620390]: Begin upload segment...
Jan 4 04:06:38.578638366 (708h37m38.449721401s) [seg_pos={0 0}, upload=620390]: Done begin upload segment.
Jan 4 04:06:38.578670537 (708h37m38.449753572s) [noise=true , address=145.79.227.4:4186 , piece_id=KO4LQG5Q73DZAETGWFVMLFUJITT4DWNBBIKILOPDTOHPGPPMOUPQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.578834680 (708h37m38.449917725s) [noise=true , address=77.90.31.17:4773 , piece_id=7AWYBAD2H7BC66E4PFYUW2ZYCLB3PDQ2HB5GK56ZUSPZEE4AI67A, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.579227888 (708h37m38.450310923s) [noise=true , address=77.90.31.11:4553 , piece_id=ESJ74PGR6QPE55FZ766CRIOMXIJGMMS4KV6DNZXXE6VW2S5QWDTA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.579600087 (708h37m38.450683132s) [noise=true , address=145.79.227.11:4731 , piece_id=XX5IWNR6ISEKE3HSDZ2FIGN7C3GOY4Q2XQGDFQGUZMCAZ5RVCX5Q, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.579898695 (708h37m38.45098174s) [noise=true , address=45.91.116.4:4940 , piece_id=D3NQTQR6FNH6T7MFFJ4YZ7UGZE3HJTRGYPESMH74CMQM7XTNKPRA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.580226330 (708h37m38.451309365s) [noise=true , address=145.79.227.10:4402 , piece_id=7L4VALKQHX4QMO4N4IOIAX5R2XAH6NCK4EVO7PJKTST4AOU2L5QQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.580483429 (708h37m38.451566464s) [noise=true , address=45.91.116.13:4156 , piece_id=LAOD6ESCYAJMJMJSKLFBTBMBD66W5HEYYRS5US4U7JXBZMBDU2KQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.580897177 (708h37m38.451980252s) [noise=true , address=145.79.227.9:4366 , piece_id=VVXVZLQADWVYG52LLKMOXEAVBOKQUMUU7JUHYAZLKSXC3CCI4P2Q, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.580993090 (708h37m38.452076125s) [noise=true , address=77.90.31.19:4844 , piece_id=2K5JEP5QWL5DH6D4O4LL23BOA3AXJKZVCNFBWUNXQUA2JWFC2NIA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.581289814 (708h37m38.452372859s) [noise=true , address=45.91.116.12:4225 , piece_id=GTKRAAQQQCXY3YRH65S43VXPSADNUQIT4R2WKQ4X5AF4FF26FLNA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.581422497 (708h37m38.452505542s) [noise=true , address=145.79.227.42:4121 , piece_id=2P2JZD5246VVMAQFYR376RTKDF4SVV5XR4RBILAOLGB7Z5RRBLCQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.581630493 (708h37m38.452713538s) [noise=true , address=145.79.227.6:4258 , piece_id=PJLBFA2PSE3AZNX65YW7HMLCW6ZXUZN6GJLSGZ7OWFGIUBKEKFLA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.581885749 (708h37m38.452968784s) [noise=true , address=45.91.116.9:4261 , piece_id=3EQZB4GNJPQSLZ6GBAYLFJCUXVW6RE5SNT2PN7G4Z6R7NEP7LDNA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.582349853 (708h37m38.453432888s) [noise=true , address=45.91.116.5:4298 , piece_id=3732EERYGHS3KRIOUERJ3EPKSTR5MYFBQ3M2J43EUPR63D2YTGUA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.582694189 (708h37m38.453777224s) [noise=true , address=145.79.227.35:4483 , piece_id=LG5PBWCTN3D34PBKBASSQOF6EN6UTA3FVCGWZT4PCLK7MZJZXFTA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.583041019 (708h37m38.454124054s) [noise=true , address=145.79.227.3:4151 , piece_id=HRNJWXQNVL27YGOJEE3FCZ7TESCMIUHO6JZTES373WO5LUCNRSHQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:40.319477881 (708h37m40.190560926s) [upload=620390]: Next returned nil segment
Jan 4 04:06:40.319497107 (708h37m40.190580152s) [upload=620390]: Waiting for error group managing segments...
Jan 4 04:06:40.567603805 (708h37m40.43868684s) [noise=true , address=145.79.227.10:4402 , piece_id=7L4VALKQHX4QMO4N4IOIAX5R2XAH6NCK4EVO7PJKTST4AOU2L5QQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.570539815 (708h37m40.44162286s) [noise=true , address=145.79.227.6:4258 , piece_id=PJLBFA2PSE3AZNX65YW7HMLCW6ZXUZN6GJLSGZ7OWFGIUBKEKFLA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.574245372 (708h37m40.445328407s) [noise=true , address=145.79.227.11:4731 , piece_id=XX5IWNR6ISEKE3HSDZ2FIGN7C3GOY4Q2XQGDFQGUZMCAZ5RVCX5Q, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.589538542 (708h37m40.460621587s) [noise=true , address=145.79.227.9:4366 , piece_id=VVXVZLQADWVYG52LLKMOXEAVBOKQUMUU7JUHYAZLKSXC3CCI4P2Q, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.590532474 (708h37m40.461615509s) [noise=true , address=145.79.227.4:4186 , piece_id=KO4LQG5Q73DZAETGWFVMLFUJITT4DWNBBIKILOPDTOHPGPPMOUPQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.590616154 (708h37m40.461699189s) [noise=true , address=145.79.227.42:4121 , piece_id=2P2JZD5246VVMAQFYR376RTKDF4SVV5XR4RBILAOLGB7Z5RRBLCQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.601876625 (708h37m40.472959671s) [noise=true , address=145.79.227.3:4151 , piece_id=HRNJWXQNVL27YGOJEE3FCZ7TESCMIUHO6JZTES373WO5LUCNRSHQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.900091178 (708h37m40.771174223s) [noise=true , address=77.90.31.17:4773 , piece_id=7AWYBAD2H7BC66E4PFYUW2ZYCLB3PDQ2HB5GK56ZUSPZEE4AI67A, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:41.067273751 (708h37m40.938356796s) [noise=true , address=45.91.116.5:4298 , piece_id=3732EERYGHS3KRIOUERJ3EPKSTR5MYFBQ3M2J43EUPR63D2YTGUA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:41.816086248 (708h37m41.687169283s) [noise=true , address=77.90.31.19:4844 , piece_id=2K5JEP5QWL5DH6D4O4LL23BOA3AXJKZVCNFBWUNXQUA2JWFC2NIA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:41.912124381 (708h37m41.783207416s) [noise=true , address=145.79.227.35:4483 , piece_id=LG5PBWCTN3D34PBKBASSQOF6EN6UTA3FVCGWZT4PCLK7MZJZXFTA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:42.070118378 (708h37m41.941201423s) [noise=true , address=45.91.116.13:4156 , piece_id=LAOD6ESCYAJMJMJSKLFBTBMBD66W5HEYYRS5US4U7JXBZMBDU2KQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:42.416882507 (708h37m42.287965562s) [noise=true , address=77.90.31.11:4553 , piece_id=ESJ74PGR6QPE55FZ766CRIOMXIJGMMS4KV6DNZXXE6VW2S5QWDTA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:42.416906131 (708h37m42.287989176s) [seg_pos={0 0}, upload=620390]: Segment reached optimal threshold of 13 pieces.
Jan 4 04:06:42.417091425 (708h37m42.28817447s) [noise=true , address=45.91.116.12:4225 , piece_id=GTKRAAQQQCXY3YRH65S43VXPSADNUQIT4R2WKQ4X5AF4FF26FLNA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Jan 4 04:06:42.417100261 (708h37m42.288183296s) [noise=true , address=45.91.116.12:4225 , piece_id=GTKRAAQQQCXY3YRH65S43VXPSADNUQIT4R2WKQ4X5AF4FF26FLNA, seg_pos={0 0}, upload=620390]: Long tail cancellation due to optimal threshold
Jan 4 04:06:42.417272649 (708h37m42.288355695s) [noise=true , address=45.91.116.9:4261 , piece_id=3EQZB4GNJPQSLZ6GBAYLFJCUXVW6RE5SNT2PN7G4Z6R7NEP7LDNA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Jan 4 04:06:42.417279292 (708h37m42.288362337s) [noise=true , address=45.91.116.9:4261 , piece_id=3EQZB4GNJPQSLZ6GBAYLFJCUXVW6RE5SNT2PN7G4Z6R7NEP7LDNA, seg_pos={0 0}, upload=620390]: Long tail cancellation due to optimal threshold
Jan 4 04:06:42.417321512 (708h37m42.288404547s) [noise=true , address=45.91.116.4:4940 , piece_id=D3NQTQR6FNH6T7MFFJ4YZ7UGZE3HJTRGYPESMH74CMQM7XTNKPRA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:51472->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:06:42.417329497 (708h37m42.288412542s) [noise=true , address=45.91.116.4:4940 , piece_id=D3NQTQR6FNH6T7MFFJ4YZ7UGZE3HJTRGYPESMH74CMQM7XTNKPRA, seg_pos={0 0}, upload=620390]: Long tail cancellation due to optimal threshold
Jan 4 04:06:42.417351860 (708h37m42.288434895s) [seg_pos={0 0}, upload=620390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 04:06:42.417432724 (708h37m42.288515759s) [upload=620390]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:06:42.417452361 (708h37m42.288535397s) [upload=620390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:06:42.718542418 (708h37m42.589625463s) [upload=620390]: Done uploading segments...
Jan 4 04:06:42.718559781 (708h37m42.589642816s) [upload=620390]: Upload finished. err: <nil>
Duration: 7.127984096s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001706178s trace:
https://jaeger.dev.storj.io/trace/1650796e7655ef75
FAIL (2026-01-04 04:07:44 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001728572s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [XX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..XXXXXX.X] duration:
43.033016203s trace:
https://jaeger.dev.storj.io/trace/d60728a44d1176
FAIL (2026-01-04 04:07:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
[noise=true , address=145.79.227.40:4082 , piece_id=ECKVKGJLUEXIHWQNV3S4R6UDQBG5E7VPYBJCCIB2D7QY3SJMYXMQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.334688475 (708h37m40.20577152s) [noise=true , address=145.79.227.2:4134 , piece_id=KCHENWCZODHPYPWCEK4GM4BC3AVN7XM3WXTVNJTWJ53VCXO2XTWQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.334994527 (708h37m40.206077572s) [noise=true , address=145.79.227.34:4922 , piece_id=YFG35YX6AGDYZAPJO7S6Y6QOVGZL5HRLCBHYAOBAELEGH7SXUM7Q, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.335332861 (708h37m40.206415906s) [noise=true , address=145.79.227.49:4365 , piece_id=WT7OX6YQOU7ZOK7WSGAXZS3J7PDRLHLCBVDOIXDK6PE5CD4NFPRA, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.335738824 (708h37m40.206821869s) [noise=true , address=145.79.227.44:4210 , piece_id=D7CQLH36ACEMU7AA7DQSORYHWZVXXAUJNX3ANUFBKIQTINQ2LUHQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.336110783 (708h37m40.207193828s) [noise=true , address=145.79.227.33:4876 , piece_id=GRHSRNQQLF64DGAAM3SO5OAWFKDEVDKPDAMQ44XOMYYWJRU23CDQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.336509762 (708h37m40.207592797s) [noise=true , address=145.79.227.17:4993 , piece_id=5PCSLWCQK7CQWZJ7KDM4SZSKVDYM5L44NBZLLGEVLUO6Z6AQY3JQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.336908752 (708h37m40.207991797s) [noise=true , address=64.49.13.138:4481 , piece_id=LTNCEOXTKZ2P4Q3AB7QZE7ECZAOZZVLGVENKX6CBONJ6PBWFZTUA, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.337928193 (708h37m40.209011229s) [noise=true , address=145.79.227.8:4355 , piece_id=36VGVJLPJWFMDPFSDFFCMS55O34IECLUBWLGI7HEXAJP2GT7R5AQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.338667471 (708h37m40.209750546s) [noise=true , address=64.49.13.155:4916 , piece_id=NFWGXVQ44T2TAUOBY3H6W55SSN6RTOKAJGSXYJPKPUEOGGWDNHZQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.339299995 (708h37m40.21038305s) [noise=true , address=45.91.116.8:4057 , piece_id=VDNHICBOFUAVZXEYW4HDI6Q7TEOS227ZFMBV2D2ED6D7LTUWJHGA, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.339891441 (708h37m40.210974476s) [noise=true , address=193.149.164.143:4696 , piece_id=JTHZWVJ5XR6MSB2IFBDWCSUD6XUNCN6J6WFHLPRA7B3KE53DQINA, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.340634967 (708h37m40.211718012s) [noise=true , address=145.79.227.43:4172 , piece_id=M4Z55KL2PNYQOXXRVEPCDDTQDAFY4EFAHNJZFHQEXBQ5J3RIMKSQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.341441534 (708h37m40.212524579s) [noise=true , address=145.79.227.30:4779 , piece_id=BUJ2NLDOEKVLZCZLBOZWARGVCYLD7OYUL26XC2J2SZUJQUTOWVGA, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.342291221 (708h37m40.213374266s) [noise=true , address=145.79.227.18:4027 , piece_id=JAFLTGD3P3SMRFCGDBVT52YXLSTYXBKWCNGCKDITDXARJ6ZYSLKQ, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:40.343113157 (708h37m40.214196192s) [noise=true , address=77.90.31.11:4588 , piece_id=RABF7ESQEP3DIJDBFI3FCJ6NTYK5TXSY4K3QBE3UBXNREHYMCHZA, seg_pos={0 0}, upload=620391]: Uploading piece...
Jan 4 04:06:41.762998379 (708h37m41.634081415s) [upload=620391]: Next returned nil segment
Jan 4 04:06:41.763014760 (708h37m41.634097795s) [upload=620391]: Waiting for error group managing segments...
Jan 4 04:06:42.136253590 (708h37m42.007336635s) [noise=true , address=64.49.13.138:4481 , piece_id=LTNCEOXTKZ2P4Q3AB7QZE7ECZAOZZVLGVENKX6CBONJ6PBWFZTUA, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:42.195158256 (708h37m42.066241291s) [noise=true , address=145.79.227.18:4027 , piece_id=JAFLTGD3P3SMRFCGDBVT52YXLSTYXBKWCNGCKDITDXARJ6ZYSLKQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:42.199961612 (708h37m42.071044647s) [noise=true , address=145.79.227.17:4993 , piece_id=5PCSLWCQK7CQWZJ7KDM4SZSKVDYM5L44NBZLLGEVLUO6Z6AQY3JQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:42.279224401 (708h37m42.150307446s) [noise=true , address=145.79.227.2:4134 , piece_id=KCHENWCZODHPYPWCEK4GM4BC3AVN7XM3WXTVNJTWJ53VCXO2XTWQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:42.362210852 (708h37m42.233293887s) [noise=true , address=145.79.227.8:4355 , piece_id=36VGVJLPJWFMDPFSDFFCMS55O34IECLUBWLGI7HEXAJP2GT7R5AQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:42.594674262 (708h37m42.465757297s) [noise=true , address=77.90.31.11:4588 , piece_id=RABF7ESQEP3DIJDBFI3FCJ6NTYK5TXSY4K3QBE3UBXNREHYMCHZA, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:42.696618622 (708h37m42.567701667s) [noise=true , address=77.90.31.17:4781 , piece_id=SHKXBNXEY3DUA3ZHPR6MGZ76XMMEDVBATZOPQX3N4TIVV5ODVUNQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:43.042276724 (708h37m42.913359759s) [noise=true , address=145.79.227.49:4365 , piece_id=WT7OX6YQOU7ZOK7WSGAXZS3J7PDRLHLCBVDOIXDK6PE5CD4NFPRA, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:43.062686096 (708h37m42.933769141s) [noise=true , address=145.79.227.44:4210 , piece_id=D7CQLH36ACEMU7AA7DQSORYHWZVXXAUJNX3ANUFBKIQTINQ2LUHQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:43.087070859 (708h37m42.958153894s) [noise=true , address=145.79.227.43:4172 , piece_id=M4Z55KL2PNYQOXXRVEPCDDTQDAFY4EFAHNJZFHQEXBQ5J3RIMKSQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:43.105461438 (708h37m42.976544483s) [noise=true , address=45.91.116.8:4057 , piece_id=VDNHICBOFUAVZXEYW4HDI6Q7TEOS227ZFMBV2D2ED6D7LTUWJHGA, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:43.139341411 (708h37m43.010424456s) [noise=true , address=145.79.227.33:4876 , piece_id=GRHSRNQQLF64DGAAM3SO5OAWFKDEVDKPDAMQ44XOMYYWJRU23CDQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:43.156894615 (708h37m43.02797766s) [noise=true , address=145.79.227.40:4082 , piece_id=ECKVKGJLUEXIHWQNV3S4R6UDQBG5E7VPYBJCCIB2D7QY3SJMYXMQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:43.168243335 (708h37m43.03932637s) [noise=true , address=64.49.13.155:4916 , piece_id=NFWGXVQ44T2TAUOBY3H6W55SSN6RTOKAJGSXYJPKPUEOGGWDNHZQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: <nil>
Jan 4 04:06:43.168272150 (708h37m43.039355195s) [seg_pos={0 0}, upload=620391]: Segment reached optimal threshold of 14 pieces.
Jan 4 04:06:43.168604873 (708h37m43.039687928s) [noise=true , address=64.49.13.169:4706 , piece_id=4NHO4VVOWSVVT6NDE4MBWUBEOXGG3SNA3YVU6SNRC2G5FKQXXWCQ, seg_pos={0 0}, upload=620391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:48768->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:48768->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48768->64.49.13.169:4706: use of closed network connection
Jan 4 04:06:43.168616025 (708h37m43.03969906s) [noise=true , address=64.49.13.169:4706 , piece_id=4NHO4VVOWSVVT6NDE4MBWUBEOXGG3SNA3YVU6SNRC2G5FKQXXWCQ, seg_pos={0 0}, upload=620391]: Long tail cancellation due to optimal threshold
Jan 4 04:06:43.168629140 (708h37m43.039712185s) [noise=true , address=193.149.164.143:4696 , piece_id=JTHZWVJ5XR6MSB2IFBDWCSUD6XUNCN6J6WFHLPRA7B3KE53DQINA, seg_pos={0 0}, upload=620391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF
Jan 4 04:06:43.168637986 (708h37m43.039721022s) [noise=true , address=193.149.164.143:4696 , piece_id=JTHZWVJ5XR6MSB2IFBDWCSUD6XUNCN6J6WFHLPRA7B3KE53DQINA, seg_pos={0 0}, upload=620391]: Long tail cancellation due to optimal threshold
Jan 4 04:06:43.168850721 (708h37m43.039933766s) [noise=true , address=145.79.227.34:4922 , piece_id=YFG35YX6AGDYZAPJO7S6Y6QOVGZL5HRLCBHYAOBAELEGH7SXUM7Q, seg_pos={0 0}, upload=620391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF
Jan 4 04:06:43.168859359 (708h37m43.039942404s) [noise=true , address=145.79.227.34:4922 , piece_id=YFG35YX6AGDYZAPJO7S6Y6QOVGZL5HRLCBHYAOBAELEGH7SXUM7Q, seg_pos={0 0}, upload=620391]: Long tail cancellation due to optimal threshold
Jan 4 04:06:43.168911978 (708h37m43.039995023s) [noise=true , address=145.79.227.30:4779 , piece_id=BUJ2NLDOEKVLZCZLBOZWARGVCYLD7OYUL26XC2J2SZUJQUTOWVGA, seg_pos={0 0}, upload=620391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eRovJAp3AyUHP4qkuDvSSGYb9TG6U4tSNzniGJBeYLhkmiUVy): protocol: expected piece hash; context canceled; EOF
Jan 4 04:06:43.168928350 (708h37m43.040011385s) [noise=true , address=145.79.227.30:4779 , piece_id=BUJ2NLDOEKVLZCZLBOZWARGVCYLD7OYUL26XC2J2SZUJQUTOWVGA, seg_pos={0 0}, upload=620391]: Long tail cancellation due to optimal threshold
Jan 4 04:06:43.169009284 (708h37m43.040092329s) [seg_pos={0 0}, upload=620391]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 04:06:43.169122940 (708h37m43.040205985s) [upload=620391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:06:43.169129522 (708h37m43.040212567s) [upload=620391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:06:43.464213273 (708h37m43.335296308s) [upload=620391]: Done uploading segments...
Jan 4 04:06:43.464232609 (708h37m43.335315654s) [upload=620391]: Upload finished. err: <nil>
Duration: 43.033070997s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001249612s trace:
https://jaeger.dev.storj.io/trace/ef5de2d3b3bca36
FAIL (2026-01-04 04:07:08 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001277354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [X....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..XX..XXX] duration:
30.036643748s trace:
https://jaeger.dev.storj.io/trace/61cba965454e0d81
FAIL (2026-01-04 04:07:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2K5JEP5QWL5DH6D4O4LL23BOA3AXJKZVCNFBWUNXQUA2JWFC2NIA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 7AWYBAD2H7BC66E4PFYUW2ZYCLB3PDQ2HB5GK56ZUSPZEE4AI67A) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: ESJ74PGR6QPE55FZ766CRIOMXIJGMMS4KV6DNZXXE6VW2S5QWDTA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: LAOD6ESCYAJMJMJSKLFBTBMBD66W5HEYYRS5US4U7JXBZMBDU2KQ) context deadline exceeded
======LOGS======
Jan 4 04:06:38.331893281 (708h37m38.202976326s) [upload=620390]: Starting upload
Jan 4 04:06:38.331979856 (708h37m38.203062891s) [upload=620390]: Done starting upload
Jan 4 04:06:38.332071790 (708h37m38.203154825s) [upload=620390]: Uploading segments...
Jan 4 04:06:38.332220244 (708h37m38.203303279s) [upload=620390]: Got next segment. Inline: false
Jan 4 04:06:38.332253627 (708h37m38.203336662s) [upload=620390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 04:06:38.332260079 (708h37m38.203343125s) [upload=620390]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 04:06:38.578582479 (708h37m38.449665514s) [seg_pos={0 0}, upload=620390]: Begin upload segment...
Jan 4 04:06:38.578638366 (708h37m38.449721401s) [seg_pos={0 0}, upload=620390]: Done begin upload segment.
Jan 4 04:06:38.578670537 (708h37m38.449753572s) [noise=true , address=145.79.227.4:4186 , piece_id=KO4LQG5Q73DZAETGWFVMLFUJITT4DWNBBIKILOPDTOHPGPPMOUPQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.578834680 (708h37m38.449917725s) [noise=true , address=77.90.31.17:4773 , piece_id=7AWYBAD2H7BC66E4PFYUW2ZYCLB3PDQ2HB5GK56ZUSPZEE4AI67A, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.579227888 (708h37m38.450310923s) [noise=true , address=77.90.31.11:4553 , piece_id=ESJ74PGR6QPE55FZ766CRIOMXIJGMMS4KV6DNZXXE6VW2S5QWDTA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.579600087 (708h37m38.450683132s) [noise=true , address=145.79.227.11:4731 , piece_id=XX5IWNR6ISEKE3HSDZ2FIGN7C3GOY4Q2XQGDFQGUZMCAZ5RVCX5Q, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.579898695 (708h37m38.45098174s) [noise=true , address=45.91.116.4:4940 , piece_id=D3NQTQR6FNH6T7MFFJ4YZ7UGZE3HJTRGYPESMH74CMQM7XTNKPRA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.580226330 (708h37m38.451309365s) [noise=true , address=145.79.227.10:4402 , piece_id=7L4VALKQHX4QMO4N4IOIAX5R2XAH6NCK4EVO7PJKTST4AOU2L5QQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.580483429 (708h37m38.451566464s) [noise=true , address=45.91.116.13:4156 , piece_id=LAOD6ESCYAJMJMJSKLFBTBMBD66W5HEYYRS5US4U7JXBZMBDU2KQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.580897177 (708h37m38.451980252s) [noise=true , address=145.79.227.9:4366 , piece_id=VVXVZLQADWVYG52LLKMOXEAVBOKQUMUU7JUHYAZLKSXC3CCI4P2Q, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.580993090 (708h37m38.452076125s) [noise=true , address=77.90.31.19:4844 , piece_id=2K5JEP5QWL5DH6D4O4LL23BOA3AXJKZVCNFBWUNXQUA2JWFC2NIA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.581289814 (708h37m38.452372859s) [noise=true , address=45.91.116.12:4225 , piece_id=GTKRAAQQQCXY3YRH65S43VXPSADNUQIT4R2WKQ4X5AF4FF26FLNA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.581422497 (708h37m38.452505542s) [noise=true , address=145.79.227.42:4121 , piece_id=2P2JZD5246VVMAQFYR376RTKDF4SVV5XR4RBILAOLGB7Z5RRBLCQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.581630493 (708h37m38.452713538s) [noise=true , address=145.79.227.6:4258 , piece_id=PJLBFA2PSE3AZNX65YW7HMLCW6ZXUZN6GJLSGZ7OWFGIUBKEKFLA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.581885749 (708h37m38.452968784s) [noise=true , address=45.91.116.9:4261 , piece_id=3EQZB4GNJPQSLZ6GBAYLFJCUXVW6RE5SNT2PN7G4Z6R7NEP7LDNA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.582349853 (708h37m38.453432888s) [noise=true , address=45.91.116.5:4298 , piece_id=3732EERYGHS3KRIOUERJ3EPKSTR5MYFBQ3M2J43EUPR63D2YTGUA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.582694189 (708h37m38.453777224s) [noise=true , address=145.79.227.35:4483 , piece_id=LG5PBWCTN3D34PBKBASSQOF6EN6UTA3FVCGWZT4PCLK7MZJZXFTA, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:38.583041019 (708h37m38.454124054s) [noise=true , address=145.79.227.3:4151 , piece_id=HRNJWXQNVL27YGOJEE3FCZ7TESCMIUHO6JZTES373WO5LUCNRSHQ, seg_pos={0 0}, upload=620390]: Uploading piece...
Jan 4 04:06:40.319477881 (708h37m40.190560926s) [upload=620390]: Next returned nil segment
Jan 4 04:06:40.319497107 (708h37m40.190580152s) [upload=620390]: Waiting for error group managing segments...
Jan 4 04:06:40.567603805 (708h37m40.43868684s) [noise=true , address=145.79.227.10:4402 , piece_id=7L4VALKQHX4QMO4N4IOIAX5R2XAH6NCK4EVO7PJKTST4AOU2L5QQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.570539815 (708h37m40.44162286s) [noise=true , address=145.79.227.6:4258 , piece_id=PJLBFA2PSE3AZNX65YW7HMLCW6ZXUZN6GJLSGZ7OWFGIUBKEKFLA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.574245372 (708h37m40.445328407s) [noise=true , address=145.79.227.11:4731 , piece_id=XX5IWNR6ISEKE3HSDZ2FIGN7C3GOY4Q2XQGDFQGUZMCAZ5RVCX5Q, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.589538542 (708h37m40.460621587s) [noise=true , address=145.79.227.9:4366 , piece_id=VVXVZLQADWVYG52LLKMOXEAVBOKQUMUU7JUHYAZLKSXC3CCI4P2Q, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.590532474 (708h37m40.461615509s) [noise=true , address=145.79.227.4:4186 , piece_id=KO4LQG5Q73DZAETGWFVMLFUJITT4DWNBBIKILOPDTOHPGPPMOUPQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.590616154 (708h37m40.461699189s) [noise=true , address=145.79.227.42:4121 , piece_id=2P2JZD5246VVMAQFYR376RTKDF4SVV5XR4RBILAOLGB7Z5RRBLCQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.601876625 (708h37m40.472959671s) [noise=true , address=145.79.227.3:4151 , piece_id=HRNJWXQNVL27YGOJEE3FCZ7TESCMIUHO6JZTES373WO5LUCNRSHQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:40.900091178 (708h37m40.771174223s) [noise=true , address=77.90.31.17:4773 , piece_id=7AWYBAD2H7BC66E4PFYUW2ZYCLB3PDQ2HB5GK56ZUSPZEE4AI67A, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:41.067273751 (708h37m40.938356796s) [noise=true , address=45.91.116.5:4298 , piece_id=3732EERYGHS3KRIOUERJ3EPKSTR5MYFBQ3M2J43EUPR63D2YTGUA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:41.816086248 (708h37m41.687169283s) [noise=true , address=77.90.31.19:4844 , piece_id=2K5JEP5QWL5DH6D4O4LL23BOA3AXJKZVCNFBWUNXQUA2JWFC2NIA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:41.912124381 (708h37m41.783207416s) [noise=true , address=145.79.227.35:4483 , piece_id=LG5PBWCTN3D34PBKBASSQOF6EN6UTA3FVCGWZT4PCLK7MZJZXFTA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:42.070118378 (708h37m41.941201423s) [noise=true , address=45.91.116.13:4156 , piece_id=LAOD6ESCYAJMJMJSKLFBTBMBD66W5HEYYRS5US4U7JXBZMBDU2KQ, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:42.416882507 (708h37m42.287965562s) [noise=true , address=77.90.31.11:4553 , piece_id=ESJ74PGR6QPE55FZ766CRIOMXIJGMMS4KV6DNZXXE6VW2S5QWDTA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: <nil>
Jan 4 04:06:42.416906131 (708h37m42.287989176s) [seg_pos={0 0}, upload=620390]: Segment reached optimal threshold of 13 pieces.
Jan 4 04:06:42.417091425 (708h37m42.28817447s) [noise=true , address=45.91.116.12:4225 , piece_id=GTKRAAQQQCXY3YRH65S43VXPSADNUQIT4R2WKQ4X5AF4FF26FLNA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Jan 4 04:06:42.417100261 (708h37m42.288183296s) [noise=true , address=45.91.116.12:4225 , piece_id=GTKRAAQQQCXY3YRH65S43VXPSADNUQIT4R2WKQ4X5AF4FF26FLNA, seg_pos={0 0}, upload=620390]: Long tail cancellation due to optimal threshold
Jan 4 04:06:42.417272649 (708h37m42.288355695s) [noise=true , address=45.91.116.9:4261 , piece_id=3EQZB4GNJPQSLZ6GBAYLFJCUXVW6RE5SNT2PN7G4Z6R7NEP7LDNA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Jan 4 04:06:42.417279292 (708h37m42.288362337s) [noise=true , address=45.91.116.9:4261 , piece_id=3EQZB4GNJPQSLZ6GBAYLFJCUXVW6RE5SNT2PN7G4Z6R7NEP7LDNA, seg_pos={0 0}, upload=620390]: Long tail cancellation due to optimal threshold
Jan 4 04:06:42.417321512 (708h37m42.288404547s) [noise=true , address=45.91.116.4:4940 , piece_id=D3NQTQR6FNH6T7MFFJ4YZ7UGZE3HJTRGYPESMH74CMQM7XTNKPRA, seg_pos={0 0}, upload=620390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:51472->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:06:42.417329497 (708h37m42.288412542s) [noise=true , address=45.91.116.4:4940 , piece_id=D3NQTQR6FNH6T7MFFJ4YZ7UGZE3HJTRGYPESMH74CMQM7XTNKPRA, seg_pos={0 0}, upload=620390]: Long tail cancellation due to optimal threshold
Jan 4 04:06:42.417351860 (708h37m42.288434895s) [seg_pos={0 0}, upload=620390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 04:06:42.417432724 (708h37m42.288515759s) [upload=620390]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:06:42.417452361 (708h37m42.288535397s) [upload=620390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:06:42.718542418 (708h37m42.589625463s) [upload=620390]: Done uploading segments...
Jan 4 04:06:42.718559781 (708h37m42.589642816s) [upload=620390]: Upload finished. err: <nil>
Duration: 30.03667115s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.002040217s trace:
https://jaeger.dev.storj.io/trace/45aa74398e738015
FAIL (2026-01-04 04:06:51 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002053762s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..XX..XX] duration:
30.002830087s trace:
https://jaeger.dev.storj.io/trace/4f365345b4d1fcdf
FAIL (2026-01-04 04:06:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: RI3POM6DRHZSW2JGO65DNYHCQ5S3J57VQ7JALUQGAKIUSGXYAHRA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FZY37TXFW4P3QGWUO4WQGHRXOZVRDPOTB6LBD3FKJ2MBV36PUFKA) context deadline exceeded
======LOGS======
Jan 4 04:05:38.124457389 (708h36m37.995540435s) [upload=620379]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 04:05:38.124463833 (708h36m37.995546868s) [upload=620379]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 04:05:38.373411420 (708h36m38.244494465s) [seg_pos={0 0}, upload=620379]: Begin upload segment...
Jan 4 04:05:38.373473388 (708h36m38.244556434s) [seg_pos={0 0}, upload=620379]: Done begin upload segment.
Jan 4 04:05:38.373507684 (708h36m38.244590729s) [noise=true , address=145.79.227.50:4373 , piece_id=7HC3VJVLZ3QB773PJALK7QNNH7BHTQ6BAF3FJMWXOPAX4EYVGOLQ, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.373656437 (708h36m38.244739472s) [noise=true , address=145.79.227.9:4365 , piece_id=OULDSSJLYVFHPLJ77TKZMC5HEHSULAJZFYKAZXZMV64CUG6FOA4A, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.374450399 (708h36m38.245533434s) [noise=true , address=45.91.116.7:4009 , piece_id=ZLVFPB7YARB55E6OFFO7Y42UZMVNK2EAIPZZ2CC4SYWT7XOUQTQA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.376249906 (708h36m38.247332941s) [noise=true , address=145.79.227.14:4838 , piece_id=NT73DSX7BOEO2XHVDJRGCTVOYBQEMWK326MGNNZ4NLRNWKREYLTA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.376271717 (708h36m38.247354762s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTYOL3V4TSOCTG2CTAN5BNNYWS2PAOPECY3EEN2FOZQVLHX6VQA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.376622906 (708h36m38.247705951s) [noise=true , address=77.90.31.14:4662 , piece_id=RI3POM6DRHZSW2JGO65DNYHCQ5S3J57VQ7JALUQGAKIUSGXYAHRA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.376913479 (708h36m38.247996524s) [noise=true , address=145.79.227.45:4230 , piece_id=3ZLFSRYOU2UVOVPII5664O2N5UO73TFCXIFXFKXX2MQ2PVEJAVGA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.376985376 (708h36m38.248068421s) [noise=true , address=145.79.227.17:4972 , piece_id=GQHVNKJP3HEADR4IU4FWOQMVO35GBCJVI6RNV7BFCSC4ZNLKAMBQ, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.377321416 (708h36m38.248404461s) [noise=true , address=77.90.31.11:4556 , piece_id=FZY37TXFW4P3QGWUO4WQGHRXOZVRDPOTB6LBD3FKJ2MBV36PUFKA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.377685429 (708h36m38.248768474s) [noise=true , address=145.79.227.21:4474 , piece_id=FSVDF4TZBMAIURXLJR5OUH4ETMIYLTKGSCUN5O3NAJQWKGQ7YUBA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.377745423 (708h36m38.248828458s) [noise=true , address=45.91.116.6:4975 , piece_id=OWKDY2FHYUXQLAJ2WNYB5M3425XV76KULQQ3NR4NLEFCEHM44HGQ, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.378158410 (708h36m38.249241455s) [noise=true , address=145.79.227.38:4979 , piece_id=7F3PXGVPF3MSFDSCUP46NVRNO2TZTDR6MNTWPW3EBII6URRPVNFQ, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.378476415 (708h36m38.24955946s) [noise=true , address=145.79.227.5:4223 , piece_id=67MPVICGGFKQREHMFAHAYKV6WDITXAAHUN47SITUSWLAPMSCXZFQ, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.378490913 (708h36m38.249573958s) [noise=true , address=145.79.227.20:4438 , piece_id=4IDNQZVI25VAXKN4EJHSGLHFHUJOKRIRMIVRMMTJ7HTMOO2UYGAA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.379101596 (708h36m38.250184651s) [noise=true , address=145.79.227.52:4445 , piece_id=NT3SYT7JAWBPAKLA4WDKAQJIEQDR2FYVSARC7EZXYEPGUX3MN6IA, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:38.379729772 (708h36m38.250812807s) [noise=true , address=45.91.116.12:4228 , piece_id=BCUICLDQVEXK2F7EXWBYWZFHLN3YGL7LBHFOKFMQXPXBGKLH3GIQ, seg_pos={0 0}, upload=620379]: Uploading piece...
Jan 4 04:05:40.072684326 (708h36m39.943767371s) [upload=620379]: Next returned nil segment
Jan 4 04:05:40.072702670 (708h36m39.943785715s) [upload=620379]: Waiting for error group managing segments...
Jan 4 04:05:40.367565980 (708h36m40.238649025s) [noise=true , address=145.79.227.52:4445 , piece_id=NT3SYT7JAWBPAKLA4WDKAQJIEQDR2FYVSARC7EZXYEPGUX3MN6IA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.368088435 (708h36m40.23917147s) [noise=true , address=145.79.227.21:4474 , piece_id=FSVDF4TZBMAIURXLJR5OUH4ETMIYLTKGSCUN5O3NAJQWKGQ7YUBA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.383289290 (708h36m40.254372325s) [noise=true , address=145.79.227.17:4972 , piece_id=GQHVNKJP3HEADR4IU4FWOQMVO35GBCJVI6RNV7BFCSC4ZNLKAMBQ, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.385466306 (708h36m40.256549341s) [noise=true , address=145.79.227.5:4223 , piece_id=67MPVICGGFKQREHMFAHAYKV6WDITXAAHUN47SITUSWLAPMSCXZFQ, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.388578953 (708h36m40.259661998s) [noise=true , address=145.79.227.14:4838 , piece_id=NT73DSX7BOEO2XHVDJRGCTVOYBQEMWK326MGNNZ4NLRNWKREYLTA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.392812054 (708h36m40.263895099s) [noise=true , address=145.79.227.50:4373 , piece_id=7HC3VJVLZ3QB773PJALK7QNNH7BHTQ6BAF3FJMWXOPAX4EYVGOLQ, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.400735020 (708h36m40.271818055s) [noise=true , address=145.79.227.38:4979 , piece_id=7F3PXGVPF3MSFDSCUP46NVRNO2TZTDR6MNTWPW3EBII6URRPVNFQ, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.403596609 (708h36m40.274679654s) [noise=true , address=145.79.227.20:4438 , piece_id=4IDNQZVI25VAXKN4EJHSGLHFHUJOKRIRMIVRMMTJ7HTMOO2UYGAA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.448714501 (708h36m40.319797536s) [noise=true , address=145.79.227.45:4230 , piece_id=3ZLFSRYOU2UVOVPII5664O2N5UO73TFCXIFXFKXX2MQ2PVEJAVGA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.452452139 (708h36m40.323535194s) [noise=true , address=145.79.227.9:4365 , piece_id=OULDSSJLYVFHPLJ77TKZMC5HEHSULAJZFYKAZXZMV64CUG6FOA4A, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.775993967 (708h36m40.647077012s) [noise=true , address=77.90.31.11:4556 , piece_id=FZY37TXFW4P3QGWUO4WQGHRXOZVRDPOTB6LBD3FKJ2MBV36PUFKA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.822059804 (708h36m40.693142849s) [noise=true , address=77.90.31.14:4662 , piece_id=RI3POM6DRHZSW2JGO65DNYHCQ5S3J57VQ7JALUQGAKIUSGXYAHRA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.958337941 (708h36m40.829420986s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTYOL3V4TSOCTG2CTAN5BNNYWS2PAOPECY3EEN2FOZQVLHX6VQA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: <nil>
Jan 4 04:05:40.958384720 (708h36m40.829467765s) [seg_pos={0 0}, upload=620379]: Segment reached optimal threshold of 13 pieces.
Jan 4 04:05:40.958738254 (708h36m40.829821309s) [noise=true , address=45.91.116.12:4228 , piece_id=BCUICLDQVEXK2F7EXWBYWZFHLN3YGL7LBHFOKFMQXPXBGKLH3GIQ, seg_pos={0 0}, upload=620379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:60016->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:60016->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60016->45.91.116.12:4228: use of closed network connection
Jan 4 04:05:40.958751439 (708h36m40.829834474s) [noise=true , address=45.91.116.12:4228 , piece_id=BCUICLDQVEXK2F7EXWBYWZFHLN3YGL7LBHFOKFMQXPXBGKLH3GIQ, seg_pos={0 0}, upload=620379]: Long tail cancellation due to optimal threshold
Jan 4 04:05:40.958788279 (708h36m40.829871314s) [noise=true , address=45.91.116.6:4975 , piece_id=OWKDY2FHYUXQLAJ2WNYB5M3425XV76KULQQ3NR4NLEFCEHM44HGQ, seg_pos={0 0}, upload=620379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49868->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49868->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49868->45.91.116.6:4975: use of closed network connection
Jan 4 04:05:40.958796133 (708h36m40.829879168s) [noise=true , address=45.91.116.6:4975 , piece_id=OWKDY2FHYUXQLAJ2WNYB5M3425XV76KULQQ3NR4NLEFCEHM44HGQ, seg_pos={0 0}, upload=620379]: Long tail cancellation due to optimal threshold
Jan 4 04:05:40.959021483 (708h36m40.830104528s) [noise=true , address=45.91.116.7:4009 , piece_id=ZLVFPB7YARB55E6OFFO7Y42UZMVNK2EAIPZZ2CC4SYWT7XOUQTQA, seg_pos={0 0}, upload=620379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: read tcp 188.166.225.170:41224->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:05:40.959028867 (708h36m40.830111912s) [noise=true , address=45.91.116.7:4009 , piece_id=ZLVFPB7YARB55E6OFFO7Y42UZMVNK2EAIPZZ2CC4SYWT7XOUQTQA, seg_pos={0 0}, upload=620379]: Long tail cancellation due to optimal threshold
Jan 4 04:05:40.959048033 (708h36m40.830131068s) [seg_pos={0 0}, upload=620379]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 04:05:40.959166168 (708h36m40.830249203s) [upload=620379]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:05:40.959171849 (708h36m40.830254884s) [upload=620379]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:05:41.271165097 (708h36m41.142248142s) [upload=620379]: Done uploading segments...
Jan 4 04:05:41.271182911 (708h36m41.142265956s) [upload=620379]: Upload finished. err: <nil>
Duration: 30.002940649s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000751082s trace:
https://jaeger.dev.storj.io/trace/1c7f44d21e764f49
FAIL (2026-01-04 04:06:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000775359s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [XXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..XXXXXX.] duration:
15.562681474s trace:
https://jaeger.dev.storj.io/trace/754018c7d5055863
FAIL (2026-01-04 04:05:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: RLSLFISITJBTUP3Y3O6HDSZ3Q72FETTGGW4HOD2SAYSQMSNQB54Q) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: IEO7FZHAAXLCOMVBXESOIJM5W4RTOXG3FO4S2WMQSKZHFW3PAUJQ) context deadline exceeded; piecestore: (Node ID: 1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, Piece ID: K3TQ72NQSA5553AF4DANLL5NUC6A537C4NCFI4WSKK3SBDRSJWDQ) context deadline exceeded; piecestore: (Node ID: 13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, Piece ID: I7PEVL6NZU7SO6UY35ISW6SGM27BBVTJNL5BKOVR6MFZ3XRZMDPA) context deadline exceeded
======LOGS======
CMJIM5G36Z4TEN4VG7PYFS6YNIDIHGVVDXQQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.720078047 (708h35m30.591161082s) [noise=true , address=45.91.116.10:4091 , piece_id=RLSLFISITJBTUP3Y3O6HDSZ3Q72FETTGGW4HOD2SAYSQMSNQB54Q, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.720555978 (708h35m30.591639023s) [noise=true , address=45.91.116.5:4317 , piece_id=IEO7FZHAAXLCOMVBXESOIJM5W4RTOXG3FO4S2WMQSKZHFW3PAUJQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.720746119 (708h35m30.591829164s) [noise=true , address=145.79.227.35:4508 , piece_id=SM5JPUYGM5LPQZ4YCWGYCPMQFKFX6GKNLMTIUCBTZXAWBCHI5QTA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721001355 (708h35m30.5920844s) [noise=true , address=45.91.116.13:4169 , piece_id=K3TQ72NQSA5553AF4DANLL5NUC6A537C4NCFI4WSKK3SBDRSJWDQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721212467 (708h35m30.592295502s) [noise=true , address=64.49.13.155:4914 , piece_id=QN5MTGT34HS5DMZVH4EJLFZNEETTYAX2D4TRZPEX3AP3FVPI7KMA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721327006 (708h35m30.592410051s) [noise=true , address=145.79.227.15:4899 , piece_id=ZBRCMTAXW6P4ZYIROH7TYZYZKCZWD3W6KJRSDMDAX34G4YWRW7JA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721637287 (708h35m30.592720332s) [noise=true , address=145.79.227.33:4872 , piece_id=4BGZDAB7KWCXKS7IO7KJ35JBWHY54ARTNHCBED5262RGU45YLWWA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721657404 (708h35m30.59274046s) [noise=true , address=145.79.227.49:4351 , piece_id=V4QNW2QPGZLSADELGEOSPJRUOP7LPFDLNQ2LUBPJFRMXIOHIH2YQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722189477 (708h35m30.593272522s) [noise=true , address=145.79.227.30:4791 , piece_id=G72CYPH4CNMN25HNF23UV3V4U7PFNYIBGNMORNNOIMHTSCJ7G4NA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722270332 (708h35m30.593353367s) [noise=true , address=145.79.227.34:4938 , piece_id=COWDYKVA474PYLQSCFBQYQQ3DCBWHWWES4VT4QG3CI7CUJACCZQQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722514447 (708h35m30.593597482s) [noise=true , address=77.90.31.10:4545 , piece_id=UALP3U753DPYMGBO4KI4PDIV5QZBV6QTFQAQTDGGGGYLYIKDJZUA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722821011 (708h35m30.593904046s) [noise=true , address=145.79.227.7:4315 , piece_id=3MSJD3BH4AGQ6W7634EOWDAI74PWLEW5UFBBTEQ6XZ53AX7FHTZQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722914508 (708h35m30.593997543s) [noise=true , address=145.79.227.43:4167 , piece_id=QJUDSHALRAXNZZQSVLAKP6PXPPKNYWJEYO33B6GOVHZ23AOAMK7Q, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.723408369 (708h35m30.594491414s) [noise=true , address=145.79.227.40:4084 , piece_id=EAKUXFJZFMGPXOHNUEA5MRBGUO4B3FSBJB5GOGEDTA4AIHDOGIUA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.725316592 (708h35m30.596399627s) [noise=true , address=64.49.13.26:4954 , piece_id=BUX6VO7N7IS6FXZUJNP3YEOPCRKZQ672EGKRHDE6YGUXHKI4FGKA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:32.106865978 (708h35m31.977949023s) [upload=620365]: Next returned nil segment
Jan 4 04:04:32.106885435 (708h35m31.97796848s) [upload=620365]: Waiting for error group managing segments...
Jan 4 04:04:32.492702626 (708h35m32.363785671s) [noise=true , address=64.49.13.26:4954 , piece_id=BUX6VO7N7IS6FXZUJNP3YEOPCRKZQ672EGKRHDE6YGUXHKI4FGKA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:32.584475839 (708h35m32.455558884s) [noise=true , address=145.79.227.7:4315 , piece_id=3MSJD3BH4AGQ6W7634EOWDAI74PWLEW5UFBBTEQ6XZ53AX7FHTZQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:32.599690639 (708h35m32.470773684s) [noise=true , address=145.79.227.15:4899 , piece_id=ZBRCMTAXW6P4ZYIROH7TYZYZKCZWD3W6KJRSDMDAX34G4YWRW7JA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.052306267 (708h35m32.923389302s) [noise=true , address=45.91.116.13:4169 , piece_id=K3TQ72NQSA5553AF4DANLL5NUC6A537C4NCFI4WSKK3SBDRSJWDQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.077039784 (708h35m32.948122839s) [noise=true , address=45.91.116.10:4091 , piece_id=RLSLFISITJBTUP3Y3O6HDSZ3Q72FETTGGW4HOD2SAYSQMSNQB54Q, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.097588252 (708h35m32.968671287s) [noise=true , address=45.91.116.11:4151 , piece_id=I7PEVL6NZU7SO6UY35ISW6SGM27BBVTJNL5BKOVR6MFZ3XRZMDPA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.538301715 (708h35m33.40938476s) [noise=true , address=45.91.116.5:4317 , piece_id=IEO7FZHAAXLCOMVBXESOIJM5W4RTOXG3FO4S2WMQSKZHFW3PAUJQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.552147581 (708h35m33.423230626s) [noise=true , address=145.79.227.30:4791 , piece_id=G72CYPH4CNMN25HNF23UV3V4U7PFNYIBGNMORNNOIMHTSCJ7G4NA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.567745842 (708h35m33.438828897s) [noise=true , address=145.79.227.33:4872 , piece_id=4BGZDAB7KWCXKS7IO7KJ35JBWHY54ARTNHCBED5262RGU45YLWWA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.568240133 (708h35m33.439323178s) [noise=true , address=145.79.227.40:4084 , piece_id=EAKUXFJZFMGPXOHNUEA5MRBGUO4B3FSBJB5GOGEDTA4AIHDOGIUA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.617292547 (708h35m33.488375592s) [noise=true , address=145.79.227.49:4351 , piece_id=V4QNW2QPGZLSADELGEOSPJRUOP7LPFDLNQ2LUBPJFRMXIOHIH2YQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.672616053 (708h35m33.543699088s) [noise=true , address=145.79.227.35:4508 , piece_id=SM5JPUYGM5LPQZ4YCWGYCPMQFKFX6GKNLMTIUCBTZXAWBCHI5QTA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.751520720 (708h35m33.622603785s) [noise=true , address=145.79.227.34:4938 , piece_id=COWDYKVA474PYLQSCFBQYQQ3DCBWHWWES4VT4QG3CI7CUJACCZQQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:34.126797051 (708h35m33.997880086s) [noise=true , address=145.79.227.43:4167 , piece_id=QJUDSHALRAXNZZQSVLAKP6PXPPKNYWJEYO33B6GOVHZ23AOAMK7Q, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:34.126821097 (708h35m33.997904142s) [seg_pos={0 0}, upload=620365]: Segment reached optimal threshold of 14 pieces.
Jan 4 04:04:34.127383487 (708h35m33.998466532s) [noise=true , address=193.149.164.143:4695 , piece_id=PYHM6CRMS6CKYUZNF5B3DLOQISSJNXXIII3EMAFKUITM4XQI6D6A, seg_pos={0 0}, upload=620365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Jan 4 04:04:34.127399057 (708h35m33.998482102s) [noise=true , address=193.149.164.143:4695 , piece_id=PYHM6CRMS6CKYUZNF5B3DLOQISSJNXXIII3EMAFKUITM4XQI6D6A, seg_pos={0 0}, upload=620365]: Long tail cancellation due to optimal threshold
Jan 4 04:04:34.128213979 (708h35m33.999297014s) [noise=true , address=64.49.13.155:4914 , piece_id=QN5MTGT34HS5DMZVH4EJLFZNEETTYAX2D4TRZPEX3AP3FVPI7KMA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: read tcp 188.166.225.170:46178->64.49.13.155:4914: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:04:34.128225731 (708h35m33.999308766s) [noise=true , address=64.49.13.155:4914 , piece_id=QN5MTGT34HS5DMZVH4EJLFZNEETTYAX2D4TRZPEX3AP3FVPI7KMA, seg_pos={0 0}, upload=620365]: Long tail cancellation due to optimal threshold
Jan 4 04:04:34.128387840 (708h35m33.999470885s) [noise=true , address=77.90.31.10:4545 , piece_id=UALP3U753DPYMGBO4KI4PDIV5QZBV6QTFQAQTDGGGGYLYIKDJZUA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX): protocol: read tcp 188.166.225.170:33466->77.90.31.10:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:04:34.128397889 (708h35m33.999480924s) [noise=true , address=77.90.31.10:4545 , piece_id=UALP3U753DPYMGBO4KI4PDIV5QZBV6QTFQAQTDGGGGYLYIKDJZUA, seg_pos={0 0}, upload=620365]: Long tail cancellation due to optimal threshold
Jan 4 04:04:34.128551962 (708h35m33.999634997s) [noise=true , address=45.91.116.7:4040 , piece_id=F5LI3BSK74KVDDPKCMJIM5G36Z4TEN4VG7PYFS6YNIDIHGVVDXQQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126XVFQJR5dbxDwfCwsYFUh8XM7ossbNEdNR4LP9vNAb9Zz1k8g): protocol: read tcp 188.166.225.170:33032->45.91.116.7:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:04:34.128562262 (708h35m33.999645297s) [noise=true , address=45.91.116.7:4040 , piece_id=F5LI3BSK74KVDDPKCMJIM5G36Z4TEN4VG7PYFS6YNIDIHGVVDXQQ, seg_pos={0 0}, upload=620365]: Long tail cancellation due to optimal threshold
Jan 4 04:04:34.128578062 (708h35m33.999661097s) [seg_pos={0 0}, upload=620365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 04:04:34.128650920 (708h35m33.999733965s) [upload=620365]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:04:34.128656601 (708h35m33.999739636s) [upload=620365]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:04:34.417608094 (708h35m34.288691139s) [upload=620365]: Done uploading segments...
Jan 4 04:04:34.417634364 (708h35m34.288717399s) [upload=620365]: Upload finished. err: <nil>
Duration: 15.562708917s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001607242s trace:
https://jaeger.dev.storj.io/trace/85172db08ca61c9
FAIL (2026-01-04 04:05:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001634322s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
35 out of 60 is failed (threshold=3) [XXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..XXXXXX] duration:
1m0.00239234s trace:
https://jaeger.dev.storj.io/trace/754018c7d5055863
FAIL (2026-01-04 04:05:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: RLSLFISITJBTUP3Y3O6HDSZ3Q72FETTGGW4HOD2SAYSQMSNQB54Q) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: IEO7FZHAAXLCOMVBXESOIJM5W4RTOXG3FO4S2WMQSKZHFW3PAUJQ) context deadline exceeded; piecestore: (Node ID: 1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, Piece ID: K3TQ72NQSA5553AF4DANLL5NUC6A537C4NCFI4WSKK3SBDRSJWDQ) context deadline exceeded; piecestore: (Node ID: 13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, Piece ID: I7PEVL6NZU7SO6UY35ISW6SGM27BBVTJNL5BKOVR6MFZ3XRZMDPA) context deadline exceeded
======LOGS======
CMJIM5G36Z4TEN4VG7PYFS6YNIDIHGVVDXQQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.720078047 (708h35m30.591161082s) [noise=true , address=45.91.116.10:4091 , piece_id=RLSLFISITJBTUP3Y3O6HDSZ3Q72FETTGGW4HOD2SAYSQMSNQB54Q, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.720555978 (708h35m30.591639023s) [noise=true , address=45.91.116.5:4317 , piece_id=IEO7FZHAAXLCOMVBXESOIJM5W4RTOXG3FO4S2WMQSKZHFW3PAUJQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.720746119 (708h35m30.591829164s) [noise=true , address=145.79.227.35:4508 , piece_id=SM5JPUYGM5LPQZ4YCWGYCPMQFKFX6GKNLMTIUCBTZXAWBCHI5QTA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721001355 (708h35m30.5920844s) [noise=true , address=45.91.116.13:4169 , piece_id=K3TQ72NQSA5553AF4DANLL5NUC6A537C4NCFI4WSKK3SBDRSJWDQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721212467 (708h35m30.592295502s) [noise=true , address=64.49.13.155:4914 , piece_id=QN5MTGT34HS5DMZVH4EJLFZNEETTYAX2D4TRZPEX3AP3FVPI7KMA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721327006 (708h35m30.592410051s) [noise=true , address=145.79.227.15:4899 , piece_id=ZBRCMTAXW6P4ZYIROH7TYZYZKCZWD3W6KJRSDMDAX34G4YWRW7JA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721637287 (708h35m30.592720332s) [noise=true , address=145.79.227.33:4872 , piece_id=4BGZDAB7KWCXKS7IO7KJ35JBWHY54ARTNHCBED5262RGU45YLWWA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.721657404 (708h35m30.59274046s) [noise=true , address=145.79.227.49:4351 , piece_id=V4QNW2QPGZLSADELGEOSPJRUOP7LPFDLNQ2LUBPJFRMXIOHIH2YQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722189477 (708h35m30.593272522s) [noise=true , address=145.79.227.30:4791 , piece_id=G72CYPH4CNMN25HNF23UV3V4U7PFNYIBGNMORNNOIMHTSCJ7G4NA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722270332 (708h35m30.593353367s) [noise=true , address=145.79.227.34:4938 , piece_id=COWDYKVA474PYLQSCFBQYQQ3DCBWHWWES4VT4QG3CI7CUJACCZQQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722514447 (708h35m30.593597482s) [noise=true , address=77.90.31.10:4545 , piece_id=UALP3U753DPYMGBO4KI4PDIV5QZBV6QTFQAQTDGGGGYLYIKDJZUA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722821011 (708h35m30.593904046s) [noise=true , address=145.79.227.7:4315 , piece_id=3MSJD3BH4AGQ6W7634EOWDAI74PWLEW5UFBBTEQ6XZ53AX7FHTZQ, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.722914508 (708h35m30.593997543s) [noise=true , address=145.79.227.43:4167 , piece_id=QJUDSHALRAXNZZQSVLAKP6PXPPKNYWJEYO33B6GOVHZ23AOAMK7Q, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.723408369 (708h35m30.594491414s) [noise=true , address=145.79.227.40:4084 , piece_id=EAKUXFJZFMGPXOHNUEA5MRBGUO4B3FSBJB5GOGEDTA4AIHDOGIUA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:30.725316592 (708h35m30.596399627s) [noise=true , address=64.49.13.26:4954 , piece_id=BUX6VO7N7IS6FXZUJNP3YEOPCRKZQ672EGKRHDE6YGUXHKI4FGKA, seg_pos={0 0}, upload=620365]: Uploading piece...
Jan 4 04:04:32.106865978 (708h35m31.977949023s) [upload=620365]: Next returned nil segment
Jan 4 04:04:32.106885435 (708h35m31.97796848s) [upload=620365]: Waiting for error group managing segments...
Jan 4 04:04:32.492702626 (708h35m32.363785671s) [noise=true , address=64.49.13.26:4954 , piece_id=BUX6VO7N7IS6FXZUJNP3YEOPCRKZQ672EGKRHDE6YGUXHKI4FGKA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:32.584475839 (708h35m32.455558884s) [noise=true , address=145.79.227.7:4315 , piece_id=3MSJD3BH4AGQ6W7634EOWDAI74PWLEW5UFBBTEQ6XZ53AX7FHTZQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:32.599690639 (708h35m32.470773684s) [noise=true , address=145.79.227.15:4899 , piece_id=ZBRCMTAXW6P4ZYIROH7TYZYZKCZWD3W6KJRSDMDAX34G4YWRW7JA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.052306267 (708h35m32.923389302s) [noise=true , address=45.91.116.13:4169 , piece_id=K3TQ72NQSA5553AF4DANLL5NUC6A537C4NCFI4WSKK3SBDRSJWDQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.077039784 (708h35m32.948122839s) [noise=true , address=45.91.116.10:4091 , piece_id=RLSLFISITJBTUP3Y3O6HDSZ3Q72FETTGGW4HOD2SAYSQMSNQB54Q, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.097588252 (708h35m32.968671287s) [noise=true , address=45.91.116.11:4151 , piece_id=I7PEVL6NZU7SO6UY35ISW6SGM27BBVTJNL5BKOVR6MFZ3XRZMDPA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.538301715 (708h35m33.40938476s) [noise=true , address=45.91.116.5:4317 , piece_id=IEO7FZHAAXLCOMVBXESOIJM5W4RTOXG3FO4S2WMQSKZHFW3PAUJQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.552147581 (708h35m33.423230626s) [noise=true , address=145.79.227.30:4791 , piece_id=G72CYPH4CNMN25HNF23UV3V4U7PFNYIBGNMORNNOIMHTSCJ7G4NA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.567745842 (708h35m33.438828897s) [noise=true , address=145.79.227.33:4872 , piece_id=4BGZDAB7KWCXKS7IO7KJ35JBWHY54ARTNHCBED5262RGU45YLWWA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.568240133 (708h35m33.439323178s) [noise=true , address=145.79.227.40:4084 , piece_id=EAKUXFJZFMGPXOHNUEA5MRBGUO4B3FSBJB5GOGEDTA4AIHDOGIUA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.617292547 (708h35m33.488375592s) [noise=true , address=145.79.227.49:4351 , piece_id=V4QNW2QPGZLSADELGEOSPJRUOP7LPFDLNQ2LUBPJFRMXIOHIH2YQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.672616053 (708h35m33.543699088s) [noise=true , address=145.79.227.35:4508 , piece_id=SM5JPUYGM5LPQZ4YCWGYCPMQFKFX6GKNLMTIUCBTZXAWBCHI5QTA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:33.751520720 (708h35m33.622603785s) [noise=true , address=145.79.227.34:4938 , piece_id=COWDYKVA474PYLQSCFBQYQQ3DCBWHWWES4VT4QG3CI7CUJACCZQQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:34.126797051 (708h35m33.997880086s) [noise=true , address=145.79.227.43:4167 , piece_id=QJUDSHALRAXNZZQSVLAKP6PXPPKNYWJEYO33B6GOVHZ23AOAMK7Q, seg_pos={0 0}, upload=620365]: Done uploading piece. err: <nil>
Jan 4 04:04:34.126821097 (708h35m33.997904142s) [seg_pos={0 0}, upload=620365]: Segment reached optimal threshold of 14 pieces.
Jan 4 04:04:34.127383487 (708h35m33.998466532s) [noise=true , address=193.149.164.143:4695 , piece_id=PYHM6CRMS6CKYUZNF5B3DLOQISSJNXXIII3EMAFKUITM4XQI6D6A, seg_pos={0 0}, upload=620365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Jan 4 04:04:34.127399057 (708h35m33.998482102s) [noise=true , address=193.149.164.143:4695 , piece_id=PYHM6CRMS6CKYUZNF5B3DLOQISSJNXXIII3EMAFKUITM4XQI6D6A, seg_pos={0 0}, upload=620365]: Long tail cancellation due to optimal threshold
Jan 4 04:04:34.128213979 (708h35m33.999297014s) [noise=true , address=64.49.13.155:4914 , piece_id=QN5MTGT34HS5DMZVH4EJLFZNEETTYAX2D4TRZPEX3AP3FVPI7KMA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: read tcp 188.166.225.170:46178->64.49.13.155:4914: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:04:34.128225731 (708h35m33.999308766s) [noise=true , address=64.49.13.155:4914 , piece_id=QN5MTGT34HS5DMZVH4EJLFZNEETTYAX2D4TRZPEX3AP3FVPI7KMA, seg_pos={0 0}, upload=620365]: Long tail cancellation due to optimal threshold
Jan 4 04:04:34.128387840 (708h35m33.999470885s) [noise=true , address=77.90.31.10:4545 , piece_id=UALP3U753DPYMGBO4KI4PDIV5QZBV6QTFQAQTDGGGGYLYIKDJZUA, seg_pos={0 0}, upload=620365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX): protocol: read tcp 188.166.225.170:33466->77.90.31.10:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:04:34.128397889 (708h35m33.999480924s) [noise=true , address=77.90.31.10:4545 , piece_id=UALP3U753DPYMGBO4KI4PDIV5QZBV6QTFQAQTDGGGGYLYIKDJZUA, seg_pos={0 0}, upload=620365]: Long tail cancellation due to optimal threshold
Jan 4 04:04:34.128551962 (708h35m33.999634997s) [noise=true , address=45.91.116.7:4040 , piece_id=F5LI3BSK74KVDDPKCMJIM5G36Z4TEN4VG7PYFS6YNIDIHGVVDXQQ, seg_pos={0 0}, upload=620365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126XVFQJR5dbxDwfCwsYFUh8XM7ossbNEdNR4LP9vNAb9Zz1k8g): protocol: read tcp 188.166.225.170:33032->45.91.116.7:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:04:34.128562262 (708h35m33.999645297s) [noise=true , address=45.91.116.7:4040 , piece_id=F5LI3BSK74KVDDPKCMJIM5G36Z4TEN4VG7PYFS6YNIDIHGVVDXQQ, seg_pos={0 0}, upload=620365]: Long tail cancellation due to optimal threshold
Jan 4 04:04:34.128578062 (708h35m33.999661097s) [seg_pos={0 0}, upload=620365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 04:04:34.128650920 (708h35m33.999733965s) [upload=620365]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:04:34.128656601 (708h35m33.999739636s) [upload=620365]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:04:34.417608094 (708h35m34.288691139s) [upload=620365]: Done uploading segments...
Jan 4 04:04:34.417634364 (708h35m34.288717399s) [upload=620365]: Upload finished. err: <nil>
Duration: 1m0.002403301s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..XX..X] duration:
30.005553252s trace:
https://jaeger.dev.storj.io/trace/6831a534c0a52a25
FAIL (2026-01-04 04:05:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
4 04:04:33.863574550 (708h35m33.734657585s) [upload=620366]: Starting upload
Jan 4 04:04:33.863641197 (708h35m33.734724242s) [upload=620366]: Done starting upload
Jan 4 04:04:33.863736980 (708h35m33.734820025s) [upload=620366]: Uploading segments...
Jan 4 04:04:33.863880503 (708h35m33.734963548s) [upload=620366]: Got next segment. Inline: false
Jan 4 04:04:33.863921420 (708h35m33.735004456s) [upload=620366]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 04:04:33.863926380 (708h35m33.735009425s) [upload=620366]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 04:04:34.108106901 (708h35m33.979189956s) [seg_pos={0 0}, upload=620366]: Begin upload segment...
Jan 4 04:04:34.108169240 (708h35m33.979252285s) [seg_pos={0 0}, upload=620366]: Done begin upload segment.
Jan 4 04:04:34.108205229 (708h35m33.979288274s) [noise=true , address=145.79.227.40:4049 , piece_id=VXKMM7OZ3VR2NJ5TIZKMN22LVNSGFPRIZCNJVQRLYKD7PHGMMCAA, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.108215007 (708h35m33.979298052s) [noise=true , address=145.79.227.3:4150 , piece_id=VJLPIQXRT7FEUGQW63VMLT64HZX5PIOVGOQQBAEQH6MQUGVUXNVA, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.108786505 (708h35m33.97986955s) [noise=true , address=145.79.227.45:4231 , piece_id=PZYLBIU5ZBZVEQG5YQAVZHGODB42NFULZ7MZQBLT2HOYPOMW5U5Q, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.109087248 (708h35m33.980170293s) [noise=true , address=77.90.31.17:4770 , piece_id=HV4KE2RAUGK76VI2ZICUGPGINRS5T6OCURYLA5JFM3N44OH7I4UA, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.109307186 (708h35m33.980390231s) [noise=true , address=145.79.227.20:4438 , piece_id=YNY4HIRVQBUJZEMUMFYQTVKKTWLMHAGRXMTMNNQIVQZMCZC3GAXQ, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.109432555 (708h35m33.9805156s) [noise=true , address=145.79.227.15:4876 , piece_id=PTJXYVEPIQMGFRIFOGUXQBO25BD2ZKPQ3KHZM27BOO6P3XPR2THQ, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.109860109 (708h35m33.980943144s) [noise=true , address=145.79.227.52:4445 , piece_id=EO4ARM7CTXZB26ZMQCAAZBPYFIBRZBN5QH5SH5HIJDTPZNVX7OWA, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.109905396 (708h35m33.980988431s) [noise=true , address=45.91.116.5:4299 , piece_id=7S4XVPEEL7OC3RPE2CBLD23ANV3QIQOI2TXI3DN6HKUNPKMU47DA, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.110210386 (708h35m33.981293431s) [noise=true , address=145.79.227.46:4602 , piece_id=4B36IIFSLN54W4FFG43F2SVPWJMJ6GVXEFTWN5XCH7UUYSNJLOQQ, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.110546196 (708h35m33.981629231s) [noise=true , address=77.90.31.16:4735 , piece_id=CPXQS7DSMTPZ4552RIHEI4GQLUATWMOUOP36EGNM2TZEPAFTI6LA, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.110672136 (708h35m33.981755181s) [noise=true , address=145.79.227.14:4837 , piece_id=ZKKG7YTPXMCH5SG36HYMCBHBDST2EYWIPLWJESKZH7KW6LXZTTXA, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.110935056 (708h35m33.982018091s) [noise=true , address=77.90.31.11:4556 , piece_id=4NA3RZ4N77SBG5D4J3XR7P6324BGM6SH5X74DSFKTVPBJL7MALGQ, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.111123335 (708h35m33.98220637s) [noise=true , address=145.79.227.4:4187 , piece_id=OK5RTYLGB54W65CPTK5RPMMKOZWLHO4QWDP5CLGSTFFZE7IVVOVQ, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.111268922 (708h35m33.982351957s) [noise=true , address=145.79.227.10:4403 , piece_id=2AMB22ZCMXRBYJEFOYPBV4ID6SAL5S35BNH3UMG7VUH3JTEQVPIQ, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.111673041 (708h35m33.982756076s) [noise=true , address=45.91.116.4:4940 , piece_id=KFVTPH7A4ZPMURYXE3OLYLJIEZHCHFD4X5AEP2EEA2SNEQYPBWGQ, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:34.112081129 (708h35m33.983164164s) [noise=true , address=145.79.227.31:4798 , piece_id=6FEZGAO24JZJ3FZAFYMA3TBFK4G4B2QMZ2SVGFDANFECXBMC44OA, seg_pos={0 0}, upload=620366]: Uploading piece...
Jan 4 04:04:35.801162948 (708h35m35.672245993s) [upload=620366]: Next returned nil segment
Jan 4 04:04:35.801180803 (708h35m35.672263848s) [upload=620366]: Waiting for error group managing segments...
Jan 4 04:04:36.087096776 (708h35m35.958179821s) [noise=true , address=145.79.227.14:4837 , piece_id=ZKKG7YTPXMCH5SG36HYMCBHBDST2EYWIPLWJESKZH7KW6LXZTTXA, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.094778572 (708h35m35.965861617s) [noise=true , address=145.79.227.3:4150 , piece_id=VJLPIQXRT7FEUGQW63VMLT64HZX5PIOVGOQQBAEQH6MQUGVUXNVA, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.098730418 (708h35m35.969813483s) [noise=true , address=145.79.227.46:4602 , piece_id=4B36IIFSLN54W4FFG43F2SVPWJMJ6GVXEFTWN5XCH7UUYSNJLOQQ, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.098972258 (708h35m35.970055313s) [noise=true , address=145.79.227.15:4876 , piece_id=PTJXYVEPIQMGFRIFOGUXQBO25BD2ZKPQ3KHZM27BOO6P3XPR2THQ, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.123775057 (708h35m35.994858092s) [noise=true , address=145.79.227.52:4445 , piece_id=EO4ARM7CTXZB26ZMQCAAZBPYFIBRZBN5QH5SH5HIJDTPZNVX7OWA, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.124881704 (708h35m35.995964739s) [noise=true , address=145.79.227.10:4403 , piece_id=2AMB22ZCMXRBYJEFOYPBV4ID6SAL5S35BNH3UMG7VUH3JTEQVPIQ, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.125383108 (708h35m35.996466154s) [noise=true , address=145.79.227.31:4798 , piece_id=6FEZGAO24JZJ3FZAFYMA3TBFK4G4B2QMZ2SVGFDANFECXBMC44OA, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.125540268 (708h35m35.996623303s) [noise=true , address=145.79.227.4:4187 , piece_id=OK5RTYLGB54W65CPTK5RPMMKOZWLHO4QWDP5CLGSTFFZE7IVVOVQ, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.182776947 (708h35m36.053859982s) [noise=true , address=145.79.227.45:4231 , piece_id=PZYLBIU5ZBZVEQG5YQAVZHGODB42NFULZ7MZQBLT2HOYPOMW5U5Q, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.189324594 (708h35m36.060407629s) [noise=true , address=145.79.227.20:4438 , piece_id=YNY4HIRVQBUJZEMUMFYQTVKKTWLMHAGRXMTMNNQIVQZMCZC3GAXQ, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.492526242 (708h35m36.363609287s) [noise=true , address=77.90.31.11:4556 , piece_id=4NA3RZ4N77SBG5D4J3XR7P6324BGM6SH5X74DSFKTVPBJL7MALGQ, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:36.914365936 (708h35m36.785448971s) [noise=true , address=45.91.116.5:4299 , piece_id=7S4XVPEEL7OC3RPE2CBLD23ANV3QIQOI2TXI3DN6HKUNPKMU47DA, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:37.154998051 (708h35m37.026081086s) [noise=true , address=77.90.31.16:4735 , piece_id=CPXQS7DSMTPZ4552RIHEI4GQLUATWMOUOP36EGNM2TZEPAFTI6LA, seg_pos={0 0}, upload=620366]: Done uploading piece. err: <nil>
Jan 4 04:04:37.155017007 (708h35m37.026100042s) [seg_pos={0 0}, upload=620366]: Segment reached optimal threshold of 13 pieces.
Jan 4 04:04:37.155185528 (708h35m37.026268563s) [noise=true , address=77.90.31.17:4770 , piece_id=HV4KE2RAUGK76VI2ZICUGPGINRS5T6OCURYLA5JFM3N44OH7I4UA, seg_pos={0 0}, upload=620366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:52948->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:52948->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52948->77.90.31.17:4770: use of closed network connection
Jan 4 04:04:37.155190918 (708h35m37.026273953s) [noise=true , address=77.90.31.17:4770 , piece_id=HV4KE2RAUGK76VI2ZICUGPGINRS5T6OCURYLA5JFM3N44OH7I4UA, seg_pos={0 0}, upload=620366]: Long tail cancellation due to optimal threshold
Jan 4 04:04:37.155500768 (708h35m37.026583803s) [noise=true , address=45.91.116.4:4940 , piece_id=KFVTPH7A4ZPMURYXE3OLYLJIEZHCHFD4X5AEP2EEA2SNEQYPBWGQ, seg_pos={0 0}, upload=620366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Jan 4 04:04:37.155507701 (708h35m37.026590736s) [noise=true , address=45.91.116.4:4940 , piece_id=KFVTPH7A4ZPMURYXE3OLYLJIEZHCHFD4X5AEP2EEA2SNEQYPBWGQ, seg_pos={0 0}, upload=620366]: Long tail cancellation due to optimal threshold
Jan 4 04:04:37.155626848 (708h35m37.026709883s) [noise=true , address=145.79.227.40:4049 , piece_id=VXKMM7OZ3VR2NJ5TIZKMN22LVNSGFPRIZCNJVQRLYKD7PHGMMCAA, seg_pos={0 0}, upload=620366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Jan 4 04:04:37.155632358 (708h35m37.026715403s) [noise=true , address=145.79.227.40:4049 , piece_id=VXKMM7OZ3VR2NJ5TIZKMN22LVNSGFPRIZCNJVQRLYKD7PHGMMCAA, seg_pos={0 0}, upload=620366]: Long tail cancellation due to optimal threshold
Jan 4 04:04:37.155647868 (708h35m37.026730903s) [seg_pos={0 0}, upload=620366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 04:04:37.155729133 (708h35m37.026812168s) [upload=620366]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:04:37.155736476 (708h35m37.026819522s) [upload=620366]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:04:37.440981362 (708h35m37.312064407s) [upload=620366]: Done uploading segments...
Jan 4 04:04:37.441004385 (708h35m37.31208743s) [upload=620366]: Upload finished. err: <nil>
Duration: 30.005577799s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXXXX] duration:
5.001461104s trace:
https://jaeger.dev.storj.io/trace/5b668761b342b43d
FAIL (2026-01-04 04:04:51 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001482023s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
35 out of 60 is failed (threshold=3) [XXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..XXXXX] duration:
1m0.002908717s trace:
https://jaeger.dev.storj.io/trace/23530d0ad68eda7
FAIL (2026-01-04 04:04:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.7:4036 , piece_id=K7BATF4ISQ7BMIMZGUN7QSLK6PMHU2DN23AZ6XUGKAXYSPG6HKFA, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.498255601 (708h34m30.369338646s) [noise=true , address=145.79.227.34:4920 , piece_id=GITTHINULTPZTIRZED6DEVG2B3AIJP2CW3HB7E3RNH7DFGUSGLFQ, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.498826298 (708h34m30.369909343s) [noise=true , address=145.79.227.33:4899 , piece_id=N5PZDEBYACSL74GTJ3J2DTMH3PVVI62WAKTIMIROH3JCKDGF5MJQ, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.498994298 (708h34m30.370077344s) [noise=true , address=145.79.227.29:4976 , piece_id=3GZZTPTAUL7MEOPIFX3XTQP6ZOLTRI6VGNZHC65HK3U46JIPMZVQ, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.499212834 (708h34m30.370295869s) [noise=true , address=45.91.116.8:4074 , piece_id=NAHCUAYWPKZ7PJBOJY5S633Q2KLRBLWPUYLYMIXTZIQE6FY66SZQ, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.499330648 (708h34m30.370413683s) [noise=true , address=145.79.227.49:4344 , piece_id=DSDKDHKP6VZOI7VMLIDFBVVBUE3RHJ3PXRE5KRMJVGBXORQZOFTQ, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.499629738 (708h34m30.370712783s) [noise=true , address=145.79.227.2:4117 , piece_id=6XZFNAJ3SF4P737VDJHBGXFBOLQMJ5JHVS7DEB6L7IQ7TL5SGVGQ, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.499774894 (708h34m30.37085794s) [noise=true , address=64.49.13.140:4513 , piece_id=B26VH6VVENS6RNENGRXI4XGFR6T7CYCXEF6XFCRD5FJB3M5NTH5A, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.500043416 (708h34m30.371126461s) [noise=true , address=145.79.227.40:4066 , piece_id=IRY4Q5BUKBTOPJ7UCYS23KM5GGAMYY3V2IHQPZWU4IPWBCXWOKPA, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.500058374 (708h34m30.371141409s) [noise=true , address=145.79.227.30:4788 , piece_id=U6GYQNV4IF2HDY2YSANBC4OCJH47Z5DRZ7H2FEAWO6GPUAGDT6LA, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.500381128 (708h34m30.371464173s) [noise=true , address=145.79.227.32:4864 , piece_id=2PMR2NYBLDJLZXXRIRCGODBSU5IMN3GG7FGBMXBKBP3TI466NG5Q, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.500447284 (708h34m30.37153032s) [noise=true , address=64.49.13.169:4708 , piece_id=IP7OAH7OVYINRCYMKUOFUJLCJ7BIDWNVI5OXWUWBAUAH7FOSN7WA, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.500828661 (708h34m30.371911696s) [noise=true , address=77.90.31.18:4828 , piece_id=JETZA74IXX6DAOLWMUL4JYFLXDFCWFW5TGXP7YPLNRH6D557CRCA, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.500842207 (708h34m30.371925252s) [noise=true , address=77.90.31.16:4757 , piece_id=JGPLJVYDGZSX3Z5I73MMYA5DDOWQAWDKLQ6CYAVMMXRQWW6EB6GQ, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:30.502192147 (708h34m30.373275192s) [noise=true , address=145.79.227.6:4284 , piece_id=RBXGLXFLPOCMDZXGWZHPDCABZ5WJXGAFZGWHL4G2E2P34YGZKXJA, seg_pos={0 0}, upload=620353]: Uploading piece...
Jan 4 04:03:31.951160019 (708h34m31.822243054s) [upload=620353]: Next returned nil segment
Jan 4 04:03:31.951182973 (708h34m31.822266018s) [upload=620353]: Waiting for error group managing segments...
Jan 4 04:03:32.360304801 (708h34m32.231387846s) [noise=true , address=145.79.227.2:4117 , piece_id=6XZFNAJ3SF4P737VDJHBGXFBOLQMJ5JHVS7DEB6L7IQ7TL5SGVGQ, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:32.384846632 (708h34m32.255929668s) [noise=true , address=145.79.227.6:4284 , piece_id=RBXGLXFLPOCMDZXGWZHPDCABZ5WJXGAFZGWHL4G2E2P34YGZKXJA, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:32.528870559 (708h34m32.399953604s) [noise=true , address=64.49.13.140:4513 , piece_id=B26VH6VVENS6RNENGRXI4XGFR6T7CYCXEF6XFCRD5FJB3M5NTH5A, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:32.661540203 (708h34m32.532623248s) [noise=true , address=77.90.31.16:4757 , piece_id=JGPLJVYDGZSX3Z5I73MMYA5DDOWQAWDKLQ6CYAVMMXRQWW6EB6GQ, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:32.839916482 (708h34m32.710999527s) [noise=true , address=77.90.31.18:4828 , piece_id=JETZA74IXX6DAOLWMUL4JYFLXDFCWFW5TGXP7YPLNRH6D557CRCA, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.363447665 (708h34m33.2345307s) [noise=true , address=145.79.227.30:4788 , piece_id=U6GYQNV4IF2HDY2YSANBC4OCJH47Z5DRZ7H2FEAWO6GPUAGDT6LA, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.383485661 (708h34m33.254568706s) [noise=true , address=145.79.227.34:4920 , piece_id=GITTHINULTPZTIRZED6DEVG2B3AIJP2CW3HB7E3RNH7DFGUSGLFQ, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.408535810 (708h34m33.279618855s) [noise=true , address=145.79.227.40:4066 , piece_id=IRY4Q5BUKBTOPJ7UCYS23KM5GGAMYY3V2IHQPZWU4IPWBCXWOKPA, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.528074532 (708h34m33.399157578s) [noise=true , address=145.79.227.32:4864 , piece_id=2PMR2NYBLDJLZXXRIRCGODBSU5IMN3GG7FGBMXBKBP3TI466NG5Q, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.566819809 (708h34m33.437902844s) [noise=true , address=145.79.227.29:4976 , piece_id=3GZZTPTAUL7MEOPIFX3XTQP6ZOLTRI6VGNZHC65HK3U46JIPMZVQ, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.617630702 (708h34m33.488713747s) [noise=true , address=145.79.227.49:4344 , piece_id=DSDKDHKP6VZOI7VMLIDFBVVBUE3RHJ3PXRE5KRMJVGBXORQZOFTQ, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.640116740 (708h34m33.511199775s) [noise=true , address=64.49.13.155:4921 , piece_id=3RJYK2X6QZ2K4SL3YF64KKNEVSLWNPM5I4KAMI4AXNO2LVFCPTQA, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.765688060 (708h34m33.636771095s) [noise=true , address=45.91.116.8:4074 , piece_id=NAHCUAYWPKZ7PJBOJY5S633Q2KLRBLWPUYLYMIXTZIQE6FY66SZQ, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.959802105 (708h34m33.83088514s) [noise=true , address=145.79.227.33:4899 , piece_id=N5PZDEBYACSL74GTJ3J2DTMH3PVVI62WAKTIMIROH3JCKDGF5MJQ, seg_pos={0 0}, upload=620353]: Done uploading piece. err: <nil>
Jan 4 04:03:33.959824096 (708h34m33.830907142s) [seg_pos={0 0}, upload=620353]: Segment reached optimal threshold of 14 pieces.
Jan 4 04:03:33.960031391 (708h34m33.831114426s) [noise=true , address=64.49.13.169:4708 , piece_id=IP7OAH7OVYINRCYMKUOFUJLCJ7BIDWNVI5OXWUWBAUAH7FOSN7WA, seg_pos={0 0}, upload=620353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:46160->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:46160->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46160->64.49.13.169:4708: use of closed network connection
Jan 4 04:03:33.960039767 (708h34m33.831122812s) [noise=true , address=64.49.13.169:4708 , piece_id=IP7OAH7OVYINRCYMKUOFUJLCJ7BIDWNVI5OXWUWBAUAH7FOSN7WA, seg_pos={0 0}, upload=620353]: Long tail cancellation due to optimal threshold
Jan 4 04:03:33.960093550 (708h34m33.831176595s) [noise=true , address=193.149.164.143:4699 , piece_id=GCZVMCXIAGPAMKJI6YECHBNKXFBXV2KNP3NH6QD35MH3G2G3D33A, seg_pos={0 0}, upload=620353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF
Jan 4 04:03:33.960106023 (708h34m33.831189058s) [noise=true , address=193.149.164.143:4699 , piece_id=GCZVMCXIAGPAMKJI6YECHBNKXFBXV2KNP3NH6QD35MH3G2G3D33A, seg_pos={0 0}, upload=620353]: Long tail cancellation due to optimal threshold
Jan 4 04:03:33.960293310 (708h34m33.831376345s) [noise=true , address=64.49.13.139:4478 , piece_id=FIH2BB6YBFTPXUIPBJFQ5BI4CUKOYEUIZ4AUSDJIHU2DC36DZ7RQ, seg_pos={0 0}, upload=620353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J): protocol: read tcp 188.166.225.170:36498->64.49.13.139:4478: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:03:33.960300394 (708h34m33.831383439s) [noise=true , address=64.49.13.139:4478 , piece_id=FIH2BB6YBFTPXUIPBJFQ5BI4CUKOYEUIZ4AUSDJIHU2DC36DZ7RQ, seg_pos={0 0}, upload=620353]: Long tail cancellation due to optimal threshold
Jan 4 04:03:33.960386287 (708h34m33.831469332s) [noise=true , address=45.91.116.7:4036 , piece_id=K7BATF4ISQ7BMIMZGUN7QSLK6PMHU2DN23AZ6XUGKAXYSPG6HKFA, seg_pos={0 0}, upload=620353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5): protocol: read tcp 188.166.225.170:34742->45.91.116.7:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:03:33.960394012 (708h34m33.831477057s) [noise=true , address=45.91.116.7:4036 , piece_id=K7BATF4ISQ7BMIMZGUN7QSLK6PMHU2DN23AZ6XUGKAXYSPG6HKFA, seg_pos={0 0}, upload=620353]: Long tail cancellation due to optimal threshold
Jan 4 04:03:33.960413469 (708h34m33.831496504s) [seg_pos={0 0}, upload=620353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 04:03:33.960489744 (708h34m33.831572789s) [upload=620353]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:03:33.960495586 (708h34m33.831578621s) [upload=620353]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:03:34.248178714 (708h34m34.119261749s) [upload=620353]: Done uploading segments...
Jan 4 04:03:34.248198120 (708h34m34.119281155s) [upload=620353]: Upload finished. err: <nil>
Duration: 1m0.002921902s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXXX] duration:
5.000998393s trace:
https://jaeger.dev.storj.io/trace/77a820189eda8b0d
FAIL (2026-01-04 04:04:21 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001040013s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXXX] duration:
5.000745362s trace:
https://jaeger.dev.storj.io/trace/442f2947c0b506d0
FAIL (2026-01-04 04:04:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000775309s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [..XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..XX..] duration:
6.847658198s trace:
https://jaeger.dev.storj.io/trace/695820a92d75c5e7
FAIL (2026-01-04 04:03:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA) context deadline exceeded
======LOGS======
Jan 4 04:01:39.947630645 (708h32m39.81871368s) [upload=620328]: Starting upload
Jan 4 04:01:39.947700528 (708h32m39.818783563s) [upload=620328]: Done starting upload
Jan 4 04:01:39.947795158 (708h32m39.818878203s) [upload=620328]: Uploading segments...
Jan 4 04:01:39.947909907 (708h32m39.818992952s) [upload=620328]: Got next segment. Inline: false
Jan 4 04:01:39.947958859 (708h32m39.819041894s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 04:01:39.947964580 (708h32m39.819047625s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 04:01:40.192231657 (708h32m40.063314712s) [seg_pos={0 0}, upload=620328]: Begin upload segment...
Jan 4 04:01:40.192286240 (708h32m40.063369275s) [seg_pos={0 0}, upload=620328]: Done begin upload segment.
Jan 4 04:01:40.192312410 (708h32m40.063395445s) [noise=true , address=145.79.227.48:4302 , piece_id=Q4IJXB2KAPEX3WFQ6W2QEPGQOPZBKVQW6YSRZ2OAKIX33KJO3SGQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.192322890 (708h32m40.063405935s) [noise=true , address=145.79.227.21:4474 , piece_id=XAS6LZOZCZZ3AO67YATB6FBYTKEF2L6DJZBDTIESHDJR7LBERGAA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.192824275 (708h32m40.06390732s) [noise=true , address=45.91.116.11:4118 , piece_id=HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.193532203 (708h32m40.064615238s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.193849126 (708h32m40.064932171s) [noise=true , address=77.90.31.16:4736 , piece_id=UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194049769 (708h32m40.065132804s) [noise=true , address=77.90.31.15:4697 , piece_id=I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194291058 (708h32m40.065374103s) [noise=true , address=145.79.227.13:4802 , piece_id=GPN7ZUKCDN2OUSHD55SJV7ACG3BYMZLA2HQDXGSJC3JTOZ6JSYYQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194384946 (708h32m40.065467981s) [noise=true , address=145.79.227.41:4085 , piece_id=3XCI2KIJ72AONJXPX2N7TWSJVE5BHKSULW6ENWDLU24WCRAKUN3Q, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194661523 (708h32m40.065744558s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.209114475 (708h32m40.08019752s) [noise=true , address=145.79.227.11:4730 , piece_id=T5BLORMXA2XLLNE6IESXGTRXG5Q54NFRXA4TMNYTKDEOSXO3G7IA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.209135624 (708h32m40.080218679s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.210677250 (708h32m40.081760285s) [noise=true , address=145.79.227.3:4151 , piece_id=4KML6THW6OWTYZYSLKCX7JVGVQCMXZZ2LJHYO5XKFKPDT3JH5WMA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.212095761 (708h32m40.083178796s) [noise=true , address=145.79.227.47:4265 , piece_id=XTBTZ4NLGYFG3QER2YSF2GCPZG3YJTBHPYLKO4E7DUCNMRJB7OEQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.213929422 (708h32m40.085012467s) [noise=true , address=145.79.227.50:4375 , piece_id=OF3MTRXG3E5NVPMYVSK3Y43M6AHIL7ZKMOFXQEW3NHBSINO5KFCA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.219098505 (708h32m40.09018154s) [noise=true , address=145.79.227.38:4977 , piece_id=SFOF2UCEGAGAMO7SXXPJ3WWP273YCNEUBTF2KEIORAUXG74LJDJA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.220540901 (708h32m40.091623936s) [noise=true , address=145.79.227.37:4566 , piece_id=OIYHEJ377M24GIMMH72OQWJDX3NVT6VOINI2MQPJZ5B47FFQZ5DA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:41.810366911 (708h32m41.681449956s) [upload=620328]: Next returned nil segment
Jan 4 04:01:41.810384545 (708h32m41.68146758s) [upload=620328]: Waiting for error group managing segments...
Jan 4 04:01:42.195975145 (708h32m42.06705819s) [noise=true , address=145.79.227.3:4151 , piece_id=4KML6THW6OWTYZYSLKCX7JVGVQCMXZZ2LJHYO5XKFKPDT3JH5WMA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.200965087 (708h32m42.072048122s) [noise=true , address=145.79.227.41:4085 , piece_id=3XCI2KIJ72AONJXPX2N7TWSJVE5BHKSULW6ENWDLU24WCRAKUN3Q, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.202003935 (708h32m42.07308697s) [noise=true , address=145.79.227.48:4302 , piece_id=Q4IJXB2KAPEX3WFQ6W2QEPGQOPZBKVQW6YSRZ2OAKIX33KJO3SGQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.208343717 (708h32m42.079426752s) [noise=true , address=145.79.227.11:4730 , piece_id=T5BLORMXA2XLLNE6IESXGTRXG5Q54NFRXA4TMNYTKDEOSXO3G7IA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.210222815 (708h32m42.08130586s) [noise=true , address=145.79.227.13:4802 , piece_id=GPN7ZUKCDN2OUSHD55SJV7ACG3BYMZLA2HQDXGSJC3JTOZ6JSYYQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.214718616 (708h32m42.085801651s) [noise=true , address=145.79.227.38:4977 , piece_id=SFOF2UCEGAGAMO7SXXPJ3WWP273YCNEUBTF2KEIORAUXG74LJDJA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.222562752 (708h32m42.093645797s) [noise=true , address=145.79.227.47:4265 , piece_id=XTBTZ4NLGYFG3QER2YSF2GCPZG3YJTBHPYLKO4E7DUCNMRJB7OEQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.223289326 (708h32m42.094372371s) [noise=true , address=145.79.227.21:4474 , piece_id=XAS6LZOZCZZ3AO67YATB6FBYTKEF2L6DJZBDTIESHDJR7LBERGAA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.224967150 (708h32m42.096050185s) [noise=true , address=145.79.227.50:4375 , piece_id=OF3MTRXG3E5NVPMYVSK3Y43M6AHIL7ZKMOFXQEW3NHBSINO5KFCA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.243411852 (708h32m42.114494897s) [noise=true , address=145.79.227.37:4566 , piece_id=OIYHEJ377M24GIMMH72OQWJDX3NVT6VOINI2MQPJZ5B47FFQZ5DA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.024848161 (708h32m42.895931206s) [noise=true , address=45.91.116.11:4118 , piece_id=HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.278733412 (708h32m43.149816447s) [noise=true , address=77.90.31.16:4736 , piece_id=UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.650148299 (708h32m43.521231344s) [noise=true , address=77.90.31.15:4697 , piece_id=I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.650174009 (708h32m43.521257064s) [seg_pos={0 0}, upload=620328]: Segment reached optimal threshold of 13 pieces.
Jan 4 04:01:43.650409576 (708h32m43.521492621s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Jan 4 04:01:43.650418003 (708h32m43.521501048s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650514897 (708h32m43.521597932s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Jan 4 04:01:43.650530617 (708h32m43.521613662s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650853832 (708h32m43.521936898s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: context canceled; context canceled
Jan 4 04:01:43.650863831 (708h32m43.521946876s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650898517 (708h32m43.521981562s) [seg_pos={0 0}, upload=620328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 04:01:43.651032182 (708h32m43.522115237s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:01:43.651038885 (708h32m43.52212192s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:01:43.937500978 (708h32m43.808584013s) [upload=620328]: Done uploading segments...
Jan 4 04:01:43.937524522 (708h32m43.808607567s) [upload=620328]: Upload finished. err: <nil>
Duration: 6.84768501s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
35 out of 60 is failed (threshold=3) [XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..XXXX] duration:
1m0.003272848s trace:
https://jaeger.dev.storj.io/trace/53082bcaf27583a0
FAIL (2026-01-04 04:03:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
4I25DBSFW6ENYVADKIT2DGLXBPEKWOK74VE6WMA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.275630350 (708h33m30.146713385s) [noise=true , address=77.90.31.11:4559 , piece_id=H7ZDZF345OX5ILABNQ7KIV6IPS6WCP55TKJ6QQFFZQQFUDV3WTQA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.275874515 (708h33m30.14695756s) [noise=true , address=45.91.116.5:4301 , piece_id=Z5DEYK7Z7G2UAFNPZPU5QV6P2PTA7SMF4DE6M6SI3HCYKAWWJAEA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.275957372 (708h33m30.147040408s) [noise=true , address=45.91.116.6:4982 , piece_id=ZW5QEMWU3LXXQJEB76BV6QEMQAWDA2LT6HOCIWET3OVMDDRONL3A, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.276122437 (708h33m30.147205472s) [noise=true , address=145.79.227.6:4283 , piece_id=NU5V3C3EYC72L25IUO2QI5HMII6KXPK7LJBMPUTM5C6PBUQF6LNA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.276241484 (708h33m30.147324519s) [noise=true , address=145.79.227.40:4079 , piece_id=F5GA3DZH23AZYRTPGUAVLQGW4HMCIMGDGZBFBAQEEDN4VIVAXGSA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.276371161 (708h33m30.147454196s) [noise=true , address=145.79.227.34:4923 , piece_id=PBB3JILXR7DUFUDZ52O4T5FPSKAFMRHBIH4GHZBMNSVKUCEHDQMA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.276486681 (708h33m30.147569726s) [noise=true , address=145.79.227.5:4231 , piece_id=JPSLMK46KN2QP4H4G6UDLQPTQHFTIUVOS4PK66KP24SS2IS4SILA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.276613001 (708h33m30.147696036s) [noise=true , address=193.149.164.143:4700 , piece_id=VQQN3J4CMQANAGPZFOI5Y5JFW5AQTKD4L2E6TZM2A6RPCSEZA5IA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.276736437 (708h33m30.147819472s) [noise=true , address=77.90.31.21:4901 , piece_id=7EQ23H6JDEQEMGDBLETVFB3WAZHDE3AC5MI57ERM4A3L2DXUZY3A, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.276853389 (708h33m30.147936424s) [noise=true , address=45.91.116.14:4209 , piece_id=KBFURY26WTF72ID7MTBJTQPQE6VMNYPPPHUCD4FIUG3EQWMLX66Q, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.276976354 (708h33m30.148059399s) [noise=true , address=64.49.13.169:4715 , piece_id=YMN7IHE5LOGMRDJOIZ2GJW5LDTJPBMA7CZPPPNG5VFDGT3VY7HHQ, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.277115078 (708h33m30.148198113s) [noise=true , address=77.90.31.13:4648 , piece_id=6ADZVQDSB4S2PRGAYZP4ILHMKZINII7DMW7DTFJKB5SDJA3FDHZQ, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:30.277599841 (708h33m30.148682886s) [noise=true , address=145.79.227.29:4961 , piece_id=B2SJZHB3PTRLJHVXUZJMD5QRDJMR7CO2BUHRROHWL6QOT3IFNJEA, seg_pos={0 0}, upload=620336]: Uploading piece...
Jan 4 04:02:31.696151905 (708h33m31.567234951s) [upload=620336]: Next returned nil segment
Jan 4 04:02:31.696171512 (708h33m31.567254557s) [upload=620336]: Waiting for error group managing segments...
Jan 4 04:02:32.129915044 (708h33m32.000998079s) [noise=true , address=145.79.227.5:4232 , piece_id=KJGPRBJGVRE5ISWITF4CYWS2YRLV3MITEO7KKGTOSRFZE2LYD3WA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:32.150744507 (708h33m32.021827542s) [noise=true , address=145.79.227.5:4231 , piece_id=JPSLMK46KN2QP4H4G6UDLQPTQHFTIUVOS4PK66KP24SS2IS4SILA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:32.194680799 (708h33m32.065763844s) [noise=true , address=145.79.227.6:4283 , piece_id=NU5V3C3EYC72L25IUO2QI5HMII6KXPK7LJBMPUTM5C6PBUQF6LNA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:32.575877643 (708h33m32.446960679s) [noise=true , address=77.90.31.21:4901 , piece_id=7EQ23H6JDEQEMGDBLETVFB3WAZHDE3AC5MI57ERM4A3L2DXUZY3A, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:32.967820870 (708h33m32.838903905s) [noise=true , address=193.149.164.143:4694 , piece_id=KOBAR6CECYAUT4I25DBSFW6ENYVADKIT2DGLXBPEKWOK74VE6WMA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.023661440 (708h33m32.894744485s) [noise=true , address=145.79.227.30:4784 , piece_id=IVCD7RQXERZUCKDTT4IN6GVSS2OOICBEHXI4UNOFAYVCI4ALGUJQ, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.070570844 (708h33m32.941653879s) [noise=true , address=145.79.227.29:4961 , piece_id=B2SJZHB3PTRLJHVXUZJMD5QRDJMR7CO2BUHRROHWL6QOT3IFNJEA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.091918804 (708h33m32.963001839s) [noise=true , address=145.79.227.33:4886 , piece_id=OJQ5PYSZPFCGU4KJESZGCF2ZM5WZNWHH2DSADUA7TGQBKCSFNWQA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.195298577 (708h33m33.066381622s) [noise=true , address=145.79.227.34:4923 , piece_id=PBB3JILXR7DUFUDZ52O4T5FPSKAFMRHBIH4GHZBMNSVKUCEHDQMA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.231061054 (708h33m33.10214409s) [noise=true , address=145.79.227.43:4192 , piece_id=HAHUX25DOMWVAD4RPMKYAQRIRKXXWQTADWEPYBPW3CV3P2TX2SIA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.253322423 (708h33m33.124405459s) [noise=true , address=145.79.227.40:4079 , piece_id=F5GA3DZH23AZYRTPGUAVLQGW4HMCIMGDGZBFBAQEEDN4VIVAXGSA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.306067058 (708h33m33.177150093s) [noise=true , address=45.91.116.14:4209 , piece_id=KBFURY26WTF72ID7MTBJTQPQE6VMNYPPPHUCD4FIUG3EQWMLX66Q, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.342923409 (708h33m33.214006454s) [noise=true , address=193.149.164.143:4700 , piece_id=VQQN3J4CMQANAGPZFOI5Y5JFW5AQTKD4L2E6TZM2A6RPCSEZA5IA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.583010797 (708h33m33.454093842s) [noise=true , address=45.91.116.5:4301 , piece_id=Z5DEYK7Z7G2UAFNPZPU5QV6P2PTA7SMF4DE6M6SI3HCYKAWWJAEA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: <nil>
Jan 4 04:02:33.583034422 (708h33m33.454117478s) [seg_pos={0 0}, upload=620336]: Segment reached optimal threshold of 14 pieces.
Jan 4 04:02:33.583221318 (708h33m33.454304363s) [noise=true , address=77.90.31.13:4648 , piece_id=6ADZVQDSB4S2PRGAYZP4ILHMKZINII7DMW7DTFJKB5SDJA3FDHZQ, seg_pos={0 0}, upload=620336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh): protocol: write tcp 188.166.225.170:50930->77.90.31.13:4648: use of closed network connection; write tcp 188.166.225.170:50930->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50930->77.90.31.13:4648: use of closed network connection
Jan 4 04:02:33.583230084 (708h33m33.454313129s) [noise=true , address=77.90.31.13:4648 , piece_id=6ADZVQDSB4S2PRGAYZP4ILHMKZINII7DMW7DTFJKB5SDJA3FDHZQ, seg_pos={0 0}, upload=620336]: Long tail cancellation due to optimal threshold
Jan 4 04:02:33.583360693 (708h33m33.454443738s) [noise=true , address=77.90.31.11:4559 , piece_id=H7ZDZF345OX5ILABNQ7KIV6IPS6WCP55TKJ6QQFFZQQFUDV3WTQA, seg_pos={0 0}, upload=620336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF
Jan 4 04:02:33.583374219 (708h33m33.454457274s) [noise=true , address=77.90.31.11:4559 , piece_id=H7ZDZF345OX5ILABNQ7KIV6IPS6WCP55TKJ6QQFFZQQFUDV3WTQA, seg_pos={0 0}, upload=620336]: Long tail cancellation due to optimal threshold
Jan 4 04:02:33.583391492 (708h33m33.454474537s) [noise=true , address=45.91.116.6:4982 , piece_id=ZW5QEMWU3LXXQJEB76BV6QEMQAWDA2LT6HOCIWET3OVMDDRONL3A, seg_pos={0 0}, upload=620336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: write tcp 188.166.225.170:49050->45.91.116.6:4982: use of closed network connection; write tcp 188.166.225.170:49050->45.91.116.6:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49050->45.91.116.6:4982: use of closed network connection
Jan 4 04:02:33.583398926 (708h33m33.454481971s) [noise=true , address=45.91.116.6:4982 , piece_id=ZW5QEMWU3LXXQJEB76BV6QEMQAWDA2LT6HOCIWET3OVMDDRONL3A, seg_pos={0 0}, upload=620336]: Long tail cancellation due to optimal threshold
Jan 4 04:02:33.583460143 (708h33m33.454543188s) [noise=true , address=64.49.13.169:4715 , piece_id=YMN7IHE5LOGMRDJOIZ2GJW5LDTJPBMA7CZPPPNG5VFDGT3VY7HHQ, seg_pos={0 0}, upload=620336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39244->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39244->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39244->64.49.13.169:4715: use of closed network connection
Jan 4 04:02:33.583468709 (708h33m33.454551744s) [noise=true , address=64.49.13.169:4715 , piece_id=YMN7IHE5LOGMRDJOIZ2GJW5LDTJPBMA7CZPPPNG5VFDGT3VY7HHQ, seg_pos={0 0}, upload=620336]: Long tail cancellation due to optimal threshold
Jan 4 04:02:33.583610439 (708h33m33.454693484s) [seg_pos={0 0}, upload=620336]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 04:02:33.583681955 (708h33m33.454765s) [upload=620336]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:02:33.583686874 (708h33m33.454769919s) [upload=620336]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:02:33.880610507 (708h33m33.751693552s) [upload=620336]: Done uploading segments...
Jan 4 04:02:33.880635836 (708h33m33.751718881s) [upload=620336]: Upload finished. err: <nil>
Duration: 1m0.003299188s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXXX] duration:
5.000624781s trace:
https://jaeger.dev.storj.io/trace/51bf2b07a80a5a9d
FAIL (2026-01-04 04:03:08 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000648627s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [...XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..XX.] duration:
5.73789552s trace:
https://jaeger.dev.storj.io/trace/695820a92d75c5e7
FAIL (2026-01-04 04:02:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA) context deadline exceeded
======LOGS======
Jan 4 04:01:39.947630645 (708h32m39.81871368s) [upload=620328]: Starting upload
Jan 4 04:01:39.947700528 (708h32m39.818783563s) [upload=620328]: Done starting upload
Jan 4 04:01:39.947795158 (708h32m39.818878203s) [upload=620328]: Uploading segments...
Jan 4 04:01:39.947909907 (708h32m39.818992952s) [upload=620328]: Got next segment. Inline: false
Jan 4 04:01:39.947958859 (708h32m39.819041894s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 04:01:39.947964580 (708h32m39.819047625s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 04:01:40.192231657 (708h32m40.063314712s) [seg_pos={0 0}, upload=620328]: Begin upload segment...
Jan 4 04:01:40.192286240 (708h32m40.063369275s) [seg_pos={0 0}, upload=620328]: Done begin upload segment.
Jan 4 04:01:40.192312410 (708h32m40.063395445s) [noise=true , address=145.79.227.48:4302 , piece_id=Q4IJXB2KAPEX3WFQ6W2QEPGQOPZBKVQW6YSRZ2OAKIX33KJO3SGQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.192322890 (708h32m40.063405935s) [noise=true , address=145.79.227.21:4474 , piece_id=XAS6LZOZCZZ3AO67YATB6FBYTKEF2L6DJZBDTIESHDJR7LBERGAA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.192824275 (708h32m40.06390732s) [noise=true , address=45.91.116.11:4118 , piece_id=HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.193532203 (708h32m40.064615238s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.193849126 (708h32m40.064932171s) [noise=true , address=77.90.31.16:4736 , piece_id=UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194049769 (708h32m40.065132804s) [noise=true , address=77.90.31.15:4697 , piece_id=I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194291058 (708h32m40.065374103s) [noise=true , address=145.79.227.13:4802 , piece_id=GPN7ZUKCDN2OUSHD55SJV7ACG3BYMZLA2HQDXGSJC3JTOZ6JSYYQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194384946 (708h32m40.065467981s) [noise=true , address=145.79.227.41:4085 , piece_id=3XCI2KIJ72AONJXPX2N7TWSJVE5BHKSULW6ENWDLU24WCRAKUN3Q, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194661523 (708h32m40.065744558s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.209114475 (708h32m40.08019752s) [noise=true , address=145.79.227.11:4730 , piece_id=T5BLORMXA2XLLNE6IESXGTRXG5Q54NFRXA4TMNYTKDEOSXO3G7IA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.209135624 (708h32m40.080218679s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.210677250 (708h32m40.081760285s) [noise=true , address=145.79.227.3:4151 , piece_id=4KML6THW6OWTYZYSLKCX7JVGVQCMXZZ2LJHYO5XKFKPDT3JH5WMA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.212095761 (708h32m40.083178796s) [noise=true , address=145.79.227.47:4265 , piece_id=XTBTZ4NLGYFG3QER2YSF2GCPZG3YJTBHPYLKO4E7DUCNMRJB7OEQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.213929422 (708h32m40.085012467s) [noise=true , address=145.79.227.50:4375 , piece_id=OF3MTRXG3E5NVPMYVSK3Y43M6AHIL7ZKMOFXQEW3NHBSINO5KFCA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.219098505 (708h32m40.09018154s) [noise=true , address=145.79.227.38:4977 , piece_id=SFOF2UCEGAGAMO7SXXPJ3WWP273YCNEUBTF2KEIORAUXG74LJDJA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.220540901 (708h32m40.091623936s) [noise=true , address=145.79.227.37:4566 , piece_id=OIYHEJ377M24GIMMH72OQWJDX3NVT6VOINI2MQPJZ5B47FFQZ5DA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:41.810366911 (708h32m41.681449956s) [upload=620328]: Next returned nil segment
Jan 4 04:01:41.810384545 (708h32m41.68146758s) [upload=620328]: Waiting for error group managing segments...
Jan 4 04:01:42.195975145 (708h32m42.06705819s) [noise=true , address=145.79.227.3:4151 , piece_id=4KML6THW6OWTYZYSLKCX7JVGVQCMXZZ2LJHYO5XKFKPDT3JH5WMA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.200965087 (708h32m42.072048122s) [noise=true , address=145.79.227.41:4085 , piece_id=3XCI2KIJ72AONJXPX2N7TWSJVE5BHKSULW6ENWDLU24WCRAKUN3Q, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.202003935 (708h32m42.07308697s) [noise=true , address=145.79.227.48:4302 , piece_id=Q4IJXB2KAPEX3WFQ6W2QEPGQOPZBKVQW6YSRZ2OAKIX33KJO3SGQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.208343717 (708h32m42.079426752s) [noise=true , address=145.79.227.11:4730 , piece_id=T5BLORMXA2XLLNE6IESXGTRXG5Q54NFRXA4TMNYTKDEOSXO3G7IA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.210222815 (708h32m42.08130586s) [noise=true , address=145.79.227.13:4802 , piece_id=GPN7ZUKCDN2OUSHD55SJV7ACG3BYMZLA2HQDXGSJC3JTOZ6JSYYQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.214718616 (708h32m42.085801651s) [noise=true , address=145.79.227.38:4977 , piece_id=SFOF2UCEGAGAMO7SXXPJ3WWP273YCNEUBTF2KEIORAUXG74LJDJA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.222562752 (708h32m42.093645797s) [noise=true , address=145.79.227.47:4265 , piece_id=XTBTZ4NLGYFG3QER2YSF2GCPZG3YJTBHPYLKO4E7DUCNMRJB7OEQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.223289326 (708h32m42.094372371s) [noise=true , address=145.79.227.21:4474 , piece_id=XAS6LZOZCZZ3AO67YATB6FBYTKEF2L6DJZBDTIESHDJR7LBERGAA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.224967150 (708h32m42.096050185s) [noise=true , address=145.79.227.50:4375 , piece_id=OF3MTRXG3E5NVPMYVSK3Y43M6AHIL7ZKMOFXQEW3NHBSINO5KFCA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.243411852 (708h32m42.114494897s) [noise=true , address=145.79.227.37:4566 , piece_id=OIYHEJ377M24GIMMH72OQWJDX3NVT6VOINI2MQPJZ5B47FFQZ5DA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.024848161 (708h32m42.895931206s) [noise=true , address=45.91.116.11:4118 , piece_id=HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.278733412 (708h32m43.149816447s) [noise=true , address=77.90.31.16:4736 , piece_id=UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.650148299 (708h32m43.521231344s) [noise=true , address=77.90.31.15:4697 , piece_id=I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.650174009 (708h32m43.521257064s) [seg_pos={0 0}, upload=620328]: Segment reached optimal threshold of 13 pieces.
Jan 4 04:01:43.650409576 (708h32m43.521492621s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Jan 4 04:01:43.650418003 (708h32m43.521501048s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650514897 (708h32m43.521597932s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Jan 4 04:01:43.650530617 (708h32m43.521613662s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650853832 (708h32m43.521936898s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: context canceled; context canceled
Jan 4 04:01:43.650863831 (708h32m43.521946876s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650898517 (708h32m43.521981562s) [seg_pos={0 0}, upload=620328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 04:01:43.651032182 (708h32m43.522115237s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:01:43.651038885 (708h32m43.52212192s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:01:43.937500978 (708h32m43.808584013s) [upload=620328]: Done uploading segments...
Jan 4 04:01:43.937524522 (708h32m43.808607567s) [upload=620328]: Upload finished. err: <nil>
Duration: 5.737927351s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXXX] duration:
5.001410968s trace:
https://jaeger.dev.storj.io/trace/5091707c57562fe3
FAIL (2026-01-04 04:02:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001429131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [.XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..XXX] duration:
1m0.005311115s trace:
https://jaeger.dev.storj.io/trace/751469b455ccd6da
FAIL (2026-01-04 04:02:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
F776EVWBDEA3IAC47LUJ43BO6ZZQ, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.261154851 (708h32m20.132237896s) [noise=true , address=145.79.227.3:4179 , piece_id=RMQYSZ2JYRGPTO7ORDLNVYS62PE2BHUVSWGOVXCLCOAZ4EI2J6TQ, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.261678849 (708h32m20.132761894s) [noise=true , address=145.79.227.49:4347 , piece_id=SDU665JQSUBXAJPTCDAPT7R3YH5LSVLOYFQOPQECA6CE5NOBP3SQ, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.262152571 (708h32m20.133235616s) [noise=true , address=193.149.164.143:4703 , piece_id=2YEPFQDYUDWOTE2TGXVR5MH3ZMTEOAA5SIUVEXB5FA34COCLZITA, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.262649798 (708h32m20.133732833s) [noise=true , address=64.49.13.170:4725 , piece_id=F4EE5IWSYX747URLNBKIB5WSEDOR7EIYSS3J6RQBHRA2FVBX7VVA, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.266062606 (708h32m20.137145641s) [noise=true , address=145.79.227.34:4919 , piece_id=UZ4QXYQYMSXBLIWDPBG5A6LMVNJFDOFHTPPIUHHCTFFAJMDTB6DQ, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.267238716 (708h32m20.138321761s) [noise=true , address=145.79.227.10:4425 , piece_id=SKDGF7BNI4J3PMBTT6EOHVRJBRCF53F5GOPQBPXNLSOLKOANARBQ, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.268628392 (708h32m20.139711427s) [noise=true , address=77.90.31.18:4826 , piece_id=BRTOFJWDMZRSQ7GZN563S75F4QD5DTLJPAB2AE47Y5TD7JJUXWIA, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.270086840 (708h32m20.141169885s) [noise=true , address=145.79.227.52:4452 , piece_id=SR4WN4JELNLHLCT3M33YJBTIRQS6NECLWIE4FK4X2MUXT7AIFJRA, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.271552150 (708h32m20.142635185s) [noise=true , address=145.79.227.43:4179 , piece_id=7U6AHAWOTMUD5EZSC7O2UFUQ4754W3QRDEVUFKMAZ7QO7TTY6L5A, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.273031627 (708h32m20.144114672s) [noise=true , address=145.79.227.8:4343 , piece_id=J5IR6SW7NJIKJQ4LS7DEBOJGLJN3ZQEFK67VR3AYHIQ3FEWMIDPQ, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.274565528 (708h32m20.145648563s) [noise=true , address=64.49.13.169:4710 , piece_id=OYH47BDVWBVWLVACD7ZCJ5DP6RYWGUR4IIGS2RHZ2LKUNIBK5MAA, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.277226555 (708h32m20.1483096s) [noise=true , address=145.79.227.44:4198 , piece_id=A6SKDNOKN6LO3TQWQWJTMPKJSQHLN6A3CJQ3A5OD3QEVY23R64HQ, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:20.279776811 (708h32m20.150859856s) [noise=true , address=45.91.116.5:4327 , piece_id=P5TNB5UTRE4G2XCQXTH23PPCL5ZUWPFLY3GXXCNHENBVCJDCTCDQ, seg_pos={0 0}, upload=620321]: Uploading piece...
Jan 4 04:01:21.706575028 (708h32m21.577658063s) [upload=620321]: Next returned nil segment
Jan 4 04:01:21.706588243 (708h32m21.577671288s) [upload=620321]: Waiting for error group managing segments...
Jan 4 04:01:22.111361120 (708h32m21.982444165s) [noise=true , address=64.49.13.170:4725 , piece_id=F4EE5IWSYX747URLNBKIB5WSEDOR7EIYSS3J6RQBHRA2FVBX7VVA, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:22.131926290 (708h32m22.003009335s) [noise=true , address=145.79.227.3:4179 , piece_id=RMQYSZ2JYRGPTO7ORDLNVYS62PE2BHUVSWGOVXCLCOAZ4EI2J6TQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:22.180662131 (708h32m22.051745166s) [noise=true , address=145.79.227.52:4452 , piece_id=SR4WN4JELNLHLCT3M33YJBTIRQS6NECLWIE4FK4X2MUXT7AIFJRA, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:22.490400866 (708h32m22.361483911s) [noise=true , address=77.90.31.14:4688 , piece_id=4QKEFLZFHZ4Q3YD5RJSIECL3RQRETTE5XFHFS4OAFHHPU4FULAPQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:22.491680743 (708h32m22.362763778s) [noise=true , address=77.90.31.13:4654 , piece_id=RP3KSSVPIJJKJC4G25N5QZEHF776EVWBDEA3IAC47LUJ43BO6ZZQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:22.578674173 (708h32m22.449757208s) [noise=true , address=145.79.227.45:4264 , piece_id=GWFKP6A7GY4B7NQR7XUNL5FTIN2TM6QQKJC5NS3W257JGH55ZFUA, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:22.762092744 (708h32m22.633175789s) [noise=true , address=77.90.31.18:4826 , piece_id=BRTOFJWDMZRSQ7GZN563S75F4QD5DTLJPAB2AE47Y5TD7JJUXWIA, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:23.009589690 (708h32m22.880672725s) [noise=true , address=145.79.227.49:4347 , piece_id=SDU665JQSUBXAJPTCDAPT7R3YH5LSVLOYFQOPQECA6CE5NOBP3SQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:23.022412286 (708h32m22.893495321s) [noise=true , address=145.79.227.43:4179 , piece_id=7U6AHAWOTMUD5EZSC7O2UFUQ4754W3QRDEVUFKMAZ7QO7TTY6L5A, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:23.071586162 (708h32m22.942669197s) [noise=true , address=145.79.227.30:4794 , piece_id=UFLX4YUIX3L2FGO7C5GZ37NPZLKRFHWSMS3CQ4XHJNSEXFWB3HBQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:23.127725911 (708h32m22.998808946s) [noise=true , address=145.79.227.34:4919 , piece_id=UZ4QXYQYMSXBLIWDPBG5A6LMVNJFDOFHTPPIUHHCTFFAJMDTB6DQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:23.173618599 (708h32m23.044701634s) [noise=true , address=145.79.227.44:4198 , piece_id=A6SKDNOKN6LO3TQWQWJTMPKJSQHLN6A3CJQ3A5OD3QEVY23R64HQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:23.334341363 (708h32m23.205424398s) [noise=true , address=193.149.164.143:4703 , piece_id=2YEPFQDYUDWOTE2TGXVR5MH3ZMTEOAA5SIUVEXB5FA34COCLZITA, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:23.676884573 (708h32m23.547967608s) [noise=true , address=145.79.227.40:4052 , piece_id=E3ZLYY4GEJTTAQX3DSCRXRBCWVTWNS6R7AP5ZX4Q6ZKZPIPV6J3Q, seg_pos={0 0}, upload=620321]: Done uploading piece. err: <nil>
Jan 4 04:01:23.676906736 (708h32m23.547989781s) [seg_pos={0 0}, upload=620321]: Segment reached optimal threshold of 14 pieces.
Jan 4 04:01:23.677106355 (708h32m23.5481894s) [noise=true , address=45.91.116.5:4327 , piece_id=P5TNB5UTRE4G2XCQXTH23PPCL5ZUWPFLY3GXXCNHENBVCJDCTCDQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: expected piece hash; context canceled; EOF
Jan 4 04:01:23.677113550 (708h32m23.548196585s) [noise=true , address=45.91.116.5:4327 , piece_id=P5TNB5UTRE4G2XCQXTH23PPCL5ZUWPFLY3GXXCNHENBVCJDCTCDQ, seg_pos={0 0}, upload=620321]: Long tail cancellation due to optimal threshold
Jan 4 04:01:23.677404413 (708h32m23.548487448s) [noise=true , address=145.79.227.8:4343 , piece_id=J5IR6SW7NJIKJQ4LS7DEBOJGLJN3ZQEFK67VR3AYHIQ3FEWMIDPQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9phoqjnKRLEuWqACueioVzs3kYPd1wZx9JAizQLdwzaR3bCw): protocol: write tcp 188.166.225.170:54412->145.79.227.8:4343: use of closed network connection; write tcp 188.166.225.170:54412->145.79.227.8:4343: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54412->145.79.227.8:4343: use of closed network connection
Jan 4 04:01:23.677411276 (708h32m23.548494311s) [noise=true , address=145.79.227.8:4343 , piece_id=J5IR6SW7NJIKJQ4LS7DEBOJGLJN3ZQEFK67VR3AYHIQ3FEWMIDPQ, seg_pos={0 0}, upload=620321]: Long tail cancellation due to optimal threshold
Jan 4 04:01:23.677505315 (708h32m23.548588361s) [noise=true , address=145.79.227.10:4425 , piece_id=SKDGF7BNI4J3PMBTT6EOHVRJBRCF53F5GOPQBPXNLSOLKOANARBQ, seg_pos={0 0}, upload=620321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD): protocol: write tcp 188.166.225.170:52274->145.79.227.10:4425: use of closed network connection; write tcp 188.166.225.170:52274->145.79.227.10:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52274->145.79.227.10:4425: use of closed network connection
Jan 4 04:01:23.677511277 (708h32m23.548594322s) [noise=true , address=145.79.227.10:4425 , piece_id=SKDGF7BNI4J3PMBTT6EOHVRJBRCF53F5GOPQBPXNLSOLKOANARBQ, seg_pos={0 0}, upload=620321]: Long tail cancellation due to optimal threshold
Jan 4 04:01:23.677601980 (708h32m23.548685025s) [noise=true , address=64.49.13.169:4710 , piece_id=OYH47BDVWBVWLVACD7ZCJ5DP6RYWGUR4IIGS2RHZ2LKUNIBK5MAA, seg_pos={0 0}, upload=620321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55438->64.49.13.169:4710: use of closed network connection
Jan 4 04:01:23.677647236 (708h32m23.548730271s) [noise=true , address=64.49.13.169:4710 , piece_id=OYH47BDVWBVWLVACD7ZCJ5DP6RYWGUR4IIGS2RHZ2LKUNIBK5MAA, seg_pos={0 0}, upload=620321]: Long tail cancellation due to optimal threshold
Jan 4 04:01:23.677664889 (708h32m23.548747924s) [seg_pos={0 0}, upload=620321]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 04:01:23.677891912 (708h32m23.548974947s) [upload=620321]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:01:23.677898524 (708h32m23.548981559s) [upload=620321]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:01:23.968972278 (708h32m23.840055313s) [upload=620321]: Done uploading segments...
Jan 4 04:01:23.968990472 (708h32m23.840073507s) [upload=620321]: Upload finished. err: <nil>
Duration: 1m0.005327956s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X...XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..XX] duration:
30.002523246s trace:
https://jaeger.dev.storj.io/trace/695820a92d75c5e7
FAIL (2026-01-04 04:02:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA) context deadline exceeded
======LOGS======
Jan 4 04:01:39.947630645 (708h32m39.81871368s) [upload=620328]: Starting upload
Jan 4 04:01:39.947700528 (708h32m39.818783563s) [upload=620328]: Done starting upload
Jan 4 04:01:39.947795158 (708h32m39.818878203s) [upload=620328]: Uploading segments...
Jan 4 04:01:39.947909907 (708h32m39.818992952s) [upload=620328]: Got next segment. Inline: false
Jan 4 04:01:39.947958859 (708h32m39.819041894s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 04:01:39.947964580 (708h32m39.819047625s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 04:01:40.192231657 (708h32m40.063314712s) [seg_pos={0 0}, upload=620328]: Begin upload segment...
Jan 4 04:01:40.192286240 (708h32m40.063369275s) [seg_pos={0 0}, upload=620328]: Done begin upload segment.
Jan 4 04:01:40.192312410 (708h32m40.063395445s) [noise=true , address=145.79.227.48:4302 , piece_id=Q4IJXB2KAPEX3WFQ6W2QEPGQOPZBKVQW6YSRZ2OAKIX33KJO3SGQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.192322890 (708h32m40.063405935s) [noise=true , address=145.79.227.21:4474 , piece_id=XAS6LZOZCZZ3AO67YATB6FBYTKEF2L6DJZBDTIESHDJR7LBERGAA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.192824275 (708h32m40.06390732s) [noise=true , address=45.91.116.11:4118 , piece_id=HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.193532203 (708h32m40.064615238s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.193849126 (708h32m40.064932171s) [noise=true , address=77.90.31.16:4736 , piece_id=UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194049769 (708h32m40.065132804s) [noise=true , address=77.90.31.15:4697 , piece_id=I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194291058 (708h32m40.065374103s) [noise=true , address=145.79.227.13:4802 , piece_id=GPN7ZUKCDN2OUSHD55SJV7ACG3BYMZLA2HQDXGSJC3JTOZ6JSYYQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194384946 (708h32m40.065467981s) [noise=true , address=145.79.227.41:4085 , piece_id=3XCI2KIJ72AONJXPX2N7TWSJVE5BHKSULW6ENWDLU24WCRAKUN3Q, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.194661523 (708h32m40.065744558s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.209114475 (708h32m40.08019752s) [noise=true , address=145.79.227.11:4730 , piece_id=T5BLORMXA2XLLNE6IESXGTRXG5Q54NFRXA4TMNYTKDEOSXO3G7IA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.209135624 (708h32m40.080218679s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.210677250 (708h32m40.081760285s) [noise=true , address=145.79.227.3:4151 , piece_id=4KML6THW6OWTYZYSLKCX7JVGVQCMXZZ2LJHYO5XKFKPDT3JH5WMA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.212095761 (708h32m40.083178796s) [noise=true , address=145.79.227.47:4265 , piece_id=XTBTZ4NLGYFG3QER2YSF2GCPZG3YJTBHPYLKO4E7DUCNMRJB7OEQ, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.213929422 (708h32m40.085012467s) [noise=true , address=145.79.227.50:4375 , piece_id=OF3MTRXG3E5NVPMYVSK3Y43M6AHIL7ZKMOFXQEW3NHBSINO5KFCA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.219098505 (708h32m40.09018154s) [noise=true , address=145.79.227.38:4977 , piece_id=SFOF2UCEGAGAMO7SXXPJ3WWP273YCNEUBTF2KEIORAUXG74LJDJA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:40.220540901 (708h32m40.091623936s) [noise=true , address=145.79.227.37:4566 , piece_id=OIYHEJ377M24GIMMH72OQWJDX3NVT6VOINI2MQPJZ5B47FFQZ5DA, seg_pos={0 0}, upload=620328]: Uploading piece...
Jan 4 04:01:41.810366911 (708h32m41.681449956s) [upload=620328]: Next returned nil segment
Jan 4 04:01:41.810384545 (708h32m41.68146758s) [upload=620328]: Waiting for error group managing segments...
Jan 4 04:01:42.195975145 (708h32m42.06705819s) [noise=true , address=145.79.227.3:4151 , piece_id=4KML6THW6OWTYZYSLKCX7JVGVQCMXZZ2LJHYO5XKFKPDT3JH5WMA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.200965087 (708h32m42.072048122s) [noise=true , address=145.79.227.41:4085 , piece_id=3XCI2KIJ72AONJXPX2N7TWSJVE5BHKSULW6ENWDLU24WCRAKUN3Q, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.202003935 (708h32m42.07308697s) [noise=true , address=145.79.227.48:4302 , piece_id=Q4IJXB2KAPEX3WFQ6W2QEPGQOPZBKVQW6YSRZ2OAKIX33KJO3SGQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.208343717 (708h32m42.079426752s) [noise=true , address=145.79.227.11:4730 , piece_id=T5BLORMXA2XLLNE6IESXGTRXG5Q54NFRXA4TMNYTKDEOSXO3G7IA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.210222815 (708h32m42.08130586s) [noise=true , address=145.79.227.13:4802 , piece_id=GPN7ZUKCDN2OUSHD55SJV7ACG3BYMZLA2HQDXGSJC3JTOZ6JSYYQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.214718616 (708h32m42.085801651s) [noise=true , address=145.79.227.38:4977 , piece_id=SFOF2UCEGAGAMO7SXXPJ3WWP273YCNEUBTF2KEIORAUXG74LJDJA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.222562752 (708h32m42.093645797s) [noise=true , address=145.79.227.47:4265 , piece_id=XTBTZ4NLGYFG3QER2YSF2GCPZG3YJTBHPYLKO4E7DUCNMRJB7OEQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.223289326 (708h32m42.094372371s) [noise=true , address=145.79.227.21:4474 , piece_id=XAS6LZOZCZZ3AO67YATB6FBYTKEF2L6DJZBDTIESHDJR7LBERGAA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.224967150 (708h32m42.096050185s) [noise=true , address=145.79.227.50:4375 , piece_id=OF3MTRXG3E5NVPMYVSK3Y43M6AHIL7ZKMOFXQEW3NHBSINO5KFCA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:42.243411852 (708h32m42.114494897s) [noise=true , address=145.79.227.37:4566 , piece_id=OIYHEJ377M24GIMMH72OQWJDX3NVT6VOINI2MQPJZ5B47FFQZ5DA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.024848161 (708h32m42.895931206s) [noise=true , address=45.91.116.11:4118 , piece_id=HDJROFJ6GXQRBD6SUNSZHJOI6MPG7KI3ORSQXFMYZTR2PRCZGBFA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.278733412 (708h32m43.149816447s) [noise=true , address=77.90.31.16:4736 , piece_id=UGM7YEDASB7KXVDJWTNSKRFSTDJCPPIGEFLMS2OYWNIGACR5F2TA, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.650148299 (708h32m43.521231344s) [noise=true , address=77.90.31.15:4697 , piece_id=I3UC4GERLHEYHHZUEXHGVXTY4PBW6CIIS6WNRMVDA7UIFILJJSUQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: <nil>
Jan 4 04:01:43.650174009 (708h32m43.521257064s) [seg_pos={0 0}, upload=620328]: Segment reached optimal threshold of 13 pieces.
Jan 4 04:01:43.650409576 (708h32m43.521492621s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Jan 4 04:01:43.650418003 (708h32m43.521501048s) [noise=true , address=145.79.227.43:4158 , piece_id=27Y7QJKCA4FXOJ4RLC5HQTMBDJRQGVAWUGRZQLEXQ5TKYGWIN7SQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650514897 (708h32m43.521597932s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Jan 4 04:01:43.650530617 (708h32m43.521613662s) [noise=true , address=77.90.31.14:4663 , piece_id=SG2CMRPV267ASTRED3XUN6S66KRGPNNTRESGMF766EVNYJY7CKLQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650853832 (708h32m43.521936898s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: context canceled; context canceled
Jan 4 04:01:43.650863831 (708h32m43.521946876s) [noise=true , address=45.91.116.5:4300 , piece_id=7XGY5O7FPWZZXTG7JKN6DN3O7LE4XGNUOFIUEIFIU4LZIWVZY4SQ, seg_pos={0 0}, upload=620328]: Long tail cancellation due to optimal threshold
Jan 4 04:01:43.650898517 (708h32m43.521981562s) [seg_pos={0 0}, upload=620328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 04:01:43.651032182 (708h32m43.522115237s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:01:43.651038885 (708h32m43.52212192s) [upload=620328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:01:43.937500978 (708h32m43.808584013s) [upload=620328]: Done uploading segments...
Jan 4 04:01:43.937524522 (708h32m43.808607567s) [upload=620328]: Upload finished. err: <nil>
Duration: 30.002544817s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXXX] duration:
5.000833888s trace:
https://jaeger.dev.storj.io/trace/3aa83078ea39f694
FAIL (2026-01-04 04:01:50 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000858425s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [X.XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..XX] duration:
1m0.00276728s trace:
https://jaeger.dev.storj.io/trace/6adb4d67dd40b2f8
FAIL (2026-01-04 04:01:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, Piece ID: DXF2BZTHUKDQ7ILZYDH3OGK3TORP7WQ3L4ZCV4F6TUIGQ2XCI3WA) context deadline exceeded; piecestore: (Node ID: 12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, Piece ID: LTZTJNDF3HKKKK4DSSTMDHWFFB2NIL6NS6N5PUR7KBBHQWLZCUUQ) context deadline exceeded; piecestore: (Node ID: 194doyGN13Ha8hsyjrERgTZqdyzFhdTQyiTGQg9rFA7CxvKmkM, Piece ID: IXZRFE6RU5253WBZ6B4UHTZ62PCQVHJDDEDRYP7GZHNGAXKY3JSQ) context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.8:4072 , piece_id=DBX7BMPF2BEVVL4NMJCGDNORMK2OB7WVXMASBVKEZUIJ55VF32YQ, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.043306681 (708h31m19.914389726s) [noise=true , address=145.79.227.43:4182 , piece_id=H7ELGS4AKSUNWLTIVSWFSLEXC6GBHPAVBPQMDY6XICE4Q2X7VDLQ, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.043690913 (708h31m19.914773948s) [noise=true , address=145.79.227.33:4878 , piece_id=NE4MNSX6P5JCRG6D4COYA6OPX6DBC7LS7ZZBKMS5NXZBOYKF6D7A, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.045066152 (708h31m19.916149197s) [noise=true , address=145.79.227.40:4081 , piece_id=OHRBYTFCNOVW2HT5VA2BAWAM67IGQ7PGULEQN44GFM2YB2MEMMLA, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.045321548 (708h31m19.916404583s) [noise=true , address=145.79.227.43:4173 , piece_id=HSRKXOYDY7XCROYNB3L3GPJ326PIHAZ5Q6QHBM3HKAN4OCJNR75A, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.046615762 (708h31m19.917698807s) [noise=true , address=193.149.164.143:4699 , piece_id=GSKE5XMTVDMPFJL6UCSRE5EK4T442BKFEL5UJEWGPWB7MWHUECYA, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.047126505 (708h31m19.91820954s) [noise=true , address=77.90.31.18:4826 , piece_id=DXF2BZTHUKDQ7ILZYDH3OGK3TORP7WQ3L4ZCV4F6TUIGQ2XCI3WA, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.048320659 (708h31m19.919403704s) [noise=true , address=145.79.227.7:4323 , piece_id=5BLY5I63Y3FSTJPJOIVAHBQX2XJGOCYKHBVZF5UCV5RFFQYPFB4A, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.048555055 (708h31m19.9196381s) [noise=true , address=145.79.227.49:4354 , piece_id=5DHUMKFWAXUF5G7QURTDQVU44MJ3D23IG5OJGR2KE4OIS67MX6VA, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.050153239 (708h31m19.921236274s) [noise=true , address=145.79.227.29:4975 , piece_id=LHUTYSRP7IPFDZ3U2V6R6DMFX2QJKF34SOV2DCQWNOYEK2C2WZEA, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.050258399 (708h31m19.921341434s) [noise=true , address=77.90.31.17:4778 , piece_id=WGB73QH4ZYTFFNILAERFJOK44VB55BMX7X4T6BO44B6MMJI2NSOQ, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.051722827 (708h31m19.922805862s) [noise=true , address=45.91.116.7:4018 , piece_id=LTZTJNDF3HKKKK4DSSTMDHWFFB2NIL6NS6N5PUR7KBBHQWLZCUUQ, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.051806987 (708h31m19.922890042s) [noise=true , address=45.91.116.11:4135 , piece_id=IXZRFE6RU5253WBZ6B4UHTZ62PCQVHJDDEDRYP7GZHNGAXKY3JSQ, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.053501083 (708h31m19.924584128s) [noise=true , address=145.79.227.35:4504 , piece_id=MT3ZOSRYCJSVWIFJN6QKVQGMYPJQQQN22IUNI7BDKQQLWZH3MMLQ, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.058080252 (708h31m19.929163327s) [noise=true , address=145.79.227.44:4223 , piece_id=OCDAFWBE3PV2SUL3CX5WOY2PDNSLMMTBTZHETRJWGZZMQZAY72GQ, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:20.060405620 (708h31m19.931488665s) [noise=true , address=64.49.13.169:4706 , piece_id=6VMBNLKCTQO6KNDSEGIMBEZROCVXUMLU6KHT5USPKA7BEE4JEDMA, seg_pos={0 0}, upload=620308]: Uploading piece...
Jan 4 04:00:21.452376447 (708h31m21.323459492s) [upload=620308]: Next returned nil segment
Jan 4 04:00:21.452393029 (708h31m21.323476074s) [upload=620308]: Waiting for error group managing segments...
Jan 4 04:00:21.947230860 (708h31m21.818313905s) [noise=true , address=145.79.227.35:4504 , piece_id=MT3ZOSRYCJSVWIFJN6QKVQGMYPJQQQN22IUNI7BDKQQLWZH3MMLQ, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:21.959261919 (708h31m21.830345084s) [noise=true , address=145.79.227.7:4323 , piece_id=5BLY5I63Y3FSTJPJOIVAHBQX2XJGOCYKHBVZF5UCV5RFFQYPFB4A, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:21.962352654 (708h31m21.833435699s) [noise=true , address=145.79.227.32:4838 , piece_id=M4XJ2XWHHXUEYDRFEXHHRG4MXZLILW76SYP4VGMKLYOZ5HQ4GABQ, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.010444909 (708h31m21.881527954s) [noise=true , address=145.79.227.29:4975 , piece_id=LHUTYSRP7IPFDZ3U2V6R6DMFX2QJKF34SOV2DCQWNOYEK2C2WZEA, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.360390474 (708h31m22.231473519s) [noise=true , address=77.90.31.17:4778 , piece_id=WGB73QH4ZYTFFNILAERFJOK44VB55BMX7X4T6BO44B6MMJI2NSOQ, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.452819382 (708h31m22.323902427s) [noise=true , address=45.91.116.11:4135 , piece_id=IXZRFE6RU5253WBZ6B4UHTZ62PCQVHJDDEDRYP7GZHNGAXKY3JSQ, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.483714179 (708h31m22.354797225s) [noise=true , address=45.91.116.7:4018 , piece_id=LTZTJNDF3HKKKK4DSSTMDHWFFB2NIL6NS6N5PUR7KBBHQWLZCUUQ, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.618874135 (708h31m22.48995718s) [noise=true , address=77.90.31.18:4826 , piece_id=DXF2BZTHUKDQ7ILZYDH3OGK3TORP7WQ3L4ZCV4F6TUIGQ2XCI3WA, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.771014746 (708h31m22.642097791s) [noise=true , address=145.79.227.44:4223 , piece_id=OCDAFWBE3PV2SUL3CX5WOY2PDNSLMMTBTZHETRJWGZZMQZAY72GQ, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.862949937 (708h31m22.734032972s) [noise=true , address=145.79.227.43:4173 , piece_id=HSRKXOYDY7XCROYNB3L3GPJ326PIHAZ5Q6QHBM3HKAN4OCJNR75A, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.868221565 (708h31m22.73930461s) [noise=true , address=145.79.227.33:4878 , piece_id=NE4MNSX6P5JCRG6D4COYA6OPX6DBC7LS7ZZBKMS5NXZBOYKF6D7A, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.903033313 (708h31m22.774116358s) [noise=true , address=145.79.227.43:4182 , piece_id=H7ELGS4AKSUNWLTIVSWFSLEXC6GBHPAVBPQMDY6XICE4Q2X7VDLQ, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:22.911531224 (708h31m22.782614269s) [noise=true , address=145.79.227.30:4767 , piece_id=66PXKB7PBZJ6PMF7YCOFKM2HD4TAXVCYH4PPH2ABBVV5Z5ASJHDA, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:23.033400864 (708h31m22.904483899s) [noise=true , address=145.79.227.49:4354 , piece_id=5DHUMKFWAXUF5G7QURTDQVU44MJ3D23IG5OJGR2KE4OIS67MX6VA, seg_pos={0 0}, upload=620308]: Done uploading piece. err: <nil>
Jan 4 04:00:23.033557793 (708h31m22.904640869s) [seg_pos={0 0}, upload=620308]: Segment reached optimal threshold of 14 pieces.
Jan 4 04:00:23.033993302 (708h31m22.905076347s) [noise=true , address=193.149.164.143:4699 , piece_id=GSKE5XMTVDMPFJL6UCSRE5EK4T442BKFEL5UJEWGPWB7MWHUECYA, seg_pos={0 0}, upload=620308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF
Jan 4 04:00:23.034005566 (708h31m22.905088601s) [noise=true , address=193.149.164.143:4699 , piece_id=GSKE5XMTVDMPFJL6UCSRE5EK4T442BKFEL5UJEWGPWB7MWHUECYA, seg_pos={0 0}, upload=620308]: Long tail cancellation due to optimal threshold
Jan 4 04:00:23.034162725 (708h31m22.90524577s) [noise=true , address=64.49.13.169:4706 , piece_id=6VMBNLKCTQO6KNDSEGIMBEZROCVXUMLU6KHT5USPKA7BEE4JEDMA, seg_pos={0 0}, upload=620308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:58308->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:58308->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58308->64.49.13.169:4706: use of closed network connection
Jan 4 04:00:23.034169529 (708h31m22.905252574s) [noise=true , address=64.49.13.169:4706 , piece_id=6VMBNLKCTQO6KNDSEGIMBEZROCVXUMLU6KHT5USPKA7BEE4JEDMA, seg_pos={0 0}, upload=620308]: Long tail cancellation due to optimal threshold
Jan 4 04:00:23.034334773 (708h31m22.905417818s) [noise=true , address=45.91.116.8:4072 , piece_id=DBX7BMPF2BEVVL4NMJCGDNORMK2OB7WVXMASBVKEZUIJ55VF32YQ, seg_pos={0 0}, upload=620308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1toupLRE5KrBcmqoVvKng6HZeybkwSzzrzwzqH3ZNukQEuEZWu): protocol: expected piece hash; context canceled; EOF
Jan 4 04:00:23.034344522 (708h31m22.905427567s) [noise=true , address=45.91.116.8:4072 , piece_id=DBX7BMPF2BEVVL4NMJCGDNORMK2OB7WVXMASBVKEZUIJ55VF32YQ, seg_pos={0 0}, upload=620308]: Long tail cancellation due to optimal threshold
Jan 4 04:00:23.034806241 (708h31m22.905889306s) [noise=true , address=145.79.227.40:4081 , piece_id=OHRBYTFCNOVW2HT5VA2BAWAM67IGQ7PGULEQN44GFM2YB2MEMMLA, seg_pos={0 0}, upload=620308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF): protocol: expected piece hash; context canceled; EOF
Jan 4 04:00:23.034828462 (708h31m22.905911518s) [noise=true , address=145.79.227.40:4081 , piece_id=OHRBYTFCNOVW2HT5VA2BAWAM67IGQ7PGULEQN44GFM2YB2MEMMLA, seg_pos={0 0}, upload=620308]: Long tail cancellation due to optimal threshold
Jan 4 04:00:23.034889699 (708h31m22.905972744s) [seg_pos={0 0}, upload=620308]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 04:00:23.035005560 (708h31m22.906088595s) [upload=620308]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:00:23.035014227 (708h31m22.906097272s) [upload=620308]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:00:23.326600787 (708h31m23.197683832s) [upload=620308]: Done uploading segments...
Jan 4 04:00:23.326621697 (708h31m23.197704742s) [upload=620308]: Upload finished. err: <nil>
Duration: 1m0.002779183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [XX...XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..X] duration:
30.003208339s trace:
https://jaeger.dev.storj.io/trace/4fce773d3b1db74d
FAIL (2026-01-04 04:01:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
08h31m35.801041388s) [upload=620312]: Done starting upload
Jan 4 04:00:35.930117867 (708h31m35.801200902s) [upload=620312]: Uploading segments...
Jan 4 04:00:35.930338316 (708h31m35.801421351s) [upload=620312]: Got next segment. Inline: false
Jan 4 04:00:35.930382200 (708h31m35.801465245s) [upload=620312]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 04:00:35.930388060 (708h31m35.801471106s) [upload=620312]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 04:00:36.176212252 (708h31m36.047295298s) [seg_pos={0 0}, upload=620312]: Begin upload segment...
Jan 4 04:00:36.176271516 (708h31m36.047354561s) [seg_pos={0 0}, upload=620312]: Done begin upload segment.
Jan 4 04:00:36.176310139 (708h31m36.047393174s) [noise=true , address=45.91.116.13:4156 , piece_id=EVVXHP7YGFZRVHQ3M53MKDM7BAX6XY44Z7C4X3L2BGJYNR3MZCRQ, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.176370574 (708h31m36.047453619s) [noise=true , address=45.91.116.5:4298 , piece_id=ACYFG4JH7UEZCMJZDNKPSV2AZP7VCD6PUZL2YIO77KNDP62BHSOQ, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.176895614 (708h31m36.047978659s) [noise=true , address=145.79.227.4:4186 , piece_id=RML7IV5QG77JFU664Y7K3WD6V5A7EUGD7HGGYE6KPO42YDU2DLFA, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.177239860 (708h31m36.048322895s) [noise=true , address=145.79.227.9:4366 , piece_id=55QH33HI4XRET6LNSOLOCTSEZHMKKJFROCYYUPBHQ4ARRHZSGFLA, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.177554228 (708h31m36.048637273s) [noise=true , address=145.79.227.8:4329 , piece_id=EQUP22JCOJQQGJ5OUDN3ZGEMZBBFCEU67N7FQOZ2Q4PEWTJFUSAA, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.178055733 (708h31m36.049138778s) [noise=true , address=145.79.227.37:4567 , piece_id=5HGX6JBKLGXMLOOMFPKSHDDTUIXNE65LWAGF7FC3P3AJ65SJQTHA, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.178451686 (708h31m36.049534721s) [noise=true , address=45.91.116.12:4225 , piece_id=RY4JFCSOWHPBTQHOMUKYWIDJRJMQEA5PKZJHSGKBDH2PTR6PXM2A, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.178943574 (708h31m36.050026609s) [noise=true , address=45.91.116.10:4082 , piece_id=HZFCFKSKBILPGPFH4FDR7ALKENADKI7XBATDYYCL653HDSWEURUA, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.179369685 (708h31m36.05045272s) [noise=true , address=145.79.227.52:4447 , piece_id=E5BRCITGD76ROWC26E3EEPIBLQB7NCF2TAQNHOITGFBSLPSXNBTQ, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.179824070 (708h31m36.050907105s) [noise=true , address=145.79.227.26:4653 , piece_id=FLR56C3RKQNU63BTTOCBXH7A3VHQPC342TSFCI6X5HT2UF4RRJTA, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.181170113 (708h31m36.052253148s) [noise=true , address=45.91.116.4:4940 , piece_id=LOHUVP7OTCRJCWGXYNEZPJWPQGE4VDOGNENQK2ODDQ3L7F7DUPJQ, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.182655121 (708h31m36.053738156s) [noise=true , address=145.79.227.15:4873 , piece_id=6K4VBR5ZL6FY7IXF57PSYSGHPSL7GVZOIAE5R2ZDUEXV6GWEYZUQ, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.186238584 (708h31m36.057321619s) [noise=true , address=145.79.227.14:4837 , piece_id=7VPM66GCOQZNQZZOTZS572DPAZ2WTXSRF2KHUCANRJFLBVVJJPWQ, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.187913333 (708h31m36.058996378s) [noise=true , address=145.79.227.12:4765 , piece_id=CHWRWXIC7SLP245XUMFRHMMGQGPLOI7QHWMIFROPU3EPPS2KB3AQ, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.191261128 (708h31m36.062344163s) [noise=true , address=145.79.227.32:4836 , piece_id=KIFNMYPMZM5XTEKYYMPGRRBYWC5AILDEWJ6CKWMH7ESU3IAHZ2YA, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:36.192723302 (708h31m36.063806337s) [noise=true , address=77.90.31.18:4805 , piece_id=EZBKD7IGQFC4QLTJIDQZ6UAMCTOACIR27QX63JXDJTVTTVTROAIA, seg_pos={0 0}, upload=620312]: Uploading piece...
Jan 4 04:00:37.875608571 (708h31m37.746691616s) [upload=620312]: Next returned nil segment
Jan 4 04:00:37.875640422 (708h31m37.746723457s) [upload=620312]: Waiting for error group managing segments...
Jan 4 04:00:38.167275485 (708h31m38.03835852s) [noise=true , address=145.79.227.9:4366 , piece_id=55QH33HI4XRET6LNSOLOCTSEZHMKKJFROCYYUPBHQ4ARRHZSGFLA, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.171273989 (708h31m38.042357034s) [noise=true , address=145.79.227.4:4186 , piece_id=RML7IV5QG77JFU664Y7K3WD6V5A7EUGD7HGGYE6KPO42YDU2DLFA, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.173180640 (708h31m38.044263685s) [noise=true , address=145.79.227.8:4329 , piece_id=EQUP22JCOJQQGJ5OUDN3ZGEMZBBFCEU67N7FQOZ2Q4PEWTJFUSAA, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.190593697 (708h31m38.061676742s) [noise=true , address=145.79.227.15:4873 , piece_id=6K4VBR5ZL6FY7IXF57PSYSGHPSL7GVZOIAE5R2ZDUEXV6GWEYZUQ, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.220116233 (708h31m38.091199278s) [noise=true , address=145.79.227.14:4837 , piece_id=7VPM66GCOQZNQZZOTZS572DPAZ2WTXSRF2KHUCANRJFLBVVJJPWQ, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.252850494 (708h31m38.123933529s) [noise=true , address=145.79.227.26:4653 , piece_id=FLR56C3RKQNU63BTTOCBXH7A3VHQPC342TSFCI6X5HT2UF4RRJTA, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.259170469 (708h31m38.130253514s) [noise=true , address=145.79.227.37:4567 , piece_id=5HGX6JBKLGXMLOOMFPKSHDDTUIXNE65LWAGF7FC3P3AJ65SJQTHA, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.262549754 (708h31m38.133632789s) [noise=true , address=145.79.227.52:4447 , piece_id=E5BRCITGD76ROWC26E3EEPIBLQB7NCF2TAQNHOITGFBSLPSXNBTQ, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.274627442 (708h31m38.145710477s) [noise=true , address=145.79.227.12:4765 , piece_id=CHWRWXIC7SLP245XUMFRHMMGQGPLOI7QHWMIFROPU3EPPS2KB3AQ, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.598544044 (708h31m38.469627079s) [noise=true , address=45.91.116.5:4298 , piece_id=ACYFG4JH7UEZCMJZDNKPSV2AZP7VCD6PUZL2YIO77KNDP62BHSOQ, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.603621272 (708h31m38.474704317s) [noise=true , address=77.90.31.18:4805 , piece_id=EZBKD7IGQFC4QLTJIDQZ6UAMCTOACIR27QX63JXDJTVTTVTROAIA, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.694841771 (708h31m38.565924806s) [noise=true , address=45.91.116.4:4940 , piece_id=LOHUVP7OTCRJCWGXYNEZPJWPQGE4VDOGNENQK2ODDQ3L7F7DUPJQ, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.744088626 (708h31m38.615171671s) [noise=true , address=145.79.227.32:4836 , piece_id=KIFNMYPMZM5XTEKYYMPGRRBYWC5AILDEWJ6CKWMH7ESU3IAHZ2YA, seg_pos={0 0}, upload=620312]: Done uploading piece. err: <nil>
Jan 4 04:00:38.744109354 (708h31m38.61519239s) [seg_pos={0 0}, upload=620312]: Segment reached optimal threshold of 13 pieces.
Jan 4 04:00:38.744332340 (708h31m38.615415385s) [noise=true , address=45.91.116.10:4082 , piece_id=HZFCFKSKBILPGPFH4FDR7ALKENADKI7XBATDYYCL653HDSWEURUA, seg_pos={0 0}, upload=620312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:51344->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:51344->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51344->45.91.116.10:4082: use of closed network connection
Jan 4 04:00:38.744340385 (708h31m38.61542342s) [noise=true , address=45.91.116.10:4082 , piece_id=HZFCFKSKBILPGPFH4FDR7ALKENADKI7XBATDYYCL653HDSWEURUA, seg_pos={0 0}, upload=620312]: Long tail cancellation due to optimal threshold
Jan 4 04:00:38.744531628 (708h31m38.615614663s) [noise=true , address=45.91.116.13:4156 , piece_id=EVVXHP7YGFZRVHQ3M53MKDM7BAX6XY44Z7C4X3L2BGJYNR3MZCRQ, seg_pos={0 0}, upload=620312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: read tcp 188.166.225.170:32890->45.91.116.13:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 04:00:38.744542219 (708h31m38.615625284s) [noise=true , address=45.91.116.13:4156 , piece_id=EVVXHP7YGFZRVHQ3M53MKDM7BAX6XY44Z7C4X3L2BGJYNR3MZCRQ, seg_pos={0 0}, upload=620312]: Long tail cancellation due to optimal threshold
Jan 4 04:00:38.744721551 (708h31m38.615804586s) [noise=true , address=45.91.116.12:4225 , piece_id=RY4JFCSOWHPBTQHOMUKYWIDJRJMQEA5PKZJHSGKBDH2PTR6PXM2A, seg_pos={0 0}, upload=620312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Jan 4 04:00:38.744728874 (708h31m38.615811919s) [noise=true , address=45.91.116.12:4225 , piece_id=RY4JFCSOWHPBTQHOMUKYWIDJRJMQEA5PKZJHSGKBDH2PTR6PXM2A, seg_pos={0 0}, upload=620312]: Long tail cancellation due to optimal threshold
Jan 4 04:00:38.744745006 (708h31m38.615828041s) [seg_pos={0 0}, upload=620312]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 04:00:38.744829576 (708h31m38.615912631s) [upload=620312]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 04:00:38.744837070 (708h31m38.615920105s) [upload=620312]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 04:00:39.029405607 (708h31m38.900488652s) [upload=620312]: Done uploading segments...
Jan 4 04:00:39.029430975 (708h31m38.90051401s) [upload=620312]: Upload finished. err: <nil>
Duration: 30.003232395s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [.X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXXX] duration:
5.001448128s trace:
https://jaeger.dev.storj.io/trace/6a3ad9779ea2c7a3
FAIL (2026-01-04 04:00:41 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001472674s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [XX.XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..X] duration:
1m0.004035009s trace:
https://jaeger.dev.storj.io/trace/59ba0323369f8ca4
FAIL (2026-01-04 04:00:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: H5X6QSOLKEYVC2U5DTOA7IK43QN44G5LUKYZ2TJEG75HPTYMFB7A) context deadline exceeded; piecestore: (Node ID: 12uBxL7Gg8kYxEpqfa8jkyG7CJKtx8zYD5vcWveLB7cRrb2UpAo, Piece ID: 3S6BXCFQOY6NP2LYRMTLWIWQ425FEFB73KQIWOZXMF7SWFRJWLPQ) context deadline exceeded; piecestore: (Node ID: 1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx, Piece ID: YSZM42F5K5AWCR2EQXENHN5GJCHH3VRTK7S3F3CZT62SFONRWIFA) context deadline exceeded
======LOGS======
.49:4349 , piece_id=SCZLPQA5EQHN3BD5XMDHDUVV3VOKJM7XYNGLLRNXWXJUYTQKGZCQ, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.812612692 (708h30m19.683695737s) [noise=true , address=145.79.227.44:4223 , piece_id=TLTPHP2V7QE3L3Z5IS7YA5YRQ3UXVKLNXLXCLVSOSKXTFUSANA4A, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.813605242 (708h30m19.684688277s) [noise=true , address=45.91.116.4:4963 , piece_id=VB77XSKDPOLOXD5ASQUOHU7QYEGQK4QEIQDQ27HQ5G3PK7T3YDSA, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.814739302 (708h30m19.685822337s) [noise=true , address=145.79.227.8:4341 , piece_id=IHPBTZ7WVNT25TLPMUCQARL4CB2AQX4TVO4K7GND3KFMRXMJ6SDA, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.814830265 (708h30m19.68591331s) [noise=true , address=77.90.31.10:4523 , piece_id=7GBBBCHTTLDMRYRBV7USEG3IBTUBNS2M6AAIP3QSLKJRBV4QZ4DA, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.815386895 (708h30m19.68646993s) [noise=true , address=77.90.31.21:4905 , piece_id=MI4ZQQSWSQQTMPW7WIFWUQHEPZOCRY26CXPDM35KTP3XABQAW6BA, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.815621060 (708h30m19.686704095s) [noise=true , address=145.79.227.43:4164 , piece_id=UMCGGVD7ZMLUIMG7AC6O5ERA6S2XWQVHW46FYGYZRA52FEGT3GYA, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.815896555 (708h30m19.68697959s) [noise=true , address=64.49.13.26:4954 , piece_id=3S6BXCFQOY6NP2LYRMTLWIWQ425FEFB73KQIWOZXMF7SWFRJWLPQ, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.816303580 (708h30m19.687386626s) [noise=true , address=145.79.227.34:4934 , piece_id=LNZLE3JXG4AN2QM4NDJCN4QHN7Y3UUL7WVHZABVSKYSRNIWRD6QQ, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.816866442 (708h30m19.687949487s) [noise=true , address=145.79.227.8:4363 , piece_id=JUAXSTBZA52ZHGRYAWDIDBIVKWBFEM3BXSUWWCCRSD2ZHC6PUGQA, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.817982096 (708h30m19.689065131s) [noise=true , address=77.90.31.14:4679 , piece_id=YSZM42F5K5AWCR2EQXENHN5GJCHH3VRTK7S3F3CZT62SFONRWIFA, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.818402507 (708h30m19.689485552s) [noise=true , address=77.90.31.11:4570 , piece_id=H5X6QSOLKEYVC2U5DTOA7IK43QN44G5LUKYZ2TJEG75HPTYMFB7A, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.818853416 (708h30m19.689936451s) [noise=true , address=45.91.116.13:4182 , piece_id=UMHJPHJYHUZKGOEMCD5LAZBRJDJBE2YYH4SENZBE4WNDXUZZMH4Q, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.819301218 (708h30m19.690384263s) [noise=true , address=145.79.227.35:4509 , piece_id=KTAUKJO7G6YKX2YSY2GIAFE5MK7CGVVLTVE64MK7T2ZVRA6K246Q, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:19.820332813 (708h30m19.691415848s) [noise=true , address=45.91.116.11:4123 , piece_id=QQRVWKONPFRXYB2P2QCKW5OPQRBRO65JUVYHRLOLYRTWDDUSICRQ, seg_pos={0 0}, upload=620295]: Uploading piece...
Jan 4 03:59:21.214764417 (708h30m21.085847452s) [upload=620295]: Next returned nil segment
Jan 4 03:59:21.214782741 (708h30m21.085865787s) [upload=620295]: Waiting for error group managing segments...
Jan 4 03:59:21.699129857 (708h30m21.570212892s) [noise=true , address=145.79.227.43:4164 , piece_id=UMCGGVD7ZMLUIMG7AC6O5ERA6S2XWQVHW46FYGYZRA52FEGT3GYA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:21.722287071 (708h30m21.593370116s) [noise=true , address=145.79.227.8:4363 , piece_id=JUAXSTBZA52ZHGRYAWDIDBIVKWBFEM3BXSUWWCCRSD2ZHC6PUGQA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:21.764610311 (708h30m21.635693356s) [noise=true , address=145.79.227.35:4509 , piece_id=KTAUKJO7G6YKX2YSY2GIAFE5MK7CGVVLTVE64MK7T2ZVRA6K246Q, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:21.962545086 (708h30m21.833628121s) [noise=true , address=77.90.31.10:4523 , piece_id=7GBBBCHTTLDMRYRBV7USEG3IBTUBNS2M6AAIP3QSLKJRBV4QZ4DA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.047051729 (708h30m21.918134764s) [noise=true , address=77.90.31.11:4570 , piece_id=H5X6QSOLKEYVC2U5DTOA7IK43QN44G5LUKYZ2TJEG75HPTYMFB7A, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.175655979 (708h30m22.046739024s) [noise=true , address=45.91.116.13:4182 , piece_id=UMHJPHJYHUZKGOEMCD5LAZBRJDJBE2YYH4SENZBE4WNDXUZZMH4Q, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.234055500 (708h30m22.105138535s) [noise=true , address=45.91.116.11:4123 , piece_id=QQRVWKONPFRXYB2P2QCKW5OPQRBRO65JUVYHRLOLYRTWDDUSICRQ, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.503536731 (708h30m22.374619766s) [noise=true , address=193.149.164.143:4696 , piece_id=RPXIMXUKMVPEVWPSBKY4I5NIU6OTMMH7DZQACY3Y5FGLL4BBSYSA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.551408516 (708h30m22.422491551s) [noise=true , address=193.149.164.143:4702 , piece_id=EVCTP5OBE6AFG65Z4CH53QXWCACL2QYTC5O4UPJY4FTKHTGSVMDA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.593643687 (708h30m22.464726732s) [noise=true , address=145.79.227.44:4223 , piece_id=TLTPHP2V7QE3L3Z5IS7YA5YRQ3UXVKLNXLXCLVSOSKXTFUSANA4A, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.635841658 (708h30m22.506924693s) [noise=true , address=145.79.227.34:4934 , piece_id=LNZLE3JXG4AN2QM4NDJCN4QHN7Y3UUL7WVHZABVSKYSRNIWRD6QQ, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.741406057 (708h30m22.612489092s) [noise=true , address=145.79.227.49:4349 , piece_id=SCZLPQA5EQHN3BD5XMDHDUVV3VOKJM7XYNGLLRNXWXJUYTQKGZCQ, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.771196893 (708h30m22.642279938s) [noise=true , address=64.49.13.26:4954 , piece_id=3S6BXCFQOY6NP2LYRMTLWIWQ425FEFB73KQIWOZXMF7SWFRJWLPQ, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.923276354 (708h30m22.794359389s) [noise=true , address=77.90.31.14:4679 , piece_id=YSZM42F5K5AWCR2EQXENHN5GJCHH3VRTK7S3F3CZT62SFONRWIFA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: <nil>
Jan 4 03:59:22.923296052 (708h30m22.794379097s) [seg_pos={0 0}, upload=620295]: Segment reached optimal threshold of 14 pieces.
Jan 4 03:59:22.923467457 (708h30m22.794550492s) [noise=true , address=77.90.31.21:4905 , piece_id=MI4ZQQSWSQQTMPW7WIFWUQHEPZOCRY26CXPDM35KTP3XABQAW6BA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: write tcp 188.166.225.170:42068->77.90.31.21:4905: use of closed network connection; write tcp 188.166.225.170:42068->77.90.31.21:4905: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42068->77.90.31.21:4905: use of closed network connection
Jan 4 03:59:22.923474371 (708h30m22.794557406s) [noise=true , address=77.90.31.21:4905 , piece_id=MI4ZQQSWSQQTMPW7WIFWUQHEPZOCRY26CXPDM35KTP3XABQAW6BA, seg_pos={0 0}, upload=620295]: Long tail cancellation due to optimal threshold
Jan 4 03:59:22.923568129 (708h30m22.794651164s) [noise=true , address=45.91.116.4:4963 , piece_id=VB77XSKDPOLOXD5ASQUOHU7QYEGQK4QEIQDQ27HQ5G3PK7T3YDSA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: write tcp 188.166.225.170:35546->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:35546->45.91.116.4:4963: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35546->45.91.116.4:4963: use of closed network connection
Jan 4 03:59:22.923572979 (708h30m22.794656014s) [noise=true , address=45.91.116.4:4963 , piece_id=VB77XSKDPOLOXD5ASQUOHU7QYEGQK4QEIQDQ27HQ5G3PK7T3YDSA, seg_pos={0 0}, upload=620295]: Long tail cancellation due to optimal threshold
Jan 4 03:59:22.923620008 (708h30m22.794703043s) [noise=true , address=145.79.227.33:4879 , piece_id=2HSIETHLLJJNSWILK5PB3MPT4OMAL6EUTJ4BFTU6MBPP4WL7SHQQ, seg_pos={0 0}, upload=620295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Jan 4 03:59:22.923636048 (708h30m22.794719084s) [noise=true , address=145.79.227.33:4879 , piece_id=2HSIETHLLJJNSWILK5PB3MPT4OMAL6EUTJ4BFTU6MBPP4WL7SHQQ, seg_pos={0 0}, upload=620295]: Long tail cancellation due to optimal threshold
Jan 4 03:59:22.923673039 (708h30m22.794756074s) [noise=true , address=145.79.227.8:4341 , piece_id=IHPBTZ7WVNT25TLPMUCQARL4CB2AQX4TVO4K7GND3KFMRXMJ6SDA, seg_pos={0 0}, upload=620295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfZXtm2GymdBAaEBFaX9jeYbK3PveonNNskUq9cLmuW7vbuu5): protocol: expected piece hash; context canceled; EOF
Jan 4 03:59:22.923678971 (708h30m22.794762006s) [noise=true , address=145.79.227.8:4341 , piece_id=IHPBTZ7WVNT25TLPMUCQARL4CB2AQX4TVO4K7GND3KFMRXMJ6SDA, seg_pos={0 0}, upload=620295]: Long tail cancellation due to optimal threshold
Jan 4 03:59:22.923874763 (708h30m22.794957798s) [seg_pos={0 0}, upload=620295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 03:59:22.923973010 (708h30m22.795056055s) [upload=620295]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:59:22.923979883 (708h30m22.795062928s) [upload=620295]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:59:23.212065680 (708h30m23.083148715s) [upload=620295]: Done uploading segments...
Jan 4 03:59:23.212095336 (708h30m23.083178371s) [upload=620295]: Upload finished. err: <nil>
Duration: 1m0.004058943s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [..X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXXX] duration:
5.000935358s trace:
https://jaeger.dev.storj.io/trace/78267b9ff483cad1
FAIL (2026-01-04 04:00:06 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000964353s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [X..X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXXX] duration:
5.001538635s trace:
https://jaeger.dev.storj.io/trace/15e2bb4cce115aa8
FAIL (2026-01-04 03:59:45 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001565386s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.XX...XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX..] duration:
6.540432654s trace:
https://jaeger.dev.storj.io/trace/1dac913198e4bce8
FAIL (2026-01-04 03:59:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA) context deadline exceeded
======LOGS======
load=620274]: Uploading segments...
Jan 4 03:57:59.230697680 (708h28m59.101780715s) [upload=620274]: Got next segment. Inline: false
Jan 4 03:57:59.230737095 (708h28m59.10182013s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 03:57:59.230743196 (708h28m59.101826231s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 03:57:59.478585542 (708h28m59.349668597s) [seg_pos={0 0}, upload=620274]: Begin upload segment...
Jan 4 03:57:59.478645196 (708h28m59.349728241s) [seg_pos={0 0}, upload=620274]: Done begin upload segment.
Jan 4 03:57:59.478682467 (708h28m59.349765512s) [noise=true , address=145.79.227.9:4365 , piece_id=24OOJNJUYJN7D4DMTASKANSTETTAJ662SB753SKR4F5JMTCBMKRQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.478693137 (708h28m59.349776182s) [noise=true , address=45.91.116.11:4119 , piece_id=FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.478881116 (708h28m59.349964161s) [noise=true , address=145.79.227.16:4911 , piece_id=FE56N37GPUKGEDPFIVSCUAQOLJPG5K2FQIJGQJBRVEROTE6IAZNQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.479549548 (708h28m59.350632613s) [noise=true , address=145.79.227.48:4303 , piece_id=MSIS6IJ4YWYQQFMKGF34SS6CFGH2BWMGHDSDJM335FRGM5JNFYXA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.479692530 (708h28m59.350775575s) [noise=true , address=145.79.227.51:4409 , piece_id=JBB7ZKOWH3WPY4LMWK4TIM3UE33C7VNF2DRGSJH7AB5V5HDLUCHQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480064850 (708h28m59.351147885s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480407532 (708h28m59.351490567s) [noise=true , address=145.79.227.4:4186 , piece_id=KUBCVDGZYTGDGVONIWGI7BHURGYIFRL52RQZ2JAHBA7HUTEVCQQA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480636718 (708h28m59.351719763s) [noise=true , address=45.91.116.13:4153 , piece_id=SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480779089 (708h28m59.351862134s) [noise=true , address=145.79.227.13:4804 , piece_id=FELWRL54QLTY4A45JT4ECWBYSDUM2EUDDQ7PCO7XGKY3P4MIMPFQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481139536 (708h28m59.352222571s) [noise=true , address=45.91.116.12:4226 , piece_id=MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481208928 (708h28m59.352291963s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481668233 (708h28m59.352751278s) [noise=true , address=145.79.227.42:4122 , piece_id=XBJ7WEX7MER4YP6GELQGK5GFE5ZDRX3W2JZVRJXODFRTVLGTM3MQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.482247114 (708h28m59.353330149s) [noise=true , address=145.79.227.37:4566 , piece_id=WRO3PDTBY4YVRCS56E7GWSYFZMUXJ6D7KIM76QED2VE3AV46SG6A, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.484921106 (708h28m59.356004141s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.485409376 (708h28m59.356492421s) [noise=true , address=145.79.227.22:4510 , piece_id=B7SQLJUEOCMEDCA56ODLID2IUJKSLL224QEHOSQDL6RZCKVG66BA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.485914699 (708h28m59.356997734s) [noise=true , address=145.79.227.20:4438 , piece_id=SOPZ32JMEACTQWF5P3CQSBLBA2CFXHJRIZNY7RWPUEPZCQYRFILA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:58:01.190842079 (708h29m1.061925124s) [upload=620274]: Next returned nil segment
Jan 4 03:58:01.190858711 (708h29m1.061941746s) [upload=620274]: Waiting for error group managing segments...
Jan 4 03:58:01.475552237 (708h29m1.346635282s) [noise=true , address=145.79.227.37:4566 , piece_id=WRO3PDTBY4YVRCS56E7GWSYFZMUXJ6D7KIM76QED2VE3AV46SG6A, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.477087481 (708h29m1.348170526s) [noise=true , address=145.79.227.9:4365 , piece_id=24OOJNJUYJN7D4DMTASKANSTETTAJ662SB753SKR4F5JMTCBMKRQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.483350486 (708h29m1.354433532s) [noise=true , address=145.79.227.48:4303 , piece_id=MSIS6IJ4YWYQQFMKGF34SS6CFGH2BWMGHDSDJM335FRGM5JNFYXA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.484149919 (708h29m1.355232974s) [noise=true , address=145.79.227.13:4804 , piece_id=FELWRL54QLTY4A45JT4ECWBYSDUM2EUDDQ7PCO7XGKY3P4MIMPFQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.492725057 (708h29m1.363808102s) [noise=true , address=145.79.227.4:4186 , piece_id=KUBCVDGZYTGDGVONIWGI7BHURGYIFRL52RQZ2JAHBA7HUTEVCQQA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.493582259 (708h29m1.364665314s) [noise=true , address=145.79.227.22:4510 , piece_id=B7SQLJUEOCMEDCA56ODLID2IUJKSLL224QEHOSQDL6RZCKVG66BA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.498938428 (708h29m1.370021463s) [noise=true , address=145.79.227.16:4911 , piece_id=FE56N37GPUKGEDPFIVSCUAQOLJPG5K2FQIJGQJBRVEROTE6IAZNQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.537908171 (708h29m1.408991217s) [noise=true , address=145.79.227.51:4409 , piece_id=JBB7ZKOWH3WPY4LMWK4TIM3UE33C7VNF2DRGSJH7AB5V5HDLUCHQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.566501758 (708h29m1.437584803s) [noise=true , address=145.79.227.42:4122 , piece_id=XBJ7WEX7MER4YP6GELQGK5GFE5ZDRX3W2JZVRJXODFRTVLGTM3MQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.884314868 (708h29m1.755397914s) [noise=true , address=45.91.116.13:4153 , piece_id=SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.000869986 (708h29m1.871953041s) [noise=true , address=145.79.227.20:4438 , piece_id=SOPZ32JMEACTQWF5P3CQSBLBA2CFXHJRIZNY7RWPUEPZCQYRFILA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.027163753 (708h29m1.898246798s) [noise=true , address=45.91.116.11:4119 , piece_id=FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.236344783 (708h29m2.107427828s) [noise=true , address=45.91.116.12:4226 , piece_id=MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.236373517 (708h29m2.107456573s) [seg_pos={0 0}, upload=620274]: Segment reached optimal threshold of 13 pieces.
Jan 4 03:58:02.236638382 (708h29m2.107721427s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection
Jan 4 03:58:02.236649844 (708h29m2.107732889s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.236957099 (708h29m2.108040144s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 03:58:02.236969793 (708h29m2.108052838s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.237049936 (708h29m2.108132971s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection
Jan 4 03:58:02.237062439 (708h29m2.108145485s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.237084251 (708h29m2.108167296s) [seg_pos={0 0}, upload=620274]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 03:58:02.237175615 (708h29m2.10825866s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:58:02.237181716 (708h29m2.108264751s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:58:02.533167946 (708h29m2.404250991s) [upload=620274]: Done uploading segments...
Jan 4 03:58:02.533194517 (708h29m2.404277562s) [upload=620274]: Upload finished. err: <nil>
Duration: 6.540449547s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X.XX...XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX.] duration:
29.964620704s trace:
https://jaeger.dev.storj.io/trace/1dac913198e4bce8
FAIL (2026-01-04 03:59:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA) context deadline exceeded
======LOGS======
load=620274]: Uploading segments...
Jan 4 03:57:59.230697680 (708h28m59.101780715s) [upload=620274]: Got next segment. Inline: false
Jan 4 03:57:59.230737095 (708h28m59.10182013s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 03:57:59.230743196 (708h28m59.101826231s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 03:57:59.478585542 (708h28m59.349668597s) [seg_pos={0 0}, upload=620274]: Begin upload segment...
Jan 4 03:57:59.478645196 (708h28m59.349728241s) [seg_pos={0 0}, upload=620274]: Done begin upload segment.
Jan 4 03:57:59.478682467 (708h28m59.349765512s) [noise=true , address=145.79.227.9:4365 , piece_id=24OOJNJUYJN7D4DMTASKANSTETTAJ662SB753SKR4F5JMTCBMKRQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.478693137 (708h28m59.349776182s) [noise=true , address=45.91.116.11:4119 , piece_id=FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.478881116 (708h28m59.349964161s) [noise=true , address=145.79.227.16:4911 , piece_id=FE56N37GPUKGEDPFIVSCUAQOLJPG5K2FQIJGQJBRVEROTE6IAZNQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.479549548 (708h28m59.350632613s) [noise=true , address=145.79.227.48:4303 , piece_id=MSIS6IJ4YWYQQFMKGF34SS6CFGH2BWMGHDSDJM335FRGM5JNFYXA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.479692530 (708h28m59.350775575s) [noise=true , address=145.79.227.51:4409 , piece_id=JBB7ZKOWH3WPY4LMWK4TIM3UE33C7VNF2DRGSJH7AB5V5HDLUCHQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480064850 (708h28m59.351147885s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480407532 (708h28m59.351490567s) [noise=true , address=145.79.227.4:4186 , piece_id=KUBCVDGZYTGDGVONIWGI7BHURGYIFRL52RQZ2JAHBA7HUTEVCQQA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480636718 (708h28m59.351719763s) [noise=true , address=45.91.116.13:4153 , piece_id=SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480779089 (708h28m59.351862134s) [noise=true , address=145.79.227.13:4804 , piece_id=FELWRL54QLTY4A45JT4ECWBYSDUM2EUDDQ7PCO7XGKY3P4MIMPFQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481139536 (708h28m59.352222571s) [noise=true , address=45.91.116.12:4226 , piece_id=MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481208928 (708h28m59.352291963s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481668233 (708h28m59.352751278s) [noise=true , address=145.79.227.42:4122 , piece_id=XBJ7WEX7MER4YP6GELQGK5GFE5ZDRX3W2JZVRJXODFRTVLGTM3MQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.482247114 (708h28m59.353330149s) [noise=true , address=145.79.227.37:4566 , piece_id=WRO3PDTBY4YVRCS56E7GWSYFZMUXJ6D7KIM76QED2VE3AV46SG6A, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.484921106 (708h28m59.356004141s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.485409376 (708h28m59.356492421s) [noise=true , address=145.79.227.22:4510 , piece_id=B7SQLJUEOCMEDCA56ODLID2IUJKSLL224QEHOSQDL6RZCKVG66BA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.485914699 (708h28m59.356997734s) [noise=true , address=145.79.227.20:4438 , piece_id=SOPZ32JMEACTQWF5P3CQSBLBA2CFXHJRIZNY7RWPUEPZCQYRFILA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:58:01.190842079 (708h29m1.061925124s) [upload=620274]: Next returned nil segment
Jan 4 03:58:01.190858711 (708h29m1.061941746s) [upload=620274]: Waiting for error group managing segments...
Jan 4 03:58:01.475552237 (708h29m1.346635282s) [noise=true , address=145.79.227.37:4566 , piece_id=WRO3PDTBY4YVRCS56E7GWSYFZMUXJ6D7KIM76QED2VE3AV46SG6A, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.477087481 (708h29m1.348170526s) [noise=true , address=145.79.227.9:4365 , piece_id=24OOJNJUYJN7D4DMTASKANSTETTAJ662SB753SKR4F5JMTCBMKRQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.483350486 (708h29m1.354433532s) [noise=true , address=145.79.227.48:4303 , piece_id=MSIS6IJ4YWYQQFMKGF34SS6CFGH2BWMGHDSDJM335FRGM5JNFYXA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.484149919 (708h29m1.355232974s) [noise=true , address=145.79.227.13:4804 , piece_id=FELWRL54QLTY4A45JT4ECWBYSDUM2EUDDQ7PCO7XGKY3P4MIMPFQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.492725057 (708h29m1.363808102s) [noise=true , address=145.79.227.4:4186 , piece_id=KUBCVDGZYTGDGVONIWGI7BHURGYIFRL52RQZ2JAHBA7HUTEVCQQA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.493582259 (708h29m1.364665314s) [noise=true , address=145.79.227.22:4510 , piece_id=B7SQLJUEOCMEDCA56ODLID2IUJKSLL224QEHOSQDL6RZCKVG66BA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.498938428 (708h29m1.370021463s) [noise=true , address=145.79.227.16:4911 , piece_id=FE56N37GPUKGEDPFIVSCUAQOLJPG5K2FQIJGQJBRVEROTE6IAZNQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.537908171 (708h29m1.408991217s) [noise=true , address=145.79.227.51:4409 , piece_id=JBB7ZKOWH3WPY4LMWK4TIM3UE33C7VNF2DRGSJH7AB5V5HDLUCHQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.566501758 (708h29m1.437584803s) [noise=true , address=145.79.227.42:4122 , piece_id=XBJ7WEX7MER4YP6GELQGK5GFE5ZDRX3W2JZVRJXODFRTVLGTM3MQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.884314868 (708h29m1.755397914s) [noise=true , address=45.91.116.13:4153 , piece_id=SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.000869986 (708h29m1.871953041s) [noise=true , address=145.79.227.20:4438 , piece_id=SOPZ32JMEACTQWF5P3CQSBLBA2CFXHJRIZNY7RWPUEPZCQYRFILA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.027163753 (708h29m1.898246798s) [noise=true , address=45.91.116.11:4119 , piece_id=FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.236344783 (708h29m2.107427828s) [noise=true , address=45.91.116.12:4226 , piece_id=MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.236373517 (708h29m2.107456573s) [seg_pos={0 0}, upload=620274]: Segment reached optimal threshold of 13 pieces.
Jan 4 03:58:02.236638382 (708h29m2.107721427s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection
Jan 4 03:58:02.236649844 (708h29m2.107732889s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.236957099 (708h29m2.108040144s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 03:58:02.236969793 (708h29m2.108052838s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.237049936 (708h29m2.108132971s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection
Jan 4 03:58:02.237062439 (708h29m2.108145485s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.237084251 (708h29m2.108167296s) [seg_pos={0 0}, upload=620274]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 03:58:02.237175615 (708h29m2.10825866s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:58:02.237181716 (708h29m2.108264751s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:58:02.533167946 (708h29m2.404250991s) [upload=620274]: Done uploading segments...
Jan 4 03:58:02.533194517 (708h29m2.404277562s) [upload=620274]: Upload finished. err: <nil>
Duration: 29.96465014s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [XXX.XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X..] duration:
15.445101379s trace:
https://jaeger.dev.storj.io/trace/317f2fa634aadcb8
FAIL (2026-01-04 03:58:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.500836722 (708h28m13.371919767s) [noise=true , address=145.79.227.40:4075 , piece_id=64V3PD4673C3HHVK7OV4AZDYCQCI2FAJQJQR6I57NOJUOKNX54YQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.501370288 (708h28m13.372453343s) [noise=true , address=77.90.31.21:4908 , piece_id=JP4AOX5Y5EHBBM5LGO4BEZN2VP2IYMINTK4SAUTQTLK7DV2IEVMA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.503483382 (708h28m13.374566417s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.506809435 (708h28m13.37789248s) [noise=true , address=77.90.31.19:4850 , piece_id=2PXGLWVHEZMTVZT5FJMFA5FEZHRSIIQW5ZXFILR2CESYXFJXBJAQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.508094151 (708h28m13.379177196s) [noise=true , address=145.79.227.34:4928 , piece_id=LYY6BBHNNDFRJRGR7L45SEMYQUGEGXDUIQDYACTKFAJT3YADCZ3A, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.510791498 (708h28m13.381874543s) [noise=true , address=77.90.31.18:4839 , piece_id=RMRIZHVQGHRSWJDN2PESTDCL5XN2AZE5B4AKBPV3XM6FADFUSXXQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.512222142 (708h28m13.383305187s) [noise=true , address=64.49.13.26:4946 , piece_id=U6PYY2KRWXF5DTWDRAVSQC3QZQPYH2HCFBMWYJBMN6B2RO4AE2DA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.514224324 (708h28m13.385307379s) [noise=true , address=77.90.31.12:4606 , piece_id=OP3W5AWMNKTYRBIBSNFRWNWYCIJXFTN3265XZNHIM37MEJB2ZNSA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.514473418 (708h28m13.385556463s) [noise=true , address=145.79.227.44:4217 , piece_id=AX5UJBRBB6ZMQ6KKZNXY6PS2KAVQBQ6A6QABH3IGWEGDTJRZJTMA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.516818914 (708h28m13.387901959s) [noise=true , address=145.79.227.25:4645 , piece_id=CRE6TPTH6SPVNV4HK7V6VIX6ITYBDRCDECE6E2DQGZ62SI3IG6ZQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.517063730 (708h28m13.388146775s) [noise=true , address=145.79.227.43:4164 , piece_id=7CDVHBK47GVX4QRH3GIDKEVSIKNINJQCT2EQQTAW3NHNURNTEGYQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.519958112 (708h28m13.391041167s) [noise=true , address=145.79.227.29:4962 , piece_id=CWXFZMN45ZRDQCCH5SJVFD7FHEZK4UXRXLUVXTJB7HTPYLEH7CZA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.520783343 (708h28m13.391866388s) [noise=true , address=145.79.227.35:4484 , piece_id=EVGCRX7ZYXABOF3EABL3FDQJX5GWPYX4RWJZLY5PVMYVXHPWSVXQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.523536816 (708h28m13.394619861s) [noise=true , address=145.79.227.16:4932 , piece_id=M2AGJU2A6PYD2SHFVCY54H5ZHV3EF64IG32OFMOF7G2IXTSA6XQA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.524736241 (708h28m13.395819286s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:14.917200007 (708h28m14.788283052s) [upload=620262]: Next returned nil segment
Jan 4 03:57:14.917216399 (708h28m14.788299444s) [upload=620262]: Waiting for error group managing segments...
Jan 4 03:57:15.330420287 (708h28m15.201503322s) [noise=true , address=64.49.13.26:4946 , piece_id=U6PYY2KRWXF5DTWDRAVSQC3QZQPYH2HCFBMWYJBMN6B2RO4AE2DA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.382535652 (708h28m15.253618687s) [noise=true , address=145.79.227.11:4754 , piece_id=OTUTAUK7V6JLXAUR5IFJPVOS3DEJGHBTPAQ5MVR5C35SMDTZXP5Q, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.401033114 (708h28m15.272116149s) [noise=true , address=145.79.227.43:4164 , piece_id=7CDVHBK47GVX4QRH3GIDKEVSIKNINJQCT2EQQTAW3NHNURNTEGYQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.406385305 (708h28m15.27746834s) [noise=true , address=145.79.227.16:4932 , piece_id=M2AGJU2A6PYD2SHFVCY54H5ZHV3EF64IG32OFMOF7G2IXTSA6XQA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.424319284 (708h28m15.295402329s) [noise=true , address=145.79.227.40:4075 , piece_id=64V3PD4673C3HHVK7OV4AZDYCQCI2FAJQJQR6I57NOJUOKNX54YQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.564462616 (708h28m15.435545651s) [noise=true , address=145.79.227.25:4645 , piece_id=CRE6TPTH6SPVNV4HK7V6VIX6ITYBDRCDECE6E2DQGZ62SI3IG6ZQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.801676355 (708h28m15.67275939s) [noise=true , address=77.90.31.12:4606 , piece_id=OP3W5AWMNKTYRBIBSNFRWNWYCIJXFTN3265XZNHIM37MEJB2ZNSA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.802560438 (708h28m15.673643473s) [noise=true , address=77.90.31.21:4908 , piece_id=JP4AOX5Y5EHBBM5LGO4BEZN2VP2IYMINTK4SAUTQTLK7DV2IEVMA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.244067610 (708h28m16.115150905s) [noise=true , address=145.79.227.35:4484 , piece_id=EVGCRX7ZYXABOF3EABL3FDQJX5GWPYX4RWJZLY5PVMYVXHPWSVXQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.276851114 (708h28m16.147934159s) [noise=true , address=145.79.227.34:4928 , piece_id=LYY6BBHNNDFRJRGR7L45SEMYQUGEGXDUIQDYACTKFAJT3YADCZ3A, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.358570980 (708h28m16.229654016s) [noise=true , address=145.79.227.44:4217 , piece_id=AX5UJBRBB6ZMQ6KKZNXY6PS2KAVQBQ6A6QABH3IGWEGDTJRZJTMA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.417875835 (708h28m16.28895887s) [noise=true , address=77.90.31.18:4839 , piece_id=RMRIZHVQGHRSWJDN2PESTDCL5XN2AZE5B4AKBPV3XM6FADFUSXXQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.574599499 (708h28m16.445682534s) [noise=true , address=145.79.227.29:4962 , piece_id=CWXFZMN45ZRDQCCH5SJVFD7FHEZK4UXRXLUVXTJB7HTPYLEH7CZA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:17.175314254 (708h28m17.046397299s) [noise=true , address=77.90.31.19:4850 , piece_id=2PXGLWVHEZMTVZT5FJMFA5FEZHRSIIQW5ZXFILR2CESYXFJXBJAQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:17.175338430 (708h28m17.046421475s) [seg_pos={0 0}, upload=620262]: Segment reached optimal threshold of 14 pieces.
Jan 4 03:57:17.175581002 (708h28m17.046664037s) [noise=true , address=64.49.13.138:4481 , piece_id=AIXOG6H7KPQWWYJHX4LTBY3AQC3X7RB5SXX2TJJGSJLPI27EZPIA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16SmJMbycEDFbqMUkmhZ5bPJHdcehG16mQdgN4s6fHZ7VcshPT): protocol: expected piece hash; context canceled; EOF
Jan 4 03:57:17.175592334 (708h28m17.046675379s) [noise=true , address=64.49.13.138:4481 , piece_id=AIXOG6H7KPQWWYJHX4LTBY3AQC3X7RB5SXX2TJJGSJLPI27EZPIA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.175705960 (708h28m17.046789005s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Zs9PFmdpQVHe1UjYapzPRUpsopENKECrNUhVVBZ5nmhpGA9Ws): protocol: expected piece hash; context canceled; EOF
Jan 4 03:57:17.175740967 (708h28m17.046824012s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.175974972 (708h28m17.047058018s) [noise=true , address=145.79.227.48:4309 , piece_id=ZKQP3WF6WGDG46N76YSANGNPZIK5RGF3PDTBZ7G2IDOP4MQ7R2EA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12txGN8RDcyMycMhkRtdZAUkMK9L2ZnqPihEGZ2YNXJvu89RUU1): protocol: write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection; write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection
Jan 4 03:57:17.175987155 (708h28m17.04707019s) [noise=true , address=145.79.227.48:4309 , piece_id=ZKQP3WF6WGDG46N76YSANGNPZIK5RGF3PDTBZ7G2IDOP4MQ7R2EA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.176093107 (708h28m17.047176162s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu): protocol: read tcp 188.166.225.170:50396->45.91.116.8:4050: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 03:57:17.176103206 (708h28m17.047186251s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.176124888 (708h28m17.047207923s) [seg_pos={0 0}, upload=620262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 03:57:17.176234506 (708h28m17.047317561s) [upload=620262]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:57:17.176242932 (708h28m17.047325977s) [upload=620262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:57:17.502405262 (708h28m17.373488317s) [upload=620262]: Done uploading segments...
Jan 4 03:57:17.502431651 (708h28m17.373514696s) [upload=620262]: Upload finished. err: <nil>
Duration: 15.445132237s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XX..X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXXX] duration:
5.001440686s trace:
https://jaeger.dev.storj.io/trace/fcd145d76cab540
FAIL (2026-01-04 03:58:30 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001462476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
35 out of 60 is failed (threshold=3) [XXXX.XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X.] duration:
15.336543477s trace:
https://jaeger.dev.storj.io/trace/317f2fa634aadcb8
FAIL (2026-01-04 03:58:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.500836722 (708h28m13.371919767s) [noise=true , address=145.79.227.40:4075 , piece_id=64V3PD4673C3HHVK7OV4AZDYCQCI2FAJQJQR6I57NOJUOKNX54YQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.501370288 (708h28m13.372453343s) [noise=true , address=77.90.31.21:4908 , piece_id=JP4AOX5Y5EHBBM5LGO4BEZN2VP2IYMINTK4SAUTQTLK7DV2IEVMA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.503483382 (708h28m13.374566417s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.506809435 (708h28m13.37789248s) [noise=true , address=77.90.31.19:4850 , piece_id=2PXGLWVHEZMTVZT5FJMFA5FEZHRSIIQW5ZXFILR2CESYXFJXBJAQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.508094151 (708h28m13.379177196s) [noise=true , address=145.79.227.34:4928 , piece_id=LYY6BBHNNDFRJRGR7L45SEMYQUGEGXDUIQDYACTKFAJT3YADCZ3A, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.510791498 (708h28m13.381874543s) [noise=true , address=77.90.31.18:4839 , piece_id=RMRIZHVQGHRSWJDN2PESTDCL5XN2AZE5B4AKBPV3XM6FADFUSXXQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.512222142 (708h28m13.383305187s) [noise=true , address=64.49.13.26:4946 , piece_id=U6PYY2KRWXF5DTWDRAVSQC3QZQPYH2HCFBMWYJBMN6B2RO4AE2DA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.514224324 (708h28m13.385307379s) [noise=true , address=77.90.31.12:4606 , piece_id=OP3W5AWMNKTYRBIBSNFRWNWYCIJXFTN3265XZNHIM37MEJB2ZNSA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.514473418 (708h28m13.385556463s) [noise=true , address=145.79.227.44:4217 , piece_id=AX5UJBRBB6ZMQ6KKZNXY6PS2KAVQBQ6A6QABH3IGWEGDTJRZJTMA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.516818914 (708h28m13.387901959s) [noise=true , address=145.79.227.25:4645 , piece_id=CRE6TPTH6SPVNV4HK7V6VIX6ITYBDRCDECE6E2DQGZ62SI3IG6ZQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.517063730 (708h28m13.388146775s) [noise=true , address=145.79.227.43:4164 , piece_id=7CDVHBK47GVX4QRH3GIDKEVSIKNINJQCT2EQQTAW3NHNURNTEGYQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.519958112 (708h28m13.391041167s) [noise=true , address=145.79.227.29:4962 , piece_id=CWXFZMN45ZRDQCCH5SJVFD7FHEZK4UXRXLUVXTJB7HTPYLEH7CZA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.520783343 (708h28m13.391866388s) [noise=true , address=145.79.227.35:4484 , piece_id=EVGCRX7ZYXABOF3EABL3FDQJX5GWPYX4RWJZLY5PVMYVXHPWSVXQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.523536816 (708h28m13.394619861s) [noise=true , address=145.79.227.16:4932 , piece_id=M2AGJU2A6PYD2SHFVCY54H5ZHV3EF64IG32OFMOF7G2IXTSA6XQA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.524736241 (708h28m13.395819286s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:14.917200007 (708h28m14.788283052s) [upload=620262]: Next returned nil segment
Jan 4 03:57:14.917216399 (708h28m14.788299444s) [upload=620262]: Waiting for error group managing segments...
Jan 4 03:57:15.330420287 (708h28m15.201503322s) [noise=true , address=64.49.13.26:4946 , piece_id=U6PYY2KRWXF5DTWDRAVSQC3QZQPYH2HCFBMWYJBMN6B2RO4AE2DA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.382535652 (708h28m15.253618687s) [noise=true , address=145.79.227.11:4754 , piece_id=OTUTAUK7V6JLXAUR5IFJPVOS3DEJGHBTPAQ5MVR5C35SMDTZXP5Q, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.401033114 (708h28m15.272116149s) [noise=true , address=145.79.227.43:4164 , piece_id=7CDVHBK47GVX4QRH3GIDKEVSIKNINJQCT2EQQTAW3NHNURNTEGYQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.406385305 (708h28m15.27746834s) [noise=true , address=145.79.227.16:4932 , piece_id=M2AGJU2A6PYD2SHFVCY54H5ZHV3EF64IG32OFMOF7G2IXTSA6XQA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.424319284 (708h28m15.295402329s) [noise=true , address=145.79.227.40:4075 , piece_id=64V3PD4673C3HHVK7OV4AZDYCQCI2FAJQJQR6I57NOJUOKNX54YQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.564462616 (708h28m15.435545651s) [noise=true , address=145.79.227.25:4645 , piece_id=CRE6TPTH6SPVNV4HK7V6VIX6ITYBDRCDECE6E2DQGZ62SI3IG6ZQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.801676355 (708h28m15.67275939s) [noise=true , address=77.90.31.12:4606 , piece_id=OP3W5AWMNKTYRBIBSNFRWNWYCIJXFTN3265XZNHIM37MEJB2ZNSA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.802560438 (708h28m15.673643473s) [noise=true , address=77.90.31.21:4908 , piece_id=JP4AOX5Y5EHBBM5LGO4BEZN2VP2IYMINTK4SAUTQTLK7DV2IEVMA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.244067610 (708h28m16.115150905s) [noise=true , address=145.79.227.35:4484 , piece_id=EVGCRX7ZYXABOF3EABL3FDQJX5GWPYX4RWJZLY5PVMYVXHPWSVXQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.276851114 (708h28m16.147934159s) [noise=true , address=145.79.227.34:4928 , piece_id=LYY6BBHNNDFRJRGR7L45SEMYQUGEGXDUIQDYACTKFAJT3YADCZ3A, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.358570980 (708h28m16.229654016s) [noise=true , address=145.79.227.44:4217 , piece_id=AX5UJBRBB6ZMQ6KKZNXY6PS2KAVQBQ6A6QABH3IGWEGDTJRZJTMA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.417875835 (708h28m16.28895887s) [noise=true , address=77.90.31.18:4839 , piece_id=RMRIZHVQGHRSWJDN2PESTDCL5XN2AZE5B4AKBPV3XM6FADFUSXXQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.574599499 (708h28m16.445682534s) [noise=true , address=145.79.227.29:4962 , piece_id=CWXFZMN45ZRDQCCH5SJVFD7FHEZK4UXRXLUVXTJB7HTPYLEH7CZA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:17.175314254 (708h28m17.046397299s) [noise=true , address=77.90.31.19:4850 , piece_id=2PXGLWVHEZMTVZT5FJMFA5FEZHRSIIQW5ZXFILR2CESYXFJXBJAQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:17.175338430 (708h28m17.046421475s) [seg_pos={0 0}, upload=620262]: Segment reached optimal threshold of 14 pieces.
Jan 4 03:57:17.175581002 (708h28m17.046664037s) [noise=true , address=64.49.13.138:4481 , piece_id=AIXOG6H7KPQWWYJHX4LTBY3AQC3X7RB5SXX2TJJGSJLPI27EZPIA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16SmJMbycEDFbqMUkmhZ5bPJHdcehG16mQdgN4s6fHZ7VcshPT): protocol: expected piece hash; context canceled; EOF
Jan 4 03:57:17.175592334 (708h28m17.046675379s) [noise=true , address=64.49.13.138:4481 , piece_id=AIXOG6H7KPQWWYJHX4LTBY3AQC3X7RB5SXX2TJJGSJLPI27EZPIA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.175705960 (708h28m17.046789005s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Zs9PFmdpQVHe1UjYapzPRUpsopENKECrNUhVVBZ5nmhpGA9Ws): protocol: expected piece hash; context canceled; EOF
Jan 4 03:57:17.175740967 (708h28m17.046824012s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.175974972 (708h28m17.047058018s) [noise=true , address=145.79.227.48:4309 , piece_id=ZKQP3WF6WGDG46N76YSANGNPZIK5RGF3PDTBZ7G2IDOP4MQ7R2EA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12txGN8RDcyMycMhkRtdZAUkMK9L2ZnqPihEGZ2YNXJvu89RUU1): protocol: write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection; write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection
Jan 4 03:57:17.175987155 (708h28m17.04707019s) [noise=true , address=145.79.227.48:4309 , piece_id=ZKQP3WF6WGDG46N76YSANGNPZIK5RGF3PDTBZ7G2IDOP4MQ7R2EA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.176093107 (708h28m17.047176162s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu): protocol: read tcp 188.166.225.170:50396->45.91.116.8:4050: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 03:57:17.176103206 (708h28m17.047186251s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.176124888 (708h28m17.047207923s) [seg_pos={0 0}, upload=620262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 03:57:17.176234506 (708h28m17.047317561s) [upload=620262]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:57:17.176242932 (708h28m17.047325977s) [upload=620262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:57:17.502405262 (708h28m17.373488317s) [upload=620262]: Done uploading segments...
Jan 4 03:57:17.502431651 (708h28m17.373514696s) [upload=620262]: Upload finished. err: <nil>
Duration: 15.336574155s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.X.XX...XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..XX] duration:
30.004553771s trace:
https://jaeger.dev.storj.io/trace/1dac913198e4bce8
FAIL (2026-01-04 03:58:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA) context deadline exceeded
======LOGS======
load=620274]: Uploading segments...
Jan 4 03:57:59.230697680 (708h28m59.101780715s) [upload=620274]: Got next segment. Inline: false
Jan 4 03:57:59.230737095 (708h28m59.10182013s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 03:57:59.230743196 (708h28m59.101826231s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 03:57:59.478585542 (708h28m59.349668597s) [seg_pos={0 0}, upload=620274]: Begin upload segment...
Jan 4 03:57:59.478645196 (708h28m59.349728241s) [seg_pos={0 0}, upload=620274]: Done begin upload segment.
Jan 4 03:57:59.478682467 (708h28m59.349765512s) [noise=true , address=145.79.227.9:4365 , piece_id=24OOJNJUYJN7D4DMTASKANSTETTAJ662SB753SKR4F5JMTCBMKRQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.478693137 (708h28m59.349776182s) [noise=true , address=45.91.116.11:4119 , piece_id=FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.478881116 (708h28m59.349964161s) [noise=true , address=145.79.227.16:4911 , piece_id=FE56N37GPUKGEDPFIVSCUAQOLJPG5K2FQIJGQJBRVEROTE6IAZNQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.479549548 (708h28m59.350632613s) [noise=true , address=145.79.227.48:4303 , piece_id=MSIS6IJ4YWYQQFMKGF34SS6CFGH2BWMGHDSDJM335FRGM5JNFYXA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.479692530 (708h28m59.350775575s) [noise=true , address=145.79.227.51:4409 , piece_id=JBB7ZKOWH3WPY4LMWK4TIM3UE33C7VNF2DRGSJH7AB5V5HDLUCHQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480064850 (708h28m59.351147885s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480407532 (708h28m59.351490567s) [noise=true , address=145.79.227.4:4186 , piece_id=KUBCVDGZYTGDGVONIWGI7BHURGYIFRL52RQZ2JAHBA7HUTEVCQQA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480636718 (708h28m59.351719763s) [noise=true , address=45.91.116.13:4153 , piece_id=SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.480779089 (708h28m59.351862134s) [noise=true , address=145.79.227.13:4804 , piece_id=FELWRL54QLTY4A45JT4ECWBYSDUM2EUDDQ7PCO7XGKY3P4MIMPFQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481139536 (708h28m59.352222571s) [noise=true , address=45.91.116.12:4226 , piece_id=MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481208928 (708h28m59.352291963s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.481668233 (708h28m59.352751278s) [noise=true , address=145.79.227.42:4122 , piece_id=XBJ7WEX7MER4YP6GELQGK5GFE5ZDRX3W2JZVRJXODFRTVLGTM3MQ, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.482247114 (708h28m59.353330149s) [noise=true , address=145.79.227.37:4566 , piece_id=WRO3PDTBY4YVRCS56E7GWSYFZMUXJ6D7KIM76QED2VE3AV46SG6A, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.484921106 (708h28m59.356004141s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.485409376 (708h28m59.356492421s) [noise=true , address=145.79.227.22:4510 , piece_id=B7SQLJUEOCMEDCA56ODLID2IUJKSLL224QEHOSQDL6RZCKVG66BA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:57:59.485914699 (708h28m59.356997734s) [noise=true , address=145.79.227.20:4438 , piece_id=SOPZ32JMEACTQWF5P3CQSBLBA2CFXHJRIZNY7RWPUEPZCQYRFILA, seg_pos={0 0}, upload=620274]: Uploading piece...
Jan 4 03:58:01.190842079 (708h29m1.061925124s) [upload=620274]: Next returned nil segment
Jan 4 03:58:01.190858711 (708h29m1.061941746s) [upload=620274]: Waiting for error group managing segments...
Jan 4 03:58:01.475552237 (708h29m1.346635282s) [noise=true , address=145.79.227.37:4566 , piece_id=WRO3PDTBY4YVRCS56E7GWSYFZMUXJ6D7KIM76QED2VE3AV46SG6A, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.477087481 (708h29m1.348170526s) [noise=true , address=145.79.227.9:4365 , piece_id=24OOJNJUYJN7D4DMTASKANSTETTAJ662SB753SKR4F5JMTCBMKRQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.483350486 (708h29m1.354433532s) [noise=true , address=145.79.227.48:4303 , piece_id=MSIS6IJ4YWYQQFMKGF34SS6CFGH2BWMGHDSDJM335FRGM5JNFYXA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.484149919 (708h29m1.355232974s) [noise=true , address=145.79.227.13:4804 , piece_id=FELWRL54QLTY4A45JT4ECWBYSDUM2EUDDQ7PCO7XGKY3P4MIMPFQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.492725057 (708h29m1.363808102s) [noise=true , address=145.79.227.4:4186 , piece_id=KUBCVDGZYTGDGVONIWGI7BHURGYIFRL52RQZ2JAHBA7HUTEVCQQA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.493582259 (708h29m1.364665314s) [noise=true , address=145.79.227.22:4510 , piece_id=B7SQLJUEOCMEDCA56ODLID2IUJKSLL224QEHOSQDL6RZCKVG66BA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.498938428 (708h29m1.370021463s) [noise=true , address=145.79.227.16:4911 , piece_id=FE56N37GPUKGEDPFIVSCUAQOLJPG5K2FQIJGQJBRVEROTE6IAZNQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.537908171 (708h29m1.408991217s) [noise=true , address=145.79.227.51:4409 , piece_id=JBB7ZKOWH3WPY4LMWK4TIM3UE33C7VNF2DRGSJH7AB5V5HDLUCHQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.566501758 (708h29m1.437584803s) [noise=true , address=145.79.227.42:4122 , piece_id=XBJ7WEX7MER4YP6GELQGK5GFE5ZDRX3W2JZVRJXODFRTVLGTM3MQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:01.884314868 (708h29m1.755397914s) [noise=true , address=45.91.116.13:4153 , piece_id=SIRHLHIGF5D7K4ZSVDOUB23UCK7H2W3Z6DQUEOVZNGB6W63TUXTA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.000869986 (708h29m1.871953041s) [noise=true , address=145.79.227.20:4438 , piece_id=SOPZ32JMEACTQWF5P3CQSBLBA2CFXHJRIZNY7RWPUEPZCQYRFILA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.027163753 (708h29m1.898246798s) [noise=true , address=45.91.116.11:4119 , piece_id=FYXNRH4PTBEVP7NFURHQQNZIE3J75O47DSRC6DYRIHSCPRSZ7IUA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.236344783 (708h29m2.107427828s) [noise=true , address=45.91.116.12:4226 , piece_id=MNKQOSCWRZUA6EJV5V4Z4MITGBHP65RW3VCOKLHNPWHOCARAJ7FQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: <nil>
Jan 4 03:58:02.236373517 (708h29m2.107456573s) [seg_pos={0 0}, upload=620274]: Segment reached optimal threshold of 13 pieces.
Jan 4 03:58:02.236638382 (708h29m2.107721427s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55062->77.90.31.14:4664: use of closed network connection
Jan 4 03:58:02.236649844 (708h29m2.107732889s) [noise=true , address=77.90.31.14:4664 , piece_id=7OGKT546AYANWDTXNWZDPIBY5PU3Y6JIJLOEWZFGYC4QANLZUZ3A, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.236957099 (708h29m2.108040144s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 03:58:02.236969793 (708h29m2.108052838s) [noise=true , address=145.79.227.34:4907 , piece_id=PLQJ4E2WED26MP76JSWHZ245VC6S5Y5GJSC6VKESSKPDKS3CHFFA, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.237049936 (708h29m2.108132971s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39388->77.90.31.10:4519: use of closed network connection
Jan 4 03:58:02.237062439 (708h29m2.108145485s) [noise=true , address=77.90.31.10:4519 , piece_id=LUC77PRP7UW72CAG7OOJG5CMQ3LERFKPWKP4U2QRAIVTIV63AJFQ, seg_pos={0 0}, upload=620274]: Long tail cancellation due to optimal threshold
Jan 4 03:58:02.237084251 (708h29m2.108167296s) [seg_pos={0 0}, upload=620274]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 03:58:02.237175615 (708h29m2.10825866s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:58:02.237181716 (708h29m2.108264751s) [upload=620274]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:58:02.533167946 (708h29m2.404250991s) [upload=620274]: Done uploading segments...
Jan 4 03:58:02.533194517 (708h29m2.404277562s) [upload=620274]: Upload finished. err: <nil>
Duration: 30.004576315s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
35 out of 60 is failed (threshold=3) [.XXXX.XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.X] duration:
1m0.00380797s trace:
https://jaeger.dev.storj.io/trace/317f2fa634aadcb8
FAIL (2026-01-04 03:58:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.500836722 (708h28m13.371919767s) [noise=true , address=145.79.227.40:4075 , piece_id=64V3PD4673C3HHVK7OV4AZDYCQCI2FAJQJQR6I57NOJUOKNX54YQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.501370288 (708h28m13.372453343s) [noise=true , address=77.90.31.21:4908 , piece_id=JP4AOX5Y5EHBBM5LGO4BEZN2VP2IYMINTK4SAUTQTLK7DV2IEVMA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.503483382 (708h28m13.374566417s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.506809435 (708h28m13.37789248s) [noise=true , address=77.90.31.19:4850 , piece_id=2PXGLWVHEZMTVZT5FJMFA5FEZHRSIIQW5ZXFILR2CESYXFJXBJAQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.508094151 (708h28m13.379177196s) [noise=true , address=145.79.227.34:4928 , piece_id=LYY6BBHNNDFRJRGR7L45SEMYQUGEGXDUIQDYACTKFAJT3YADCZ3A, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.510791498 (708h28m13.381874543s) [noise=true , address=77.90.31.18:4839 , piece_id=RMRIZHVQGHRSWJDN2PESTDCL5XN2AZE5B4AKBPV3XM6FADFUSXXQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.512222142 (708h28m13.383305187s) [noise=true , address=64.49.13.26:4946 , piece_id=U6PYY2KRWXF5DTWDRAVSQC3QZQPYH2HCFBMWYJBMN6B2RO4AE2DA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.514224324 (708h28m13.385307379s) [noise=true , address=77.90.31.12:4606 , piece_id=OP3W5AWMNKTYRBIBSNFRWNWYCIJXFTN3265XZNHIM37MEJB2ZNSA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.514473418 (708h28m13.385556463s) [noise=true , address=145.79.227.44:4217 , piece_id=AX5UJBRBB6ZMQ6KKZNXY6PS2KAVQBQ6A6QABH3IGWEGDTJRZJTMA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.516818914 (708h28m13.387901959s) [noise=true , address=145.79.227.25:4645 , piece_id=CRE6TPTH6SPVNV4HK7V6VIX6ITYBDRCDECE6E2DQGZ62SI3IG6ZQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.517063730 (708h28m13.388146775s) [noise=true , address=145.79.227.43:4164 , piece_id=7CDVHBK47GVX4QRH3GIDKEVSIKNINJQCT2EQQTAW3NHNURNTEGYQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.519958112 (708h28m13.391041167s) [noise=true , address=145.79.227.29:4962 , piece_id=CWXFZMN45ZRDQCCH5SJVFD7FHEZK4UXRXLUVXTJB7HTPYLEH7CZA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.520783343 (708h28m13.391866388s) [noise=true , address=145.79.227.35:4484 , piece_id=EVGCRX7ZYXABOF3EABL3FDQJX5GWPYX4RWJZLY5PVMYVXHPWSVXQ, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.523536816 (708h28m13.394619861s) [noise=true , address=145.79.227.16:4932 , piece_id=M2AGJU2A6PYD2SHFVCY54H5ZHV3EF64IG32OFMOF7G2IXTSA6XQA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:13.524736241 (708h28m13.395819286s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Uploading piece...
Jan 4 03:57:14.917200007 (708h28m14.788283052s) [upload=620262]: Next returned nil segment
Jan 4 03:57:14.917216399 (708h28m14.788299444s) [upload=620262]: Waiting for error group managing segments...
Jan 4 03:57:15.330420287 (708h28m15.201503322s) [noise=true , address=64.49.13.26:4946 , piece_id=U6PYY2KRWXF5DTWDRAVSQC3QZQPYH2HCFBMWYJBMN6B2RO4AE2DA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.382535652 (708h28m15.253618687s) [noise=true , address=145.79.227.11:4754 , piece_id=OTUTAUK7V6JLXAUR5IFJPVOS3DEJGHBTPAQ5MVR5C35SMDTZXP5Q, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.401033114 (708h28m15.272116149s) [noise=true , address=145.79.227.43:4164 , piece_id=7CDVHBK47GVX4QRH3GIDKEVSIKNINJQCT2EQQTAW3NHNURNTEGYQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.406385305 (708h28m15.27746834s) [noise=true , address=145.79.227.16:4932 , piece_id=M2AGJU2A6PYD2SHFVCY54H5ZHV3EF64IG32OFMOF7G2IXTSA6XQA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.424319284 (708h28m15.295402329s) [noise=true , address=145.79.227.40:4075 , piece_id=64V3PD4673C3HHVK7OV4AZDYCQCI2FAJQJQR6I57NOJUOKNX54YQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.564462616 (708h28m15.435545651s) [noise=true , address=145.79.227.25:4645 , piece_id=CRE6TPTH6SPVNV4HK7V6VIX6ITYBDRCDECE6E2DQGZ62SI3IG6ZQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.801676355 (708h28m15.67275939s) [noise=true , address=77.90.31.12:4606 , piece_id=OP3W5AWMNKTYRBIBSNFRWNWYCIJXFTN3265XZNHIM37MEJB2ZNSA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:15.802560438 (708h28m15.673643473s) [noise=true , address=77.90.31.21:4908 , piece_id=JP4AOX5Y5EHBBM5LGO4BEZN2VP2IYMINTK4SAUTQTLK7DV2IEVMA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.244067610 (708h28m16.115150905s) [noise=true , address=145.79.227.35:4484 , piece_id=EVGCRX7ZYXABOF3EABL3FDQJX5GWPYX4RWJZLY5PVMYVXHPWSVXQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.276851114 (708h28m16.147934159s) [noise=true , address=145.79.227.34:4928 , piece_id=LYY6BBHNNDFRJRGR7L45SEMYQUGEGXDUIQDYACTKFAJT3YADCZ3A, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.358570980 (708h28m16.229654016s) [noise=true , address=145.79.227.44:4217 , piece_id=AX5UJBRBB6ZMQ6KKZNXY6PS2KAVQBQ6A6QABH3IGWEGDTJRZJTMA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.417875835 (708h28m16.28895887s) [noise=true , address=77.90.31.18:4839 , piece_id=RMRIZHVQGHRSWJDN2PESTDCL5XN2AZE5B4AKBPV3XM6FADFUSXXQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:16.574599499 (708h28m16.445682534s) [noise=true , address=145.79.227.29:4962 , piece_id=CWXFZMN45ZRDQCCH5SJVFD7FHEZK4UXRXLUVXTJB7HTPYLEH7CZA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:17.175314254 (708h28m17.046397299s) [noise=true , address=77.90.31.19:4850 , piece_id=2PXGLWVHEZMTVZT5FJMFA5FEZHRSIIQW5ZXFILR2CESYXFJXBJAQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: <nil>
Jan 4 03:57:17.175338430 (708h28m17.046421475s) [seg_pos={0 0}, upload=620262]: Segment reached optimal threshold of 14 pieces.
Jan 4 03:57:17.175581002 (708h28m17.046664037s) [noise=true , address=64.49.13.138:4481 , piece_id=AIXOG6H7KPQWWYJHX4LTBY3AQC3X7RB5SXX2TJJGSJLPI27EZPIA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16SmJMbycEDFbqMUkmhZ5bPJHdcehG16mQdgN4s6fHZ7VcshPT): protocol: expected piece hash; context canceled; EOF
Jan 4 03:57:17.175592334 (708h28m17.046675379s) [noise=true , address=64.49.13.138:4481 , piece_id=AIXOG6H7KPQWWYJHX4LTBY3AQC3X7RB5SXX2TJJGSJLPI27EZPIA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.175705960 (708h28m17.046789005s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Zs9PFmdpQVHe1UjYapzPRUpsopENKECrNUhVVBZ5nmhpGA9Ws): protocol: expected piece hash; context canceled; EOF
Jan 4 03:57:17.175740967 (708h28m17.046824012s) [noise=true , address=45.91.116.12:4245 , piece_id=2M6AUTFBKLYDERG6YPQA66HZQ6IPPBRXY2XE4IGRWQSNKSBW7OHA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.175974972 (708h28m17.047058018s) [noise=true , address=145.79.227.48:4309 , piece_id=ZKQP3WF6WGDG46N76YSANGNPZIK5RGF3PDTBZ7G2IDOP4MQ7R2EA, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12txGN8RDcyMycMhkRtdZAUkMK9L2ZnqPihEGZ2YNXJvu89RUU1): protocol: write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection; write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53940->145.79.227.48:4309: use of closed network connection
Jan 4 03:57:17.175987155 (708h28m17.04707019s) [noise=true , address=145.79.227.48:4309 , piece_id=ZKQP3WF6WGDG46N76YSANGNPZIK5RGF3PDTBZ7G2IDOP4MQ7R2EA, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.176093107 (708h28m17.047176162s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu): protocol: read tcp 188.166.225.170:50396->45.91.116.8:4050: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 03:57:17.176103206 (708h28m17.047186251s) [noise=true , address=45.91.116.8:4050 , piece_id=SGGGIQQDZH7JSGKGFSNNH4SIOZBMNUB73PPK3KXZTTYJABSF5VKQ, seg_pos={0 0}, upload=620262]: Long tail cancellation due to optimal threshold
Jan 4 03:57:17.176124888 (708h28m17.047207923s) [seg_pos={0 0}, upload=620262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 03:57:17.176234506 (708h28m17.047317561s) [upload=620262]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:57:17.176242932 (708h28m17.047325977s) [upload=620262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:57:17.502405262 (708h28m17.373488317s) [upload=620262]: Done uploading segments...
Jan 4 03:57:17.502431651 (708h28m17.373514696s) [upload=620262]: Upload finished. err: <nil>
Duration: 1m0.003848839s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X.X.XX...XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..X] duration:
30.002848631s trace:
https://jaeger.dev.storj.io/trace/456db84ea9f0bb0f
FAIL (2026-01-04 03:57:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
4 03:57:17.416685589 (708h28m17.287768634s) [upload=620264]: Starting upload
Jan 4 03:57:17.416758047 (708h28m17.287841122s) [upload=620264]: Done starting upload
Jan 4 03:57:17.416848840 (708h28m17.287931875s) [upload=620264]: Uploading segments...
Jan 4 03:57:17.417024705 (708h28m17.28810774s) [upload=620264]: Got next segment. Inline: false
Jan 4 03:57:17.417074520 (708h28m17.288157555s) [upload=620264]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 03:57:17.417080270 (708h28m17.288163315s) [upload=620264]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 03:57:17.662666231 (708h28m17.533749276s) [seg_pos={0 0}, upload=620264]: Begin upload segment...
Jan 4 03:57:17.662711427 (708h28m17.533794472s) [seg_pos={0 0}, upload=620264]: Done begin upload segment.
Jan 4 03:57:17.662754980 (708h28m17.533838015s) [noise=true , address=45.91.116.10:4082 , piece_id=JNTGL3YMCSGRGPWP4VXD2R45NW3YFOL4FKVXMPPPEAW6MQRKH33A, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.662759138 (708h28m17.533842183s) [noise=true , address=145.79.227.7:4293 , piece_id=LBMEEOV2GPY2G52EY4AKHE54TV5SMUKLAW7QAWN3B36IXONOM3MQ, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.663180430 (708h28m17.534263465s) [noise=true , address=145.79.227.13:4803 , piece_id=JQGXFYZF5W6N7B3IBDGKLUZQ3KM23UEFLHUGAHC3G3SB3VULWZMQ, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.663414897 (708h28m17.534497932s) [noise=true , address=145.79.227.39:4014 , piece_id=MOK3H3GNG3FJWAHGBO4Q3ZD7S4HP62OQSJLXM5XZDOL2OKHSMSAQ, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.663523243 (708h28m17.534606278s) [noise=true , address=77.90.31.17:4773 , piece_id=YWOWPHPUJTUSRITONHZMYR55QQR6IXTVN7OFFZ6UBPO6GMGMVXIQ, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.663705560 (708h28m17.534788595s) [noise=true , address=145.79.227.21:4474 , piece_id=WXUTKUBOGU3E6N3J5O62GOLWMH3X5OASK33X54CDIX6HK6KLLGPA, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.663885953 (708h28m17.534968998s) [noise=true , address=45.91.116.14:4189 , piece_id=7P22OEUMZX7KQL3KTVQW7CQMW2PVGGY7RVTZFYD25TMVLETKDURQ, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.664065385 (708h28m17.53514842s) [noise=true , address=145.79.227.26:4653 , piece_id=VLFK7JWRMQ6WT76E4S4VPX5DQ7EHLQT67BF37MHCMNROR6XSHCUQ, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.664198498 (708h28m17.535281533s) [noise=true , address=45.91.116.13:4155 , piece_id=WDK5EZ6SPRUAM55ONWJ6HCVOC6FILUYJJMSJSA3GSFSK54OMLOUA, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.664467591 (708h28m17.535550897s) [noise=true , address=77.90.31.10:4518 , piece_id=R4XGKCVP2KQOJJ5MFQ5H7JG2TCM6QELB6SJD4BVJYODFEANFECDA, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.664923980 (708h28m17.536007015s) [noise=true , address=77.90.31.21:4877 , piece_id=EWGLFAPZ53FVAJDUI54ZEBDLD266JVO6IGBUYJ2VLPN7VP5BYVSQ, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.665179186 (708h28m17.536262221s) [noise=true , address=145.79.227.52:4447 , piece_id=RZJ4Y4JUGEFFPX2VQAKADJCZ4SJAQZBCTRM32NPYV64KPNJJD5GA, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.665536015 (708h28m17.53661906s) [noise=true , address=145.79.227.42:4121 , piece_id=ELZHAUI23PRG4VFSTIDNRLFQGDAA4ZPIIOJ5327O5XEXW2AZZPOA, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.665570661 (708h28m17.536653696s) [noise=true , address=145.79.227.45:4231 , piece_id=DR3XPZ6JFSVC7CF2XE3H55S77EHJLHE3TOOHBNH3KJYG6BK3QT4Q, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.666065644 (708h28m17.537148679s) [noise=true , address=45.91.116.5:4300 , piece_id=THEH3OYOPVD4B6RVR2UVAXRFFGMEYPDVUQAM75F2C5KCLJZO4G7A, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:17.666449505 (708h28m17.53753254s) [noise=true , address=145.79.227.4:4187 , piece_id=O7WOGCO6GYRXPH2TVDF4QOERLCMXI34UQZPAWZ23SGAFIQAQHSIQ, seg_pos={0 0}, upload=620264]: Uploading piece...
Jan 4 03:57:19.370079947 (708h28m19.241162982s) [upload=620264]: Next returned nil segment
Jan 4 03:57:19.370097149 (708h28m19.241180184s) [upload=620264]: Waiting for error group managing segments...
Jan 4 03:57:19.657153666 (708h28m19.528236711s) [noise=true , address=145.79.227.42:4121 , piece_id=ELZHAUI23PRG4VFSTIDNRLFQGDAA4ZPIIOJ5327O5XEXW2AZZPOA, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:19.658226087 (708h28m19.529309142s) [noise=true , address=145.79.227.7:4293 , piece_id=LBMEEOV2GPY2G52EY4AKHE54TV5SMUKLAW7QAWN3B36IXONOM3MQ, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:19.660908886 (708h28m19.531991931s) [noise=true , address=145.79.227.21:4474 , piece_id=WXUTKUBOGU3E6N3J5O62GOLWMH3X5OASK33X54CDIX6HK6KLLGPA, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:19.661574815 (708h28m19.53265785s) [noise=true , address=145.79.227.26:4653 , piece_id=VLFK7JWRMQ6WT76E4S4VPX5DQ7EHLQT67BF37MHCMNROR6XSHCUQ, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:19.671918109 (708h28m19.543001154s) [noise=true , address=145.79.227.13:4803 , piece_id=JQGXFYZF5W6N7B3IBDGKLUZQ3KM23UEFLHUGAHC3G3SB3VULWZMQ, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:19.679803684 (708h28m19.550886729s) [noise=true , address=145.79.227.39:4014 , piece_id=MOK3H3GNG3FJWAHGBO4Q3ZD7S4HP62OQSJLXM5XZDOL2OKHSMSAQ, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:19.707286285 (708h28m19.57836932s) [noise=true , address=145.79.227.52:4447 , piece_id=RZJ4Y4JUGEFFPX2VQAKADJCZ4SJAQZBCTRM32NPYV64KPNJJD5GA, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:19.741377299 (708h28m19.612460354s) [noise=true , address=145.79.227.45:4231 , piece_id=DR3XPZ6JFSVC7CF2XE3H55S77EHJLHE3TOOHBNH3KJYG6BK3QT4Q, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:20.368648903 (708h28m20.239731938s) [noise=true , address=77.90.31.10:4518 , piece_id=R4XGKCVP2KQOJJ5MFQ5H7JG2TCM6QELB6SJD4BVJYODFEANFECDA, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:20.489259281 (708h28m20.360342316s) [noise=true , address=45.91.116.10:4082 , piece_id=JNTGL3YMCSGRGPWP4VXD2R45NW3YFOL4FKVXMPPPEAW6MQRKH33A, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:21.000478634 (708h28m20.871561709s) [noise=true , address=77.90.31.21:4877 , piece_id=EWGLFAPZ53FVAJDUI54ZEBDLD266JVO6IGBUYJ2VLPN7VP5BYVSQ, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:21.162227577 (708h28m21.033310622s) [noise=true , address=45.91.116.14:4189 , piece_id=7P22OEUMZX7KQL3KTVQW7CQMW2PVGGY7RVTZFYD25TMVLETKDURQ, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:21.316298940 (708h28m21.187381975s) [noise=true , address=45.91.116.5:4300 , piece_id=THEH3OYOPVD4B6RVR2UVAXRFFGMEYPDVUQAM75F2C5KCLJZO4G7A, seg_pos={0 0}, upload=620264]: Done uploading piece. err: <nil>
Jan 4 03:57:21.316319550 (708h28m21.187402585s) [seg_pos={0 0}, upload=620264]: Segment reached optimal threshold of 13 pieces.
Jan 4 03:57:21.316508479 (708h28m21.187591514s) [noise=true , address=145.79.227.4:4187 , piece_id=O7WOGCO6GYRXPH2TVDF4QOERLCMXI34UQZPAWZ23SGAFIQAQHSIQ, seg_pos={0 0}, upload=620264]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:40758->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:40758->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40758->145.79.227.4:4187: use of closed network connection
Jan 4 03:57:21.316514310 (708h28m21.187597345s) [noise=true , address=145.79.227.4:4187 , piece_id=O7WOGCO6GYRXPH2TVDF4QOERLCMXI34UQZPAWZ23SGAFIQAQHSIQ, seg_pos={0 0}, upload=620264]: Long tail cancellation due to optimal threshold
Jan 4 03:57:21.316660729 (708h28m21.187743764s) [noise=true , address=45.91.116.13:4155 , piece_id=WDK5EZ6SPRUAM55ONWJ6HCVOC6FILUYJJMSJSA3GSFSK54OMLOUA, seg_pos={0 0}, upload=620264]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Jan 4 03:57:21.316665869 (708h28m21.187748914s) [noise=true , address=45.91.116.13:4155 , piece_id=WDK5EZ6SPRUAM55ONWJ6HCVOC6FILUYJJMSJSA3GSFSK54OMLOUA, seg_pos={0 0}, upload=620264]: Long tail cancellation due to optimal threshold
Jan 4 03:57:21.316772262 (708h28m21.187855297s) [noise=true , address=77.90.31.17:4773 , piece_id=YWOWPHPUJTUSRITONHZMYR55QQR6IXTVN7OFFZ6UBPO6GMGMVXIQ, seg_pos={0 0}, upload=620264]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Jan 4 03:57:21.316777722 (708h28m21.187860757s) [noise=true , address=77.90.31.17:4773 , piece_id=YWOWPHPUJTUSRITONHZMYR55QQR6IXTVN7OFFZ6UBPO6GMGMVXIQ, seg_pos={0 0}, upload=620264]: Long tail cancellation due to optimal threshold
Jan 4 03:57:21.316793682 (708h28m21.187876727s) [seg_pos={0 0}, upload=620264]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 03:57:21.316883924 (708h28m21.187966969s) [upload=620264]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:57:21.316890166 (708h28m21.187973211s) [upload=620264]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:57:21.613466048 (708h28m21.484549093s) [upload=620264]: Done uploading segments...
Jan 4 03:57:21.613491206 (708h28m21.484574251s) [upload=620264]: Upload finished. err: <nil>
Duration: 30.002862347s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXX..X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXXX] duration:
5.001784622s trace:
https://jaeger.dev.storj.io/trace/5b45b9c32fbdac2
FAIL (2026-01-04 03:57:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001818096s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXXX..X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXXX] duration:
5.001360112s trace:
https://jaeger.dev.storj.io/trace/459634bcabdd56cc
FAIL (2026-01-04 03:57:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001370963s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [.XXXX..X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXXX] duration:
5.000816879s trace:
https://jaeger.dev.storj.io/trace/4acc8b5318c5e367
FAIL (2026-01-04 03:56:51 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000839351s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [..XXXX..X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXXX] duration:
5.001480449s trace:
https://jaeger.dev.storj.io/trace/7c27d8942cd49dba
FAIL (2026-01-04 03:56:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001495137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [..XXXX.XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X.] duration:
15.322893821s trace:
https://jaeger.dev.storj.io/trace/335f7a951c15fcc9
FAIL (2026-01-04 03:56:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2]: Uploading piece...
Jan 4 03:54:59.665964167 (708h25m59.537047212s) [noise=true , address=64.49.13.170:4722 , piece_id=SNPTYRFXTVQFSP6ODBJSNPSVQCWUWRDNE2HDSLJZH6QEJ7SBTDHQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.666615126 (708h25m59.537698171s) [noise=true , address=145.79.227.32:4868 , piece_id=TBI4X5ODKX7RHDDI7PPMDU4RRSX3AH7ITM5AVA3ONJVYWO57GIJQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.667510672 (708h25m59.538593717s) [noise=true , address=145.79.227.30:4790 , piece_id=ERARTNVKLOQUMXLI62UZN4F22NRSBAQGXYVWKWWJR4GJDJVZAN3A, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.667669865 (708h25m59.53875291s) [noise=true , address=145.79.227.27:4715 , piece_id=TLBIIAGXYIWYY4K4URWOGCZI4CC4FMREEVG5PCK44YBUKWCNKR5Q, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.668188091 (708h25m59.539271136s) [noise=true , address=64.49.13.155:4917 , piece_id=WBHQFRLYUDRRU4QZHHG7ACHNSDR2KWEKSUCNZFJFTZTOLCNHULNA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.668604935 (708h25m59.53968798s) [noise=true , address=145.79.227.43:4170 , piece_id=SLCRKGUE53KQD7LINKLR6OSJOAIJYB27O6HIXBABKI5PD2TRBRVA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.668657135 (708h25m59.53974017s) [noise=true , address=145.79.227.49:4355 , piece_id=SBWATWUIHAUSGJLX2IUZUSSGBFNR6JGZNG5RW7F5VDEN2YADJQLA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669031698 (708h25m59.540114733s) [noise=true , address=45.91.116.10:4113 , piece_id=PPJZYL6EPEJEAWBY3BKUL5R5PE64MKEFQP66BNEONVT7V4KJUTLA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669172146 (708h25m59.540255191s) [noise=true , address=145.79.227.35:4491 , piece_id=4TWONEFVZXBUNUOZN7ZF7653ZMNMBLKDADBQZGQHE6AKGG7JK2EA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669452179 (708h25m59.540535214s) [noise=true , address=77.90.31.16:4743 , piece_id=24YEQADNWVQHVMTPJ63HFYWDRCSESE6FFQHNN32P2SSDRQ6F62GA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669669793 (708h25m59.540752838s) [noise=true , address=145.79.227.7:4297 , piece_id=RNQGX6GRG6GL3EM22VKPY3BYEARINRKLK3PDGFXZAKAHHFH6JBGQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669994171 (708h25m59.541077216s) [noise=true , address=45.91.116.13:4163 , piece_id=I3AH64TGSZBQCM6XT6PYJEDEAWZOLUI35HGAQNXT44QZN6OJFMNQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.670368083 (708h25m59.541451118s) [noise=true , address=145.79.227.29:4956 , piece_id=HZBIKPID75T7CMNJHMGKDMEDRE3ZD25ZFGVQSXKLCV34JTEVPEWQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.670441282 (708h25m59.541524327s) [noise=true , address=77.90.31.21:4882 , piece_id=QVNUDOIIX3FC7CAYVW5ALQTTS6J7YUEK5KN7WYWYAPZFMTFOFKUQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.670914534 (708h25m59.541997579s) [noise=true , address=145.79.227.14:4870 , piece_id=M4GIOL4H5QKXWQAE3HKIT4N7G5UAFTZ7IOQSP65OICB54CIHR5YA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.671405448 (708h25m59.542488493s) [noise=true , address=77.90.31.11:4574 , piece_id=5CXKPREM3WEHOV653WAPKLQ3O5RC6L4P22BRUJDYPYIHL55ZMYSQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.671901012 (708h25m59.542984057s) [noise=true , address=77.90.31.13:4656 , piece_id=PYJGURNUIHXBRYHQTVJF4VD324YU56EOH55S5KBFCIWGAVZKLTJQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:55:01.109274522 (708h26m0.980357567s) [upload=620232]: Next returned nil segment
Jan 4 03:55:01.109294080 (708h26m0.980377125s) [upload=620232]: Waiting for error group managing segments...
Jan 4 03:55:01.490405950 (708h26m1.361488996s) [noise=true , address=64.49.13.155:4917 , piece_id=WBHQFRLYUDRRU4QZHHG7ACHNSDR2KWEKSUCNZFJFTZTOLCNHULNA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.543575672 (708h26m1.414658718s) [noise=true , address=64.49.13.170:4722 , piece_id=SNPTYRFXTVQFSP6ODBJSNPSVQCWUWRDNE2HDSLJZH6QEJ7SBTDHQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.543660865 (708h26m1.4147439s) [noise=true , address=145.79.227.7:4297 , piece_id=RNQGX6GRG6GL3EM22VKPY3BYEARINRKLK3PDGFXZAKAHHFH6JBGQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.587030196 (708h26m1.458113241s) [noise=true , address=145.79.227.27:4715 , piece_id=TLBIIAGXYIWYY4K4URWOGCZI4CC4FMREEVG5PCK44YBUKWCNKR5Q, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.598081249 (708h26m1.469164304s) [noise=true , address=145.79.227.38:4002 , piece_id=ZSTKF43VUA6QT3RI436BUR3HDY5MVK4HEIPC3Y66YVRRI6MZZLSQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.630902966 (708h26m1.501986011s) [noise=true , address=145.79.227.14:4870 , piece_id=M4GIOL4H5QKXWQAE3HKIT4N7G5UAFTZ7IOQSP65OICB54CIHR5YA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.796424984 (708h26m1.667508039s) [noise=true , address=77.90.31.13:4656 , piece_id=PYJGURNUIHXBRYHQTVJF4VD324YU56EOH55S5KBFCIWGAVZKLTJQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.907831917 (708h26m1.778914962s) [noise=true , address=77.90.31.21:4882 , piece_id=QVNUDOIIX3FC7CAYVW5ALQTTS6J7YUEK5KN7WYWYAPZFMTFOFKUQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.178142516 (708h26m2.049225571s) [noise=true , address=45.91.116.10:4113 , piece_id=PPJZYL6EPEJEAWBY3BKUL5R5PE64MKEFQP66BNEONVT7V4KJUTLA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.481641152 (708h26m2.352724197s) [noise=true , address=145.79.227.49:4355 , piece_id=SBWATWUIHAUSGJLX2IUZUSSGBFNR6JGZNG5RW7F5VDEN2YADJQLA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.589048459 (708h26m2.460131514s) [noise=true , address=145.79.227.35:4491 , piece_id=4TWONEFVZXBUNUOZN7ZF7653ZMNMBLKDADBQZGQHE6AKGG7JK2EA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.600182240 (708h26m2.471265285s) [noise=true , address=145.79.227.32:4868 , piece_id=TBI4X5ODKX7RHDDI7PPMDU4RRSX3AH7ITM5AVA3ONJVYWO57GIJQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.692843210 (708h26m2.563926275s) [noise=true , address=77.90.31.16:4743 , piece_id=24YEQADNWVQHVMTPJ63HFYWDRCSESE6FFQHNN32P2SSDRQ6F62GA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.772503363 (708h26m2.643586408s) [noise=true , address=145.79.227.30:4790 , piece_id=ERARTNVKLOQUMXLI62UZN4F22NRSBAQGXYVWKWWJR4GJDJVZAN3A, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.772830066 (708h26m2.643913111s) [seg_pos={0 0}, upload=620232]: Segment reached optimal threshold of 14 pieces.
Jan 4 03:55:02.773221080 (708h26m2.644304155s) [noise=true , address=145.79.227.29:4956 , piece_id=HZBIKPID75T7CMNJHMGKDMEDRE3ZD25ZFGVQSXKLCV34JTEVPEWQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pf65Tg83xM523bgdJK2BxiC6uzJ1Hh528DY8CUDxGTnxZSd2W): protocol: expected piece hash; context canceled; EOF
Jan 4 03:55:02.773233092 (708h26m2.644316148s) [noise=true , address=145.79.227.29:4956 , piece_id=HZBIKPID75T7CMNJHMGKDMEDRE3ZD25ZFGVQSXKLCV34JTEVPEWQ, seg_pos={0 0}, upload=620232]: Long tail cancellation due to optimal threshold
Jan 4 03:55:02.773631662 (708h26m2.644714717s) [noise=true , address=145.79.227.43:4170 , piece_id=SLCRKGUE53KQD7LINKLR6OSJOAIJYB27O6HIXBABKI5PD2TRBRVA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: expected piece hash; context canceled; EOF
Jan 4 03:55:02.773643935 (708h26m2.64472698s) [noise=true , address=145.79.227.43:4170 , piece_id=SLCRKGUE53KQD7LINKLR6OSJOAIJYB27O6HIXBABKI5PD2TRBRVA, seg_pos={0 0}, upload=620232]: Long tail cancellation due to optimal threshold
Jan 4 03:55:02.774162292 (708h26m2.645245357s) [noise=true , address=45.91.116.13:4163 , piece_id=I3AH64TGSZBQCM6XT6PYJEDEAWZOLUI35HGAQNXT44QZN6OJFMNQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRpjHEvHzBoYqq6DXKC5Z3SPsP4HEZz2tAiuheHkgzzcnsVU2): protocol: expected piece hash; context canceled; EOF
Jan 4 03:55:02.774173934 (708h26m2.645256989s) [noise=true , address=45.91.116.13:4163 , piece_id=I3AH64TGSZBQCM6XT6PYJEDEAWZOLUI35HGAQNXT44QZN6OJFMNQ, seg_pos={0 0}, upload=620232]: Long tail cancellation due to optimal threshold
Jan 4 03:55:02.775108143 (708h26m2.646191188s) [noise=true , address=77.90.31.11:4574 , piece_id=5CXKPREM3WEHOV653WAPKLQ3O5RC6L4P22BRUJDYPYIHL55ZMYSQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF
Jan 4 03:55:02.775155975 (708h26m2.64623903s) [noise=true , address=77.90.31.11:4574 , piece_id=5CXKPREM3WEHOV653WAPKLQ3O5RC6L4P22BRUJDYPYIHL55ZMYSQ, seg_pos={0 0}, upload=620232]: Long tail cancellation due to optimal threshold
Jan 4 03:55:02.775226549 (708h26m2.646309594s) [seg_pos={0 0}, upload=620232]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 03:55:02.775564712 (708h26m2.646647757s) [upload=620232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:55:02.775592576 (708h26m2.646675621s) [upload=620232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:55:03.076544249 (708h26m2.947627304s) [upload=620232]: Done uploading segments...
Jan 4 03:55:03.076573285 (708h26m2.94765633s) [upload=620232]: Upload finished. err: <nil>
Duration: 15.322933867s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.X.X.XX...XX....X.XX....XX...X...XX..X....X...XX.X...XX.XX..] duration:
5.744481652s trace:
https://jaeger.dev.storj.io/trace/19d4536160a96a90
FAIL (2026-01-04 03:56:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
03:54:26.371135946 (708h25m26.242218981s) [upload=620222]: Starting upload
Jan 4 03:54:26.371349913 (708h25m26.242432988s) [upload=620222]: Done starting upload
Jan 4 03:54:26.371539974 (708h25m26.242623009s) [upload=620222]: Uploading segments...
Jan 4 03:54:26.371626880 (708h25m26.242710466s) [upload=620222]: Got next segment. Inline: false
Jan 4 03:54:26.371675332 (708h25m26.242758377s) [upload=620222]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 03:54:26.371683297 (708h25m26.242766342s) [upload=620222]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 03:54:26.614033807 (708h25m26.485116852s) [seg_pos={0 0}, upload=620222]: Begin upload segment...
Jan 4 03:54:26.614091046 (708h25m26.485174081s) [seg_pos={0 0}, upload=620222]: Done begin upload segment.
Jan 4 03:54:26.614177270 (708h25m26.485260305s) [noise=true , address=145.79.227.21:4475 , piece_id=U7KK6CZ56PBBYZD7HNHHUAKKLXFKSKFQLPMKBK2L7T6SDLN2TP2A, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.614196827 (708h25m26.485279872s) [noise=true , address=77.90.31.10:4520 , piece_id=PMC2UA2NQI4LCCC5FWFUBF6WTHPP7TS34SQZLEMMJN5S4WKD33GQ, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.614817770 (708h25m26.485900805s) [noise=true , address=145.79.227.10:4403 , piece_id=4CUWJHUPX7OJ46PS3H4T62VN7F2O63KMSJKLI34Y4KV3H76ZJPRQ, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.615249712 (708h25m26.486332757s) [noise=true , address=145.79.227.44:4193 , piece_id=3PJ26L5H5AE7WM3CRBMBTRH5JB67RB66PMO3QX5RIEGYM7Y45TKA, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.615604117 (708h25m26.486687152s) [noise=true , address=45.91.116.4:4940 , piece_id=YGDQUHMSOEJNJ4JHAMXOVL4BORDFUX5WMOJVVK6FU6YOR3WUOU6Q, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.615932622 (708h25m26.487015658s) [noise=true , address=45.91.116.9:4263 , piece_id=346JASIIMLBZUSXK5W2EDCLHRSNINAOQSIDFOY6MTNQBSRLMXLVA, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.616247652 (708h25m26.487330687s) [noise=true , address=145.79.227.38:4979 , piece_id=DJOYZ36NJYCJPFFJ646URV475BEYUGF6GMTUOUNWLCZ4VNO7WKWQ, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.617603194 (708h25m26.488686229s) [noise=true , address=145.79.227.14:4838 , piece_id=F3G5CQLZM54NFGZI7HXVKNP5OV5ES4G75ZTWX2LXGOUVMULTQ7EQ, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.619886992 (708h25m26.490970027s) [noise=true , address=145.79.227.22:4510 , piece_id=VGVTZD4UAOS3JAB3RZ6ZDUNVS363QQ7AIFWIVYYXCHLKX6VFAPWA, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.622808705 (708h25m26.49389174s) [noise=true , address=77.90.31.14:4663 , piece_id=T2CZMYNMDF7IKVLHEBSUDAJDFS6FAD5TANTGIDF2W6NSYUYCNX2Q, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.624706970 (708h25m26.495790005s) [noise=true , address=145.79.227.37:4566 , piece_id=ITIY7CLWUQTISZQ2SSDCTDM7QBETGGXJVEXSDBGSLJYZ7PRVPIXQ, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.625854405 (708h25m26.49693745s) [noise=true , address=145.79.227.41:4087 , piece_id=UCZBKD2O35M52LBVGKAXTYKSYMYBC77BBSHCPSEK5GHTTBOBIUQA, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.627565072 (708h25m26.498648108s) [noise=true , address=145.79.227.12:4765 , piece_id=G4RGSRPDEMVD5UISK6MO4M6C7M2B36L6IU7RQFYTFSEPAV4HU6LA, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.628719410 (708h25m26.499802445s) [noise=true , address=145.79.227.43:4157 , piece_id=MQTWFH2N5EQFJO6LICGRBZIRC57O5HSEM4HK3UOONJSJPT5ZYXDA, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.631019650 (708h25m26.502102685s) [noise=true , address=145.79.227.46:4602 , piece_id=RVIOWNACJJA6PA7AOI4AGUMHBNZLDIVZBKXGA5YIUPTBYKYIOK7A, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:26.635507525 (708h25m26.50659056s) [noise=true , address=145.79.227.42:4123 , piece_id=WI2ORQJITIJPLKXENDYRPXVNNBJ5SUG3EWAB7JFDIAYTVXVEDU3Q, seg_pos={0 0}, upload=620222]: Uploading piece...
Jan 4 03:54:28.297171798 (708h25m28.168254833s) [upload=620222]: Next returned nil segment
Jan 4 03:54:28.297187107 (708h25m28.168270142s) [upload=620222]: Waiting for error group managing segments...
Jan 4 03:54:28.621389264 (708h25m28.492472309s) [noise=true , address=145.79.227.38:4979 , piece_id=DJOYZ36NJYCJPFFJ646URV475BEYUGF6GMTUOUNWLCZ4VNO7WKWQ, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.624113351 (708h25m28.495196386s) [noise=true , address=145.79.227.37:4566 , piece_id=ITIY7CLWUQTISZQ2SSDCTDM7QBETGGXJVEXSDBGSLJYZ7PRVPIXQ, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.636328570 (708h25m28.507411605s) [noise=true , address=145.79.227.42:4123 , piece_id=WI2ORQJITIJPLKXENDYRPXVNNBJ5SUG3EWAB7JFDIAYTVXVEDU3Q, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.639719577 (708h25m28.510802623s) [noise=true , address=145.79.227.41:4087 , piece_id=UCZBKD2O35M52LBVGKAXTYKSYMYBC77BBSHCPSEK5GHTTBOBIUQA, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.641461204 (708h25m28.512544239s) [noise=true , address=145.79.227.21:4475 , piece_id=U7KK6CZ56PBBYZD7HNHHUAKKLXFKSKFQLPMKBK2L7T6SDLN2TP2A, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.645250649 (708h25m28.516333684s) [noise=true , address=145.79.227.22:4510 , piece_id=VGVTZD4UAOS3JAB3RZ6ZDUNVS363QQ7AIFWIVYYXCHLKX6VFAPWA, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.655550381 (708h25m28.526633426s) [noise=true , address=145.79.227.46:4602 , piece_id=RVIOWNACJJA6PA7AOI4AGUMHBNZLDIVZBKXGA5YIUPTBYKYIOK7A, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.683966008 (708h25m28.555049053s) [noise=true , address=145.79.227.12:4765 , piece_id=G4RGSRPDEMVD5UISK6MO4M6C7M2B36L6IU7RQFYTFSEPAV4HU6LA, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.707833786 (708h25m28.578916831s) [noise=true , address=145.79.227.14:4838 , piece_id=F3G5CQLZM54NFGZI7HXVKNP5OV5ES4G75ZTWX2LXGOUVMULTQ7EQ, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:28.997111211 (708h25m28.868194256s) [noise=true , address=77.90.31.10:4520 , piece_id=PMC2UA2NQI4LCCC5FWFUBF6WTHPP7TS34SQZLEMMJN5S4WKD33GQ, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:29.055548514 (708h25m28.926631559s) [noise=true , address=45.91.116.4:4940 , piece_id=YGDQUHMSOEJNJ4JHAMXOVL4BORDFUX5WMOJVVK6FU6YOR3WUOU6Q, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:29.307842767 (708h25m29.178925812s) [noise=true , address=77.90.31.14:4663 , piece_id=T2CZMYNMDF7IKVLHEBSUDAJDFS6FAD5TANTGIDF2W6NSYUYCNX2Q, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:29.892767877 (708h25m29.763850912s) [noise=true , address=45.91.116.9:4263 , piece_id=346JASIIMLBZUSXK5W2EDCLHRSNINAOQSIDFOY6MTNQBSRLMXLVA, seg_pos={0 0}, upload=620222]: Done uploading piece. err: <nil>
Jan 4 03:54:29.892787925 (708h25m29.76387096s) [seg_pos={0 0}, upload=620222]: Segment reached optimal threshold of 13 pieces.
Jan 4 03:54:29.892958270 (708h25m29.764041305s) [noise=true , address=145.79.227.44:4193 , piece_id=3PJ26L5H5AE7WM3CRBMBTRH5JB67RB66PMO3QX5RIEGYM7Y45TKA, seg_pos={0 0}, upload=620222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Jan 4 03:54:29.892964552 (708h25m29.764047597s) [noise=true , address=145.79.227.44:4193 , piece_id=3PJ26L5H5AE7WM3CRBMBTRH5JB67RB66PMO3QX5RIEGYM7Y45TKA, seg_pos={0 0}, upload=620222]: Long tail cancellation due to optimal threshold
Jan 4 03:54:29.893220258 (708h25m29.764303303s) [noise=true , address=145.79.227.43:4157 , piece_id=MQTWFH2N5EQFJO6LICGRBZIRC57O5HSEM4HK3UOONJSJPT5ZYXDA, seg_pos={0 0}, upload=620222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Jan 4 03:54:29.893228645 (708h25m29.76431169s) [noise=true , address=145.79.227.43:4157 , piece_id=MQTWFH2N5EQFJO6LICGRBZIRC57O5HSEM4HK3UOONJSJPT5ZYXDA, seg_pos={0 0}, upload=620222]: Long tail cancellation due to optimal threshold
Jan 4 03:54:29.893272067 (708h25m29.764355112s) [noise=true , address=145.79.227.10:4403 , piece_id=4CUWJHUPX7OJ46PS3H4T62VN7F2O63KMSJKLI34Y4KV3H76ZJPRQ, seg_pos={0 0}, upload=620222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: write tcp 188.166.225.170:59084->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:59084->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59084->145.79.227.10:4403: use of closed network connection
Jan 4 03:54:29.893287446 (708h25m29.764370481s) [noise=true , address=145.79.227.10:4403 , piece_id=4CUWJHUPX7OJ46PS3H4T62VN7F2O63KMSJKLI34Y4KV3H76ZJPRQ, seg_pos={0 0}, upload=620222]: Long tail cancellation due to optimal threshold
Jan 4 03:54:29.893361106 (708h25m29.764444141s) [seg_pos={0 0}, upload=620222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 03:54:29.893463151 (708h25m29.764546186s) [upload=620222]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:54:29.893468330 (708h25m29.764551366s) [upload=620222]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:54:30.185219105 (708h25m30.05630215s) [upload=620222]: Done uploading segments...
Jan 4 03:54:30.185242910 (708h25m30.056325945s) [upload=620222]: Upload finished. err: <nil>
Duration: 5.744514754s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
18.111836424s trace:
https://jaeger.dev.storj.io/trace/1ee36f24231792d5
FAIL (2026-01-04 03:56:00 UTC)
orbiter warning: context deadline exceeded
Duration: 18.111863476s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
34 out of 60 is failed (threshold=3) [...XXXX.XXXXXX.X....X..XXXX.XXXX.XXX..XX.XX.X.XX..X..X.X...X] duration:
1m0.00335549s trace:
https://jaeger.dev.storj.io/trace/335f7a951c15fcc9
FAIL (2026-01-04 03:55:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2]: Uploading piece...
Jan 4 03:54:59.665964167 (708h25m59.537047212s) [noise=true , address=64.49.13.170:4722 , piece_id=SNPTYRFXTVQFSP6ODBJSNPSVQCWUWRDNE2HDSLJZH6QEJ7SBTDHQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.666615126 (708h25m59.537698171s) [noise=true , address=145.79.227.32:4868 , piece_id=TBI4X5ODKX7RHDDI7PPMDU4RRSX3AH7ITM5AVA3ONJVYWO57GIJQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.667510672 (708h25m59.538593717s) [noise=true , address=145.79.227.30:4790 , piece_id=ERARTNVKLOQUMXLI62UZN4F22NRSBAQGXYVWKWWJR4GJDJVZAN3A, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.667669865 (708h25m59.53875291s) [noise=true , address=145.79.227.27:4715 , piece_id=TLBIIAGXYIWYY4K4URWOGCZI4CC4FMREEVG5PCK44YBUKWCNKR5Q, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.668188091 (708h25m59.539271136s) [noise=true , address=64.49.13.155:4917 , piece_id=WBHQFRLYUDRRU4QZHHG7ACHNSDR2KWEKSUCNZFJFTZTOLCNHULNA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.668604935 (708h25m59.53968798s) [noise=true , address=145.79.227.43:4170 , piece_id=SLCRKGUE53KQD7LINKLR6OSJOAIJYB27O6HIXBABKI5PD2TRBRVA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.668657135 (708h25m59.53974017s) [noise=true , address=145.79.227.49:4355 , piece_id=SBWATWUIHAUSGJLX2IUZUSSGBFNR6JGZNG5RW7F5VDEN2YADJQLA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669031698 (708h25m59.540114733s) [noise=true , address=45.91.116.10:4113 , piece_id=PPJZYL6EPEJEAWBY3BKUL5R5PE64MKEFQP66BNEONVT7V4KJUTLA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669172146 (708h25m59.540255191s) [noise=true , address=145.79.227.35:4491 , piece_id=4TWONEFVZXBUNUOZN7ZF7653ZMNMBLKDADBQZGQHE6AKGG7JK2EA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669452179 (708h25m59.540535214s) [noise=true , address=77.90.31.16:4743 , piece_id=24YEQADNWVQHVMTPJ63HFYWDRCSESE6FFQHNN32P2SSDRQ6F62GA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669669793 (708h25m59.540752838s) [noise=true , address=145.79.227.7:4297 , piece_id=RNQGX6GRG6GL3EM22VKPY3BYEARINRKLK3PDGFXZAKAHHFH6JBGQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.669994171 (708h25m59.541077216s) [noise=true , address=45.91.116.13:4163 , piece_id=I3AH64TGSZBQCM6XT6PYJEDEAWZOLUI35HGAQNXT44QZN6OJFMNQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.670368083 (708h25m59.541451118s) [noise=true , address=145.79.227.29:4956 , piece_id=HZBIKPID75T7CMNJHMGKDMEDRE3ZD25ZFGVQSXKLCV34JTEVPEWQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.670441282 (708h25m59.541524327s) [noise=true , address=77.90.31.21:4882 , piece_id=QVNUDOIIX3FC7CAYVW5ALQTTS6J7YUEK5KN7WYWYAPZFMTFOFKUQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.670914534 (708h25m59.541997579s) [noise=true , address=145.79.227.14:4870 , piece_id=M4GIOL4H5QKXWQAE3HKIT4N7G5UAFTZ7IOQSP65OICB54CIHR5YA, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.671405448 (708h25m59.542488493s) [noise=true , address=77.90.31.11:4574 , piece_id=5CXKPREM3WEHOV653WAPKLQ3O5RC6L4P22BRUJDYPYIHL55ZMYSQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:54:59.671901012 (708h25m59.542984057s) [noise=true , address=77.90.31.13:4656 , piece_id=PYJGURNUIHXBRYHQTVJF4VD324YU56EOH55S5KBFCIWGAVZKLTJQ, seg_pos={0 0}, upload=620232]: Uploading piece...
Jan 4 03:55:01.109274522 (708h26m0.980357567s) [upload=620232]: Next returned nil segment
Jan 4 03:55:01.109294080 (708h26m0.980377125s) [upload=620232]: Waiting for error group managing segments...
Jan 4 03:55:01.490405950 (708h26m1.361488996s) [noise=true , address=64.49.13.155:4917 , piece_id=WBHQFRLYUDRRU4QZHHG7ACHNSDR2KWEKSUCNZFJFTZTOLCNHULNA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.543575672 (708h26m1.414658718s) [noise=true , address=64.49.13.170:4722 , piece_id=SNPTYRFXTVQFSP6ODBJSNPSVQCWUWRDNE2HDSLJZH6QEJ7SBTDHQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.543660865 (708h26m1.4147439s) [noise=true , address=145.79.227.7:4297 , piece_id=RNQGX6GRG6GL3EM22VKPY3BYEARINRKLK3PDGFXZAKAHHFH6JBGQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.587030196 (708h26m1.458113241s) [noise=true , address=145.79.227.27:4715 , piece_id=TLBIIAGXYIWYY4K4URWOGCZI4CC4FMREEVG5PCK44YBUKWCNKR5Q, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.598081249 (708h26m1.469164304s) [noise=true , address=145.79.227.38:4002 , piece_id=ZSTKF43VUA6QT3RI436BUR3HDY5MVK4HEIPC3Y66YVRRI6MZZLSQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.630902966 (708h26m1.501986011s) [noise=true , address=145.79.227.14:4870 , piece_id=M4GIOL4H5QKXWQAE3HKIT4N7G5UAFTZ7IOQSP65OICB54CIHR5YA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.796424984 (708h26m1.667508039s) [noise=true , address=77.90.31.13:4656 , piece_id=PYJGURNUIHXBRYHQTVJF4VD324YU56EOH55S5KBFCIWGAVZKLTJQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:01.907831917 (708h26m1.778914962s) [noise=true , address=77.90.31.21:4882 , piece_id=QVNUDOIIX3FC7CAYVW5ALQTTS6J7YUEK5KN7WYWYAPZFMTFOFKUQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.178142516 (708h26m2.049225571s) [noise=true , address=45.91.116.10:4113 , piece_id=PPJZYL6EPEJEAWBY3BKUL5R5PE64MKEFQP66BNEONVT7V4KJUTLA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.481641152 (708h26m2.352724197s) [noise=true , address=145.79.227.49:4355 , piece_id=SBWATWUIHAUSGJLX2IUZUSSGBFNR6JGZNG5RW7F5VDEN2YADJQLA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.589048459 (708h26m2.460131514s) [noise=true , address=145.79.227.35:4491 , piece_id=4TWONEFVZXBUNUOZN7ZF7653ZMNMBLKDADBQZGQHE6AKGG7JK2EA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.600182240 (708h26m2.471265285s) [noise=true , address=145.79.227.32:4868 , piece_id=TBI4X5ODKX7RHDDI7PPMDU4RRSX3AH7ITM5AVA3ONJVYWO57GIJQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.692843210 (708h26m2.563926275s) [noise=true , address=77.90.31.16:4743 , piece_id=24YEQADNWVQHVMTPJ63HFYWDRCSESE6FFQHNN32P2SSDRQ6F62GA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.772503363 (708h26m2.643586408s) [noise=true , address=145.79.227.30:4790 , piece_id=ERARTNVKLOQUMXLI62UZN4F22NRSBAQGXYVWKWWJR4GJDJVZAN3A, seg_pos={0 0}, upload=620232]: Done uploading piece. err: <nil>
Jan 4 03:55:02.772830066 (708h26m2.643913111s) [seg_pos={0 0}, upload=620232]: Segment reached optimal threshold of 14 pieces.
Jan 4 03:55:02.773221080 (708h26m2.644304155s) [noise=true , address=145.79.227.29:4956 , piece_id=HZBIKPID75T7CMNJHMGKDMEDRE3ZD25ZFGVQSXKLCV34JTEVPEWQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pf65Tg83xM523bgdJK2BxiC6uzJ1Hh528DY8CUDxGTnxZSd2W): protocol: expected piece hash; context canceled; EOF
Jan 4 03:55:02.773233092 (708h26m2.644316148s) [noise=true , address=145.79.227.29:4956 , piece_id=HZBIKPID75T7CMNJHMGKDMEDRE3ZD25ZFGVQSXKLCV34JTEVPEWQ, seg_pos={0 0}, upload=620232]: Long tail cancellation due to optimal threshold
Jan 4 03:55:02.773631662 (708h26m2.644714717s) [noise=true , address=145.79.227.43:4170 , piece_id=SLCRKGUE53KQD7LINKLR6OSJOAIJYB27O6HIXBABKI5PD2TRBRVA, seg_pos={0 0}, upload=620232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: expected piece hash; context canceled; EOF
Jan 4 03:55:02.773643935 (708h26m2.64472698s) [noise=true , address=145.79.227.43:4170 , piece_id=SLCRKGUE53KQD7LINKLR6OSJOAIJYB27O6HIXBABKI5PD2TRBRVA, seg_pos={0 0}, upload=620232]: Long tail cancellation due to optimal threshold
Jan 4 03:55:02.774162292 (708h26m2.645245357s) [noise=true , address=45.91.116.13:4163 , piece_id=I3AH64TGSZBQCM6XT6PYJEDEAWZOLUI35HGAQNXT44QZN6OJFMNQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRpjHEvHzBoYqq6DXKC5Z3SPsP4HEZz2tAiuheHkgzzcnsVU2): protocol: expected piece hash; context canceled; EOF
Jan 4 03:55:02.774173934 (708h26m2.645256989s) [noise=true , address=45.91.116.13:4163 , piece_id=I3AH64TGSZBQCM6XT6PYJEDEAWZOLUI35HGAQNXT44QZN6OJFMNQ, seg_pos={0 0}, upload=620232]: Long tail cancellation due to optimal threshold
Jan 4 03:55:02.775108143 (708h26m2.646191188s) [noise=true , address=77.90.31.11:4574 , piece_id=5CXKPREM3WEHOV653WAPKLQ3O5RC6L4P22BRUJDYPYIHL55ZMYSQ, seg_pos={0 0}, upload=620232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF
Jan 4 03:55:02.775155975 (708h26m2.64623903s) [noise=true , address=77.90.31.11:4574 , piece_id=5CXKPREM3WEHOV653WAPKLQ3O5RC6L4P22BRUJDYPYIHL55ZMYSQ, seg_pos={0 0}, upload=620232]: Long tail cancellation due to optimal threshold
Jan 4 03:55:02.775226549 (708h26m2.646309594s) [seg_pos={0 0}, upload=620232]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 4 03:55:02.775564712 (708h26m2.646647757s) [upload=620232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 03:55:02.775592576 (708h26m2.646675621s) [upload=620232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 03:55:03.076544249 (708h26m2.947627304s) [upload=620232]: Done uploading segments...
Jan 4 03:55:03.076573285 (708h26m2.94765633s) [upload=620232]: Upload finished. err: <nil>
Duration: 1m0.003375499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
18.292532094s trace:
https://jaeger.dev.storj.io/trace/1ee36f24231792d5
FAIL (2026-01-04 03:55:28 UTC)
orbiter warning: context deadline exceeded
Duration: 18.292627265s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [...XXXX..X.XXXXXXXX..XXXXX.XXXX.XXXXXX...X.XXX.XX.X.XXXXXXXX] duration:
5.001547551s trace:
https://jaeger.dev.storj.io/trace/1971aaf7f95d0ac5
FAIL (2026-01-04 03:55:27 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001567729s
core
uplink
us-select-1
us1-placement-20
|