Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.03312521s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 18:30:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 6.033165157s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....XX........X.............................................] duration: 3.575621601s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:29:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5756527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.665705092s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 18:29:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.665735378s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....XX........X............................................] duration: 3.485540494s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:28:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485571654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.825172779s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 18:28:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.825192037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002971278s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 18:27:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 30.002986957s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......XX........X...........................................] duration: 3.485810704s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:27:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485833146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X......XX........X..........................................] duration: 3.554981286s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:26:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555006884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X......XX........X.........................................] duration: 3.486619092s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:26:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486650903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X......XX........X........................................] duration: 3.523963775s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:25:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523991278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X......XX........X.......................................] duration: 3.402742358s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:25:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.402784388s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X...X......XX........X......................................] duration: 3.63544304s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:24:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.635471325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X...X......XX........X.....................................] duration: 3.465419095s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:23:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.465450904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X...X......XX........X....................................] duration: 3.453352448s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:22:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.453407482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X...X......XX........X...................................] duration: 3.651789206s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:22:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.651881332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X...X......XX........X..................................] duration: 3.598667791s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:21:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.598697699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....X...X......XX........X.................................] duration: 3.462910127s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:20:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463054462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.218925234s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:20:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.218950121s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......X...X......XX........X................................] duration: 3.597309267s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:20:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.597367477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.138759796s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:19:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.138796956s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......X...X......XX........X...............................] duration: 3.475342912s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:19:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475368851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.170251781s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:19:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.170280114s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X...X......XX........X..............................] duration: 3.64553074s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:18:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.64557829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.........X...X......XX........X.............................] duration: 3.605408578s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:18:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.605439236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.195710961s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:18:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.195742201s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........X...X......XX........X............................] duration: 3.513415811s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:17:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513500322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.237766071s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:17:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.237790398s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...........X...X......XX........X...........................] duration: 3.55518209s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:17:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555210315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.281440666s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:16:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.281464259s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [............X...X......XX........X..........................] duration: 3.373486067s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:16:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.373627986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.148863195s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:16:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.148892299s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.............X...X......XX........X.........................] duration: 4.363158381s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:15:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.363183709s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.317071774s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:14:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.317092814s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.209918456s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:14:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.210185163s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............X...X......XX........X........................] duration: 3.559008726s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:14:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559031421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.790177029s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:13:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.790206656s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............X...X......XX........X.......................] duration: 3.553304705s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:13:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553335773s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.299409281s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:13:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.299437204s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.714325328s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:13:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.71435752s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.818858676s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:12:44 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 1.81888675s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X...X......XX........X......................] duration: 3.66358609s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:12:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.663612701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.566675323s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:12:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.566719557s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.828915219s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:12:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.828947098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.720589459s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:11:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.720611033s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.32831239s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:11:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.328356102s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................X...X......XX........X.....................] duration: 3.675329483s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:11:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.675355832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.207976086s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:10:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.208007285s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.750084629s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:10:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.750111962s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................X...X......XX........X....................] duration: 3.545943368s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:10:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546058857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................X...X......XX........X...................] duration: 3.677017647s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:10:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.677048766s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)